YT YouTube2Text

YouTube2Text → Transcripts → Learn RAG From Scratch – Python AI Tutorial from a LangChain Engineer

Learn RAG From Scratch – Python AI Tutorial from a LangChain Engineer — Transcript

by freeCodeCamp.org · 27,073 words · 3,910 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00in this course Lance Martin will teach
0:02you how to implement rag from scratch
0:06Lance is a software engineer at Lang
0:08chain and Lang chain is one of the most
0:10common ways to implement rag Lance will
0:14help you understand how to use rag to
0:16combine custom data with llms hi this is
0:19Lance Martin I'm a software engineer at
0:21Lang chain I'm going to be giving a
0:23short course focused on rag or retrieval
0:26augmented generation which is one of the
0:28most popular kind of ideas and
0:30in llms
0:32today so really the motivation for this
0:34is that most of the world's data is
0:37private um whereas llms are trained on
0:41publicly available data so you can kind
0:43of see on the bottom on the x-axis the
0:45number of tokens using pre-training
0:47various llms so it kind of varies from
0:50say 1.5 trillion tokens in the case of
0:52smaller models like
0:5452 out to some very large number that we
0:56actually don't know for proprietary
0:58models like GPT 4 CLA
1:00three but what's really interesting is
1:03that the context window or the ability
1:06to feed external information into these
1:08LMS is actually getting larger so about
1:11a year ago context windows were between
1:134 and 8,000 tokens you know that's like
1:16maybe a dozen pages of text we've
1:18recently seen models all the way out to
1:20a million tokens which is thousands of
1:22pages of text so while these llms are
1:24trained on large scale public data it's
1:28increasingly feasible to feed them
1:30this huge mass of private data that
1:33they've never seen that private data can
1:35be your kind of personal data it can be
1:37corporate data or you know other
1:40information that you want to pass to an
1:42LM that's not natively in his training
1:45set and so this is kind of the main
1:48motivation for rag it's really the idea
1:52that llms one are kind of the the center
1:54of a new kind of operating
1:56system and two it's increasingly
1:59critical to be able to feed information
2:01from external sources such as private
2:04data into llms for processing so that's
2:07kind of the overarching motivation for
2:10Rag and now rag refers to retrieval
2:13augmented generation and you can think
2:14of it in three very general steps
2:17there's a process of indexing of
2:19external data so you can think about
2:22this as you know building a database for
2:24example um many companies already have
2:27large scale databases in different forms
2:29they could be SQL DBS relational DBS um
2:32they could be Vector Stores um or
2:35otherwise but the point is that
2:38documents are indexed such that they can
2:40be retrieved based upon some heuristics
2:43relative to an input like a question and
2:45those relevant documents can be passed
2:47to an llm and the llm can produce
2:50answers that are grounded in that
2:52retrieved information so that's kind of
2:54the centerpiece or central idea behind
2:56Rag and why it's really powerful
2:58technology because it's really uniting
3:01the the knowledge and processing
3:03capacity of llms with large scale
3:06private external data source for which
3:09most of the important data in the world
3:11still
3:13lives and in the following short videos
3:16we're going to kind of build up a
3:17complete understanding of the rag
3:20landscape and we're going to be covering
3:22a bunch of interesting papers and
3:24techniques that explain kind of how to
3:25do rag and I've really broken it down
3:28into a few different sections
3:30so starting with a question on the left
3:32the first kind of section is what I call
3:35query trans translation so this captures
3:37a bunch of different methods to take a
3:39question from a user and modify it in
3:42some way to make it better suited for
3:44retrieval from you know one of these
3:45indexes we've talked
3:47about that can use methods like query
3:50writing it can be decomposing the query
3:52into you know constituent sub
3:55questions then there's a question of
3:57routing so taking that decomposed a
4:00Rewritten question and routing it to the
4:02right place you might have multiple
4:03Vector stores a relational DB graph DB
4:06and a vector store so it's the challenge
4:08of getting a question to the right
4:10Source then there's a there's kind of
4:12the challenge of query construction
4:15which is basically taking natural
4:16language and converting it into the DSL
4:19necessary for whatever data source you
4:21want to work with a classic example here
4:23is text a SQL which is kind of a very
4:25kind of well studied process but text a
4:28cipher for graph DV is very interesting
4:31text to metadata filters for Vector DBS
4:34is also a very big area of
4:36study um then there's indexing so that's
4:40the process of taking your documents and
4:42processing them in some way so they can
4:44be easily retrieved and there's a bunch
4:46of techniques for that we'll talk
4:47through we'll talk through different
4:48embedding methods we'll talk about
4:50different indexing
4:51strategies after
4:53retrieval there are different techniques
4:56to rerank or filter retrieve documents
4:59um and then finally we'll talk about
5:00generation and kind of an interesting
5:02new set of methods to do what we might
5:05call as active rag so in that retrieval
5:08or generation stage grade documents
5:12grade answers um grade for relevance to
5:15the question grade for faithfulness to
5:18the documents I.E check for
5:19hallucinations and if either fail
5:22feedback uh re- retrieve or rewrite the
5:25question uh regenerate the qu regenerate
5:28the answer and so forth so there's a
5:30really interesting set of methods we're
5:31going to talk through that cover that
5:33like retrieval and generation with
5:37feedback and you know in terms of
5:39General outline we'll cover the basics
5:41first it'll go through indexing
5:43retrieval and generation kind of in the
5:44Bare Bones and then we'll talk through
5:47more advanced techniques that we just
5:48saw on the prior slide career
5:50Transformations routing uh construction
5:52and so forth hi this is Lance from Lang
5:55chain this the second video in our
5:57series rack from scratch focused on
5:59indexing
6:01so in the past video you saw the main
6:04kind of overall components of rag
6:06pipelines indexing retrieval and
6:09generation and here we're going to kind
6:10of Deep dive on indexing and give like
6:13just a quick overview of it so the first
6:16aspect of indexing is we have some
6:18external documents that we actually want
6:20to load and put into what we're trying
6:22to call Retriever and the goal of this
6:25retriever is simply given an input
6:27question I want to fish out doents that
6:30are related to my question in some
6:32way now the way to establish that
6:35relationship or relevance or similarity
6:37is typically done using some kind of
6:39numerical representation of documents
6:42and the reason is that it's very easy to
6:44compare vectors for example of numbers
6:47uh relative to you know just free form
6:50text and so a lot of approaches have
6:53been a developed over the years to take
6:56text documents and compress them down
6:58into a numerical rep presentation that
7:01then can be very easily
7:03searched now there's a few ways to do
7:06that so Google and others came up with
7:08many interesting statistical methods
7:11where you take a document you look at
7:13the frequency of words and you build
7:15what they call sparse vectors such that
7:18the vector locations are you know a
7:20large vocabulary of possible words each
7:23value represents the number of
7:24occurrences of that particular word and
7:27it's sparse because there's of course
7:28many zeros it's a very large vocabulary
7:31relative to what's present in the
7:32document and there's very good search
7:34methods over this this type of numerical
7:37representation now a bit more recently
7:40uh embedding methods that are machine
7:41learned so you take a document and you
7:43build a compressed fixed length
7:45representation of that
7:47document um have been developed with
7:51correspondingly very strong search
7:53methods over
7:54embeddings um so the intuition here is
7:59that we take documents and we typically
8:01split them because embedding models
8:04actually have limited context windows so
8:07you know on the order of maybe 512
8:09tokens up to 8,000 tokens or Beyond but
8:12they're not infinitely large so
8:14documents are split and each document is
8:16compressed into a vector and that Vector
8:20captures a semantic meaning of the
8:22document itself the vectors are indexed
8:25questions can be embedded in the exactly
8:27same way and then numerical kind of
8:30comparison in some form you know using
8:33very different types of methods can be
8:34performed on these vectors to fish out
8:37relevant documents relative to my
8:41question um and let's just do a quick
8:43code walk through on some of these
8:46points so I have my notebook here I've
8:51installed here um now I've set a few API
8:55keys for lsmith which are very useful
8:58for tracing which we'll see
9:00shortly um previously I walked through
9:03this this kind of quick start that just
9:04showed overall how to lay out these rag
9:07pipelines and here what I'll do is I'll
9:09Deep dive a little bit more on indexing
9:11and I'm going to take a question and a
9:13document and first I'm just going to
9:16compute the number of tokens in for
9:18example the question and this is
9:19interesting because embedding models in
9:22llms more generally operate on tokens
9:24and so it's kind of nice to understand
9:27how large the documents are that I'm
9:28trying to feed in in this case it's
9:30obviously a very small in this case
9:33question now I'm going to specify open
9:35eye embeddings I specify an embedding
9:38model here and I just say embed embed
9:40query I can pass my question my document
9:43and what you can see here is that runs
9:47and this is mapped to now a vector of
9:49length 1536 and that fixed length Vector
9:53representation will be computed for both
9:56documents and really for any document so
9:58you're always is kind of computing this
10:00fix length Vector that encodes the
10:02semantics of the text that you've passed
10:05now I can do things like cosine
10:06similarity to compare
10:08them and as we'll see here I can load
10:13some documents this is just like we saw
10:16previously I can split
10:18them and I can index them here just like
10:22we did before but we can see under the
10:23hood really what we're doing is we're
10:25taking each split we're embedding it
10:27using open eye embeddings into this this
10:29kind of this Vector representation and
10:31that's stored with a link to the rod
10:33document itself in our Vector store and
10:36next we'll see how to actually do
10:38retrieval using this Vector store hi
10:41this is Lance from Lang chain and this
10:43is the third video in our series rag
10:45from scratch building up a lot of the
10:47motivations for rag uh from the very
10:50basic
10:51components um so we're going to be
10:54talking about retrieval today in the
10:56last two uh short videos I outlined
10:59indexing and gave kind of an overview of
11:01this flow which starts with indexing of
11:04our documents retrieval of documents
11:06relevant to our question and then
11:07generation of answers based on the
11:09retriev
11:11documents and so we saw that the
11:13indexing process basically makes
11:15documents easy to retrieve and it goes
11:18through a flow that basically looks like
11:20you take our documents you split them in
11:22some way into these smaller chunks that
11:25can be easily embedded um those
11:28embeddings are then numerical
11:30representations of those documents that
11:31are easily
11:32searchable and they're stored in an
11:35index when given a question that's also
11:38embedded the index performs a similarity
11:41search and returns splits that are
11:43relevant to the
11:45question now if we dig a little bit more
11:47under the hood we can think about it
11:49like this if we take a document and
11:52embed it let's imagine that embedding
11:54just had three dimensions so you know
11:56each document is projected into some
11:58point in this 3D
12:00space now the point is that the location
12:03in space is determined by the semantic
12:06meaning or content in that document so
12:09to follow that then documents in similar
12:12locations in space contain similar
12:15semantic information and this very
12:17simple idea is really the Cornerstone
12:19for a lot of search and retrieval
12:21methods that you'll see with modern
12:22Vector stores so in particular we take
12:25our documents we embed them into this in
12:27this case a toy 3D space
12:30we take our question do the
12:32same we can then do a search like a
12:35local neighborhood search you can think
12:37about in this 3D space around our
12:39question to say hey what documents are
12:42nearby and these nearby neighbors are
12:45then retrieved because they can they
12:47have similar semantics relative to our
12:50question and that's really what's going
12:53on here so again we took our documents
12:56we split them we embed them and now they
12:58exist in this high dimensional space
13:00we've taken our question embedded it
13:02projected in that same space and we just
13:04do a search around the question from
13:07nearby documents and grab ones that are
13:09close and we can pick some number we can
13:12say we want one or two or three or n
13:14documents close to my question in this
13:17embedding space and there's a lot of
13:19really interesting methods that
13:20implement this very effectively I I link
13:22one
13:24here um and we have a lot of really nice
13:28uh Integrations to play with this
13:30general idea so many different embedding
13:32models many different indexes lots of
13:35document loaders um and lots of
13:37Splitters that can be kind of recombined
13:39to test different ways of doing this
13:41kind of indexing or
13:43retrieval um so now I'll show a bit of a
13:45code walkth through so here we defined
13:50um we kind of had walked through this
13:52previously this is our notebook we've
13:54installed a few packages we've set a few
13:57environment variables using lsmith
14:00and we showed this previously this is
14:02just an overview showing how to run rag
14:04like kind of end to end in the last uh
14:07short talk we went through
14:09indexing um and what I'm going to do
14:11very simply is I'm just going to reload
14:14our
14:15documents so now I have our documents
14:19I'm going to resplit
14:21them and we saw before how we can build
14:23our
14:24index now here let's actually do the
14:27same thing but in the slide we actually
14:29showed kind of that notion of search in
14:31that 3D
14:33space and a nice parameter to think
14:35about in building your your retriever is
14:38K so K tells you the number of nearby
14:41neighbors to fetch when you do that
14:42retrieval process and we talked about
14:44you know in that 3D space do I want one
14:47nearby neighbor or two or three so here
14:49we can specify k equals 1 for example
14:53now we're building our index so we're
14:54taking every split embedding it storing
14:57it now what's nice is I asked a a
14:59question what is Task decomposition this
15:01is related to the blog post and I'm
15:03going to run get relevant documents so I
15:05run that and now how many documents do I
15:08get back I get one as expected based
15:10upon k equals 1 so this retrieve
15:12document should be related to my
15:14question now I can go to lsmith and we
15:17can open it up and we can look at our
15:19Retriever and we can see here was our
15:21question here's the one document we got
15:23back and okay so that makes sense this
15:27document pertains to task ke
15:29decomposition in particular and it kind
15:31of lays out a number of different
15:33approaches that can be used to do that
15:35this all kind of makes sense and this
15:37shows kind of in practice how you can
15:38implement this this NE this kind of KNN
15:42or k nearest neighbor search uh really
15:45easily uh just using a few lines of code
15:48and next we're going to talk about
15:50generation
15:52thanks hey this is Lance from Lang chain
15:55this is the fourth uh short video in our
15:57rack from scratch series
15:59that's going to be focused on
16:01generation now in the past few videos we
16:04walked through the general flow uh for
16:07kind of basic rag starting with indexing
16:10Fall by
16:11retrieval then
16:13generation of an answer based upon the
16:15documents that we retrieved that are
16:17relevant to our question this is kind of
16:19the the very basic
16:21flow now an important consideration in
16:25generation is really what's happening is
16:28we're taking the documents you retrieve
16:30and we're stuffing them into the llm
16:32context window so if we kind of walk
16:34back through the process we take
16:37documents we split them for convenience
16:39or embedding we then embed each split
16:43and we store that in a vector store as
16:45this kind of easily searchable numerical
16:47representation or vector and we take a
16:50question embed it to produce a similar
16:52kind of numerical representation we can
16:55then search for example using something
16:57like KN andn in this kind of dimensional
17:00space for documents that are similar to
17:02our question based on their proximity or
17:05location in this space in this case you
17:07can see 3D is a toy kind of toy
17:10example now we've recovered relevant
17:12splits to our question we pack those
17:15into the context window and we produce
17:17our
17:19answer now this introduces the notion of
17:22a prompt so the prompt is kind of a you
17:25can think have a placeholder that has
17:27for example you know in our case B keys
17:30so those keys can be like context and
17:33question so they basically are like
17:35buckets that we're going to take those
17:37retrieve documents and Slot them in
17:40we're going to take our question and
17:41also slot it in and if you kind of walk
17:44through this flow you can kind of see
17:45that we can build like a dictionary from
17:48our retrieve documents and from our
17:50question and then we can basically
17:52populate our prompt template with the
17:54values from the dict and then becomes a
17:57prompt value which can be passed to llm
17:59like a chat model resulting in chat
18:01messages which we then parse into a
18:03string and get our answer so that's like
18:06the basic workflow that we're going to
18:07see and let's just walk through that in
18:09code very quickly to kind of give you
18:11like a Hands-On intuition so we had our
18:14notebook we walk through previously
18:16install a few packages I'm setting a few
18:19lsmith environment variables we'll see
18:21it's it's nice for uh kind of observing
18:23and debugging our
18:25traces um previously we did this quick
18:27start we're going to skip that over
18:30um and what I will do is I'm going to
18:33build our retriever so again I'm going
18:36to take documents and load them uh and
18:38then I'm going to split them here we've
18:40kind of done this previously so I'll go
18:42through this kind of quickly and then
18:43we're going to embed them and store them
18:45in our index so now we have this
18:47retriever object here now I'm going to
18:50jump down here now here's where it's
18:51kind of fun this is the generation bit
18:54and you can see here I'm defining
18:55something new this is a prompt template
18:58and what my prompt template is something
18:59really simple it's just going to say
19:00answer the following question based on
19:02this context it's going to have this
19:04context variable and a question so now
19:06I'm building my prompt so great now I
19:08have this prompt let's define an llm
19:11I'll choose
19:1235 now this introdu the notion of a
19:15chain so in Lang chain we have an
19:17expression language called L Cel Lang
19:20chain expression language which lets you
19:22really easily compose things like
19:24prompts LMS parsers retrievers and other
19:27things but the very simple kind of you
19:30know example here is just let's just
19:32take our prompt which you defined right
19:33here and connect it to an LM which you
19:35defined right here into this chain so
19:37there's our chain now all we're doing is
19:39we're invoking that chain so every L
19:42expression language chain has a few
19:44common methods like invoke bat stream in
19:47this case we just invoke it with a dict
19:50so context and question that maps to the
19:54expected Keys here in our template
19:58and so if we run invoke what we see is
20:01it's just going to execute that chain
20:02and we get our answer now if we zoom
20:05over to Langs Smith we should see that
20:07it's been populated so yeah we see a
20:09very simple runable
20:11sequence here was our
20:14document um and here's our output and
20:18here is our prompt answer the following
20:21question based on the context here's the
20:24document we passed in here is the
20:26question and then we get our answer so
20:28that's pretty nice um now there's a lot
20:32of other options for rag prompts I'll
20:34pull one in from our prompt tub this
20:36one's like kind of a popular prompt so
20:39it just like has a little bit more
20:41detail but you know it's the main the
20:43main intuition is the same um you're
20:47passing in documents you're asking them
20:48to reason about the documents given a
20:50question produce an answer and now here
20:53I'm going to find a rag chain which will
20:55automatically do the retrieval for us
20:57and all I have to do is specify here's
20:59my retriever which we defined
21:01before here's our question we which we
21:03invoke with the question gets passed
21:06through to the key question in our dict
21:10and it automatically will trigger the
21:12retriever which will return documents
21:14which get passed into our context so
21:16it's exactly what we did up here except
21:18before we did this
21:20manually and
21:22now um this is all kind of automated for
21:25us we pass that dick which is autop
21:28populated
21:29into our prompt llm out to parser now
21:31let invoke it and that should all just
21:34run and great we get an answer and we
21:37can look at the
21:39trace and we can see everything that
21:41happened so we can see our retriever was
21:44run these documents were
21:46retrieved they get passed into our
21:49LM and we get our final answer so this
21:53kind of the end of our overview um where
21:56we talked about I'll go back to the
21:58slide here quickly we talked about
22:00indexing retrieval and now
22:02generation and follow-up short videos
22:04we'll kind of dig into some of the more
22:06com complex or detailed themes that
22:09address some limitations that can arise
22:11in this very simple pipeline
22:14thanks hi my from Lang chain over the
22:17next few videos we're going to be
22:18talking about career
22:20translation um and in this first video
22:23we're going to cover the topic of
22:24multi-query
22:26so query translation sits kind of at the
22:29first stage of an advanced rag Pipeline
22:34and the goal of career translation is
22:36really to take an input user question
22:39and to translate in some way in order to
22:42improve
22:44retrieval so the problem statement is
22:47pretty intuitive user queries um can be
22:51ambiguous and if the query is poorly
22:53written because we're typically doing
22:56some kind of semantic similarity search
22:57between the query and our documents if
23:01the query is poorly written or ill
23:02opposed we won't retrieve the proper
23:05documents from our
23:07index so there's a few approaches to
23:10attack this problem and you can kind of
23:13group them in a few different ways so
23:15here's one way I like to think about it
23:17a few approaches has involveed query
23:19rewriting so taking a query and
23:22reframing it like writing from a
23:23different perspective um and that's what
23:26we're going to talk about a little bit
23:27here in depth using approaches like
23:29multi-query or rag Fusion which we'll
23:32talk about in the next video you can
23:34also do things like take a question and
23:36break it down to make it less abstract
23:38like into sub questions and there's a
23:40bunch of interesting papers focused on
23:42that like least to most from
23:44Google you can also take the opposite
23:46approach of take a question to make it
23:48more abstract uh and there's actually
23:50approach we're going to talk about later
23:51in a future video called stepback
23:53prompting that focuses on like kind of
23:56higher a higher level question from the
23:59input so the intuition though for this
24:03multier approach is we're taking a
24:05question and we're going to break it
24:06down into a few differently worded
24:08questions uh from different
24:10perspectives and the intuition here is
24:13simply that um it is possible that the
24:17way a question is initially worded once
24:21embedded it is not well aligned or in
24:24close proximity in this High dimensional
24:26embedding space to a document that we
24:27want to R that's actually related so the
24:30thinking is that by kind of rewriting it
24:32in a few different ways you actually
24:34increase the likel of actually
24:36retrieving the document that you really
24:37want to um because of nuances in the way
24:41that documents and questions are
24:43embedded this kind of more shotgun
24:46approach of taking a question Fanning it
24:47out into a few different perspectives
24:50May improve and increase the reliability
24:51of retrieval that's like the intuition
24:54really um and of course we can com
24:56combine this with retrieval so we can
24:59take our our kind of fan out questions
25:01do retrieval on each one and combine
25:03them in some way and perform rag so
25:06that's kind of the overview and now
25:07let's what let's go over to um our code
25:11so this is a notebook and we're going to
25:13share all
25:14this um we're just installing a few
25:18packages we're setting a lsmith API Keys
25:21which we'll see why that's quite useful
25:22here shortly there's our diagram now
25:25first I'm going to Index this blog post
25:27on agents I'm going to split it um well
25:31I'm going to load it I'm going to split
25:32it and then I'm going to index it in
25:34chroma locally so this is a vector store
25:37we've done this previously so now I have
25:38my index defined so here is where I'm
25:41defining my prompt for multiquery which
25:44is your your assistant your task is to
25:47basically reframe this question into a
25:48few different sub
25:50questions um so there's our
25:53prompt um right here we'll pass that to
25:56an llm part it um into a string and then
26:01split the string by new lines and so
26:02we'll get a list of questions out of
26:04this chain that's really all we're doing
26:06here now all we're doing is here's a
26:09sample input question there's our
26:11generate queries chain which we defined
26:13we're going to take that list and then
26:15simply apply each question to retriever
26:19so we'll do retrieval per question and
26:21this little function here is just going
26:22to take the unique Union of documents uh
26:25across all those retrievals so let's run
26:27this and see what happens so we're going
26:29to run this and we're going to get some
26:32set of questions uh or documents back so
26:36let's go to Langs Smith now we can
26:37actually see what happened under the
26:39hood so here's the key
26:42point we ran our initial chain to
26:44generate a set of of reframed questions
26:48from our input and here was that prompt
26:51and here is that set of questions that
26:52we generated now what happened is for
26:55every one of those questions we did an
26:57independent retrieval that's what we're
26:58showing here so that's kind of the first
27:00step which is great now I can go back to
27:03the notebook and we can show this
27:05working end to end so now we're going to
27:07take that retrieval chain we'll pass it
27:09into context of our final rag prompt
27:12we'll also pass through the question
27:14we'll pass that to our rag prompt here
27:16pass it to an LM and then Pary output
27:19now let's let's kind of see how that
27:21works so again that's okay there it is
27:24so let's actually go into langth and see
27:26what happened under the hood so this was
27:28our final chain so this is great we took
27:31our input question we broke it out to
27:33these like five rephrase questions for
27:36every one of those we did a retrieval
27:38that's all great we then took the unique
27:41Union of documents and you can see in
27:42our final llm prompt answer the
27:45following cont following question based
27:46on the context this is the final set of
27:50unique documents that we retrieved from
27:52all of our sub
27:54questions um here's our initial question
27:57there's our answer so that kind of shows
27:59you how you can set this up really
28:00easily how you can use l Smith to kind
28:02of investigate what's going on and in
28:04particular use l Smith to investigate
28:06those intermediate questions that you
28:08generate in that like kind of question
28:11generation phase and in a future talks
28:13we're going to go through um some of
28:15these other methods that we kind of
28:17introduced at the start of this one
28:18thank
28:20you last L chain this is the second
28:23video of our Deep dive on query
28:25translation in our rag from scratch
28:27series focused on a method called rag
28:30Fusion so as we kind of showed before
28:33career translation you can think of as
28:35the first stage in an advanced rag
28:38pipeline we're taking an input user
28:40question and We're translating it some
28:42way in order to improve
28:44retrievable now we showed this General
28:48mapping of approaches previously so
28:50again you have kind of like rewriting so
28:52you can take a question and like kind of
28:54break it down into uh differently worded
28:58are different different perspectives of
29:00the same question so that's kind of
29:02rewriting there's sub questions where
29:04you take a question break it down into
29:06smaller problems solve each one
29:08independently and then there step back
29:10where you take a question and kind of go
29:11more abstract where you kind of ask a
29:14higher level question as a precondition
29:16to answer the user question so those are
29:18the approaches and we're going to dig
29:20into one of the particular approaches
29:22for rewriting called rat Fusion now this
29:25is really similar to what we just saw
29:27with multiquery
29:28the difference being we actually apply a
29:31a kind of a clever rank ranking step of
29:33our retriev documents um which you call
29:36reciprocal rank Fusion that's really the
29:38only difference the the input stage of
29:41taking a question breaking it out into a
29:44few kind of differently worded questions
29:47retrieval on each one is all the same
29:50and we're going to see that in the code
29:51here shortly so let's just hop over
29:54there and then look at this so again
29:56here is a notebook that we introduced
29:59previously here's the packages we've
30:01installed we've set a few API keys for
30:04lsmith which we see why is quite
30:07useful um and you can kind of go down
30:11here to a rag Fusion
30:13section and the first thing you'll note
30:16is what our prompt is so it looks really
30:18similar to The Prompt we just saw with
30:19multiquery and simply your helpful
30:22assistant that generates multiple search
30:23queries based upon user input and here's
30:26the question output for queries so let's
30:30define our prompt and here was our query
30:33Generation chain again this looks a lot
30:34like we just saw we take our prompt Plum
30:37that into an llm and then basically
30:39parse by new lines and that'll basically
30:42split out these questions into a list
30:46that's all it's going to happen here so
30:47that's cool now here's where the novelty
30:51comes
30:52in each time we do retrieval from one of
30:56those questions we're going to get back
30:58a list of documents from our Retriever
31:01and so we do it over that we generate
31:03four questions here based on our prompt
31:06we do the over four questions well like
31:07a list of lists
31:09basically now reciprocal rank Fusion is
31:12really well suited for this exact
31:13problem we want to take this list to
31:15list and build a single Consolidated
31:17list and really all that's going on is
31:20it's looking at the documents in each
31:22list and kind of aggregating them into a
31:24final output ranking um and that's
31:28really the intuition around what's
31:29happening
31:33here um so let's go ahead
31:39and so
31:46let's so let's go ahead and look at that
31:49in some detail so we can see we
31:53run
31:55retrieval that's great now let's go over
31:57to Lang Smith and have a look at what's
32:00going on here so we can see that here
32:05was our prompt to your helpful assistant
32:06that generates multiple search queries
32:08based on a single input and here is our
32:10search queries and then here are our
32:14four retrievals so that's that's really
32:16good so we know that all is
32:19working um and then those retrievals
32:23simply went into this rank
32:25function and our correspondingly ranked
32:29to a final list of six unique rank
32:31documents that's really all we
32:34did so let's actually put that all
32:37together into an a full rag chain that's
32:41going to run
32:43retrieval return that final list of rank
32:47documents and pass it to our context
32:50pass through our question send that to a
32:53rag prompt pass it to an LM parse it to
32:56an output and let's run all that
32:58together and see that
33:01working cool so there's our final
33:07answer now let's have a look in lsmith
33:10we can see here was our four questions
33:13here's our retrievals and then our final
33:15rag prompt plumed through the final list
33:18of ranked six questions which we can see
33:22laid out here and our final answer so
33:25this can be really convenient
33:26particularly if we're operating across
33:29like maybe different Vector stores uh or
33:32we want to do like retrieval across a
33:34large number of of kind of differently
33:36worded questions this reciprocal rank
33:38Fusion step is really nice um for
33:41example if we wanted to only take the
33:43top three documents or something um it
33:46can be really nice to build that
33:47Consolidated ranking across all these
33:49independent retrievals then pass that to
33:52for the final generation so that's
33:54really the intuition about what's
33:55happening here thanks
33:58hi this is Lance from Lang chain this is
34:00our third video focused on query
34:02translation in the rag from scratch
34:04series and we're going to be talking
34:06about
34:07decomposition so query translation in
34:09general is a set of approaches that sits
34:11kind of towards the front of this
34:13overall rag Pipeline and the objective
34:16is to modify or rewrite or otherwise
34:18decompose an input question from a user
34:21in order improve
34:24retrieval so we can talk through some of
34:26these approaches previously in
34:28particular various ways to do query
34:30writing like rag fusion and multiquery
34:32there's a separate set of techniques
34:34that become pretty popular and are
34:35really interesting for certain problems
34:37which we might call like kind of
34:39breaking down or decomposing an input
34:41question into a set of sub
34:43questions um so some of the papers here
34:46that are are pretty cool are for example
34:49this work from
34:50Google um and the objective really is
34:54first to take an input question and
34:57decompose it into a set of sub problems
35:00so this particular example from the
35:02paper was the problem of um last letter
35:07concatenation and so it took the inut
35:10question of three words think machine
35:13learning and broke it down into three
35:15sub problems think think machine think
35:17machine learning as the third sub
35:19problem and then you can see in this
35:20bottom panel it solves each one
35:23individually so it shows for example in
35:25green solving the problem think machine
35:28where you can catenate the last letter
35:30of k with the last letter of machine or
35:32last letter think K less machine e can
35:36concatenate those to K and then for the
35:39overall problem taking that solution and
35:43then and basically building on it to get
35:45the overall solution of keg so that's
35:47kind of one concept of decomposing into
35:50sub problems solving them
35:52sequentially now a related work called
35:55IRC or in leap retrieval combines
35:58retrieval with Chain of Thought
36:01reasoning and so you can kind of put
36:03these together into one approach which
36:05you can think of as kind of dynamically
36:08retrieval um to solve a set of sub
36:11problems kind of that retrieval kind of
36:13interleaving with Chain of Thought as
36:15noted in the second paper and a set of
36:19decomposed questions based on your
36:21initial question from the first work
36:23from Google so really the idea here is
36:25we're taking one sub question we're
36:28answering it we're taking that answer
36:29and using it to help answer the second
36:31sub question and so forth so let's
36:34actually just walk through this in code
36:35to show how this might
36:37work so this is The Notebook we've been
36:40working with from some of the other uh
36:42videos you can see we already have a
36:43retriever to find uh up here at the top
36:47and what we're going to do
36:49is we're first going to find a prompt
36:52that's basically going to say given an
36:54input question let's break it down to
36:57set of sub problems or sub question
36:59which can be solved individually so we
37:01can do that and this blog post is
37:03focused on agents so let's ask a
37:05question about what are the main
37:06components of an LM powerered autonomous
37:08agent system so let's run this and
37:13see what the decomposed questions are so
37:16you can see the decomposed questions are
37:18what is LM technology how does it work
37:21um what are components and then how the
37:23components interact so it's kind of a
37:25sane way to kind of break down this
37:27problem into a few sub problems which
37:28you might attack individually now here's
37:32where um we Define a prompt that very
37:35simply is going to take our question
37:38we'll take any prior questions we've
37:40answered and we'll take our retrieval
37:42and basically just combine them and we
37:45can Define this very simple
37:47chain um actually let's go back and make
37:49sure retriever is defined up at the
37:52top so now we are building our
37:56retriever good we have that now so we
37:59can go back down here and let's run this
38:02so
38:03now we are
38:06running and what's happening is we're
38:10trying to solve each of these questions
38:12individually using retrieval and using
38:15any prior question answers so okay very
38:18good looks like that's been done and we
38:20can see here's our answer now let's go
38:23over to langth and actually see what
38:25happened under the hood so here's what's
38:27kind of of interesting and helpful to
38:28see for the first question so here's our
38:31first one it looks like it just does
38:33retrieval which is we expect and then it
38:36uses that to answer this initial
38:37question now for the second question
38:40should be a little bit more interesting
38:42because if you look at our prompt here's
38:44our question now here is our background
38:47available question answer pair so this
38:49was the answer question answer pair from
38:51the first question which we add to our
38:52prompt and then here's the retrieval for
38:55this particular question so we're kind
38:56of building up up the solution because
38:58we're pending the question answer pair
39:00from question one and then likewise with
39:03question three it should combine all of
39:05that so we can look at here here's our
39:08question here's question one here's
39:10question two great now here's additional
39:14retrieval related to this particular
39:16question and we get our final answer so
39:18that's like a really nice way you can
39:20kind of build up Solutions um using this
39:23kind of
39:24interleaved uh retrieval and
39:26concatenating question answer pairs I do
39:29want to mention very briefly that we can
39:31also take a different approach where we
39:33can just answer these all individually
39:36and then just concatenate all those
39:37answers to produce a final answer and
39:39I'll show that really quickly here um
39:43it's like a little bit less interesting
39:44maybe because you're not using answers
39:46from each uh question to inform the next
39:50one you're just answering them all in
39:51parallel this might be better for cases
39:53where it's not really like a sub
39:55question decomposition but maybe it's
39:57like like a set of set of several in
40:00independent questions whose answers
40:02don't depend on each other that might be
40:04relevant for some
40:05problems um and we can go ahead and run
40:08okay so this ran as well we can look at
40:11our
40:12trace and in this
40:14case um yeah we can see that this
40:16actually just kind of concatenates all
40:19of our QA pairs to produce the final
40:22answer so this gives you a sense for how
40:24you can use quer decomposition employ
40:27IDE IDE from uh from two different
40:28papers that are pretty cool thanks hi
40:32this is Lance from Lang chain this is
40:34the fourth video uh in our Deep dive on
40:37queer translation in the rag from
40:39scratch series and we're going to be
40:40focused on step back
40:42prompting so queer translation as we
40:45said in some of the prior videos kind of
40:48sits at the the kind of first stage of
40:51kind of a a a rag pipeline or flow and
40:56the main aim is to take an question and
40:58to translate it or modify in such a way
41:00that it improves
41:02retrieval now we talked through a few
41:04different ways to approach this problem
41:07so one General approach involves
41:08rewriting a question and we talk about
41:10two ways to do that rag fusion
41:12multiquery and again this is this is
41:14really about taking a question and
41:16modifying it to capture a few different
41:19perspectives um which may improve the
41:21retrieval
41:22process now another approach is to take
41:25a question and kind of make it less
41:26abstract like break it down into sub
41:29questions um and then solve each of
41:31those independently so that's what we
41:32saw with like least to most prompting um
41:35and a bunch of other variants kind of in
41:37that in that vein of sub problem solving
41:42and then consolidating those Solutions
41:44into a final
41:45answer now a different approach
41:48presented um by again Google as well is
41:52stepback prompting so stepback prompting
41:55kind of takes the the the opposite
41:57approach where it tries to ask a more
42:00abstract question so the paper talks a
42:04lot
42:05about um using F shot
42:09prompting to produce what they call the
42:12stepback or more abstract questions and
42:15the way it does it is it provides a
42:17number of
42:19examples of stepb back questions given
42:22your original question so like this is
42:24like this is for example they like for
42:26prompt temp
42:27you're an expert World Knowledge I asked
42:29you a question your response should be
42:30comprehensive not contradict with the
42:32following um and this is kind of where
42:35you provide your like original and then
42:38step back so here's like some example um
42:43questions so like um
42:46like uh at year saw the creation of the
42:50region where the country is located
42:53which region of the
42:54country um is the county of of herir
42:58related um Janell was born in what
43:01country what is janell's personal
43:04history so that that's maybe a more
43:05intuitive example so it's like you ask a
43:07very specific question about like the
43:09country someone's born the more abstract
43:11question is like just give me the
43:12general history of this individual
43:14without worrying about that particular
43:16um more specific question um so let's
43:21actually just walk through how this can
43:22be done in practice um so again here's
43:25kind of like a a diagram of uh the
43:28various approaches um from less
43:31abstraction to more
43:32abstraction now here is where we're
43:35formulating our prompt using a few of
43:38the few shot examples from the
43:40paper um so again like input um yeah
43:45something about like the police perform
43:46wful arrests and what what camp members
43:48of the police do so like it it basically
43:51gives the model a few examples um we
43:54basically formulate this into a prompt
43:56that's really all going on here again we
43:58we repeat um this overall prompt which
44:02we saw from the paper your expert World
44:04Knowledge your test is to step back and
44:06paraphrase a question generate more a
44:08generic step back question which is
44:10easier to answer here are some examples
44:12so it's like a very intuitive prompt so
44:16okay let's start with the question what
44:18is Task composition for llm agents and
44:21we're going to say generate stack
44:23question okay so this is pretty
44:24intuitive right what is a process of
44:26task compos I so like not worrying as
44:28much about agents but what is that
44:30process of task composition in general
44:33and then hopefully that can be
44:36independently um retrieved we we can
44:39independently retrieve documents related
44:41to the stepb back question and in
44:43addition retrieve documents related to
44:45the the actual question and combine
44:47those to produce kind of final answer so
44:49that's really all that's going on um and
44:52here's the response template where we're
44:53Plumbing in the stepback context and our
44:57question context and so what we're going
45:00to do here is we're going to take our
45:01input question and perform retrieval on
45:04that we're also going to generate our
45:06stepb back question and perform
45:07retrieval on that we're going to plumb
45:10those into the prompt as here's our very
45:13here's our basically uh our prompt Keys
45:15normal question step back question um
45:19and our overall question again we
45:21formulate those as a dict we Plum those
45:23into our response prompt um and then we
45:29go ahead and attempt to answer our
45:31overall question so we're going to run
45:33that that's
45:35running and okay we have our answer now
45:38I want to hop over to Langs Smith and
45:41attempt to show you um kind of what that
45:45looked like under the hood so let's see
45:48let's like go into each of these steps
45:51so here was our prompt right you're an
45:53expert World Knowledge your test to to
45:55step back and paraph as a question
45:58um so um here were our few shot prompts
46:03and this was our this was our uh stepb
46:06question so what is the process of task
46:08composition um good from the input what
46:12is Tas composition for LM agents we
46:14perform retrieval on both what is
46:16process
46:17composition uh and what is for LM agents
46:20we perform both retrievals we then
46:23populate our prompt with both
46:27uh original question answer and then
46:29here's the context retrieve from both
46:31the question and the stepb back question
46:33here was our final answer so again this
46:35is kind of a nice technique um probably
46:38depends on a lot of the types of like
46:41the type of domain you want to perform
46:43retrieval on um but in some domains
46:47where for example there's a lot of kind
46:48of conceptual knowledge that underpins
46:52questions you expect users to ask this
46:54stepback approach could be really
46:55convenient to automatically formulate a
46:59higher level question um to for example
47:02try to improve retrieval I can imagine
47:04if you're working with like kind of
47:05textbooks or like technical
47:07documentation where you make independent
47:10chapters focused on more highlevel kind
47:12of like Concepts and then other chapters
47:15on like more detailed uh like
47:18implementations this kind of like stepb
47:19back approach and independent retrieval
47:22could be really helpful thanks hi this
47:25is Lance from Lang chain
47:27this is the fifth video focused on queer
47:29translation in our rack from scratch
47:31series we're going to be talking about a
47:33technique called
47:34hide so again queer translation sits
47:37kind of at the front of the overall rag
47:39flow um and the objective is to take an
47:41input question and translate it in some
47:43way that improves
47:46retrieval now hide is an interesting
47:49approach that takes advantage of a very
47:51simple idea the basic rag flow takes a
47:55question and embeds it takes a document
47:57and embeds it and looks for similarity
48:00between an embedded document and
48:02embedded question but questions and
48:04documents are very different text
48:06objects so documents can be like very
48:08large chunks taken from dense um
48:11Publications or other sources whereas
48:14questions are short kind of tur
48:16potentially ill worded from users and
48:19the intuition behind hide is take
48:22questions and map them into document
48:25space using a hypothetical document or
48:28by generating a hypothetical document um
48:31that's the basic intuition and the idea
48:34kind of shown here visually is that in
48:36principle for certain cases a
48:38hypothetical document is closer to a
48:41desired document you actually want to
48:42retrieve in this you know High
48:45dimensional embedding space than the
48:47sparse raw input question itself so
48:50again it's just kind of means of trans
48:52translating raw questions into these
48:54hypothetical documents that are better
48:57suited for
48:58retrieval so let's actually do a Code
49:00walkthrough to see how this works and
49:02it's actually pretty easy to implement
49:04which is really nice so first we're just
49:06starting with a prompt and we're using
49:08the same notebook that we've used for
49:09prior videos we have a blog post on
49:12agents r index um so what we're going to
49:15do is Define a prompt to generate a
49:17hypothetical documents in this case
49:19we'll say write a write a paper passage
49:22uh to answer a given question so let's
49:25just run this and see what happens again
49:26we're taking our prompt piping it to to
49:29open Ai chck gpte and then using string
49:32Opa parer and so here's a hypothetical
49:34document section related to our question
49:38okay and this is derived of course lm's
49:40kind of embedded uh kind of World
49:43Knowledge which is you know a sane place
49:45to generate hypothetical documents now
49:48let's now take that hypothetical
49:50document and basically we're going to
49:52pipe that into a retriever so this means
49:54we're going to fetch documents from our
49:56index related to this hypothetical
49:59document that's been embedded and you
50:01can see we get a few qu a few retrieved
50:04uh chunks that are related to uh this
50:08hypothetical document that's all we've
50:11done um and then let's take the final
50:15step where we take those retrieve
50:17documents here which we
50:19defined and our question we're going to
50:22pipe that into this rag prompt and then
50:25we're going to run our kind of rag chain
50:27right here which you've seen before and
50:29we get our answer so that's really it we
50:32can go to lsmith and we can actually
50:33look at what happened um so here for
50:37example this was our final um rag prompt
50:43answer the following question based on
50:44this context and here is the retrieve
50:47documents that we passed in so that
50:48part's kind of straightforward we can
50:50also look at um okay this is our
50:55retrieval okay now this is this is
50:57actually what we we generated a
51:01hypothetical document here um okay so
51:05this is our hypothetical document so
51:08we've run chat open AI we generated this
51:10passage with our hypothetical document
51:12and then we've run
51:13retrieval here so this is basically
51:15showing hypothetical document generation
51:17followed by retrieval um so again here
51:21was our passage which we passed in and
51:24then here's our retrieve documents from
51:25the retriever which are related to the
51:27passage content so again in this
51:30particular index case it's possible that
51:32the input question was sufficient to
51:34retrieve these documents in fact given
51:36prior examples uh I know that some of
51:38these same documents are indeed
51:39retrieved just from the raw question but
51:42in other context it may not be the case
51:44so folks have reported nice performance
51:46using Hyde uh for certain domains and
51:49the Really convenient thing is that you
51:52can take this this document generation
51:55prompt you can tune this arbitrarily for
51:57your domain of Interest so it's
51:59absolutely worth experimenting with it's
52:01a it's a need approach uh that can
52:03overcome some of the challenges with
52:04retrieval uh thanks very much hi this is
52:08Lance from Lang chain this is the 10th
52:10video in our rack from scratch series
52:12focused on
52:13routing so we talk through query
52:16translation which is the process of
52:17taking a question and translating in
52:19some way it could be decomposing it
52:21using stepback prompting or otherwise
52:24but the idea here was take our question
52:26change it into a form that's better
52:27suited for retrieval now routing is the
52:30next step which is basically routing
52:32that potentially decomposed question to
52:35the right source and in many cases that
52:37could be a different database so let's
52:38say in this toy example we have a vector
52:40store a relational DB and a graph DB the
52:43what we redo with routing is we simply
52:45route the question based upon the cont
52:47of the question to the relevant data
52:50source so there's a few different ways
52:52to do that one is what we call logical
52:54routing in this case we basically give
52:56an llm knowledge of the various data
52:59sources that we have at our disposal and
53:02we let the llm kind of Reason about
53:04which one to apply the question to so
53:07it's kind of like the the LM is applying
53:09some logic to determine you which which
53:11data sour for example to to use
53:13alternatively you can use semantic
53:15routing which is where we take a
53:17question we embed it and for example we
53:19embed prompts we then compute the
53:22similarity between our question and
53:24those prompts and then we choose a
53:27prompt based upon the similarity so the
53:29general idea is in our diagram we talk
53:31about routing to for example a different
53:33database but it can be very general can
53:35be routing to different prompt it can be
53:37you know really arbitrarily taking this
53:40question and sending it at different
53:41places be at different prompts be at
53:43different Vector
53:44stores so let's walk through the code a
53:46little bit so you can see just like
53:48before we've done a few pip installs we
53:51set up lsmith and let's talk through uh
53:54logical routing first so so in this toy
53:57example let's say we had for example uh
54:00three different docs like we had python
54:02docs we had JS docs we had goang docs
54:05what we want to do is take a question
54:07route it to one of those three so what
54:10we're actually doing is we're setting up
54:11a data model which is basically going to
54:15U be bound to our llm and allow the llm
54:19to Output one of these three options as
54:23a structured object so you really think
54:26about this as like
54:27classification classification plus
54:29function calling to produce a structured
54:31output which is constrained to these
54:33three
54:34possibilities so the way we do that is
54:37let's just zoom in here a little bit we
54:39can Define like a structured object that
54:41we want to get out from our llm like in
54:44this case we want for example you know
54:46one of these three data sources to be
54:49output we can take this and we can
54:52actually convert it into open like open
54:54for example function schema
54:57and then we actually pass that in and
54:58bind it to our llm so what happens is we
55:01ask a question our llm invokes this
55:04function on the output to produce an
55:07output that adheres to the schema that
55:09we specify so in this case for example
55:12um we output like you know in this toy
55:15example let's say we wanted like you
55:17know an output to be data source Vector
55:19store or SQL database the output will
55:22contain a data source object and it'll
55:23be you know one of the options we
55:25specify as a Json string we also
55:28instantiate a parser from this object to
55:32parse that Json string to an output like
55:35a pantic object for example so that's
55:37just one toy example and let's show one
55:39up here so in this case again we had our
55:41three doc sources um we bind that to our
55:46llm so you can see we do with structured
55:49output basically under the hood that's
55:51taking that object definition turning
55:54into function schema and binding that
55:55function schema to our llm and we call
55:58our prompt you're an expert at routing a
56:00user question based on you know
56:03programming language um that user
56:05referring to so let's define our router
56:08here now what we're going to do is we'll
56:10ask a question that is python code so
56:14we'll call that and now it's done and
56:16you see the object we get out is indeed
56:18it's a route query object so it's
56:20exactly it aderes to this data model
56:22we've set up and in this case it's it's
56:26it's correct so it's calling this python
56:27doc so you can we can extract that right
56:29here as a string now once we have this
56:33you can really easily set up like a
56:35route so this could be like our full
56:37chain where we take this router we
56:39should defined here and then this choose
56:41route function can basically take that
56:44output and do something with it so for
56:46example if python docs this could then
56:49apply the question to like a retriever
56:51full of python information uh or JS same
56:55thing so this is where you would hook
56:57basically that question up to different
56:59chains that are like you know retriever
57:02chain one for python retriever chain two
57:04for JS and so forth so this is kind of
57:06like the routing mechanism but this is
57:08really doing the heavy lifting of taking
57:10an input question and turning into a
57:12structured object that restricts the
57:14output to one of a few output types that
57:18we care about in our like routing
57:20problem so that's really kind of the way
57:22this all hooks
57:23together now semantic outing is actually
57:26maybe even a little bit more
57:27straightforward based on what we've seen
57:29previously so in that case let's say we
57:32have two prompts we have a physics
57:33prompt we have a math
57:35prompt we can embed those prompts no
57:38problem we do that here now let's say we
57:41have an input question from a user like
57:43in this case what is a black hole we
57:45pass that through we then apply this
57:47runnable Lambda function which is
57:49defined right here what we're doing here
57:51is we're embedding the question we're
57:53Computing similarity between the
57:54question and the prompts uh we're taking
57:57the most similar and then we're
58:00basically choosing the prompt based on
58:02that similarity and you can see let's
58:03run that and try it
58:05out and we're using the physics prompt
58:07and there we go black holes region and
58:09space so that just shows you kind of how
58:11you can use semantic routing uh to
58:15basically embed a question embed for
58:17example various prompts pick the prompt
58:19based on sematic similarity so that
58:22really gives you just two ways to do
58:23routing one is logical routing with
58:25function in uh can be used very
58:27generally in this case we applied it to
58:29like different coding languages but
58:31imagine these could be swapped out for
58:33like you know my python uh my like
58:36vector store versus My Graph DB versus
58:38my relational DB and you could just very
58:42simply have some description of what
58:43each is and you know then not only will
58:46the llm do reasoning but it'll also
58:49return an object uh that can be parsed
58:52very cleanly to produce like one of a
58:54few very specific types which then you
58:56can reason over like we did here in your
58:59routing function so that kind of gives
59:01you the general idea and these are
59:02really very useful tools and I encourage
59:05you to experiment with them
59:07thanks hi this is Lance from Lang chain
59:10this is the 11th part of our rag from
59:12scratch video series focused on query
59:15construction so we previously talked
59:17through uh query translation which is
59:19the process of taking a question and
59:21converting it or translating it into a
59:24question that's better optimized for
59:25retrieval then we talked about routing
59:27which is the process of going taking
59:29that question routing it to the right
59:30Source be it a given Vector store graph
59:33DB um or SQL DB for example now we're
59:37going to talk about the process of query
59:38construction which is basically taking
59:40natural language and converting it into
59:42particular domain specific language uh
59:45for one of these sources now we're going
59:47to talk specifically about the process
59:49of going from natural language to uh
59:52meditated filters for Vector
59:54Stores um the problem statement is
59:56basically this let's imagine we had an
59:58index of Lang Chain video transcripts um
1:00:01you might want to ask a question give me
1:00:04you know or find find me videos on chat
1:00:06Lang chain published after 2024 for
1:00:09example um the the process of query
1:00:13structuring basically converts this
1:00:15natural language question into a
1:00:17structured query that can be applied to
1:00:19the metadata uh filters on your vector
1:00:22store so most Vector stores will have
1:00:24some kind of meditative filters that can
1:00:26do kind of structur querying on top of
1:00:29uh the chunks that are indexed um so for
1:00:32example this type of query will retrieve
1:00:34all chunks uh that talk about the topic
1:00:36of chat Lang chain uh published after
1:00:39the date 2024 that's kind of the problem
1:00:41statement and to do this we're going to
1:00:43use function calling um in this case you
1:00:46can use for example open AI or other
1:00:48providers to do that and we're going to
1:00:50do is at a high level take the metadata
1:00:53fields that are present in our Vector
1:00:55store and divide them to the model as
1:00:57kind of information and the model then
1:01:00can take those and produce queries that
1:01:03adhere to the schema provided um and
1:01:06then we can parse those out to a
1:01:07structured object like a identic object
1:01:10which again which can then be used in
1:01:12search so that's kind of the problem
1:01:14statement and let's actually walk
1:01:15through
1:01:17code um so here's our notebook which
1:01:20we've kind of gone through previously
1:01:22and I'll just show you as an example
1:01:24let's take a example YouTube video and
1:01:26let's look at the metadata that you get
1:01:28with the transcript so you can see you
1:01:30get stuff like description uh URL um
1:01:34yeah publish date length things like
1:01:36that now let's say we had an index that
1:01:39had um basically a that had a number of
1:01:42different metadata fields and filters uh
1:01:46that allowed us to do range filtering on
1:01:47like view count publication date the
1:01:49video length um or unstructured search
1:01:52on contents and title so those are kind
1:01:54of like the imagine we had an index that
1:01:56had uh those kind of filters available
1:02:00to us what we can do is capture that
1:02:03information about the available filters
1:02:05in an object so we're calling that this
1:02:06tutorial search object kind of
1:02:08encapsulates that information about the
1:02:10available searches that we can do and so
1:02:12we basically enumerate it here content
1:02:14search and title search or semantic
1:02:15searches that can be done over those
1:02:19fields um and then these filters then
1:02:22are various types of structure searches
1:02:24we can do on like the length um The View
1:02:28count and so forth and so we can just
1:02:30kind of build that object now we can set
1:02:33this up really easily with a basic
1:02:35simple prompt that says you know you're
1:02:36an expert can bring natural language
1:02:38into database queries you have access to
1:02:40the database tutorial videos um given a
1:02:43question return a database query
1:02:44optimize retrieval so that's kind of it
1:02:47now here's the key point though when you
1:02:49call this LM with structured output
1:02:51you're binding this pantic object which
1:02:53contains all the information about our
1:02:55index to the llm which is exactly what
1:02:58we talked about previously it's really
1:03:00this process right here you're taking
1:03:02this object you're converting it to a
1:03:04function schema for example open AI
1:03:05you're binding that to your model and
1:03:07then you're going to be able to get um
1:03:10structured object out versus a Json
1:03:13string from a natural language question
1:03:15which can then be parsed into a pantic
1:03:18object which you get out so that's
1:03:19really the flow and it's taking
1:03:21advantage of function calling as we said
1:03:23so if we go back down we set up our
1:03:26query analyzer chain right here now
1:03:28let's try to run that just on a on a
1:03:30purely semantic input so rag from
1:03:32scratch let's run that and you can see
1:03:35this just does like a Content search and
1:03:36a title search that's exactly what you
1:03:38would expect now if we pass a question
1:03:41that includes like a date filter let's
1:03:43just see if that would work
1:03:45and there we go so you kind of still get
1:03:48that semantic search um but you also get
1:03:52um search over for example publish date
1:03:54earliest and latest publish date kind of
1:03:56as as you would expect let's try another
1:03:58one here so videos focus on the topic of
1:04:01chat Lang chain they're published before
1:04:022024 this is just kind of a rewrite of
1:04:04this question in slightly different way
1:04:06using a different date filter and then
1:04:08you can see we can get we get content
1:04:10search title search and then we can get
1:04:12kind of a date search so this is a very
1:04:14general strategy that can be applied
1:04:16kind of broadly to um different kinds of
1:04:19querying you want to do it's really the
1:04:21process of going from an unstructured
1:04:23input to a structured query object out
1:04:26following an arbitrary schema that you
1:04:29provide and so as noted really this
1:04:32whole thing we created here this
1:04:33tutorial search is based upon the
1:04:35specifics of our Vector store of
1:04:37interest and if you want to learn more
1:04:39about this I link to some documentation
1:04:41here that talks a lot about different uh
1:04:44types of of Integrations we have with
1:04:46different Vector store providers to do
1:04:47exactly this so it's a very useful trick
1:04:50um it allows you to do kind of query uh
1:04:54uh say metadata filter filtering on the
1:04:56fly from a natural language question
1:04:58it's a very convenient trick uh that
1:05:01works with many different Vector DBS so
1:05:03encourage you to play with it
1:05:06thanks this is Lance from Lang chain I'm
1:05:09going to talk about indexing uh and
1:05:11mulation indexing in particular for the
1:05:1412th part of our rag from scratch series
1:05:17here so we previously talked about a few
1:05:20different major areas we talk about
1:05:22query translation which takes a question
1:05:24and translates it in some way to
1:05:26optimize for retrieval we talk about
1:05:28routing which is the process of taking a
1:05:30question routing it to the right data
1:05:32source be it a vector store graph DB uh
1:05:35SQL DB we talked about queer
1:05:37construction we dug into uh basically
1:05:39queer construction for Vector stores but
1:05:41of course there's also text SQL text to
1:05:44Cipher um so now we're going to talk
1:05:46about indexing a bit in particular we're
1:05:48going to talk about indexing indexing
1:05:50techniques for Vector Stores um and I
1:05:53want to highlight one particular method
1:05:54today called multi-representation
1:05:57indexing so the high LEL idea here is
1:06:01derived a bit from a paper called
1:06:03proposition indexing which kind of makes
1:06:05a simple
1:06:06observation you can think about
1:06:10decoupling raw documents and the unit
1:06:13you use for
1:06:15retrieval so in the typical case you
1:06:18take a document you split it up in some
1:06:21way to index it and then you embed the
1:06:24split directly
1:06:26um this paper talks about actually
1:06:29taking a document splitting it in some
1:06:32way but then using an llm to produce
1:06:35what they call a proposition which you
1:06:37can think of as like kind of a
1:06:38distillation of that split so it's kind
1:06:41of like using an llm to modify that
1:06:43split in some way to distill it or make
1:06:45it like a crisper uh like summary so to
1:06:49speak that's better optimized for
1:06:51retrieval so that's kind of one
1:06:52highlight one piece of intuition so we
1:06:55actually taken that idea and we've kind
1:06:57of built on it a bit in kind of a really
1:06:59nice way that I think is very well
1:07:01suited actually for long context llms so
1:07:04the idea is pretty simple you take a
1:07:07document and you you actually distill it
1:07:10or create a proposition like they show
1:07:11in the prior paper I kind of typically
1:07:14think of this as just produce a summary
1:07:16of the document and you embed that
1:07:18summary so that summary is meant to be
1:07:21optimized for retrieval so might contain
1:07:23a bunch of keywords from the document or
1:07:25like the big
1:07:26ideas such that when you embed the
1:07:29summary you embed a question you do
1:07:31search you basically can find that
1:07:34document based upon this highly
1:07:35optimized summary for retrieval so
1:07:38that's kind of represented here in your
1:07:39vector store but here's the catch you
1:07:42independently store the raw document in
1:07:44a dock store and when you when you
1:07:47basically retrieve the summary in the
1:07:50vector store you return the full
1:07:52document for the llm to perform
1:07:54generation and this is a nice trick
1:07:56because at generation time now with long
1:07:59condex LMS for example the LM can handle
1:08:02that entire document you don't need to
1:08:03worry about splitting it or anything you
1:08:05just simply use the summary to prod like
1:08:08to create a really nice representation
1:08:10for fishing out that full dock use that
1:08:13full dock in generation there might be a
1:08:14lot of reasons you want to do that you
1:08:16want to make sure the LM has the full
1:08:17context to actually answer the question
1:08:19so that's the big idea it's a nice trick
1:08:22and let's walk through some code
1:08:23here we have a notebook all set up uh
1:08:26just like before we done some pip
1:08:28installs um set to maybe I Keys here for
1:08:32lsmith um kind of here's a diagram now
1:08:35let me show an example let's just load
1:08:37two different uh blog posts uh one is
1:08:39about agents one is about uh you know
1:08:41human data quality um and what we're
1:08:45going to do is let's create a summary of
1:08:46each of those so this is kind of the
1:08:48first step of that process where we're
1:08:50going from like the raw documents to
1:08:51summaries let's just have a look and
1:08:53make sure those ran So Okay cool so the
1:08:57first DOC discusses you know building
1:08:58autonomous agents the second doc
1:09:00contains the importance of high quality
1:09:01human data and training okay so that's
1:09:03pretty nice we have our summaries now
1:09:06we're going to go through a process
1:09:07that's pretty
1:09:08simple first we Define a vector store
1:09:10that's going to index those
1:09:12summaries now we're going to Define what
1:09:14we call like our our document storage is
1:09:16going to store the full documents okay
1:09:19so this multiv Vector retriever kind of
1:09:21just pulls those two things together we
1:09:23basically add our Dock Store we had this
1:09:25bite store is basically the the the full
1:09:28document store uh the vector store is
1:09:30our Vector store um and now this ID is
1:09:32what we're going to use to reference
1:09:34between the chunks or the summaries and
1:09:37the full documents that's really it so
1:09:40now for every document we'll Define a
1:09:42new Doc ID um and then we're basically
1:09:44going to like take our summary documents
1:09:47um and we're going to extract um for
1:09:51each of our summaries we're going to get
1:09:53the associated doc ID so we go um so
1:09:58let's go ahead and do that so we have
1:10:01our summary docs which we add to the
1:10:03vector store we have our full documents
1:10:06uh our doc IDs and the full raw
1:10:08documents which are added to our doc
1:10:10store and then let's just do a query
1:10:12Vector store like a similarity search on
1:10:14our Vector store so memory and agents
1:10:16and we can see okay so we can extract
1:10:19you know from the summaries we can get
1:10:22for example the summary that pertains to
1:10:24um a agents so that's a good thing now
1:10:27let's go ahead and run a query get
1:10:30relevant documents on our retriever
1:10:32which basically combines the summaries
1:10:35uh which we use for retrieval then the
1:10:38doc store which we use to get the full
1:10:40doc back so we're going to apply our
1:10:42query we're going to basically run this
1:10:45and here's the key Point we've gotten
1:10:48back the entire
1:10:49article um and we can actually if you
1:10:53want to look at the whole thing we we
1:10:55can just go ahead and do this here we go
1:10:58so this is the entire article that we
1:10:59get back from that search so it's a
1:11:02pretty nice trick again we query with
1:11:04just memory and agents um and we can
1:11:07kind of go back to our diagram here we
1:11:09quered for memory and agents it started
1:11:11our summaries it found the summary
1:11:13related to memory and agents it uses
1:11:15that doc ID to reference between the
1:11:17vector store and the doc store it fishes
1:11:19out the right full doc returns us the
1:11:21full document in this case the full web
1:11:23page that's really it simple idea nice
1:11:27way to go from basically like nice
1:11:30simple proposition style or summary
1:11:32style indexing to full document
1:11:34retrieval which is very useful
1:11:36especially with long contact LMS thank
1:11:40you hi this is Lance from Lang chain
1:11:43this is the 13th part of our rag from
1:11:45scratch series focused on a technique
1:11:47called
1:11:48Raptor so Raptor sits within kind of an
1:11:51array of different indexing techniques
1:11:54that can be applied on Vector Stores um
1:11:57we just talked about
1:11:58multi-representation indexing um we I
1:12:01priv a link to a video that's very good
1:12:04talking about the different means of
1:12:05chunking so I encourage you to look at
1:12:06that and we're going to talk today about
1:12:09a technique called Raptor which you can
1:12:10kind of think of it as a technique for
1:12:12hierarchical
1:12:14indexing so the highle intuition is
1:12:17this some questions require very
1:12:20detailed information from a corpus to
1:12:23answer like pertain to a single document
1:12:25or single chunk so like we can call
1:12:28those low-level
1:12:29questions some questions require
1:12:31consolidation across kind broad swast of
1:12:34a document so across like many documents
1:12:37or many chunks within a document and you
1:12:39can call those like higher level
1:12:41questions and so there's kind of this
1:12:44challenge in retrieval and that
1:12:46typically we do like K nearest neighbors
1:12:48retrieval like we've been talking about
1:12:50you're fishing out some number of chunks
1:12:53but what if you have a question that
1:12:55requires information across like five
1:12:57six you know or a number of different
1:12:59chunks which may exceed you know the K
1:13:02parameter in your retrieval so again
1:13:04when you typically do retrieval you
1:13:06might set a k parameter of three which
1:13:08means you're retrieving three chunks
1:13:10from your vector store um and maybe you
1:13:12have a high very high level question
1:13:14that could benefit from infation across
1:13:16more than three so this technique called
1:13:18raptor is basically a way to build a
1:13:21hierarchical index of document summaries
1:13:25and the intuition is this you start with
1:13:27a set of documents as your Leafs here on
1:13:30the left you cluster them and then you
1:13:33Summarize each cluster so each cluster
1:13:36of similar documents um will consult
1:13:39information from across your context
1:13:42which is you know your context could be
1:13:44a bunch of different splits or could
1:13:45even be across a bunch of different
1:13:47documents you're basically capturing
1:13:49similar ones and you're consolidating
1:13:51the information across them in a summary
1:13:53and here's the interesting thing you do
1:13:54that
1:13:55recursively until either you hit like a
1:13:57limit or you end up with one single
1:13:59cluster that's a kind of very high level
1:14:01summary of all of your
1:14:03documents and what the paper shows is
1:14:05that if you basically just collapse all
1:14:08these and index them together as a big
1:14:10pool you end up with a really nice array
1:14:12of chunks that span the abstraction
1:14:15hierarchy like you have a bunch of
1:14:17chunks from Individual documents that
1:14:19are just like more detailed chunks
1:14:21pertaining to that you know single
1:14:23document but you also have chunks from
1:14:25these summaries or I would say like you
1:14:27know maybe not chunks but in this case
1:14:29the summary is like a distillation so
1:14:31you know raw chunks on the left that
1:14:33represent your leavs are kind of like
1:14:35the rawest form of information either
1:14:37raw chunks or raw documents and then you
1:14:40have these higher level summaries which
1:14:42are all indexed together so if you have
1:14:44higher level questions they should
1:14:46basically be more similar uh in sematic
1:14:49search for example to these higher level
1:14:50summary chunks if you have lower level
1:14:53questions then they'll retrieve these
1:14:55more lower level chunks and so you have
1:14:56better semantic coverage across like the
1:14:59abstraction hierarchy of question types
1:15:01that's the intuition they do a bunch of
1:15:02nice studies to show that this works
1:15:04pretty well um I actually did a deep
1:15:07dive video just on this which I link
1:15:09below um I did want to cover it briefly
1:15:12just at a very high level um so let's
1:15:15actually just do kind of a code walkr
1:15:17and I've added it to this rack from
1:15:19scratch course notebook but I link over
1:15:21to my deep dive video as well as the
1:15:23paper and the the full code notebook
1:15:27which is already checked in is discussed
1:15:28at more length in the Deep dive the
1:15:31technique is a little bit detailed so I
1:15:33only want to give you very high levels
1:15:35kind of overview here and you can look
1:15:37at the Deep dive video if you want to go
1:15:39in more depth again we talked through
1:15:41this abstraction
1:15:43hierarchy um I applied this to a large
1:15:46set of Lang chain documents um so this
1:15:49is me loading basically all of our Lang
1:15:51chain expression language docs so this
1:15:53is on the order of 30 documents you can
1:15:55see I do a histogram here of the token
1:15:57counts per document some are pretty big
1:16:00most are fairly small less than you know
1:16:024,000 tokens um and what I did is I
1:16:05indexed all of them um individually so
1:16:09the all those raw documents you can kind
1:16:11of Imagine are here on the left and then
1:16:14I do um I do embedding I do clustering
1:16:18summarization and I do that recursively
1:16:21um until I end up with in this case I
1:16:24believe I only set like three levels of
1:16:27recursion and then I save them all my
1:16:29Vector store so that's like the highle
1:16:31idea I'm applying this Raptor technique
1:16:33to a whole bunch of Lang chain documents
1:16:36um that have fairly large number of
1:16:38tokens um so I do that um and yeah I use
1:16:45actually use both CLA as well as open AI
1:16:48here um this talks through the
1:16:50clustering method which they that they
1:16:52use which is pretty interesting you can
1:16:54kind of dig into that on your own if if
1:16:55you're really um interested this is a
1:16:58lot of their code um which I cite
1:17:00accordingly um this is basically
1:17:02implementing the clustering method that
1:17:03they use um and this is just simply the
1:17:08document embedding stage um this is like
1:17:12basically embedding uh and clustering
1:17:15that's really it uh some text formatting
1:17:19um summarizing of the clusters right
1:17:22here um and then this is just running
1:17:24that whole process recursively that's
1:17:26really it um this is tree building so
1:17:30basically I have the RO the rod docs
1:17:33let's just go back and look at Doc texts
1:17:35so this should be all my raw documents
1:17:37uh so that's right you can see it here
1:17:39doc text is basically just the text in
1:17:41all those Lang chain documents that I
1:17:43pulled
1:17:45um and so I run this process on them
1:17:49right
1:17:50here uh so this is that recursive
1:17:52embedding cluster basically runs and
1:17:54produces is that tree here's the results
1:17:57um this is me just going through the
1:17:59results and basically adding the result
1:18:02text to this list of uh texts um oh okay
1:18:07so here's what I do this Leaf text is
1:18:09all the raw documents and I'm appending
1:18:12to that all the summaries that's all
1:18:14it's going on and then I'm indexing them
1:18:16all together that's the key Point rag
1:18:19chain and there you have it that's
1:18:21really all you do um so anyway I
1:18:23encourage you to look at this in depth
1:18:24it's a pretty interesting technique it
1:18:26works well long with long contexts so
1:18:29for example one of the arguments I made
1:18:30is that it's kind of a nice approach to
1:18:32consult information across like a span
1:18:35of large
1:18:36documents like in this particular case
1:18:38my individual documents were lch
1:18:40expression language docs uh each each
1:18:42being somewhere in the order of you know
1:18:44in this case like you know most of them
1:18:46are less than 4,000 tokens some pretty
1:18:48big but I index them all I cluster them
1:18:51without any splits uh embed them cluster
1:18:54them build this tree um and go from
1:18:56there and it all works because we now
1:18:58have llms that can go out to you know
1:19:01100 or 200,000 up to million tokens and
1:19:03Contex so you can actually just do this
1:19:05process for big swats of documents in
1:19:08place without any without any splitting
1:19:10uh it's a pretty nice approach so I
1:19:12encourage you to think about it look at
1:19:13it watch the deep that video If you
1:19:14really want to go deeper on this um
1:19:20thanks hi this is Lance from Lang chain
1:19:22this is the 14th part of our rag from
1:19:24scratch series we're going to I'm going
1:19:26to be talking about an approach called
1:19:28cold
1:19:29bear um so we've talked about a few
1:19:34different approaches for indexing and
1:19:37just as kind of a refresher indexing
1:19:38Falls uh kind of right down here in our
1:19:41flow we started initially with career
1:19:43translation taking a question
1:19:45translating it in some way to optimize
1:19:47retrieval we talked about routing it to
1:19:49a particular database we then talked
1:19:51about query construction so going from
1:19:53natural language to the DSL or domain
1:19:56specific language for E any of the
1:19:59databases that you want to work with
1:20:01those are you know metadata filters for
1:20:02Vector stores or Cipher
1:20:05for graph DB or SQL for relational DB so
1:20:09that's kind of the flow we talked about
1:20:10today we talked about some indexing
1:20:12approaches like multi-representation
1:20:14indexing we gave a small shout out to
1:20:16greet camer in the series on chunking uh
1:20:19we talked about hierarchical indexing
1:20:22and I want to include one Advanced kind
1:20:24embedding approach so we talked a lot
1:20:26about embeddings are obviously very
1:20:28Central to semantic similarity search um
1:20:31and
1:20:32retrieval so one of the interesting
1:20:35points that's been brought up is that
1:20:38embedding models of course take a
1:20:39document you can see here on the top and
1:20:42embed it basically compress it to a
1:20:45vector so it's kind of a compression
1:20:48process you representing all the
1:20:49semantics of that document in a single
1:20:51Vector you're doing the same to your
1:20:53question you're doing similarity search
1:20:55between the question embedding and the
1:20:57document embedding um in order to
1:20:59perform retrieval you're typically
1:21:01taking the you know K most similar um
1:21:05document abetting is given a question
1:21:07and that's really how you're doing it
1:21:10now a lot of people said well hey the
1:21:12compressing a full document with all
1:21:13this Nuance to single Vector seems a
1:21:15little bit um overly restrictive right
1:21:18and this is a fair question to ask um
1:21:21there's been some interesting approaches
1:21:22to try to address that and one is this
1:21:24this this approach method called Co bear
1:21:28so the intuition is actually pretty
1:21:30straightforward there's a bunch of good
1:21:31articles I link down here this is my
1:21:33little cartoon to explain it which I
1:21:35think is hopefully kind of helpful but
1:21:37here's the main idea instead of just
1:21:39taking a document and compressing it
1:21:41down to a single Vector basically single
1:21:44uh what we might call embedding Vector
1:21:46we take the document we break it up into
1:21:49tokens so tokens are just like you know
1:21:51units of of content it depends on the
1:21:54token areas you use we talked about this
1:21:56earlier so you basically tokenize it and
1:21:59you produce basically an embedding or
1:22:01vector for every token and there's some
1:22:03kind of positional uh waiting that
1:22:05occurs when you do this process so you
1:22:08obviously you look to look at the
1:22:09implementation understand the details
1:22:10but the intuition is that you're
1:22:12producing some kind of representation
1:22:14for every token okay and you're doing
1:22:17the same thing for your question so
1:22:19you're taking your question you're
1:22:20breaking into a tokens and you have some
1:22:22representation or vector per token
1:22:25and then what you're doing is for every
1:22:27token in the question you're Computing
1:22:29the similarity across all the tokens in
1:22:33the document and you're finding the max
1:22:36you're taking the max you're storing
1:22:38that and you're doing that process for
1:22:42all the tokens in the question so again
1:22:45token two you compare it to every token
1:22:48in the in the document compute the
1:22:51Max and then the final score is in this
1:22:54case the sum of the max similarities uh
1:22:57between every question token and any
1:23:00document token so it's an interesting
1:23:03approach uh it reports very strong
1:23:06performance latency is definitely a
1:23:08question um so kind of production
1:23:11Readiness is something you should look
1:23:12into but it's a it's an approach that's
1:23:14worth mentioning here uh because it's
1:23:16pretty
1:23:18interesting um and let's walk through
1:23:20the
1:23:21code
1:23:22so there's actually nice Library called
1:23:25rouille which makes it very easy to play
1:23:27with Co bear um she's pip install it
1:23:30here I've already done that and we can
1:23:33use one of their pre-train models to
1:23:35mediate this process so I'm basically
1:23:36following their documentation this is
1:23:38kind of what they recommended um so I'm
1:23:40running this
1:23:41now hopefully this runs somewhat quickly
1:23:43I'm not sure I I previously have loaded
1:23:45this model so hopefully it won't take
1:23:47too long and yeah you can see it's
1:23:48pretty quick uh I'm on a Mac M2 with 32
1:23:51gigs um so just as like a context in
1:23:54terms of my my system um this is from
1:23:56their documentation we're just grabbing
1:23:57a Wikipedia page this is getting a full
1:23:59document on Miyazaki so that's cool
1:24:03we're going to grab that now this is
1:24:05just from their docs this is basically
1:24:06how we create an index so we provide the
1:24:09you know some index name the collection
1:24:11um the max document length and yeah you
1:24:13should look at their documentation for
1:24:15these flags these are just the defaults
1:24:16so I'm going to create my index um so I
1:24:19get some logging here so it it's working
1:24:21under the hood um and by the way I
1:24:24actually have their documentation open
1:24:26so you can kind of follow along um
1:24:30so
1:24:31um let's see yeah right about here so
1:24:35you can kind of follow this indexing
1:24:36process to create an index you need to
1:24:37load a train uh a trained model this can
1:24:41be either your own pre-train model or
1:24:42one of ours from The Hub um and this is
1:24:44kind of the process we're doing right
1:24:45now create index is just a few lines of
1:24:48code and this is exactly what we're
1:24:49doing um so this is the you know my
1:24:52documents and this is the indexing step
1:24:54that we just we just kind of walk
1:24:56through and it looks like it's done um
1:24:58so you get a bunch of logging here
1:25:00that's fine um now let's actually see if
1:25:02this works so we're going to run drag
1:25:04search what an emotion Studio did Miaki
1:25:06found set our K parameter and we get
1:25:09some results okay so it's running and
1:25:13cool we get some documents out so you
1:25:15know it seems to work now what's nice is
1:25:17you can run this within lighting chain
1:25:18as a liting chain retriever so that
1:25:20basically wraps this as a lighting chain
1:25:22Retriever and then you can use it freely
1:25:24as a retriever within Lang chain it
1:25:25works with all the other different LMS
1:25:27and all the other components like
1:25:28rankers and so forth that we talk
1:25:30through so you can use this directly as
1:25:32a retriever let's try this out and boom
1:25:35nice and fast um and we get our
1:25:38documents again this is a super simple
1:25:40test example you should run this maybe
1:25:42on more complex cases but it's pretty
1:25:44pretty easy spin up it's a really
1:25:45interesting alternative indexing
1:25:47approach um using again like we talked
1:25:49through um a very different algorithm
1:25:53for computing do similarity that may
1:25:55work better I think an interesting
1:25:57regime to consider this would be longer
1:25:59documents so if you want like longer um
1:26:02yeah if if you basically want kind of
1:26:04long context embedding I think you
1:26:06should look into for example the uh Max
1:26:09token limits for this approach because
1:26:11it partitions the document into into
1:26:13each token um I would be curious to dig
1:26:16into kind of what the overall context
1:26:18limits are for this approach of coar but
1:26:21it's really interesting to consider and
1:26:22it reports very strong performance so
1:26:24again I encourage you to play with it
1:26:26and this is just kind of an intro to how
1:26:27to get set up and to start experimenting
1:26:29with it really quickly
1:26:32thanks hi this is Lance from Lang chain
1:26:35I'm going to be talking about using
1:26:36langra to build a diverse and
1:26:39sophisticated rag
1:26:40flows so just to set the stage the basic
1:26:44rag flow you can see here starts with a
1:26:47question retrieval of relevant documents
1:26:49from an index which are passed into the
1:26:52context window of an llm for generation
1:26:54of an answer ground in your documents
1:26:57that's kind of the basic
1:26:59outline and we can see it's like a very
1:27:01linear
1:27:02path um in practice though you often
1:27:06encounter a few different types of
1:27:08questions like when do we actually want
1:27:10to
1:27:11retrieve based upon the context of the
1:27:13question um are the retrieve documents
1:27:16actually good or not and if they're not
1:27:18good should we discard them and then how
1:27:21do we loot back and retry retrieval with
1:27:23for example and improved
1:27:25question so these types of questions
1:27:29motivate an idea of active rag which is
1:27:32a process where an llm actually decides
1:27:34when and where to retrieve based upon
1:27:36like existing
1:27:37retrievals or existing
1:27:40Generations now when you think about
1:27:43this there's a few different levels of
1:27:45control that you have over an llm in a
1:27:48rag
1:27:49application the base case like we saw
1:27:52with our chain is just use an llm to
1:27:55choose a single steps output so for
1:27:58example in traditional rag you feed it
1:28:00documents and it decides to generation
1:28:04so it's just kind of one step now a lot
1:28:07of rag workflows will use the idea of
1:28:09routing so like given a question should
1:28:12I route it to a vector store or a graph
1:28:16DB um and we have seen this quite a
1:28:20bit now this newer idea that I want to
1:28:23introduce
1:28:24is how do we build more sophisticated
1:28:28logical
1:28:29flows um in a rag
1:28:32pipeline um that you let the llm choose
1:28:36between different steps but specify all
1:28:41the transitions that are
1:28:43available and this is known as we call a
1:28:45state
1:28:46machine now there's a few different
1:28:50architectures that have emerged uh to
1:28:53build different types of rag
1:28:54chains and of course chains are
1:28:58traditionally used just for like very
1:28:59basic rag but this notion of State
1:29:02machine is a bit newer and Lang graph
1:29:06which we recently released provides a
1:29:07really nice way to build State machines
1:29:10for Rag and for other
1:29:13things and the general idea here is that
1:29:16you can lay out more diverse and
1:29:18complicated rag
1:29:20flows and then Implement them as graphs
1:29:24and it kind of motivates this more broad
1:29:26idea of of like flow engineering and
1:29:28thinking through the actual like
1:29:30workflow that you want and then
1:29:32implementing it um and we're gonna
1:29:35actually do that right now so I'm GNA Pi
1:29:39a recent paper called CAG corrective rag
1:29:42which is really a nice method um for
1:29:46active rag that incorporates a few
1:29:48different
1:29:50ideas um so first you retrieve documents
1:29:54and then you grade
1:29:55them now if at least one document
1:29:59exceeds the threshold for
1:30:01relevance you go to generation you
1:30:04generate your
1:30:06answer um and it does this knowledge
1:30:10refinement stage after that but let's
1:30:13not worry about that for right now it's
1:30:15kind of not essential for understanding
1:30:16the basic flow here so again you do a
1:30:20grade for relevance for every document
1:30:23if any is relevant you
1:30:25generate now if they're all ambiguous or
1:30:29incorrect based upon your
1:30:31grader you retrieve from an external
1:30:34Source they use web
1:30:36search and then they pass that as their
1:30:39context for answer
1:30:42generation so it's a really neat
1:30:44workflow where you're doing retrieval
1:30:46just like with basic rag but then you're
1:30:48reasoning about the documents if they're
1:30:50relevant go ahead and at least one is
1:30:53relevant go ahead and generate if
1:30:54they're not retrieve from alternative
1:30:57source and then pack that into the
1:30:59context and generate your
1:31:02answer so let's see how we would
1:31:04implement this as a estate machine using
1:31:09Lang
1:31:10graph um we'll make a few
1:31:13simplifications
1:31:15um we're going to first decide if any
1:31:19documents are relevant we'll go ahead
1:31:22and do the the web search
1:31:25um to supplement the output so that's
1:31:28just like kind of one minor
1:31:29modification um we'll use tab search for
1:31:32web search um we use Query writing to
1:31:35optimize the search for uh to optimize
1:31:38the web search but it follows a lot of
1:31:40the the intuitions of the main paper uh
1:31:44small note here we set the Tav API key
1:31:48and another small mode I've already set
1:31:50my lsmith API key um with which we'll
1:31:54see is useful a bit later for observing
1:31:57the resulting
1:31:59traces now I'm going to index three blog
1:32:02posts that I
1:32:03like um I'm going to use chroma DB I'm G
1:32:06use open ey embeddings I'm going to run
1:32:09this right now this will create a vector
1:32:11store for me from these three blog
1:32:15posts and then what I'm going to do is
1:32:19Define
1:32:20State now this is kind of the core
1:32:23object that going to be passed around my
1:32:25graph that I'm going to
1:32:27modify and right here is where I Define
1:32:29it and the key point to note right now
1:32:32is it's just a dictionary and it can
1:32:35contain things that are relevant for rag
1:32:37like question documents generation and
1:32:40we'll see how we update that in in in a
1:32:42little bit but the first thing to note
1:32:44is we Define our state and this is
1:32:47what's going to be modified in every Noe
1:32:49of our
1:32:50graph now here's really the Crux of it
1:32:53and this is the thing I want to zoom in
1:32:54on a little bit um
1:32:58so when you kind of move from just
1:33:00thinking about promps to thinking about
1:33:02overall flows it it's like kind of a fun
1:33:05and interesting exercise I kind of think
1:33:07about this as it's been mentioned on
1:33:09Twitter a little bit more like flow
1:33:12engineering so let's think through what
1:33:15was actually done in the paper and what
1:33:19modifications to our state are going to
1:33:21happen in each stage so we start with a
1:33:24question you can see that on the far
1:33:25left and this kind of state is represent
1:33:28as a dictionary like we have we start
1:33:29with a question we perform retrieval
1:33:31from our Vector store which we just
1:33:33created that's going to give us
1:33:35documents so that's one node we made an
1:33:38an adjustment to our state by adding
1:33:41documents that's step
1:33:43one now we have a second node where
1:33:45we're going to grade the documents and
1:33:48in this node we might filter some out so
1:33:50we are making a modification to state
1:33:52which is why it's a node so we're going
1:33:54to have a
1:33:55greater then we're going to have what
1:33:58we're going to call a conditional Edge
1:34:00so we saw we went from question to
1:34:02retrieval retrieval always goes to
1:34:05grading and now we have a
1:34:07decision if any document is
1:34:10irrelevant we're going to go ahead and
1:34:14do web search to
1:34:15supplement and if they're all relevant
1:34:17will go to generation it's a minor kind
1:34:20of a minor kind of logical uh decision
1:34:23ision that we're going to
1:34:25make um if any are not relevant we'll
1:34:28transform the query and we'll do web
1:34:31search and we'll use that for Generation
1:34:33so that's really it and that's how we
1:34:35can kind of think about our flow and how
1:34:37our States can be modified throughout
1:34:39this
1:34:40flow now all we then need to do and I I
1:34:44kind of found
1:34:46spending 10 minutes thinking carefully
1:34:48through your flow
1:34:50engineering is really valuable because
1:34:52from here it's really implementation
1:34:55details um and it's pretty easy as
1:34:58you'll see so basically I'm going to run
1:35:02this code block but then we can like
1:35:03walk through some of it I won't show you
1:35:04everything so it'll get a little bit
1:35:06boring but really all we're doing is
1:35:09we're finding functions for every node
1:35:12that take in the state and modify in
1:35:15some way that's all it's going on so
1:35:17think about retrieval we run retrieval
1:35:19we take in state remember it's a dict we
1:35:22get our state dick like this
1:35:24we extract one keyy question from our
1:35:26dick we pass that to a retriever we get
1:35:29documents and we write back out State
1:35:32now with documents key added that's
1:35:36all generate going to be similar we take
1:35:39in state now we have our question and
1:35:41documents we pull in a prompt we Define
1:35:44an llm we do minor post processing on
1:35:47documents we set up a chain for
1:35:49retrieval uh or sorry for Generation
1:35:51which is just going to be take our
1:35:52prompt pump Plum that to an llm
1:35:55partially output a string and we run it
1:35:57right here invoking our documents in our
1:36:01question to get our answer we write that
1:36:04back to State that's
1:36:06it and you can kind of follow here for
1:36:09every node we just Define a function
1:36:12that performs the state modification
1:36:13that we want to do on that
1:36:15node grading documents is going to be
1:36:17the same um in this case I do a little
1:36:21thing extra here because I actually
1:36:23Define a identic data model for my
1:36:25grader so that the output of that
1:36:28particular grading chain is a binary yes
1:36:31or no you can look at the code make sure
1:36:33it's all shared um and that just makes
1:36:36sure that our output is is very
1:36:38deterministic so that we then can down
1:36:41here perform logical filtering so what
1:36:45you can see here is um we Define this
1:36:49search value no and we iterate through
1:36:53our documents we grade them if any
1:36:56document uh is graded as not relevant we
1:37:00flag this search thing to yes that means
1:37:04we're going to perform web search we
1:37:06then add that to our state dict at the
1:37:08end so run web search now that value is
1:37:10true that's
1:37:12it and you can kind of see we go through
1:37:14some other nodes here there's web search
1:37:16node um now here is where our one
1:37:20conditional Edge we Define right here
1:37:23this is where where we decide to
1:37:24generate or not based on that search key
1:37:27so we again get our state let's extract
1:37:30the various values so we have this
1:37:32search value now if search is yes we
1:37:37return the next no that we want to go to
1:37:40so in this case it'll be transform query
1:37:42which will then go to web search else we
1:37:46go to
1:37:47generate so what we can see is we laid
1:37:51out our graph which you can kind of see
1:37:53up
1:37:55here and now we Define functions for all
1:37:57those nodes as well as the conditional
1:38:01Edge and now we scroll down all we have
1:38:05to do is just lay that out here again as
1:38:08our flow and this is kind of what you
1:38:10might think of as like kind of flow
1:38:11engineering where you're just laying out
1:38:13the graph as you drew it where we have
1:38:17set our entry point as retrieve we're
1:38:19adding an edge between retrieve and
1:38:21grade documents so we went retrieval
1:38:23grade documents we add our conditional
1:38:25Edge depending on the grade either
1:38:28transform the query go to web search or
1:38:31just go to generate we create an edge
1:38:34between transform the query and web
1:38:35search then web search to generate and
1:38:38then we also have an edge generate to
1:38:40end and that's our whole graph that's it
1:38:42so we can just run
1:38:44this and now I'm going to ask a question
1:38:47so let's just say um how does agent
1:38:51memory work for example let's just try
1:38:53that and what this is going to do is
1:38:55going to print out what's going on as we
1:38:58run through this graph so um first we
1:39:01going to see output from
1:39:03retrieve this is going to be all of our
1:39:05documents that we retrieved so that's
1:39:07that's fine this just from our our
1:39:08retriever then you can see that we're
1:39:11doing a relevance check across our
1:39:13documents and this is kind of
1:39:15interesting right you can see we grading
1:39:17them here one is grade as not
1:39:20relevant um and okay you can see the
1:39:23documents are now filtered because we
1:39:24removed the one that's not relevant and
1:39:26because one is not relevant we decide
1:39:29okay we're going to just transform the
1:39:31query and run web
1:39:33search and um you can see after query
1:39:37transformation we rewrite the question
1:39:39slightly we then run web
1:39:42search um and you can see from web
1:39:44search it searched from some additional
1:39:47sources um which you can actually see
1:39:50here it's
1:39:51appended as a so here it is so here it's
1:39:55a new document appended from web search
1:39:57which is from memory knowledge
1:39:59requirements so it it basically looked
1:40:01up some AI architecture related to
1:40:03memory uh web results so that's fine
1:40:06that's exactly what we want to
1:40:08do and then um we generate a
1:40:11response so that's great and this is
1:40:14just showing you everything in kind of
1:40:15gory detail but I'm going to show you
1:40:18one other thing that's that's really
1:40:19nice about this if I go to lsmith
1:40:24I have my AP I ke set so all my
1:40:26Generations are just logged to to lsmith
1:40:29and I can see my Lang graph run here now
1:40:33what's really cool is this shows me all
1:40:36of my nodes so remember we had retrieve
1:40:42grade we evaluated the grade because one
1:40:45was irrelevant we then went ahead and
1:40:47transformed the query we did a web
1:40:50search we pended that to our context you
1:40:52can see all those steps are laid out
1:40:54here in fact you can even look at every
1:40:56single uh grader and its output I will
1:40:59move this up
1:41:01slightly um so you can see the the
1:41:04different scores for grades okay so this
1:41:06particular retrieval was graded as as
1:41:09not relevant so that's fine that that
1:41:11can happen in some cases and because of
1:41:15that um we did a query transformation so
1:41:18we modified the question slightly how
1:41:21does memory how does the memory system
1:41:23an artificial agents function so it's
1:41:25just a minor rephrasing of the question
1:41:27we did this Tav web search this is where
1:41:30it queried from this particular blog
1:41:33post from medium so it's like a sing web
1:41:35query we can like sanity check it and
1:41:37then what's need is we can go to our
1:41:39generate step look at open Ai and here's
1:41:41our full prompt how does the memory
1:41:43system in our official agents function
1:41:46and then here's all of our documents so
1:41:49this is the this is the web search as
1:41:51well as we still have the Rel chunks
1:41:54that were retrieved from our blog posts
1:41:57um and then here's our answer so that's
1:42:01really it you can see how um really
1:42:05moving from the notion of just like I'll
1:42:09actually go back to the original um
1:42:12moving
1:42:14from uh I will try to open this up a
1:42:17little
1:42:18bit um
1:42:24yeah I can see my face
1:42:26still um the transition from laying out
1:42:33simple
1:42:35chains to
1:42:38flows is a really interesting and
1:42:40helpful way of thinking about why graphs
1:42:42are really interesting because you can
1:42:45encode more sophisticated logical
1:42:48reasoning
1:42:49workflows but in a
1:42:51very like clean and well-engineered way
1:42:56where you can specify all the
1:42:58transitions that you actually want to
1:43:00have
1:43:01executed um and I actually find this way
1:43:05of thinking and building kind of logical
1:43:08uh like workflows really
1:43:10intuitive um we have a blog post coming
1:43:13out uh tomorrow that discusses both
1:43:17implementing self rag as well as C rag
1:43:20for two different active rag approaches
1:43:22using using uh this idea of of State
1:43:25machines and Lang graph um so I
1:43:28encourage you to play with it uh I found
1:43:31it really uh intuitive to work with um I
1:43:35also found uh inspection of traces to be
1:43:38quite intuitive using Lang graph because
1:43:43every node is enumerated pretty clearly
1:43:46for you which is not always the case
1:43:48when you're using other types of of more
1:43:50complex reasoning approaches for example
1:43:52like agents so in any case um I hope
1:43:56this was helpful and I definitely
1:43:58encourage you to check out um kind of
1:44:00this notion of like flow engineering
1:44:02using Lang graph and in the context of
1:44:04rag it can be really powerful hopefully
1:44:06as you've seen here thank
1:44:09you hey this is Lance from Lang chain I
1:44:12want to talk to a recent paper that I
1:44:14saw called adaptive rag which brings
1:44:16together some interesting ideas that
1:44:17have kind of been covered in other
1:44:18videos but this actually ties them all
1:44:21together in kind of a fun way so the the
1:44:23two big ideas to talk about here are one
1:44:27of query analysis so we've actually done
1:44:30kind of a whole rag from scratch series
1:44:31that walks through each of these things
1:44:32in detail but this is a very nice
1:44:35example of how this comes together um
1:44:37with some other ideas we've been talking
1:44:39about so query analysis is typically the
1:44:41process of taking an input question and
1:44:44modifying in some way uh to better
1:44:46optimize retrieval there's a bunch of
1:44:48different methods for this it could be
1:44:50decomposing it into sub questions it
1:44:52could be using some clever techniques
1:44:54like stepb back prompting um but that's
1:44:57kind of like the first stage of query
1:44:59analysis then typically you can do
1:45:01routing so you route a question to one
1:45:03of multiple potential sources it could
1:45:05be one or two different Vector stores it
1:45:08could be relational DB versus Vector
1:45:10store it could be web search it could
1:45:12just be like an llm fallback right so
1:45:15this is like one kind of big idea query
1:45:17analysis right it's kind of like the
1:45:18front end of your rag pipeline it's
1:45:20taking your question it's modifying it
1:45:22in some way it's sending it to the right
1:45:24place be it a web search be it a vector
1:45:26store be it a relational DB so that's
1:45:29kind of topic one now topic two is
1:45:32something that's been brought up in a
1:45:34few other videos um of what I kind of
1:45:36call Flow engineering or adaptive rag
1:45:40which is the idea of doing tests in your
1:45:43rag pipeline or in your rag inference
1:45:45flow uh to do things like check
1:45:47relevance documents um check whether or
1:45:51not the answer contains hallucinations
1:45:53so this recent blog post from Hamil
1:45:55Hussein actually covers evaluation in in
1:45:58some really nice detail and one of the
1:46:00things he highlighted
1:46:01explicitly is actually this topic so he
1:46:04talks about unit tests and in particular
1:46:06he says something really interesting
1:46:08here he says you know unlike typical
1:46:09unit tests you want to organize these
1:46:11assertions in places Beyond typical unit
1:46:14testing such as data cleaning and here's
1:46:17the key Point automatic retries during
1:46:19model inference that's the key thing I
1:46:21want to like draw your attention to to
1:46:23it's a really nice approach we've talked
1:46:25about some other papers that do that
1:46:26like corrective rag self rag but it's
1:46:28also cool to see it here and kind of
1:46:30encapsulated in this way the main idea
1:46:33is that you're using kind of unit tests
1:46:34in your flow to make Corrections like if
1:46:38your retrieval is bad you can correct
1:46:39from that if your generation has
1:46:41hallucinations you can correct from that
1:46:43so I'm going to kind of draw out like a
1:46:45cartoon diagram of what we're going to
1:46:47do here and you can kind of see it here
1:46:50we're starting with a question we talked
1:46:52about query analysis we're going to take
1:46:53our question and we're going to decide
1:46:55where it needs to go and for this
1:46:57particular toy example I'm going to say
1:46:59either send it to a vector store send it
1:47:01to web search or just have the llm
1:47:03answer it right so that's like kind of
1:47:05my fallback
1:47:06Behavior then we're going to bring in
1:47:08that idea of kind of online flow
1:47:10engineering or unit testing where I'm
1:47:13going to have my retrieval either from
1:47:15the VOR store or web search I'm then
1:47:17going to ask is this actually relevant
1:47:19to the question if it isn't I'm actually
1:47:21going to kick back to web sech so this
1:47:23is a little bit more relevant in the
1:47:24case if I've routed to to the vector
1:47:27store done retrieval documents aren't
1:47:29relevant I'll have a fallback
1:47:32mechanism um then I'm going to generate
1:47:34I check for hallucinations in my
1:47:36generation and then I check for um for
1:47:39whether or not the the generation
1:47:41actually answers the question then I
1:47:42return my answer so again we're tying
1:47:44together two ideas one is query analysis
1:47:47like basically taking a question routing
1:47:49it to the right place modifying it as
1:47:51needed and then kind of online unit
1:47:54testing and iterative flow
1:47:56feedback so to do this I've actually
1:48:00heard a lot of people talk online about
1:48:02command r a new model release from gooh
1:48:04here it has some pretty nice properties
1:48:07that I was kind of reading about
1:48:08recently so one it has nice support for
1:48:11Tool use and it does support query
1:48:14writing in the context of tool use uh so
1:48:18this all rolls up in really nice
1:48:20capabilities for routing it's kind of
1:48:22one now two it's small it's 35 billion
1:48:26parameter uh it's actually open weight
1:48:28so you can actually run this locally and
1:48:30I've tried that we can we can talk about
1:48:31that later uh so and it's also fast
1:48:34served via the API so it's kind of a
1:48:36small model and it's well tuned for rag
1:48:39so I heard a lot of people talking about
1:48:40using coher for Rag and it has a large
1:48:43context 120,000 tokens so this like a
1:48:45nice combination of properties it
1:48:47supports to and routing it's small and
1:48:49fast so it's like quick for grading and
1:48:51it's well tuned for rag so it's actually
1:48:54a really nice fit for this particular
1:48:55workflow where I want to do query
1:48:57analysis and routing and I want to do
1:48:59kind of online checking uh and rag so
1:49:02kind of there you go now let's just get
1:49:05to the coding bit so I have a notebook
1:49:07kind of like usual I've done a few pip
1:49:09installs you can see it's nothing exotic
1:49:11I'm bringing Lang chain coh here I set
1:49:13my coher API key now I'm just going to
1:49:16set this Lang chain project within
1:49:18lsmith so all my traces for this go to
1:49:20that project and I have enabled tracing
1:49:23so I'm using Langs Smith here so we're
1:49:25going to walk through this flow and
1:49:27let's do the first thing let's just
1:49:28build a vector store so I'm going to
1:49:30build a vector store using coherent
1:49:32beddings with chroma open source Vector
1:49:34DB runs locally from three different web
1:49:37pages on blog post that I like so it
1:49:39pertains to agents prompt engineering
1:49:41and adversarial attacks so now I have a
1:49:43retriever I can run retriever invoke and
1:49:46I can ask a question about you know
1:49:48agent
1:49:50memory agent
1:49:53memory and there we go so we get
1:49:56documents back so there we go we have a
1:49:58retriever now now here's where I'm going
1:50:01to bring in coh here I also want a
1:50:03router so you look at our flow the first
1:50:06step is this routing stage right so what
1:50:08I'm going to do is I'm guess we going to
1:50:10find two tools a web search tool and a
1:50:14vector store tool okay in my Preamble is
1:50:17just going to say you're an expert
1:50:18routing user questions to Vector store
1:50:20web search now here's the key I tell it
1:50:23what the vector store has so again my
1:50:25index my Vector store has agents prompt
1:50:28engineering adial tax I just repeat that
1:50:30here agents prompt adversarial tax so it
1:50:32knows what's in the vector store um so
1:50:35use it for questions on these topics
1:50:36otherwise use web search so that's it I
1:50:40use command R here now I'm going to bind
1:50:42these tools to the model and attach the
1:50:44Preamble and I have a structured LM
1:50:47router so let's give it a let's give
1:50:49this a few tests just to like kind of
1:50:51sandbox this a little bit
1:50:53so I can inval here's my chain I have a
1:50:55router prompt I pass that to the
1:50:56structured LM router which I defined
1:50:58right here and um let's ask a few
1:51:01different questions like who will the
1:51:02Bears draft in the NFL draft with types
1:51:05of agent memory and Hi how are you so
1:51:08I'm going to kick that off and you can
1:51:10see you know it does web search it does
1:51:13it goes to Vector store and then
1:51:14actually returns this false so that's
1:51:16kind of interesting um this is actually
1:51:19just
1:51:20saying if it does not use either tool so
1:51:23for that particular query web search or
1:51:26the vector store was inappropriate it'll
1:51:28just say hey I didn't call one of those
1:51:30tools so that's interesting we'll use
1:51:31that later so that's my router tool now
1:51:35the second thing is my grader and here's
1:51:38where I want to show you something
1:51:40really nice that is generally useful uh
1:51:43for many different problems you might
1:51:45encounter so here's what I'm doing I'm
1:51:48defining a data model uh for My Grade so
1:51:51basically grade documents it's going to
1:51:54have this is a pantic object it is just
1:51:57basically a binary score here um field
1:52:00specified here uh documents are relevant
1:52:03to the question yes no I have a preamble
1:52:05your grer assessing relevance of
1:52:06retrieve documents to a user question um
1:52:09blah blah blah so you know and then
1:52:11basically give it a b score yes no I'm
1:52:13using command R but here's the catch I'm
1:52:16using this wi structured outputs thing
1:52:18and I'm passing my grade documents uh
1:52:21data model to that that so this is the
1:52:23key thing we can test this right now as
1:52:26well it's going to return an object
1:52:29based on the schema I give it which is
1:52:31extremely useful for all sorts of use
1:52:33cases and let's actually Zoom back up so
1:52:36we're actually right here so this
1:52:39greater stage right I want to constrain
1:52:41the output to yes no I don't want any
1:52:43preambles I want anything because the
1:52:45logic I'm going to build in this graph
1:52:48is going to require a yes no binary
1:52:50response from this particular Edge in
1:52:52our graph
1:52:53so that's why this greater tool is
1:52:55really
1:52:56useful and I'm asking like a mock
1:52:59question types of agent memory I do a
1:53:01retriever I do a retrieval from our
1:53:03Vector store I get the tuck and I test
1:53:05it um I invoke our greater retrieval
1:53:08grater chain with the question the doc
1:53:11text and it's relevant as we would
1:53:13expect so that's good but again let's
1:53:16just kind of look at that a little bit
1:53:17more closely what's actually happening
1:53:19under the hood here here's the pantic
1:53:21object we passed
1:53:23here's the document in question I'm
1:53:24providing basically it's converting this
1:53:26object into coher function schema it's
1:53:29binding that to the
1:53:31llm we pass in the document question it
1:53:34returns an object basic a Json string
1:53:37per our pantic schema that's it and then
1:53:41it's just going to like parse that into
1:53:42a pantic object which we get at the end
1:53:44of the day so that's what's happening
1:53:45under the hood with this with structured
1:53:47output thing but it's extremely useful
1:53:49and you'll see we're going to use that a
1:53:50few different places um um because we
1:53:53want to ensure that in our in our flow
1:53:57here we have three different grading
1:53:58steps and each time we want to constrain
1:54:00the output to yes no we're going to use
1:54:02that structured output more than
1:54:04once um this is just my generation so
1:54:07this is good Old Rag let's just make
1:54:09sure that works um I'm using rag chain
1:54:12typical rag prompt again I'm using
1:54:15cohere for rag pretty easy and yeah so
1:54:19the rag piece works that's totally fine
1:54:21nothing to it crazy there um I'm going
1:54:24to find this llm fallback so this is
1:54:27basically if you saw a router chain if
1:54:31it doesn't use a tool I want to fall
1:54:33back and just fall back to the llm so
1:54:35I'm going to kind of build that as a
1:54:37little chain here so okay this is just a
1:54:39fallback I have my Preamble just you're
1:54:42you're an assistant answer the question
1:54:43based upon your internal knowledge so
1:54:46again that fallback behavior is what we
1:54:48have here so what we've done already is
1:54:51we defined our router piece we've
1:54:53defined our our basic retrieval our
1:54:56Vector store we already have here um
1:54:58we've defined our first logic or like
1:55:01grade check and we defined our fallback
1:55:04and we're just kind of roll through the
1:55:05parts of our graph and Define each piece
1:55:08um so I'm going to have two other
1:55:10graders and they're going to use the
1:55:11same thing we just talked about slightly
1:55:14different data model I mean same output
1:55:16but actually just slightly different uh
1:55:19prompt um and you know descript destion
1:55:22this in this case is the aners grounded
1:55:24the facts yes no this is my
1:55:25hallucination
1:55:26grater uh and then I have an answer
1:55:28grader as well and I've also run a test
1:55:31on each one and you can see I'm getting
1:55:32binary this this these objects out have
1:55:35a binary score so this a pantic object
1:55:38with a binary score uh and that's
1:55:39exactly what we want cool
1:55:43and I have a Search tool so that's
1:55:46really nice we've actually gone through
1:55:48and we've kind of laid out I have like a
1:55:50router I've tested it we have a vector
1:55:51story tested we've tested each of our
1:55:54graders here we've also tested
1:55:55generation of just doing rag so we have
1:55:58a bunch of pieces built here we have a
1:56:00fallback piece we have web search now
1:56:02the question is how do I Stitch these
1:56:04together into this kind of flow and for
1:56:06that I I like to use Lang graph we'll
1:56:09talk a little about Lang graph versus
1:56:10agents a bit later but I want to show
1:56:12you why this is really easy to do using
1:56:14Lang graph so what's kind of nice is
1:56:17I've kind of laid out all my logic here
1:56:19we've tested individually and now all
1:56:21I'm going to do
1:56:22is I'm going to first lay out uh the
1:56:25parts of my graph so what you're going
1:56:27to notice here is first there's a graph
1:56:30state so this state represents kind of
1:56:32the key parts of the graph or the key
1:56:35kind of objects within the graph that
1:56:36we're going to be modifying so this is
1:56:38basically a graph centered around rag
1:56:40we're going to have question generation
1:56:42and documents that's really kind of the
1:56:44main things we're going to be working
1:56:45with in our
1:56:46graph so then you're going to see
1:56:49something that's pretty intuitive I
1:56:51think what you're going to see is we're
1:56:53going to basically walk through this
1:56:55flow and for each of these little
1:56:57circles we're just going to find a
1:56:58function and these uh little squares or
1:57:02these these you can think about every
1:57:04Circle as a node and every kind of
1:57:06diamond here as as an edge or
1:57:08conditional Edge so that's actually what
1:57:10we're going to do right now we're going
1:57:11to lay out all of our nodes and edges
1:57:14and each one of them are just going to
1:57:15be a function and you're going to see
1:57:16how we do that right now so I'm going to
1:57:20go down here I def find my graph state
1:57:22so this is what's going to be kind of
1:57:23modified and propagated throughout my
1:57:24graph now all I'm going to do is I'm
1:57:27just going to find a function uh for
1:57:29each of those nodes so let me kind of go
1:57:32side by side and show you the diagram
1:57:33and then like kind of show the nodes
1:57:35next to it so here's the
1:57:38diagram so we have uh a retrieve node so
1:57:42that kind of represents our Vector store
1:57:43we have a fallback node that's this
1:57:46piece we have a generate node so that's
1:57:49basically going to do our rag you can
1:57:51see there we have a grade documents node
1:57:54kind of right
1:57:55here um and we have a web search node so
1:57:58that's right here cool now here's where
1:58:02we're actually to find the edges so you
1:58:04can see our edges are the pieces of the
1:58:05graph that are kind of making different
1:58:06decisions so this route question Edge
1:58:09basic conditional Edge is basically
1:58:12going to take an input question and
1:58:14decide where it needs to go and that's
1:58:15all we're doing down here it kind of
1:58:18follows what we did up at the top where
1:58:20we tested this individually so recall we
1:58:23basically just invoke that question
1:58:24router returns our source now remember
1:58:27if tool calls were not in the source we
1:58:30do our fall back so we show actually
1:58:32showed that all the way up here remember
1:58:34this if tool calls is not in the
1:58:36response this thing will just be false
1:58:38so that means we didn't either we didn't
1:58:39call web search and we didn't call uh
1:58:43our retriever tool so then we're just
1:58:45going to fall
1:58:46back
1:58:48um yep right here and this is just like
1:58:53uh you know a catch just in case a tool
1:58:54could make a decision but most
1:58:57interestingly here's where we choose a
1:58:59data source basically so um this is the
1:59:02output of our tool call we're just going
1:59:05to fish out the name of the tool so
1:59:08that's data source and then here we go
1:59:10if the data source is web search I'm
1:59:12returning web search as basically the
1:59:14next node to go to um otherwise if it's
1:59:18Vector store we return Vector store as
1:59:19the next node to go to so what's this
1:59:22search thing well remember we right up
1:59:24here Define this node web search that's
1:59:27it we're just going to go to that node
1:59:30um what's this Vector store um you'll
1:59:32see below how we can kind of tie these
1:59:35strings that we returned from the
1:59:36conditional Edge to the node we want to
1:59:38go to that's really it um same kind of
1:59:42thing here decide to generate that's
1:59:44going to roll in these two conditional
1:59:46edges into one um and basically it's
1:59:49going to do if there's no documents so
1:59:51basic basically if we filtered out all
1:59:54of our documents from this first test
1:59:56here um then what we're going to do is
2:00:01we've decided all documents are not
2:00:02relevant to the question and we're going
2:00:04to kick back to web search exactly as we
2:00:06show here so that's this piece um
2:00:09otherwise we're going to go to generate
2:00:11so that's this piece so again in these
2:00:14conditional edges you're basically
2:00:15implementing the logic that you see in
2:00:17our diagram right here that's all that's
2:00:19going on um and again this is just
2:00:23implementing the final two checks uh for
2:00:25hallucinations and and answer
2:00:27relevance um
2:00:30and um yep so here's our hallucination
2:00:34grader we then extract the grade if the
2:00:38if basically there are
2:00:39hallucinations um oh sorry in this case
2:00:42the grade actually yes means that the
2:00:45answer is grounded so we say answer is
2:00:48actually grounded and then we go to the
2:00:50next step we go to the next test that's
2:00:52all this is doing it's just basically
2:00:53wrapping this logic that we're
2:00:55implementing here in our graph so that's
2:00:57all that's going on and let's go ahead
2:00:59and Define all those things so nice we
2:01:02have all that um now we can actually go
2:01:05down a little bit and we can pull
2:01:09um this is actually where we stitch
2:01:12together everything so all it's
2:01:13happening here is you see we defined all
2:01:16these functions up here we just add them
2:01:18as nodes in our graph here and then we
2:01:21build our graph here basically by by
2:01:24basically laying out the flow or the
2:01:26connectivity between our nodes and edges
2:01:29so you know you can look at this
2:01:30notebook to kind of study in a bit of
2:01:31detail what's going on but frankly what
2:01:33I like to do here typically just draw
2:01:36out a graph kind of like we did up
2:01:38here and then Implement uh the Lo
2:01:42logical flow here in your graph as nodes
2:01:45and edges just like we're doing here
2:01:47that's all that's happening uh so again
2:01:49we have like our entry point is the
2:01:51router
2:01:52um this is like the output is this is
2:01:55basically directing like here's what the
2:01:57router is outputting and here's the next
2:02:00node to go to so that's it um and then
2:02:03for each node we're kind of applying
2:02:05like we're saying like what's what's the
2:02:06flow so web search goes to generate
2:02:09after um and retrieve goes to grade
2:02:12documents grade documents um kind of is
2:02:16is like is a conditional Edge um
2:02:18depending on the results we either do
2:02:19web search or generate and then our
2:02:21second one we go from generate to uh
2:02:25basically this grade uh generation
2:02:27versus documents in question based on
2:02:29the output of that we either have
2:02:31hallucinations we regenerate uh we found
2:02:34that the answer is not useful we kick
2:02:36back to web search or we end um finally
2:02:39we have that llm fallback and that's
2:02:41also if we go to the fallback we end so
2:02:44what you're seeing here is actually the
2:02:46the logic flow we're laying out in this
2:02:49graph matches the diagram
2:02:52that we laid out up top I'm just going
2:02:54to copy these over and I'll actually go
2:02:56then back to the diagram and and kind of
2:02:57underscore that a little bit more so
2:03:01here is the flow we've laid out again
2:03:03here is our diagram and you can kind of
2:03:05look at them side by side and see how
2:03:07they basically match up so here's kind
2:03:10of our flow diagram going from basically
2:03:12query analysis that's this thing this
2:03:15route question and you can see web
2:03:18search Vector store LM fallback LM
2:03:20fallback web search vector store so
2:03:22those are like the three options that
2:03:23can come out of this conditional Edge
2:03:26and then here's where we connect so if
2:03:27we go to web search then basically we
2:03:30next go to
2:03:31generate so that's kind of this whole
2:03:34flow um now if we go to
2:03:38retrieve um then we're going to grade so
2:03:41that's
2:03:42it um and you know it follows kind of as
2:03:46you can see here that's really it uh so
2:03:49it's just nice to draw the these
2:03:51diagrams out first and then it's pretty
2:03:54quick to implement each node and each
2:03:56Edge just as a function and then stitch
2:03:59them together in a graph just like I
2:04:00show here and of course we'll make sure
2:04:01this code's publ so you can use it as a
2:04:03reference um so there we go now let's
2:04:07try a few a few different test questions
2:04:10so like what player the Bears to draft
2:04:12and NFL draft right let's have a look at
2:04:14that and they should print everything
2:04:16it's doing as we go so okay this is
2:04:19important route question it just decides
2:04:21to route to web search that's good it
2:04:23doesn't go to our Vector store this is a
2:04:24current event not related to our Vector
2:04:26store at all it goes to web search um
2:04:29and then it goes to generate so that's
2:04:30what we'd expect so basically web search
2:04:32goes through to generate
2:04:35um and we check hallucinations
2:04:38Generations ground the documents we
2:04:40check generation versus question the
2:04:42generation addresses the question the
2:04:43Chicago Bears expected to draft Caleb
2:04:45Williams that's right that's that's the
2:04:48consensus so cool that works now let's
2:04:50ask a question related to our Vector
2:04:52store what are the types of agent memory
2:04:54we'll kick this off so we're routing
2:04:56okay we're routing to rag now look how
2:04:58fast this is that's really fast so we
2:05:01basically whip through that document
2:05:03grading determine they're all relevant
2:05:06uh we go to decision to
2:05:07generate um we check hallucinations we
2:05:10check answer versus question and there
2:05:13are several types of memory stored in
2:05:15the human brain memory can also be
2:05:16stored in G of Agents you have LM agents
2:05:19memory stream retrieval model and and
2:05:20reflection mechanism so it's
2:05:22representing what's captured on the blog
2:05:23post pretty reasonably now let me show
2:05:25you something else is kind of nice I can
2:05:27go to Langs Smith and I can go to my
2:05:29projects we create this new project
2:05:31coher adaptive rag at the start and
2:05:33everything is actually logged there
2:05:35everything we just did so I can open
2:05:37this up and I can actually just kind of
2:05:40look through all the stages of my Lang
2:05:42graph to here's my retrieval stage um
2:05:45here's my grade document stage and we
2:05:47can kind of audit the grading itself we
2:05:49kind of looked at this one by one
2:05:51previously but it's actually pretty nice
2:05:53we can actually audit every single
2:05:54individual document grade to see what's
2:05:56happening um we can basically go through
2:06:00um to this is going to be one of the
2:06:02other graders here
2:06:05um yep so this is actually going to be
2:06:09the hallucination grading right here uh
2:06:12and then this is going to be the answer
2:06:13grading right here so that's really it
2:06:15you can kind of walk through the entire
2:06:16graph you can you can kind of study
2:06:18what's going on um which is actually
2:06:20very useful so it looks like this worked
2:06:23pretty well um and finally let's just
2:06:25ask a question that should go to that
2:06:27fallback uh path down at the bottom like
2:06:30not related at all to our Vector store
2:06:32current events and yeah hello I'm doing
2:06:34well so it's pretty neat we've seen in
2:06:36maybe 15 minutes we've from scratch
2:06:38built basically a semi- sophisticated
2:06:40rag system that has agentic properties
2:06:42we've done in Lang graph we've done with
2:06:44coher uh command R you can see it's
2:06:46pretty darn fast in fact we can go to
2:06:48Langs Smith and look at so this whole
2:06:50thing took 7 seconds uh that is not bad
2:06:54let's look at the most recent one so
2:06:56this takes one second so the fallback
2:06:58mechanism to the LM is like 1 second um
2:07:01the let's just look here so 6 seconds
2:07:04for the initial uh land graph so this is
2:07:08not bad at all it's quite fast it done
2:07:10it does quite a few different checks we
2:07:12do routing uh and then we have kind of a
2:07:15bunch of nice fallback behavior and
2:07:17inline checking uh for both relevance
2:07:20hallucinations and and answer uh kind of
2:07:23groundedness or answer usefulness so you
2:07:27know this is pretty nice I definitely
2:07:28encourage you to play with a notebook
2:07:30command R is a really nice option for
2:07:31this due to the fact that is tool use
2:07:33routing uh small and quite fast and it's
2:07:36really good for Rags it's a very nice
2:07:38kind of uh a very nice option for
2:07:41workflows like this and I think you're
2:07:43going to see more and more of this kind
2:07:44of like uh adaptive or self-reflective
2:07:47rag um just because this is something
2:07:50that a lot systems can benefit from like
2:07:53a a lot of production rack systems kind
2:07:55of don't necessarily have
2:07:57fallbacks uh depending on for example
2:08:00like um you know if the documents
2:08:03retrieved are not relevant uh if the
2:08:05answer contains hallucinations and so
2:08:07forth so this opportunity to apply
2:08:09inline checking along with rag is like a
2:08:11really nice theme I think we're going to
2:08:13see more and more of especially as model
2:08:15inference gets faster and faster and
2:08:17these checks get cheaper and cheaper to
2:08:18do kind of in the inference Loop
2:08:22now as a final thing I do want to bring
2:08:23up the a point about you know we've
2:08:25shown this Lang graph stuff what about
2:08:27agents you know how do you think about
2:08:28agents versus Lang graph right and and I
2:08:30think the way I like to frame this is
2:08:33that um Lang graph is really good
2:08:37for um flows that you have kind of very
2:08:41clearly defined that don't have like
2:08:43kind of open-endedness but like in this
2:08:45case we know the steps we want to take
2:08:47every time we want to do um basically
2:08:49query analysis routing and then we want
2:08:51to do a three grading steps and that's
2:08:53it um Lang graph is really good for
2:08:56building very reliable flows uh it's
2:08:58kind of like putting an agent on guard
2:09:00rails and it's really nice uh it's less
2:09:04flexible but highly reliable and so you
2:09:07can actually use smaller faster models
2:09:08with langra so that's the thing I like
2:09:10about we saw here command R 35 billion
2:09:12parameter model works really well with
2:09:14langra quite quick we' were able to
2:09:16implement a pretty sophisticated rag
2:09:18flow really quickly 15 minutes um in
2:09:21time is on the order of like less than
2:09:23you know around 5 to 6 seconds so so
2:09:25pretty good right now what about agents
2:09:28right so I think Agents come into play
2:09:30when you want more flexible workflows
2:09:32you don't want to necessarily follow a
2:09:34defined pattern a priori you want an
2:09:36agent to be able to kind of reason and
2:09:38make of open-end decisions which is
2:09:40interesting for certain like long
2:09:41Horizon planning problems you know
2:09:43agents are really
2:09:44interesting the catch is that
2:09:46reliability is a bit worse with agents
2:09:48and so you know that's a big question a
2:09:50lot of people bring up and that's kind
2:09:52of where larger LMS kind of come into
2:09:53play with a you know there's been a lot
2:09:55of questions about using small LMS even
2:09:57open source models with agents and
2:09:59reliabilities kind of continuously being
2:10:01an issue whereas I've been able to run
2:10:03these types of land graphs with um with
2:10:06uh like mraw or you know command R
2:10:09actually is open weights you can run it
2:10:10locally um I've been able to run them
2:10:12very reproducibly with open source
2:10:14models on my laptop um so you know I
2:10:17think there's a tradeoff and Comm
2:10:20actually there's a new coher model
2:10:21coming out uh believe command R plus
2:10:25which uh is a larger model so it's
2:10:27probably more suitable for kind of more
2:10:29open-ended agentic use cases and there's
2:10:32actually a new integration with Lang
2:10:33chain that support uh coher agents um
2:10:36which is quite nice so I think it's it's
2:10:38worth experimenting for certain problems
2:10:40in workflows you may need more
2:10:42open-ended reasoning in which case use
2:10:43an agent with a larger model otherwise
2:10:46you can use like Lang graph for more uh
2:10:49a more reliable potential
2:10:51but con strain flow and it can also use
2:10:53smaller models faster LMS so those are
2:10:56some of the trade-offs to keep in mind
2:10:57but anyway encourage you play with a
2:10:59notebook explore for yourself I think
2:11:01command R is a really nice model um I've
2:11:03also been experimenting with running it
2:11:04locally with AMA uh currently the
2:11:07quantise model is like uh two bit
2:11:09quantise is like 13 billion uh or so uh
2:11:14yeah 13 gigs it's it's a little bit too
2:11:17large to run quickly locally for me
2:11:22um inference for things like rag we're
2:11:25on the order of 30 seconds so again it's
2:11:27not great for a live demo but it does
2:11:29work it is available on a llama so I
2:11:30encourage you to play with that I have a
2:11:32Mac M2 32 gig um so you know if I if
2:11:35you're a larger machine then it
2:11:36absolutely could be worth working with
2:11:38locally so encourage you to play with
2:11:40that anyway hopefully this was useful
2:11:41and interesting I think this is a cool
2:11:43paper cool flow um coher command R is a
2:11:46nice option for these types of like
2:11:48routing uh it's quick good with Lang
2:11:52graph good for rag good for Tool use so
2:11:55you know have a have a look and uh you
2:11:57know reply anything uh any feedback in
2:12:00the comments
2:12:06thanks hi this is Lance from Lang chain
2:12:09this is a talk I gave at two recent
2:12:10meetups in San Francisco called is rag
2:12:13really dead um and I figured since you
2:12:16know a lot of people actually weren't
2:12:17able to make those meetups uh I just
2:12:19record this and put this on YouTube and
2:12:21see if this is of interest to folks um
2:12:24so we all kind of recognize that Contex
2:12:26windows are getting larger for llms so
2:12:29on the x-axis you can see the tokens
2:12:31used in pre-training that's of course
2:12:33you know getting larger as well um
2:12:35proprietary models are somewhere over
2:12:37the two trillion token regime we don't
2:12:39quite know where they sit uh and we've
2:12:41all the way down to smaller models like
2:12:4352 trained on far fewer tokens um but
2:12:46what's really notable is on the y axis
2:12:49you can see about a year ago da the art
2:12:52models were on the order of 4,000 to
2:12:548,000 tokens and that's you know dozens
2:12:56of pages um we saw Claude 2 come out
2:12:59with the 200,000 token model earlier I
2:13:02think it was last year um gbd4 128,000
2:13:06tokens now that's hundreds of pages and
2:13:09now we're seeing Claud 3 and Gemini come
2:13:11out with million token models so this is
2:13:13hundreds to thousands of pages so
2:13:16because of this phenomenon people have
2:13:17been kind of wondering is rag dead if
2:13:19you can stuff you know many thousands of
2:13:22pages into the context window llm why do
2:13:24you need a reteval system um it's a good
2:13:27question spoke sparked a lot of
2:13:29interesting debate on Twitter um and
2:13:32it's maybe first just kind of grounding
2:13:33on what is rag so rag is really the
2:13:35process of reasoning and retrieval over
2:13:37chunks of of information that have been
2:13:40retrieved um it's starting with you know
2:13:42documents that are indexed um they're
2:13:45retrievable through some mechanism
2:13:47typically some kind of semantic
2:13:48similarity search or keyword search
2:13:50other mechanisms
2:13:51retriev docs should then pass to an llm
2:13:54and the llm reasons about them to ground
2:13:56response to the question in the retrieve
2:13:58document so that's kind of the overall
2:14:00flow but the important point to make is
2:14:02that typically it's multiple documents
2:14:04and involve some form of
2:14:06reasoning so one of the questions I
2:14:08asked recently is you know if long
2:14:10condex llms can replace rag it should be
2:14:12able to perform you know multia
2:14:15retrieval and reasoning from its own
2:14:16context really effectively so I teamed
2:14:19up with Greg Cameron uh to kind of
2:14:21pressure test this and he had done some
2:14:23really nice needle the Haack analyses
2:14:25already focused on kind of single facts
2:14:28called needles placed in a Hy stack of
2:14:31Paul Graham essays um so I kind of
2:14:33extended that to kind of mirror the rag
2:14:36use case or kind of the rag context uh
2:14:39where I took multiple facts so I call it
2:14:41multi needle um I buil on a funny needle
2:14:45in the HTO challenge published by
2:14:46anthropic where they add they basically
2:14:48placed Pizza ingredients in the context
2:14:51uh and asked the LM to retrieve this
2:14:53combination of pizza ingredients I did I
2:14:56kind of Rift on that and I basically
2:14:58split the pizza ingredients up into
2:14:59three different needles and place those
2:15:01three ingredients in different places in
2:15:03the context and then ask the um to
2:15:06recover those three ingredients um from
2:15:09the context so again the setup is the
2:15:12question is what the secret ingredients
2:15:13need to build a perfect Pizza the
2:15:15needles are the ingredients figs Pudo
2:15:17goat cheese um I place them in the
2:15:21context at some specified intervals the
2:15:23way this test works is you can basically
2:15:25set the percent of context you want to
2:15:28place the first needle and the remaining
2:15:30two are placed at roughly equal
2:15:31intervals in the remaining context after
2:15:33the first so that's kind of the way the
2:15:35test is set up now it's all open source
2:15:36by the way the link is below so needs
2:15:39are placed um you ask a question you
2:15:42promp L them with with kind of um with
2:15:45this context and the question and then
2:15:47produces the answer and now the the
2:15:48framework will grade the response
2:15:51both one are you know all are all the
2:15:55the specified ingredients present in the
2:15:56answer and two if not which ones are
2:16:00missing so I ran a bunch of analysis on
2:16:03this with GPD 4 and came kind of came up
2:16:05with some with some fun results um so
2:16:07you can see on the left here what this
2:16:09is looking at is different numbers of
2:16:11needles placed in 120,000 token context
2:16:14window for
2:16:15gbd4 and I'm asking um gbd4 to retrieve
2:16:20either one three or 10 needles now I'm
2:16:24also asking it to do reasoning on those
2:16:26needles that's what you can see in those
2:16:28red bars so green is just retrieve the
2:16:30ingredients red is reasoning and the
2:16:32reasoning challenge here is just return
2:16:34the first letter of each ingredient so
2:16:37we find is basically two things the
2:16:39performance or the percentage of needles
2:16:42retrieved drops with respect to the
2:16:44number of needles that's kind of
2:16:45intuitive you place more facts
2:16:48performance gets worse but also it gets
2:16:50worse if you ask it to reason so if you
2:16:53say um just return the needles it does a
2:16:56little bit better than if you say return
2:16:58the needles and tell me the first letter
2:17:00so you overlay reasoning so this is the
2:17:02first observation more facts is harder
2:17:06uh and reasoning is harder uh than just
2:17:09retrieval now the second question we ask
2:17:11is where are these needles actually
2:17:12present in the context that we're
2:17:14missing right so we know for example um
2:17:18retrieval of um 10 needles is around 60%
2:17:23so where are the missing needles in the
2:17:26context so on the right you can see
2:17:28results telling us actually which
2:17:30specific needles uh are are the model
2:17:33fails to retrieve so what we can see is
2:17:36as you go from a th000 tokens up to
2:17:39120,000 tokens on the X here and you
2:17:42look at needle one place at the start of
2:17:44the document to needle 10 placed at the
2:17:46end at a th000 token context link you
2:17:49can retrieve them all so again kind of
2:17:52match what we see over here small well
2:17:54actually sorry over here everything I'm
2:17:56looking at is 120,000 tokens so that's
2:17:59really not the point uh the point is
2:18:01actually smaller context uh better
2:18:04retrieval so that's kind of point one um
2:18:08as I increase the context window I
2:18:10actually see that uh there is increased
2:18:14failure to retrieve needles which you
2:18:15see can see in red here towards the
2:18:18start of the
2:18:19document um and so this is an
2:18:21interesting result um and it actually
2:18:23matches what Greg saw with single needle
2:18:25case as well so the way to think about
2:18:27it is it appears that um you know if you
2:18:31for example read a book and I asked you
2:18:33a question about the first chapter you
2:18:34might have forgotten it same kind of
2:18:36phenomenon appears to happen here with
2:18:38retrieval where needles towards the
2:18:40start of the context are are kind of
2:18:42Forgotten or are not well retrieved
2:18:45relative to those of the end so this is
2:18:47an effect we see with gbd4 it's been
2:18:49reproduced quite a bit so ran nine
2:18:51different trials here Greg's also seen
2:18:53this repeatedly with single needle so it
2:18:55seems like a pretty consistent
2:18:57result and there's an interesting point
2:18:59I put this on Twitter and a number of
2:19:00folks um you know replied and someone
2:19:03sent me this paper which is pretty
2:19:04interesting and it mentions recency bias
2:19:07is one possible reason so the most
2:19:09informative tokens for predicting the
2:19:10next token uh you know are are are
2:19:14present close to or recent to kind of
2:19:17where you're doing your generation and
2:19:18so there's a bias to attend to recent
2:19:20tokens which is obviously not great for
2:19:23the retrieval problem as we saw here so
2:19:26again the results show us that um
2:19:30reasoning is a bit harder than retrieval
2:19:32more needles is more difficult and
2:19:34needles towards the start of your
2:19:36context are harder to retrieve than
2:19:38towards the end those are three main
2:19:40observations from this and it maybe
2:19:42indeed due to this recency bias so
2:19:45overall what this kind of tells you is
2:19:47be wary of just context stuffing in
2:19:49large long context
2:19:51there are no retrieval
2:19:52guarantees and also there's some recent
2:19:55results that came out actually just
2:19:56today suggesting that single needle may
2:19:58be misleadingly easy um you know there's
2:20:02no reasoning it's retrieving a single
2:20:04needle um and also these guys I'm I
2:20:08showed this tweet here show that um the
2:20:12in a lot of these needle and Haack
2:20:13challenges including mine the facts that
2:20:16we look for are very different than um
2:20:20the background kind of Hy stack of Paul
2:20:21Graham essays and so that may be kind of
2:20:23an interesting artifact they note that
2:20:25indeed if the needle is more subtle
2:20:28retrievals is worse so I think basically
2:20:31when you see these really strong
2:20:33performing needle and hyack analyses put
2:20:35up by model providers you should be
2:20:37skeptical um you shouldn't necessarily
2:20:39assume that you're going to get high
2:20:40quality retrieval from these long
2:20:41contact LMS uh for numerous reasons you
2:20:45need to think about retrieval of
2:20:46multiple facts um you need to think
2:20:48about reasoning on top of retrieval you
2:20:50need need to think about the subtlety of
2:20:52the retrieval relative to the background
2:20:54context because for many of these needle
2:20:56and the Haack challenges it's a single
2:20:57needle no reasoning and the needle
2:21:00itself is very different from the
2:21:01background so anyway those may all make
2:21:03the challenge a bit easier than a real
2:21:05world scenario of fact retrieval so I
2:21:07just want to like kind of lay out that
2:21:09those cautionary notes but you know I
2:21:13think it is fair to say this will
2:21:14certainly get better and I think it's
2:21:17also fair to say that rag will change
2:21:19and this is just like a nearly not a
2:21:21great joke but Frank zap a musician made
2:21:24the point Jazz isn't dead it just smells
2:21:26funny you know I think same for rag rag
2:21:28is not dead but it will change I think
2:21:30that's like kind of the key Point here
2:21:32um so just as a followup on that rag
2:21:35today's focus on precise retrieval of
2:21:37relevant doc chunks so it's very focused
2:21:39on typically taking documents chunking
2:21:42them in some particular way often using
2:21:44very OS syncratic chunking methods
2:21:46things like chunk size are kind of
2:21:48picked almost arbitrarily embeding them
2:21:50storing them in an index taking a
2:21:52question embedding it doing K&N uh
2:21:55similarity search to retrieve relevant
2:21:57chunks you're often setting a k
2:21:59parameter which is the number of chunks
2:22:00you retrieve you often will do some kind
2:22:02of filtering or Pro processing on the
2:22:04retrieve chunks and then ground your
2:22:06answer in those retrieved chunks so it's
2:22:08very focused on precise retrieval of
2:22:10just the right chunks now in a world
2:22:14where you have very long context models
2:22:16I think there's a fair question to ask
2:22:18is is this really kind of the most most
2:22:20reasonable approach so kind of on the
2:22:22left here you can kind of see this
2:22:25notion closer to today of I need the
2:22:27exact relevant chunk you can risk over
2:22:29engineering you can have you know higher
2:22:31complexity sensitivity to these odd
2:22:33parameters like chunk size k um and you
2:22:36can indeed suffer lower recall because
2:22:38you're really only picking very precise
2:22:40chunks you're beholden to very
2:22:41particular embedding models so you know
2:22:44I think going forward as long context
2:22:46models get better and better there are
2:22:48definitely question you should certainly
2:22:50question the current kind of very
2:22:52precise chunking rag Paradigm but on the
2:22:54flip side I think just throwing all your
2:22:56docs into context probably will also not
2:22:59be the preferred approach you'll suffer
2:23:01higher latency higher token usage I
2:23:03should note that today 100,000 token GPD
2:23:064 is like $1 per generation I spent a
2:23:09lot of money on Lang Chain's account uh
2:23:11on that multile analysis I don't want to
2:23:13tell Harrison how much I spent uh so
2:23:16it's it's you know it's not good right
2:23:18um You Can't audit retrieve
2:23:21um and security and and authentication
2:23:23are issues if for example you need
2:23:25different users different different
2:23:26access to certain kind of retriev
2:23:28documents or chunks in the Contex
2:23:30stuffing case you you kind of can't
2:23:32manage security as easily so there's
2:23:33probably some predo optimal regime kind
2:23:35of here in the middle and um you know I
2:23:39I put this out on Twitter I think
2:23:40there's some reasonable points raised I
2:23:42think you know this inclusion at the
2:23:44document level is probably pretty sane
2:23:46documents are self-contained chunks of
2:23:48context um so you know what about
2:23:51document Centric rag so no chunking uh
2:23:54but just like operate on the context of
2:23:56full documents so you know if you think
2:23:59forward to the rag Paradigm that's
2:24:01document Centric you still have the
2:24:03problem of taking an input question
2:24:05routing it to the right document um this
2:24:07doesn't change so I think a lot of
2:24:09methods that we think about for kind of
2:24:11query analysis um taking an input
2:24:14question rewriting it in a certain way
2:24:16to optimize retrieval things like
2:24:18routing taking a question routing to the
2:24:20right database be it a relational
2:24:22database graph database Vector store um
2:24:25and quer construction methods so for
2:24:27example text to SQL text to Cipher for
2:24:30graphs um or text to even like metadata
2:24:32filters for for Vector stores those are
2:24:35all still relevant in the world that you
2:24:37have long Contex llms um you're probably
2:24:40not going to dump your entire SQL DB and
2:24:42feed that to the llm you're still going
2:24:43to have SQL queries you're still going
2:24:45to have graph queries um you may be more
2:24:48permissive with what you extract but it
2:24:50still is very reasonable to store the
2:24:52majority of your structured data in
2:24:53these in these forms likewise with
2:24:55unstructured data like documents like we
2:24:58said before it still probably makes
2:25:00sense to ENC to you know store documents
2:25:02independently but just simply aim to
2:25:04retrieve full documents rather than
2:25:05worrying about these idiosyncratic
2:25:07parameters like like chunk size um and
2:25:10along those lines there's a lot of
2:25:12methods out there we've we've done a few
2:25:14of these that are kind of well optimized
2:25:16for document retrieval so one I want a
2:25:18flag is what we call multi repesent
2:25:20presentation indexing and there's
2:25:21actually a really nice paper on this
2:25:23called dense X retriever or proposition
2:25:25indexing but the main point is simply
2:25:27this would you do is you take your OD
2:25:29document you produce a representation
2:25:31like a summary of that document you
2:25:33index that summary right and then um at
2:25:37retrieval time you ask your question you
2:25:39embed your question and you simply use a
2:25:41highle summary to just retrieve the
2:25:43right document you pass the full
2:25:45document to the LM for a kind of final
2:25:48generation so it's kind of a trick where
2:25:51you don't have to worry about embedding
2:25:52full documents in this particular case
2:25:54you can use kind of very nice
2:25:56descriptive summarization prompts to
2:25:58build descriptive summaries and the
2:26:00problem you're solving here is just get
2:26:02me the right document it's an easier
2:26:04problem than get me the right chunk so
2:26:06this is kind of a nice approach it
2:26:08there's also different variants of it
2:26:10which I share below one is called parent
2:26:11document retriever where you could use
2:26:13in principle if you wanted smaller
2:26:15chunks but then just return full
2:26:17documents but anyway the point is
2:26:19preserving full documents for Generation
2:26:21but using representations like summaries
2:26:23or chunks for retrieval so that's kind
2:26:25of like approach one that I think is
2:26:27really interesting approach two is this
2:26:30idea of raptor is a cool paper came out
2:26:32of Stamper somewhat recently and this
2:26:34solves the problem of what if for
2:26:36certain questions I need to integrate
2:26:38information across many documents so
2:26:41what this approach does is it takes
2:26:42documents and it it embeds them and
2:26:45clusters them and then it summarizes
2:26:47each cluster um and it does this
2:26:49recursively in up with only one very
2:26:51high level summary for the entire Corpus
2:26:53of documents and what they do is they
2:26:55take this kind of this abstraction
2:26:57hierarchy so to speak of different
2:26:59document summarizations and they just
2:27:01index all of it and they use this in
2:27:03retrieval and so basically if you have a
2:27:05question that draws an information
2:27:07across numerous documents you probably
2:27:09have a summary present and and indexed
2:27:12that kind of has that answer captured so
2:27:15it's a nice trick to consolidate
2:27:16information across documents um they
2:27:19they paper actually reports you know
2:27:22these documents in their case or the
2:27:23leavs are actually document chunks or
2:27:25slices but I actually showed I have a
2:27:28video on it and a notebook that this
2:27:29works across full documents as well um
2:27:33and this and I segue into to do this you
2:27:36do need to think about long context
2:27:37embedding models because you're
2:27:38embedding full documents and that's a
2:27:40really interesting thing to track um the
2:27:43you know hazy research uh put out a
2:27:45really nice um uh blog post on this
2:27:48using uh what the Monch mixer so it's
2:27:50kind of a new architecture that tends to
2:27:53longer context they have a 32,000 token
2:27:56embedding model that's pres that's
2:27:58available on together AI absolutely
2:27:59worth experimenting with I think this is
2:28:01really interesting Trend so long long
2:28:03Contex and beddings kind of play really
2:28:05well with this kind of idea you take
2:28:07full documents embed them using for
2:28:09example long Contex embedding models and
2:28:11you can kind of build these document
2:28:12summarization trees um really
2:28:14effectively so I think this another nice
2:28:16trick for working with full documents in
2:28:19the long context kind of llm regime um
2:28:23one other thing I'll note I think
2:28:25there's also going to Mo be move away
2:28:27from kind of single shot rag well
2:28:29today's rag we typically you know we
2:28:31chunk documents uh uh embed them store
2:28:34them in an index you know do retrieval
2:28:36and then do generation but there's no
2:28:38reason why you shouldn't kind of do
2:28:40reasoning on top of the generation or
2:28:42reasoning on top of the retrieval and
2:28:44feedback if there are errors so there's
2:28:46a really nice paper called selfrag um
2:28:49that kind of reports this we implemented
2:28:50this using Lang graph works really well
2:28:53and the simp the idea is simply to you
2:28:55know grade the relevance of your
2:28:57documents relative to your question
2:28:59first if they're not relevant you
2:29:01rewrite the question you can do you can
2:29:02do many things in this case we do
2:29:04question rewriting and try again um we
2:29:06also grade for hallucinations we grade
2:29:09for answer relevance but anyway it kind
2:29:11of moves rag from like a single shot
2:29:12Paradigm to a kind of a cyclic flow uh
2:29:16in which you actually do various
2:29:17gradings Downstream and this is all
2:29:19relev in the long context llm regime as
2:29:22well in fact you know it you you
2:29:24absolutely should take advantage of of
2:29:27for example increasingly fast and
2:29:30Performing LMS to do this grading um
2:29:33Frameworks like langra allow you to
2:29:35build these kind of these flows which
2:29:37build which allows you to kind of have a
2:29:38more performant uh kind of kind of
2:29:42self-reflective rag pipeline now I did
2:29:44get a lot of questions about latency
2:29:46here and I completely agree there's a
2:29:47trade-off between kind of performance
2:29:49accuracy and latency that's present here
2:29:51I think the real answer is you can opt
2:29:54to use very fast uh for example models
2:29:57like grock where seeing um you know gp35
2:30:00turbos very fast these are fairly easy
2:30:03grading challenges so you can use very
2:30:05very fast LMS to do the grading and for
2:30:07example um you you can also restrict
2:30:11this to only do one turn of of kind of
2:30:13cyclic iteration so you can kind of
2:30:14restrict the latency in that way as well
2:30:16so anyway I think it's a really cool
2:30:18approach still relevant in the world as
2:30:20we move towards longer context so it's
2:30:22kind of like building reasoning on top
2:30:23of rag um in the uh generation and
2:30:28retrieval stages and a related point one
2:30:30of the challenges with rag is that your
2:30:33index for example you you may have a
2:30:36question that is that asks something
2:30:38that's outside the scope of your index
2:30:39and this is kind of always a problem so
2:30:41a really cool paper called c c rag or
2:30:44corrective rag came out you know a
2:30:45couple months ago that basically does
2:30:47grading just like we talked about before
2:30:50and then if the documents are not
2:30:51relevant you kick off and do a web
2:30:53search and basically return the search
2:30:55results to the LM for final generation
2:30:57so it's a nice fallback in cases where
2:31:00um your you the questions out of the
2:31:02domain of your retriever so you know
2:31:04again nice trick overlaying reasoning on
2:31:07top of rag I think this trend you know
2:31:09continues um because you know it it just
2:31:12it makes rag systems you know more
2:31:14performant uh and less brittle to
2:31:18questions that are out of domain so you
2:31:19know you know that's another kind of
2:31:21nice idea this particular approach also
2:31:23we showed works really well with with uh
2:31:25with open source models so I ran this
2:31:27with mraw 7B it can run locally on my
2:31:29laptop using a llama so again really
2:31:32nice approach I encourage you to look
2:31:33into this um and this is all kind of
2:31:35independent of the llm kind of context
2:31:37length these are reasoning you can add
2:31:40on top of the retrieval stage that that
2:31:42can kind of improve overall performance
2:31:45and so the overall picture kind of looks
2:31:47like this where you know I think that
2:31:49the the the the problem of routing your
2:31:52question to the right database Andor to
2:31:54the right document kind of remains in
2:31:56place query analysis is still quite
2:31:58relevant routing is still relevant query
2:32:00construction is still relevant um in the
2:32:02long Contex regime I think there is less
2:32:04of an emphasis on document chunking
2:32:07working with full documents is probably
2:32:08kind of more parto optimal so to speak
2:32:11um there's some some clever tricks for
2:32:13IND indexing of documents like the
2:32:15multi-representation indexing we talked
2:32:16about the hierarchical indexing using
2:32:19Raptor that we talked about as well are
2:32:20two interesting ideas for document
2:32:22Centric indexing um and then kind of
2:32:25reasoning in generation post retrieval
2:32:28on retrieval itself tog grade on the
2:32:30generations themselves checking for
2:32:32hallucinations those are all kind of
2:32:34interesting and relevant parts of a rag
2:32:36system that I think we'll probably will
2:32:38see more and more of as we move more
2:32:40away from like a more naive prompt
2:32:42response Paradigm more to like a flow
2:32:44Paradigm we're seeing that actually
2:32:45already in codenation it's probably
2:32:47going to carry over to rag as well where
2:32:49we kind of build rag systems that have
2:32:51kind of a cyclic flow to them operate on
2:32:53documents use longc Comics llms um and
2:32:55still use kind of routing and query
2:32:57analysis so reasoning pre- retrieval
2:32:59reasoning post- retrieval so anyway that
2:33:01was kind of my talk um and yeah feel
2:33:04free to leave any comments on the video
2:33:05and I'll try to answer any questions but
2:33:07um yeah that's that's probably about it
2:33:09thank you

About this transcript

This page contains the full transcript of Learn RAG From Scratch – Python AI Tutorial from a LangChain Engineer by freeCodeCamp.org, generated from the public captions YouTube serves with the video. The transcript has 27,073 words across 3,910 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.