YouTube2Text

Agent Memory EXPLAINED - Complete Architecture — Transcript

by Hugging Face · 4,935 words · 688 segments · language en · Watch on YouTube

Full transcript

  1. 0:01Good morning everyone. If you want your
  2. 0:03agent to remember things across
  3. 0:04conversations about yourself, about
  4. 0:06itself, and what it learns about the
  5. 0:08world, you're going to need an agent
  6. 0:10memory system. And today, that's what
  7. 0:13we're going to be explaining. First of
  8. 0:14all, we're going to see what agent
  9. 0:16memory systems are. And then we're going
  10. 0:18to be taking a deep dive into one
  11. 0:20existing very popular memory system
  12. 0:22called MEM0. We're going to explore what
  13. 0:25stores it uses to create its full
  14. 0:28long-term memory. We're going to explore
  15. 0:30the workflows for ingestion and
  16. 0:32retrieval of the memories that it
  17. 0:34creates. And we're going to take a look
  18. 0:35at what it does when it just wants to
  19. 0:37delete one memory. And finally, I'm
  20. 0:39going to explain to you how you can
  21. 0:41build this yourself if you want to build
  22. 0:42it your own. And how you can run the
  23. 0:45whole thing with your own local models
  24. 0:46and some recommendations of local models
  25. 0:48that will be useful on every step of the
  26. 0:51whole thing. So without any further ado,
  27. 0:54let's get right into it. All right. So
  28. 0:57the first thing to talk about is what
  29. 0:59actually is long form or long-term
  30. 1:02memory for agents. And something to keep
  31. 1:04in mind is that this is not the same
  32. 1:06thing as conversational memory. So
  33. 1:09remember that LLMs are stateless
  34. 1:12machines. In other words, when you send
  35. 1:14the prompt to your LLM, your LLM is
  36. 1:16going to process it and give you a
  37. 1:19completion. If you send another prompt
  38. 1:21to it after that, it will not remember
  39. 1:23anything about the previous prompt or
  40. 1:25the previous completion. So, if you want
  41. 1:28your agent or your your assistant to be
  42. 1:31able to remember what it said before,
  43. 1:33you're going to have to send over the
  44. 1:35entire history of the conversation with
  45. 1:38your prom so that it knows what it
  46. 1:40talked about before. And that is
  47. 1:42essentially conversational memory. This
  48. 1:44is what happens actually when you create
  49. 1:46an agent. So an agent scaffold or an
  50. 1:48agent harness will handle that context
  51. 1:51for you. In other words, whenever you
  52. 1:53send a message to your agent, your agent
  53. 1:55will process it with an LLM. Then we'll
  54. 1:58get the response back to from the LLM
  55. 2:01and will append the response both your
  56. 2:03message and the response to the history
  57. 2:06that it keeps in its state. And then
  58. 2:09whenever there is a new message, it will
  59. 2:11add the new message right here and send
  60. 2:13the whole thing back again to the LLM.
  61. 2:16That's the way it will remember things
  62. 2:18about its conversation. And this way
  63. 2:20when you mention something right here,
  64. 2:22the agent will remember it right here in
  65. 2:25the conversation. But this is
  66. 2:27conversational memory. If you only have
  67. 2:30this and you, for example, you start a
  68. 2:32new conversation and you send a new
  69. 2:34message, your agent will not remember
  70. 2:37anything from this other conversation
  71. 2:39because it is in another context. So
  72. 2:41what we do when we want to create long
  73. 2:43form memory is we create another service
  74. 2:46right here which is going to be the
  75. 2:47long-term memory. And this one right
  76. 2:50here is going to store things from every
  77. 2:53single conversation you have. And on
  78. 2:55every turn of your agent, it is going to
  79. 2:57check its long-term memory to be able to
  80. 3:00respond with facts that it knows about
  81. 3:02you and about itself which will be
  82. 3:04stored in the long form memory which
  83. 3:06will be independent from the
  84. 3:09conversational memory. Okay. And what
  85. 3:11we're going to be doing today is we're
  86. 3:13going to be taking a look at how this
  87. 3:14long for memory works in mem which is
  88. 3:17one very popular example of long-term
  89. 3:20memory implementations. And we're going
  90. 3:22to take a look at the entire
  91. 3:24architecture and how it works. So let's
  92. 3:27get right into that. And before I
  93. 3:29forget, just remember that for long form
  94. 3:31memory to work, you're going to have to
  95. 3:33make it external to your conversation.
  96. 3:36So it should not be within the same uh
  97. 3:38file or the same store as the session
  98. 3:40itself. Uh it [snorts] should be
  99. 3:42persistent across sessions. In other
  100. 3:44words, it should be at a different place
  101. 3:47from where you store your sessions. And
  102. 3:49this also means that you can also use
  103. 3:51the same memory for multiple agents if
  104. 3:54you're going to have it centered on the
  105. 3:56user. So you have the same memory for
  106. 3:58one user like for example myself and my
  107. 4:01tastes and my current projects that I'm
  108. 4:03working on and have that memory be
  109. 4:05shared among different agents. So for
  110. 4:07example with JHPT with claude with PI
  111. 4:10etc. [snorts]
  112. 4:11And then ideally you want to give your
  113. 4:13agent the tools to explore its own
  114. 4:16memory or at least you want to do
  115. 4:18ingestion and retrieval automatically on
  116. 4:21every agent turn. Okay? [snorts] or at
  117. 4:24least at after each conversation. All
  118. 4:27right, so with that out of the way,
  119. 4:28let's take a look at which stores Meim
  120. 4:31Zero uses. And let's just store it like
  121. 4:33this. Let's say stores. And it uses
  122. 4:36three main stores. Okay, so the first
  123. 4:39one is going to be the main memory. And
  124. 4:43this one is going to be the main uh
  125. 4:45place where all the memories are going
  126. 4:47to be stored. So the memories
  127. 4:48themselves, the short paragraph or short
  128. 4:51sentences that contain the memory itself
  129. 4:54are going to be stored right here. So
  130. 4:57the memories plus their metadata. A
  131. 5:00memory can be something like for example
  132. 5:02the user prefers vegan restaurants and
  133. 5:05that memory is going to contain a bunch
  134. 5:07of metadata associated to it and it's
  135. 5:09going to be in this uh vector store as
  136. 5:11well. Uh the metadata can include things
  137. 5:14such as the date when the memory was
  138. 5:17created. Uh whether the memory belongs
  139. 5:20to the user or the agent. So if it's
  140. 5:23information for example about the user
  141. 5:25as we mentioned about the user
  142. 5:27preferring vegan restaurants for example
  143. 5:29or about the agent for example learning
  144. 5:32about current events or about a new way
  145. 5:34to perform some procedure etc. It's also
  146. 5:37going to contain the hash of the memory
  147. 5:40itself. And this is used strictly for
  148. 5:42the duplication. We're also going to
  149. 5:44have the leatized version of this. Um,
  150. 5:47and this one right here is used mainly
  151. 5:50for keyword uh search uh which we're
  152. 5:53going to be explaining in a bit. And um
  153. 5:55after that, it also includes the dates
  154. 5:57such as when it was the dates when it
  155. 6:00was created and updated. And it also
  156. 6:02create uh mentions the expiration date
  157. 6:05and it will only have also have the
  158. 6:08attribution of who created this this
  159. 6:11memory. So for example if you're using
  160. 6:12multiple agents which agent created this
  161. 6:15memory and that's the main memory store
  162. 6:17the vector store and on top of that we
  163. 6:20have another store called the entity
  164. 6:22store or the entity memory and what's
  165. 6:25going to happen is that me zero is going
  166. 6:28to extract the entities from the main
  167. 6:30memory. So the places, the proper names,
  168. 6:34the people, etc. And it's going to save
  169. 6:37them in this separate vector store. So
  170. 6:40this one right here is also a vector
  171. 6:42database, but every point in that vector
  172. 6:45database is going to be an entity. And
  173. 6:48each entity, we're going to have in its
  174. 6:51metadata that it's linked to one or more
  175. 6:55main memories. So for example, if my
  176. 6:57memory is that my favorite neighborhood
  177. 7:00in Paris is Mat, then the entity is
  178. 7:04going to be Mat and Paris and whenever I
  179. 7:07am asking a question about Paris, it
  180. 7:10will retrieve the entity about Paris and
  181. 7:13it will see all the memories linked to
  182. 7:16that entity and among which there is
  183. 7:19going to be my memory about my favorite
  184. 7:22um neighborhood in Paris. Okay, that is
  185. 7:24the second one. And the third one is a
  186. 7:27SQLite database which is used for two
  187. 7:31things. First of all, basically a
  188. 7:33logging system to keep track of all the
  189. 7:36history of all the changes in your
  190. 7:38vector stores. And second, it's going to
  191. 7:41keep the latest 10 messages that were
  192. 7:45sent to the pipeline so that it has
  193. 7:49access to those um most recent messages
  194. 7:52to figure out what the user is talking
  195. 7:54about when creating a new memory. And
  196. 7:56I'm going to explain to you a little bit
  197. 7:57more about this in a moment. So those
  198. 7:59are the three different stores that we
  199. 8:01have. Let's take a look at how Mezero
  200. 8:04uses them in ingestion and in retrieval.
  201. 8:07All right. So now we have a much better
  202. 8:09understanding first of all of what is
  203. 8:11long-term memory for agents and also uh
  204. 8:14in this particular example which are the
  205. 8:16stores that are used by our agent. Now
  206. 8:19let's talk about the ingestion part
  207. 8:21which is the part that is going to run
  208. 8:22when the agent wants to save new
  209. 8:25memories. Okay. So um the ingestion part
  210. 8:28is going to be executed or it's going to
  211. 8:30be run after every agent turn. So when
  212. 8:33you ask the agent to do something, it
  213. 8:35will continue the task, finish it or
  214. 8:38when it finishes one turn, it will take
  215. 8:40those messages produced by that turn and
  216. 8:43they will be sent to this pipeline.
  217. 8:45Okay, so the input in this case, it's
  218. 8:47going to be messages. And just uh for
  219. 8:51reference, there are three main ways in
  220. 8:54which those messages can be turned into
  221. 8:56memories in your database. The first one
  222. 9:00is going to be procedural memory.
  223. 9:04And this way of ingesting the memory is
  224. 9:07going to uh take a list of messages and
  225. 9:10summarize the process or the actions
  226. 9:12that the agent took so that in the
  227. 9:15future it is able to retrieve this
  228. 9:17memory and reproduce that exact
  229. 9:19procedure. This is useful if you want to
  230. 9:22if you finished the a if the agent
  231. 9:24finished doing something and you wanted
  232. 9:26to remember exactly what it did with
  233. 9:28which tool calls it uh made and which
  234. 9:31results it got. Uh this is the one that
  235. 9:34you probably want to use. But [snorts]
  236. 9:36uh in general I don't really uh feel
  237. 9:38like it's very used anymore precisely
  238. 9:40because you can just extract a scale
  239. 9:42from a from a a conversation transcript
  240. 9:45and that works much better. Um, the
  241. 9:48second way is to set the infer parameter
  242. 9:51to false. And what this is going to do
  243. 9:53is that it's going to take all these
  244. 9:54input messages that you're taking
  245. 9:56sending into the pipeline, embed them,
  246. 9:58and put them straight into your
  247. 10:01database. Just like that, the messages
  248. 10:03themselves. Okay. And the third way is
  249. 10:06to set infer equal to true. And this one
  250. 10:10right here is going to trigger the
  251. 10:12extraction with LLM pipeline, which is
  252. 10:14the more sophisticated and more advanced
  253. 10:16way to uh do memory for long-term
  254. 10:20agents. And this is the one that we're
  255. 10:22going to be focusing on today. Okay. So
  256. 10:26let's suppose that you have set infer to
  257. 10:28true and you send the input messages to
  258. 10:31the ingestion pipeline. What is going to
  259. 10:33happen after that? So the second step
  260. 10:35right here is going to be to load
  261. 10:39recent
  262. 10:41context. And this includes a bunch of
  263. 10:44things because this extraction is going
  264. 10:45to happen with an LLM. So what's going
  265. 10:47to happen is that you're going to send
  266. 10:49all these messages to an LLM
  267. 10:53and the LLM is going to export
  268. 10:56a structured JSON with all the memories
  269. 11:00that it extracted from it. Okay. And in
  270. 11:03order to do that, the LLM is going to
  271. 11:05have a very specific prompt and a very
  272. 11:07specific context about the user and
  273. 11:10about what they are talking about so
  274. 11:12that it is able to extract relevant um
  275. 11:15memories from those messages and this uh
  276. 11:18context includes a bunch of different
  277. 11:20things. So let me tell you the first of
  278. 11:23all um let me just give you a tour on
  279. 11:25this prompt. The prompt includes just
  280. 11:27the role of the LLM which is going to be
  281. 11:31memory extractor. Then also includes a
  282. 11:34summary of the user. So user summary. It
  283. 11:38is also going to include the messages of
  284. 11:40course that we are adding right here as
  285. 11:43input. It is also going to include the
  286. 11:45recent memories
  287. 11:48that were saved to the database so that
  288. 11:50we know what we're talking about and
  289. 11:52maybe some relevant memories that come
  290. 11:53into play. right here, it's going to
  291. 11:55include relevant memories. So that means
  292. 11:58that it is going to look into the
  293. 11:59database to find relevant memories to
  294. 12:02these messages. And this one right here
  295. 12:04is actually done by just flattening the
  296. 12:07messages. So turning the messages into a
  297. 12:09single uh string, embedding that list of
  298. 12:12me that that messages and searching the
  299. 12:16existing database for relevant memories
  300. 12:19with this particular um that are related
  301. 12:22to that message. And those relevant
  302. 12:24messages are also going to be appended
  303. 12:26to context. And it is also going to
  304. 12:28include the last n number of messages.
  305. 12:31Ideally some these are the 10 latest
  306. 12:33messages. So last 10 messages. And these
  307. 12:36ones right here are going to come from
  308. 12:38the SQLite database that we mentioned
  309. 12:40before. So every time this uh pipeline
  310. 12:43is triggered, it is going to save that
  311. 12:45message into the SQLite database and
  312. 12:49it's going to keep only the 10 latest
  313. 12:51messages. And this one right here is
  314. 12:53very useful to identify and to figure
  315. 12:56out what pronouns mean for example in
  316. 12:58that particular new message. So for
  317. 13:00example, let's suppose that we sent a
  318. 13:03new input messages a new input message
  319. 13:05that includes things like it is great or
  320. 13:09he's really good at this. Those it and
  321. 13:12he don't really have context about what
  322. 13:15they mean unless you send the past few
  323. 13:17messages so that the this LLM knows what
  324. 13:20you're talking about. So this one right
  325. 13:21here comes to comes from SQLite and this
  326. 13:24one right here come from the main vector
  327. 13:27DB. Then on top of that it lists the
  328. 13:29conversation date and the current date
  329. 13:32and this prompt is going to export this
  330. 13:35very nice JSON which is going to be the
  331. 13:37memory that we're going to be importing
  332. 13:40into our system. Okay. So let's suppose
  333. 13:42that we now have passed this and as a
  334. 13:45result we got something like the user
  335. 13:48prefers
  336. 13:50vegan
  337. 13:52restaurants and this is going to be the
  338. 13:54memory of the user. Okay, that's just an
  339. 13:56example. Now what's going to happen is
  340. 13:58that this memory is going to be saved
  341. 14:01into our main memory store. It's going
  342. 14:04to be added. It's going to uh get the
  343. 14:06date where it was added or updated. Uh
  344. 14:09whether it's from the user or the agent
  345. 14:11as we mentioned before. It's going to be
  346. 14:13hashed and uh dduplicated against the
  347. 14:17existing database. And of course, since
  348. 14:19we're using hash duplication in this
  349. 14:21particular case, it only works for
  350. 14:24memories that are the exact same
  351. 14:26wording. So probably not the most useful
  352. 14:29to dduplication system that we have, but
  353. 14:31that's what we have right here. and it's
  354. 14:33also going to contain its leatized
  355. 14:35version. So this sentence right here is
  356. 14:38going to be leatized and it's going to
  357. 14:39be saved alongside the memory for uh
  358. 14:43keyword search later on. And that is how
  359. 14:45it's going to be stored into the main
  360. 14:48main vector
  361. 14:51DB and also into our SQLite database as
  362. 14:56we mentioned before to keep track of the
  363. 14:57latest messages that were added to the
  364. 15:00pipeline. Okay. And that's essentially
  365. 15:03how the system is going to store the the
  366. 15:07new memories. And the most important
  367. 15:09part is the LLM based extraction. And
  368. 15:12this right here can be done with a very
  369. 15:14small LLM. If I'm not mistaken, uh,
  370. 15:17Mez0ero does it with GPT5 mini, but you
  371. 15:20can use a very small model, open source,
  372. 15:23and run this for free in your in your
  373. 15:25machine because it is very
  374. 15:27straightforward and very simple. going
  375. 15:28to be giving you some recommendations of
  376. 15:30models in a bit. All right, so we have
  377. 15:33effectively understood pretty well what
  378. 15:35is long-term memory, the stores that
  379. 15:37come into play with me zero and how the
  380. 15:40ingestion flow works. Now let's talk a
  381. 15:42little bit about retrieval and this is a
  382. 15:44very important part and there are
  383. 15:47essentially two main parts or two main
  384. 15:49moments where you are going to want uh
  385. 15:51retrieval to work. Retrieval essentially
  386. 15:54means uh getting relevant memories to
  387. 15:57whatever topic you're talking about. Uh
  388. 15:59so you want to retrieve those relevant
  389. 16:01memories in two different scenarios. The
  390. 16:04first one is when your agent wants to um
  391. 16:07explicitly search its memory and ideally
  392. 16:10this is exposed to the agent via an
  393. 16:12agent tool. So for example a tool called
  394. 16:15search memory database or something like
  395. 16:17that. That way your agent is going to be
  396. 16:20able to call the tool uh search it with
  397. 16:22a query and find relevant memories
  398. 16:24related to that particular query. That
  399. 16:27is explicit memory searching. And
  400. 16:29another way of searching is to have it
  401. 16:32be automatic on every agent turn. So
  402. 16:35when you send a message to the agent um
  403. 16:38the memory is going to trigger
  404. 16:39automatically. It's going to find
  405. 16:41relevant memories related to your query,
  406. 16:44append them to the to the context of
  407. 16:46your agent and send that to your LLM. So
  408. 16:48your LLM has context about what um your
  409. 16:53user is talking about and some previous
  410. 16:55conversations. Okay. [snorts] And this
  411. 16:57retrieval the pipeline for the retrieval
  412. 17:00is the same for both the tool and
  413. 17:02automatic retrieval. And what's going to
  414. 17:04happen is the following. So first of
  415. 17:06all, we're going to take a few
  416. 17:08parameters as input. We're going to take
  417. 17:10first of all the query of course. So
  418. 17:12whatever we're talking about or what
  419. 17:14we're using if it is the user message
  420. 17:17then the user message becomes the query
  421. 17:18itself. There might be some query
  422. 17:21rewriting at this step and that can be
  423. 17:23pretty useful in your pipeline. I um as
  424. 17:26far as I'm not aware mem does not do
  425. 17:28that by default. So you would have to
  426. 17:30implement that on the harness side. But
  427. 17:32um just keep that in mind. The query is
  428. 17:34the first part. The top k memories that
  429. 17:37you want. So how many memories do you
  430. 17:39want by retrieval search? So for
  431. 17:42example, let's say that you want 10
  432. 17:44memories. A threshold of the minimum
  433. 17:46score of memories to be added to the
  434. 17:49list of related memories. And finally
  435. 17:52the identity of who is looking for these
  436. 17:55memories. Okay. [snorts] So not the ID,
  437. 17:57the identity. So is this uh memory
  438. 18:00related to the agent, a memory related
  439. 18:02to the user or to the run itself? Okay.
  440. 18:05And this is very important because agent
  441. 18:07memory can be things about what it
  442. 18:10learned about the world or some
  443. 18:11procedures about uh how to do some
  444. 18:14different stuff and about the user it
  445. 18:17will be related to the user preferences
  446. 18:19the user's previous experience etc. And
  447. 18:23with that input what we're going to do
  448. 18:24is we're going to retrieve relevant
  449. 18:26memories for that conversation. So the
  450. 18:29first step right here is to embed the
  451. 18:32query. And in order to do that, we're
  452. 18:34going to use the same embedding system,
  453. 18:36the same embedding uh model as we used
  454. 18:38to create the vector database. And in
  455. 18:41order to do that, we can use some open
  456. 18:43source models. And here on the screen is
  457. 18:45a list of some of the most popular
  458. 18:47embedding models that you can use for
  459. 18:49this particular task. And you can run
  460. 18:51them locally, of course, in your
  461. 18:52machine. Men zero by default uses closed
  462. 18:55models. Just know that you can also do
  463. 18:57this locally completely with your own
  464. 18:59open- source embedded models. Once the
  465. 19:02uh query is uh embedded, we're going to
  466. 19:06use it to retrieve semantically similar
  467. 19:09memories in our main vector store. Okay,
  468. 19:12so right here we're going to search
  469. 19:15vector DB semantically and this is done
  470. 19:18with aN. Okay, so nearest neighbors
  471. 19:22and actually here is where your top K
  472. 19:25value comes into play [snorts] because
  473. 19:28remember that top K is the number of
  474. 19:30memories that you want to be added to
  475. 19:32your context of your agent. And let's
  476. 19:34suppose that you said uh that you want
  477. 19:36the top K of 10. So you want 10 related
  478. 19:39me uh memories to enrich your context of
  479. 19:42your user of or your or about your
  480. 19:44agent. And uh what's going to happen is
  481. 19:47that this vector search is not going to
  482. 19:49look for only 10 messages. It's going to
  483. 19:52look for many more because it's going to
  484. 19:53do some reranking. And reranking works
  485. 19:55better, of course, if you have a larger
  486. 19:57pool of results. And then it's going to
  487. 20:00select the top 10 from those uh that
  488. 20:03bigger pool. And in order to do that,
  489. 20:05what's going to happen is that it's
  490. 20:06going to either multiply your top k * 4
  491. 20:11or select uh 60 um as the minimum uh
  492. 20:18number of messages memories that are
  493. 20:20queried. Okay. So if you set a top k of
  494. 20:2310, you're going to actually get 60
  495. 20:25messages from this vector search. And
  496. 20:28this is very important because on top of
  497. 20:30this 60 messages uh there are going to
  498. 20:33be a couple of extra steps to uh rerank
  499. 20:37and to boost some memories over others.
  500. 20:41And here is where you have the keyword
  501. 20:44matching score. And on the other side
  502. 20:47you have the entity boost. Now, this one
  503. 20:52right here is essentially going to
  504. 20:54search for overlaps in words between
  505. 20:57your query and the memories within the
  506. 21:01that would were uh that were returned
  507. 21:03from here. So, uh remember that the
  508. 21:05memories returned from here um are the
  509. 21:08base pool on which there is going to be
  510. 21:10the reanking. None of these two steps
  511. 21:13can add additional memories to what was
  512. 21:15retrieved right here. So, the first part
  513. 21:17right here is keyword matching score. So
  514. 21:20what's going to happen is that both the
  515. 21:22query uh is going to be leatized and
  516. 21:24actually remember that all the memories
  517. 21:26already already contain their own lema
  518. 21:30in their metadata as we mentioned
  519. 21:31before. So we're going to compare these
  520. 21:33two and we're going to give it a score
  521. 21:36depending on how similar uh the words
  522. 21:38are. So word so in other words it's
  523. 21:41measuring the word overlap and it is
  524. 21:43doing that in case you're curious about
  525. 21:45it with BM25.
  526. 21:47Okay. And by the end, we're going to uh
  527. 21:50translate that into a score between zero
  528. 21:53and one. Okay, [clears throat]
  529. 21:56so that's going to be this score right
  530. 21:58here of keyword matching. And for entity
  531. 22:00boost, what's going to happen is that it
  532. 22:02is going to first extract the entities
  533. 22:06from the query. Then it is going to
  534. 22:08search the entity database for this
  535. 22:12query and the resulting entities contain
  536. 22:16memories that are linked to them. Right?
  537. 22:18As we mentioned before, that is how the
  538. 22:20entity uh memory works. So every entry
  539. 22:24in this vector database is an entity. So
  540. 22:26for example, Paris, restaurant, etc. And
  541. 22:30every single one of those entities is
  542. 22:32linked to one or more memories. Okay?
  543. 22:36And what's going to happen is that
  544. 22:38whatever entities we get from that
  545. 22:40vector search, we're going to take the
  546. 22:43memories associated to those entities
  547. 22:46and we're going to only select the ones
  548. 22:49that are in our initial 60 or our top K
  549. 22:54initial pool. And in order to do that,
  550. 22:57we're going to select those and we're
  551. 22:58going to give them an additional score.
  552. 23:00And this score right here is between
  553. 23:02zero and five. If you're interested in
  554. 23:05the exact formula of how they get this
  555. 23:08specific score, you can take a look at
  556. 23:11the screen. I will be showing it right
  557. 23:13now. But to keep things simple, just
  558. 23:16remember that whichever entity,
  559. 23:18whichever memories were retrieved that
  560. 23:20were retrieved from this entity database
  561. 23:23are going to get a a score and the score
  562. 23:26is going to be higher if there are less
  563. 23:29memories associated to that particular
  564. 23:30entity. Okay. See for example if for
  565. 23:33example if my entity is Paris and I have
  566. 23:35a thousand memories associated to that
  567. 23:37one then the score is going to be much
  568. 23:39lower. If my uh entity is Paris and I
  569. 23:43have just two memories associated to
  570. 23:44that one then it is more likely that
  571. 23:46those memories are related to whatever I
  572. 23:48am um trying to retrieve right here. So
  573. 23:51that's going to give us a larger score
  574. 23:54between 0.5 uh uh closer to 0.5. In the
  575. 23:58end, all these three scores are going to
  576. 24:00be added. Um, the vector search also is
  577. 24:03translated into a score between zero and
  578. 24:06one. So, we have from 0 to one here,
  579. 24:09from 0 to 1 here, and from 0 to 0.5
  580. 24:11here. That gives us a total of 2.5
  581. 24:15potential score. So, what's going to
  582. 24:17happen for each one of the messages in
  583. 24:20our initial pool is we're going to add
  584. 24:23both three scores. So this one, this one
  585. 24:26plus this one plus this one. And we're
  586. 24:28going to divide them over 2.5. And that
  587. 24:32is going to give us the final score. And
  588. 24:36then given that score, we're going to
  589. 24:38select the actual top K. And those are
  590. 24:43the ones that are going to be returned
  591. 24:45uh to the user or to your agent. Okay.
  592. 24:48And that's essentially how retrieval
  593. 24:50works. As you can see, it's a little bit
  594. 24:52more complex than the ingestion part. Uh
  595. 24:54[snorts] there are three different uh
  596. 24:57scores that are added together in order
  597. 24:59to actually rank the most relevant
  598. 25:01memories for that particular query. And
  599. 25:03something else remember I mean I already
  600. 25:05mentioned it but remember it's very
  601. 25:06important to remember that the query
  602. 25:09right here is any it's uh it can be
  603. 25:12anything and it's usually uh whatever
  604. 25:15you send to this pipeline. But this
  605. 25:17pipeline does not necessarily have a
  606. 25:19query rewriting system. If you want the
  607. 25:22retrieval to work better, I would
  608. 25:23recommend that you add a query rewriting
  609. 25:25system right here on the side of your
  610. 25:28harness. So whenever the user asks about
  611. 25:30something, you ideally have a small LM
  612. 25:34translate that into a better query so
  613. 25:37that you can get better results right
  614. 25:38here in the retrieval. [snorts] And
  615. 25:41there you go. This is essentially
  616. 25:43everything that you need to know about
  617. 25:45how MEMS works. And let's take a look
  618. 25:47very quickly at some models that you may
  619. 25:49want to use in each one of these stages.
  620. 25:52Uh if you want to do this with
  621. 25:54completely local setups with open source
  622. 25:56models. So if you go to hugging phase,
  623. 25:58we have right here if you go into
  624. 26:01models, you're going to see a bunch of
  625. 26:02different models. uh I would recommend
  626. 26:05that if you want to use an open- source
  627. 26:08model for the extraction part which is
  628. 26:10one of the most important parts I would
  629. 26:12recommend that you choose a small model
  630. 26:14because extraction is a relatively
  631. 26:16simple uh task. So between 1 billion and
  632. 26:1912 billion parameters should be enough.
  633. 26:22I would not go be under 1 billion unless
  634. 26:25you fine-tune the model in order to make
  635. 26:27it more accurate. But um in this
  636. 26:30ballpark you already have good
  637. 26:32suggestions. And if you click on text
  638. 26:33generation, it's going to uh filter only
  639. 26:35for text generation. So for example,
  640. 26:37right here you can see that you can use
  641. 26:40some lama models, you can use Quen. Uh
  642. 26:43probably you could I would recommend
  643. 26:45something like Quen 38B. I I think that
  644. 26:478 billion parameters is pretty good. You
  645. 26:49can choose the latest version of the
  646. 26:52model at the time of watching. For
  647. 26:55embeddings, if you want to take a look
  648. 26:56at them as well, I recommend that you go
  649. 26:59also right here to models. And in tasks,
  650. 27:02you're going to go right here and filter
  651. 27:05by feature extraction. And that way, you
  652. 27:08will see mostly embedding models. And
  653. 27:10you can select the one that you like the
  654. 27:12best. And if you want to see a ranking
  655. 27:15of the embedding models, you can take a
  656. 27:17look at it right here. Here, which is
  657. 27:20one of the most reputable benchmarks for
  658. 27:22a bunch of different um embedding models
  659. 27:26for montilingual, for English, for
  660. 27:28different languages, etc. And here you
  661. 27:30have the benchmarks exactly depending on
  662. 27:32which one you're trying to figure out.
  663. 27:33So for example, let's suppose that you
  664. 27:35want something related specifically to
  665. 27:37medical. Well, here you have the
  666. 27:39benchmark for embedding models that
  667. 27:42perform better with medical contexts.
  668. 27:45And of course, there's always the
  669. 27:46possibility of fine-tuning your own
  670. 27:48small models. I would actually recommend
  671. 27:50that if you're going to be doing this in
  672. 27:52a long term, you fine-tune your models
  673. 27:55for this to work even better and to be
  674. 27:57able to run this locally. And uh there
  675. 27:59you go. That is essentially a very quick
  676. 28:02overview of u long form or long-term
  677. 28:05memory for agents with a very nice use
  678. 28:07case like me zero. I will be dissecting
  679. 28:10other memory libraries such as super
  680. 28:13memory for example etc to show you a
  681. 28:15little bit more of different ways of
  682. 28:18dealing with long form memory. But uh I
  683. 28:20hope that this was educational and uh do
  684. 28:22let me know if there are other things
  685. 28:24that you would like to see in this
  686. 28:25format. And thanks a lot for watching.
  687. 28:27It has been a great pleasure. I'll see
  688. 28:30you in the next one.

About this transcript

This page contains the full transcript of Agent Memory EXPLAINED - Complete Architecture by Hugging Face, generated from the public captions YouTube serves with the video. The transcript has 4,935 words across 688 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.