i gave AI a memory upgrade—and found a serious problem | HINDSIGHT review — Transcript

Full transcript

1. 0:00Okay, so let's talk about Agent Memory today,
2. 0:03because I've been going deeply on this lately.
3. 0:05I went through two open source memory systems back
4. 0:07to back, and after leaving with them for a while,
5. 0:10the feeling is complicated. There is this part
6. 0:13where I'm like, wow, he actually remembered that.
7. 0:15And then there's this other part where I'm
8. 0:17like, I never said that. So this video isn't a
9. 0:21recommendation. I'm not going to see here
10. 0:23telling you to install hindsight or any other memory
11. 0:25system right now. I just want to walk you
12. 0:27through what it's actually like to use one.
13. 0:30Actually, hold on, let me back up a bit. Before
14. 0:32I get into hindsight, I want to talk about why I
15. 0:34wanted the memory system in the first place.
16. 0:37Because you might be thinking, it's just memory,
17. 0:39right? We used to make a few markdown
18. 0:41files, write things down, where is the big deal?
19. 0:43That's exactly what I thought at first. Then I
20. 0:46found out I had underestimated it. Anyone who
21. 0:48used an agent for a while already knows. The
22. 0:51difference between an agent that remembers and
23. 0:53doesn't is huge. And the weight of that difference is not the same for every agent.
24. 0:58Some of them you only use to write code, say
25. 1:01DeepSeek or Codex. For those remembering
26. 1:04what happened inside this project is enough.
27. 1:07You only need project level memory, but a personal
28. 1:09assistant agents like Hermes, that's not just
29. 1:11a tool for writing code. It handles my life,
30. 1:14my work, my creative work. So what it needs to
31. 1:16remember isn't one project, it's me. And that
32. 1:19pushes my requirement on its memory up a
33. 1:21whole notch. Let's be clear, Hermes is not without
34. 1:24memory. It has memory, it's actually known
35. 1:27for its memory. That memory lives in a document
36. 1:30called memory dot MD. And that document has
37. 1:32a ceiling on it 2200 characters by default,
38. 1:36it's so limited, you write two paragraphs and
39. 1:38you're out. So it keeps writing and then deleting
40. 1:40the old stuff, squeezing in new stuff,
41. 1:42condensing over and over. Put bluntly, it can hold a short
42. 1:47about you, what you like how you tend to work
43. 1:50and the occasional project detail. Anything beyond
44. 1:52that, you're done. The capacity is just what it
45. 1:55is. These days, I watched you tell me the memory
46. 1:57is full several times a day, and then
47. 1:59something gets delayed to something gets condensed, it
48. 2:02either shows everything in or gives up and here
49. 2:05is the interesting thing. I finally opened memory
50. 2:08dot MD myself one day, everything in there
51. 2:10had been condensed into something like an
52. 2:12ancient literature. I couldn't even read it. And
53. 2:15Hermes defaults that memory dot MD that small for
54. 2:18a reason, you can raise it sure. But the
55. 2:20default is small because every time you open a new
56. 2:23conversation, that whole memory file gets sent
57. 2:26again, it just hands in your context the entire
58. 2:28time. So anything you put in there that isn't
59. 2:31about the current project, there's two things,
60. 2:33it pollutes your context, and it burns tokens for
61. 2:36absolutely nothing. So is 2200 characters enough?
62. 2:40Well, for most people who's just playing with
63. 2:42this, I think it is. Hermes can afford to keep
64. 2:44its memory that light because it has other
65. 2:47tricks. There is a skills or one skills aren't really
66. 2:50memory strictly speaking, but they hold a ton of
67. 2:53detail about how to do specific kinds of tasks.
68. 2:55Having one of those for a regular job is honestly
69. 2:58enough most days. And if it really isn't enough,
70. 3:00there is one last layer when it needs to
71. 3:02find something, it can go dig through the entire
72. 3:05session history, everything you told it and
73. 3:07everything it said back, all of that sitting in
74. 3:09a database, when it has to, it goes in there.
75. 3:12And honestly, it's pretty accurate. So Hermes setup
76. 3:14isn't bad day to day, it's genuinely enough.
77. 3:17Once you get to where I am, dozens of conversations
78. 3:20every day, and you let that pile up for months,
79. 3:22you start to feel the friction. It's really hard
80. 3:25for you to find something you said before or
81. 3:26your deep before the data is just too big, you're
82. 3:29needling his tech in it and that waste time with
83. 3:32tokens and drag samples accuracy and speed. So for
84. 3:35me using a personal agent heavily over a long
85. 3:37stretch, long term memory is still one of the
86. 3:40clearest gaps to close that gap. I went and
87. 3:43looked at the open source project called hindsight.
88. 3:46Hindsight means looking back after the fact.
89. 3:48I like the name, which honestly fits what the
90. 3:50product is doing. So what is hindsight exactly?
91. 3:54You can think of it as a long term memory layer
92. 3:56built for agents. The core is three verbs
93. 3:59retain, recall, reflect, remember, look it back up,
94. 4:03and there isn't over the memories to form a
95. 4:05judgment. And underneath, it isn't just storing
96. 4:08text when a conversation comes in, it pulls out
97. 4:10whatever seems worth remembering around embeddings,
98. 4:14build relationships between people and events
99. 4:16and records the timing too. So when you ask the
100. 4:18question later, it isn't searching keywords, it
101. 4:21goes out from semantics, relationships and the
102. 4:24time at the same time, then stitches the most
103. 4:27relevant memories back together, which makes it
104. 4:29feel like an external memory organ hanging off
105. 4:31the back of the agent. The agent handles talking
106. 4:34and working hindsight sits back there quietly
107. 4:36remembering pulling every conversation apart,
108. 4:39summarizing it, weaving it into a web of memory. And when you need to look back,
109. 4:44it pitches the relevant parts out. Let me show
110. 4:46you what that looks like. I was testing a model
111. 4:48called the space bunny at the time and I casually
112. 4:51asked one question, what do I actually think about
113. 4:53this model? It went through hindsight, recalled
114. 4:56and put together a pretty complete memory right
115. 4:59on the spot and the accuracy held up, not word
116. 5:01for word, but close enough. All of it was genuinely
117. 5:04things I'd said or things I'd talked through
118. 5:06with the agent, my reaction at the time, what I'd
119. 5:09actually done, how I changed my mind. So my
120. 5:12reaction at the time was like, this thing is real,
121. 5:15it works. I still kept an eye on it though. I
122. 5:17asked the follow up question, did you pull all
123. 5:19that from the memory or did you go dig through
124. 5:22some other files? Because that's Hermes's habit,
125. 5:25you give her a task he doesn't know about and go
126. 5:27searching session history for keywords and then
127. 5:30pieces something together out of them. And he said
128. 5:32no, the result was from hindsight only. Think about
129. 5:35that, no session search at all. And it still handed
130. 5:37me my entire take on one thing in a single answer.
131. 5:40That's what a memory system actually buys you. As
132. 5:43a comparison, I asked it to do the same job with
133. 5:46hindsight turned off. It was noticeably
134. 5:48slower. You can see it went through dozens of files,
135. 5:50it took forever to produce a similar answer
136. 5:53and the fact that it could produce one at all
137. 5:55is largely because I happen to have a written
138. 5:57transcript about space bunny already. Plus the
139. 5:59daily conversation logs my agent and I generated
140. 6:02it just found those files and summarise it for me
141. 6:05again. So same destination, completely different
142. 6:09experience getting there. Now, a normal project
143. 6:11intro video would probably end right about here.
144. 6:14Memory system, it's great, go use it. See you next
145. 6:17time. But anyone who's watched my video for a
146. 6:19while knows the real experience of running something
147. 6:22better doesn't go that smoothly. Things break.
148. 6:24And the thing that broke is exactly why I can't
149. 6:27recommend this system to you outright. And why I
150. 6:29personally hide on fully trusted high bar. Because
151. 6:32it will remember the wrong things very seriously.
152. 6:35During one writing session, I had the agent to go
153. 6:37look up some material. My agent pulled a video,
154. 6:40summarised it from the subtitles, I handed me the
155. 6:42information. Some other creator runs a batch of
156. 6:45experiments plus their opinions on it. There was
157. 6:48never me seeing anything of my agent find all of
158. 6:51that on its own. But when I asked for memories to
159. 6:54work from, it had recorded it as me running the
160. 6:56experiments and me making the judgment. It learned
161. 6:59to invent a life history for me. Honestly, that
162. 7:02part is a little unsettling because this happened
163. 7:05yesterday. So I can point out exactly where it
164. 7:07went wrong. Or if it had happened six months ago,
165. 7:10or even a year ago, and I don't remember it
166. 7:12anywhere near that clearly myself. If that can't
167. 7:15be fixed, the system isn't trustworthy. So I went
168. 7:18back and traced it. And it turns out that at the
169. 7:20level of how it works, this kind of error is
170. 7:23hard to avoid. Because this memory system takes each
171. 7:26conversation between me and the agent summarises
172. 7:29it attributes it then links it up. And the moment
173. 7:31the context gets long and the roles aren't
174. 7:34separated clearly enough, say all the you he and it that
175. 7:38showed up in a conversation of a few times me and
176. 7:41the agent in that session obviously know who those
177. 7:44pronouns point to. But the memory system only
178. 7:46sees that one turn so you can very easily map that
179. 7:50eat onto the wrong person. My workaround for
180. 7:53now was to modify Hermes's Hindsight plugin.
181. 7:56So when it hands a conversation over to
182. 7:58hindsight summarize, it clearly marks which lines were me
183. 8:02and which came from the agent and reserves
184. 8:04every pronoun whether that helps out no, it's an L I
185. 8:08m everything's everything is a probability.
186. 8:11But at least next time we make something up,
187. 8:13I'll trace it faster than this time. And there
188. 8:15were two smaller problems. I Wow, not as bad as
189. 8:18inventing a life history. But annoying enough
190. 8:20that I want to mention them. The first one is
191. 8:23maintenance. The whole system isn't something
192. 8:25like a simple skill. It's infrastructure you have to
193. 8:28raise yourself and the chain is long on top of
194. 8:30being external. It's not like a memory dot MD,
195. 8:33where your Hermes Agent writes one sentence and
196. 8:35that's it. It has to pull your turn apart first,
197. 8:38then summarize it with an L M, then my
198. 8:41embedding, build the index and store all of it in its own
199. 8:44memory system. And then when something needs to
200. 8:46recall it, that's another set of moves. And that's
201. 8:49the simplified version. In practice, there's more
202. 8:51moving parts than that. The chain that don't means
203. 8:54any link in the middle can break and then either
204. 8:57you never got stored or it never gets called the
205. 8:59worst one I hid was that it breaks silently. You
206. 9:03have no idea at all. My hindsight had just been
207. 9:05deployed. I spent the first day thinking it was
208. 9:07working. Then the next day I call it an entire
209. 9:10previous day of memory was just empty. It turns
210. 9:13out that there's a silent bug in the plugin. So
211. 9:15during all of my conversations, the previous
212. 9:18day, nothing got sweetened, but it reported nothing.
213. 9:20But bugs like that usually don't come back once
214. 9:23you've fixed them. So I think that's an acceptable
215. 9:25trade. And the other one is latency. It's an
216. 9:28external system. So you don't get the memory the
217. 9:31second you finish talking, it might be ready the
218. 9:33next turn or the one after that. And that's also
219. 9:36something you just just have to live with. So you
220. 9:39might be put off by now, or you might be dying to
221. 9:41try it if you're a tinkerer like me. But
222. 9:44before you go tinker, here's what running this thing
223. 9:46actually costs me. It's open source. And thank God
224. 9:49for the open source community and everyone in it.
225. 9:51But open source doesn't mean it's free to
226. 9:53run. Settling aside the machine and the disk,
227. 9:56it's four bills. Bill one is summarization. Every
228. 9:59single turn goes through an LLM, you can configure
229. 10:02it to every few turns or even every few steps,
230. 10:05depending on how much detail you want to keep. And
231. 10:07bill two is embedding. If you've got the hardware,
232. 10:10you can run it local. If you'd rather not bother,
233. 10:13you can use the cloud. And honestly, the cloud is
234. 10:15very cheap, like pennies a day. And bill three is
235. 10:18re ranking once you have 10s of 1000s of
236. 10:20memories. Once every relevant stuff is retrieved, you need
237. 10:24a general ranker to resource everything by
238. 10:26relevance. So the result is actually usable. And bill four is
239. 10:30deployment resources. In practice, one core CPU and
240. 10:33two gigabytes of RAM will not run this comfortably,
241. 10:36because your cloud server isn't running
242. 10:38hindsight alone. You have Hermes in there, plus whatever
243. 10:41else you deploy yourself. I bumped it to two
244. 10:44core CPU and four gigabytes of RAM. And that barely
245. 10:47works. And that's with the Linux version they have,
246. 10:49because my particular VPS is already tight on disk.
247. 10:52I just I just have too many things on it. So the
248. 10:54full database and all the dependencies were a bit
249. 10:57too much. So put it all together. There are costs
250. 11:00and there are things that's not so great about it.
251. 11:03But I do plan to keep using it for exactly one
252. 11:05reason. It actually feels the gap in Hermes's memory,
253. 11:09a memory web that writes itself in continuously
254. 11:11with timestamps with original text, summarized and
255. 11:14linked a single MD file just cannot do that. What
256. 11:18this kind of memory system actually gives you isn't
257. 11:20making the agent smarter on the spot. It's
258. 11:22quietly sorting through everything you've said, done and
259. 11:25thought about ahead of time. And then later when
260. 11:27around the task and you need something, you notice
261. 11:30it remembers more accurately. It gets what you
262. 11:32want better. And it understands why you made a
263. 11:35decision back then. Because everything you see
264. 11:37right there in the memory web, instead of you
265. 11:39searching a session from three months ago. So this
266. 11:41thing has to be raised. Your memory accumulates a
267. 11:44day at a time, kind of like keeping a plant.
268. 11:47If you stare at it every day and you feel like
269. 11:50there's nothing happening, nothing is changing.
270. 11:52But when you come back a month later, you'll find
271. 11:55it's dead. Just growing; it's more alive. So
272. 11:57that's it for today. If you got something out of this,
273. 11:59leave a like and don't forget to subscribe.
274. 12:01If not, watch another one. See you next time.