Запускаю ИИ-агента с локальными нейросетями на MacBook — Transcript

Full transcript

1. 0:19Lately, everyone has been talking about
2. 0:21AI agents. Some are specialized, like
3. 0:24coding agents, and others are
4. 0:26general-purpose, like AutoGPT, for
5. 0:28example. AutoGPT at one point became
6. 0:32the most important application ever
7. 0:33created by people. At least, if you
8. 0:36judge by the stars on GitHub, it has
9. 0:39over 260,000 stars. And that is an
10. 0:42absolute record among all software ever
11. 0:44written. Knowing how to create your own
12. 0:46agents, and especially how to interact
13. 0:48with them, is one of the most essential
14. 0:50skills for every programmer today. And
15. 0:52you know what I decided to do? I
16. 0:54decided to write my own agent. It is
17. 0:56also for general use. It is a bit
18. 0:57different in how it is built and how it
19. 0:59works. But its main feature is that it
20. 1:02runs with local neural networks.
21. 1:05Meaning the neural networks also run on
22. 1:07your computer, no need for
23. 1:08subscriptions or paying for anything.
24. 1:10You can run an AutoGPT-style agent that
25. 1:12way too, but my agent, which I called
26. 1:14MANbot, only works that way, though it
27. 1:16is easy to fix and connect an API. I
28. 1:18won't be singing its praises here. In
29. 1:21this video, I just wanted to show how
30. 1:23to install the agent on your computer,
31. 1:24how to install various local models,
32. 1:26how to connect these models to the
33. 1:28agent, and what the agent can actually
34. 1:29do. This is what my agent's page looks
35. 1:32like. It says all sorts of things about
36. 1:35it here, but it also has a GitHub where
37. 1:37you can download everything and read
38. 1:39about how to install this agent. But
39. 1:42before installing this agent, you need
40. 1:44to install another program, because to
41. 1:45run neural networks, we need a model
42. 1:47runtime. I use Ollama. You might ask,
43. 1:51why Ollama? Well, because it can work
44. 1:54with different CPUs and GPUs, and
45. 1:56secondly, it supports audio models out
46. 1:58of the box, like Whisper, so you can
47. 2:01send voice messages to the bot, and
48. 2:03images, so you can send pictures and
49. 2:05have it do something with them. And, of
50. 2:08course, text generation. On macOS,
51. 2:10Ollama is installed from a package file
52. 2:13, so you can download it and install
53. 2:15the server on your computer right away.
54. 2:17It is a fairly simple procedure. We
55. 2:19click install, enter your computer's
56. 2:22secret password, and everything is
57. 2:24installed. In this case, it's a Mac,
58. 2:26but it works exactly the same way for
59. 2:28Windows and Linux. And you can just run
60. 2:30the command "ollama serve". You can see
61. 2:33it says that another server is already
62. 2:35running. That's because on Mac and
63. 2:38Windows, you have a graphical
64. 2:40application you can launch as well. In
65. 2:44this case, you can manage all the
66. 2:46settings directly from this application
67. 2:49. If, for example, you just run the
68. 2:52Ollama server, you will have the exact
69. 2:54same interface available in your
70. 2:55browser. And here you can add and
71. 2:58remove models. Look, since I started it
72. 3:02on a Mac, audio models aren't supported
73. 3:04here, but I definitely install a model
74. 3:07like Qwen 2.5 VL Instruct. This is the
75. 3:10one that knows how to recognize images.
76. 3:12We click download, and it installs
77. 3:15quite quickly. Then, of course, we will
78. 3:18need models that can generate text. And
79. 3:21lately, I prefer to use Qwen 3.5. You
80. 3:24see, we have several options available:
81. 3:260.8 B, 2B, and 4B. For this example or
82. 3:29within the scope of this video, we'll
83. 3:31download one, but generally, I use
84. 3:33three models in my bot: small, medium,
85. 3:35and large for different tasks. Simpler
86. 3:37task, smaller model, faster execution
87. 3:39speed. Let's take this 4B model and
88. 3:43download it. And, of course, agents
89. 3:46absolutely need models that know how to
90. 3:48work with tools. Otherwise, it will
91. 3:50just be a chat that answers some
92. 3:52questions and isn't particularly smart.
93. 3:55Therefore, when choosing models, make
94. 3:57sure they have tool support. And one
95. 4:00more model we will need is this small
96. 4:04model called Nomic Embed Text. This is
97. 4:08a model that will allow us to turn our
98. 4:10chats into tokens and store them in
99. 4:12long-term memory. Because my agent in
100. 4:15this case supports RAG services, so it
101. 4:17can search for something. And that is
102. 4:20why it needs such a special model. And
103. 4:23that is exactly why I choose Ollama,
104. 4:24because it handles all these models and
105. 4:26all these variations itself. Except for
106. 4:28audio on Mac, and even that can
107. 4:29probably be fixed. But downloading the
108. 4:31model is not enough; it also needs to
109. 4:32be configured. And the main parameter
110. 4:35we need to adjust for text models in
111. 4:38this case is the context window size
112. 4:40and the engine it runs on. To do this,
113. 4:44we go to our text model that we intend
114. 4:47to use. We click on its settings right
115. 4:50here. And since I am using 3.5, I know
116. 4:53it supports a 256,000-token context
117. 4:56window. But that is too much. The
118. 4:59computer works or the model on the
119. 5:00computer runs slowly. Through trial and
120. 5:02error, I have determined for myself
121. 5:04that 128,000 or 131,072. tokens is
122. 5:09ideal for running on a local computer.
123. 5:14Also, if I were running it on Windows
124. 5:16or Linux, I would have another choice
125. 5:17here regarding which engine I want to
126. 5:18use. I always choose Vulkan, but on Mac
127. 5:21we have no alternatives. We always have
128. 5:23Metal. Now I can click, for example,
129. 5:25Load and see how fast my model is
130. 5:27running. It will load into memory now,
131. 5:30appear at the top in active models, and
132. 5:32we can ask it something. My standard
133. 5:35question is always "hi". Let's type it
134. 5:38right away to see if the model has
135. 5:40loaded or not. See, it says that
136. 5:41loading is still in progress. And here
137. 5:43it is loaded. And it replies quite fast
138. 5:45to us, at a speed of 18 tokens per
139. 5:47second in this case. Well, that's as
140. 5:48fast as it managed to ramp up. All the
141. 5:50necessary models are downloaded. We can
142. 5:51install the bot itself. Let's go to the
143. 5:53repository. Of course, we clone this
144. 5:56repository. We type Git Clone and
145. 5:57specify the path to the repository. It
146. 5:59copies into the Manbot folder. We enter
147. 6:03it and run npm install, because we need
148. 6:06to install all dependencies. Next, we
149. 6:10copy config example to just config json
150. 6:14to work with it. We open it in, for
151. 6:18example, nano. And here we need to fix
152. 6:22a few places. Well, first of all, we
153. 6:25need a token for our Telegram, so we go
154. 6:29to Telegram now, and there is a special
155. 6:33bot called BotFather. We will create a
156. 6:38new bot, name it, for example, Manbot.
157. 6:45Manbot. Let's give it the name
158. 6:50Manbotestbot. Underscore. Bot. Once
159. 6:56again. Bot. I don't really care. We
160. 6:59create it. We go over here. And we need
161. 7:02this key. We copy the key. We go to our
162. 7:07config and paste it into config under
163. 7:09bot token. This way, our script, our
164. 7:14agent will be able to send messages to
165. 7:16us. Next, we need our Telegram account,
166. 7:20or several users who will be able to
167. 7:21use the bot, because it will only
168. 7:23accept messages from these Telegram
169. 7:25accounts. To find your own, there is,
170. 7:27for example, the Scan My ID bot. When
171. 7:29you start a chat with this bot, it
172. 7:31gives you your ID. We copy it and paste
173. 7:34it right here. And then there are
174. 7:37various settings for where everything
175. 7:39is stored. We are also interested in
176. 7:41this embed model; it is the one that
177. 7:43turns our chats into tokens. And to
178. 7:46make sure we don't mix up the name, we
179. 7:49go over here, click on settings, and
180. 7:51copy its name up here. Copy, close. We
181. 7:56go to our file and paste it. We are
182. 7:58using this model. And the three models
183. 8:02that will be used for user interaction.
184. 8:05I only downloaded one, so I will enter
185. 8:07just that one. We go over here and
186. 8:10click edit. Here is our model. We copy
187. 8:13its name. We paste it here and replace
188. 8:20the medium model with the same one and
189. 8:23the small model with the same one as
190. 8:25well. Since my bot plans what it's
191. 8:29going to do first and then executes it,
192. 8:30I need to select which model to use for
193. 8:32planning. In our case, it doesn't
194. 8:34matter, all three are the same, but I
195. 8:35usually set it to medium. Then there
196. 8:37are some other odd settings and the
197. 8:39Whisper setup, regarding which model to
198. 8:40use. I don't have it here in Limonade
199. 8:43on my Mac, but I have it installed on
200. 8:46my Linux server. That's it. Everything
201. 8:49else can be left as is. We save this
202. 8:52file, exit it, and theoretically, we
203. 8:56can start our bot. We type npm run
204. 9:00build to build the main components. and
205. 9:05run npm run dev orchestrator. And if
206. 9:12nothing went wrong, we can see that our
207. 9:14bot has started and the model has
208. 9:17warmed up. Now we can go to our chat
209. 9:20and find our bot. Manbot test bot. Bot,
210. 9:28message. Hi. See, it replied to us. It
211. 9:31says that it is Manbot. And we can
212. 9:33communicate with it. For example, by
213. 9:34default, I have a weather skill.
214. 9:36Several bot skills are built-in for
215. 9:38searching and interacting with websites
216. 9:40. And for instance, the weather. I can
217. 9:42ask: "What's the weather in Amsterdam
218. 9:45tomorrow?" That's it, it has started
219. 9:50thinking. Since I don't really
220. 9:52understand what it's doing, I, of
221. 9:55course, have a special dashboard at
222. 9:57Localhost 3001 where I can monitor its
223. 10:00actions; see, this is my only request
224. 10:02so far. First run. The planning process
225. 10:05is currently underway. I repeat,
226. 10:07everything is being done on the same
227. 10:09Mac, and the video is recording a bit
228. 10:11slowly. See, it has planned everything,
229. 10:13using the weather skill. We can see
230. 10:15exactly what it is going to call here.
231. 10:18See, it called weather Amsterdam, the
232. 10:21necessary request. And if we look at
233. 10:24Telegram, we will see that when the
234. 10:25planning finished, it notified us that
235. 10:27the planning was complete, and we can
236. 10:28follow along. It is polling the weather
237. 10:31here, for some reason, several times.
238. 10:33Well, that's just the model. And I hope
239. 10:35that when it finishes, the status here
240. 10:37will change, and it will give us an
241. 10:39answer. But for now, it's working
242. 10:40properly. And the site is a very
243. 10:42strange site, see, it's not returning
244. 10:44success. It is trying to get the
245. 10:46weather in various different ways. I
246. 10:48guess it won't succeed. Overall, the
247. 10:50idea is that it takes the data here and
248. 10:52sends it. And I wanted to show you what
249. 10:54it looks like because I have an actual
250. 10:56working agent. And you can see that
251. 10:58this is what the dashboard of my active
252. 11:00agent looks like. I cleared its
253. 11:02database yesterday because it was
254. 11:04experimenting with a new version, so it
255. 11:05has been relatively fresh since
256. 11:07yesterday. But you can see that it is
257. 11:09currently working on a task. It uses
258. 11:11the following models. There is a small
259. 11:152B model, a medium 4B Qwen 3.5, and
260. 11:18this GPTOS S20B. A large model that it
261. 11:22works with. Everything works relatively
262. 11:25quickly. And here it is being used; it
263. 11:28is searching for something online,
264. 11:31seeing and opening various links,
265. 11:33retrieving and learning things. But the
266. 11:37point is that this is how it works, and
267. 11:39the chat and its responses look like
268. 11:41this. I already have a fairly large
269. 11:42chat with it, from back when I was
270. 11:43interacting with it before the cleanup.
271. 11:45Well, for example, I could ask it: "
272. 11:48Find interesting high-growth potential
273. 11:50industries for investment." And this is
274. 11:53the kind of answer it found for me. It
275. 11:54dug through various industries and
276. 11:57various information. I can also, for
277. 12:00instance, ask it, well, it manages my
278. 12:03email every two hours. You see, it
279. 12:06cleans something there, deletes, finds,
280. 12:08and writes some replies. But just as
281. 12:11easily, I can ask it about the weather,
282. 12:13and about—scrolling a long time. See,
283. 12:17another cleanup of my email. Right now
284. 12:19it is working. An error occurred. It
285. 12:21restarted itself. Again, working with
286. 12:24email. More email work. It can add
287. 12:27events to the calendar. In general, it
288. 12:29can do various things. It supports
289. 12:31skills, so you can add new capabilities
290. 12:33to it. And at the same time, it works
291. 12:35completely locally. I have a small
292. 12:37server in the room with a Ryzen AI9 and
293. 12:4064 GB of RAM. And it runs those models
294. 12:43there. So running all of this is quite
295. 12:45easy. It all works easily too, it just
296. 12:48takes maybe 10 minutes, but in return,
297. 12:51you get at least a secretary like this
298. 12:53for your email, because it cleans my
299. 12:56busiest inbox, and I literally open the
300. 12:58mail app once a day to check my other
301. 13:00emails or other accounts. I am sure
302. 13:03that the number of bots or the number
303. 13:06of such agents in the world will grow.
304. 13:08Everyone will have their own personal
305. 13:10agent. And right now, as you could see,
306. 13:12doing this is quite simple. You can
307. 13:14download Manbot, Open-interpreter, or
308. 13:17any other agent to run models locally
309. 13:19that allow your computer to perform at
310. 13:21least simple operations. Like this
311. 13:25video if you enjoyed it. Write in the
312. 13:27comments what you think about AI agents
313. 13:29, and Manbot in particular. Subscribe
314. 13:31to the channel so you don't miss my
315. 13:33other videos. Follow me on Telegram to
316. 13:35get information faster and more often.
317. 13:37And I'm not saying goodbye, this is the
318. 13:39internet. See you soon. Yeah.