I Built a Local AI Assistant: 100% Free & No Subscriptions! — Transcript

Full transcript

1. 0:00This is my AI assistant Ada, and she
2. 0:01could do a lot of cool things. Whether
3. 0:03it's controlling my home, doing some
4. 0:06shopping for me on the web, or a whole
5. 0:08lot of other stuff. But, Ada does have
6. 0:10one big problem. Every time we ask her
7. 0:12to do something, she needs to operate on
8. 0:14Wi-Fi. So, she sends a prompt to
9. 0:16Google's big data centers, we have to
10. 0:18wait for a little bit, and then we
11. 0:19eventually get a reply back. The problem
12. 0:21with this is, one, she needs the
13. 0:23internet to work. Also, our prompts are
14. 0:25not private. And lastly, if you use this
15. 0:27enough, it ends up not being free
16. 0:30anymore, and you have to pay. So, today
17. 0:31we're going to make a new AI assistant
18. 0:33that's 100% local, 100% private, and
19. 0:36100% free. So, the first step in
20. 0:38building our AI assistant is picking an
21. 0:40open-source model. So, I started with
22. 0:42these three choices: Gemma 3, Qwen 3,
23. 0:45[music]
24. 0:45or DeepSeek-R 1. And each of these three
25. 0:48open-source models comes [music] in
26. 0:50different sizes, which are measured in
27. 0:52parameters. In most cases, the more
28. 0:53parameters a model has, the smarter it
29. 0:56is. But, also, the more parameter a
30. 0:58model does have, the more
31. 1:00resource-intensive it's going to be. So,
32. 1:02a very rough estimate is 1 billion
33. 1:05parameters is equal to about 1 GB of
34. 1:07VRAM. So, to start off with, I decided
35. 1:09I'm going to pick a model between 1
36. 1:11billion and 2 billion parameters. And
37. 1:13that's because my GPU only has about 8
38. 1:15GB of RAM. And if you take away the RAM
39. 1:17that my operating system already uses,
40. 1:19we only have about 6 GB of RAM to work
41. 1:22with. So, we have to be very strategic
42. 1:24on the models that we're choosing. So,
43. 1:25these are the three models and sizes
44. 1:27that I want to test. So, I went ahead
45. 1:29and wrote a test script that's going to
46. 1:31test the model in four different [music]
47. 1:32areas. And we're going to be measuring
48. 1:34accuracy, how much VRAM the models are
49. 1:36taking up, and lastly, the latency, how
50. 1:38long we have to wait before [music] the
51. 1:39model starts typing up the answer. Okay,
52. 1:41so after running the test, the results
53. 1:43are
54. 1:44kind of weird. So, first, each model
55. 1:46performed the great for accuracy. All of
56. 1:49them scored 90% or higher, with Qwen 3
57. 1:52scoring the highest with 95%. But, also,
58. 1:55Qwen 3 being the largest model, it took
59. 1:57almost 2 GB of VRAM. And with Gemma 3
60. 2:00and DeepSeek, only a little more than 1
61. 2:02GB of VRAM. But, what really surprised
62. 2:05me was the latency test. DeepSeek and
63. 2:08Qwen both took about 100 ms, but Gemma 3
64. 2:11took twice as long, which is a little
65. 2:14weird just because it's a smaller model.
66. 2:15I would assume it would be the fastest.
67. 2:17But, with that, it really cuts down our
68. 2:19options to either DeepSeek or Qwen. But,
69. 2:22one cool thing about DeepSeek and Qwen
70. 2:25is that they're both thinking models,
71. 2:26which means that they could kind of
72. 2:27think through problems. So, I'm curious
73. 2:29to see how each of those models will
74. 2:31perform if I run the same exact test,
75. 2:34but with thinking on. So, after running
76. 2:36the test with thinking on, both models
77. 2:38did perform better on accuracy. Qwen
78. 2:41scored 100% and DeepSeek got one more
79. 2:44question right with 95%. But, the main
80. 2:46downfall of using thinking is how long
81. 2:48it takes the model to respond. It took
82. 2:50Qwen 3 1,700 ms to respond back, and it
83. 2:53took DeepSeek 1,900 ms to reply back.
84. 2:56But, in comparison, Qwen 3 performed
85. 2:58better on accuracy and is faster. So,
86. 3:00that's going to be the model that we're
87. 3:01using for this project. But, now we kind
88. 3:03of have a problem. Are we going to use
89. 3:05Qwen 3 with thinking or without
90. 3:07thinking? But, I thought about it and I
91. 3:09said, "Why not use both?" So, we could
92. 3:12first send the user's prompt through a
93. 3:14small quick model, and that quick model
94. 3:16then decides to send it to either the
95. 3:18Qwen version that is thinking or the
96. 3:20Qwen version that is not thinking. And
97. 3:22at first, I thought I came up with some
98. 3:24new brilliant architecture for AI, but I
99. 3:27did some research and this is just
100. 3:29called a router. You have one AI that is
101. 3:31smart enough to send specific prompts to
102. 3:33a more specialized AI model. And
103. 3:34thankfully, I already have a model in
104. 3:36mind to do the job,
105. 3:38>> [music]
106. 3:38>> and that's Gemma. We're not using any
107. 3:40version of Gemma, we're using a specific
108. 3:43version of Gemma called Function Gemma.
109. 3:45And what makes Function Gemma so special
110. 3:46is that one, it's super small, only 270
111. 3:49million parameters, and also, it was
112. 3:51specifically trained
113. 3:53>> [music]
114. 3:53>> to output functions. If you don't know
115. 3:54what a function is, it's just a tiny
116. 3:56snippet of code that can be ran to
117. 3:58execute a task. So first, to get a
118. 4:00baseline, I wrote a script that's going
119. 4:02to test the model on about 21 different
120. 4:05variations on what it's to think or not
121. 4:07to think. And I'm not expecting it to do
122. 4:09well because in the model's
123. 4:11documentation it says you are supposed
124. 4:12to fine-tune the model on your own
125. 4:15actual functions, so let's just see how
126. 4:17well it does. And okay, to no one's
127. 4:19surprise, the model performed bad as we
128. 4:22expected. It scored a 52% getting 11 out
129. 4:25of 21 questions right, and the average
130. 4:28routing time was about 1,700
131. 4:30milliseconds, which is
132. 4:31>> [music]
133. 4:32>> pretty slow. So now we're going to train
134. 4:33the model. So basically you got to have
135. 4:35a whole bunch of examples on what your
136. 4:37input is and what the model should say.
137. 4:40I did that with a total of 200 examples,
138. 4:42100 for non-thinking, 100 for thinking.
139. 4:44We're going to train the model on this
140. 4:46and then we're going to see how it does.
141. 4:47So the trained model did a lot better.
142. 4:49It scored a 95% getting 20 out of 21
143. 4:52right, and also the average routing time
144. 4:54was 1,000 milliseconds, which is pretty
145. 4:57[music] fast. So if we do a quick VRAM
146. 4:59check, we know our operating system
147. 5:00takes about 2.1 GB of VRAM, and now with
148. 5:03Function Gemma combined with Qwen 3
149. 5:05we're using another 2.3 GB of VRAM,
150. 5:08which means we only have about 3.5 GB of
151. 5:11RAM left. So we're starting to cut it
152. 5:13close, and we still need to be able to
153. 5:14speak to it and for it to speak back. So
154. 5:17let's set that up. Well, first let's
155. 5:18handle text-to-speech. And the model
156. 5:20that we're going to use for this is
157. 5:21called Piper TTS. And what makes Piper
158. 5:23TTS so good is one, it's ran on the CPU,
159. 5:26not the GPU. So we can run this and it's
160. 5:29not going to slow down our Qwen 3 and
161. 5:30Function Gemma. But also, usually if you
162. 5:33run a model on your CPU, it's slow, but
163. 5:35Piper is insanely fast.
164. 5:37>> [music]
165. 5:38>> So if we go ahead and run our
166. 5:41demo file, let's just say type in
167. 5:44"Hello." And let's see.
168. 5:46Hello. Okay.
169. 5:48>> assist you today?
170. 5:48>> All right, that's That's pretty fast.
171. 5:51Let's ask the little harder question.
172. 5:52Let's see if it thinks. Explain inverse
173. 5:55kinematics.
174. 5:58All right, we see the thinking coming
175. 5:59up. Perfect.
176. 6:02>> Inverse kinematics is the process of
177. 6:04determining the joint angles of a
178. 6:05robotic arm or mechanism required to
179. 6:07achieve a specific end effector position
180. 6:09and orientation.
181. 6:11>> Okay, so it's fast. The voice sounds It
182. 6:14sounds half decent. I mean, we're
183. 6:15working with what we have. So now, we
184. 6:17need to add two things. We need to add a
185. 6:18wake word, and we also need to add
186. 6:20speech to text. So if you guys have
187. 6:22watched the channel for any amount of
188. 6:24time, you will know my go-to wake word
189. 6:26is Ada. But in order to do this locally,
190. 6:29I'll have to train the speech to text
191. 6:30model on me saying Ada a trillion times.
192. 6:33>> [music]
193. 6:33>> And honestly, I really don't want to do
194. 6:35that, especially because all my code is
195. 6:37open source. As soon as I release it,
196. 6:39you guys always ask, "How do I change
197. 6:41the wake word name? Does the name have
198. 6:42to be Ada?" So instead, we're going to
199. 6:44use a pre-trained model that uses a wake
200. 6:46word that hopefully everyone is happy
201. 6:48with, which is Jarvis. So the way this
202. 6:50is going to work is that as soon as I
203. 6:52say Jarvis, the wake word model is going
204. 6:54to flag the speech to text model to
205. 6:57start converting my speech to text. Once
206. 6:59that speech is converted to text, it's
207. 7:01going to go to function Gemma. Function
208. 7:03Gemma is going to say, "Hey, is this a
209. 7:04thinking question or a non-thinking
210. 7:06question?" It's going to send that to
211. 7:07Quinn. Quinn is going to give us our
212. 7:09answer, and that answer is going to be
213. 7:10converted to speech. All right, so we
214. 7:13activated speech to text, and I'm
215. 7:15talking. As you can see, nothing's
216. 7:17happening. But as soon as I say Jarvis,
217. 7:20how are you?
218. 7:22It listens.
219. 7:23>> I'm good. How is everything going?
220. 7:25>> Beautiful.
221. 7:26Jarvis,
222. 7:28explain AI to me.
223. 7:33It does the thinking.
224. 7:34>> AI, artificial intelligence, is a
225. 7:36technology that allows machines to
226. 7:37perform tasks that would typically
227. 7:39require human intelligence, like
228. 7:41learning, reasoning, or problem solving.
229. 7:43>> All right, so it works and it works
230. 7:45pretty well, but there's no user
231. 7:47interface, there's no GUI. If it was up
232. 7:50to me, I would have just reused my Ada
233. 7:52GUI, but we have a problem. We're
234. 7:55already using 56% of our RAM and the Ada
235. 7:59GUI is very, very resource intensive.
236. 8:02It's completely modular, has a whole
237. 8:04bunch of animations, it has hand and
238. 8:06gesture control. It's just too much for
239. 8:09a local AI. So, we're going to remake
240. 8:11the GUI with the focus on one being
241. 8:14lightweight and two having a lot of
242. 8:16features that Jarvis can directly
243. 8:17interact with. Also, if you guys have
244. 8:19been liking the content, make sure you
245. 8:20like the video and if you want to see
246. 8:22more of this type of stuff, subscribe to
247. 8:24the channel. So, when designing the user
248. 8:26interface, the first thing I thought
249. 8:27about were all of the daily tasks that I
250. 8:29struggle with that I'm going to need
251. 8:31help with. And the first thing is
252. 8:32obvious, I suck at planning things. So,
253. 8:34I split the planning task up into
254. 8:36multiple sections. The first, a regular
255. 8:38to-do list. So, let's say if I need to
256. 8:40do shopping, I can add that to my to-do
257. 8:43list, I could complete it, I could
258. 8:47uncomplete something and I could also
259. 8:49delete. The next thing that I wanted to
260. 8:51add was just adding things to calendar.
261. 8:53So, you have your calendar
262. 8:55and then let's say
263. 8:57dinner,
264. 8:59we set the time
265. 9:01for later in the day,
266. 9:04set it, we say it's personal,
267. 9:06and boom, we have dinner.
268. 9:09Then the last two things are just pretty
269. 9:10simple, a simple timer and then also
270. 9:14setting alarm so I can wake up in the
271. 9:15morning. The next thing I wanted to add
272. 9:17was just keeping up with news and
273. 9:19current events. So, I wrote a custom
274. 9:21script that runs in the background and
275. 9:23it checks for news and it also pulls the
276. 9:24top 10 or so headlines. And then of
277. 9:26course, I made sure to add smart home
278. 9:28control so we can turn on and off the
279. 9:30lights and also change the colors. So,
280. 9:34now our UI has all this functionality,
281. 9:37but how do we get Jarvis to control it?
282. 9:39And that's honestly the easy part
283. 9:40because we already done it. So,
284. 9:42originally we trained Function
285. 9:43Juggernaut on two functions, thinking or
286. 9:46non-thinking. But, what if we trained it
287. 9:48on turn on the lights or add this task
288. 9:51or add this event to my calendar? So, I
289. 9:53put together about 10 functions for me
290. 9:56to retrain Function Juggernaut on so it
291. 9:58can have complete control over
292. 10:00everything that the UI could do. So,
293. 10:03let's put it all together and let's test
294. 10:05it. Jarvis,
295. 10:06add buy groceries to my to-do list.
296. 10:11>> I've added buy groceries to your to-do
297. 10:12[music] list.
298. 10:13>> Nice. Jarvis,
299. 10:15set an alarm for 8:00 a.m.
300. 10:20>> Alarm set for 8:00 a.m. [music]
301. 10:22>> Jarvis,
302. 10:23set a timer for 1 minute.
303. 10:27>> Timer set for 1 minute.
304. 10:29>> Jarvis,
305. 10:30anything important happening on the
306. 10:32news?
307. 10:38>> The top news includes Trump's potential
308. 10:39[music] visits to Denmark and seven
309. 10:41countries, the red purple star sapphire
310. 10:44from Sri Lanka, a Ford scandal, [music]
311. 10:46and ICE arrests impacting 2026 races.
312. 10:49>> Jarvis, can you go ahead and turn off
313. 10:52the lights in the office?
314. 10:58Nice.
315. 11:00>> I'll turn off the office lights.
316. 11:02>> Nice.
317. 11:02>> [music]
318. 11:03>> Jarvis,
319. 11:04can you turn on the office lights?
320. 11:10>> I'll turn on the office lights.
321. 11:12>> Jarvis, [music]
322. 11:14change the office lights color to blue.
323. 11:21>> I'll change the office lights to blue.
324. 11:24>> So, those are the main features. Also,
325. 11:26the one thing that I really wanted to
326. 11:28add that I just could not get to work
327. 11:30was the browser agent. But, if by any
328. 11:32chance you are interested in the browser
329. 11:34agent and you want to see it working
330. 11:35completely, doing some [music] shopping
331. 11:37for me, doing some research for me, you
332. 11:39can check out this video of Ada because
333. 11:40the browser agent is working perfectly
334. 11:42there.