Run AI Models Locally: Zero API Keys, Zero Fees with Docker Desktop Model Runner — Transcript

Full transcript

1. 0:00Hey developers, tired of sending your
2. 0:02data to the cloud APIs just to use AI in
3. 0:05your applications? What if you could run
4. 0:07powerful AI models right on your Mac
5. 0:09with zero API keys, zero data sharing,
6. 0:12and zero monthly fees? Today, I'm going
7. 0:15to show you how Docker Desktop's new
8. 0:17model runner feature lets you run AI
9. 0:20models locally and how to integrate them
10. 0:22with your Spring Boot applications using
11. 0:24Spring AI. In just 15 minutes, you have
12. 0:27a fully functional AI powered Java
13. 0:29application running completely on your
14. 0:32own
15. 0:34machine. So here I am over on the docker
16. 0:38model runner uh documentation. Uh this
17. 0:41is a new feature for Docker Desktop. You
18. 0:43can see here that it requires Docker
19. 0:45Desktop 4.40 and later. And I think this
20. 0:49is only for Docker Desktop with Mac and
21. 0:52Apple silicone at the time. I believe
22. 0:55this will uh branch out into other
23. 0:57platforms later on. So the Docker model
24. 1:00runner plugin lets you pull models from
25. 1:03DockerHub. So you can run open- source
26. 1:05models. You can run AI models directly
27. 1:08from the command line. You can manage
28. 1:10local models. You can add new ones, list
29. 1:12them, remove them, interact with models
30. 1:15using a submitted prompt or in a chat
31. 1:17mode. So this is very similar to
32. 1:20something like I've covered in the past.
33. 1:22If you want, I'll leave a link in the
34. 1:24description below. Uh, but this is a new
35. 1:27feature from Docker. I saw this
36. 1:28announced at Java 1. I was really
37. 1:30excited about this. Another way to kind
38. 1:32of interact with local models. So once
39. 1:36you have it running, um, once you have
40. 1:38it installed, you can do a couple of
41. 1:40things. You can check the model status.
42. 1:42You can run help, which will give you a
43. 1:44bunch of commands that you can run. So
44. 1:46if you want to list the models that
45. 1:47you've already installed locally, if you
46. 1:49want to pull a new one down from Docker
47. 1:51Hub, you can. You can remove, you can
48. 1:54run status and version. So here's where
49. 1:57it starts. Um, we'll go into Docker
50. 1:59Desktop. I'll show you this in a second,
51. 2:00but I just want to go through the
52. 2:02documentation first. So you docker model
53. 2:05pull and then pull a specific model that
54. 2:07you want to use. Today I'm going to use
55. 2:09Google Gemini's Gemma 3, which is their
56. 2:11open- source model. uh I've been finding
57. 2:13it uh really useful uh for testing on my
58. 2:16local machine. And so you'll say docker
59. 2:19model pull and then AI and then the name
60. 2:21of the model. Once it's done, it's
61. 2:24available to use. You can use it right
62. 2:25in the terminal on your machine and and
63. 2:28chat with it interactively and do some
64. 2:31things there. But then there's also a
65. 2:32way that we can use it in our
66. 2:34applications.
67. 2:36So if you ever want to see which ones
68. 2:38you have available, you can list um you
69. 2:40can run a particular model and then uh
70. 2:43it goes into some other things. The big
71. 2:46thing here is that if you want to
72. 2:48integrate it into say an AI application
73. 2:50that we're building using something like
74. 2:52Java and Spring, then you'll need to
75. 2:54make sure that there is a feature
76. 2:57toggled on for making TCP available.
77. 3:00Once you do that, it exposes a localhost
78. 3:04um network over HTTP and you can do a
79. 3:08bunch of things to it like you can get a
80. 3:10list of engines, you can do some other
81. 3:12things with it. So um that's the
82. 3:15documentation and then uh you can
83. 3:18interact with it through the OpenAI API.
84. 3:21So in Java and in Spring AI, we'll use
85. 3:23the OpenAI API and we'll talk to it just
86. 3:26like we would uh OpenAI's
87. 3:31GPT40. So that's the documentation. I'll
88. 3:34leave a link to that in the description
89. 3:36below. Uh but with that, let's take a
90. 3:38look at Docker Desktop. Uh we'll see
91. 3:40what we need to configure there and then
92. 3:42we'll go ahead and take a look at how to
93. 3:44pull a model. And um oh before we do
94. 3:47that, I want to look at one more thing.
95. 3:49So if you're using something like Olama,
96. 3:52there's a list of models that are
97. 3:54available. The same thing is here. So
98. 3:56what models are available? You can open
99. 3:57up this link and you can see the
100. 3:59different models. So there's uh Llama
101. 4:023.3 um and so on. So I'm going to use
102. 4:06Gemma. So if you go down here, you can
103. 4:08see Gemma 3. There are some different
104. 4:11flavors of it, but it gives you the
105. 4:12command here. So Docker model pool AI
106. 4:14Gemma 3. So you can copy that if you
107. 4:16want. And then uh there are a couple
108. 4:19different models here. One is 7.7 GB,
109. 4:22one is 2.5 GB, and I believe I just
110. 4:25installed this. So if you say pull AI
111. 4:28Gemma 3, it's going to pull the latest,
112. 4:30which is this one. If you want the
113. 4:32larger one, uh then you'll want to be
114. 4:34specific and give it that as a
115. 4:37parameter. Cool. So with that, let's
116. 4:39head over to the desktop and take a look
117. 4:41at Docker Desktop. All right. So once
118. 4:43you have the correct version of Docker
119. 4:45Desktop installed, you want to go up to
120. 4:47settings and down to our uh features and
121. 4:51development. And this is where it says
122. 4:53enable docker model runner. And you'll
123. 4:56want to make sure that this check box is
124. 4:59checked. So enable host side TCP
125. 5:01support. I'm going to leave it on the
126. 5:03default port of 1 2 4 3 4. With that,
127. 5:06you have everything you need to get up
128. 5:08and running. So again, you can go here
129. 5:10and say dock docker model list and it
130. 5:14will list out all of the uh available
131. 5:16models that you have pulled. Now I went
132. 5:18ahead and pulled down that Gemma 3 model
133. 5:20just so we didn't have to wait for me to
134. 5:22install it. But if you wanted to
135. 5:24yourself, you can say docker model pull
136. 5:27uh and then the name of the model that
137. 5:29you want to pull down. But because I
138. 5:31already have that running, I can go
139. 5:32ahead and say docker model run and give
140. 5:35it that name. So AI Gemma 3. And now I
141. 5:38can interact with it locally uh in my
142. 5:40terminal. So I can say what is an
143. 5:44interesting fact about Docker and we'll
144. 5:48see this and it will come back and we
145. 5:50get that nice streaming response. So
146. 5:52similar to how we would with something
147. 5:54like chat GBT, we get this nice kind of
148. 5:56conversational um text window that we
149. 5:59can just kind of chat back and forth
150. 6:01with. Again, the important thing to
151. 6:03remember here is now that we are running
152. 6:05a model locally on our machine to be
153. 6:08able to do this and we are not paying
154. 6:10for this. We are not sending our data to
155. 6:12the cloud. Uh this is all running on my
156. 6:15on my machine and running pretty fast, I
157. 6:17might add. So, um I don't know how it
158. 6:21says this works with Docker or Apple Mac
159. 6:25silicone. So, I'm running an M4 pretty
160. 6:28decent machine. So that's why I'm
161. 6:30probably getting some pretty good
162. 6:32performance out of this. Um, but I'd be
163. 6:34curious if anyone anybody's running uh
164. 6:36something else, let me know how the
165. 6:38performance is for you. So that is um
166. 6:42running. With that running, uh we can go
167. 6:45ahead and create an app to talk to this
168. 6:46model locally. All right. So I do have a
169. 6:48GitHub repo here under docker model
170. 6:51runner. Uh this has all the explanation
171. 6:53on how to get up and running with this.
172. 6:55Basically what we're going to do here
173. 6:56today. Uh but I'll walk you through
174. 6:58that. So, I'm going to go over to
175. 7:01start.spring.io, pick the latest version
176. 7:03of Spring Boot. I'm going to call this
177. 7:05dev.dan Vega. We'll call this Docker
178. 7:08Model Runner. And I'll put this in a
179. 7:12package called Docker. So, I'm going to
180. 7:14pick a couple of dependencies here. I'm
181. 7:16going to choose web and then I'm going
182. 7:18to choose OpenAI because again, this is
183. 7:21OpenAI compatible. So, this will work
184. 7:23with that. That's all of the settings or
185. 7:26all of the dependencies that we need. We
186. 7:28can go ahead and click generate. This
187. 7:29will download a zip file. You can open
188. 7:31up this in whatever ID or text editor
189. 7:34you're most productive in. For me,
190. 7:36that's going to be Intelligj. But with
191. 7:37that, I think we have everything we
192. 7:39need. So, let's write some code. All
193. 7:40right. So, here I am in my application.
194. 7:43Uh, the first thing I need to do is set
195. 7:44up a couple of properties. Whenever
196. 7:46you're working with a large language
197. 7:47model here in Spring AI, you need to
198. 7:49define a couple of properties. Now, the
199. 7:51first one is the Open AI API key. We
200. 7:55actually don't need an API key because
201. 7:57we are not talking to OpenAI. We are
202. 7:59talking to a local model. But uh Spring
203. 8:02AI requires an OpenAI key when you're
204. 8:04using the OpenAI Spring Boot starter. So
205. 8:07I'm just going to put in an empty API
206. 8:09key. We don't actually need one here. Uh
207. 8:11then I'm going to go ahead and set the
208. 8:14base URL. So this is going to be because
209. 8:17we uh enabled that TCP option in Docker
210. 8:20Desktop. We are going to have something
211. 8:23running on localhost one two uh what was
212. 8:26it? Uh 434 I believe. Yes. And then uh
213. 8:30the base URL for this is going to be
214. 8:32engines
215. 8:35llama.cpp. Now the other thing is there
216. 8:38is a completions path. This is what it
217. 8:41looks like. I'm not going to enable that
218. 8:43because we are not changing that. This
219. 8:44is what we need. So this plus the base
220. 8:47URL is the full URL to uh the API path.
221. 8:52Uh with that we also need one more. We
222. 8:53got to define the model that we're using
223. 8:55because again with docker model runner
224. 8:57we could be using one of num a number of
225. 8:59models. So here we're going to define
226. 9:02the model that we're using as AI Gemma
227. 9:053. All right with that we have
228. 9:07everything we need to go ahead and write
229. 9:09some code. I'm going to keep this pretty
230. 9:11simple today. I'm going to do this right
231. 9:14in my application class. I'm going to
232. 9:16create a new bean of type command line
233. 9:18runner. Command line runner. And we are
234. 9:21going to pull in the chat client. So if
235. 9:24you're new to Spring AI, you can check
236. 9:25out a lot of videos on my channel about
237. 9:27this. But the chat client is basically
238. 9:30the abstraction to talk to whatever
239. 9:34model we've configured. So we've
240. 9:36configured Gemma 3 underneath the hood.
241. 9:38But the nice thing about this is this
242. 9:40chat client is going to be kind of
243. 9:42universal, right? Like we don't need to
244. 9:44change code based on whatever model
245. 9:46we're talking to. So now I can use that
246. 9:48builder and build a new client. I'm
247. 9:50going to call this client. And then I
248. 9:53can use the client to send a prompt. So
249. 9:56I'm going to say
250. 9:58client.prompt. And in that prompt, I'm
251. 10:00going to ask a similar question. Um,
252. 10:02something like when was Docker created?
253. 10:07And then we can go ahead and make a call
254. 10:09to it. This is a blocking call. If you
255. 10:11wanted to do a streaming call, you
256. 10:12could. Uh, but this will be just fine
257. 10:15for this example. And then what I want
258. 10:17to get back is just the response. I just
259. 10:19want that string response that comes
260. 10:21back. I'll create a variable from this
261. 10:23called response. And then I'll go ahead
262. 10:25and output the response. So with that,
263. 10:28that is all we need to get up and
264. 10:30running. Hopefully this uh works. Let's
265. 10:32go ahead and give it a try. And if we
266. 10:34run this, this will execute after the
267. 10:38application context has uh started up.
268. 10:41And we do have an error here. But that's
269. 10:43okay. This is fun. Let's figure out
270. 10:45what's going on together here and see if
271. 10:48we can't figure this out.
272. 10:50Um, so we are sending this to so I uh
273. 10:56post for request. Um, null. Oh, and it
274. 11:00looks like I have a typo here. That
275. 11:03would definitely do it. Let's go ahead
276. 11:05and fix
277. 11:07that. And it looks like it's running. So
278. 11:10here, Docker was officially created in
279. 11:12July of 2013. Here's a breakdown of the
280. 11:16key
281. 11:17milestones. So that's awesome. We were
282. 11:19able to use Docker Desktop's new um
283. 11:23Docker model runner and we were able to
284. 11:26pull in a local open-source free model
285. 11:31that we can use right on our laptops.
286. 11:33This is really great because we don't
287. 11:34have to have any API keys. We don't
288. 11:36occur any monthly billing. We just kind
289. 11:39of get to test some things out locally.
290. 11:41And again, uh we could take these models
291. 11:44to production. So if you wanted to host
292. 11:46these models uh within your
293. 11:48organization, you could as well. Uh but
294. 11:51for me as a developer, this is really
295. 11:53great to be able to kind of test this
296. 11:55stuff out locally and using a tool that
297. 11:57I love to use, which is Docker Desktop.
298. 11:59So I hope you found this useful. I'm
299. 12:01really excited for this new feature and
300. 12:03taking a look at their road map and
301. 12:04seeing what they're they're interested
302. 12:06in in putting together on this. I think
303. 12:08this is a great alternative to a llama.
304. 12:11Uh so hopefully you do, too. And
305. 12:14hopefully you learned something new
306. 12:15today. I I had a lot of fun putting this
307. 12:16together. If you learned something new
308. 12:18today, do me a big favor. Leave me a
309. 12:20thumbs up, subscribe to the channel, and
310. 12:23as always, happy coding, friends.
311. 12:26[Music]