Microsoft Joins the Local AI Push — Transcript

Full transcript

1. 0:00Okay, so yesterday Microsoft held their
2. 0:02Windows and Surface event in San
3. 0:04Francisco. In this video I want to talk
4. 0:06about some of the interesting things
5. 0:07that they showed there. So they had an
6. 0:09interesting section where they showed
7. 0:111.6 million tokens. That was how much
8. 0:13one coding agent session sent into a
9. 0:16model during one of the demos. And they
10. 0:18made it clear that the bill for that was
11. 0:20zero. Not because Microsoft was picking
12. 0:22up the tab or anything like that, but
13. 0:24because the model was running on the
14. 0:25laptop itself. And I got to say that it
15. 0:28didn't look like a toy demo either. The
16. 0:30agent was investigating some GitHub
17. 0:32issues on the Windows terminal code
18. 0:34base. And maybe even the most
19. 0:36interesting part here was that they
20. 0:37claimed that it wasn't actually even the
21. 0:39developer who decided it should run
22. 0:41locally. It was the tool. And while most
23. 0:44of the coverage that I've seen about
24. 0:45this event really focused on the Surface
25. 0:47Laptop Ultra and the Nvidia RTX Spark
26. 0:50chip here, both of those looked really
27. 0:53cool. Although definitely a steep price
28. 0:55tag there. A lot of that was actually
29. 0:57announced back in June. Really the big
30. 0:58thing here I guess is the shipping
31. 1:00dates, the pricing, that kind of thing.
32. 1:02But for me the interesting thing here
33. 1:03was what is Windows actually going to do
34. 1:06with this powerful processor? And it
35. 1:09seems one of the key things that it's
36. 1:11going to do is decide which of your
37. 1:13tokens run on your machine and which of
38. 1:15them go to the cloud. So Microsoft is
39. 1:18calling this hybrid intelligence. They
40. 1:20had the person who leads the Windows
41. 1:22team actually described it as
42. 1:24intelligent routing. Basically the
43. 1:25models, the runtime, and the chip set
44. 1:28all working together. So let's start
45. 1:30with the router that's the piece that
46. 1:32actually made the demo work. So in the
47. 1:34GitHub Copilot demo, the presenter had
48. 1:36the model picker set to auto. And the
49. 1:39new version of auto doesn't just choose
50. 1:41between cloud models anymore. It can
51. 1:44also choose a local model. So when she
52. 1:46gave it a pretty basic sort of clean up
53. 1:48task on the Windows terminal code base,
54. 1:51it actually sent that to the Microsoft
55. 1:53AI code flash model that was running on
56. 1:56the laptop. And you could actually see
57. 1:58the GPU ramping up on the screen. She
58. 2:01even pointed out that you could put the
59. 2:02whole machine into airplane mode and it
60. 2:05would still keep going. After that
61. 2:07though, she then gave it something
62. 2:08harder to go through all the issues that
63. 2:11were tagged needs repo. When she did
64. 2:13that, what happened there was a whole
65. 2:15different thing. So basically a cloud
66. 2:17model, it was one of the open AI models
67. 2:19GPT-6 or something, did the planning,
68. 2:22then three sub agents ran locally on
69. 2:24that Microsoft AI code model. So this is
70. 2:27a very standard pattern we've seen,
71. 2:29right? The expensive model does the
72. 2:30thinking, the local cheap models do the
73. 2:33bulk reading. And that's where we get to
74. 2:35the actual 1.6 million tokens coming in
75. 2:37here, right? The majority of those
76. 2:40tokens, in fact, I think all but maybe
77. 2:42like 10,000 or so, were running locally.
78. 2:45And that's where the 1.6 million tokens
79. 2:47come from. 1.6 million in and a bit over
80. 2:5110,000 out. That's roughly 150 to 1. And
81. 2:55more and more this is the kind of
82. 2:56pattern that you see if you're sort of
83. 2:59building things with multiple agents,
84. 3:01where the context window keeps getting
85. 3:02longer and you keep sending it through
86. 3:04to generate something small out. And I
87. 3:06thought this this was really interesting
88. 3:08to see basically how they handled this
89. 3:11routing between the local model and the
90. 3:13cloud model. Because I do think that
91. 3:15this is something we're going to see
92. 3:16happen more and more going forward.
93. 3:18While it's not in the interest of
94. 3:20companies like OpenAI or Anthropic to
95. 3:23really talk about that, it is in the
96. 3:25interest of any of the hardware makers
97. 3:27and even companies like Microsoft who
98. 3:29are making some of the software and the
99. 3:31hardware, not just the actual models. So
100. 3:34this is going to ship in the GitHub
101. 3:35Copilot app on October 15th. And it
102. 3:38seems that that app is getting both
103. 3:39local context, local actions, local
104. 3:42models over the next few months. And
105. 3:44it's really going to be something that's
106. 3:45very interesting to sort of look out
107. 3:47for. Because we know that every router
108. 3:49lives or dies by what happens when it
109. 3:51gets it wrong. If the locomotive can't
110. 3:53handle the task it was sent, does it
111. 3:55then know that and send it back to the
112. 3:57cloud, or do you just get a worse answer
113. 4:01and never actually know what failed and
114. 4:02what didn't work out the way you wanted
115. 4:04it. Unfortunately, that wasn't something
116. 4:06that they showed here, but it's
117. 4:07certainly going to be interesting to see
118. 4:09when we've seen things like the whole
119. 4:11model router idea with things like PAIR
120. 4:14from NVIDIA through to other projects
121. 4:16and ideas that people have put out there
122. 4:18of how this is actually going to work.
123. 4:20All right, so what did they actually say
124. 4:22runs locally? So, first off, this was
125. 4:24the Microsoft AI code 1.1 flash model.
126. 4:28This is the coding model that Microsoft
127. 4:30introduced at the Build conference. And
128. 4:32the thing that kind of amazes me here is
129. 4:34that they've taken it down to three
130. 4:36bits, which has cut the size by close to
131. 4:4080%.
132. 4:41And they've got it running on these
133. 4:43devices with a 256K context window fully
134. 4:47local in there. Now, I've got to say
135. 4:49that I'm on the fence a lot about a
136. 4:51three-bit quantized version of a model,
137. 4:54especially when you're trying to use it
138. 4:56for something like coding. I think
139. 4:58time's going to tell either they've
140. 4:59worked out something really good that
141. 5:01they're actually doing there, or this is
142. 5:03going to be calling the cloud for any
143. 5:05sort of coding task that's remotely
144. 5:07difficult. And that same idea carries
145. 5:09into the second model, right? So, they
146. 5:11talked about a new coming Nemotron model
147. 5:14that runs at two bits in about 20 gigs
148. 5:17of memory. And interestingly, they're
149. 5:19pitching that as a good fit for the RTX
150. 5:21Spark. So, don't forget the RTX Spark
151. 5:24now is not just going to come in a 128
152. 5:26gig version. We've already seen the DGX
153. 5:29Sparks include a 64 gig version, and it
154. 5:32seems that the RTX Spark might even have
155. 5:34variations that go down to a 24 gig
156. 5:37version. Now, the third one that they
157. 5:39showed, and I got to say I'm a bit
158. 5:40suspicious about this one. I really hope
159. 5:42I'm proven wrong, but this was them
160. 5:44running DeepSeek 4 locally at 1.6 bits
161. 5:49in about 60 gigs of memory. The line
162. 5:52that they used on stage was something
163. 5:53along the lines of that a year ago this
164. 5:56capability was only available from the
165. 5:58Frontier cloud models. And for most
166. 6:00people that is kind of true. Now you can
167. 6:02run DeepSeek 4.1 on multiple DGXs. I
168. 6:06actually do that. It's one of my
169. 6:07favorite models that I really like. But
170. 6:09it's going to be really interesting to
171. 6:11see what a 1.6 bit version of that model
172. 6:14actually is like. Now to be fair,
173. 6:16they're actually saying that the 1.6
174. 6:18bits isn't average. So in practice that
175. 6:21means some layers are going to be kept
176. 6:22at higher precision and a lot of them
177. 6:24are going to be pushed very low. So you
178. 6:26try and keep the layers that are the
179. 6:28most important at higher precision
180. 6:30levels and then layers that don't get
181. 6:32activated as much, you then go for lower
182. 6:34precision on those. The real question is
183. 6:36though, how much of that quality
184. 6:38actually survives? So the claim on stage
185. 6:40was that it outperforms on coding and
186. 6:43reasoning benchmarks. But they didn't
187. 6:45really clarify what that meant. And I
188. 6:47would say that there really sort of
189. 6:48three key things here. One, does the
190. 6:51coding pass rate hold up against a
191. 6:53higher precision version of the same
192. 6:55model? I'm very suspicious of whether
193. 6:57that actually does. Secondly, does it
194. 6:59fall apart at long context? And then
195. 7:01thirdly, if you've got sort of
196. 7:03complicated use of tools, does it
197. 7:06actually still produce clean tool calls?
198. 7:09Because structured output is often the
199. 7:11first thing to break when you push
200. 7:13quantization too far. And for any sort
201. 7:16of agentic task, a broken tool call is
202. 7:18generally going to be a failed task. The
203. 7:20other thing you got to remember too is
204. 7:22that if you're really going for a sort
205. 7:24of long context like 256K,
206. 7:27that's not going to be free, right? At
207. 7:29that length your KV cache takes up a lot
208. 7:31of memory on top of the actual weight.
209. 7:34So there's a lot of nice tricks of where
210. 7:35people quantize that KV cache where
211. 7:38that's down to 8 bit, 4 bit, perhaps
212. 7:40even lower than that now. But my guess
213. 7:43is you're probably going to need one of
214. 7:44the bigger versions of this machine with
215. 7:46a large amount of memory to be able to
216. 7:48do that. Now, another interesting point
217. 7:50just to raise here that I haven't really
218. 7:52talked about on the channel before is
219. 7:54that it's a very interesting debate of
220. 7:57do you go for a bigger model with a
221. 8:00lower quantization rate, or do you go
222. 8:03for a higher quantization rate with a
223. 8:05smaller model? And I personally I've
224. 8:07been seeing as long as your
225. 8:08quantizations don't go below 4-bit,
226. 8:11actually going for the bigger model
227. 8:13usually gives you a better win. But,
228. 8:15that's mostly me saying that
229. 8:16anecdotally. I can't recall off the top
230. 8:18of my head any sort of papers or
231. 8:20anything that have been talking about
232. 8:21that. All right. So, to be able to run
233. 8:23these models on the machine, you're
234. 8:25going to need a runtime. And one of the
235. 8:27announcements that they had, I think, is
236. 8:29just a clear win here was the whole idea
237. 8:32of llama.cpp inside of Windows ML. So,
238. 8:35Windows ML is Microsoft's runtime for
239. 8:37deploying models across GPUs, NPUs,
240. 8:41CPUs. And the cool thing here is that
241. 8:43they're bringing llama.cpp into it. So,
242. 8:45why does that matter? Basically, it's
243. 8:47going to allow for a lot of
244. 8:48compatibility. It means open models on
245. 8:50day one. So, when a new model drops and
246. 8:53there's a GGUF quant that shows up a few
247. 8:55hours later, you're not having to wait
248. 8:57for Microsoft to package it. And that's
249. 9:00especially true if you're not just using
250. 9:02the normal versions, perhaps you're
251. 9:03using an uncensored version, a
252. 9:05specifically fine-tuned version, which
253. 9:08Microsoft is never going to convert for
254. 9:10their sort of system. So, having
255. 9:12llama.cpp in there now gives you that
256. 9:14ability to just pick the models that you
257. 9:16want to run and actually run them. The
258. 9:19other cool thing here is that if you are
259. 9:21building a Windows app that ships with a
260. 9:23local model, you've now got this one
261. 9:26runtime that handles whatever the
262. 9:28hardware your user happens to have. I'm
263. 9:30guessing as long as that hardware
264. 9:32probably is reasonably recent. The thing
265. 9:34I I know yet, and I'll be looking into,
266. 9:36is how you can actually plug your own
267. 9:38local model into the Copilot router here
268. 9:41or whether it's only the Microsoft
269. 9:43picks. I got to say, if you could plug
270. 9:45in any model there, this could be a huge
271. 9:48win for Microsoft in that you could just
272. 9:51basically decide, "Oh, I want Qwen 4 35B
273. 9:56mixture of expert model plugged into my
274. 9:58OS and it just gets automatically used
275. 10:00by a whole bunch of different apps or
276. 10:02something." All right, the next thing up
277. 10:04was once you've got capable local models
278. 10:06here is obviously how do you stop them
279. 10:09from doing things that you don't want
280. 10:10them to do. And in many ways what
281. 10:12Microsoft announced here was their
282. 10:15version of Nvidia's open shell or their
283. 10:18version of Docker sandboxes, etc. And
284. 10:21this is called the Microsoft execution
285. 10:23containers, MXC. It's basically a
286. 10:25runtime sandbox that organizations can
287. 10:28set policies for what an agent can
288. 10:30access, and then Windows itself enforces
289. 10:33them while the agent runs. The bit that
290. 10:35I think here is really quite interesting
291. 10:37is the whole thing around agent
292. 10:38identity. Every action the agent takes
293. 10:41is attributed to the agent, not to you.
294. 10:44And so you can imagine that this helps
295. 10:46both in sort of tracing things, but also
296. 10:49when you're actually working on
297. 10:51something, if the agent actually goes to
298. 10:53do something, you can actually block it
299. 10:55and then it loses access to take that
300. 10:58action, but you as the logged-in user
301. 11:00still can have access. All right, lastly
302. 11:02up I guess was the hardware itself and
303. 11:04I'm going to say Microsoft's done a
304. 11:06really nice job at design here. These
305. 11:08look really impressive. And don't
306. 11:10forget, these are just the first of the
307. 11:12RTX sparks that we're seeing coming out.
308. 11:14These machines are up to 128 gigs of
309. 11:17unified memory, they have full CUDA
310. 11:19support, and it's all running on a
311. 11:21Windows laptop. You don't have to go
312. 11:23into Linux to actually do this. So
313. 11:26reading some of the coverage, I think
314. 11:27Engadget puts this GPU at around about
315. 11:30an RTX 5070 level
316. 11:33with 20 CPU cores and it won't draw more
317. 11:35than 80 watts. Now, the prices for this
318. 11:37started I think around about $2,600,
319. 11:40but obviously the more memory that you
320. 11:42put into this, the more that price goes
321. 11:44up. And I guess there's really nothing
322. 11:46that Microsoft or Nvidia can really do
323. 11:49to prevent that. The whole memory and
324. 11:51RAM issues that are going on at the
325. 11:52moment is just making all of this
326. 11:54hardware get more and more expensive.
327. 11:57Along with a laptop version, they've
328. 11:58also got a desktop hearing there. And
329. 12:01interestingly, they even teased a
330. 12:02Windows version of the DGX Station
331. 12:05coming later on this year. So, that's
332. 12:07going to be interesting to see. I think
333. 12:09more for unlocking the ability for other
334. 12:11people to be able to actually use these.
335. 12:14So, hopefully at some point I will get
336. 12:15my hands on an RTX Spark and be able to
337. 12:17run some tests on it myself and see just
338. 12:20how these things perform and work out
339. 12:22for which group of people is this going
340. 12:23to be a viable alternative to a DGX
341. 12:26Spark, etc. Anyway, so to pull it all
342. 12:28together, I thought this was actually
343. 12:30interesting than usual event from
344. 12:32Microsoft because it really planted a
345. 12:34flag of them saying that you're going to
346. 12:36need to be able to run models locally
347. 12:38and not just use the cloud. And I think
348. 12:41by them getting behind it really is an
349. 12:43important step in the direction that
350. 12:45local AI is going. So, no longer do we
351. 12:47just have sort of enthusiasts or ML
352. 12:50people or even just startups sort of
353. 12:53pushing for local AI now. And while
354. 12:55we've had Nvidia clearly pushing for
355. 12:57local AI this year, having Microsoft
356. 13:00join that bandwagon is definitely a step
357. 13:02forward in making all of this the norm
358. 13:04rather than the exception. So, anyway,
359. 13:06let me know in the comments what you
360. 13:07think. What's your take on the RTX
361. 13:09Spark? How do you feel about Windows
362. 13:11adopting Llama.cpp rather than just
363. 13:14making their own protocol or their own
364. 13:16way to do it? I'm really interested to
365. 13:18hear from people about this one. I do
366. 13:20think Microsoft's gone in an interesting
367. 13:21direction here. Anyway, as always, if
368. 13:24you enjoyed the video, please click like
369. 13:25and subscribe and I will talk to you in
370. 13:26the next video. Bye for now.