NVIDIA Doubles Down on Local AI With PAIR — Transcript

Full transcript

1. 0:00Okay, so this past week has been to
2. 0:03steal a YouTube cliche a huge week in
3. 0:06AI. And in many ways this week we've
4. 0:08seen a lot of things that were bubbling
5. 0:10away below the surface for perhaps even
6. 0:136 months for some of these things rise
7. 0:15up to the top and actually get released.
8. 0:18So of course I'm talking about this week
9. 0:20we had the launch of Fable 5.1 and
10. 0:23Mythos from Anthropic. Probably a little
11. 0:25bit more interesting we had the whole
12. 0:28launch of GPT-6 Astra. And I feel that
13. 0:31those models really show that on the
14. 0:33continuum, and I really stress the word
15. 0:36continuum there, we're moving speedily
16. 0:39along towards AGI and perhaps even ASI.
17. 0:43But while there are a sort of 10,000
18. 0:45videos out there covering these models
19. 0:47and showing you all the cool games that
20. 0:49you can build and all those things are
21. 0:51true and very cool. And I've definitely
22. 0:53enjoyed this weekend playing with some
23. 0:54of these models. In this video I want to
24. 0:57focus on Nvidia. Now later on in the
25. 1:00video I'm going to talk about a new
26. 1:01release that they had that still is very
27. 1:03early days but kind of signals where
28. 1:05they're going for a lot of this stuff.
29. 1:07But first off let me show you a few
30. 1:09interesting things that happened with
31. 1:11Nvidia this week. So if you've been
32. 1:13watching my videos for a while you won't
33. 1:15be surprised that one of the things I've
34. 1:17consistently complimented Nvidia on when
35. 1:20they released their open weight models
36. 1:22is that they're not just actually
37. 1:23releasing weights. They're releasing
38. 1:26papers. They're releasing recipes for
39. 1:28training. And often most importantly
40. 1:30they're releasing these data sets which
41. 1:32allow people to sort of see how these
42. 1:34models are being made and then to work
43. 1:37out improvements that they can make for
44. 1:39fine-tuning their own versions. Now
45. 1:41because of that not surprisingly
46. 1:44this year Nvidia has overtaken all the
47. 1:46other enterprise players to be the
48. 1:48number one contributor of repos on
49. 1:51Hugging Face. Now the key thing is that
50. 1:53Nvidia is making a huge bet on open
51. 1:57weights and to use a term that perhaps
52. 1:59was corrupted by another company on open
53. 2:02AI. And by open AI here, I am talking
54. 2:05about the word of open weights, open
55. 2:08models, open data sets, etc. So, back at
56. 2:11the end of July, the CEO of Nvidia
57. 2:13joined Twitter or joined x.com for the
58. 2:16first time. And his first post was
59. 2:19literally this letter talking about open
60. 2:22weights and what Nvidia was doing to be
61. 2:25committed to this and encouraged other
62. 2:27companies to sign it to get on board
63. 2:29with supporting the concepts around open
64. 2:32weights and the open use of AI and not
65. 2:35having it dominated by any one
66. 2:37particular company. And I think in the
67. 2:38end, pretty much all the AI companies
68. 2:40with the exception of Anthropic ended up
69. 2:43signing it. Jump forward back to this
70. 2:45week, Nvidia's doubling down even more
71. 2:48on the concept of open weight models and
72. 2:50making this stuff even more available to
73. 2:52people by buying Hugging Face. And they
74. 2:55certainly paid a large amount for it.
75. 2:57This is just under $13 billion.
76. 3:00So, congratulations to all my friends
77. 3:02who work at Hugging Face. I hope you
78. 3:03weighted your compensation in shares
79. 3:05more than salary because you probably
80. 3:06made a very nice payday here. And one of
81. 3:09the funny things with that acquisition,
82. 3:11the actual amount turns out to be the
83. 3:13decimal value for what now is called the
84. 3:16Hugging Face emoji here. So, contrary to
85. 3:18what a lot of people are saying, I don't
86. 3:20see that Nvidia buying Hugging Face is
87. 3:22actually going to be really bad for the
88. 3:24open community. I think in some ways
89. 3:26this may have actually defended off a
90. 3:28number of other companies that were
91. 3:30trying to buy it that perhaps would have
92. 3:32made changes to it. It's in Nvidia's
93. 3:34interest to keep this open because they
94. 3:36are benefiting so much from this. In
95. 3:39many ways, you could think of Hugging
96. 3:40Face at the moment being the hub of
97. 3:43local AI. And this is what Nvidia really
98. 3:46wants to own a large chunk of. They can
99. 3:49see clearly the changes that are
100. 3:51starting to take effect. And the changes
101. 3:53that I'm talking about is the whole play
102. 3:55of AI hitting mass adoption. So, if
103. 3:58you're listening to this video, you're
104. 4:00almost certainly not one of the masses.
105. 4:02If you've been listening to my videos
106. 4:04for a year or two, you're one of the
107. 4:06super early adopters here. And that may
108. 4:09have been because of your job, that may
109. 4:10have been cuz your interests, etc. But
110. 4:13don't be fooled. While people like you
111. 4:14and I have been watching all these
112. 4:16models come out and getting amazed over
113. 4:18the last X number of years, we are not
114. 4:22the masses. The masses are coming along
115. 4:25and starting to realize that one, AI can
116. 4:27be useful for a lot of tasks. And two,
117. 4:31that probably they don't want to give
118. 4:33away all their data after they've seen
119. 4:35what's happened with a lot of the social
120. 4:36media companies over the years. That's
121. 4:39creating this huge market in local AI.
122. 4:42And rightly so, Nvidia wants to
123. 4:44capitalize on that and capture as much
124. 4:46of that as possible. Now, these are some
125. 4:48of the interesting stats that if you
126. 4:50look at the world of open weight models
127. 4:52and local AI, you can see that
128. 4:55year-to-date already on Hugging Face,
129. 4:57open models are well over four times
130. 5:00what they were last year. And we've
131. 5:02still got a long way to go in the year.
132. 5:04We can see that the whole sort of
133. 5:05generative AI models just in general are
134. 5:08going up massively as well with things
135. 5:11like ComfyUI. And as much as I love
136. 5:13ComfyUI, that's probably not a mass
137. 5:15product yet. Already, it's well over
138. 5:18double this year compared to last year
139. 5:20as well. And the reasons driving this is
140. 5:22that we're just seeing the whole open
141. 5:25weight system catch up faster and faster
142. 5:28to the proprietary models. Now, I'm not
143. 5:30saying that the open weight models are
144. 5:32better than the proprietary models. I
145. 5:34think this week's releases show that
146. 5:36yes, the proprietary models still ahead
147. 5:38in many ways. But look at things like
148. 5:40this. It's only a few months ago
149. 5:43that we had something like Claude Opus
150. 5:45Max being released in April 2026 that
151. 5:48was reaching a score of 55 on the
152. 5:50artificial analysis intelligence index.
153. 5:53Quen 3.8 flash next has already
154. 5:55surpassed that, and it came out in
155. 5:57August this year. If we look at video
156. 6:00models, Gemini Omni flash, which came
157. 6:03out just in May around at the time of
158. 6:05Google IO, again looking at artificial
159. 6:07intelligence scores for video models,
160. 6:10the original version of it has been
161. 6:12surpassed by Mini Max H3 in August this
162. 6:17year. Now, yes, these are not models
163. 6:19that are going to fit on your phone or
164. 6:20anything. They require a serious amount
165. 6:22to compute to run them locally, but they
166. 6:25are able to be run locally. So, that
167. 6:28brings us to what Invidia launched this
168. 6:30week. Now, they've launched this new
169. 6:32kind of router. So, this is called
170. 6:34Invidia PAIR, and PAIR stands for
171. 6:38personal AI router here. And the whole
172. 6:40idea here is that Invidia is kind of
173. 6:43betting that in the future people are
174. 6:46going to have multiple machines in their
175. 6:48houses that are GPU enabled or that are
176. 6:50going to be accelerators. And you've got
177. 6:52to remember that this is coming just
178. 6:54before next month's launch of the RTX
179. 6:56Spark laptops. And there are already a
180. 6:59number of hardware makers that have
181. 7:00announced that they're going to be
182. 7:02bringing out their own RTX Spark. Now,
183. 7:05those are Windows systems with RTX built
184. 7:08into them. And you could imagine in the
185. 7:10future, as people start to upgrade their
186. 7:12laptops, their computers over the next 5
187. 7:15years, many of these machines are going
188. 7:17to have some form of accelerator in
189. 7:20there, whether it's a GPU, whether it's
190. 7:23an NPU, whatever people are going to
191. 7:25call it. It's going to be a chip or part
192. 7:27of a chipset that is built to process AI
193. 7:30models locally. And what Invidia PAIR
194. 7:33here is trying to get ahead of is this
195. 7:35whole idea of if you've got multiple
196. 7:38machines in your house, how do you
197. 7:40actually get the best use out of them
198. 7:43for agents like Hermes, like Open Claw?
199. 7:46So, this is fundamentally a router
200. 7:48that's quite different than something
201. 7:50like SwitchYard, which we looked at last
202. 7:52month. SwitchYard was all about picking
203. 7:55the right model across an API, whether
204. 7:58that was cloud-hosted or self-hosted.
205. 8:01And it would pick that model based on a
206. 8:03variety of different conditions or
207. 8:05models to actually go through this. So,
208. 8:07you could sort of think of SwitchYard as
209. 8:09being the quality and cost decisions for
210. 8:13things that may leave your network.
211. 8:15Pair, on the other hand, is for hardware
212. 8:17that you control. And the problem that
213. 8:19this is going to solve here is that as
214. 8:22we've got a lot more agentic software,
215. 8:23things like Hermes and Open Claw, we
216. 8:26know that one of the key things that
217. 8:27makes them so good is that they have a
218. 8:30lead agent that can delegate, whether
219. 8:32that's sort of research jobs or coding
220. 8:34jobs or other sort of tasks, to parallel
221. 8:38sub-agents. And those sub-agents can
222. 8:40then go off and do their bit and come
223. 8:42back and deliver their results to the
224. 8:45orchestrator. Now, if you're trying to
225. 8:47run that fully locally, the challenge is
226. 8:50often going to be is those sub-agents
227. 8:52are not actually working in parallel at
228. 8:54all. They get into a queue waiting for
229. 8:57the GPU to be available. So, where Pair
230. 8:59comes in is it allows you to basically
231. 9:02serve other models on multiple machines
232. 9:05on your network and have those
233. 9:06sub-agents be spread out across your
234. 9:09entire network and not just queued up to
235. 9:11work on the one machine. So, in some
236. 9:13ways you can think of Pair as like your
237. 9:16sort of virtual inference traffic cop.
238. 9:18It sees that, "Okay, machine number one
239. 9:20is busy, so it then delegates these
240. 9:22calls over to machine number two." And
241. 9:24the cool thing here is that your agent
242. 9:26never really needs to know this. So,
243. 9:29your agent thinks that it's just calling
244. 9:31a llama or just calling LM Studio, and
245. 9:34it makes the calls just like they would
246. 9:36before, but PA then sits in the middle
247. 9:38directing the traffic over to your
248. 9:40machine number one, machine number two,
249. 9:42etc. as you go through. So, as people
250. 9:45start to have multiple machines that can
251. 9:47do this, and Nvidia points out that a
252. 9:49large percentage of households in the US
253. 9:52already have two or more machines in
254. 9:54their house, probably not with
255. 9:57state-of-the-art GPUs, but the idea here
256. 9:59is that you've got something which
257. 10:01allows you to take advantage of your
258. 10:04gaming machine with perhaps an Nvidia
259. 10:06GPU in there, of a laptop with an RTX
260. 10:09Spark in there, perhaps a bigger
261. 10:10workstation that you've got in an
262. 10:12office, that kind of thing. And of
263. 10:14course, this clearly is early days. It's
264. 10:16not something that everyone is going to
265. 10:17have lots of these machines lying
266. 10:19around, but it does show that Nvidia's
267. 10:21thinking about where local AI is
268. 10:23actually going to go. Now, one of the
269. 10:24things that's really important to stress
270. 10:26here is that this is not VRAM pooling,
271. 10:29right? It's not building a cluster out
272. 10:31of your multiple machines to make one
273. 10:34bigger, stronger machine with more VRAM
274. 10:37here. Each request is running on a
275. 10:39single machine and just that one
276. 10:41machine. More nodes on your network are
277. 10:43going to give you more parallel
278. 10:45requests, not faster performance for a
279. 10:48single prompt, etc. And it's not going
280. 10:50to allow you to then load huge models or
281. 10:52something. Now, for me, one of the
282. 10:54biggest things that makes all of this
283. 10:55interesting is that this is not just
284. 10:58software that Nvidia has launched. This
285. 11:01is an open source Apache 2 licensed
286. 11:05project. Anyone can fork this and can
287. 11:08change it and can customize it in any
288. 11:10way they want. Currently, this is built
289. 11:13just for your own personal network, but
290. 11:15you could imagine if you wanted to give
291. 11:17this to one of the latest coding models,
292. 11:19you could probably get it to work with
293. 11:21networks in the cloud. And I do wonder,
294. 11:24eventually, do we see something like
295. 11:26this and SwitchYard merge together
296. 11:29rather than be separate pieces of
297. 11:30software? So, it's all up on GitHub. It
298. 11:33is still literally at a 0.1 release
299. 11:36here. This is very early days for where
300. 11:38this is going. But, you can see already
301. 11:40out of the box they're supporting
302. 11:42Windows, Linux, and Mac. And yes, if
303. 11:44you're running an Ollama on a Mac
304. 11:47machine, you can actually add that to
305. 11:49pair. It's not just for Nvidia GPUs
306. 11:51here. And currently, it's only
307. 11:53supporting Ollama and LM Studio. But,
308. 11:56I've been on calls where Nvidia
309. 11:57employees who've been working on this
310. 11:58have said openly that they're open to
311. 12:02pull requests. And sure enough, you can
312. 12:03see that there are already pull requests
313. 12:05in there to support things like the
314. 12:08Anthropic style of API, rather than just
315. 12:10the OpenAI endpoint. People are already
316. 12:13submitting pull requests for supporting
317. 12:15llama.cpp. And it's going to be
318. 12:17interesting to see what does Nvidia
319. 12:19actually accept, which pull requests do
320. 12:21they actually accept, and which do they
321. 12:23not like, etc. But, the cool thing is
322. 12:25because it's Apache 2.0, people are
323. 12:28going to be able to fork this. If Nvidia
324. 12:30ends up blocking these kinds of things
325. 12:32too much, the community is just going to
326. 12:34fork it and create their own version
327. 12:36going forward. And I think Nvidia is
328. 12:38very conscious of that just by the fact
329. 12:40that they've made it Apache 2 licensed
330. 12:43in the first place. Okay, so how does
331. 12:45all of this work? Well, pair actually
332. 12:47just proxy sort of takes over the ports
333. 12:50for Ollama and LM Studio. It just
334. 12:53changes the ports for those inference
335. 12:55engines, so that it can sit on those
336. 12:57ports and basically direct the traffic.
337. 13:00It is then connected over your network
338. 13:02directing where the requests go to which
339. 13:05machine running which engine, etc. So,
340. 13:07the cool thing here is all your existing
341. 13:09apps keep the same ports, and you don't
342. 13:11need to make any changes in those,
343. 13:13whether that's Hermes or another coding
344. 13:15agent, etc. But, pair will then redirect
345. 13:18that traffic to the right place. So, the
346. 13:20setup for this is pretty simple. You can
347. 13:22basically just install it. Here you can
348. 13:24see I've installed the version for the
349. 13:26Mac. And straight away it's picking up
350. 13:28that I've got a Llama installed. I
351. 13:30actually have LM Studio installed as
352. 13:32well. Not sure why it's not picking that
353. 13:34one up in there. So, I think technically
354. 13:36they only support Apple M4 and above.
355. 13:40So, I'm using an Apple M2 Pro, an older
356. 13:43Mac mini in here. But, you can see that
357. 13:45okay, with a Llama I can see like the
358. 13:47VRAM that I've got, uh how much memory
359. 13:49I've got available, etc. And then I can
360. 13:52actually come in here and look at a
361. 13:54Llama. And I don't actually use a Llama
362. 13:56that much nowadays. So, it's got some
363. 13:58old models in there still. Yeah, some of
364. 14:00the embedding models, some of the other
365. 14:02ones that I was using for research and
366. 14:03stuff like that. So, just looking at
367. 14:05their video demo here, you can sort of
368. 14:07see that okay, once this is set up, if
369. 14:10something pings it, it can allocate
370. 14:13different models to different machines.
371. 14:16So, you can see in the demo here that
372. 14:18they've got basically a Qwen model
373. 14:20running on an RTX 5090 on the DGX Spock
374. 14:25there. It looks like they've got an
375. 14:27another version of the Qwen 3.6 mixture
376. 14:30of experts model going on there. And
377. 14:32this is able to basically ping these,
378. 14:35and obviously you need to have them
379. 14:37loaded to and ready to go in here. It's
380. 14:40pretty cool to see that like they're
381. 14:41showing that this is not only working
382. 14:43with the RTX 5090 in here, but also with
383. 14:46a Radeon graphics and stuff like that in
384. 14:48there as well. That said for all of
385. 14:51this, it's still early days. I think
386. 14:53we're going to see, you know, a lot of
387. 14:55changes with this. I I think Nvidia is
388. 14:58also looking to the community to see
389. 15:00what changes people want to make, and
390. 15:03what changes people are interested to
391. 15:05actually have here. Right. So, to finish
392. 15:07up, this is definitely worth checking
393. 15:09out if you've got multiple machines with
394. 15:13GPUs in them. It could be useful for a
395. 15:15lot of different things. In the past,
396. 15:18I've tended to use Tailscale for this,
397. 15:21and I've had special models for the sub
398. 15:24agents on particular nodes that were
399. 15:27running. So, for example, on the DGX
400. 15:29Spark, I'd have multiple DGX Sparks
401. 15:32running smaller models, and then
402. 15:34something like the RTX Pro 6000 running
403. 15:36the Qwen 27B model or something like
404. 15:40that. I don't think the 0.1 version of
405. 15:42this is necessarily going to change
406. 15:44hugely how I do things, but I can see
407. 15:48that this is a project that's going to
408. 15:49be really interesting because as people
409. 15:52start to integrate things perhaps like
410. 15:54SwitchYard to it or other routers in
411. 15:56here, then for sure the whole thing gets
412. 15:58a lot more interesting to be able to run
413. 16:01with your agents. The biggest downside I
414. 16:04see of this currently is just the speed.
415. 16:07So, while I'm very lucky to have a lot
416. 16:09of accelerators that I can use, etc.,
417. 16:11the challenge always becomes about speed
418. 16:15of like how do you get the tokens to be
419. 16:17faster, how do you sort of improve that.
420. 16:20Now, since model,
421. 16:24I've got that up to around about 300
422. 16:26tokens per second now. There's versions
423. 16:29out there that are doing faster than
424. 16:30that. If I'm doing things in batches, I
425. 16:33can average around about 380 tokens per
426. 16:36second out of that, and that's certainly
427. 16:38been a workhorse there. And I do wonder
428. 16:40as I delegate this using paired to other
429. 16:43machines for doing inference, is the
430. 16:45speed hit just going to slow it down too
431. 16:48much at the moment. But anyway, go over
432. 16:50to the repo, give it a star. It's
433. 16:52definitely worth checking out and
434. 16:54following this going forward, and feel
435. 16:56free to chime in on the issues and pull
436. 16:59requests for this, and let's see where
437. 17:00this actually goes in the future. I'm
438. 17:03also interested in testing out other
439. 17:05routers. If you're using anything else
440. 17:06that you think is really good, please
441. 17:08let me know in the comments cuz I'm
442. 17:09definitely looking for what is the best
443. 17:12out there with this kind of thing.
444. 17:14Anyway, if you found the video useful,
445. 17:16please click like and subscribe, and I
446. 17:17will talk to you in the next video. Bye
447. 17:19for now.