HP ZGX Fury - 3,000 Tokens/Sec Locally — Transcript

Full transcript

1. 0:00Okay, so you probably already know the
2. 0:01DGX Spock. I've made videos about them.
3. 0:04Many people online have been talking
4. 0:06about them and made videos. Today, I
5. 0:08want to introduce you to its bigger
6. 0:10brother, the DGX Station, and
7. 0:13specifically the HP ZGX Fury DGX
8. 0:17Station, which actually has a number of
9. 0:19things that set it apart from other DGX
10. 0:21stations that are out there on the
11. 0:22market. If you haven't seen one of these
12. 0:24before, you can kind of think of the DGX
13. 0:27Spock or the ZGX Nano as being a machine
14. 0:30that sits quite neatly on top of your
15. 0:32desk by itself and allows you to run a
16. 0:35small model. And they can be fantastic
17. 0:37for different use cases. The ZGX Fury,
18. 0:40on the other hand, is the machine that
19. 0:42sits next to your desk. Not only can it
20. 0:45power the models that you want to use by
21. 0:46yourself, but it has the bandwidth to
22. 0:48power models across a whole team of
23. 0:51people using it. Now, HP has been kind
24. 0:53enough to sponsor this video and to go
25. 0:55to quite a lot of effort to get me a
26. 0:57loaner unit halfway across the world so
27. 1:00that I could run a bunch of tests on
28. 1:01this and see how one of these actually
29. 1:03stacks up for real-world use. All right,
30. 1:05so what actually is this? Officially,
31. 1:07it's called the HP ZGX Fury G1 NAI
32. 1:12Station. And this is HP's own unique
33. 1:14take on Nvidia's DGX Station. Inside,
34. 1:18you've got Nvidia's Blackwell GB3 chip
35. 1:21paired with a 72-core Grace CPU. And
36. 1:25together, that gets you 20 petaflops of
37. 1:27AI compute at FP4. And it's not just the
38. 1:30compute that's impressive here. There's
39. 1:32also the memory. So, in total, you've
40. 1:35got 748 GB of RAM shared between the CPU
41. 1:39and GPU. And I'll go into the unique
42. 1:41architecture of that in a bit. You've
43. 1:43also got two 400 GB per second network
44. 1:46ports. So, if one of these isn't enough,
45. 1:48you can actually link multiple of these
46. 1:50together just like you can with the DGX
47. 1:52Spock. Now, the whole thing is liquid
48. 1:54cooled, so you can run it as a tower
49. 1:56next to your desk, but it's also
50. 1:58actually built so that you can mount it
51. 1:59in a server rack and is actually fully
52. 2:01optimized for airflow in that rack as
53. 2:04well, which is one of a number of things
54. 2:06that sets this apart from the other DGX
55. 2:08stations out there. Now, let's take a
56. 2:10look at that memory. With the way most
57. 2:12people are running local models these
58. 2:14days, you kind of got two choices. You
59. 2:16can go with unified memory like the DGX
60. 2:18Spark does and get quite a lot of it, or
61. 2:20you can go with the route of having an
62. 2:22individual GPU card like an RTX 6000,
63. 2:25which is going to get you faster
64. 2:26results, but then you're generally
65. 2:28capped at like 96 GB per card, which
66. 2:31really makes it difficult to run a lot
67. 2:32of the models that are out there. The
68. 2:34Fury here doesn't make you choose. You
69. 2:37get 748 GB to fit the really big open
70. 2:40models, and you got 252 GB of that
71. 2:43memory being HBM, basically
72. 2:45high-bandwidth memory. And that
73. 2:47bandwidth is pretty amazing. You're
74. 2:50looking at 7.1 TB per second for moving
75. 2:53the weights around. The fastest
76. 2:55professional workstation card that
77. 2:56Nvidia makes is the RTX Pro 6000, and
78. 3:00you're only getting 1.8 TB on that. So,
79. 3:03this box is really giving you the kind
80. 3:05of setup that you would normally only
81. 3:07find in a multi-GPU server. You've got
82. 3:10both the high-speed, high-bandwidth
83. 3:12memory, but you've also got a bunch of
84. 3:14unified memory in there as well. Now,
85. 3:16one of the things that's really
86. 3:17impressive about the way that HP has
87. 3:19approached their own ZGX Fury is that
88. 3:22they wanted to make sure that they're
89. 3:23not just doing what everyone else is
90. 3:25doing in taking the Nvidia reference
91. 3:28architecture out of the box and not
92. 3:30adding anything to it. The ZGX Fury is
93. 3:32actually the only DGX station that's
94. 3:35been allowed to customize what they
95. 3:37actually ship out on the box and then
96. 3:39include some of their own unique
97. 3:40software. So, there's basically a tool
98. 3:42called HP Z Runtime, or ZRT for short.
99. 3:46It's a command-line tool that takes a
100. 3:48lot of the pain out of getting models
101. 3:50running on the box and then optimizing
102. 3:52them for serving. You run ZRT setup,
103. 3:54then ZRT serve, and then name the model
104. 3:57that you want. And then they've even got
105. 3:59things like ZRT chat if you want to
106. 4:01basically just test it out quickly. And
107. 4:03this will go out there and pull down a
108. 4:04model from hugging face automatically.
109. 4:07So you're not having to hunt around for
110. 4:08the weights and the config files for
111. 4:11actually setting it up with this
112. 4:12machine. So the ZRT toolkit basically
113. 4:15lets you do a lot of different useful
114. 4:17tasks that everyone's going to want to
115. 4:19do with this machine. If you need to
116. 4:21check the status and see what's on the
117. 4:23GPU, you can see that really easily. And
118. 4:25if you want to download a model from
119. 4:27hugging face and serve it, that's
120. 4:29literally just the ZRT serve command
121. 4:31here. And that's going to take care of
122. 4:32everything from downloading the weights,
123. 4:34configuring it for VLLM, and then
124. 4:36loading the model in RAM, and giving you
125. 4:38an endpoint that you can share with your
126. 4:40team straight away. Once the model is
127. 4:42running, you can use ZRT then to monitor
128. 4:44the usage, but you can also use it then
129. 4:46to run a benchmark on a dataset that you
130. 4:48choose to get all the relevant stats out
131. 4:51of things like time to first token, the
132. 4:53number of tokens per second, or the
133. 4:55speed is going to be like when you've
134. 4:57got lots of concurrent users actually
135. 4:59using the system. And it's even got some
136. 5:01really useful helper functions in there
137. 5:03of just starting up an interactive chat
138. 5:05session with a model, so that you can
139. 5:07actually see whether it's configured
140. 5:08right, whether it's getting the
141. 5:10responses back that you think it should
142. 5:12be. And in many ways the ZRT toolkit
143. 5:14really just emphasizes HP's thinking of
144. 5:18taking the reference architecture from
145. 5:19Nvidia, but making it easy to use for
146. 5:22people in enterprise situations. And if
147. 5:24you're actually a developer that's using
148. 5:26VS code, etc., they've also got an HPZ
149. 5:29tool kit, which is an extension that
150. 5:31lets you work right out of the box from
151. 5:33your code editor. Another thing that
152. 5:35sets the Fury apart from the other DGX
153. 5:37stations out there is that while this
154. 5:39ships with Ubuntu out of the box, just
155. 5:41like the DGX Spark and most other AI
156. 5:43machines, but HP has actually gone to
157. 5:45quite a lot of effort to make the ZGX
158. 5:47Fury the first GB300 station certified
159. 5:51for Red Hat Enterprise Linux. And what
160. 5:53most people don't know is that when you
161. 5:55actually look at what people are running
162. 5:56in data centers, what they're actually
163. 5:58training on in their clusters, it
164. 6:01usually turns out to be Red Hat
165. 6:03Enterprise Linux. And the whole thing of
166. 6:05HP customizing the machine is not
167. 6:08actually stopping with the OS or their
168. 6:10ZRT software. They've also teamed up
169. 6:12with Infinity AI to actually ship a
170. 6:14specific quantization of the Nemotron
171. 6:16super model that is customized and
172. 6:18optimized specifically for this box. So,
173. 6:21what does that actually get you? Well,
174. 6:23Infinity AI used their auto research
175. 6:25system to tune in videos Nemotron 3
176. 6:27super specifically for the GB300 ship
177. 6:31inside the Fury. Same hardware, nothing
178. 6:33swapped out, but they've doubled the
179. 6:35number of people the box can actually
180. 6:37serve at once and doubled the overall
181. 6:39token throughput without slowing down
182. 6:41anyone's individual responses. And
183. 6:44that's actually what matters on a
184. 6:45machine like this. The Fury isn't just
185. 6:47for you or for one particular person.
186. 6:50It's for your whole team. Twice the
187. 6:52concurrent users on the same box is
188. 6:54basically like getting a second machine
189. 6:56for free. If you look at it as it's
190. 6:58going through this, I'm testing it here
191. 7:01with lots of different streams going at
192. 7:04the same time. You can see in aggregate
193. 7:06we're getting over 3,000 tokens per
194. 7:09second coming out here. And amazingly,
195. 7:11you can see in the rag test here, when
196. 7:14I've got 16 concurrent users at the same
197. 7:16time, we're getting well over 1,300
198. 7:18tokens a second and the fans on the
199. 7:20machine haven't even kicked in yet. And
200. 7:22don't forget here, this is not a small
201. 7:25model. This is the Qwen 3.8 flash next
202. 7:28model. Even here simulating 64 users at
203. 7:32the same time, the machine's able to
204. 7:34basically take that and keep going.
205. 7:37You can see that even with 64 concurrent
206. 7:39users here, we're still getting good
207. 7:41token speeds out of the machine.
208. 7:44You can see when I come back and look
209. 7:45through this. In just a matter of
210. 7:47minutes, this has outputted over 500,000
211. 7:50tokens. And on other tests, when we
212. 7:53actually come through and look at these,
213. 7:54we can see we're still getting up to
214. 7:57around high 80s into 90 tokens per
215. 8:00second with 64 concurrent users once it
216. 8:03starts generating. This totally just
217. 8:05underscores the power of this machine in
218. 8:08allowing you to run really big workloads
219. 8:10with high concurrency. So, my current
220. 8:12plan is in the next video, I want to
221. 8:14actually see how many models can I run
222. 8:16at once on this machine and how I can
223. 8:19actually combine both LLMs and
224. 8:22multimodal models for image generation
225. 8:24and video generation. And that actually
226. 8:26brings me to who is this for? Up till
227. 8:28now, the model for local AI has
228. 8:30generally been one box per person.
229. 8:33People would need to get their own Spark
230. 8:34or DGX Nano, and everyone runs their own
231. 8:37little models on their desk. The DGX
232. 8:40Station and the Fury specifically flips
233. 8:42that. It's not built to be your personal
234. 8:44machine. It's built to be your team's
235. 8:46machine. One box that a whole group of
236. 8:49people can be hitting at the same time.
237. 8:51And that's exactly what you can do here.
238. 8:53You can have multiple people running
239. 8:55inference against a big model, a bunch
240. 8:57of AI agents running in the background
241. 8:59on a different model, someone else doing
242. 9:01some fine-tuning, and someone else
243. 9:03experimenting with something new all on
244. 9:05the same machine at the same time. And
245. 9:07don't forget, none of this comes with a
246. 9:09per-token API bill. One of the big
247. 9:11things that has changed this year is how
248. 9:13teams are actually using LLMs and AI.
249. 9:17People are no longer just chatting with
250. 9:18a model. They're actually using fully
251. 9:21agentic systems. And that's one of the
252. 9:23things that the DGX Fury shines at.
253. 9:26Because the machine is so powerful and
254. 9:28you've got so much memory in it, you can
255. 9:30actually run multiple LLMs or have
256. 9:33things like embedding models, video
257. 9:35models, image generation models all
258. 9:37running at the same time. And then
259. 9:39because you've got the great CPU with so
260. 9:42many cores, it can be running things
261. 9:44like lots of tools, sandbox
262. 9:46environments, and jobs that require
263. 9:48standard processing with a variety of
264. 9:50different libraries and use cases.
265. 9:52Having a main orchestrator model along
266. 9:54with multiple models that you can use
267. 9:56for sub-agents, and at the same time
268. 9:58being able to run a bunch of tools in
269. 10:00sandbox environments is just not
270. 10:03something that other machines can do
271. 10:05anywhere near at this scale. And HP has
272. 10:08leaned into this with the Fury. It's
273. 10:10rack-mountable, but it's also designed
274. 10:12to just sit in an office. So, instead of
275. 10:14buying a card for everyone on the team,
276. 10:16you buy one Fury, put it in the office
277. 10:19or on a rack, and everybody shares it.
278. 10:21And I'm going to say HP's own tagline
279. 10:23for this sums it up pretty well.
280. 10:25Sharable on-prem AI, scale without token
281. 10:28cost. And the good thing here is because
282. 10:31it's on-prem, your data never leaves the
283. 10:33building. If you're in healthcare,
284. 10:35government, or manufacturing, it's not a
285. 10:37nice to have these days, that's becoming
286. 10:39a requirement. More and more people
287. 10:41don't want to be giving their data to
288. 10:43the big proprietary model companies. And
289. 10:45it's not just about where the data
290. 10:46lives, it's about control over the whole
291. 10:48stack. Take a cybersecurity team, they
292. 10:51need to analyze malware, write exploits,
293. 10:53they need to be able to write proof of
294. 10:55concepts, and they need to be able to
295. 10:56red team on their own systems. Ask one
296. 10:59of the big closed models to help with
297. 11:00that, and there's a good chance that it
298. 11:02just refuses. Or in many ways, what's
299. 11:05worse is that it perhaps worked last
300. 11:07week, and then the provider quietly
301. 11:08changed the guardrails, and now it
302. 11:10doesn't. That team can't be at the mercy
303. 11:12of someone else's model policy. With a
304. 11:15box like this, they pick the model, they
305. 11:17set the guardrails, and nothing changes
306. 11:20unless they decide to change it. The DGX
307. 11:23Fury is basically giving you a data
308. 11:25center under your desk. So, back when
309. 11:27Nvidia announced the DGX Station
310. 11:29alongside the DGX Spark, the thing that
311. 11:31jumped out at me was the chip. The GB300
312. 11:34isn't some cut-down desktop part.
313. 11:38Ultra, the same architecture Nvidia puts
314. 11:40in its GB300 NVL72 racks in a data
315. 11:44center. Until now, if you wanted that
316. 11:46kind of compute, you were renting it
317. 11:48from a cloud provider or putting in a
318. 11:50request with an IT department for a
319. 11:53space in a server room. Now, it can just
320. 11:55sit right next to your desk. And it's
321. 11:57not just the chip, you've got the HBM
322. 11:59memory, the 400 gigabits per second
323. 12:01networking, the liquid cooling, and the
324. 12:04Red Hat Enterprise Linux. And again,
325. 12:06normally all of those things are things
326. 12:08that you would find in a data center.
327. 12:10And one of the reasons that that matters
328. 12:12is because whatever you build on this
329. 12:14box uses the same stack as the big
330. 12:17systems. If you prototype an agent or
331. 12:19fine-tune a model on the DGX Fury, when
332. 12:22it comes time to scale up to a data
333. 12:24center or cloud, you're not having to
334. 12:27rewrite everything. If one box isn't
335. 12:29enough, you can actually link two
336. 12:30together over those 400 gigabit ports.
337. 12:33All of these things take this from
338. 12:34beyond just being a workstation to
339. 12:37actually being more like a small data
340. 12:38center. So, overall, just to finish up,
341. 12:41the DGX Fury clearly is its own product.
342. 12:44While DGX Station is NVIDIA's reference
343. 12:47design, and the Fury is using the same
344. 12:49GB300 chipset, in the end here, HP's
345. 12:52taken all of that and made their own
346. 12:54product. And it's because of things like
347. 12:56the software that they've added, the
348. 12:58support through things like Red Hat
349. 13:00Enterprise Linux, and even just the way
350. 13:03that they've built the hardware to make
351. 13:05it rack-mountable and something that you
352. 13:06can use both in an office, but also in
353. 13:09an on-prem data center. Well, I know for
354. 13:11most individuals, this is totally out of
355. 13:14our budget, but for enterprise use or a
356. 13:16team looking to build a really solid
357. 13:19on-prem AI solution, the DGX Fury really
358. 13:23is delivering state-of-the-art on-prem
359. 13:25local AI. Of course, if you want to find
360. 13:28out more about the DGX Fury and also
361. 13:31about the DGX Nano, I've put links in
362. 13:34the description where you can check
363. 13:35those out and find some more detailed
364. 13:36information.
365. 13:38Anyway, over the next few weeks, I plan
366. 13:40to do a number of videos while I've got
367. 13:42this machine on loan. Let me know in the
368. 13:44comments what you would like to see me
369. 13:45do with this machine. What kind of tests
370. 13:48and models would you like to see me run
371. 13:50on this? Anyway, as always, if you
372. 13:52enjoyed the video, please click like and
373. 13:53subscribe, and I will talk to you in the
374. 13:55next video. Bye for now.