YouTube2Text

Every Ways to Get 128GB VRAM for Local AI at Full Context — Transcript

by Kai · 3,252 words · 464 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Okay, so 6 months ago I finally got fed
  2. 0:03up. I was running a local model at
  3. 0:05something like four tokens per second.
  4. 0:07My laptop fan sounded like it was
  5. 0:09auditioning for a jet engine role. And
  6. 0:11meanwhile, I was watching people on
  7. 0:13Reddit post screenshots of 60 or 70
  8. 0:16tokens a second on their own machines
  9. 0:17while I sat there thinking,
  10. 0:19"What am I even doing wrong?" So, I did
  11. 0:21what any reasonable person does, which
  12. 0:23is open a browser and type in, "How much
  13. 0:25VRAM do I actually need for local AI?"
  14. 0:28And that's how I fell into a rabbit hole
  15. 0:30that I'm still trying to climb out of.
  16. 0:33Everyone gives the same answer, and it's
  17. 0:35128 GB. That's the golden threshold
  18. 0:38where you get to run the big models and
  19. 0:40never pay an API bill again, and it
  20. 0:42sounded a lot like buying a dream. And
  21. 0:45then I started pricing it out, and I
  22. 0:47want you to feel what I felt here
  23. 0:49because the cheapest legit option I
  24. 0:50found was $3,000, and the most expensive
  25. 0:53was almost 7,000, and somewhere in
  26. 0:55between those two numbers I started
  27. 0:57questioning every decision I've ever
  28. 0:59made. But what nobody told me is that
  29. 1:01not all of these options are actually
  30. 1:03buying you the same thing. Some of them
  31. 1:05are really just buying you a fourth bit,
  32. 1:07and that will make a lot more sense once
  33. 1:09we've priced everything. So, let's go
  34. 1:11through every route to 128 gigs, and I
  35. 1:14mean every route. We'll look at the mini
  36. 1:16PC, the Mac Studio, and the Nvidia box,
  37. 1:19then four used server cards, and then
  38. 1:22the option that made me feel stupid for
  39. 1:23not trying it first. Now, before we
  40. 1:26price anything, we need to understand
  41. 1:28why 128 GB became the number in the
  42. 1:31first place. And it comes down to one
  43. 1:33model, which is Qwen 3.8 Flash Next.
  44. 1:37If you spend any time on the local Llama
  45. 1:38subreddit, and I basically live there at
  46. 1:40this point, this model is everywhere,
  47. 1:43and it's become the default answer to,
  48. 1:45"What do I run on a big memory machine?"
  49. 1:47One person posted that they haven't
  50. 1:49called a cloud API since the day it came
  51. 1:51out, which gives you a sense of the
  52. 1:52energy around it. And the reason 128
  53. 1:55matters is boring but real. The four-bit
  54. 1:58builds of this model land between 94 and
  55. 2:01124 GB depending on the format, which
  56. 2:04fits in 128 and in nothing smaller. So,
  57. 2:07the logic writes itself. Where you get
  58. 2:09128 gigs, load the 4-bit file, and
  59. 2:12you're done. But, as always, it's not
  60. 2:14actually that simple. Now, this is where
  61. 2:16it got interesting for me, and also a
  62. 2:18little embarrassing because I'd been
  63. 2:20thinking about it all wrong. I assumed,
  64. 2:22pretty logically I thought, that if a
  65. 2:24model has 125 billion parameters, you
  66. 2:26need all 125 billion sitting in fast
  67. 2:29memory to run it fast. And FlashNext
  68. 2:31just doesn't work that way. The model
  69. 2:33does have 125 billion parameters, and it
  70. 2:37also has a 51 billion parameter lookup
  71. 2:39table for common word sequences, which
  72. 2:41sounds enormous.
  73. 2:43But, for each word it writes, it only
  74. 2:45actually reads about 6 billion of them.
  75. 2:47And at 3 bits per weight, that's a
  76. 2:48couple of gigabytes per word.
  77. 2:50Everything else just needs to be
  78. 2:51somewhere your chip can reach when a
  79. 2:53word calls for it. And when we look at
  80. 2:55the model card, it actually says the
  81. 2:57lookup table is more amenable to
  82. 2:59offloading than the rest of the model.
  83. 3:01So, there's an engine called Strata that
  84. 3:03keeps the active part on your GPU and
  85. 3:05the rest in normal system RAM.
  86. 3:07One person ran it on an RTX 3090, which
  87. 3:10is a card with only 24 GB of VRAM, and
  88. 3:13used 23 GB of VRAM plus 47 GB of RAM in
  89. 3:17total. In other words, nothing on that
  90. 3:19machine was 128 gigs.
  91. 3:22This idea has a name, which is expert
  92. 3:24offloading.
  93. 3:25The way I picture it, the 128 gig box is
  94. 3:28a desk big enough to hold every book you
  95. 3:30own open at once, while the PC setup is
  96. 3:33a smaller desk next to a shelf.
  97. 3:35And if each word only opens a few pages
  98. 3:37at a time, the shelf is fine and you
  99. 3:39don't need the big desk. That was the
  100. 3:41realization that shifted everything for
  101. 3:43me, because it means capacity and speed
  102. 3:45are no longer the same purchase. Now,
  103. 3:48let's say we still want the big pool,
  104. 3:50and we're buying 128 gigs anyway.
  105. 3:52There's a detail here that will mildly
  106. 3:54ruin your week.
  107. 3:56On Windows, AMD's Ryzen AI Max mini PCs,
  108. 4:00which are one of the main routes to 128
  109. 4:02gigs, let the GPU side claim at most 96
  110. 4:06GB by default. So, the box was sold as
  111. 4:09128, and the other 32 just sit there
  112. 4:12with the CPU. On Linux, it can go past
  113. 4:15100, and some owners report getting
  114. 4:17close to 120. And a 128 gig Mac does
  115. 4:20much the same thing because macOS caps
  116. 4:23what the GPU can lock at around 96 gigs.
  117. 4:26The developer of one of the main Mac
  118. 4:28engines actually tells you to go into
  119. 4:30the terminal and raise that limit by
  120. 4:32hand before you run anything at full
  121. 4:34context. And when we hold those 96 gigs
  122. 4:37against the 4-bit builds, which start at
  123. 4:3994 gigs before any context window at
  124. 4:42all, it's pretty clear where this is
  125. 4:44going. Even the $6,950
  126. 4:47Nvidia box, the DGX Spark, only sees
  127. 4:50about 122 usable gigs. And since
  128. 4:53Nvidia's own 4-bit file is 124 GB, the
  129. 4:57most popular recipe for running it
  130. 4:59leaves about 48 gigs on the SSD and
  131. 5:01reads it from there. So, the nearly
  132. 5:03$7,000 machine is also offloading.
  133. 5:06Microsoft at least has the decency to
  134. 5:08mention it because their new laptops say
  135. 5:11in a footnote that the memory the GPU
  136. 5:13can address is less than the total. But,
  137. 5:15they never say how much less. And the
  138. 5:18128 on the box is really a ceiling, but
  139. 5:20the default sitting well under it. Now,
  140. 5:23if we've decided we want to pull anyway,
  141. 5:25we should start with the cheapest way to
  142. 5:27get it. That's AMD's Ryzen AI Max + 395
  143. 5:31chip, which everyone calls Strix Halo.
  144. 5:34It starts at $3,099 in a mini PC from a
  145. 5:37brand called Bosque Game, while
  146. 5:39Framework makes one for $3,449
  147. 5:42that's currently out of stock, and
  148. 5:44Minisforum's version is $3,799.
  149. 5:48And it's the same chip with the same 256
  150. 5:50GB per second of memory bandwidth in all
  151. 5:53of them. But, the engine you run on this
  152. 5:54machine really matters more than the
  153. 5:56brand of the box. Someone ran Flash Next
  154. 5:59on a Strix Halo machine with two
  155. 6:01different inference engines, 160,000
  156. 6:05tokens deep into a conversation. The
  157. 6:07first engine read that prompt at 113
  158. 6:10tokens a second, and the second one, a
  159. 6:12community fork called Gufo, read it at
  160. 6:141,227.
  161. 6:16And that's a 160,000
  162. 6:19token prompt loading in either 23
  163. 6:21minutes or 2 minutes on the same
  164. 6:23hardware with the same model, where the
  165. 6:25only thing that changed was the engine.
  166. 6:27The newer engines go even further.
  167. 6:29Strata posts 47 to 54 tokens per second
  168. 6:32on the 4-bit build, and around 1,300
  169. 6:35tokens per second on prompt reading. And
  170. 6:37Halogen, which is another engine a lot
  171. 6:39of owners swear by, posts similar
  172. 6:41numbers. It's closed source, though, and
  173. 6:43its forums are mostly people asking when
  174. 6:45that changes. And I'm right there with
  175. 6:47them. A bigger sibling is also coming,
  176. 6:49which is the Ryzen AI Max Pro 495 with
  177. 6:53192 GB at about $6,799
  178. 6:57from Framework, and that gives you more
  179. 6:59headroom at a similar speed. But for
  180. 7:01right now, if you put Linux on a Strix
  181. 7:03Halo box and pick the right engine, it's
  182. 7:05the cheapest machine that runs the 4-bit
  183. 7:07build well. Now we get to the two
  184. 7:09expensive options, and they split the
  185. 7:11work in really different ways. The Mac
  186. 7:13Studio with an M5 Max and 128 gigs runs
  187. 7:16$5,099.
  188. 7:18The 40-core GPU config is the only way
  189. 7:21to get 128 gigs, and it also brings 614
  190. 7:25GB per second, which is more than twice
  191. 7:28what Strix Halo has. And going from 64
  192. 7:30to 128 gigs costs $1,600
  193. 7:34on the Mac, which works out to about $25
  194. 7:36a gigabyte. On Flash Next specifically,
  195. 7:40the developer of the MLX serve engine
  196. 7:41reports about 40 tokens per second on
  197. 7:44pros and 75 on code. And the machine is
  198. 7:47quiet and polished with a software story
  199. 7:49on the Mac that's really good. But to be
  200. 7:52fair to the other boxes, nobody has
  201. 7:54published prompt reading speeds for this
  202. 7:56model on this machine yet. And that's a
  203. 7:58real gap because it matters for long
  204. 8:00context agent work. The DGX Spark is the
  205. 8:03CUDA machine at $6,950
  206. 8:06as of early October after Nvidia raised
  207. 8:08the price 74% from launch. It writes at
  208. 8:11about 41 tokens per second median and 27
  209. 8:14on pros, and it has a draft ahead trick
  210. 8:17that helps, because without it, the
  211. 8:18speed drops to 17. Where the spark used
  212. 8:21to shine was prompt reading. On an older
  213. 8:23model, it was five times faster than
  214. 8:25Strix Halo, which one benchmark
  215. 8:27translated into saving 83 seconds every
  216. 8:30time a coding agent re-reads a 30,000
  217. 8:33token context window. And that was a big
  218. 8:35deal.
  219. 8:36Now, on flash next with today's engines,
  220. 8:39the gap is 1,666
  221. 8:42tokens per second against 1,370.
  222. 8:45So, the 83-second advantage is now about
  223. 8:484 seconds, and you're paying a
  224. 8:50significant premium for those 4 seconds.
  225. 8:53And Nvidia just launched a 64 gig spark
  226. 8:55at $4,999,
  227. 8:58so two of them for 128 gigs comes to
  228. 9:01$10,000, and I'm not sure that needs any
  229. 9:04commentary from me. On October 7th,
  230. 9:06Nvidia quietly launched something called
  231. 9:08the RTX spark, which is a completely
  232. 9:11different chip from the DGX spark,
  233. 9:13despite the name. It's in Windows
  234. 9:15laptops and small desktops now.
  235. 9:17Microsoft's Surface Laptop Ultra with
  236. 9:19128 gigs is $5,899.99,
  237. 9:24and the desktop version, the Surface RTX
  238. 9:27spark dev box, is $5,999.99
  239. 9:31with 128 gigs. Microsoft is claiming 2.1
  240. 9:35times faster first word latency and 6.2
  241. 9:39times faster video than an M5 Pro
  242. 9:41MacBook Pro.
  243. 9:43But when we look at the footnotes, those
  244. 9:44numbers were measured on pre-production
  245. 9:4664 GB units, and the one petaflop figure
  246. 9:50uses four-bit math with sparsity turned
  247. 9:52on, which is not how you'd normally
  248. 9:54compare performance. Reported memory
  249. 9:56bandwidth puts it somewhere around 273
  250. 9:59to 307 GB per second, which is next to
  251. 10:02Strix Halo rather than next to the Mac
  252. 10:04Studio. So, my take is to treat it as a
  253. 10:06laptop that holds the four-bit build,
  254. 10:08and to wait for someone outside
  255. 10:10Microsoft to actually time it on this
  256. 10:12model before spending $6,000 on one.
  257. 10:14Now, we get to the chaotic option
  258. 10:16because the top voted 128 gig build on
  259. 10:19Local Llama this quarter came from
  260. 10:20someone who bought four used Radeon Pro
  261. 10:23V620 cards for $1,400
  262. 10:26total. They threw in a used EPYC server
  263. 10:29CPU, 256 gigs of DDR4 RAM for $610, and
  264. 10:34a 1,600 W power supply, and walked out
  265. 10:38with a whole machine for around $3,000.
  266. 10:41On Flash next at four bit, this thing
  267. 10:43writes 60 to 70 tokens per second, which
  268. 10:46on Pros is faster than every other box
  269. 10:48in this video. So, the server route wins
  270. 10:50on raw output speed. But, it also draws
  271. 10:53500 to 600 W while it's writing. At 4
  272. 10:56hours a day and $0.15 per kilowatt-hour,
  273. 10:59that works out to about $120 a year in
  274. 11:02electricity against around $38 a year
  275. 11:05for a Strix Halo box under similar load.
  276. 11:08So, the power bill really is part of the
  277. 11:09price tag, and the deal is gone because
  278. 11:12commenters started reporting that V620s
  279. 11:15climbed to the price of a 32 gig V100
  280. 11:18after that build went viral. New
  281. 11:20alternatives like Intel Arc Pro B70s and
  282. 11:23AMD Radeon AI Pro R9700s
  283. 11:27land around $6,800
  284. 11:29for four.
  285. 11:30Four used Instinct MI50s are $2,300 to
  286. 11:34$2,600 before a host machine, and they
  287. 11:37run on a chip AMD put in software
  288. 11:39maintenance mode years ago.
  289. 11:41And a single RTX Pro 6000 Blackwell with
  290. 11:4496 gigs runs $8,900
  291. 11:47to $13,250
  292. 11:50if you can find one. So, the server
  293. 11:51route made sense at $3,000, but at
  294. 11:54today's prices, it's a different
  295. 11:55calculation. Now, there's one route that
  296. 11:58none of the 128 GB guys price.
  297. 12:01StrataZone Readme lists a test setup
  298. 12:03with an RTX 5070 with 12 GB, a six-core
  299. 12:07Ryzen 5, and 64 GB of RAM. So, nothing
  300. 12:10fancy at all. On the two-bit build, it
  301. 12:12writes 94 tokens per second. On the
  302. 12:153-bit build, it writes 50 to 62. And it
  303. 12:18reads a 32,000 token prompt at 1,750
  304. 12:22tokens per second. And that's the
  305. 12:24writing speed of the $3,000 to $7,000
  306. 12:27boxes with faster reading than any of
  307. 12:29them on a card you might already own.
  308. 12:31Someone running a 3090, which is a card
  309. 12:33from 2020, in an external enclosure
  310. 12:36connected to a laptop with 64 gigs of
  311. 12:39DDR4 posted at 60 to 81 tokens per
  312. 12:42second on the 3-bit build. You might be
  313. 12:44wondering what happens if you just throw
  314. 12:46more RAM at the 4-bit build. One owner
  315. 12:48tried exactly that with an RTX 5090
  316. 12:51laptop GPU paired with 192 GB of RAM.
  317. 12:55The 3.5-bit build ran at 53 to 59 tokens
  318. 12:59per second, but the 4-bit build ran at
  319. 13:0121 to 33, which is about half the speed.
  320. 13:04So, the PC matches the pool at three
  321. 13:06bits and falls behind at four. And
  322. 13:09that's the line every route in this
  323. 13:11video lands on one side of.
  324. 13:13And this is the question nobody has put
  325. 13:15a number on, even though it's the one
  326. 13:17that actually matters. The lab that made
  327. 13:19the 3-bit file says it keeps 99.4%
  328. 13:23of the full model score across three
  329. 13:25benchmark tests. But on one of those
  330. 13:27tests, the full model already gets
  331. 13:29everything right. So, 99.4% of perfect
  332. 13:32is still perfect. Owners don't fully
  333. 13:35agree, either. Where one person called
  334. 13:37the two and three bit builds lobotomized
  335. 13:39and another runs the four bit build
  336. 13:41across three 3090s and says it blows 27B
  337. 13:44out of the water.
  338. 13:45And nobody has published both versions
  339. 13:47side by side on a long agent session,
  340. 13:49which is the actual work you'd buy 128
  341. 13:51gigs for. So, that data gap is real.
  342. 13:54Now, we can at least put a number on the
  343. 13:56price. If you already own a gaming PC
  344. 13:59with a 12 gig card, 64 GB of DDR5 costs
  345. 14:02about $880 at the cheapest listing. And
  346. 14:06that's your 3-bit setup. The cheapest
  347. 14:08machine that runs four bit well is the
  348. 14:10$3,099
  349. 14:11Strix Halo, which means the fourth bit
  350. 14:14costs you about $2,200
  351. 14:16as an upgrade. If you're building from
  352. 14:18scratch, the 5070, the RAM, and about
  353. 14:21$800 for a basic PC comes to around
  354. 14:23$2,400,
  355. 14:25and against the Strix Halo, the fourth
  356. 14:27bit costs $700. And $700 is the one
  357. 14:31number in this whole video that's easy
  358. 14:33to say yes to if you're building new and
  359. 14:35four-bit quality really matters to your
  360. 14:37workflow. Now, I want to be clear about
  361. 14:39the ceiling here because it's lower than
  362. 14:41it looks. Strider runs one model family,
  363. 14:45which is Flash Next and its variants.
  364. 14:47And the speed gains come directly from
  365. 14:49how that specific model is designed with
  366. 14:52a small active weight set and a lookup
  367. 14:54table built offload. So, if we point the
  368. 14:56same PC at a different large model,
  369. 14:59we're back to whatever standard
  370. 15:00llama.cpp does with it. Context is
  371. 15:02tighter, too, because one person running
  372. 15:05a 3.5-bit build on two 3090s with 128 GB
  373. 15:09of RAM got 130,000
  374. 15:12tokens of context and mentioned they
  375. 15:14might go back to Qwen 3.8-27B,
  376. 15:17which gave them 256,000 if things don't
  377. 15:20improve. And the next model might not
  378. 15:21cooperate at all. Qwen calls Flash Next
  379. 15:24a preview of the Qwen 4 architecture,
  380. 15:26and somebody noted the flash models have
  381. 15:28gotten bigger with every version, so the
  382. 15:30PC trick relies on design choices the
  383. 15:32next model might not make. But the pool
  384. 15:35doesn't care about any of that because a
  385. 15:37128 gig box loads whatever the next
  386. 15:39model is and whatever shape it takes as
  387. 15:41long as someone writes an engine for it.
  388. 15:44There are also two kinds of buyers the
  389. 15:45PC route doesn't work for, and I want to
  390. 15:47name them plainly. The first is anyone
  391. 15:50serving multiple people or running a
  392. 15:52swarm of agents at the same time, where
  393. 15:54the math changes completely. Strider's
  394. 15:56notes show four R9700 cards serving
  395. 15:59eight clients at 165 tokens per second
  396. 16:02total, and the DGX Spark holds 26 tokens
  397. 16:05per second per stream with four running,
  398. 16:08while a gaming PC with one 12-gig card
  399. 16:10is a one-person machine. The second is
  400. 16:13anyone who barely uses it because RunPod
  401. 16:15rents an RTX 6000 with 96 GB of real
  402. 16:19VRAM for $2.09 an hour, and your $3,099
  403. 16:23Strix Halo buys you 1,483
  404. 16:27of those hours, which is about a year at
  405. 16:294 hours a day. And Alibaba's hosted Qwen
  406. 16:323.8 flash costs 47 cents per million
  407. 16:35output tokens, so $3,000 buys you 6.6
  408. 16:39billion output tokens. At 50 tokens per
  409. 16:42second, that's 4 years of writing
  410. 16:44without a break. So, if your box is
  411. 16:46going to sit idle most of the week, it
  412. 16:48makes more sense to rent the bit than to
  413. 16:50buy it. Now, this is how I'd put money
  414. 16:52down today. If you already own a gaming
  415. 16:55PC with a 12 gig card or better, buy 64
  416. 16:58GB of RAM and run the 3-bit build in
  417. 17:00Strata, and try that before you spend
  418. 17:02$3,000 on anything.
  419. 17:04If you're buying one box for 4-bit
  420. 17:06quality, get the Strix Halo and put
  421. 17:08Linux on it. And pick your engine before
  422. 17:10you pick your brand, because the default
  423. 17:12engine is where 7 tokens per second came
  424. 17:14from, and the right engine is where 50
  425. 17:17comes from.
  426. 17:18Buy the Mac Studio if silence and
  427. 17:20writing speed are worth the premium to
  428. 17:21you, and raise its memory limit on day
  429. 17:24one before you do anything else. Buy the
  430. 17:26DGX Spark only if your work specifically
  431. 17:28needs CUDA, not because you've heard
  432. 17:30CUDA's good, but because you have a
  433. 17:32reason it's required. Wait on the RTX
  434. 17:34Spark until someone outside Microsoft
  435. 17:36has actually timed it on this model. And
  436. 17:39only build four cards if you're serving
  437. 17:40a team and your power bill can absorb
  438. 17:42it. But never judge any of these
  439. 17:44machines on the default engine, because
  440. 17:46you'd be paying $3,000 for a number the
  441. 17:48box was never going to hit. I started
  442. 17:51this whole thing frustrated at four
  443. 17:52tokens per second on a laptop, and I
  444. 17:55ended up down a rabbit hole that
  445. 17:56included used server auction sites,
  446. 17:59Linux kernel flags, and
  447. 18:01at least one moment where I seriously
  448. 18:03considered buying four graphics cards
  449. 18:05and explaining it to my partner. What
  450. 18:07surprised me most wasn't the price, but
  451. 18:10that the bit, that one extra bit of
  452. 18:12precision per weight, is actually
  453. 18:14something you can buy with a dollar
  454. 18:16amount on it. Everything else is really
  455. 18:18just packaging around that number, and
  456. 18:20once you price the bit, the machine you
  457. 18:22need is often already under your desk.
  458. 18:25If this was useful, subscribe because
  459. 18:27I'm going to follow up with hands-on
  460. 18:28benchmarks once I stop arguing with
  461. 18:30myself about which box to buy. And drop
  462. 18:32a comment if you've got a setup running
  463. 18:34since I really want to know what engine
  464. 18:35you're on. See you in the next one.

About this transcript

This page contains the full transcript of Every Ways to Get 128GB VRAM for Local AI at Full Context by Kai, generated from the public captions YouTube serves with the video. The transcript has 3,252 words across 464 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.