Every Ways to Get 128GB VRAM for Local AI at Full Context — Transcript
Full transcript
- 0:00Okay, so 6 months ago I finally got fed
- 0:03up. I was running a local model at
- 0:05something like four tokens per second.
- 0:07My laptop fan sounded like it was
- 0:09auditioning for a jet engine role. And
- 0:11meanwhile, I was watching people on
- 0:13Reddit post screenshots of 60 or 70
- 0:16tokens a second on their own machines
- 0:17while I sat there thinking,
- 0:19"What am I even doing wrong?" So, I did
- 0:21what any reasonable person does, which
- 0:23is open a browser and type in, "How much
- 0:25VRAM do I actually need for local AI?"
- 0:28And that's how I fell into a rabbit hole
- 0:30that I'm still trying to climb out of.
- 0:33Everyone gives the same answer, and it's
- 0:35128 GB. That's the golden threshold
- 0:38where you get to run the big models and
- 0:40never pay an API bill again, and it
- 0:42sounded a lot like buying a dream. And
- 0:45then I started pricing it out, and I
- 0:47want you to feel what I felt here
- 0:49because the cheapest legit option I
- 0:50found was $3,000, and the most expensive
- 0:53was almost 7,000, and somewhere in
- 0:55between those two numbers I started
- 0:57questioning every decision I've ever
- 0:59made. But what nobody told me is that
- 1:01not all of these options are actually
- 1:03buying you the same thing. Some of them
- 1:05are really just buying you a fourth bit,
- 1:07and that will make a lot more sense once
- 1:09we've priced everything. So, let's go
- 1:11through every route to 128 gigs, and I
- 1:14mean every route. We'll look at the mini
- 1:16PC, the Mac Studio, and the Nvidia box,
- 1:19then four used server cards, and then
- 1:22the option that made me feel stupid for
- 1:23not trying it first. Now, before we
- 1:26price anything, we need to understand
- 1:28why 128 GB became the number in the
- 1:31first place. And it comes down to one
- 1:33model, which is Qwen 3.8 Flash Next.
- 1:37If you spend any time on the local Llama
- 1:38subreddit, and I basically live there at
- 1:40this point, this model is everywhere,
- 1:43and it's become the default answer to,
- 1:45"What do I run on a big memory machine?"
- 1:47One person posted that they haven't
- 1:49called a cloud API since the day it came
- 1:51out, which gives you a sense of the
- 1:52energy around it. And the reason 128
- 1:55matters is boring but real. The four-bit
- 1:58builds of this model land between 94 and
- 2:01124 GB depending on the format, which
- 2:04fits in 128 and in nothing smaller. So,
- 2:07the logic writes itself. Where you get
- 2:09128 gigs, load the 4-bit file, and
- 2:12you're done. But, as always, it's not
- 2:14actually that simple. Now, this is where
- 2:16it got interesting for me, and also a
- 2:18little embarrassing because I'd been
- 2:20thinking about it all wrong. I assumed,
- 2:22pretty logically I thought, that if a
- 2:24model has 125 billion parameters, you
- 2:26need all 125 billion sitting in fast
- 2:29memory to run it fast. And FlashNext
- 2:31just doesn't work that way. The model
- 2:33does have 125 billion parameters, and it
- 2:37also has a 51 billion parameter lookup
- 2:39table for common word sequences, which
- 2:41sounds enormous.
- 2:43But, for each word it writes, it only
- 2:45actually reads about 6 billion of them.
- 2:47And at 3 bits per weight, that's a
- 2:48couple of gigabytes per word.
- 2:50Everything else just needs to be
- 2:51somewhere your chip can reach when a
- 2:53word calls for it. And when we look at
- 2:55the model card, it actually says the
- 2:57lookup table is more amenable to
- 2:59offloading than the rest of the model.
- 3:01So, there's an engine called Strata that
- 3:03keeps the active part on your GPU and
- 3:05the rest in normal system RAM.
- 3:07One person ran it on an RTX 3090, which
- 3:10is a card with only 24 GB of VRAM, and
- 3:13used 23 GB of VRAM plus 47 GB of RAM in
- 3:17total. In other words, nothing on that
- 3:19machine was 128 gigs.
- 3:22This idea has a name, which is expert
- 3:24offloading.
- 3:25The way I picture it, the 128 gig box is
- 3:28a desk big enough to hold every book you
- 3:30own open at once, while the PC setup is
- 3:33a smaller desk next to a shelf.
- 3:35And if each word only opens a few pages
- 3:37at a time, the shelf is fine and you
- 3:39don't need the big desk. That was the
- 3:41realization that shifted everything for
- 3:43me, because it means capacity and speed
- 3:45are no longer the same purchase. Now,
- 3:48let's say we still want the big pool,
- 3:50and we're buying 128 gigs anyway.
- 3:52There's a detail here that will mildly
- 3:54ruin your week.
- 3:56On Windows, AMD's Ryzen AI Max mini PCs,
- 4:00which are one of the main routes to 128
- 4:02gigs, let the GPU side claim at most 96
- 4:06GB by default. So, the box was sold as
- 4:09128, and the other 32 just sit there
- 4:12with the CPU. On Linux, it can go past
- 4:15100, and some owners report getting
- 4:17close to 120. And a 128 gig Mac does
- 4:20much the same thing because macOS caps
- 4:23what the GPU can lock at around 96 gigs.
- 4:26The developer of one of the main Mac
- 4:28engines actually tells you to go into
- 4:30the terminal and raise that limit by
- 4:32hand before you run anything at full
- 4:34context. And when we hold those 96 gigs
- 4:37against the 4-bit builds, which start at
- 4:3994 gigs before any context window at
- 4:42all, it's pretty clear where this is
- 4:44going. Even the $6,950
- 4:47Nvidia box, the DGX Spark, only sees
- 4:50about 122 usable gigs. And since
- 4:53Nvidia's own 4-bit file is 124 GB, the
- 4:57most popular recipe for running it
- 4:59leaves about 48 gigs on the SSD and
- 5:01reads it from there. So, the nearly
- 5:03$7,000 machine is also offloading.
- 5:06Microsoft at least has the decency to
- 5:08mention it because their new laptops say
- 5:11in a footnote that the memory the GPU
- 5:13can address is less than the total. But,
- 5:15they never say how much less. And the
- 5:18128 on the box is really a ceiling, but
- 5:20the default sitting well under it. Now,
- 5:23if we've decided we want to pull anyway,
- 5:25we should start with the cheapest way to
- 5:27get it. That's AMD's Ryzen AI Max + 395
- 5:31chip, which everyone calls Strix Halo.
- 5:34It starts at $3,099 in a mini PC from a
- 5:37brand called Bosque Game, while
- 5:39Framework makes one for $3,449
- 5:42that's currently out of stock, and
- 5:44Minisforum's version is $3,799.
- 5:48And it's the same chip with the same 256
- 5:50GB per second of memory bandwidth in all
- 5:53of them. But, the engine you run on this
- 5:54machine really matters more than the
- 5:56brand of the box. Someone ran Flash Next
- 5:59on a Strix Halo machine with two
- 6:01different inference engines, 160,000
- 6:05tokens deep into a conversation. The
- 6:07first engine read that prompt at 113
- 6:10tokens a second, and the second one, a
- 6:12community fork called Gufo, read it at
- 6:141,227.
- 6:16And that's a 160,000
- 6:19token prompt loading in either 23
- 6:21minutes or 2 minutes on the same
- 6:23hardware with the same model, where the
- 6:25only thing that changed was the engine.
- 6:27The newer engines go even further.
- 6:29Strata posts 47 to 54 tokens per second
- 6:32on the 4-bit build, and around 1,300
- 6:35tokens per second on prompt reading. And
- 6:37Halogen, which is another engine a lot
- 6:39of owners swear by, posts similar
- 6:41numbers. It's closed source, though, and
- 6:43its forums are mostly people asking when
- 6:45that changes. And I'm right there with
- 6:47them. A bigger sibling is also coming,
- 6:49which is the Ryzen AI Max Pro 495 with
- 6:53192 GB at about $6,799
- 6:57from Framework, and that gives you more
- 6:59headroom at a similar speed. But for
- 7:01right now, if you put Linux on a Strix
- 7:03Halo box and pick the right engine, it's
- 7:05the cheapest machine that runs the 4-bit
- 7:07build well. Now we get to the two
- 7:09expensive options, and they split the
- 7:11work in really different ways. The Mac
- 7:13Studio with an M5 Max and 128 gigs runs
- 7:16$5,099.
- 7:18The 40-core GPU config is the only way
- 7:21to get 128 gigs, and it also brings 614
- 7:25GB per second, which is more than twice
- 7:28what Strix Halo has. And going from 64
- 7:30to 128 gigs costs $1,600
- 7:34on the Mac, which works out to about $25
- 7:36a gigabyte. On Flash Next specifically,
- 7:40the developer of the MLX serve engine
- 7:41reports about 40 tokens per second on
- 7:44pros and 75 on code. And the machine is
- 7:47quiet and polished with a software story
- 7:49on the Mac that's really good. But to be
- 7:52fair to the other boxes, nobody has
- 7:54published prompt reading speeds for this
- 7:56model on this machine yet. And that's a
- 7:58real gap because it matters for long
- 8:00context agent work. The DGX Spark is the
- 8:03CUDA machine at $6,950
- 8:06as of early October after Nvidia raised
- 8:08the price 74% from launch. It writes at
- 8:11about 41 tokens per second median and 27
- 8:14on pros, and it has a draft ahead trick
- 8:17that helps, because without it, the
- 8:18speed drops to 17. Where the spark used
- 8:21to shine was prompt reading. On an older
- 8:23model, it was five times faster than
- 8:25Strix Halo, which one benchmark
- 8:27translated into saving 83 seconds every
- 8:30time a coding agent re-reads a 30,000
- 8:33token context window. And that was a big
- 8:35deal.
- 8:36Now, on flash next with today's engines,
- 8:39the gap is 1,666
- 8:42tokens per second against 1,370.
- 8:45So, the 83-second advantage is now about
- 8:484 seconds, and you're paying a
- 8:50significant premium for those 4 seconds.
- 8:53And Nvidia just launched a 64 gig spark
- 8:55at $4,999,
- 8:58so two of them for 128 gigs comes to
- 9:01$10,000, and I'm not sure that needs any
- 9:04commentary from me. On October 7th,
- 9:06Nvidia quietly launched something called
- 9:08the RTX spark, which is a completely
- 9:11different chip from the DGX spark,
- 9:13despite the name. It's in Windows
- 9:15laptops and small desktops now.
- 9:17Microsoft's Surface Laptop Ultra with
- 9:19128 gigs is $5,899.99,
- 9:24and the desktop version, the Surface RTX
- 9:27spark dev box, is $5,999.99
- 9:31with 128 gigs. Microsoft is claiming 2.1
- 9:35times faster first word latency and 6.2
- 9:39times faster video than an M5 Pro
- 9:41MacBook Pro.
- 9:43But when we look at the footnotes, those
- 9:44numbers were measured on pre-production
- 9:4664 GB units, and the one petaflop figure
- 9:50uses four-bit math with sparsity turned
- 9:52on, which is not how you'd normally
- 9:54compare performance. Reported memory
- 9:56bandwidth puts it somewhere around 273
- 9:59to 307 GB per second, which is next to
- 10:02Strix Halo rather than next to the Mac
- 10:04Studio. So, my take is to treat it as a
- 10:06laptop that holds the four-bit build,
- 10:08and to wait for someone outside
- 10:10Microsoft to actually time it on this
- 10:12model before spending $6,000 on one.
- 10:14Now, we get to the chaotic option
- 10:16because the top voted 128 gig build on
- 10:19Local Llama this quarter came from
- 10:20someone who bought four used Radeon Pro
- 10:23V620 cards for $1,400
- 10:26total. They threw in a used EPYC server
- 10:29CPU, 256 gigs of DDR4 RAM for $610, and
- 10:34a 1,600 W power supply, and walked out
- 10:38with a whole machine for around $3,000.
- 10:41On Flash next at four bit, this thing
- 10:43writes 60 to 70 tokens per second, which
- 10:46on Pros is faster than every other box
- 10:48in this video. So, the server route wins
- 10:50on raw output speed. But, it also draws
- 10:53500 to 600 W while it's writing. At 4
- 10:56hours a day and $0.15 per kilowatt-hour,
- 10:59that works out to about $120 a year in
- 11:02electricity against around $38 a year
- 11:05for a Strix Halo box under similar load.
- 11:08So, the power bill really is part of the
- 11:09price tag, and the deal is gone because
- 11:12commenters started reporting that V620s
- 11:15climbed to the price of a 32 gig V100
- 11:18after that build went viral. New
- 11:20alternatives like Intel Arc Pro B70s and
- 11:23AMD Radeon AI Pro R9700s
- 11:27land around $6,800
- 11:29for four.
- 11:30Four used Instinct MI50s are $2,300 to
- 11:34$2,600 before a host machine, and they
- 11:37run on a chip AMD put in software
- 11:39maintenance mode years ago.
- 11:41And a single RTX Pro 6000 Blackwell with
- 11:4496 gigs runs $8,900
- 11:47to $13,250
- 11:50if you can find one. So, the server
- 11:51route made sense at $3,000, but at
- 11:54today's prices, it's a different
- 11:55calculation. Now, there's one route that
- 11:58none of the 128 GB guys price.
- 12:01StrataZone Readme lists a test setup
- 12:03with an RTX 5070 with 12 GB, a six-core
- 12:07Ryzen 5, and 64 GB of RAM. So, nothing
- 12:10fancy at all. On the two-bit build, it
- 12:12writes 94 tokens per second. On the
- 12:153-bit build, it writes 50 to 62. And it
- 12:18reads a 32,000 token prompt at 1,750
- 12:22tokens per second. And that's the
- 12:24writing speed of the $3,000 to $7,000
- 12:27boxes with faster reading than any of
- 12:29them on a card you might already own.
- 12:31Someone running a 3090, which is a card
- 12:33from 2020, in an external enclosure
- 12:36connected to a laptop with 64 gigs of
- 12:39DDR4 posted at 60 to 81 tokens per
- 12:42second on the 3-bit build. You might be
- 12:44wondering what happens if you just throw
- 12:46more RAM at the 4-bit build. One owner
- 12:48tried exactly that with an RTX 5090
- 12:51laptop GPU paired with 192 GB of RAM.
- 12:55The 3.5-bit build ran at 53 to 59 tokens
- 12:59per second, but the 4-bit build ran at
- 13:0121 to 33, which is about half the speed.
- 13:04So, the PC matches the pool at three
- 13:06bits and falls behind at four. And
- 13:09that's the line every route in this
- 13:11video lands on one side of.
- 13:13And this is the question nobody has put
- 13:15a number on, even though it's the one
- 13:17that actually matters. The lab that made
- 13:19the 3-bit file says it keeps 99.4%
- 13:23of the full model score across three
- 13:25benchmark tests. But on one of those
- 13:27tests, the full model already gets
- 13:29everything right. So, 99.4% of perfect
- 13:32is still perfect. Owners don't fully
- 13:35agree, either. Where one person called
- 13:37the two and three bit builds lobotomized
- 13:39and another runs the four bit build
- 13:41across three 3090s and says it blows 27B
- 13:44out of the water.
- 13:45And nobody has published both versions
- 13:47side by side on a long agent session,
- 13:49which is the actual work you'd buy 128
- 13:51gigs for. So, that data gap is real.
- 13:54Now, we can at least put a number on the
- 13:56price. If you already own a gaming PC
- 13:59with a 12 gig card, 64 GB of DDR5 costs
- 14:02about $880 at the cheapest listing. And
- 14:06that's your 3-bit setup. The cheapest
- 14:08machine that runs four bit well is the
- 14:10$3,099
- 14:11Strix Halo, which means the fourth bit
- 14:14costs you about $2,200
- 14:16as an upgrade. If you're building from
- 14:18scratch, the 5070, the RAM, and about
- 14:21$800 for a basic PC comes to around
- 14:23$2,400,
- 14:25and against the Strix Halo, the fourth
- 14:27bit costs $700. And $700 is the one
- 14:31number in this whole video that's easy
- 14:33to say yes to if you're building new and
- 14:35four-bit quality really matters to your
- 14:37workflow. Now, I want to be clear about
- 14:39the ceiling here because it's lower than
- 14:41it looks. Strider runs one model family,
- 14:45which is Flash Next and its variants.
- 14:47And the speed gains come directly from
- 14:49how that specific model is designed with
- 14:52a small active weight set and a lookup
- 14:54table built offload. So, if we point the
- 14:56same PC at a different large model,
- 14:59we're back to whatever standard
- 15:00llama.cpp does with it. Context is
- 15:02tighter, too, because one person running
- 15:05a 3.5-bit build on two 3090s with 128 GB
- 15:09of RAM got 130,000
- 15:12tokens of context and mentioned they
- 15:14might go back to Qwen 3.8-27B,
- 15:17which gave them 256,000 if things don't
- 15:20improve. And the next model might not
- 15:21cooperate at all. Qwen calls Flash Next
- 15:24a preview of the Qwen 4 architecture,
- 15:26and somebody noted the flash models have
- 15:28gotten bigger with every version, so the
- 15:30PC trick relies on design choices the
- 15:32next model might not make. But the pool
- 15:35doesn't care about any of that because a
- 15:37128 gig box loads whatever the next
- 15:39model is and whatever shape it takes as
- 15:41long as someone writes an engine for it.
- 15:44There are also two kinds of buyers the
- 15:45PC route doesn't work for, and I want to
- 15:47name them plainly. The first is anyone
- 15:50serving multiple people or running a
- 15:52swarm of agents at the same time, where
- 15:54the math changes completely. Strider's
- 15:56notes show four R9700 cards serving
- 15:59eight clients at 165 tokens per second
- 16:02total, and the DGX Spark holds 26 tokens
- 16:05per second per stream with four running,
- 16:08while a gaming PC with one 12-gig card
- 16:10is a one-person machine. The second is
- 16:13anyone who barely uses it because RunPod
- 16:15rents an RTX 6000 with 96 GB of real
- 16:19VRAM for $2.09 an hour, and your $3,099
- 16:23Strix Halo buys you 1,483
- 16:27of those hours, which is about a year at
- 16:294 hours a day. And Alibaba's hosted Qwen
- 16:323.8 flash costs 47 cents per million
- 16:35output tokens, so $3,000 buys you 6.6
- 16:39billion output tokens. At 50 tokens per
- 16:42second, that's 4 years of writing
- 16:44without a break. So, if your box is
- 16:46going to sit idle most of the week, it
- 16:48makes more sense to rent the bit than to
- 16:50buy it. Now, this is how I'd put money
- 16:52down today. If you already own a gaming
- 16:55PC with a 12 gig card or better, buy 64
- 16:58GB of RAM and run the 3-bit build in
- 17:00Strata, and try that before you spend
- 17:02$3,000 on anything.
- 17:04If you're buying one box for 4-bit
- 17:06quality, get the Strix Halo and put
- 17:08Linux on it. And pick your engine before
- 17:10you pick your brand, because the default
- 17:12engine is where 7 tokens per second came
- 17:14from, and the right engine is where 50
- 17:17comes from.
- 17:18Buy the Mac Studio if silence and
- 17:20writing speed are worth the premium to
- 17:21you, and raise its memory limit on day
- 17:24one before you do anything else. Buy the
- 17:26DGX Spark only if your work specifically
- 17:28needs CUDA, not because you've heard
- 17:30CUDA's good, but because you have a
- 17:32reason it's required. Wait on the RTX
- 17:34Spark until someone outside Microsoft
- 17:36has actually timed it on this model. And
- 17:39only build four cards if you're serving
- 17:40a team and your power bill can absorb
- 17:42it. But never judge any of these
- 17:44machines on the default engine, because
- 17:46you'd be paying $3,000 for a number the
- 17:48box was never going to hit. I started
- 17:51this whole thing frustrated at four
- 17:52tokens per second on a laptop, and I
- 17:55ended up down a rabbit hole that
- 17:56included used server auction sites,
- 17:59Linux kernel flags, and
- 18:01at least one moment where I seriously
- 18:03considered buying four graphics cards
- 18:05and explaining it to my partner. What
- 18:07surprised me most wasn't the price, but
- 18:10that the bit, that one extra bit of
- 18:12precision per weight, is actually
- 18:14something you can buy with a dollar
- 18:16amount on it. Everything else is really
- 18:18just packaging around that number, and
- 18:20once you price the bit, the machine you
- 18:22need is often already under your desk.
- 18:25If this was useful, subscribe because
- 18:27I'm going to follow up with hands-on
- 18:28benchmarks once I stop arguing with
- 18:30myself about which box to buy. And drop
- 18:32a comment if you've got a setup running
- 18:34since I really want to know what engine
- 18:35you're on. See you in the next one.
About this transcript
This page contains the full transcript of Every Ways to Get 128GB VRAM for Local AI at Full Context by Kai, generated from the public captions YouTube serves with the video. The transcript has 3,252 words across 464 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.