54GB VRAM For $35 The Cheapest Local AI Hack? — Transcript
Full transcript
- 0:00Somebody in Russia paid 3,000 rubles,
- 0:03about 35 bucks, for a rusty old mining
- 0:05rig with 54 gigs of video memory,
- 0:08and it runs a 35 billion parameter AI
- 0:11model at over 30 tokens a second.
- 0:14But the number that actually made it
- 0:15fast isn't the one in the headline, and
- 0:17we'll get to which one, because you
- 0:18might already own it. For context, the
- 0:21expensive gaming cards have 24 gigs, and
- 0:23the really expensive ones have 48,
- 0:25right? And if you want 80,
- 0:27congratulations, you're now shopping for
- 0:29an Nvidia H100, which costs about the
- 0:32same as a small car, a good small car
- 0:34with heated seats. So yeah, the internet
- 0:37lost its mind. Tom's Hardware covered
- 0:39it, the UCCF Tech covered it. The Reddit
- 0:41post hit around 2,000 upvotes, and
- 0:44roughly 47 YouTube thumbnails showed up
- 0:46with a shocked face next to a pile of
- 0:48graphics cards. So what's the catch?
- 0:51His model only used 23 of those 54 gigs,
- 0:54and a second-hand GTX 1080 in a 2015
- 0:57desktop runs the exact same model at 25
- 1:00tokens a second.
- 1:02So if you've got an 8 gig card and 32
- 1:04gigs of RAM, there's a decent chance you
- 1:06already own this rig and you just don't
- 1:08know it yet.
- 1:09So I want to show you why nine graphics
- 1:11cards behave like one, why this thing is
- 1:14fast anyway, which number actually made
- 1:16it fast, and the one flag that gets you
- 1:19the same speed on hardware you probably
- 1:21already have. And at the end, we'll get
- 1:23to whether you should actually buy one
- 1:25of these at all.
- 1:26If you're new here, I'm Kai, and on this
- 1:28channel I work out what it really costs
- 1:30to run these models at home. And I'll be
- 1:32honest, I came to this story the
- 1:34expensive way. I had a little side
- 1:36project running, nothing crazy, just an
- 1:39agent that reads documents, summarizes
- 1:42them, and fires off some emails. For the
- 1:44first two weeks, I genuinely thought I
- 1:46was Tony Stark. Then the bill came in,
- 1:48and I was not Tony Stark. I was a guy
- 1:50who'd spent $140 running Claude over a
- 1:54document pipeline that could have been a
- 1:55Python script. Sound familiar?
- 1:58So what does any reasonable person do?
- 1:59At p.m. with my laptop cooking my legs
- 2:02and a cup of tea going cold on the desk,
- 2:04I Googled, how to run AI locally for
- 2:07free. That sent me down a rabbit hole
- 2:09that took like weeks through llama
- 2:12setups, through llama CPP builds that
- 2:15failed three times, and through one
- 2:17quantized model that answered every
- 2:19prompt like a philosophy student having
- 2:21a stroke. And at the end of every one of
- 2:23those roads, you hit the same wall,
- 2:25right? VRAM, the memory on your graphics
- 2:28card, and the reason a model flies on a
- 2:31rented server and types through mud on
- 2:33your three-year-old laptop.
- 2:35So when a $35 rig with 54 gigs of it
- 2:38showed up, I watched pretty much every
- 2:40video about it. And some of them got it
- 2:42right, but a lot of them quietly skipped
- 2:44the part that actually matters to you,
- 2:46which is what this means for the
- 2:47computer you already own. So let's start
- 2:49with the story because honestly, it's
- 2:51one of my favorite internet finds of the
- 2:53year.
- 2:54A guy in Russia buys a garage. Inside
- 2:56the garage, like a relic from a previous
- 2:58civilization, is an old crypto mining
- 3:01farm. Nine graphics cards hanging off a
- 3:03frame in a case that's gone rusty from
- 3:06years of sitting in the damp. The owner
- 3:08lists it on Avito, which if you're not
- 3:10aware, is basically Russia's eBay or
- 3:12Craigslist for 5,000 rubles, so about
- 3:15$60. And he's not even sure it works,
- 3:18and he flat out refuses to let anyone
- 3:20test it before buying.
- 3:22Now, most sensible people walk away at
- 3:24this point, right? A Reddit user called
- 3:26Mark Pruncken did not walk away. He took
- 3:29the guy down to 3,000 rubles, paid
- 3:31blind, dragged the rusty thing home, and
- 3:34booted it off a USB stick because there
- 3:36wasn't even an SSD inside.
- 3:38Did it work? All nine cards came up. He
- 3:41loads Quen 3.6, a 35 billion parameter
- 3:44model, and it runs it over 30 tokens a
- 3:47second, which is faster than most people
- 3:49read.
- 3:50And the math is kind of absurd, right?
- 3:52$35 for 54 gigs is about 65 cents per
- 3:55gigabyte, while a used RTX 3090 with 24
- 3:59gigs was was for a median of around
- 4:021,552
- 4:04on eBay over the last 90 days, which is
- 4:07$65 a gig.
- 4:09Same kind of memory 100 times the price.
- 4:12So what did I do? Exactly what you're
- 4:14thinking. I went straight to that place
- 4:16where you're typing P106 mining rig into
- 4:19Google and telling yourself it's an
- 4:21investment. I stayed there for like
- 4:23three days.
- 4:25I made a spreadsheet, I priced out power
- 4:27supplies,
- 4:28and then I read the comments. Because
- 4:30with viral tech posts, the headline is
- 4:33the sizzle, and the truth is buried
- 4:35somewhere around comment 47 posted by
- 4:37someone with a 10-year-old account and
- 4:39no flair who actually knows what they're
- 4:41talking about. This one was no
- 4:42different, and we'll get to what those
- 4:44comments said in a bit because they
- 4:47changed the whole picture. First though,
- 4:49you need to know what these cards
- 4:50actually are. These are NVIDIA P10600s
- 4:55and they came out in mid 2017 right in
- 4:58the middle of the first big crypto boom.
- 5:00NVIDIA took the chip from a GTX 1060, a
- 5:03totally normal gaming card from that
- 5:05era, ripped out every display output,
- 5:08and sold the results straight to miners.
- 5:10No HDMI, no display port. You can't even
- 5:13plug a monitor into one of these. So
- 5:15it's a graphics card that was never
- 5:17meant to show you any graphics. Its
- 5:19whole job from day one was solving
- 5:21hashes for Ethereum as fast as it could.
- 5:24Each card has 6 gigs of GDDR5 on a
- 5:27192-bit bus running at 8,008 MHz
- 5:31effective. And if we run those numbers,
- 5:33we get about 192 GB per second of memory
- 5:36bandwidth. I want to highlight that one
- 5:38because in about 2 minutes it's going to
- 5:39be the most important number in this
- 5:41video. Each card pulls up to 120 W and
- 5:44there are no tensor cores, which are the
- 5:47dedicated matrix math units NVIDIA
- 5:49started shipping after Pascal and the
- 5:51reason a Turing or Ampere RTX card chews
- 5:54through AI works so efficiently. The
- 5:56P106 is Pascal, so it's doing AI maths
- 6:00on hardware that was designed to do
- 6:01mining maths in 2017. Then on September
- 6:0415th, 2022, Ethereum switched from proof
- 6:08of work to proof of stake, and
- 6:10overnight, the only reason these cards
- 6:12existed just evaporated. Millions of
- 6:15mining cards suddenly had no job, so
- 6:17they flooded the secondhand market,
- 6:19prices collapsed, and they ended up in
- 6:21crates, in garages, and in the hands of
- 6:24people like Mark Pronkin.
- 6:25So that's the hardware. Nine cards, 54
- 6:28gigs combined, old architecture, no
- 6:31display outputs, cheapest chips. What
- 6:34could possibly go wrong?
- 6:36Hold that thought. We're coming back to
- 6:38it.
- 6:38Okay, here's the part that tripped me up
- 6:40completely when I started with multi-GPU
- 6:42setups, and I think it trips up most
- 6:44people watching videos about this rig.
- 6:46You'd assume nine cards act like one big
- 6:4954 gig card, right? Everything working
- 6:51at once, the job split nine ways, nine
- 6:55times the speed.
- 6:56Nope.
- 6:57This rig has 54 gigs of memory and
- 7:00roughly the speed of one card. So why
- 7:02one card?
- 7:03Every time a language model writes a
- 7:05token, which is roughly a word, it has
- 7:07to pull its weights out of memory and
- 7:10into the chip. The weights are the
- 7:12billions of numbers that hold everything
- 7:14the model learned. And for a normal
- 7:16model, it has to read all of them for
- 7:18every single token.
- 7:20Now here's the problem. These nine cards
- 7:22are hanging off single lanes of PCI
- 7:24Express 2.0, which is a pretty slow
- 7:27link. So the software splits the job by
- 7:29layers. Card one gets the first chunk of
- 7:32layers, card two gets the next chunk,
- 7:34and so on down to card nine. Card one
- 7:37runs its layers, hands a small result to
- 7:39card two, and waits. Card two does its
- 7:42bit, hands off to card three, and waits.
- 7:45And it goes all the way down the line to
- 7:46card nine, and then the whole thing
- 7:49starts over at card one for the next
- 7:51token. So at any moment, one card is
- 7:53working and eight cards are waiting,
- 7:55which kind of defeats the point of nine
- 7:57cards, right?
- 7:59What this means is you get the memory of
- 8:00all nine cards added together, that big
- 8:0354, and the bandwidth of exactly one
- 8:06card, 192
- 8:09gigs a second. And that lets us do some
- 8:11really simple division because for a
- 8:13dense model, the kind where every
- 8:15parameter gets used on every token, you
- 8:18just take the model size divided by the
- 8:20bandwidth and that's your speed ceiling.
- 8:22So let's try the obvious test, the
- 8:24biggest model that fits.
- 8:26Llama 3.3 70B at 4-bit is about 42.5
- 8:30gigs. So it fits in 54 with room to
- 8:32spare. That's the dream the headline is
- 8:34selling, right? A 70 billion parameter
- 8:37model on $35 of hardware.
- 8:39But 42.5 divided by 192 gives you a
- 8:43ceiling of about 4.5 tokens a second and
- 8:46in the real world you'd see under three.
- 8:48So a 500 token answer, two or three
- 8:51paragraphs, takes almost three minutes.
- 8:53Three minutes for two paragraphs. I've
- 8:56chatted with people who type slower than
- 8:57that, but that's not the point. The
- 8:59point is the 54 gigs lets you load the
- 9:02model. It doesn't let you enjoy it. So
- 9:04how is he getting 30?
- 9:06The answer's in the model's name. He ran
- 9:08Quen 3.6 35B and the A3B is the bit that
- 9:12matters. This is a mixture of experts
- 9:15model or MOE and honestly I'd argue
- 9:18understanding MOE is the single most
- 9:20useful thing you can learn about local
- 9:22AI right now because it changes how you
- 9:24should think about hardware.
- 9:26A dense model like Llama 3.3 70B uses
- 9:29every parameter in every layer for every
- 9:32token. All 70 billion every time. It's
- 9:35like having 70 billion specialists and
- 9:38dragging every one of them into every
- 9:40meeting, even when most of them have
- 9:41nothing to say.
- 9:43A mixture of experts model still stores
- 9:45all its parameters, 35 billion here,
- 9:47which is the 35B in the name, but inside
- 9:50each of its 40 layers, instead of one
- 9:53big network, there are 256
- 9:56small specialist networks called
- 9:58experts. And a router, which is
- 10:00basically a tiny decision maker, looks
- 10:03at each token and switches on just eight
- 10:06of those 256
- 10:08plus one shared expert that always runs.
- 10:10So, per token, only about three billion
- 10:13parameters actually get read, and the
- 10:15other 32 billion just sit there in
- 10:17memory in case they're needed. It stores
- 10:1935 billion, but it only reads three,
- 10:22right? At four bit, that's roughly two
- 10:25gigs read per token instead of the 42
- 10:27you'd need for the dense 70B.
- 10:30And 192 gigs a second divided by two
- 10:33gigs a token gives you a ceiling of
- 10:35about 96 tokens a second. Wait. 96? He
- 10:40got 30. That's about a third of the
- 10:42ceiling, and we'll come back to where
- 10:43the rest went because it's actually good
- 10:45news for you. Either way, 30 tokens a
- 10:48second from $35 of hardware. The
- 10:51original write-up had a metaphor for
- 10:52this that I really like.
- 10:54The rig is a massive warehouse with one
- 10:57slow loading dock. If you order
- 10:59everything in the warehouse every day,
- 11:01that dock is your bottleneck. But this
- 11:03model only ever orders a few boxes at a
- 11:05time, so most of the warehouse just sits
- 11:07there, and a slow dock is fine when your
- 11:09orders are small. That's the whole
- 11:11trick. The model is cheap to read, so
- 11:13the slow dock barely matters.
- 11:15And when that clicked, I had a slightly
- 11:17uncomfortable realization because the 54
- 11:20gigs, the thing in the headline and in
- 11:22every single thumbnail, wasn't doing
- 11:24what I thought it was doing.
- 11:26He ran a 23 gig model on a 54 gig rig.
- 11:29That's 31 gigs, about five cards worth,
- 11:32holding nothing at all. So now flip it
- 11:35around. What if you don't have 54 gigs?
- 11:38What if you've got eight gigs of VRAM
- 11:40and 32 gigs of normal system RAM? Can a
- 11:4323 gig model run on that?
- 11:45This is where most of the videos on this
- 11:47rig buried the lid. Remember, the router
- 11:50picks eight experts out of 256. So in
- 11:53every layer, the other 248 experts are
- 11:56idle for that token. They don't need to
- 11:58live on the GPU, right? They just need
- 12:00to be reachable. And Llama CPP, the free
- 12:03open source engine that O Llama was
- 12:05built on and that sits under most local
- 12:07AI apps, has a flag for exactly this.
- 12:10It's called N CPU MOE, and it keeps the
- 12:13expert weights in your system RAM while
- 12:15the attention layers and the shared
- 12:17weights, the stuff that runs on every
- 12:19single token, stay on the graphics card.
- 12:21Per token, the router picks its eight
- 12:23experts. Your CPU runs those eight
- 12:26straight out of RAM. The card handles
- 12:28everything else and the other 248 never
- 12:31get touched. So an 8 gig card can run a
- 12:3423 gig model. And not just technically
- 12:37run it, run it fast. On May 11th, a
- 12:40developer called Martin Andrews posted
- 12:42his numbers. His machine was a GTX 1080
- 12:45with 8 gigs, so the same Pascal
- 12:47generation as the P106, a card he picked
- 12:50up second hand for under $200 sitting
- 12:52next to an Intel i7 6700 from 2015 and
- 12:5632 gigs of RAM. Same model, Qwen 3.6 35B
- 13:00at 4 bit with N CPU MOE set to 30. Over
- 13:03a 2,122 token answer, it wrote 24.94
- 13:08tokens a second. One card, 83% of the
- 13:11nine card rig. Then XDA developers ran
- 13:14it on a Lenovo Legion 5 gaming laptop
- 13:16with a mobile RTX 3070 and 8 gigs of
- 13:19VRAM. Without the flag, four tokens a
- 13:21second. With N CPU MOE at 28, 32.82,
- 13:26faster than the nine card rig on a
- 13:28laptop from one flag.
- 13:30Another developer tried a 12 gig RTX
- 13:323060 with 31 gigs of DDR4. Without the
- 13:36flag, it ran out of memory before it
- 13:37even finished loading. With the flag,
- 13:41on a slightly smaller three bit build.
- 13:44And at the top end, someone measured a
- 13:46single used RTX 3090 at 101.7
- 13:50tokens a second on the same model.
- 13:53That's 3.4 times the rig. So here's the
- 13:56leaderboard for the exact model in the
- 13:58headline. The RTX 3060, 23 tokens a
- 14:01second. The 2016 GTX 1080 desktop, 25.
- 14:05The $35 nine card rig, 30. The gaming
- 14:09laptop with a mobile 3070, 33. And
- 14:1213090, about 100. The rig lands right in
- 14:16middle of hardware most gamers already
- 14:18own.
- 14:19I had two feelings about this list at
- 14:20once, and you might too.
- 14:22Is this disappointing?
- 14:24Or is this kind of amazing?
- 14:25Honestly, it's both. If you've already
- 14:28got a gaming PC with 8 GB of VRAM and 32
- 14:31GB of RAM, you might already own this
- 14:33rig's performance, which is a bit
- 14:35deflating if you were excited about the
- 14:36rig.
- 14:37But a 2015 desktop with a 2016 graphics
- 14:40card running a 35 billion parameter
- 14:42model at 25 tokens a second is wild, and
- 14:46almost nobody knows the flag exists.
- 14:48Now, I've spent a few minutes tearing
- 14:50down the rig's main selling point, so
- 14:52let me be fair. The 54 GB isn't useless.
- 14:55It just isn't what made this thing fast.
- 14:57So what does it actually buy you? Three
- 15:00things that an 8 GB card plus 32 GB of
- 15:03RAM really can match.
- 15:05Number one is higher precision. The same
- 15:07Qwen model at 8-bit gives you noticeably
- 15:10better output, better reasoning, fewer
- 15:12weird hallucinations, and it needs about
- 15:1538 GB in total.
- 15:17On a 40 GB system that's also running an
- 15:19OS and a browser with 11 tabs open about
- 15:21why your llama CPP build failed,
- 15:24that doesn't fit. On the rig, it does
- 15:26with room to spare.
- 15:28Number two is long context, and this one
- 15:30surprised me.
- 15:31Every token you feed the model, every
- 15:33page of a document, every line of code,
- 15:37every message in the chat takes memory
- 15:39to hold while it's working. That's the
- 15:41KV cache, which is basically the model's
- 15:44short-term memory. So it grows with
- 15:46every token you add, right?
- 15:48On an 8 GB card, a long conversation
- 15:50eats into the room you need for the
- 15:52weights, and eventually stuff spills
- 15:54into RAM or just breaks. On the rig, all
- 15:56of it stays in fast VRAM.
- 15:59One developer estimated that a 65,000
- 16:01token context, think pasting a whole
- 16:04codebase into the chat, takes 10 to 15
- 16:07GB on top of the weights. So that 31 GB
- 16:10of empty VRAM suddenly very occupied. So
- 16:13if you do document analysis, long-form
- 16:16writing, code review across big repos,
- 16:19anything where you feed the model a lot,
- 16:21that spare VRAM is genuinely valuable.
- 16:24Number three is something the buyer
- 16:25mentioned himself, which is nine
- 16:27separate workers.
- 16:29Each P106 has 6 GB and reads at 192 GB a
- 16:33second when it isn't waiting on its
- 16:35neighbors. Split the work right, nine
- 16:37small jobs in parallel instead of one
- 16:39long chain, and you've got nine decent
- 16:41little workers going at once. That's
- 16:43less run the biggest model and more run
- 16:46lots of things at once, and his plan
- 16:48points that way. He wants to move the
- 16:49cards into a bigger 12-slot case with a
- 16:52proper CPU and add three more cards,
- 16:54which takes it to 72 GB. Those are real
- 16:56uses. They're not the use in the
- 16:58thumbnail, but they're not nothing.
- 17:00Okay, remember comment 47? Here's where
- 17:03I get nitpicky, and I'm not sorry,
- 17:05because this is what separates a useful
- 17:07tech video from a dopamine dispenser and
- 17:09a trench coat. Almost every video on
- 17:11this rig quoted the same two numbers,
- 17:14and nobody actually measured either of
- 17:16them. The first is about 400 tokens a
- 17:18second for reading your prompt, which is
- 17:20called prefill.
- 17:22It sounds plausible, right? Nine cards,
- 17:25decent raw compute, and prompt reading
- 17:27leans on raw maths more than bandwidth.
- 17:30But here's the problem. Pascal has no
- 17:32tensor cores, so the units that make
- 17:34modern cards blaze through that maths
- 17:36just aren't there. And nobody has
- 17:38measured prefill on this rig. It's a
- 17:40guess that turned into a fact by being
- 17:42repeated. I've seen it quoted in videos
- 17:44like it's settled. It isn't. The second
- 17:47is 150 W at idle. Measured? Nope. The
- 17:51one person in the thread who did
- 17:53measure, someone with two P102,
- 17:56the 10 GB cousin of the P106, reported 7
- 18:00W per card sitting idle, which times
- 18:02nine is 63 W, not 150. And that matters
- 18:07because power is the running cost that
- 18:09decides whether local AI actually saves
- 18:12you money, right?
- 18:14So let's do the real numbers. Nine cards
- 18:16at 7 W is 63 W. Leave it on for a month,
- 18:20which is 730 hours, and that's 46
- 18:23kilowatt hours. And at the US average
- 18:25home rate of about 18 cents, that's
- 18:26about $8.35 a month for the rig to sit
- 18:30there doing nothing. In California, it's
- 18:32closer to $14. Now add actual use. With
- 18:35the layer split running, one card works
- 18:38at up to 120 watts, while the other
- 18:40eight idle. So call it 176 watts for the
- 18:43cards while it's riding. And remember,
- 18:45that's a ceiling, not a measurement. At
- 18:4730 tokens a second, a million tokens
- 18:50takes about 9.26 hours, and 176 watts
- 18:54for 9.26 hours is 1.63 kilowatt hours,
- 18:58which works out to about 30 cents per
- 19:00million output tokens. So how does that
- 19:02compare to just paying someone? Open
- 19:05Router lists this exact model at about
- 19:0790 cents per million output tokens from
- 19:09most providers and as low as 10 cents
- 19:12from the cheapest one right now. So
- 19:13writing on the rig is cheaper if you're
- 19:15paying for power, right? 30 cents versus
- 19:1790. But here's the kicker, and the
- 19:19spreadsheet only shows it when you
- 19:20squint. That $8.35 idle bill at the US
- 19:24average buys you about 9.3 million
- 19:27output tokens at 90 cents a million.
- 19:30That's the rig writing nonstop,
- 19:32literally nonstop, for 86 hours a month
- 19:35just to break even on standby. A heavy
- 19:37chat user might put out a couple of
- 19:39million output tokens a month. At that
- 19:41rate, you'd spend $2 on a hosted API and
- 19:44over $8 keeping the rig awake. Not
- 19:47great. If you run agents all day, the
- 19:49rig writes for hours at a stretch, your
- 19:52power's cheap, and the math flips your
- 19:54way. If you use it for an hour in the
- 19:55evening and then go to bed, switch it
- 19:57off at the wall. And again, the 8 gig
- 20:00card in your desktop is already plugged
- 20:02in and already on your bill. It doesn't
- 20:04cost you anything extra to do AI work.
- 20:06Now you might be asking, Kai, why not
- 20:09just do what every other video said?
- 20:11Because almost all of them ended the
- 20:12same way. If you're not in Russia, buy a
- 20:15Tesla P100 or a P40 instead. Data center
- 20:19cards with more memory and similar cheap
- 20:21prices secondhand. Sounds reasonable,
- 20:23right?
- 20:24So here's what happened to those prices
- 20:26after the videos came out.
- 20:28According to GPU Dojo's price tracker,
- 20:30on August 28th, the cheapest 16 gig
- 20:33Tesla P100 was $79.97.
- 20:38By November 13th, about two and a half
- 20:40months later, it was $219.99.
- 20:45That's 2.75 times higher.
- 20:48The 24 gig P40 went from $239 to
- 20:51$383.90,
- 20:54up 61%.
- 20:56The 32 gig V100 went from $650 to
- 20:59$1,039.
- 21:01There's a comment with 179 likes that
- 21:04basically writes the epitaph for every
- 21:06cheap AI hardware video, and I'll
- 21:08paraphrase it.
- 21:09Every time a video shows cheap hardware
- 21:11running AI, that hardware stops being
- 21:14cheap. The P106 themselves are still
- 21:16cheap in the US, for now. Single cards
- 21:19list for about $20 to $60, and there's
- 21:22apparently a lot of 24 going for $499,
- 21:26which is about $21 a card. So nine of
- 21:29those is around $189 for the cards
- 21:32alone.
- 21:33But the board, the frame, nine PCIe
- 21:35risers, and power supplies that can
- 21:38handle a 1,080 watt ceiling are all
- 21:40extra.
- 21:41That's what the $35 hides. He bought a
- 21:44finished rig, right? You'd be building
- 21:46one, and piece by piece with new parts,
- 21:49you are nowhere near $35 anymore.
- 21:52So if you still want to do this, copy
- 21:54what the buyer did. Search your local
- 21:56listings for a whole mining farm, not
- 21:58single cards, because the frame and the
- 22:00power supplies are where a from scratch
- 22:02build gets expensive fast.
- 22:04And plan to run it headless. No display
- 22:06outputs means you need a board with
- 22:08built-in graphics or another machine to
- 22:10connect from. And remember, what could
- 22:13possibly go wrong? Here's the other
- 22:15thing. Nvidia has said the 580 driver
- 22:17branch is the last one that supports
- 22:19Pascal, which is the architecture these
- 22:21cards are built on. CUDA 13 has already
- 22:24dropped offline compilation for anything
- 22:26below compute capability 7.5 and the
- 22:28P106 is 6.1. Security updates for the
- 22:32580 branch run through October 2028
- 22:35according to Tom's Hardware. After that,
- 22:37nothing new. Today it mostly works fine.
- 22:39Olama ships a CUDA 12 runner that
- 22:41handles these cards and Lama CPP still
- 22:44builds for Pascal against CUDA 12. The
- 22:46community advice is basically to pin
- 22:48your driver, stop it auto updating,
- 22:51stick to the Q4 and Q8 quant formats
- 22:53that use the 8-bit integer path, and
- 22:56stay away from anything heavy on half
- 22:57precision maths because on Pascal half
- 23:00precision runs at 1/64th of full speed.
- 23:0464th. That's not a typo. So you've got
- 23:07roughly two years of runway with a fence
- 23:09at the end and after 2028 things get
- 23:12uncertain.
- 23:13The GTX 1080 from earlier is in exactly
- 23:15the same boat, Pascal being Pascal.
- 23:18For a $35 rig that paid for itself on
- 23:20day one, two more years is a great
- 23:22return, right? If you're building from
- 23:24scratch right now, factor in that
- 23:26timeline. So should you buy one? I want
- 23:28to give you a straight answer and not a
- 23:30hedge. There's one person this rig is
- 23:32exactly right for and it's the guy who
- 23:34bought it. Why him? His electricity is
- 23:37included in his rent and even if it
- 23:38weren't, he says power is cheap where he
- 23:41lives. P106 and their 10 gig cousins go
- 23:44for $10 to $20 locally. He wanted
- 23:47unlimited use, long context, and privacy
- 23:51and he doesn't want an AI agent reading
- 23:53his messages through someone else's
- 23:54server while he pays for the privilege.
- 23:56For him, almost every cost in this video
- 23:59is basically zero. A complete rig for
- 24:02$35, free power, room to run a bigger
- 24:05model at 8-bit with 65,000 tokens of
- 24:08context, all fully local. That's a great
- 24:11purchase. And the same logic works for
- 24:13you if all of this is true. You find a
- 24:15complete working rig near scrap price
- 24:18and I mean complete. So cards, frame,
- 24:21board, power supply, everything. Your
- 24:24power is cheap, included, or you'll run
- 24:26it enough to cover standby. You're fine
- 24:28on Linux with hardware that has no
- 24:30display outputs,
- 24:31and you specifically need what 54 gigs
- 24:33gives you, which is eight-bit precision,
- 24:36very long context, or lots of small jobs
- 24:39in parallel.
- 24:40If those don't all apply, your upgrade
- 24:42probably isn't a mining rig. It might be
- 24:44better RAM, it might be a machine where
- 24:46the CPU and GPU share one unified pool
- 24:49of memory, and there are some options
- 24:51going that way. Or it might just be
- 24:52turning on one flag and finding out you
- 24:55already have what you need.
- 24:56So, would running it locally have saved
- 24:59me that $140?
- 25:01Yes, just not on a mining rig.
- 25:03Don't buy a mining rig to run the model
- 25:05in this headline. Run it on what you've
- 25:07got first. So, what do you actually do
- 25:09tonight? Here are the steps so you can
- 25:11skip the three days of forum diving I
- 25:13did. If your desktop has an eight-gig
- 25:15NVIDIA card and 32 gigs of RAM, you
- 25:18probably already own the rig's 30 tokens
- 25:20a second, right?
- 25:22It's sitting right there, and the only
- 25:23thing missing is the flag. Step one,
- 25:26install llama cpp. Why not olama?
- 25:29Because n cpu moe is a llama cpp flag,
- 25:33so you want llama cpp directly. The XDA
- 25:35Developers article on this model is a
- 25:37solid reference because they ran it
- 25:39clean with and without the flag and
- 25:41published the numbers properly. Step
- 25:43two, grab a four-bit build of Quinn 3.6
- 25:4735 B. Unsloth's builds are the most
- 25:49popular and well maintained.
- 25:51The download's about 23 gigs, so clear
- 25:53some space and set n cpu moe to 30 to
- 25:57start. Step three, run llama bench,
- 26:00which is the benchmark tool built into
- 26:02llama cpp instead of a does this feel
- 26:05fast vibe check? Run it three times with
- 26:07the flag and three times without because
- 26:09one run can be noisy, and compare the
- 26:11generation tokens per second. Step four,
- 26:14if you're on olama and want to stay
- 26:16there, run olama ps before anything
- 26:19else. If it shows part of your model
- 26:21sitting on the CPU, you're already
- 26:23paying the speed cost of not using the
- 26:24flag, you just don't know it. Step five,
- 26:28still under 20 tokens a second and you
- 26:30think it should be
- 26:31Check your RAM before you look at
- 26:32graphics cards. The experts get read
- 26:34from system RAM, right? So if you're
- 26:36running one stick in single channel
- 26:38instead of two, you're leaving a lot of
- 26:40bandwidth on the table. Two sticks in
- 26:42dual channel on an older Intel platform
- 26:44can double it. That's a free upgrade if
- 26:46you've got a spare stick in a drawer.
- 26:48Only once you know what your current
- 26:49hardware actually does does it make
- 26:51sense to think about new hardware. And
- 26:53when you do, price it by what the model
- 26:55reads per token, not by what the card
- 26:57holds. That's the number that sets your
- 26:59speed.
- 27:00Which brings us back to the missing
- 27:02tokens. The math said 96 and he got 30.
- 27:06But look at what that 30 came off. A
- 27:082017 architecture, nine cards handing
- 27:11every token down a chain of single lane
- 27:13PCIe links, a Celeron with a few gigs of
- 27:16RAM, a default Ollama install booted
- 27:19from USB with zero tuning.
- 27:22My guess is those nine handoffs eat a
- 27:24big chunk of the gap, but nobody's
- 27:26tested that properly yet, so treat it as
- 27:29a guess. And that's the good news for
- 27:30you because your one card doesn't have
- 27:33nine handoffs, which is part of why a
- 27:34single GTX 1080 gets to 25. So the
- 27:38number that made this rig fast was the
- 27:40three, the 3 billion parameters it reads
- 27:43per token, never the 54. And I think
- 27:46that's the useful bit under all the
- 27:47hardware talk. Right now, almost every
- 27:49hardware conversation in AI is about how
- 27:51much you can hold. VRAM, parameters,
- 27:55context windows, gigabytes. Those
- 27:58numbers make great thumbnails, right?
- 28:00The question that actually decides fast
- 28:02or slow, cheap or expensive is how much
- 28:05you have to read. Mixture of experts is
- 28:07shaking up local AI precisely because it
- 28:10splits those two apart. So a 35 billion
- 28:12parameter model only reads 3 billion per
- 28:15token. Once that clicks, a lot of
- 28:17hardware you wrote off as two weeks
- 28:19starts looking a lot more capable. And
- 28:21my API build problem solved. I'm running
- 28:23my pipeline locally now on a machine I
- 28:25already owned. Not a flashy rig, not a
- 28:28$35 garage find, just a desktop with an
- 28:318 gig card, 32 gigs of RAM, and one flag
- 28:34in a config file. It does 25 tokens a
- 28:37second, which is the same as a nine
- 28:39card, 54 gig $35 mining rig. I did still
- 28:44make a spreadsheet about it. Old habits.
- 28:46Anyway, that's it for this one. If this
- 28:49helped you figure out your setup,
- 28:50subscribe because I do a lot of these
- 28:52where we go past the headline and find
- 28:54out what's real. And if you try this,
- 28:56drop a comment with your card and the
- 28:58number you get after you turn on the
- 29:00flag.
- 29:00I want to build a leaderboard out of
- 29:02real numbers instead of theoretical
- 29:04ones. See you in the next one.
About this transcript
This page contains the full transcript of 54GB VRAM For $35 The Cheapest Local AI Hack? by Kai, generated from the public captions YouTube serves with the video. The transcript has 5,105 words across 739 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.