YouTube2Text

54GB VRAM For $35 The Cheapest Local AI Hack? — Transcript

by Kai · 5,105 words · 739 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Somebody in Russia paid 3,000 rubles,
  2. 0:03about 35 bucks, for a rusty old mining
  3. 0:05rig with 54 gigs of video memory,
  4. 0:08and it runs a 35 billion parameter AI
  5. 0:11model at over 30 tokens a second.
  6. 0:14But the number that actually made it
  7. 0:15fast isn't the one in the headline, and
  8. 0:17we'll get to which one, because you
  9. 0:18might already own it. For context, the
  10. 0:21expensive gaming cards have 24 gigs, and
  11. 0:23the really expensive ones have 48,
  12. 0:25right? And if you want 80,
  13. 0:27congratulations, you're now shopping for
  14. 0:29an Nvidia H100, which costs about the
  15. 0:32same as a small car, a good small car
  16. 0:34with heated seats. So yeah, the internet
  17. 0:37lost its mind. Tom's Hardware covered
  18. 0:39it, the UCCF Tech covered it. The Reddit
  19. 0:41post hit around 2,000 upvotes, and
  20. 0:44roughly 47 YouTube thumbnails showed up
  21. 0:46with a shocked face next to a pile of
  22. 0:48graphics cards. So what's the catch?
  23. 0:51His model only used 23 of those 54 gigs,
  24. 0:54and a second-hand GTX 1080 in a 2015
  25. 0:57desktop runs the exact same model at 25
  26. 1:00tokens a second.
  27. 1:02So if you've got an 8 gig card and 32
  28. 1:04gigs of RAM, there's a decent chance you
  29. 1:06already own this rig and you just don't
  30. 1:08know it yet.
  31. 1:09So I want to show you why nine graphics
  32. 1:11cards behave like one, why this thing is
  33. 1:14fast anyway, which number actually made
  34. 1:16it fast, and the one flag that gets you
  35. 1:19the same speed on hardware you probably
  36. 1:21already have. And at the end, we'll get
  37. 1:23to whether you should actually buy one
  38. 1:25of these at all.
  39. 1:26If you're new here, I'm Kai, and on this
  40. 1:28channel I work out what it really costs
  41. 1:30to run these models at home. And I'll be
  42. 1:32honest, I came to this story the
  43. 1:34expensive way. I had a little side
  44. 1:36project running, nothing crazy, just an
  45. 1:39agent that reads documents, summarizes
  46. 1:42them, and fires off some emails. For the
  47. 1:44first two weeks, I genuinely thought I
  48. 1:46was Tony Stark. Then the bill came in,
  49. 1:48and I was not Tony Stark. I was a guy
  50. 1:50who'd spent $140 running Claude over a
  51. 1:54document pipeline that could have been a
  52. 1:55Python script. Sound familiar?
  53. 1:58So what does any reasonable person do?
  54. 1:59At p.m. with my laptop cooking my legs
  55. 2:02and a cup of tea going cold on the desk,
  56. 2:04I Googled, how to run AI locally for
  57. 2:07free. That sent me down a rabbit hole
  58. 2:09that took like weeks through llama
  59. 2:12setups, through llama CPP builds that
  60. 2:15failed three times, and through one
  61. 2:17quantized model that answered every
  62. 2:19prompt like a philosophy student having
  63. 2:21a stroke. And at the end of every one of
  64. 2:23those roads, you hit the same wall,
  65. 2:25right? VRAM, the memory on your graphics
  66. 2:28card, and the reason a model flies on a
  67. 2:31rented server and types through mud on
  68. 2:33your three-year-old laptop.
  69. 2:35So when a $35 rig with 54 gigs of it
  70. 2:38showed up, I watched pretty much every
  71. 2:40video about it. And some of them got it
  72. 2:42right, but a lot of them quietly skipped
  73. 2:44the part that actually matters to you,
  74. 2:46which is what this means for the
  75. 2:47computer you already own. So let's start
  76. 2:49with the story because honestly, it's
  77. 2:51one of my favorite internet finds of the
  78. 2:53year.
  79. 2:54A guy in Russia buys a garage. Inside
  80. 2:56the garage, like a relic from a previous
  81. 2:58civilization, is an old crypto mining
  82. 3:01farm. Nine graphics cards hanging off a
  83. 3:03frame in a case that's gone rusty from
  84. 3:06years of sitting in the damp. The owner
  85. 3:08lists it on Avito, which if you're not
  86. 3:10aware, is basically Russia's eBay or
  87. 3:12Craigslist for 5,000 rubles, so about
  88. 3:15$60. And he's not even sure it works,
  89. 3:18and he flat out refuses to let anyone
  90. 3:20test it before buying.
  91. 3:22Now, most sensible people walk away at
  92. 3:24this point, right? A Reddit user called
  93. 3:26Mark Pruncken did not walk away. He took
  94. 3:29the guy down to 3,000 rubles, paid
  95. 3:31blind, dragged the rusty thing home, and
  96. 3:34booted it off a USB stick because there
  97. 3:36wasn't even an SSD inside.
  98. 3:38Did it work? All nine cards came up. He
  99. 3:41loads Quen 3.6, a 35 billion parameter
  100. 3:44model, and it runs it over 30 tokens a
  101. 3:47second, which is faster than most people
  102. 3:49read.
  103. 3:50And the math is kind of absurd, right?
  104. 3:52$35 for 54 gigs is about 65 cents per
  105. 3:55gigabyte, while a used RTX 3090 with 24
  106. 3:59gigs was was for a median of around
  107. 4:021,552
  108. 4:04on eBay over the last 90 days, which is
  109. 4:07$65 a gig.
  110. 4:09Same kind of memory 100 times the price.
  111. 4:12So what did I do? Exactly what you're
  112. 4:14thinking. I went straight to that place
  113. 4:16where you're typing P106 mining rig into
  114. 4:19Google and telling yourself it's an
  115. 4:21investment. I stayed there for like
  116. 4:23three days.
  117. 4:25I made a spreadsheet, I priced out power
  118. 4:27supplies,
  119. 4:28and then I read the comments. Because
  120. 4:30with viral tech posts, the headline is
  121. 4:33the sizzle, and the truth is buried
  122. 4:35somewhere around comment 47 posted by
  123. 4:37someone with a 10-year-old account and
  124. 4:39no flair who actually knows what they're
  125. 4:41talking about. This one was no
  126. 4:42different, and we'll get to what those
  127. 4:44comments said in a bit because they
  128. 4:47changed the whole picture. First though,
  129. 4:49you need to know what these cards
  130. 4:50actually are. These are NVIDIA P10600s
  131. 4:55and they came out in mid 2017 right in
  132. 4:58the middle of the first big crypto boom.
  133. 5:00NVIDIA took the chip from a GTX 1060, a
  134. 5:03totally normal gaming card from that
  135. 5:05era, ripped out every display output,
  136. 5:08and sold the results straight to miners.
  137. 5:10No HDMI, no display port. You can't even
  138. 5:13plug a monitor into one of these. So
  139. 5:15it's a graphics card that was never
  140. 5:17meant to show you any graphics. Its
  141. 5:19whole job from day one was solving
  142. 5:21hashes for Ethereum as fast as it could.
  143. 5:24Each card has 6 gigs of GDDR5 on a
  144. 5:27192-bit bus running at 8,008 MHz
  145. 5:31effective. And if we run those numbers,
  146. 5:33we get about 192 GB per second of memory
  147. 5:36bandwidth. I want to highlight that one
  148. 5:38because in about 2 minutes it's going to
  149. 5:39be the most important number in this
  150. 5:41video. Each card pulls up to 120 W and
  151. 5:44there are no tensor cores, which are the
  152. 5:47dedicated matrix math units NVIDIA
  153. 5:49started shipping after Pascal and the
  154. 5:51reason a Turing or Ampere RTX card chews
  155. 5:54through AI works so efficiently. The
  156. 5:56P106 is Pascal, so it's doing AI maths
  157. 6:00on hardware that was designed to do
  158. 6:01mining maths in 2017. Then on September
  159. 6:0415th, 2022, Ethereum switched from proof
  160. 6:08of work to proof of stake, and
  161. 6:10overnight, the only reason these cards
  162. 6:12existed just evaporated. Millions of
  163. 6:15mining cards suddenly had no job, so
  164. 6:17they flooded the secondhand market,
  165. 6:19prices collapsed, and they ended up in
  166. 6:21crates, in garages, and in the hands of
  167. 6:24people like Mark Pronkin.
  168. 6:25So that's the hardware. Nine cards, 54
  169. 6:28gigs combined, old architecture, no
  170. 6:31display outputs, cheapest chips. What
  171. 6:34could possibly go wrong?
  172. 6:36Hold that thought. We're coming back to
  173. 6:38it.
  174. 6:38Okay, here's the part that tripped me up
  175. 6:40completely when I started with multi-GPU
  176. 6:42setups, and I think it trips up most
  177. 6:44people watching videos about this rig.
  178. 6:46You'd assume nine cards act like one big
  179. 6:4954 gig card, right? Everything working
  180. 6:51at once, the job split nine ways, nine
  181. 6:55times the speed.
  182. 6:56Nope.
  183. 6:57This rig has 54 gigs of memory and
  184. 7:00roughly the speed of one card. So why
  185. 7:02one card?
  186. 7:03Every time a language model writes a
  187. 7:05token, which is roughly a word, it has
  188. 7:07to pull its weights out of memory and
  189. 7:10into the chip. The weights are the
  190. 7:12billions of numbers that hold everything
  191. 7:14the model learned. And for a normal
  192. 7:16model, it has to read all of them for
  193. 7:18every single token.
  194. 7:20Now here's the problem. These nine cards
  195. 7:22are hanging off single lanes of PCI
  196. 7:24Express 2.0, which is a pretty slow
  197. 7:27link. So the software splits the job by
  198. 7:29layers. Card one gets the first chunk of
  199. 7:32layers, card two gets the next chunk,
  200. 7:34and so on down to card nine. Card one
  201. 7:37runs its layers, hands a small result to
  202. 7:39card two, and waits. Card two does its
  203. 7:42bit, hands off to card three, and waits.
  204. 7:45And it goes all the way down the line to
  205. 7:46card nine, and then the whole thing
  206. 7:49starts over at card one for the next
  207. 7:51token. So at any moment, one card is
  208. 7:53working and eight cards are waiting,
  209. 7:55which kind of defeats the point of nine
  210. 7:57cards, right?
  211. 7:59What this means is you get the memory of
  212. 8:00all nine cards added together, that big
  213. 8:0354, and the bandwidth of exactly one
  214. 8:06card, 192
  215. 8:09gigs a second. And that lets us do some
  216. 8:11really simple division because for a
  217. 8:13dense model, the kind where every
  218. 8:15parameter gets used on every token, you
  219. 8:18just take the model size divided by the
  220. 8:20bandwidth and that's your speed ceiling.
  221. 8:22So let's try the obvious test, the
  222. 8:24biggest model that fits.
  223. 8:26Llama 3.3 70B at 4-bit is about 42.5
  224. 8:30gigs. So it fits in 54 with room to
  225. 8:32spare. That's the dream the headline is
  226. 8:34selling, right? A 70 billion parameter
  227. 8:37model on $35 of hardware.
  228. 8:39But 42.5 divided by 192 gives you a
  229. 8:43ceiling of about 4.5 tokens a second and
  230. 8:46in the real world you'd see under three.
  231. 8:48So a 500 token answer, two or three
  232. 8:51paragraphs, takes almost three minutes.
  233. 8:53Three minutes for two paragraphs. I've
  234. 8:56chatted with people who type slower than
  235. 8:57that, but that's not the point. The
  236. 8:59point is the 54 gigs lets you load the
  237. 9:02model. It doesn't let you enjoy it. So
  238. 9:04how is he getting 30?
  239. 9:06The answer's in the model's name. He ran
  240. 9:08Quen 3.6 35B and the A3B is the bit that
  241. 9:12matters. This is a mixture of experts
  242. 9:15model or MOE and honestly I'd argue
  243. 9:18understanding MOE is the single most
  244. 9:20useful thing you can learn about local
  245. 9:22AI right now because it changes how you
  246. 9:24should think about hardware.
  247. 9:26A dense model like Llama 3.3 70B uses
  248. 9:29every parameter in every layer for every
  249. 9:32token. All 70 billion every time. It's
  250. 9:35like having 70 billion specialists and
  251. 9:38dragging every one of them into every
  252. 9:40meeting, even when most of them have
  253. 9:41nothing to say.
  254. 9:43A mixture of experts model still stores
  255. 9:45all its parameters, 35 billion here,
  256. 9:47which is the 35B in the name, but inside
  257. 9:50each of its 40 layers, instead of one
  258. 9:53big network, there are 256
  259. 9:56small specialist networks called
  260. 9:58experts. And a router, which is
  261. 10:00basically a tiny decision maker, looks
  262. 10:03at each token and switches on just eight
  263. 10:06of those 256
  264. 10:08plus one shared expert that always runs.
  265. 10:10So, per token, only about three billion
  266. 10:13parameters actually get read, and the
  267. 10:15other 32 billion just sit there in
  268. 10:17memory in case they're needed. It stores
  269. 10:1935 billion, but it only reads three,
  270. 10:22right? At four bit, that's roughly two
  271. 10:25gigs read per token instead of the 42
  272. 10:27you'd need for the dense 70B.
  273. 10:30And 192 gigs a second divided by two
  274. 10:33gigs a token gives you a ceiling of
  275. 10:35about 96 tokens a second. Wait. 96? He
  276. 10:40got 30. That's about a third of the
  277. 10:42ceiling, and we'll come back to where
  278. 10:43the rest went because it's actually good
  279. 10:45news for you. Either way, 30 tokens a
  280. 10:48second from $35 of hardware. The
  281. 10:51original write-up had a metaphor for
  282. 10:52this that I really like.
  283. 10:54The rig is a massive warehouse with one
  284. 10:57slow loading dock. If you order
  285. 10:59everything in the warehouse every day,
  286. 11:01that dock is your bottleneck. But this
  287. 11:03model only ever orders a few boxes at a
  288. 11:05time, so most of the warehouse just sits
  289. 11:07there, and a slow dock is fine when your
  290. 11:09orders are small. That's the whole
  291. 11:11trick. The model is cheap to read, so
  292. 11:13the slow dock barely matters.
  293. 11:15And when that clicked, I had a slightly
  294. 11:17uncomfortable realization because the 54
  295. 11:20gigs, the thing in the headline and in
  296. 11:22every single thumbnail, wasn't doing
  297. 11:24what I thought it was doing.
  298. 11:26He ran a 23 gig model on a 54 gig rig.
  299. 11:29That's 31 gigs, about five cards worth,
  300. 11:32holding nothing at all. So now flip it
  301. 11:35around. What if you don't have 54 gigs?
  302. 11:38What if you've got eight gigs of VRAM
  303. 11:40and 32 gigs of normal system RAM? Can a
  304. 11:4323 gig model run on that?
  305. 11:45This is where most of the videos on this
  306. 11:47rig buried the lid. Remember, the router
  307. 11:50picks eight experts out of 256. So in
  308. 11:53every layer, the other 248 experts are
  309. 11:56idle for that token. They don't need to
  310. 11:58live on the GPU, right? They just need
  311. 12:00to be reachable. And Llama CPP, the free
  312. 12:03open source engine that O Llama was
  313. 12:05built on and that sits under most local
  314. 12:07AI apps, has a flag for exactly this.
  315. 12:10It's called N CPU MOE, and it keeps the
  316. 12:13expert weights in your system RAM while
  317. 12:15the attention layers and the shared
  318. 12:17weights, the stuff that runs on every
  319. 12:19single token, stay on the graphics card.
  320. 12:21Per token, the router picks its eight
  321. 12:23experts. Your CPU runs those eight
  322. 12:26straight out of RAM. The card handles
  323. 12:28everything else and the other 248 never
  324. 12:31get touched. So an 8 gig card can run a
  325. 12:3423 gig model. And not just technically
  326. 12:37run it, run it fast. On May 11th, a
  327. 12:40developer called Martin Andrews posted
  328. 12:42his numbers. His machine was a GTX 1080
  329. 12:45with 8 gigs, so the same Pascal
  330. 12:47generation as the P106, a card he picked
  331. 12:50up second hand for under $200 sitting
  332. 12:52next to an Intel i7 6700 from 2015 and
  333. 12:5632 gigs of RAM. Same model, Qwen 3.6 35B
  334. 13:00at 4 bit with N CPU MOE set to 30. Over
  335. 13:03a 2,122 token answer, it wrote 24.94
  336. 13:08tokens a second. One card, 83% of the
  337. 13:11nine card rig. Then XDA developers ran
  338. 13:14it on a Lenovo Legion 5 gaming laptop
  339. 13:16with a mobile RTX 3070 and 8 gigs of
  340. 13:19VRAM. Without the flag, four tokens a
  341. 13:21second. With N CPU MOE at 28, 32.82,
  342. 13:26faster than the nine card rig on a
  343. 13:28laptop from one flag.
  344. 13:30Another developer tried a 12 gig RTX
  345. 13:323060 with 31 gigs of DDR4. Without the
  346. 13:36flag, it ran out of memory before it
  347. 13:37even finished loading. With the flag,
  348. 13:41on a slightly smaller three bit build.
  349. 13:44And at the top end, someone measured a
  350. 13:46single used RTX 3090 at 101.7
  351. 13:50tokens a second on the same model.
  352. 13:53That's 3.4 times the rig. So here's the
  353. 13:56leaderboard for the exact model in the
  354. 13:58headline. The RTX 3060, 23 tokens a
  355. 14:01second. The 2016 GTX 1080 desktop, 25.
  356. 14:05The $35 nine card rig, 30. The gaming
  357. 14:09laptop with a mobile 3070, 33. And
  358. 14:1213090, about 100. The rig lands right in
  359. 14:16middle of hardware most gamers already
  360. 14:18own.
  361. 14:19I had two feelings about this list at
  362. 14:20once, and you might too.
  363. 14:22Is this disappointing?
  364. 14:24Or is this kind of amazing?
  365. 14:25Honestly, it's both. If you've already
  366. 14:28got a gaming PC with 8 GB of VRAM and 32
  367. 14:31GB of RAM, you might already own this
  368. 14:33rig's performance, which is a bit
  369. 14:35deflating if you were excited about the
  370. 14:36rig.
  371. 14:37But a 2015 desktop with a 2016 graphics
  372. 14:40card running a 35 billion parameter
  373. 14:42model at 25 tokens a second is wild, and
  374. 14:46almost nobody knows the flag exists.
  375. 14:48Now, I've spent a few minutes tearing
  376. 14:50down the rig's main selling point, so
  377. 14:52let me be fair. The 54 GB isn't useless.
  378. 14:55It just isn't what made this thing fast.
  379. 14:57So what does it actually buy you? Three
  380. 15:00things that an 8 GB card plus 32 GB of
  381. 15:03RAM really can match.
  382. 15:05Number one is higher precision. The same
  383. 15:07Qwen model at 8-bit gives you noticeably
  384. 15:10better output, better reasoning, fewer
  385. 15:12weird hallucinations, and it needs about
  386. 15:1538 GB in total.
  387. 15:17On a 40 GB system that's also running an
  388. 15:19OS and a browser with 11 tabs open about
  389. 15:21why your llama CPP build failed,
  390. 15:24that doesn't fit. On the rig, it does
  391. 15:26with room to spare.
  392. 15:28Number two is long context, and this one
  393. 15:30surprised me.
  394. 15:31Every token you feed the model, every
  395. 15:33page of a document, every line of code,
  396. 15:37every message in the chat takes memory
  397. 15:39to hold while it's working. That's the
  398. 15:41KV cache, which is basically the model's
  399. 15:44short-term memory. So it grows with
  400. 15:46every token you add, right?
  401. 15:48On an 8 GB card, a long conversation
  402. 15:50eats into the room you need for the
  403. 15:52weights, and eventually stuff spills
  404. 15:54into RAM or just breaks. On the rig, all
  405. 15:56of it stays in fast VRAM.
  406. 15:59One developer estimated that a 65,000
  407. 16:01token context, think pasting a whole
  408. 16:04codebase into the chat, takes 10 to 15
  409. 16:07GB on top of the weights. So that 31 GB
  410. 16:10of empty VRAM suddenly very occupied. So
  411. 16:13if you do document analysis, long-form
  412. 16:16writing, code review across big repos,
  413. 16:19anything where you feed the model a lot,
  414. 16:21that spare VRAM is genuinely valuable.
  415. 16:24Number three is something the buyer
  416. 16:25mentioned himself, which is nine
  417. 16:27separate workers.
  418. 16:29Each P106 has 6 GB and reads at 192 GB a
  419. 16:33second when it isn't waiting on its
  420. 16:35neighbors. Split the work right, nine
  421. 16:37small jobs in parallel instead of one
  422. 16:39long chain, and you've got nine decent
  423. 16:41little workers going at once. That's
  424. 16:43less run the biggest model and more run
  425. 16:46lots of things at once, and his plan
  426. 16:48points that way. He wants to move the
  427. 16:49cards into a bigger 12-slot case with a
  428. 16:52proper CPU and add three more cards,
  429. 16:54which takes it to 72 GB. Those are real
  430. 16:56uses. They're not the use in the
  431. 16:58thumbnail, but they're not nothing.
  432. 17:00Okay, remember comment 47? Here's where
  433. 17:03I get nitpicky, and I'm not sorry,
  434. 17:05because this is what separates a useful
  435. 17:07tech video from a dopamine dispenser and
  436. 17:09a trench coat. Almost every video on
  437. 17:11this rig quoted the same two numbers,
  438. 17:14and nobody actually measured either of
  439. 17:16them. The first is about 400 tokens a
  440. 17:18second for reading your prompt, which is
  441. 17:20called prefill.
  442. 17:22It sounds plausible, right? Nine cards,
  443. 17:25decent raw compute, and prompt reading
  444. 17:27leans on raw maths more than bandwidth.
  445. 17:30But here's the problem. Pascal has no
  446. 17:32tensor cores, so the units that make
  447. 17:34modern cards blaze through that maths
  448. 17:36just aren't there. And nobody has
  449. 17:38measured prefill on this rig. It's a
  450. 17:40guess that turned into a fact by being
  451. 17:42repeated. I've seen it quoted in videos
  452. 17:44like it's settled. It isn't. The second
  453. 17:47is 150 W at idle. Measured? Nope. The
  454. 17:51one person in the thread who did
  455. 17:53measure, someone with two P102,
  456. 17:56the 10 GB cousin of the P106, reported 7
  457. 18:00W per card sitting idle, which times
  458. 18:02nine is 63 W, not 150. And that matters
  459. 18:07because power is the running cost that
  460. 18:09decides whether local AI actually saves
  461. 18:12you money, right?
  462. 18:14So let's do the real numbers. Nine cards
  463. 18:16at 7 W is 63 W. Leave it on for a month,
  464. 18:20which is 730 hours, and that's 46
  465. 18:23kilowatt hours. And at the US average
  466. 18:25home rate of about 18 cents, that's
  467. 18:26about $8.35 a month for the rig to sit
  468. 18:30there doing nothing. In California, it's
  469. 18:32closer to $14. Now add actual use. With
  470. 18:35the layer split running, one card works
  471. 18:38at up to 120 watts, while the other
  472. 18:40eight idle. So call it 176 watts for the
  473. 18:43cards while it's riding. And remember,
  474. 18:45that's a ceiling, not a measurement. At
  475. 18:4730 tokens a second, a million tokens
  476. 18:50takes about 9.26 hours, and 176 watts
  477. 18:54for 9.26 hours is 1.63 kilowatt hours,
  478. 18:58which works out to about 30 cents per
  479. 19:00million output tokens. So how does that
  480. 19:02compare to just paying someone? Open
  481. 19:05Router lists this exact model at about
  482. 19:0790 cents per million output tokens from
  483. 19:09most providers and as low as 10 cents
  484. 19:12from the cheapest one right now. So
  485. 19:13writing on the rig is cheaper if you're
  486. 19:15paying for power, right? 30 cents versus
  487. 19:1790. But here's the kicker, and the
  488. 19:19spreadsheet only shows it when you
  489. 19:20squint. That $8.35 idle bill at the US
  490. 19:24average buys you about 9.3 million
  491. 19:27output tokens at 90 cents a million.
  492. 19:30That's the rig writing nonstop,
  493. 19:32literally nonstop, for 86 hours a month
  494. 19:35just to break even on standby. A heavy
  495. 19:37chat user might put out a couple of
  496. 19:39million output tokens a month. At that
  497. 19:41rate, you'd spend $2 on a hosted API and
  498. 19:44over $8 keeping the rig awake. Not
  499. 19:47great. If you run agents all day, the
  500. 19:49rig writes for hours at a stretch, your
  501. 19:52power's cheap, and the math flips your
  502. 19:54way. If you use it for an hour in the
  503. 19:55evening and then go to bed, switch it
  504. 19:57off at the wall. And again, the 8 gig
  505. 20:00card in your desktop is already plugged
  506. 20:02in and already on your bill. It doesn't
  507. 20:04cost you anything extra to do AI work.
  508. 20:06Now you might be asking, Kai, why not
  509. 20:09just do what every other video said?
  510. 20:11Because almost all of them ended the
  511. 20:12same way. If you're not in Russia, buy a
  512. 20:15Tesla P100 or a P40 instead. Data center
  513. 20:19cards with more memory and similar cheap
  514. 20:21prices secondhand. Sounds reasonable,
  515. 20:23right?
  516. 20:24So here's what happened to those prices
  517. 20:26after the videos came out.
  518. 20:28According to GPU Dojo's price tracker,
  519. 20:30on August 28th, the cheapest 16 gig
  520. 20:33Tesla P100 was $79.97.
  521. 20:38By November 13th, about two and a half
  522. 20:40months later, it was $219.99.
  523. 20:45That's 2.75 times higher.
  524. 20:48The 24 gig P40 went from $239 to
  525. 20:51$383.90,
  526. 20:54up 61%.
  527. 20:56The 32 gig V100 went from $650 to
  528. 20:59$1,039.
  529. 21:01There's a comment with 179 likes that
  530. 21:04basically writes the epitaph for every
  531. 21:06cheap AI hardware video, and I'll
  532. 21:08paraphrase it.
  533. 21:09Every time a video shows cheap hardware
  534. 21:11running AI, that hardware stops being
  535. 21:14cheap. The P106 themselves are still
  536. 21:16cheap in the US, for now. Single cards
  537. 21:19list for about $20 to $60, and there's
  538. 21:22apparently a lot of 24 going for $499,
  539. 21:26which is about $21 a card. So nine of
  540. 21:29those is around $189 for the cards
  541. 21:32alone.
  542. 21:33But the board, the frame, nine PCIe
  543. 21:35risers, and power supplies that can
  544. 21:38handle a 1,080 watt ceiling are all
  545. 21:40extra.
  546. 21:41That's what the $35 hides. He bought a
  547. 21:44finished rig, right? You'd be building
  548. 21:46one, and piece by piece with new parts,
  549. 21:49you are nowhere near $35 anymore.
  550. 21:52So if you still want to do this, copy
  551. 21:54what the buyer did. Search your local
  552. 21:56listings for a whole mining farm, not
  553. 21:58single cards, because the frame and the
  554. 22:00power supplies are where a from scratch
  555. 22:02build gets expensive fast.
  556. 22:04And plan to run it headless. No display
  557. 22:06outputs means you need a board with
  558. 22:08built-in graphics or another machine to
  559. 22:10connect from. And remember, what could
  560. 22:13possibly go wrong? Here's the other
  561. 22:15thing. Nvidia has said the 580 driver
  562. 22:17branch is the last one that supports
  563. 22:19Pascal, which is the architecture these
  564. 22:21cards are built on. CUDA 13 has already
  565. 22:24dropped offline compilation for anything
  566. 22:26below compute capability 7.5 and the
  567. 22:28P106 is 6.1. Security updates for the
  568. 22:32580 branch run through October 2028
  569. 22:35according to Tom's Hardware. After that,
  570. 22:37nothing new. Today it mostly works fine.
  571. 22:39Olama ships a CUDA 12 runner that
  572. 22:41handles these cards and Lama CPP still
  573. 22:44builds for Pascal against CUDA 12. The
  574. 22:46community advice is basically to pin
  575. 22:48your driver, stop it auto updating,
  576. 22:51stick to the Q4 and Q8 quant formats
  577. 22:53that use the 8-bit integer path, and
  578. 22:56stay away from anything heavy on half
  579. 22:57precision maths because on Pascal half
  580. 23:00precision runs at 1/64th of full speed.
  581. 23:0464th. That's not a typo. So you've got
  582. 23:07roughly two years of runway with a fence
  583. 23:09at the end and after 2028 things get
  584. 23:12uncertain.
  585. 23:13The GTX 1080 from earlier is in exactly
  586. 23:15the same boat, Pascal being Pascal.
  587. 23:18For a $35 rig that paid for itself on
  588. 23:20day one, two more years is a great
  589. 23:22return, right? If you're building from
  590. 23:24scratch right now, factor in that
  591. 23:26timeline. So should you buy one? I want
  592. 23:28to give you a straight answer and not a
  593. 23:30hedge. There's one person this rig is
  594. 23:32exactly right for and it's the guy who
  595. 23:34bought it. Why him? His electricity is
  596. 23:37included in his rent and even if it
  597. 23:38weren't, he says power is cheap where he
  598. 23:41lives. P106 and their 10 gig cousins go
  599. 23:44for $10 to $20 locally. He wanted
  600. 23:47unlimited use, long context, and privacy
  601. 23:51and he doesn't want an AI agent reading
  602. 23:53his messages through someone else's
  603. 23:54server while he pays for the privilege.
  604. 23:56For him, almost every cost in this video
  605. 23:59is basically zero. A complete rig for
  606. 24:02$35, free power, room to run a bigger
  607. 24:05model at 8-bit with 65,000 tokens of
  608. 24:08context, all fully local. That's a great
  609. 24:11purchase. And the same logic works for
  610. 24:13you if all of this is true. You find a
  611. 24:15complete working rig near scrap price
  612. 24:18and I mean complete. So cards, frame,
  613. 24:21board, power supply, everything. Your
  614. 24:24power is cheap, included, or you'll run
  615. 24:26it enough to cover standby. You're fine
  616. 24:28on Linux with hardware that has no
  617. 24:30display outputs,
  618. 24:31and you specifically need what 54 gigs
  619. 24:33gives you, which is eight-bit precision,
  620. 24:36very long context, or lots of small jobs
  621. 24:39in parallel.
  622. 24:40If those don't all apply, your upgrade
  623. 24:42probably isn't a mining rig. It might be
  624. 24:44better RAM, it might be a machine where
  625. 24:46the CPU and GPU share one unified pool
  626. 24:49of memory, and there are some options
  627. 24:51going that way. Or it might just be
  628. 24:52turning on one flag and finding out you
  629. 24:55already have what you need.
  630. 24:56So, would running it locally have saved
  631. 24:59me that $140?
  632. 25:01Yes, just not on a mining rig.
  633. 25:03Don't buy a mining rig to run the model
  634. 25:05in this headline. Run it on what you've
  635. 25:07got first. So, what do you actually do
  636. 25:09tonight? Here are the steps so you can
  637. 25:11skip the three days of forum diving I
  638. 25:13did. If your desktop has an eight-gig
  639. 25:15NVIDIA card and 32 gigs of RAM, you
  640. 25:18probably already own the rig's 30 tokens
  641. 25:20a second, right?
  642. 25:22It's sitting right there, and the only
  643. 25:23thing missing is the flag. Step one,
  644. 25:26install llama cpp. Why not olama?
  645. 25:29Because n cpu moe is a llama cpp flag,
  646. 25:33so you want llama cpp directly. The XDA
  647. 25:35Developers article on this model is a
  648. 25:37solid reference because they ran it
  649. 25:39clean with and without the flag and
  650. 25:41published the numbers properly. Step
  651. 25:43two, grab a four-bit build of Quinn 3.6
  652. 25:4735 B. Unsloth's builds are the most
  653. 25:49popular and well maintained.
  654. 25:51The download's about 23 gigs, so clear
  655. 25:53some space and set n cpu moe to 30 to
  656. 25:57start. Step three, run llama bench,
  657. 26:00which is the benchmark tool built into
  658. 26:02llama cpp instead of a does this feel
  659. 26:05fast vibe check? Run it three times with
  660. 26:07the flag and three times without because
  661. 26:09one run can be noisy, and compare the
  662. 26:11generation tokens per second. Step four,
  663. 26:14if you're on olama and want to stay
  664. 26:16there, run olama ps before anything
  665. 26:19else. If it shows part of your model
  666. 26:21sitting on the CPU, you're already
  667. 26:23paying the speed cost of not using the
  668. 26:24flag, you just don't know it. Step five,
  669. 26:28still under 20 tokens a second and you
  670. 26:30think it should be
  671. 26:31Check your RAM before you look at
  672. 26:32graphics cards. The experts get read
  673. 26:34from system RAM, right? So if you're
  674. 26:36running one stick in single channel
  675. 26:38instead of two, you're leaving a lot of
  676. 26:40bandwidth on the table. Two sticks in
  677. 26:42dual channel on an older Intel platform
  678. 26:44can double it. That's a free upgrade if
  679. 26:46you've got a spare stick in a drawer.
  680. 26:48Only once you know what your current
  681. 26:49hardware actually does does it make
  682. 26:51sense to think about new hardware. And
  683. 26:53when you do, price it by what the model
  684. 26:55reads per token, not by what the card
  685. 26:57holds. That's the number that sets your
  686. 26:59speed.
  687. 27:00Which brings us back to the missing
  688. 27:02tokens. The math said 96 and he got 30.
  689. 27:06But look at what that 30 came off. A
  690. 27:082017 architecture, nine cards handing
  691. 27:11every token down a chain of single lane
  692. 27:13PCIe links, a Celeron with a few gigs of
  693. 27:16RAM, a default Ollama install booted
  694. 27:19from USB with zero tuning.
  695. 27:22My guess is those nine handoffs eat a
  696. 27:24big chunk of the gap, but nobody's
  697. 27:26tested that properly yet, so treat it as
  698. 27:29a guess. And that's the good news for
  699. 27:30you because your one card doesn't have
  700. 27:33nine handoffs, which is part of why a
  701. 27:34single GTX 1080 gets to 25. So the
  702. 27:38number that made this rig fast was the
  703. 27:40three, the 3 billion parameters it reads
  704. 27:43per token, never the 54. And I think
  705. 27:46that's the useful bit under all the
  706. 27:47hardware talk. Right now, almost every
  707. 27:49hardware conversation in AI is about how
  708. 27:51much you can hold. VRAM, parameters,
  709. 27:55context windows, gigabytes. Those
  710. 27:58numbers make great thumbnails, right?
  711. 28:00The question that actually decides fast
  712. 28:02or slow, cheap or expensive is how much
  713. 28:05you have to read. Mixture of experts is
  714. 28:07shaking up local AI precisely because it
  715. 28:10splits those two apart. So a 35 billion
  716. 28:12parameter model only reads 3 billion per
  717. 28:15token. Once that clicks, a lot of
  718. 28:17hardware you wrote off as two weeks
  719. 28:19starts looking a lot more capable. And
  720. 28:21my API build problem solved. I'm running
  721. 28:23my pipeline locally now on a machine I
  722. 28:25already owned. Not a flashy rig, not a
  723. 28:28$35 garage find, just a desktop with an
  724. 28:318 gig card, 32 gigs of RAM, and one flag
  725. 28:34in a config file. It does 25 tokens a
  726. 28:37second, which is the same as a nine
  727. 28:39card, 54 gig $35 mining rig. I did still
  728. 28:44make a spreadsheet about it. Old habits.
  729. 28:46Anyway, that's it for this one. If this
  730. 28:49helped you figure out your setup,
  731. 28:50subscribe because I do a lot of these
  732. 28:52where we go past the headline and find
  733. 28:54out what's real. And if you try this,
  734. 28:56drop a comment with your card and the
  735. 28:58number you get after you turn on the
  736. 29:00flag.
  737. 29:00I want to build a leaderboard out of
  738. 29:02real numbers instead of theoretical
  739. 29:04ones. See you in the next one.

About this transcript

This page contains the full transcript of 54GB VRAM For $35 The Cheapest Local AI Hack? by Kai, generated from the public captions YouTube serves with the video. The transcript has 5,105 words across 739 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.