YouTube2Text

Your Life at Every Level of Running AI Locally (Laptop to 4×3090) — Transcript

by DevPsyche · 2,407 words · 359 segments · language en · Watch on YouTube

Full transcript

  1. 0:00At some point, you had an electrician in
  2. 0:02your house. He was at your breaker panel
  3. 0:04with a voltage tester, and he asked the
  4. 0:06question electricians ask, which is,
  5. 0:09"What's it for?" And you had to decide,
  6. 0:11in your own hallway, how honest to be
  7. 0:14with a professional about why you needed
  8. 0:16a dedicated 20-amp circuit run to a
  9. 0:18spare room. You said it was basically a
  10. 0:21computer. He looked past you at four
  11. 0:24graphics cards bolted to an open frame
  12. 0:26on a shelf, humming with no case around
  13. 0:29them. He didn't say anything. That was
  14. 0:31its own kind of review. Because here's
  15. 0:34what nobody warns you about. The ceiling
  16. 0:36on this hobby isn't money, and it isn't
  17. 0:39skill. It's the copper in your walls.
  18. 0:42There's a hard limit on how much
  19. 0:44artificial intelligence you can install
  20. 0:46in a home, and the wiring sets it, not
  21. 0:48you. And all of it, the electrician, the
  22. 0:51frame, the 20 amps, started with a
  23. 0:54switch. You turned off the Wi-Fi just to
  24. 0:57check. And a thing sitting on your own
  25. 1:00hard drive, connected to nothing,
  26. 1:02answered you. Hold on to that flick,
  27. 1:05because the gesture never changes. Only
  28. 1:08the price does. Level zero. I ran it on
  29. 1:11my laptop, 10 minutes. That's the trap.
  30. 1:15You install Ollama, pull a 4-GB file,
  31. 1:18and there it is, producing words at
  32. 1:20about the speed of someone typing with
  33. 1:22one finger while thinking about
  34. 1:23something else. You showed one person.
  35. 1:26You were quite excited. They watched it
  36. 1:29crawl out a paragraph and said, kindly,
  37. 1:31"Isn't that just a worse ChatGPT?"
  38. 1:34And it is. On that laptop, in that
  39. 1:37month, it is a worse ChatGPT that also
  40. 1:40heats your legs and makes the fan sound
  41. 1:42like it's trying to leave the building.
  42. 1:44You had no answer. You changed the
  43. 1:47subject. Four out of five people stop
  44. 1:49here. They delete the folder and go back
  45. 1:52to the browser tab, and they're right
  46. 1:54to. You didn't. And the reason is the
  47. 1:57actual beginning. The question in your
  48. 1:59head changed shape. It stopped being, is
  49. 2:03this any good? And became, what would it
  50. 2:06take to make this good? The first is a
  51. 2:09review. The second is a project. And
  52. 2:12projects have budgets.
  53. 2:15>> Level one. I have a gaming GPU anyway.
  54. 2:18>> The card in your desktop plays games. It
  55. 2:21turns out it has a second job. And the
  56. 2:24second job pays in status. This is the
  57. 2:27largest doorway into the whole thing.
  58. 2:30Not researchers, not privacy obsessives,
  59. 2:33but gamers who found their toy was
  60. 2:35secretly industrial equipment. And this
  61. 2:38is where your mental model of computers
  62. 2:40gets rearranged. Because you arrive
  63. 2:42assuming the fast [music] card wins.
  64. 2:45Wrong axis. The card isn't an engine.
  65. 2:48It's a bucket. How much memory it holds
  66. 2:51decides whether a model can exist on it
  67. 2:54at all. How fast that memory moves
  68. 2:56decides how quickly it talks. Speed was
  69. 3:00never the constraint. Room was.
  70. 3:03You learn this the way everyone does.
  71. 3:05CUDA out of memory in a terminal at
  72. 3:0711:00 at night. It isn't really an
  73. 3:09error. It's a tape measure. Then the
  74. 3:13crueler lesson underneath. You abandon
  75. 3:16Ollama for llama.cpp,
  76. 3:18find a page of launch flags, and
  77. 3:20discover the same 8 GB card doing 10
  78. 3:24tokens a second will do 30 if you just
  79. 3:27tell it where to put things. You spent
  80. 3:30two weeks blaming silicon. It was a text
  81. 3:33file. And then you say it. Everyone says
  82. 3:36it. At every level, forever. I'm only 8
  83. 3:40gigs short. Nobody in the history of
  84. 3:43this hobby has ever been only 8 gigs
  85. 3:46short. Level two. The 24 GB threshold.
  86. 3:51You buy a graphics card from 2020 from a
  87. 3:54person you will never meet again.
  88. 3:56Sometimes that's a shipping label with
  89. 3:58no return address. Sometimes it's cash
  90. 4:01in a parking lot at a time you agreed
  91. 4:03over text and neither of you can work
  92. 4:05out why this feels illegal. $1,200
  93. 4:09and 2 years ago it was half that. A
  94. 4:126-year-old consumer product is
  95. 4:14appreciating, not because gamers want
  96. 4:16it, but because several hundred thousand
  97. 4:19people are trying to fit a mind into a
  98. 4:21closet and a used 3090 is the cheapest
  99. 4:24honest way to buy 24 GB. It arrives and
  100. 4:28you immediately take it apart because
  101. 4:30there's a decent chance it spent 2021 in
  102. 4:33a warehouse mining cryptocurrency and
  103. 4:35its thermal pads are now the consistency
  104. 4:37of old chewing gum. You replace them
  105. 4:40yourself on a $1,200 card from a
  106. 4:43stranger. This feels normal to you now.
  107. 4:46But the real shift isn't technical.
  108. 4:48Everything before was repurposing a
  109. 4:51laptop you had, a card you already
  110. 4:53owned. This is the first object that
  111. 4:56entered your home because of this and
  112. 4:58when somebody asks what it's for, you're
  113. 5:01going to have to say a full sentence.
  114. 5:03You tell yourself it holds its value,
  115. 5:06that you'll sell it on. Technically
  116. 5:08true. You won't. And then almost
  117. 5:11immediately the ceiling. Llama 70B at
  118. 5:144-bit needs about 42 GB. You have 24. So
  119. 5:19you settle for something smaller and
  120. 5:21it's genuinely good and you're genuinely
  121. 5:23happy and there is now a number on a
  122. 5:26screen telling you precisely what you
  123. 5:28can't have. Level three, two cards, the
  124. 5:32Rubicon.
  125. 5:33The second card arrives months later and
  126. 5:35was obviously always going to. 48 GB and
  127. 5:40for the first time the thing on your
  128. 5:41desk isn't a demonstration, it's a
  129. 5:44collie. Slower than the cloud, worse at
  130. 5:47hard problems, but holding a real
  131. 5:49conversation without anyone's
  132. 5:50permission. The case stops closing
  133. 5:53around now.
  134. 5:54>> [music]
  135. 5:54>> There's a riser cable. There may be a
  136. 5:56second power supply resting on a book.
  137. 5:59It has stopped being a computer in the
  138. 6:01way a thing with a lid is a computer and
  139. 6:03become equipment. Then, on a Tuesday in
  140. 6:06April, somebody you live with comes in
  141. 6:08looking for a phone charger. They stop
  142. 6:10in the doorway. They don't say anything
  143. 6:12for a second. Then, "Why hot in here?"
  144. 6:16And you say the fans are fine, which is
  145. 6:18true, and they say it's like a car in
  146. 6:20here, which is also true, and they take
  147. 6:22the charger and leave. No argument.
  148. 6:25Nothing happens. But you sit there
  149. 6:27afterwards understanding that a file has
  150. 6:29been opened on you, and every decision
  151. 6:32you make about this now has a second
  152. 6:34person in it. So you find the trick, and
  153. 6:36you will tell people about the trick for
  154. 6:38years.
  155. 6:39Dial the cards from 350 W down to 220,
  156. 6:43and you lose almost nothing. 39 tokens a
  157. 6:46second instead of 44. 90% of the
  158. 6:49performance for a third less heat.
  159. 6:52Everyone who finds this believes they
  160. 6:53discovered it. Every one of them is
  161. 6:55doing it because of the doorway. There's
  162. 6:58a sideways move here, and some people
  163. 7:00take it. Instead of a third card, you
  164. 7:02buy the quiet box, a Mac Studio, or that
  165. 7:05$4,000 Nvidia brick that draws less than
  166. 7:08a hairdryer and sits on a desk looking
  167. 7:10like a router. Silent, small, and
  168. 7:12requiring no conversation with anyone
  169. 7:14you live with. It has one flaw, the
  170. 7:17exact inverse of level one. Room is not
  171. 7:20the same as speed. Hand one an
  172. 7:238,000-word prompt, and it can take 14
  173. 7:26minutes and 48 seconds to produce a
  174. 7:28single character. It works. It
  175. 7:30absolutely works. It works at the speed
  176. 7:34of mail. So the advice from everyone
  177. 7:36further up is to stop here or step
  178. 7:38sideways. Two cards is easy to cool,
  179. 7:42easy to power, and does most of what
  180. 7:44you'd realistically ask. Nobody stops
  181. 7:47here. Level four, four times 3090, the
  182. 7:51summit. Four cards, 96 GB, $3,500
  183. 7:56in graphics alone. Realistically, 8 to
  184. 7:5910,000 once you've bought a server board
  185. 8:01with enough lanes to feed them. The
  186. 8:03first surprise is geometric. Four modern
  187. 8:06cards do not fit inside anything sold as
  188. 8:09a computer case. So, you buy an open
  189. 8:11frame built for mining, and your
  190. 8:13artificial intelligence now lives on a
  191. 8:15shelf in the open, like a set of
  192. 8:17weights. The second surprise is that
  193. 8:19cooling kills more of these builds than
  194. 8:21power does. Everyone plans the
  195. 8:23electricity. Nobody plans the air. Two
  196. 8:27cards made a room warm. Four cards make
  197. 8:30it a different room. You have installed,
  198. 8:33permanently, roughly the heat of a space
  199. 8:35heater you are not allowed to switch off
  200. 8:37because it's thinking. In January, this
  201. 8:40is a feature, and you will tell people
  202. 8:42it's a feature. By June, it's a room you
  203. 8:44enter briefly and at speed, like a sauna
  204. 8:47you're annoyed with. And then the
  205. 8:49arithmetic that put a man in your
  206. 8:51hallway. Four cards at stock pull
  207. 8:54roughly 2,000 W. One builder in an
  208. 8:57apartment ran those numbers, looked at
  209. 8:59his wiring, and stopped at two because
  210. 9:022,000 W is well past what his flat could
  211. 9:05tolerate while somebody was also running
  212. 9:07the dishwasher and the kettle. Read that
  213. 9:10again. The frontier of consumer
  214. 9:12artificial intelligence in this decade,
  215. 9:15in a wealthy country, is capped by a
  216. 9:18kettle. At sustained draw, that's
  217. 9:20something like $300 a month in
  218. 9:23electricity, a number you will mention
  219. 9:25to nobody, and which you have absolutely
  220. 9:27calculated. Then there's what you were
  221. 9:30promised and don't get. 96 GB is not one
  222. 9:34pool. How much you can use depends on
  223. 9:36the software, the wiring between cards,
  224. 9:39and the model's shape. And VLLM, the
  225. 9:43serving engine half this hobby runs on,
  226. 9:45quietly recommends in its own docs that
  227. 9:48you use a single GPU whenever the model
  228. 9:50fits on one. Somewhere in here, a used
  229. 9:53server board will decline to turn on.
  230. 9:56Not fail, decline. So will its
  231. 9:59replacement. One builder lost three
  232. 10:01weekends and two boards before changing
  233. 10:03brands. There's a detail in all this
  234. 10:06that says everything. Builders choose
  235. 10:08parts so the thing reads as a slightly
  236. 10:11oversized gaming PC rather than what it
  237. 10:14is, because it has to pass as normal to
  238. 10:16visitors. You are camouflaging your
  239. 10:18hobby inside a different hobby. That
  240. 10:21camouflage is the whole problem,
  241. 10:23actually. There's no uniform for this
  242. 10:25one. Nothing you can wear that explains
  243. 10:27to a stranger what's humming behind a
  244. 10:29door in your house. So we made a hoodie
  245. 10:32about it at shop.devpsych.com
  246. 10:35on the theory that it's the closest
  247. 10:36thing this has to a membership card and
  248. 10:39dramatically cheaper than another
  249. 10:40graphics card. Around now the machine
  250. 10:43begins to move. Desk, under desk,
  251. 10:46closet, garage. Every level up pushes it
  252. 10:50one room further from where people live.
  253. 10:52You tell people it's for fine-tuning.
  254. 10:54You have fine-tuned one thing. It was in
  255. 10:57March. Level five. You have left the
  256. 11:00hobby. $10,000,
  257. 11:03one card, 96 GB of error-correcting
  258. 11:06memory, 600 W.
  259. 11:09There's no build story here because
  260. 11:11there's no build. You bought a solution.
  261. 11:14The project is opening a box. And
  262. 11:17something quietly ends. A friend's
  263. 11:19company started routing work through
  264. 11:21your machine because it was there and it
  265. 11:23was free and you offered. There is now a
  266. 11:25Slack channel with four people in it
  267. 11:27where somebody occasionally types, "Is
  268. 11:29it up?" So you stop taking the side
  269. 11:32panel
  270. 11:33You stop reading release threads at
  271. 11:34midnight. You stop trying the
  272. 11:36interesting new model because it might
  273. 11:38break the thing four people are typing
  274. 11:40into.
  275. 11:41The tell is linguistic. The day you say
  276. 11:44the cluster instead of [music] my rig,
  277. 11:47something has finished. Fun's over.
  278. 11:50Uptime has begun.
  279. 11:52Here's the part nobody prepares you for.
  280. 11:55The problem was never the model. The
  281. 11:58problem was having a problem.
  282. 12:00And the moment you can bypass every
  283. 12:02constraint at once, the part you enjoyed
  284. 12:05evaporates. People at this level talk
  285. 12:08about their old two-card setup the way
  286. 12:10other men talk about a car they sold.
  287. 12:13Level six, the way back down.
  288. 12:16So, you sell it. The listing goes up on
  289. 12:18a Sunday. Four times RTX 3090, open
  290. 12:21frame, all tested, buyer collects.
  291. 12:25You light and photograph it more
  292. 12:26carefully than you have ever
  293. 12:28photographed a member of your own
  294. 12:30family. A man drives three hours to
  295. 12:32collect it. He's at level three. He's
  296. 12:35thrilled. He asks questions in the
  297. 12:37driveway that you answered for yourself
  298. 12:39two years ago, gets them answered in 40
  299. 12:41seconds, and hears none of it because
  300. 12:44nobody does.
  301. 12:46Watching him load it into a hatchback is
  302. 12:48like watching yourself leave.
  303. 12:50Then the room goes cold. The bill drops
  304. 12:53by 2/3.
  305. 12:54The frame is gone, and the shelf is just
  306. 12:57a shelf.
  307. 12:58And here's the part that stings. You use
  308. 13:01it more now because while you were
  309. 13:03climbing, the models came down to meet
  310. 13:05you. What needed 96 GB in 2024 fits in
  311. 13:0924 today and runs faster. You didn't
  312. 13:13downgrade. You waited expensively in a
  313. 13:16hot room for the software to catch up.
  314. 13:19You ask it normal questions now in a
  315. 13:21room at a normal temperature with nobody
  316. 13:24standing in the doorway, and you write
  317. 13:26it up and post it, and it does numbers
  318. 13:29because it gets read at 1:00 in the
  319. 13:30morning by people sitting next to a
  320. 13:32machine they've become slightly afraid
  321. 13:34of. The uncomfortable truth.
  322. 13:37Here's the confession nobody posts.
  323. 13:39Almost everyone at every level of this
  324. 13:42is hybrid. Local for the bulk work, the
  325. 13:45private documents, the overnight jobs,
  326. 13:48the things you'd rather not upload. The
  327. 13:50genuinely hard thinking still quietly
  328. 13:53goes to the cloud. Almost nobody leads
  329. 13:55with that because the identity of the
  330. 13:57thing is independence, and the honest
  331. 14:00version is mostly independence. So, why
  332. 14:03do 800,000 people keep going?
  333. 14:07In January 2024, a man named Tom
  334. 14:10Jobbins, who everyone knew as The Bloke,
  335. 14:13disappeared. For a year he had taken
  336. 14:15nearly every significant new model,
  337. 14:18converted it into a file ordinary people
  338. 14:20could actually run, and put it on
  339. 14:22Hugging Face for nothing.
  340. 14:24He is the reason this hobby has
  341. 14:26civilians in it. Then, he stopped. No
  342. 14:29announcement, no reason, no goodbye. And
  343. 14:33nobody really knows why. The community
  344. 14:36didn't collapse. Within weeks, other
  345. 14:38volunteers, Bartowski, Em Rademaker, a
  346. 14:41dozen more, had filled the gap. And
  347. 14:44they're still there tonight converting
  348. 14:46files for strangers, unpaid.
  349. 14:49That's what you joined when you handed
  350. 14:50cash to a stranger in a parking lot. Not
  351. 14:53a product, a relay.
  352. 14:55Because everyone else is renting their
  353. 14:57thinking from about five companies on
  354. 14:59terms that can be rewritten in a blog
  355. 15:01post with a model that can be retired on
  356. 15:04a Tuesday. And you never wanted a better
  357. 15:07model than theirs. You wanted to be the
  358. 15:10only person who could switch yours off.
  359. 15:13So, go on. Turn off the Wi-Fi.

About this transcript

This page contains the full transcript of Your Life at Every Level of Running AI Locally (Laptop to 4×3090) by DevPsyche, generated from the public captions YouTube serves with the video. The transcript has 2,407 words across 359 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.