YT YouTube2Text

YouTube2Text → Transcripts → Don't Buy a GPU Instead Build This 32GB Free AI Server Instead

Don't Buy a GPU Instead Build This 32GB Free AI Server Instead — Transcript

by Kai · 2,600 words · 377 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Okay, so picture this. It's 2:00 a.m.
0:03I'm sitting at my desk with my coffee
0:05gone cold and my laptop fan sounds like
0:07it is trying to take off from a runway
0:09all because I'm trying to run a halfway
0:11decent AI model locally. And it's not
0:14even working since it is putting out two
0:16tokens per second, which is not even a
0:18speed so much as a medical condition.
0:21And then I made the mistake of opening
0:22Twitter where someone was flexing their
0:24RTX 5090 setup with something like, "Oh,
0:27yeah, 200 tokens a second, silky smooth,
0:31totally worth the five grand."
0:33But five grand for a graphics card is
0:35real money that I just don't have
0:37sitting around to make an AI go brrr on
0:39my laptop. So, I went down the YouTube
0:41rabbit hole like any reasonable person
0:43does and that is where I found the
0:45video. You've probably seen it, too,
0:48since it's called build a free 32 GB AI
0:51server with two Tesla T4s, $0. It has
0:55260,000 views and the comments are going
0:57crazy with lines like "This changed my
1:00life. I can't believe this is free and
1:02my RTX 3090 is crying." And honestly, I
1:05believed that enough to spend an entire
1:07Sunday setting it up, getting it
1:09running, and feeling like a genius right
1:12before reality hit like a cold shower.
1:14Now, that is exactly what we are going
1:16to talk about today. There is a lot more
1:19to the story than the thumbnail tells us
1:21including something on that exact same
1:23free menu that literally nobody picked
1:26and it absolutely destroys the T4 setup.
1:29So, let's get into it. So, first, let's
1:32walk through what this whole free 32 gig
1:35AI server thing actually is in case you
1:37haven't seen those videos. There is a
1:39platform called Kaggle, which is
1:41Google's data science playground and
1:43basically where data nerds go to train
1:45models and compete in ML competitions.
1:48And Kaggle has this absolutely wild free
1:51tier where you verify a phone number,
1:53just a phone number, and get about 30
1:56GPU hours per week for free with no
1:59credit card, no trial, nothing. And one
2:02of the options on that menu is GPU T4
2:05X2, which is two Nvidia Tesla T4 cards
2:09at 16 gigs each. So, 32 gigs of VRAM
2:12total. Now, on that hardware, people
2:14have been running Qwen 3, specifically
2:16the 27 billion parameter version. And
2:18that is a serious model, Apache
2:20licensed, out since August, and
2:22downloaded 6.8 million times on Hugging
2:26Face in a single month. So, it can code,
2:29it can reason, and it is the real deal.
2:31The workflow this video show is to spin
2:33up the Kaggle notebook, download the
2:35model through Ollama, punch a hole
2:37through Cloudflare to make it public,
2:39and paste that URL into a coding agent
2:42like Cloud Code or Hermes. At that
2:44point, we have a free 32-gig AI server
2:46that we can point a whole workflow at.
2:49And the comparison they always make is
2:51free versus a $1,300 graphics card. So,
2:55free wins before the video even starts,
2:57right?
2:58That is also exactly where the story
3:00starts to get interesting. I remember
3:02watching the benchmark and Dev Zone X
3:04video, which showed 16 tokens per
3:06second, and thinking it wasn't blazing
3:08fast, but it was workable enough to ship
3:10code with. But once I actually started
3:12using it as a daily coding server,
3:15something started to feel off.
3:17Two GPUs sounds like two GPUs. So,
3:20wouldn't you expect double the power or
3:22at least twice the speed of one card?
3:25But that is not what actually happens
3:26here. The model file for Qwen 3 27B at
3:304-bit quantization is about 17 GB, while
3:33each T4 on Kaggle gives us 15.3 GB of
3:37usable VRAM. So, the file doesn't fit on
3:40one card, and what Ollama does instead
3:42is split the model across both cards by
3:45layers, where half the model lives on
3:47card one and half on card two. Now, to
3:49generate one token, card one reads its
3:52half of the weights while card two sits
3:54there doing nothing. Then it hands off
3:56the result and card two reads its half
3:59while card one sits there doing nothing
4:01and only then do we get one token. It's
4:04a relay race, not a team lift. Engineers
4:07call this a layer split, which means two
4:09T4s give us the capacity of two cards
4:12and the speed of exactly one. The T4
4:14itself launched in 2018 and it moves 320
4:18GB per second through its memory bus, so
4:21that is really our ceiling here and
4:23DevZoneX measured 16.23 tokens per
4:26second with multi-token prediction on
4:29and one commenter said they could not
4:30get past 18, which is not a coincidence,
4:33it is physics. Now, 16 to 18 tokens per
4:36second for writing code is survivable
4:39and I've used worse, but the part that
4:41absolutely destroyed me when I figured
4:43it out is the thing nobody in those
4:45tutorial videos mentions. And every time
4:47I'd fire up my coding agent and give it
4:49a task,
4:50there'd be this pause. At first I blamed
4:53the tunnel or maybe my internet being
4:55weird, so I refreshed and waited and
4:58nothing happened until the words
4:59suddenly showed up. What's happening
5:01during that silence is called prefill.
5:04Before a model writes a single word
5:06back, it has to read everything we sent
5:08it, which includes the message, the
5:10attached files, the previous
5:12conversation and the agent's system
5:15prompt. And the system prompts in tools
5:17like Claude Code are reportedly around
5:1930,000 tokens before anyone has even
5:21typed a single word. Reading is a
5:23different kind of work than writing,
5:25since it's math-heavy rather than
5:27memory-bound, and the T4 has 65
5:30teraflops of compute for this. So when
5:32we scale the numbers for a 27B model, we
5:34are looking at roughly 325 tokens per
5:37second during prefill. That puts a
5:3930,000 token agent prompt at about 92
5:42seconds of complete silence before the
5:44first word comes out. And the beautiful
5:46part is that Cloudflare's quick tunnel,
5:49the one every single one of these
5:51tutorials uses, automatically kills any
5:54request that hasn't responded in 100
5:56seconds. That is the timeout limit on
5:58their free tier, and it can't be changed
6:00below their enterprise plan. So, we've
6:02got 92 seconds of prefill and 100
6:05seconds before Cloudflare gives up and
6:07throws a 524 error, which leaves us 8
6:10seconds of margin. One of the creators,
6:12A Stud TY, literally says in their own
6:15video that the first message might time
6:17out and to just send it again. And if we
6:19scroll through the comments on these
6:20videos, there is a greatest hits
6:23collection of 502 errors, 504 errors,
6:27and provider unavailable. So, it is not
6:30actually broken. It's just math on a
6:32whiteboard that nobody drew for us.
6:34And this is where I had my own no
6:36moment, since I was actually planning to
6:38use this for my day-to-day work. I've
6:40been thinking about it as a GPU, when
6:42really it is a quota.
6:44We get about 30 GPU hours a week, and
6:47Kaggle caps a single session at 12
6:49hours. And if we walk away from the
6:51laptop and the session goes idle, which
6:53happens in as little as 20 minutes, the
6:55whole thing dies, including the model
6:57download, Ollama, the Cloudflare tunnel,
7:00and the URL we pasted into the agent.
7:02And the next time we start, we get a
7:04completely new URL, so the config needs
7:07updating again.
7:08Some people in the comments say each
7:10cold start costs them 15 minutes, and
7:12some say 30, and those minutes come out
7:14of a 30-hour weekly quota, too. So, I
7:17did the math the thumbnails don't want
7:18us to do. Now, 30 hours a week is
7:21108,000
7:23seconds. At 16.23 tokens per second, if
7:27the server was generating tokens
7:28literally every single second with no
7:31setup time, no idle time, and no reading
7:34time, it would produce about 1.75
7:37million tokens in a week. Now, that same
7:41model, Qwen 327B, is sold on OpenRouter,
7:45where about 20 different companies host
7:47it. And the average price customers
7:49actually paid is $2.46 per million
7:52output tokens, and 1.75 million tokens *
7:56$2.46
7:57is $4.31.
7:59So, our free server, at the absolute
8:01best case with every second perfectly
8:04optimized, is worth $4.31 a week. And
8:07the $4.31 version doesn't time out,
8:10doesn't need a phone number, and doesn't
8:12die when we close a tab. That is the
8:14comparison I really needed to see from
8:16the beginning. Now, this next part
8:19genuinely annoyed me when I discovered
8:21it, because it's sitting right there on
8:22the same page, and somehow nobody in the
8:25whole wave of 32 gig server videos
8:27picked it. If we scroll down on the
8:29Kaggle accelerator menu, below the T4
8:32pair, there is another free option that
8:34says TPU VM. That is eight of Google's
8:37own TPU chips with 16 gigs of HBM each,
8:41so 128 GB total. And each chip moves
8:44about 800 GB per second, 2 and 1/2 times
8:47the T4's memory bandwidth, with 197
8:51teraflops of 16-bit math per chip.
8:54Until August, there was actually a good
8:55reason nobody was using it for this,
8:58because Qwen-3 uses a type of attention
9:00that TPU software could not handle. But
9:03in August, vLLM's TPU backend added
9:06support for it in version 0.28.0.
9:10And shortly after, a project called
9:12Kaggle TPU Lab put everything together,
9:15and it is MIT licensed and sitting on
9:18GitHub with about 579 stars. What it
9:21runs is Qwen-327B
9:23at full 16-bit precision, not the 4-bit
9:26quantized version that fits on the T4s,
9:28but the full 55.6 GB model with the full
9:32262,000 token context window.
9:35And the numbers from the project's own
9:37benchmarks are kind of embarrassing for
9:39the T4 setup. Writing speed is around
9:42130 tokens per second against the T4
9:45pair 16, so that is eight times faster.
9:48Prefill, the reading speed, comes in at
9:51about 10,300 tokens per second against
9:54roughly 325 on the T4 pair, which makes
9:58it 30 times faster.
10:00So that 30,000 token agent prompt that
10:02takes 92 seconds of silence on the T4s
10:05takes about 3 seconds on the TPU. It
10:08also hands out an API key, so the
10:09endpoint is actually secured, and the
10:11whole thing is live about 22 minutes
10:13after pressing run. And it's the same
10:15free quota page, the same phone
10:18verification, and the same notebook
10:20format. Just a different drop-down menu
10:22option that literally nobody in 260,000
10:25views clicked. Now, to be fair, this is
10:28not about swapping one set of hype for
10:30another because both of these setups,
10:33the T4s and the TPU, come with real
10:35catches that nobody reads out in any of
10:38these tutorials. The first catch is
10:40Kaggle's acceptable use policy. The
10:42version in force since June 2025
10:45explicitly lists server farming and
10:48activity unrelated to ML data science as
10:51abuse of their resources. And when it
10:53comes to wiring a Cloudflare tunnel to a
10:55notebook and running a keep-alive script
10:57to dodge the idle timer, one Reddit
11:00builder who published exactly this setup
11:02called it a gray area. We could not find
11:05a documented ban, but it is still your
11:07account and it's still their rules.
11:09The second catch is that it is not
11:11local, even though every thumbnail says
11:14free local AI. The moment the code goes
11:16through a Cloudflare tunnel to Google
11:18servers, it stopped being local. And
11:20Cloudflare themselves say quick tunnels
11:22are for testing and development with no
11:25uptime guarantee. That matters because
11:28the one real reason to run AI locally is
11:31keeping code private and off someone
11:32else's machine. And that is exactly what
11:35this setup can't give us.
11:37One commenter summed it up perfectly
11:38with free local AI look inside VPS,
11:42which is really just a VPS with extra
11:44steps.
11:45The third catch is that the defaults on
11:47the T4 tutorials are genuinely kind of
11:50chaotic. Ollama has no login out of the
11:52box, and most tutorials put it behind a
11:55public link with no API key, so the URL
11:58is essentially the password. And the
12:00model most of them pull is an uncensored
12:02reupload from a single hugging face
12:04account rather than Qwen's official
12:06file. And the TPU project also wants a
12:09Kaggle API token to launch from a
12:11terminal, but it's open source, so we
12:13can read every line before handing that
12:15over, and we should.
12:17Now, I've been sitting on this for a
12:18while, and I think the honest answer is
12:21more nuanced than the thumbnail suggest.
12:23The free Kaggle setup, T4s or TPU, is
12:27genuinely perfect for learning. If the
12:29goal is understanding how an inference
12:31server works, how to split a model
12:33across cards, how to configure Ollama,
12:36and how to set up a tunnel, this is an
12:38amazing sandbox. We install the engine,
12:41watch things break, and fix them. And
12:43nothing we own gets hot, which is
12:45genuinely valuable. It is also what
12:47Kaggle was built for, which is training
12:49and evaluating models. Batch jobs,
12:52fine-tuning, running 500 prompts through
12:55a model for an eval, and saving the
12:57output aren't a gray area at all.
12:59Because that is the exact thing the
13:01platform is designed around. And for a
13:04student with no budget and no credit
13:05card, the TPU route gives something
13:08real. A strong coding model running at
13:10130 tokens a second, API key and all, 20
13:14hours a week, completely free. For a lot
13:17of people in a lot of places, that is
13:19not a rounding error, but a meaningful
13:21tool. But for a daily coding workflow
13:24that just works, where we open the
13:26laptop at 9:00 a.m. and the agent
13:27responds immediately, a full week of
13:30this free server is worth less than
13:32lunch. The hosted version of the same
13:34model, on Cerebras, for example, runs at
13:37618 tokens a second for $1.49
13:42per million tokens. At that price, the
13:4430-hour weekly Kaggle quota, converted
13:47to tokens, costs less than five bucks.
13:49So, buy $5 of credit and move on.
13:52And if the real reason for wanting local
13:54AI is that the code literally cannot
13:57leave the machine because of security,
13:59compliance, an NDA, whatever, then every
14:03option in this video is out, including
14:05the free ones. That is the actual reason
14:08to own hardware, and it's a completely
14:10legitimate reason. So, after all of
14:12this, where I actually landed is pretty
14:14simple. If you want free, skip the 2T4s,
14:17run the TPU recipe, and treat it like a
14:19lab you open for an afternoon rather
14:21than a server you leave running. If you
14:23code every day, point your harness at
14:25the hosted model because a full week of
14:28the free server is worth $4.31.
14:31That is two cups of coffee, and one of
14:33them will go cold while the agent prompt
14:35pre-fills. And buy the GPU the day your
14:38code is not allowed to leave your desk
14:40because until that day, free is just
14:42someone else's computer with a timer on
14:44it. If this broke the way you were
14:45thinking about this whole thing, then
14:47good. That was the point. Drop a comment
14:49if you've actually tried either setup
14:51because I want to know what your real
14:53numbers looked like. And if you haven't
14:54subscribed yet, that is fine, but you
14:56probably should since next time we are
14:58going to look at what actually makes
15:00sense for a coding workflow when you're
15:01not made of money. So, I'll see you
15:03then.

About this transcript

This page contains the full transcript of Don't Buy a GPU Instead Build This 32GB Free AI Server Instead by Kai, generated from the public captions YouTube serves with the video. The transcript has 2,600 words across 377 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.