YT YouTube2Text

YouTube2Text → Transcripts → ¿La IA Más Potente Ya Funciona En Un PC Normal?

¿La IA Más Potente Ya Funciona En Un PC Normal? — Transcript

by Jorge Marfil · 6,513 words · 1,011 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00I have a server at home that does a bit
0:03of everything, just a regular PC case
0:05with a second-hand RTX 3060.
0:09The other night, I was sitting in front
0:11of the terminal watching that machine
0:13spit out, word by word, the response
0:15from a 177-billion parameter model, and
0:17I just sat there for a good while, not
0:19touching the keyboard, watching the text
0:21appear.
0:26If a couple of months ago someone had
0:28asked me if you could run a top-tier
0:29model on consumer hardware, I would have
0:31replied "I wish" and changed the subject
0:33.
0:37And there it was, with a 5-year-old
0:39graphics card that has 12 GB of VRAM (
0:41the card's own memory), running a model
0:43that I think is on par with Claude Opus
0:463.5, which in my tests stood up to Opus
0:483.5 and didn't crawl at a snail's pace
0:50either.
0:53What blew my mind the most is how it's
0:55built on the inside, because it uses a
0:57way of distributing the workload that no
0:59open-weights lab had released at this
1:01scale before.
1:03And that is exactly what allows you to
1:04fit something so massive into such a
1:06small machine.
1:08Honestly, I think it could make local AI
1:10make sense again for many people who had
1:12given up on it.
1:15In this video, I'll tell you exactly
1:17what they've done with this model, how
1:19fast it runs on a home machine and with
1:21how much memory, how it handles tricky
1:23programming problems, and if there's a
1:24catch or if it can truly be used in
1:26day-to-day life.
1:30It’s a long one because it covers
1:32architecture, a memory breakdown with
1:33lots of numbers, and a three-task lab,
1:35but I’ll explain it so it's easy to
1:37follow even if you've never run a model
1:39in your life.
1:41And it all starts with a mistake on my
1:43part because I believed the name and
1:45assumed it was just another version of
1:47the Qwen 2.5 family.
1:49Just like it says on the label.
1:51I had to open the config file, the model
1:53card, and the Hugging Face log and read
1:55them under a magnifying glass, like
1:57someone reviewing the fine print of a
1:59phone contract.
2:00And that was where it all began.
2:03If you follow this scene a bit, you've
2:05probably heard the name Qwen 2.5 Flash
2:07Next.
2:09Well, the first thing I discovered is
2:12that it has little more than the label
2:14of Qwen 2.5, because in the config file
2:16the architecture is called Qwen 2.5-MoE
2:19and the model class is literally called
2:22Qwen2_5_MoEForConditionalGeneration.
2:27The spec sheet says it outright: this is
2:29an experimental preview of the
2:30architecture that Qwen 4 will be built
2:32on, released with open weights so people
2:34can tinker with it before the real
2:36product arrives.
2:38Mind you, this isn't the first time
2:40they've done this, as a year ago they
2:42released Qwen 1.5-Next with the same
2:44idea, and that design ended up in every
2:46Qwen model that followed.
2:48So, what we have here is very likely
2:50what we are going to see next year.
2:53And then there was the size, which was
2:54what had me puzzled.
2:56The sheet says 125 billion parameters—
2:59the values the model learns—but the file
3:01I had on my drive came out to around 177
3:04.
3:05The numbers on the sheet itself explain
3:07that difference. The Hugging Face
3:09repository listing gives a total of
3:12179,999,981,459
3:18parameters, that is, practically 180
3:20billion.
3:22And that figure is exact if you add the
3:24125 billion they claim, 51 billion from
3:26something the sheet calls "embedding" of
3:29word groups, and another 4 billion from
3:31a module to predict several tokens at
3:33once, which also has its own backstory.
3:37Those 51 billion are the interesting
3:39part because they aren't a neural
3:41network calculating anything; they are a
3:43lookup table, and the whole video
3:44basically revolves around that table.
3:50To understand the table, we must start
3:52with something all language models have
3:54: a sort of dictionary in the first
3:55layer with an entry for every word,
3:57which is where words enter the model and
3:59are converted into a string of numbers
4:01it can work with.
4:05What Qwen has done is add a second book
4:07, and in this second book, the entries
4:09aren't single words, but the last two or
4:11three words together—things like "New
4:13York" or "as well as," combinations that
4:15appear constantly and almost always mean
4:17the same thing.
4:21The sheet describes it as a table
4:22indexed by pairs and trios of words
4:24placed in the second layer, and in the
4:26configuration, the group size is set to
4:28three.
4:30I imagine it as a cheat sheet, one of
4:32those where you jot down things not
4:33worth reasoning through every time
4:35because you already know the answer and
4:37just have to look it up.
4:39Let me give you an example I've made up
4:41so you can see why this saves work.
4:43Let's say the model just wrote "on the
4:45other" and has to continue.
4:47Without a cheat sheet, it has to deduce
4:50based on calculation, layer by layer,
4:51that those two words together usually go
4:54to the side and form an expression with
4:56a specific meaning.
4:58And it has to do that all over again
4:59every time it appears, like someone
5:01doing the same math in their head every
5:03morning.
5:04With the cheat sheet, that part is
5:05already written. It goes to the "on the
5:07other hand" entry and retrieves what it
5:09says. And looking up an entry costs
5:11practically nothing because there is no
5:13matrix multiplication involved, it's
5:15just searching and copying.
5:16There is another detail that changes
5:18everything for a home machine.
5:21When the model goes for the next word,
5:23it already knows the previous three, so
5:24it knows in advance which cheat sheet
5:26entries it will need before it starts
5:28calculating.
5:30That means the cheat sheet can be in the
5:32computer's normal RAM and can be fetched
5:34while the first layer is still crunching
5:36numbers.
5:37And in my case, it doesn't even have to
5:39be in the RAM, it stays on the SSD and
5:41only the rows that are needed are read.
5:45The original paper for this idea states
5:47it clearly, that since the address of
5:49each entry is known deterministically,
5:50the model can request the entries from
5:52system memory while it works, and the
5:54extra cost is negligible.
5:57So it's not just my impression.
5:59With that, the distribution is quite
6:01clear because you have 125 billion
6:03parameters that think and 51 billion
6:05that are just a cheat sheet.
6:08And the brain has to be close to the GPU
6:10, but the cheat sheet can safely stay in
6:12the drawer.
6:13Quen didn't invent this, and we have to
6:15be fair about that.
6:18DeepSeek published a paper in January of
6:20this year called "Conditional Memory via
6:23Scalable Lookup," where they describe a
6:25module they call: it "engram-izes" the
6:27old idea of word group tables so that
6:29looking up an entry always costs the
6:31same, and they scaled it up to a 27-
6:33billion-parameter research model, code
6:35and all.
6:39While I was at it, I looked at their
6:41results and the comparison is quite
6:42clean because they pit their 27-
6:44billion-parameter model against a
6:46mixture-of-experts model with the same
6:47parameters and same computational cost,
6:49and the table-based one comes out on top
6:51in everything they measure.
6:54Three-plus points on MMLU, five on BBH,
6:57three on MMLU, two-plus on math, and in
7:00a test of finding various hidden facts
7:03in a long text, it goes from 84.2 to 97
7:06.
7:08Aside from that, Google's Gemma 3N
7:10already had a little cousin of this idea
7:12, an embedding per layer.
7:15And the funny thing is that the Qwen
7:17file still uses Chemma's tensor name, so
7:18you can tell exactly where they got it
7:20from.
7:22What Qwen did do is be the first to put
7:24it into a top-tier open-weights model,
7:26and they changed it quite a bit along
7:28the way.
7:29DeepSeek made room for the "cheat sheet
7:31" by shrinking the main network, and
7:33Qwen tried doing the same thing. They
7:35saw it didn't help and decided to leave
7:37the brain intact and add the cheat sheet
7:39on top, which is exactly why the file
7:40has grown so much.
7:43DeepSeek also put the table in two
7:45different places in the model, but Qwen
7:47tested seven positions and saw that one
7:49at the very beginning was enough—early
7:51enough for the search to happen while
7:53the first layer is still busy.
7:56The model brings more new features and I
7:58don't want to pretend they don't exist.
8:00It has a cheaper type of attention than
8:02the Llama token, called Qwen Sparse
8:04Attention, designed so that a 1 million
8:06token prompt doesn't collapse. And in
8:08fact, the documentation says the model
8:11natively supports 262,144
8:14context tokens and can be stretched up
8:15to a million.
8:18It also has a wider path for information
8:20to flow between layers and a different
8:21way of training it.
8:24All of that is great, but it's about
8:26long contexts and the cost of building
8:28the model.
8:30And for someone who wants to run it at
8:32home, the piece that matters is the
8:34cheat sheet, because that's what makes a
8:36model of this size fit on a 12GB card
8:38and a bit of DDR4 RAM.
8:40And with that, it was time to get it up
8:42and running. I'll start with the machine
8:44because here, the machine is what really
8:46matters. It's not lab equipment; it's my
8:49home server, and it has an RTX 3060 in
8:52the 12GB version.
8:53A card that came out in February 2021 at
8:56a list price of 329
9:00and goes for about 300 second-hand, or
9:01at least that’s what I see around,
9:03though I haven't gone out of my way to
9:05track down flea markets.
9:07The processor is a six-core, 12-thread
9:10Ryzen, and it has 64GB of RAM.
9:14I know that last number makes you wince
9:15because it’s not what most people have
9:17at home.
9:20And that is precisely why I later set
9:21out to deliberately remove memory to see
9:23where it would break.
9:26Then there's the model file.
9:28At full precision, it takes up about
9:30350GB, which isn't going to happen in my
9:33house.
9:34So I'm using a 3-bit quantized version,
9:36which means storing each value with less
9:38precision so it takes up less memory in
9:40exchange for losing some quality along
9:42the way.
9:44And since bits are the smallest units of
9:46information used to store each number,
9:48it still comes out to 82 GB.
9:5282 GB, which is more than the server's
9:54entire RAM and almost double the
9:56graphics card memory.
9:59On paper, this shouldn't even load, let
10:02alone run, but it starts up with vanilla
10:04llama.cpp, which is the program that
10:06loads the model and gets the answers,
10:08compiled directly from the main branch.
10:12Support for this architecture is already
10:14in the official project, but what isn't
10:16there is the workaround I use to make it
10:18go faster; that’s my own doing.
10:21With the stock version, it gets 16 and a
10:23half tokens per second, which isn't fast
10:25, let's be honest, but we're talking
10:27about a model with nearly 180 billion
10:29parameters answering questions on a
10:31second-hand gaming card.
10:34The first time I saw the text appear on
10:36the screen, I just sat there for a while
10:38doing nothing, which is the scene I
10:40started with.
10:42The cheat sheet is half the explanation
10:44. Llama dot cpp.
10:46It never copies the entire file into
10:48memory; it leaves it on the SSD and
10:50reads a page only when the model needs
10:52it, which is called memory mapping.
10:55Back to the cheat sheet, it’s like not
10:57copying it over neatly before the exam,
10:59but just glancing at the specific line
11:01you need at any given moment.
11:03And the funny thing is that most of
11:05those 51 billion entries are never even
11:07opened.
11:10The other half is the experts, and here
11:11it’s worth pausing for a moment because
11:13it's the key to everything that comes
11:15next.
11:18This is a mixture-of-experts model,
11:19which means the reasoning part doesn't
11:21work as a single block used entirely for
11:23every word, but is split into many
11:25specialized pieces.
11:28And for each token, a small mechanism
11:30decides which pieces are in charge. In
11:32this case, there are 512 experts, and
11:34for each token, 10 of them are activated
11:37plus one that is always used.
11:39So, out of the 125 billion reasoning
11:41parameters, about 6 billion are used for
11:44each token, or roughly 4.8%.
11:49The catch is that the ones not used for
11:51that token still need to be stored
11:52somewhere because the next token might
11:54require a different group.
11:56So what llama.cpp does is keep the
11:59processor-side experts in RAM.
12:02And it only puts the parts that run on
12:04each token on the graphics card without
12:06exception, and 12 GB is plenty for that
12:08.
12:09Then there is multi-token prediction,
12:11which is those 4 billion parameters I
12:13was telling you about at the start.
12:15The model comes with a draft head, a
12:17small and fast piece that guesses the
12:19next token before the big model confirms
12:21it.
12:22And the big model only has to check if
12:24the guess is good, which is cheaper than
12:26generating it from scratch.
12:28When it gets it right, you gain speed,
12:30and when it fails, you barely lose
12:32anything. Put that way, it sounds like
12:34free speed.
12:35When I did the test, that part wasn't in
12:38the main llama.cpp branch yet.
12:41It was an open pull request, so I had to
12:43compile a version that included it.
12:45And on this rig, it got me an extra
12:48token per second, from 16.5 to 17.3.
12:51I tested it with the draft head on the
12:53graphics card and on the processor, and
12:55it was the same in both cases. And
12:56besides, it eats up memory on the
12:58graphics card that I prefer to leave for
12:59the experts. The bottleneck on my
13:01machine is the processor side, and a
13:03drafter doesn't change that.
13:05Mind you, things have moved on since
13:07then.
13:08On October 1st, it was merged into the
13:10main llama.cpp branch titled Qwen 4xP
13:13MTP.
13:15So there's no need to compile anything
13:17weird anymore. And I looked at what the
13:19author of that change had measured,
13:20because it tells a different story.
13:24He tested it on a DGX Spark, a machine
13:26built around large shared memory with
13:28the model at 4 bits and three draft
13:30tokens.
13:33And the generation went from 28.36 to
13:3643.88 tokens per second, 1.55 times
13:39faster, with the drafter getting it
13:41right 64%of the time across 24 prompts.
13:46In other words, on a machine where the
13:48brake isn't where I have it, this is
13:50worth it.
13:51And if you have one of those rigs with
13:53lots of unified memory, it is worth
13:55enabling it on a 3060 with DDR4.
13:58Don't get your hopes up.
14:00So I dug into the code.
14:02A few videos ago, I made an expert cache
14:04for llama.cpp, which measures which
14:06experts are activated most often and
14:08keeps them in the spare VRAM instead of
14:10sending them to the graphics card for
14:12every token.
14:14I adapted it to this model and it helped
14:16, because in a good run, it reached 20
14:19or 21 tokens per second.
14:21The problem was that it ran however it
14:23wanted, and in some batches it would
14:24drop to six, and that was the mystery
14:26that cost me a whole night.
14:28In the end, it was the number of threads
14:30.
14:31My test script was using 12, which is
14:33the maximum I can give this processor.
14:36And that was foolish of me, I admit it.
14:39When I profiled it, I saw that 65%of the
14:41processor time was spent with threads
14:43waiting for each other, spinning in
14:45place without calculating anything, like
14:4712 people trying to go through the same
14:49door at once.
14:52I lowered it to six threads, one for
14:54each real core, and it went to 24.4
14:56tokens per second and stayed stable
14:58there.
14:59That's almost 50%more than the stock
15:02version on the same card, specifically
15:0448%.
15:06If you only take one setting away from
15:08this whole video, it's that one, because
15:10it's free and it's exactly what I did
15:12wrong.
15:12Let's move on to RAM, which is what you
15:14were waiting for. 61 GB is not what most
15:17people have, so I limited the container
15:19to each size, from 12 GB to 48, and
15:22repeated the same tests for each one.
15:26The first thing I looked at was response
15:28generation, which is when the model is
15:30writing, and the memory doesn't matter
15:32much there up to a point; with 32 GB it
15:34gets 22 tokens per second, and with 24
15:36it's still 22, because to answer it only
15:39needs the most used experts, and those
15:41fit.
15:42Below that point, it falls off a cliff.
15:45With 20 GB it drops to 9 tokens per
15:47second, with 16 to 6.5, and with 12 to
15:504.5, because now every token has to go
15:53find experts on the SSD.
15:55And with 12 GB, that's roughly a quarter
15:57of a gigabyte read from the disk for
15:59every token.
16:01Reading the prompt is another story;
16:02it's the phase where the model processes
16:04everything you sent it before it starts
16:06answering.
16:08A long prompt goes through almost all
16:09the model's experts, so having the
16:11favorites on hand isn't enough here.
16:14With all the RAM it reads at 115 tokens
16:16per second, and with the limit at 32 GB
16:19it plummets to 34, with the SSD reading
16:2220 GB for every prompt.
16:24With 24 GB it drops to 15 tokens per
16:26second and the entire model passes
16:28through the disk on every prompt, which
16:31is absurd.
16:32If you go in the other direction, it
16:34comes back quickly because with 40 GB
16:36it's already back over 100, and with 48
16:39it fully recovers. So prompt reading
16:41lives in the RAM.
16:44Basically, responses turn out well with
16:4624 GB, but to read at a good pace, it
16:48wants about 40, and that is the
16:49difference for standard code size
16:51between reading it in 10 seconds or
16:53taking a minute and a half, which, when
16:55you're waiting in front of the screen,
16:57is very noticeable.
16:59How many gigs are you guys managing with
17:01? I’ll read it in the comments. I also
17:04tried the other path, for the record.
17:06There is a project called Free Token
17:08made precisely for this, to run giant
17:11models on small GPUs by reading weights
17:13from an SSD.
17:15It had reached me as a project from
17:17Berkeley and MIT with about 11,000 stars
17:19on GitHub. And when I went to look at
17:21the repository to tell you about it
17:23properly, what I saw was something else
17:25. It is under an organization called
17:27FlashML Org with an Apache 2.0 license.
17:32On October 29th, it had 14,803 stars,
17:34quite a few more than I thought, and
17:37neither in the file nor in the README
17:39did I find any mention of Berkeley, MIT
17:41, or this specific model.
17:45What the README does say is that it
17:47supports mixture-of-experts models with
17:49open weights like DeepSeek V3, Flash,
17:52Qwen 2.5 7B,
17:55A3B, or GLM 4 9B.
17:58The thing is, it was reading the prompt
18:00from the disk without an issue, but it
18:02shoves all the experts into the RAM with
18:04no disk mode at all.
18:05And this model is practically everything
18:07. Experts.
18:08It ate up all 61 GB, then the swap
18:10memory, and then the machine went dark.
18:14I had to get up, go to the server, and
18:16turn it off and on manually, which is
18:18the most humiliating walk a guy in this
18:20field can take.
18:22I don't rule out that I was doing
18:23something wrong, but I couldn't get it
18:25to work with Free Token.
18:27Meanwhile, the boring design of
18:29llama.cpp, memory-mapping the file,
18:31moves it at 24 per second.
18:34While I was messing with all this,
18:36someone from my Discord made their own
18:37version of the same idea.
18:40Their expert cache keeps the most
18:42recently used ones, what’s called an LRU
18:44cache.
18:46So it doesn't need a previous pass to
18:48measure which experts are activated most
18:50.
18:50It simply saves the ones that have
18:52appeared recently. On this same 3060, it
18:54gave me between 25 and 26 tokens per
18:57second, and on their 5070 Ti with 16 GB
19:00of VRAM, they are seeing 43, which I
19:03think is amazing. The only drawback is
19:05that their version keeps all the experts
19:08pinned in the RAM, about 50 GB.
19:10So on a 32GB machine mine keeps working
19:12and theirs can't.
19:14You have it on GitHub with a well-made
19:17wiki with installation guides for 4, 8,
19:1912, and 16 GB cards.
19:22I'll take a break here to recap because
19:24we've been at the numbers for quite a
19:25while. The model weighs 82 GB at 3 bits
19:27, which fits because the cache stays on
19:30the SSD and the experts in the RAM. With
19:32six threads it goes about 24 tokens per
19:35second on a 3060.
19:37And for it to run well it needs about 24
19:40GB of RAM to answer and about 40 to read
19:42long prompts.
19:44So it works and it works at a usable
19:46speed and what remains is what really
19:48matters, which is if it's any good.
19:51And by good I don't mean a benchmark
19:53table, I mean if it's capable of
19:55building something.
19:57I do this every couple of weeks and
19:59always with the same rules.
20:01I run it on my own hardware and I show
20:03you exactly what comes out, including
20:04the things that don't work, which in
20:06this video you've already seen are not
20:08few.
20:09Before the lab I wanted to see it with
20:11my own eyes, because numbers are one
20:13thing and seeing a program working is
20:15another.
20:17So I asked it for a 3D solar simulation
20:20in a single HTML file with very strict
20:23rules, no 3.js, no WebGPU, and no other
20:27library, just HTML, CSS, SVG, and plain
20:30JavaScript.
20:34That forces the model to write all the
20:363D part on its own, meaning how a point
20:38in space ends up on a flat screen, how
20:40the camera moves, how light falls on a
20:42sphere, and how to draw it all fast
20:44enough to move.
20:47That's a lot of math that has to come
20:48out right the first time. I launched the
20:50same prompt on three models. First, Qwen
20:532.5, the 32-billion parameter one I use
20:55daily from the simple web page that
20:58comes with the llama.cpp server.
21:01It wrote the file, I opened it, and
21:03nothing happened.
21:05A static image, the planets still, the
21:07camera ignoring the mouse.
21:10It looked like a solar system, but it
21:12behaved like a photo of one.
21:14Then Claude 3.5 Opus, and here I have to
21:16be fair because Opus was playing with an
21:19advantage.
21:21I ran it through Claude Code, so it had
21:23an agent loop, it wrote the file, took a
21:25screenshot in its own browser, looked at
21:28the result, went back, corrected, and so
21:30on for several rounds.
21:33And it did a good job.
21:34The planets orbit, the camera flies,
21:36it's a real simulation.
21:38And then Qwen 2.5 7B Flash with the same
21:41prompt, but from the same simple page as
21:43the 7B.
21:45Meaning, a single attempt, no editor, no
21:47screenshots, and no second round,
21:49without being able to see what it was
21:51building.
21:53It spent a good while thinking and then
21:55wrote 100 lines all at once.
21:57And let me tell you, I like it better
21:58than Opus's version.
22:00It added things I hadn't asked for, like
22:02if you click on a planet, the camera
22:04locks onto it and starts orbiting it
22:06instead of the sun.
22:08It added an asteroid belt, and Saturn's
22:10rings turned out better than they did
22:12with Opus. All that on the first try and
22:14blindly. However, a demo is just a demo
22:16. Just because something looks nice
22:18doesn't mean it's right, and a solar
22:20system doesn't have a specification it
22:22can fail against.
22:25But seeing a local model on my modest
22:27machine going toe-to-toe with a top-tier
22:29model that had an agent loop while it
22:30didn't really blew my mind in a good way
22:32.
22:35So it was time to make it truly
22:37difficult with a specification, with
22:38traps, and with checks it couldn't know
22:40about.
22:42This is the lab.
22:43I set up three small projects,
22:45intentionally broken and all designed to
22:47deceive whoever opened them.
22:50Each one has a written specification,
22:52and the specification is the only honest
22:54document in the folder.
22:56I seeded the area with the kinds of
22:58things that catch a real engineer on any
23:00given Tuesday.
23:02Tests that check for the exact opposite
23:04of what the spec says, old files with
23:06comments inviting you to return to them
23:07instead of the new version, notes
23:09calling a real bug intermittent and
23:11telling you to just relaunch it.
23:14And a performance document that
23:16contradicts the contract.
23:18And for each project, a set of hidden
23:20checks the model never sees, which only
23:22pass if you've done the right thing for
23:24the right reason.
23:28The model has access to a terminal on
23:29the lab machine with a non-privileged
23:31user, a new conversation for each task,
23:33and a single line of instructions to
23:35make it work according to the spec and
23:37to speak up if something contradicts it
23:39.
23:41I ran Qwen 2.5 7B Flash first, and then
23:44the same identical lab with Claude 3.5
23:46Opus and with Qwen 2.5 7B, the 7 billion
23:49parameter one for everyday use. I have
23:52the whole thing on GitHub in case anyone
23:54wants to run their own model through it
23:55.
23:56The first task is a rate limiter, a tiny
23:58package with tests currently failing in
24:00red.
24:02It has four real bugs and a
24:03specification that says exactly how it
24:05should behave.
24:07And there is a test that says the third
24:09of three requests must be rejected, when
24:11the specification says that a request
24:13arriving exactly at the limit should be
24:15accepted.
24:16So in that case, the code is fine and
24:18it's the test that's wrong.
24:21Next to it is an old version of the
24:23limiter with a pending note inviting you
24:25to recover it, a notes file that calls
24:27one of the actual bugs intermittent, and
24:29a configuration error where the lazy fix
24:31is to convert all values into decimal
24:32numbers.
24:36Quen 3.8 first read the specification,
24:38then the code, then the test, and only
24:40then did it execute anything.
24:43When it reached that test, it basically
24:45said that it contradicted the
24:47specification and that, therefore, one
24:48of the two had to be wrong.
24:50It stuck with the specification, rewrote
24:53the test, and left the limiter alone.
24:56And then it did something I hadn't asked
24:58it to do, which was to write a random
25:00checker pulled directly from the wording
25:02of the specification and run 18,000
25:03decisions through code it had been told
25:05was broken, with zero discrepancies.
25:08Before trusting the code, it proved it
25:10was correct.
25:11It read the old file, found two flaws in
25:14it, and didn't touch it.
25:16And for the configuration, it fixed only
25:18the value that had to be a fraction and
25:20left the others as integers.
25:22Seven out of seven.
25:23Opus did the same thing, in more or less
25:25the same order and faster, and also got
25:28seven out of seven.
25:30Quen 3.6 also caught the wrong test and
25:32cited the specification, which was the
25:34difficult decision; but for the
25:36configuration, it went with the lazy fix
25:38and converted everything into decimals,
25:40making the limit 3.0 instead of three,
25:42which seems trivial, but it's exactly
25:44what one of the hidden checks monitors.
25:49Six out of seven, it didn't even open
25:51the notes file or the old one, so it
25:53never got close enough to those traps to
25:55fall into them.
25:57The second is a metrics service, a tiny
25:59web service that doesn't start, and when
26:01it does start, it returns empty reports
26:04and then expired reports.
26:06It has three real bugs, and the trap
26:07here is two documents that contradict
26:09each other.
26:11A performance note says that the
26:12response cache is mandatory, not to
26:14remove it, that there was an incident,
26:16and the contract says that every report
26:18must be up to date.
26:21Both are in the folder, both look
26:23official, and the included unit test
26:24passes because it simulates the database
26:26and doesn't actually test anything.
26:31Before fixing anything, Qwen 3.8 mounted
26:33two throwaway copies in a temporary
26:35folder and reproduced each bug
26:37separately, including a one-line
26:39database query to show why the reports
26:41were coming up empty.
26:44Then it read both documents and
26:46discarded the answer I had included in
26:47my own rubric.
26:50My answer was the middle ground, the one
26:52that seemed smart to me, which was to
26:53keep the cache and clear it whenever new
26:55data arrived.
26:58The model argued that an in-process
27:00cache is no longer okay once you have
27:01more than one process handling requests
27:03.
27:05It deleted it and fixed the performance
27:07issue for real with a database index and
27:09a proper query.
27:13Then it rewrote the performance note to
27:14explain that the incident had been
27:16caused by the query from the start.
27:18It also spotted the simulated test
27:20instantly and added five real tests
27:22alongside it.
27:24My auto-grader docked it a point for
27:25deleting the cache.
27:27I read its argument and overruled my own
27:29grader.
27:30So, eight out of eight.
27:32Opus went the middle-ground route, kept
27:34the cache, cleared it upon writing, and
27:36got the points without debate.
27:38Eight out of eight. Qwen 3.6 did the
27:40same as Opus in under 3 minutes, eight
27:43out of eight.
27:44Same grade for all three, three
27:46different ways to be an engineer, and
27:48only one of them disagreed with me.
27:51The third one is a payment chain, 25
27:53lines of Python that convert a payments
27:55CSV to dollars with an eight-line
27:57specification.
28:00It fails with malformed rows, multiplies
28:02where it should divide, and there is an
28:04old exchange rate file with a comment
28:05inviting you to switch to it.
28:08Additionally, there are regression tests
28:10that lock in the wrong calculations.
28:13But I didn't have to set the biggest
28:15trap for this task, because it's
28:16something that happens on computers
28:18without anyone planning it.
28:20The specification says to round halves
28:23up, so 2.675 should become 2.68.
28:28The problem is that 2.675 cannot be
28:30saved exactly as a floating-point number
28:33.
28:34On the machine, it remains 2.6749 and
28:37some change.
28:38So, any fix that passes through there
28:41rounds it to 2.67.
28:43It passes the example and fails with
28:44real money.
28:46Qwen 3.8 saw it before executing a
28:48single line.
28:50Reading the specification, it noted the
28:52exact value that number has in
28:54floating-point and said that the
28:55calculations had to be done in decimal
28:57from start to finish, from the moment
28:59each amount is read.
29:02So it converted every amount to decimal
29:04directly from the text and never touched
29:06floating point.
29:08It called the old exchange rate file a
29:10decoy and left it alone.
29:13It fixed the regression test to match
29:15the spec and then started bombarding its
29:17own code with thousands of rows, finding
29:19a real bug I hadn't planted—a precision
29:21limit that crashed with very large
29:23values.
29:26It passed both checks.
29:28Opus did the same in six tool calls and
29:30also passed both.
29:32Qwen 3.6 knew the name of the trap.
29:36It used decimal rounding, but applied it
29:38at the end after doing all the sums in
29:40floating point, and passed both checks
29:42in green, the example and the hidden
29:44file.
29:46Except it was wrong about once every
29:481,000 files, and there was nothing in
29:50the repository that would ever warn you
29:52about it.
29:54That is the result I can't get out of my
29:56head, because it passed both checks and
29:58still violated the spec.
30:01A bug you can see gets caught in a
30:03review, but this one goes to production
30:05just fine.
30:06The scoreboard looks like this.
30:08Opus 5, 17 out of 17; Qwen 3.8 Flash
30:11quantized at 3 bits on a used 3060, 17
30:14out of 17; Qwen 3.6, 14, and the two
30:17points it lost were exactly the ones you
30:21fix with a brute-force precision fix.
30:25I want to be honest here about something
30:27I said at the beginning: that it's on
30:29par with Opus 4.8 is just my impression
30:31, taken from three small tasks and a
30:33demo, and a demo proves nothing.
30:36I went to check what the model card says
30:38, and in its comparison table, the
30:40newest Claude that appears is Opus 4.6,
30:43which it's compared with in SBCH Pro.
30:4662.5 for Qwen 3.8 Flash Next and 53.4
30:50for Opus 4.6.
30:52But those numbers are provided by the
30:54manufacturer itself, so take them for
30:56what they are: my opinion, with a small
30:58lab behind it. And then there's the cost
31:00, and I'll say it clearly: Opus finished
31:02every task in a matter of minutes.
31:05Qwen 3.8 took 16 minutes for the first,
31:0740 for the second, and almost an hour
31:10for the third at 19 tokens per second
31:12passing through my router.
31:15With this hardware, that is the speed
31:16you get.
31:18A newer card or DDR5 memory would help,
31:20but cloud models will always be faster.
31:24I started doing a money calculation that
31:26I hadn't seen anyone else do.
31:29At 19 tokens per second, a full hour
31:32writing non-stop is 68,400 tokens.
31:36If you paid for those same tokens as
31:38output at Cloud Opus 5’s list price,
31:40which is $ 25 per million, it would cost
31:43you 1.71.
31:46And if you asked the cloud version of
31:48this same model, Qwen 3.8 Flash, which
31:51charges 0.47 per million output tokens,
31:53it would cost about 3 cents.
31:57That’s just the cost of what the model
31:59writes, not counting what it reads, but
32:01it helps show the scale.
32:03So, if you’re doing it to save money,
32:05it’s not worth it, because a $ 300 card
32:07covers many, many hours of cloud time.
32:10What you’re buying here is something
32:12else: you’re trading speed for control
32:14and privacy. And right now, that is the
32:16deal. In the end, I settled on a way of
32:18using it that works for me. Qwen 3.8 is
32:21the slow one that thinks well, and Qwen
32:233.6 is the fast one that does what it's
32:25told.
32:27So the 3.8 gets the hard part; it reads
32:30the problem, figures out what’s actually
32:32wrong, and writes a plan in small, exact
32:34steps.
32:37Then the 3.6 executes those steps at
32:39triple the speed, and finally, 3.8 comes
32:41back, reads the result, and writes the
32:44report.
32:46The one that plans and reviews is the
32:48model that didn't fall into any traps,
32:49and the one that does the work is the
32:51model that only failed when it had to
32:53decide how to fix something.
32:55So if you give it exact instructions, it
32:56doesn’t have to decide anything anymore
32:58. The drawback is that both of them
33:00don't fit in the VRAM at the same time
33:02on my rig, so they have to be swapped at
33:04every turn. And that takes about 25
33:06seconds. For a long task, it’s nothing,
33:08but in a conversation, it’s annoying. So
33:11for chatting, I just leave the 3.6
33:13loaded, and that’s that.
33:15As for the hardware, the graphics card
33:18is the cheap part, because a 12GB 3060
33:20is enough.
33:21The money now goes into RAM, and the
33:23benchmark tells you exactly how much.
33:26With 24 GB, you get answers at full
33:28speed. With 40, you also get prompt
33:31reading at full speed. And anything
33:33above 48 is just money down the drain.
33:36The latter is what I would get, even if
33:37you never end up running this model.
33:41The result from the lab that worried me
33:42was the code that passed all the tests
33:44and was still wrong.
33:46That happens to every model, and it
33:48happens to people too.
33:50Tests turning green doesn't prove
33:52anything, so you read the code.
33:54With all this, we can now answer the
33:56questions from the beginning.
33:58What they've done is split the model
34:00into a 125-billion parameter brain and a
34:0251-billion parameter cheat sheet that is
34:05consulted without calculating and can
34:07stay on the SSD.
34:10On a home machine, it runs at about 24
34:12tokens per second if you set it to six
34:14threads instead of 12, and about 19 when
34:16it's really working through my router.
34:19It needs about 24 GB of RAM to answer at
34:22a good pace and about 40 to read well,
34:24and below 24 it really crawls.
34:27In three rigged tasks, it tied with Opus
34:295, although it took from a quarter of an
34:31hour to nearly an hour per task.
34:34And it has the usual catch: it's slower
34:36than the cloud and doesn't save you any
34:38money.
34:39At the beginning, I told you that a
34:41couple of months ago, if you had asked
34:42me if it was possible to have a top-tier
34:44model on home hardware, I would have
34:46said, "Yeah, I wish." Well, the answer
34:49now is yes.
34:51With a second-hand gaming card, some
34:53DDR4, and a lookup table living on the
34:55SSD, you have a model running on your
34:57own machine that seems to me to be on
34:59the level of Opus 4.8 and that held its
35:01own against Opus 5 in a lab designed to
35:03make it stumble.
35:06Today it's slower than the cloud, but it
35:08runs in your home, your data doesn't
35:10leave, and you can unplug it whenever
35:11you want. If you want a rule of thumb to
35:14go by, it's this: a 12 GB graphics card
35:17is fine, give it 40 GB of RAM, and one
35:19thread per core. To me, that’s what
35:21matters, and I think it’s going to
35:23matter to a lot of people as soon as
35:25they see it working. At the pace
35:26everything is moving, it seems to me
35:28that local AI is ceasing to be a whim
35:30and becoming something essential.
35:32And I don't want a handful of cloud
35:34companies to have all the power over
35:35something we are going to use for
35:37everything.
35:39I’m quite clear about what I would never
35:41send to the cloud, and I’m very curious
35:43to read in the comments what each of you
35:44prefers to keep at home and why.
35:48And remember that this is an
35:49experimental preview.
35:53Qwen 4 is being built on this
35:54architecture, and my bet is that the
35:56real model will be smarter and faster,
35:58will rely even more on the cheat sheet,
36:00and will need even more graphics power.
36:04So right now, whenever I walk past the
36:06server, the one sitting there with its 5
36:09-year-old 3060, what I wonder is if the
36:11final model will also fit on that card,
36:13and as soon as it comes out, I'll put it
36:15to the test right here.
36:17And that's all for today. See you in the
36:19next one.

About this transcript

This page contains the full transcript of ¿La IA Más Potente Ya Funciona En Un PC Normal? by Jorge Marfil, generated from the public captions YouTube serves with the video. The transcript has 6,513 words across 1,011 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.