Mon IA locale va 11 fois plus vite (même PC) — Transcript
Full transcript
- 0:00Well, hi everyone, this is Nathan
- 0:01Innight. I hope you're all doing well.
- 0:04In this video, I'm going to show you how
- 0:06I managed to get a model with over 177
- 0:08billion parameters running at more than
- 0:1080 tokens per second on my small 16GB
- 0:12graphics card.
- 0:14This model is Qwen 3.8 Flash Next. At
- 0:17first, it was responding at 7 tokens per
- 0:19second. And today, it responds at over
- 0:2280 tokens per second. But between 7
- 0:25tokens and 80 tokens, there were 3
- 0:26months of struggle, mistakes, and
- 0:28pleasant surprises.
- 0:30My machine is a fairly standard server,
- 0:32with a 16GB RTX 5060 Ti. It's the one
- 0:35doing almost all the work. A second
- 0:37graphics card, an 8GB RTX 3070, which I
- 0:39initially thought would be very useful,
- 0:41but you'll see that's not necessarily
- 0:43the case. I also have a Ryzen 9 3900X
- 0:46and 46GB of DDR4 RAM. Remember that last
- 0:49figure well because it's what will
- 0:51decide everything. Very quickly, I run
- 0:53into a big problem.
- 0:55It's an MoE model, thousands of small
- 0:57experts, 512 experts per layer over more
- 1:00than 48 layers.
- 1:02For each token, only 10 of them work,
- 1:04but you never know which ones in advance
- 1:06. And all together, they weigh between
- 1:0840 and 55GB. They don't fit in my
- 1:11graphics card at all and barely fit in
- 1:13the VRAM. The whole challenge is putting
- 1:15the right experts in the right place. At
- 1:17the end of August, I started with
- 1:19llama.cpp, the engine everyone uses, and
- 1:21I got my first results. 7.9 tokens per
- 1:24second. It's really super slow. So, I
- 1:27tinker around, I put a few layers on the
- 1:29second card, and there, I reach 9.5
- 1:31tokens per second. I look at what's
- 1:33happening and I see that the processor
- 1:34is maxed out and my graphics cards are
- 1:36doing almost nothing. The real problem
- 1:38isn't power, it's the RAM.
- 1:41On September 10th, first bit of good
- 1:43news, I come across an engine with a
- 1:44super simple idea: keep the most used
- 1:46experts on the card.
- 1:48Hot experts stay in VRAM, the others
- 1:50wait on the side. I start this engine
- 1:53and there, 24.5 tokens per second. More
- 1:55than three times faster without changing
- 1:57any hardware. And first surprise, my
- 1:59second card is no longer of any use. So
- 2:02, I immediately want better quality. I
- 2:04take a larger quant, the Q4KM, and there
- 2:06, it's a trap. These experts exceed my
- 2:09RAM. The model takes a minute to read my
- 2:11question before writing the first word.
- 2:13But I'm not giving up. I lock the model
- 2:15in RAM and put nine layers in VRAM. The
- 2:18prefill goes from 31 to 206 tokens per
- 2:20second. The problem is solved. But I've
- 2:22understood the rules of the game. Here,
- 2:24everything must fit in RAM. At the end
- 2:27of September, I make a decision.
- 2:29I abandon llama.cpp completely and
- 2:30switch to an engine written specifically
- 2:32for this type of model.
- 2:34Its trick is enabling MTP. Guessing
- 2:37several tokens ahead, then verifying
- 2:39them all at once. The speed doubles. I
- 2:42reach 58 tokens per second. I keep
- 2:44improving, reaching 64 and then 70
- 2:47tokens per second. And the prefill even
- 2:50hits 1000 tokens per second.
- 2:52Then, one evening in October, I got
- 2:53really scared. I run one test too many,
- 2:56the RAM fills up completely, the server
- 2:58crashes, and I can't connect to it. And
- 3:00I can't go restart it manually because
- 3:02I'm not in the same place as the server
- 3:04. Since that day, I'm super careful, I
- 3:06don't run any tests without protection
- 3:08anymore. I set up a RAM limit, an
- 3:10automatic cutoff in case of issues, and
- 3:12an automatic reboot.
- 3:14At that moment, I think to myself, "
- 3:15What if I wrote my own engine, made
- 3:17specifically for this?" In one day, it
- 3:19goes from 29 to 63 tokens per second.
- 3:22It's super motivating, but it's still
- 3:24slower than the one I was already using
- 3:26. I set it aside, but I've understood
- 3:28where the time was being lost.
- 3:29At the end of October, I discover Basalt
- 3:31, an even newer engine. It shows a lot
- 3:34of promise. I install it, run it, and
- 3:36there, 57 tokens per second. Worse than
- 3:39before. I was really disappointed. But I
- 3:40can't believe it. I think there's a
- 3:42problem, so I look for where it's coming
- 3:44from. And there, I find 10 GB of RAM
- 3:46occupied for nothing, because of my old
- 3:48tests. I clean it up, run it again, and
- 3:50there, 94 tokens per second. And the
- 3:53prefill takes off, 2200 tokens per
- 3:55second.
- 3:56At the start of November, I want better
- 3:57quality. I try other quants and there,
- 4:00it's weird. Each one drops back to 63
- 4:02tokens per second. I think first it's
- 4:04the engine, then the machine, and I'm
- 4:05wrong both times, actually. The problem
- 4:07is one single layer out of 48 in a
- 4:09format that the fast mode doesn't
- 4:11recognize. Because of it, all the models
- 4:13switch to slow mode. I convert this
- 4:14layer, and there, I reach 83 tokens per
- 4:17second.
- 4:17One big question remains: which quant? I
- 4:20tried nine of them. The largest are too
- 4:22heavy; their experts don't fit in RAM.
- 4:24The surprise comes from the smallest
- 4:26ones. Unsloth's, the UDIQ2XS, keeps
- 4:28precision where it counts and hits 99.4
- 4:32%of the original quality. In perplexity
- 4:35, it gets 3.285 versus 3.273 for the
- 4:39original model. It's almost the same.
- 4:42Smaller and better than larger quants,
- 4:43this is the one I'm keeping. Alright,
- 4:46let's sum everything up. Here is the
- 4:47progress made from late August to early
- 4:50November. We see three major leaps. And
- 4:52each time, it wasn't the hardware that
- 4:54changed, just the engine. And my second
- 4:56card in all this, the RTX 3070, at first
- 4:59, it gave me 20%more speed. With the
- 5:01expert cache, nothing at all. With the
- 5:03specialized engine, 10 to 15%, and with
- 5:06Basalt, it slows everything down. The
- 5:08official version of Basalt even rejects
- 5:10it. Basalt is only compatible with cards
- 5:13starting with 50. The better the engine
- 5:15, the less useful it is. In the end, I
- 5:17won't be using this card.
- 5:18Once the model was well-tuned, I asked
- 5:20myself a question. What if it actually
- 5:22existed? I tried Swift, which thinks for
- 5:24less time. A month later, I went back to
- 5:26the original. Then, Edge Zero, which
- 5:28promises more speed with a new kind of
- 5:30expert. Converted to the classic format
- 5:32, it's fast, but its quality drops
- 5:33significantly. So, I modified the engine
- 5:35to read it without losing anything. It
- 5:37works, but it doesn't exceed 57 tokens
- 5:39per second. So, I deleted everything.
- 5:41I kept searching, digging through
- 5:43Hugging Face, and found about 1,600
- 5:45versions of this model.
- 5:47Almost all are simple quants or
- 5:49Uncensored versions, but one name stands
- 5:51out: Tinfield 1. A fine-tune made for
- 5:54code. He is a real competitor. I
- 5:56downloaded it, converted it, and
- 5:58organized a duel between the two. First
- 6:00round: speed. Tinfield wins, 80 tokens
- 6:03per second against 68.
- 6:05Second round, I asked each for a small
- 6:07application, and Tinfield's was cleaner
- 6:09. So, I told myself, that's it, I found
- 6:11something better, but I wanted to be
- 6:13sure. So, I had them work as standalone
- 6:15agents on 10 automatically graded tasks
- 6:18. On the six easy tasks, it was a
- 6:20perfect tie. On the four difficult tasks
- 6:22, Flash Next succeeded at all, while
- 6:23Tinfield got stuck on the hardest one.
- 6:25He rewrites his code twice without ever
- 6:27running his tests. Result: 9.5 out of 10
- 6:31for Flash Next and 8.95 for Tinfield.
- 6:34Faster, but less reliable. The champion
- 6:36keeps his spot.
- 6:38So, what did I learn in three? 1, the
- 6:40limit isn't the graphics card, it's the
- 6:42RAM. 2, the engine matters more than the
- 6:45hardware. It multiplies the speed by 10
- 6:47. 3, a well-made quant beats a bigger
- 6:50quant. 4, a slower second card can
- 6:52actually waste time. 5, impressions are
- 6:55deceiving. Before changing models, you
- 6:56have to measure.
- 6:58Today, my model runs at 83 tokens per
- 7:00second in decode and 2200 in predictive
- 7:02on a machine many people could have at
- 7:05home.
- 7:06I simply installed two 16GB sticks and
- 7:08two 8GB sticks of DDR4 in my PC, plus a
- 7:115060 TI graphics card, so the
- 7:13entry-level for the latest generation of
- 7:16GPUs.
- 7:17The same model, the same equipment, and
- 7:19yet 11 times faster than on day one. And
- 7:21I think the story isn't over yet. This
- 7:23was Antoine In-Line. See you next time.
About this transcript
This page contains the full transcript of Mon IA locale va 11 fois plus vite (même PC) by Nathan Inline, generated from the public captions YouTube serves with the video. The transcript has 1,422 words across 208 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.