YouTube2Text

Mon IA locale va 11 fois plus vite (même PC) — Transcript

by Nathan Inline · 1,422 words · 208 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Well, hi everyone, this is Nathan
  2. 0:01Innight. I hope you're all doing well.
  3. 0:04In this video, I'm going to show you how
  4. 0:06I managed to get a model with over 177
  5. 0:08billion parameters running at more than
  6. 0:1080 tokens per second on my small 16GB
  7. 0:12graphics card.
  8. 0:14This model is Qwen 3.8 Flash Next. At
  9. 0:17first, it was responding at 7 tokens per
  10. 0:19second. And today, it responds at over
  11. 0:2280 tokens per second. But between 7
  12. 0:25tokens and 80 tokens, there were 3
  13. 0:26months of struggle, mistakes, and
  14. 0:28pleasant surprises.
  15. 0:30My machine is a fairly standard server,
  16. 0:32with a 16GB RTX 5060 Ti. It's the one
  17. 0:35doing almost all the work. A second
  18. 0:37graphics card, an 8GB RTX 3070, which I
  19. 0:39initially thought would be very useful,
  20. 0:41but you'll see that's not necessarily
  21. 0:43the case. I also have a Ryzen 9 3900X
  22. 0:46and 46GB of DDR4 RAM. Remember that last
  23. 0:49figure well because it's what will
  24. 0:51decide everything. Very quickly, I run
  25. 0:53into a big problem.
  26. 0:55It's an MoE model, thousands of small
  27. 0:57experts, 512 experts per layer over more
  28. 1:00than 48 layers.
  29. 1:02For each token, only 10 of them work,
  30. 1:04but you never know which ones in advance
  31. 1:06. And all together, they weigh between
  32. 1:0840 and 55GB. They don't fit in my
  33. 1:11graphics card at all and barely fit in
  34. 1:13the VRAM. The whole challenge is putting
  35. 1:15the right experts in the right place. At
  36. 1:17the end of August, I started with
  37. 1:19llama.cpp, the engine everyone uses, and
  38. 1:21I got my first results. 7.9 tokens per
  39. 1:24second. It's really super slow. So, I
  40. 1:27tinker around, I put a few layers on the
  41. 1:29second card, and there, I reach 9.5
  42. 1:31tokens per second. I look at what's
  43. 1:33happening and I see that the processor
  44. 1:34is maxed out and my graphics cards are
  45. 1:36doing almost nothing. The real problem
  46. 1:38isn't power, it's the RAM.
  47. 1:41On September 10th, first bit of good
  48. 1:43news, I come across an engine with a
  49. 1:44super simple idea: keep the most used
  50. 1:46experts on the card.
  51. 1:48Hot experts stay in VRAM, the others
  52. 1:50wait on the side. I start this engine
  53. 1:53and there, 24.5 tokens per second. More
  54. 1:55than three times faster without changing
  55. 1:57any hardware. And first surprise, my
  56. 1:59second card is no longer of any use. So
  57. 2:02, I immediately want better quality. I
  58. 2:04take a larger quant, the Q4KM, and there
  59. 2:06, it's a trap. These experts exceed my
  60. 2:09RAM. The model takes a minute to read my
  61. 2:11question before writing the first word.
  62. 2:13But I'm not giving up. I lock the model
  63. 2:15in RAM and put nine layers in VRAM. The
  64. 2:18prefill goes from 31 to 206 tokens per
  65. 2:20second. The problem is solved. But I've
  66. 2:22understood the rules of the game. Here,
  67. 2:24everything must fit in RAM. At the end
  68. 2:27of September, I make a decision.
  69. 2:29I abandon llama.cpp completely and
  70. 2:30switch to an engine written specifically
  71. 2:32for this type of model.
  72. 2:34Its trick is enabling MTP. Guessing
  73. 2:37several tokens ahead, then verifying
  74. 2:39them all at once. The speed doubles. I
  75. 2:42reach 58 tokens per second. I keep
  76. 2:44improving, reaching 64 and then 70
  77. 2:47tokens per second. And the prefill even
  78. 2:50hits 1000 tokens per second.
  79. 2:52Then, one evening in October, I got
  80. 2:53really scared. I run one test too many,
  81. 2:56the RAM fills up completely, the server
  82. 2:58crashes, and I can't connect to it. And
  83. 3:00I can't go restart it manually because
  84. 3:02I'm not in the same place as the server
  85. 3:04. Since that day, I'm super careful, I
  86. 3:06don't run any tests without protection
  87. 3:08anymore. I set up a RAM limit, an
  88. 3:10automatic cutoff in case of issues, and
  89. 3:12an automatic reboot.
  90. 3:14At that moment, I think to myself, "
  91. 3:15What if I wrote my own engine, made
  92. 3:17specifically for this?" In one day, it
  93. 3:19goes from 29 to 63 tokens per second.
  94. 3:22It's super motivating, but it's still
  95. 3:24slower than the one I was already using
  96. 3:26. I set it aside, but I've understood
  97. 3:28where the time was being lost.
  98. 3:29At the end of October, I discover Basalt
  99. 3:31, an even newer engine. It shows a lot
  100. 3:34of promise. I install it, run it, and
  101. 3:36there, 57 tokens per second. Worse than
  102. 3:39before. I was really disappointed. But I
  103. 3:40can't believe it. I think there's a
  104. 3:42problem, so I look for where it's coming
  105. 3:44from. And there, I find 10 GB of RAM
  106. 3:46occupied for nothing, because of my old
  107. 3:48tests. I clean it up, run it again, and
  108. 3:50there, 94 tokens per second. And the
  109. 3:53prefill takes off, 2200 tokens per
  110. 3:55second.
  111. 3:56At the start of November, I want better
  112. 3:57quality. I try other quants and there,
  113. 4:00it's weird. Each one drops back to 63
  114. 4:02tokens per second. I think first it's
  115. 4:04the engine, then the machine, and I'm
  116. 4:05wrong both times, actually. The problem
  117. 4:07is one single layer out of 48 in a
  118. 4:09format that the fast mode doesn't
  119. 4:11recognize. Because of it, all the models
  120. 4:13switch to slow mode. I convert this
  121. 4:14layer, and there, I reach 83 tokens per
  122. 4:17second.
  123. 4:17One big question remains: which quant? I
  124. 4:20tried nine of them. The largest are too
  125. 4:22heavy; their experts don't fit in RAM.
  126. 4:24The surprise comes from the smallest
  127. 4:26ones. Unsloth's, the UDIQ2XS, keeps
  128. 4:28precision where it counts and hits 99.4
  129. 4:32%of the original quality. In perplexity
  130. 4:35, it gets 3.285 versus 3.273 for the
  131. 4:39original model. It's almost the same.
  132. 4:42Smaller and better than larger quants,
  133. 4:43this is the one I'm keeping. Alright,
  134. 4:46let's sum everything up. Here is the
  135. 4:47progress made from late August to early
  136. 4:50November. We see three major leaps. And
  137. 4:52each time, it wasn't the hardware that
  138. 4:54changed, just the engine. And my second
  139. 4:56card in all this, the RTX 3070, at first
  140. 4:59, it gave me 20%more speed. With the
  141. 5:01expert cache, nothing at all. With the
  142. 5:03specialized engine, 10 to 15%, and with
  143. 5:06Basalt, it slows everything down. The
  144. 5:08official version of Basalt even rejects
  145. 5:10it. Basalt is only compatible with cards
  146. 5:13starting with 50. The better the engine
  147. 5:15, the less useful it is. In the end, I
  148. 5:17won't be using this card.
  149. 5:18Once the model was well-tuned, I asked
  150. 5:20myself a question. What if it actually
  151. 5:22existed? I tried Swift, which thinks for
  152. 5:24less time. A month later, I went back to
  153. 5:26the original. Then, Edge Zero, which
  154. 5:28promises more speed with a new kind of
  155. 5:30expert. Converted to the classic format
  156. 5:32, it's fast, but its quality drops
  157. 5:33significantly. So, I modified the engine
  158. 5:35to read it without losing anything. It
  159. 5:37works, but it doesn't exceed 57 tokens
  160. 5:39per second. So, I deleted everything.
  161. 5:41I kept searching, digging through
  162. 5:43Hugging Face, and found about 1,600
  163. 5:45versions of this model.
  164. 5:47Almost all are simple quants or
  165. 5:49Uncensored versions, but one name stands
  166. 5:51out: Tinfield 1. A fine-tune made for
  167. 5:54code. He is a real competitor. I
  168. 5:56downloaded it, converted it, and
  169. 5:58organized a duel between the two. First
  170. 6:00round: speed. Tinfield wins, 80 tokens
  171. 6:03per second against 68.
  172. 6:05Second round, I asked each for a small
  173. 6:07application, and Tinfield's was cleaner
  174. 6:09. So, I told myself, that's it, I found
  175. 6:11something better, but I wanted to be
  176. 6:13sure. So, I had them work as standalone
  177. 6:15agents on 10 automatically graded tasks
  178. 6:18. On the six easy tasks, it was a
  179. 6:20perfect tie. On the four difficult tasks
  180. 6:22, Flash Next succeeded at all, while
  181. 6:23Tinfield got stuck on the hardest one.
  182. 6:25He rewrites his code twice without ever
  183. 6:27running his tests. Result: 9.5 out of 10
  184. 6:31for Flash Next and 8.95 for Tinfield.
  185. 6:34Faster, but less reliable. The champion
  186. 6:36keeps his spot.
  187. 6:38So, what did I learn in three? 1, the
  188. 6:40limit isn't the graphics card, it's the
  189. 6:42RAM. 2, the engine matters more than the
  190. 6:45hardware. It multiplies the speed by 10
  191. 6:47. 3, a well-made quant beats a bigger
  192. 6:50quant. 4, a slower second card can
  193. 6:52actually waste time. 5, impressions are
  194. 6:55deceiving. Before changing models, you
  195. 6:56have to measure.
  196. 6:58Today, my model runs at 83 tokens per
  197. 7:00second in decode and 2200 in predictive
  198. 7:02on a machine many people could have at
  199. 7:05home.
  200. 7:06I simply installed two 16GB sticks and
  201. 7:08two 8GB sticks of DDR4 in my PC, plus a
  202. 7:115060 TI graphics card, so the
  203. 7:13entry-level for the latest generation of
  204. 7:16GPUs.
  205. 7:17The same model, the same equipment, and
  206. 7:19yet 11 times faster than on day one. And
  207. 7:21I think the story isn't over yet. This
  208. 7:23was Antoine In-Line. See you next time.

About this transcript

This page contains the full transcript of Mon IA locale va 11 fois plus vite (même PC) by Nathan Inline, generated from the public captions YouTube serves with the video. The transcript has 1,422 words across 208 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.