YouTube2Text

Riesiges KI-Modell auf einer RTX 3090: 116 Token/s mit Strata! LLM Benchmark mit Qwen 3.8 Flash Next — Transcript

by Mario Alka · 3,974 words · 574 segments · language en · Watch on YouTube

Full transcript

  1. 0:00If you have a consumer PC and you want
  2. 0:02to run a really awesome AI model on it,
  3. 0:04then you definitely need to watch this
  4. 0:06video. With that said, hello and
  5. 0:10welcome to today's video, because even
  6. 0:12if you might not believe it, what I
  7. 0:14have here is not a special AI server
  8. 0:17like you usually see in my videos, but
  9. 0:19rather, let's take a look, a completely
  10. 0:21normal consumer PC in a rack case. What
  11. 0:25exactly is inside this server? Well,
  12. 0:27hardware that you probably have in your
  13. 0:29own gaming PC at home to play Fortnite
  14. 0:31and other games. A 750-watt power
  15. 0:35supply, nothing special. A very simple
  16. 0:38AM4 CPU, no idea which series,
  17. 0:40definitely not a high-performance one.
  18. 0:43Plus 32 GB of RAM and an RTX 3090.
  19. 0:48There’s nothing more to it than that.
  20. 0:49So, in the end, everything that you
  21. 0:50would also have yourself. I'm checking
  22. 0:52right now, does it have an NVMe? Yes.
  23. 0:54Oh, it even has an NVMe, I have to
  24. 0:56mention that. So, it actually has a
  25. 0:58decent amount of power. The difficulty
  26. 1:01now with the 3090 is that it only has
  27. 1:0424 GB of VRAM. That's a fair amount,
  28. 1:07relative to the consumer average, but
  29. 1:09when we look at the AI sector, we all
  30. 1:11know that's nothing. It's ridiculous.
  31. 1:15But someone has brought something
  32. 1:17really cool to the market called Strat,
  33. 1:20and with Strat, it's possible to run
  34. 1:23really large models on such consumer
  35. 1:25PCs. Although, "really large" is
  36. 1:28actually incorrect. Currently, it only
  37. 1:30works with—yes, over there—Qwen 1.5
  38. 1:32110B. That is what is runnable with it.
  39. 1:35A model that normally requires around
  40. 1:38180 GB of memory. Yes, and you can
  41. 1:42comfortably run that here on a 24 GB
  42. 1:45graphics card, and supposedly with
  43. 1:47quite okay performance, too. And I’d
  44. 1:51say we take a look at the whole thing,
  45. 1:53evaluate it, and benchmark whether
  46. 1:54it’s actually true and if we can
  47. 1:56really get it to perform that well. And
  48. 1:59we are talking here about a 110 billion
  49. 2:03parameter model on an RTX 3090. This
  50. 2:07means we are trying to pack a model
  51. 2:10that is so big into such a small amount
  52. 2:14of VRAM, which is quite an exciting
  53. 2:17story, and especially the Qwen 1.5
  54. 2:20model is very powerful. So, anyone who
  55. 2:23knows my channel knows that I’ve
  56. 2:24already set up a complete server with
  57. 2:26it and built entire programs with it.
  58. 2:28So, as far as that goes, you can really
  59. 2:30get a lot done with it. Of course,
  60. 2:32it’s also clear that it has certain
  61. 2:34limits, one has to say. Especially
  62. 2:36regarding the commercial license,
  63. 2:38that's more about the licensing, but
  64. 2:40software-wise it is super, super
  65. 2:42exciting. And if you're interested in
  66. 2:44the whole thing, definitely check out
  67. 2:45my channel. That's where exactly that
  68. 2:47video is, a Prompt Browser Game. Check
  69. 2:49it out. I was super surprised at what
  70. 2:52this AI model built here, and being
  71. 2:54able to run the model on a 3090, where
  72. 2:57I previously needed an RTX 6000. Yeah,
  73. 3:00if that works properly, then it is
  74. 3:02super, super good. Now you always have
  75. 3:06to add, and for me, the whole thing is
  76. 3:08a bit like Hummingbird. And by
  77. 3:10Hummingbird, I don't mean the one you
  78. 3:13might know from me, Hummingbird 1 from
  79. 3:16Aleph Alpha, the AI model from Germany,
  80. 3:19but I mean the Hummingbird, uh, where
  81. 3:21it was said: Yes, you can run giant AI
  82. 3:24models—there's news about it—on a
  83. 3:2724 GB laptop. Ultimately, exactly the
  84. 3:31same news that you read about Strater,
  85. 3:33because for Strater it is also said
  86. 3:35that you need 32 GB of RAM, 12 GB of
  87. 3:37VRAM, and it works. The deciding factor
  88. 3:40is, and you always have to see this,
  89. 3:42the hype that is created around things
  90. 3:44that aren't even there. Because with
  91. 3:46Hummingbird, it was the case back then,
  92. 3:49if we take a look at the GitHub account
  93. 3:52, it was described exactly that you
  94. 3:54need, for example, 6 RTX 5090s to
  95. 3:57generate 9 to 9.2 tokens per second. I
  96. 4:00had tested it myself on an RTX 6000
  97. 4:02with 96 GB of VRAM and I think I got
  98. 4:050.1 tokens or something like that. An
  99. 4:08absolutely massively poor value, and I
  100. 4:10have to say, that made me a bit, I
  101. 4:12don't know, with Strater at the
  102. 4:14beginning, I thought, oh, this is just
  103. 4:16another hype story again. But I can
  104. 4:19tell you in advance: It is something
  105. 4:21that is really super exciting and
  106. 4:23interesting, and it is not a hype like
  107. 4:25Hummingbird, which is quite cool in
  108. 4:27theory in the end. But rather Strater,
  109. 4:30if you keep building on it, can be a
  110. 4:32foundation for really cool future AI
  111. 4:34projects. That's why we should first
  112. 4:37take a look at what Strater actually is
  113. 4:39and how it ultimately works. And so
  114. 4:41we'll dive a little bit into, um, AI
  115. 4:44models. How is an AI model structured
  116. 4:46and how do they work? We have here a
  117. 4:49125 billion parameter model. That means
  118. 4:54the model is huge and contains a lot of
  119. 4:56content, a lot of knowledge, and a lot
  120. 4:58of data, and until now, if you wanted
  121. 5:01to run something like this, you needed
  122. 5:03server-grade cards, but in this case,
  123. 5:05we're talking about a gaming PC. And
  124. 5:09now you have to imagine the model. That
  125. 5:12is also one of the reasons why this
  126. 5:13Strater. for example, currently only
  127. 5:15works with Qwen 38B, because it's a
  128. 5:18so-called MoE model, and for an MoE
  129. 5:20model, imagine all these dots as
  130. 5:23experts; in this case, it consists of
  131. 5:25almost 25,000 experts, namely this Qwen
  132. 5:2838B Flash Next. And not every one of
  133. 5:31these experts is always active, but
  134. 5:33only the golden dots you see right now,
  135. 5:35those are. for example, the active
  136. 5:38experts that you need. This means the
  137. 5:40entire model does not have to be loaded
  138. 5:43all the time; parts of the model are
  139. 5:45sufficient. The difficulty is just
  140. 5:47which parts of the AI model actually
  141. 5:49need to be loaded. And to make this
  142. 5:53whole thing performant, it's the case
  143. 5:55with Strater that they said: "Hey,
  144. 5:57we'll distribute the experts across
  145. 5:59four locations." On one hand, there's
  146. 6:03the graphics card; here are the experts
  147. 6:04that are constantly being queried, the
  148. 6:06ones you need all the time. One expert,
  149. 6:08for example, handles logic, another
  150. 6:10handles mathematics, another handles
  151. 6:12some knowledge topics. That's just a
  152. 6:14way you can imagine it a little bit.
  153. 6:15And if I'm writing source code all the
  154. 6:17time, I don't need an archaeologist who
  155. 6:19can tell me something about dinosaurs;
  156. 6:20instead, I ultimately need ones that
  157. 6:22are responsible for my specific field.
  158. 6:24This means I have the graphics card on
  159. 6:26one side, and the experts that are
  160. 6:28especially relevant to me are loaded
  161. 6:30there. All 25,000 experts are kept in
  162. 6:35the RAM. And that is also partly the
  163. 6:38reason why, if we go to the GitHub
  164. 6:41account of Strater. I will also show it
  165. 6:44myself in a moment. You just need at
  166. 6:46least 32 GB of RAM, because it's simply
  167. 6:49necessary for this technique to work,
  168. 6:51so that you can ultimately define this
  169. 6:54waiting room. The other thing, the
  170. 6:57processor, is also involved. It helps
  171. 6:59out, it handles some of the
  172. 7:00calculations. And on the hard drive,
  173. 7:02that is ultimately the directory of
  174. 7:04experts. But very little is loaded from
  175. 7:07there. That means the hard drive, which
  176. 7:10is a slow component, isn't quite as bad
  177. 7:11in this case, although it is still
  178. 7:13recommended to use an NVME. But exactly
  179. 7:16these four locations are what work
  180. 7:17together at this point. And how should
  181. 7:21I imagine all of this? So, whenever you
  182. 7:23generate a new word or a new token, the
  183. 7:25question is always: "Hey, which experts
  184. 7:27do I need for this? Then the experts
  185. 7:30are fetched, the calculation is
  186. 7:31performed, the whole thing is finished,
  187. 7:33and it starts all over again. And I
  188. 7:36just said, well, not all experts have
  189. 7:37to be loaded all the time. This means
  190. 7:39they reside in the graphics card, and
  191. 7:41partly come from the system memory.
  192. 7:43This is a technique already known from
  193. 7:46other models, but in this case, it's
  194. 7:48optimized even more precisely, and most
  195. 7:50importantly, only those ten experts
  196. 7:52that are needed at that exact moment
  197. 7:54are loaded. Now, it is also the case—
  198. 8:00and this is an additional factor—that
  199. 8:02normally, if you calculate four words
  200. 8:04or four tokens, you need four separate
  201. 8:06passes. That means pass one, I fetch
  202. 8:09the experts, I calculate, next token,
  203. 8:11pass two, fetch expert, calculate,
  204. 8:14token, and so on and so on. But this
  205. 8:17whole process also costs time, and that
  206. 8:20is......for example, one of the tricks
  207. 8:22that Strater uses, because Strater says
  208. 8:25:" Look, we don't go and reload word by
  209. 8:28word by word, but instead I do one pass
  210. 8:30, load all the relevant experts, and
  211. 8:33then, let's say, I finish my four items
  212. 8:35and then I reload again. "And this
  213. 8:38fetching of the experts consumes
  214. 8:40resources. And in that moment, it is
  215. 8:42said that you are 1.6 to 1.8 times
  216. 8:45faster. Because you ultimately fare
  217. 8:48better with a sort of preview of which
  218. 8:51experts you likely need and keep
  219. 8:53loading things in, even in the
  220. 8:54background, instead of doing it fresh
  221. 8:57in every single run, as just shown,
  222. 8:59over and over again. And that is
  223. 9:02ultimately the actual trick of the
  224. 9:03whole technique, that you manage to do
  225. 9:05this. Well, we distribute a lot into
  226. 9:08the RAM, use it in between, and then
  227. 9:10gain this resource advantage. Yes, and
  228. 9:15why is all of this so interesting? I
  229. 9:18mean, sure, I don't think I need to
  230. 9:19explain much to you guys. Uh, it’s
  231. 9:22generally about considering what RAM
  232. 9:24and VRAM cost these days. If you manage
  233. 9:29to run 125 billion parameter models,
  234. 9:32large models, with 12 GB of VRAM and 32
  235. 9:35GB of RAM, that’s an extremely grand
  236. 9:38thing—on one hand for the entire AI
  237. 9:40market, but also for you, because you
  238. 9:43can drastically reduce your ongoing
  239. 9:46subscription costs. I also realize that
  240. 9:49if you say," I want to code, I use my
  241. 9:51Cloud, I have my Astra, I need that
  242. 9:53quality, "then there’s no question
  243. 9:55about it. It’s the same for me; I use
  244. 9:58Qwen 2.5 Coder 32B for about 50%of my
  245. 10:01AI agents. I have over 40 agents, half
  246. 10:05of them are roughly Qwen 2.5 Coder 32B,
  247. 10:07and I am extremely happy. But I can't
  248. 10:10replace it completely, because it's
  249. 10:12clear that for large coding tasks, the
  250. 10:14frontier models are simply much better.
  251. 10:17Yes, and I mean I’ve already
  252. 10:18presented the requirements a few times.
  253. 10:20It’s just important for me to explain
  254. 10:23a few things, especially regarding the
  255. 10:25recommended hardware. So, one thing
  256. 10:27that is very important for you: you can
  257. 10:29run it with Nvidia or with AMD as well.
  258. 10:31However, 64 GB of RAM is recommended. I
  259. 10:35consciously used 32 GB in this test
  260. 10:37because, relative to consumer graphics
  261. 10:39cards, I have to say I didn't have
  262. 10:42anything else in the server I could use
  263. 10:44for it. That’s why it was the RTX
  264. 10:473090, but with 32 GB. I used an NVMe
  265. 10:50drive and ran it under Linux, but those
  266. 10:52are just the possibilities of how you
  267. 10:54can run all of this yourselves. And if
  268. 10:57we take a look at the results now, I
  269. 10:59have to say, it gets extremely exciting
  270. 11:01. Because it was important to me to say
  271. 11:04: Okay, generally, vLLM is great
  272. 11:06technology on a 3090 with all the
  273. 11:08requirements we just mentioned, like
  274. 11:10the Ryzen 5 5600X. I’ve now evaluated
  275. 11:14it again with a two-bit quantized model
  276. 11:17—I’ll explain that in a moment—to
  277. 11:20compare it with an AI server using an
  278. 11:23RTX 6000 Ada, with 96 GB VRAM and 128
  279. 11:26GB RAM available, though it only used
  280. 11:2960 GB and a four-bit quantized model,
  281. 11:33to see:" Hey, is there an advantage if
  282. 11:36I run this on hardware that can already
  283. 11:39run such a model? "So, I definitely
  284. 11:43included that in the benchmark. And
  285. 11:45very importantly, I also wanted to do
  286. 11:47it on the DGX Park, but here is the
  287. 11:49problem: it's ARM instead of X86. That
  288. 11:53means the architecture didn't work, and
  289. 11:55I don't have unified memory, meaning I
  290. 11:57don't have RAM and graphics card memory
  291. 12:00combined, but rather I'm using the
  292. 12:02memory that's available together, and
  293. 12:04the current setup isn't compatible with
  294. 12:06that, but we'll talk about that in a
  295. 12:08moment. Let's first talk about
  296. 12:11quantization. I chose both 2-bit and 4-
  297. 12:14bit quantization. And now you might be
  298. 12:17asking yourselves, what is the
  299. 12:19disadvantage? So, 2-bit quantization
  300. 12:22basically means the model is more
  301. 12:25compressed. This means it's no longer
  302. 12:28the original model in its large size,
  303. 12:30but you can think of it like drawers; a
  304. 12:32lot of information that was previously
  305. 12:34in separate small drawers has been
  306. 12:36packed into one, and the quality
  307. 12:38generally suffers a bit because of that
  308. 12:40. However, the 2-bit quantization was
  309. 12:43necessary to even be able to run the
  310. 12:44whole thing, because otherwise I
  311. 12:45wouldn't have been able to load the
  312. 12:47model at all. And that is also a bit of
  313. 12:49a disadvantage. So if you are operating
  314. 12:51the whole thing with the minimum
  315. 12:53requirements here, then you have to
  316. 12:54work with 2-bit quantization. But we
  317. 12:57will talk a bit about the quality later
  318. 12:59and say, okay, is the model even usable
  319. 13:01at all, or is it just plain stupid to
  320. 13:03the point where it doesn't work anymore
  321. 13:05? Now let's take a look at the
  322. 13:07benchmark in general and the results,
  323. 13:09and I have to say, it surprised me
  324. 13:11extremely. I didn't expect it. The 2-
  325. 13:14bit quantized model on a 3090 with 32
  326. 13:18GB RAM, 116 tokens per second. That's a
  327. 13:23terrifically good value, guys. That's a
  328. 13:25really good value, I can tell you. And
  329. 13:28I'll go to the website lmbenchmark.de
  330. 13:30for that, because that's where you can
  331. 13:32also look up all the benchmarks I do.
  332. 13:35If you filter by Nvidia GeForce RTX
  333. 13:373090 there, you will see this benchmark
  334. 13:39value right here. Also very important
  335. 13:41to note there. You can see I also did a
  336. 13:45benchmark value there with a 3-bit
  337. 13:46quantization, and you can see that the
  338. 13:48performance goes down significantly
  339. 13:50because it's not recommended by the,
  340. 13:52let's say, manufacturer. They would
  341. 13:55rather say, just take the 2-bit
  342. 13:56quantized one. I included it anyway so
  343. 13:58you have some comparison values and can
  344. 14:00see: yes, if you choose the models too
  345. 14:02large, then a lot is offloaded to RAM,
  346. 14:04then maybe the hard drive is partially
  347. 14:06used as well, and then the whole thing
  348. 14:08just gets slow. And you can really see
  349. 14:11that extremely when you look at the
  350. 14:14prefill here, i.e., this loading of
  351. 14:16data; it's already relatively slow. And
  352. 14:20so you can put it into perspective: I
  353. 14:22have another benchmark for comparison,
  354. 14:25which I'll show you, a Qwen 36, which
  355. 14:27is only a 35B model. So, mind you, with
  356. 14:31the Qwen 38, we're talking about a 180B
  357. 14:33model, and here we're talking about a
  358. 14:3635B model, and the 35B model is almost
  359. 14:39just as fast with 162 tokens compared
  360. 14:41to 116 tokens. So you can really see
  361. 14:45that where you might have only had
  362. 14:47small models before, you can now
  363. 14:49suddenly load a large model yourself
  364. 14:51and it just works incredibly,
  365. 14:53incredibly well. Therefore, the
  366. 14:55benchmark results for a single request
  367. 14:57are ultra-good, and for five parallel
  368. 14:59requests—meaning when multiple
  369. 15:00processes are running at the same time
  370. 15:02—they are still ultra-good; because
  371. 15:04that is usually the problem, that one
  372. 15:06request is fine, but as soon as two or
  373. 15:08three happen, it starts to fall apart.
  374. 15:11That is not the case here at all, and
  375. 15:13even with ten parallel requests, we get
  376. 15:151100 tokens per second—sure, not per
  377. 15:17request, you have to divide by 10 to
  378. 15:19get the individual value, but those are
  379. 15:22still ultimately good values. And this
  380. 15:25so-called prefill, this token
  381. 15:27processing—meaning you give it text
  382. 15:29and it processes it at 1400—is also
  383. 15:32an extremely good value. We can compare
  384. 15:35that as well. I'll switch over again.
  385. 15:37Let's look at a small model again. Sure
  386. 15:40, if I run a Qwen 36 on a graphics card
  387. 15:42like that, I get 3000 in prefill, but
  388. 15:44we have 1400, which is still a very,
  389. 15:46very good value, because others
  390. 15:48sometimes only have 400, 500, and so on
  391. 15:50, and for such a large model, it's
  392. 15:52bombastic, I definitely have to say.
  393. 15:56Let's take a closer look at the whole
  394. 15:57thing. Um, if we now look at the 3090
  395. 16:00at the lowest level, then it's exactly
  396. 16:04what I said. Um, we quantized it with 2
  397. 16:09bits, and that costs accuracy. Normally
  398. 16:13, you would use 4-bit quantization;
  399. 16:14that's actually the typical standard.
  400. 16:16That means the model isn't shrunk down
  401. 16:19any further; four is a typical value.
  402. 16:22Eight is basically the original that
  403. 16:24you would normally use. There is also
  404. 16:2716, but that’s only used if you need
  405. 16:29to train new models. So, in other words
  406. 16:32, 8 or 4-bit would be normal. And there
  407. 16:34is the danger that people say, well,
  408. 16:36the model is getting dumber, it's
  409. 16:38hallucinating, it's claiming weird
  410. 16:40things. Those are risks that you
  411. 16:41naturally have. I also took a look at
  412. 16:43what the general quality is like. But I
  413. 16:47didn't do an extensive long-term test.
  414. 16:48We'll take a look at the results in a
  415. 16:50moment. So you don't say, yeah cool,
  416. 16:53it's great that it runs, but it's just
  417. 16:54talking nonsense. It's definitely not
  418. 16:58quite that bad at the moment. Um,
  419. 17:01exactly. Why did I measure it this way?
  420. 17:02I already explained that quite clearly
  421. 17:03here. Ultimately, the problem is the 32
  422. 17:06GB, which is why I can only do it with
  423. 17:08that quantization. As I said, I also
  424. 17:11measured it with 3-bit quantization,
  425. 17:13but the values—especially when the
  426. 17:16prefill is, for example, at 106—you
  427. 17:18can imagine 106 tokens being processed
  428. 17:20per second. Especially if you want to
  429. 17:23process ultra-large documents, so 106
  430. 17:25words, let's say roughly around that
  431. 17:27mark. It just takes forever, and in the
  432. 17:29end, it's no longer usable. So,
  433. 17:31therefore, if you have more RAM, you
  434. 17:34can use the larger variants. Otherwise,
  435. 17:37Strater only recommends 4-bit if you
  436. 17:40have 96 GB of RAM. That’s just how it
  437. 17:43is. I also did this because I was
  438. 17:46interested: Does Strater offer a
  439. 17:49performance advantage if I have a
  440. 17:51graphics card with 96 GB of VRAM, where
  441. 17:54I could fit the 4-bit quantized version
  442. 17:56anyway? And so I compared it: I ran it
  443. 17:59with Strater and with normal Llama.cpp.
  444. 18:03You can see the values are almost
  445. 18:05identical. So, there is a minimal
  446. 18:08difference with one request. With ten
  447. 18:11requests, it is a bit faster with
  448. 18:13Strater. So I would say, on average,
  449. 18:15it's about the same. Yes, that means
  450. 18:19for you, if you have a graphics card
  451. 18:20where it already fits, then you don't
  452. 18:22have this deficit where you say, yeah,
  453. 18:23it doesn't fit, so you have to keep
  454. 18:25copying back and forth. Llama CPP also
  455. 18:28had techniques to manage expert layers
  456. 18:30and such itself. That means the
  457. 18:32bottleneck is somewhere else entirely.
  458. 18:34It brings you no advantage. So in that
  459. 18:35moment, you don't need to use Strater.
  460. 18:37So it only makes sense if you really
  461. 18:39have small graphics cards and if it
  462. 18:40just doesn't work any other way. That's
  463. 18:42where it gets exciting, too. Because as
  464. 18:44of today, Strater can only manage one
  465. 18:46graphics card. I will introduce the
  466. 18:48forks to you later. There are already
  467. 18:50Strater forks that can handle two or
  468. 18:52more graphics cards. You could look at
  469. 18:54that again, you could then check the
  470. 18:56whole thing to see to what extent it
  471. 18:58brings a performance advantage. If you
  472. 19:00have three graphics cards with, for
  473. 19:03example, 32 GB, you have 96 GB. Does it
  474. 19:05bring an advantage? I don't know. You
  475. 19:07definitely have to look at that
  476. 19:08separately again. Exactly, as I said,
  477. 19:10DGXPK unfortunately didn't work. Um,
  478. 19:12it's a different platform, so I will
  479. 19:15make a separate video about that
  480. 19:17because with DGXPK it's also exciting,
  481. 19:19is there a performance advantage, yes
  482. 19:21or no? But we will definitely make a
  483. 19:23separate video for that. I already
  484. 19:26touched on the forks a bit, you have to
  485. 19:28consider, um, it is now the 13th.
  486. 19:30October. The whole thing was just Must
  487. 19:32check GitHub briefly to see if I can
  488. 19:34find that out quickly. Um. I think
  489. 19:38there are already 48 releases now.
  490. 19:40Let's take a quick look. Let's look at
  491. 19:43one of the last releases to perhaps
  492. 19:45find out when Strater actually is from.
  493. 19:48So, you can see it, two weeks ago, no
  494. 19:50idea what. So, it's really not that,
  495. 19:53it's a relatively fresh model server.
  496. 19:57Exactly, that is probably the first
  497. 19:58launch. So the whole thing is two weeks
  498. 20:01old and there are already 2,300
  499. 20:02derivatives. So that is really quite
  500. 20:04something. A rating with 23,000 stars.
  501. 20:07So that, I think that says quite a lot.
  502. 20:09But what interesting forks are there? I
  503. 20:11can definitely recommend the one for V1
  504. 20:13to you. The dual GPU thing, that is
  505. 20:15definitely currently under development.
  506. 20:17There are others that work without Cuda
  507. 20:19so that it also works better on AMD and
  508. 20:22Intel. As I said, for DGX Park and so
  509. 20:23on, there are still others. There's
  510. 20:25another one for notebooks, there's one
  511. 20:27for older AMD graphics cards, and so on
  512. 20:29. So, these are just the best ones I've
  513. 20:31picked out for you. I really have to
  514. 20:34say, this is exciting, and I believe
  515. 20:36Strater will change a lot in the market
  516. 20:38because you can really use it
  517. 20:40effectively. Now, of course, the
  518. 20:45question remains, as I mentioned
  519. 20:46earlier, how big is the difference in
  520. 20:48quality? I took a rough—really rough
  521. 20:52—look at it and gave both, meaning
  522. 20:55the two-bit quantized version and the
  523. 20:58four-bit quantized version, identical
  524. 21:01tasks; in this case, 200 of them, and
  525. 21:04one achieved 99.5, the other achieved
  526. 21:07100. That means for one task, which was
  527. 21:11about grammar or something similar, the
  528. 21:13two-bit version messed up a little bit.
  529. 21:17The point is, it’s 0.5%. That doesn't
  530. 21:20sound like much. But you have to
  531. 21:22consider one thing: in a harness, like
  532. 21:24when using OpenCL or something similar,
  533. 21:26the thing runs in a loop the whole time
  534. 21:28. That means it does hundreds of tasks
  535. 21:33per hour, and if it gets a few tasks
  536. 21:36wrong here and there, then the overall
  537. 21:39results might end up being incorrect.
  538. 21:43So, it's hard to estimate. I think to
  539. 21:47really determine if the two-bit
  540. 21:48quantized version works, you'd have to
  541. 21:51do a long-term benchmark over several
  542. 21:53hours with really complex, large tasks
  543. 21:55that build on each other, like creating
  544. 21:57a website and so on. Maybe I'll make a
  545. 22:01separate video on that, where I
  546. 22:02generally compare these quantizations,
  547. 22:05because in a short-term test, I'd say
  548. 22:07it's okay. So, you can probably use it.
  549. 22:09You probably won't even notice it, but
  550. 22:11as I said, it would need to be looked
  551. 22:13at in detail. In general, who is
  552. 22:15Strater worth it for? Clearly, for
  553. 22:17everyone with small graphics cards,
  554. 22:19because the token results are wonderful
  555. 22:21. If you have enough VRAM, you don't
  556. 22:24need it. And with Unified Memory, you
  557. 22:26don't even need to use the normal,
  558. 22:27original Strater, because as I said, it
  559. 22:29won't add any value for you and it
  560. 22:31won't work. Yeah, so a question that
  561. 22:34definitely interests me: how much VRAM
  562. 22:35do you have on average? Um, maybe write
  563. 22:38it in the comments. It helps me in the
  564. 22:40future to categorize such videos and
  565. 22:42see, okay, does it make sense to
  566. 22:44perhaps install a different graphics
  567. 22:46card? Um, if you're interested in
  568. 22:48videos like these, then definitely
  569. 22:49leave a subscription. Please feel free
  570. 22:51to leave a like as well, because that
  571. 22:52definitely helps the videos reach more
  572. 22:54people. Yes, with that in mind, I'll
  573. 22:56say see you next time. Until then, and
  574. 22:57ciao.

About this transcript

This page contains the full transcript of Riesiges KI-Modell auf einer RTX 3090: 116 Token/s mit Strata! LLM Benchmark mit Qwen 3.8 Flash Next by Mario Alka, generated from the public captions YouTube serves with the video. The transcript has 3,974 words across 574 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.