Qwen 3.8 Flash Next vs Flash Next Coder — Transcript

Full transcript

1. 0:00Okay, by popular request and also
2. 0:02because I'm curious myself, let's take
3. 0:03another look at Strata. But this time,
4. 0:04let's put Quen 3.8 Flash Next, the model
5. 0:07we looked at last time, uh, up against
6. 0:09Quen 3.8 Flash Next Coder. This is a
7. 0:11version of the model that allegedly is
8. 0:13specifically geared towards coding
9. 0:14tasks, which sounds like it's right up
10. 0:16my alley. Let's put these two models
11. 0:17headto head and see which one comes out
12. 0:19on top. No place [music] like local
13. 0:22host.
14. 0:24No place like local host.
15. 0:27>> Okay, just a quick catch up. Since the
16. 0:29review video last week, I've been using
17. 0:30Quinn 3.8 Flash Next with Strata almost
18. 0:32exclusively. It has quickly become my
19. 0:34go-to model. Uh, and I like it even
20. 0:36better than Quinn 3.8 27B dense, which
21. 0:38is saying a lot because I loved that
22. 0:40model. Uh, Flash Next is faster on my
23. 0:42hardware with Strata and it's even
24. 0:44faster now. I've tweaked it uh since
25. 0:46that last episode. In the last in the
26. 0:48review episode, I was getting about 45
27. 0:49tokens per second out of uh Flash Next
28. 0:52on my hardware with 36 gigs of VRAM and
29. 0:55128 gigs of DDR4 behind that. And a lot
30. 0:58of people in the comments were saying,
31. 0:59"You should be getting a lot a lot more
32. 1:00than that." Yeah, I I know I should.
33. 1:02Just a quick recap of my hardware. I've
34. 1:03got a dual GPU server sitting in the
35. 1:05corner there. It's called Saturn. I've
36. 1:06got an RTX 470 with 12 gigs of VRAM and
37. 1:09an RTX Pro 4000, small form factor with
38. 1:12uh 24 gigs of VRAM. Great. 36 gig VRAM
39. 1:15pool sounds great. Here's the problem.
40. 1:17The motherboard on that system is kind
41. 1:19of old. And the secondary slot, the slot
42. 1:21that the Pro 4000 is plugged into, is a
43. 1:22PCIe Gen 4 X4 slot. Ouch. That really
44. 1:26hurts uh bandwidth. Uh max transfer
45. 1:28going in and out of that slot is 8 GB
46. 1:30per second. Compared to the other slot,
47. 1:32PCIe Gen 5 X8 32 gigs per second. So,
48. 1:36when the two cards have to talk to each
49. 1:37other, they can only do so at the speed
50. 1:39of the slowest slot. And that really,
51. 1:40really slows me down. Uh, so 45 tokens
52. 1:43per second actually didn't seem that bad
53. 1:45to me. By comparison, when I run the
54. 1:47dense model, a 30 27B dense, I get 35
55. 1:50tokens per second on that same hardware
56. 1:51with the with a GPU split. Uh, so 45
57. 1:54tokens per second felt like a step up,
58. 1:56and it was, but we can do even better
59. 1:58than that. Uh, since then, I've tweaked
60. 2:00the settings a little bit. I've also
61. 2:01updated to the latest version. And when
62. 2:03I run, this is the same model we looked
63. 2:04at last time, Quinn 3.8 Flash Next, but
64. 2:06with an updated version of Strata and
65. 2:08with slightly different settings. We'll
66. 2:09talk about that in a bit. If I use my
67. 2:11favorite testing question, what does the
68. 2:13transient keyword do in Java? I get now
69. 2:1660 plus. I've seen a spike as high as
70. 2:18100. Uh it seems to be around Yeah,
71. 2:21there it goes. Look at that. 70. It's
72. 2:23it's it's more variable now than it was
73. 2:25before. But uh if we look at the
74. 2:26console, I I consistently get 60 plus
75. 2:28tokens per second. So that's like about
76. 2:30a 50% improvement over what we saw last
77. 2:32time almost. With the occasional spike
78. 2:34up to 100 tokens per second, which is
79. 2:36great. All I did other than upgrading uh
80. 2:38strata itself. I uh tweaked the settings
81. 2:40a little bit, there's options around do
82. 2:42you want to keep the KV cache in VRAMm
83. 2:44and push some experts out into system
84. 2:46memory or the other way around. Do you
85. 2:48want to put your KV cache in system
86. 2:49memory and uh leave more VRAM free for
87. 2:52uh for experts to reside? Uh you can
88. 2:54also uh enable or disable image uh
89. 2:56processing which uh either reserves a
90. 2:59gig or two of VRAM for the mm prod or it
91. 3:03evicts that and says no no no no we're
92. 3:04going to use that VRAM for uh for for
93. 3:06storing experts. What I found through my
94. 3:08tweaking and there's a bunch of
95. 3:09different settings you can you can
96. 3:10change with this. What I found is that
97. 3:12the more experts you can squeeze into
98. 3:14the VRAMm the faster this thing's going
99. 3:15to go. So I've got it tweaked now. Uh
100. 3:18okay so we've tweaked our speed. We've
101. 3:20got the model running really well. I'm
102. 3:22really really happy with it. Uh if you
103. 3:23watched the previous episode from last
104. 3:25Thursday, whatever that was, we did a
105. 3:26long running code session uh with uh
106. 3:28Quen 3.8 next running through strata.
107. 3:30Zero problems, zero. But there's another
108. 3:34option available and that is uh Quen 3.8
109. 3:37flash next coder. Let's take a quick
110. 3:39look at that. So we're going to go back
111. 3:42to our Austrian friends lab. They have
112. 3:44put together this uh flash next GSQ blah
113. 3:47blah blah coder. Okay,
114. 3:51this is a 512 expert mixture of experts
115. 3:54model with half of its experts removed.
116. 3:57Right off the bat, that strikes me as a
117. 4:00bad idea, but let's let's keep an open
118. 4:01mind here. So, uh what they did was, and
119. 4:04this is a little confusing with the way
120. 4:05they presented the numbers, they they
121. 4:06say it's uh effectively 1.89 bits per
122. 4:09parameter of the original transformer.
123. 4:11That's that's a bit of a lie. Let's
124. 4:12let's scroll down a bit. Uh what they
125. 4:16did was they they took half the experts,
126. 4:18the ones that they figured out somehow
127. 4:20have nothing to do with coding tasks and
128. 4:22they just said, "Well, we don't need
129. 4:23them." And they got rid of them. They
130. 4:24stripped them out. The remaining weights
131. 4:27are stored at 3.5 bits per weight. If we
132. 4:29scroll down a little bit further, we see
133. 4:30that no individual weight is actually
134. 4:32stored at 1.89 bits per second. Bits per
135. 4:35what am I saying? No individual weight
136. 4:37is stored at 1.89 bits per weight. Uh
137. 4:40what where they came up with that
138. 4:41number, it corresponds. So after you
139. 4:44remove half the experts and compress it,
140. 4:46the model's compressed at IQ3. I believe
141. 4:48the remaining weights are stored at 3.5
142. 4:50bits per weight. So it's the same IQ3S
143. 4:53quantization. I think they call it 1.89
144. 4:56bits per weight because if you compare
145. 4:58it to the size of the original model,
146. 5:01they're saying that effectively it's the
147. 5:03same size you'd get if you took the
148. 5:05original model and compressed it down to
149. 5:061.89 bits per weight. But they didn't do
150. 5:08that. They com they they they removed
151. 5:10half the experts, compressed the
152. 5:12remaining ones at IQ3, and the model
153. 5:14size is the same as what you'd get at
154. 5:16IQ1, but it isn't IQ1. So I my head is
155. 5:19kind of spinning a little bit. If we go
156. 5:20to their files uh tab here and look at
157. 5:23the they call it IQ1.
158. 5:26It's not. It's IQ3. The the weights are
159. 5:29compressed at IQ3, but they call it IQ1
160. 5:31because that's the effective size of it.
161. 5:33If you look at add up the two halves of
162. 5:34this file, it's 60 gigs. I believe the
163. 5:38one we downloaded last week was 80 gigs
164. 5:40or 84 somewhere in that neighborhood.
165. 5:42So, it's it's noticeably smaller. It's
166. 5:44not because they quantized it more. It's
167. 5:45because they removed half the experts.
168. 5:47Okay, a bold strategy. What they're
169. 5:49saying is true. They haven't lost
170. 5:51anything of value if your use case is
171. 5:54coding tasks. Uh they do mention here.
172. 5:58Yeah. So, they there's a disclaimer
173. 5:59here. Removing half a model's experts
174. 6:01necessarily reduces its capabilities.
175. 6:02Well, yeah, no kidding. The contribution
176. 6:04of this release is that the reduction is
177. 6:05directed towards code, agentic tool use,
178. 6:08vision, and spatial reasoning. And the
179. 6:10retained experts are therefore those
180. 6:11that support those domains. Great.
181. 6:13Degradation outside this set is an
182. 6:15accepted cost of this method. What
183. 6:16they're saying is this model is much
184. 6:18worse at other tasks that are not coding
185. 6:21tasks. If you just want to do coding
186. 6:23tasks, allegedly, this is the way to go
187. 6:25cuz it's smaller. And if we go to uh
188. 6:27Strata's page, uh we'll see this is the
189. 6:30the bargain basement entry- level uh
190. 6:31model that they support. You only need
191. 6:3332 gigs of system memory to run this.
192. 6:35Uh, but I noticed that there's a
193. 6:36disclaimer here as well. They
194. 6:38specifically say this model is weaker
195. 6:40outside code and it's not good at
196. 6:42languages. And they they kind of steer
197. 6:44you towards, you know, like maybe you
198. 6:46should be looking at Q2 or IQ3S. Uh, if
199. 6:48you're if you want to if you want a
200. 6:50well-rounded general purpose model,
201. 6:51don't use this one. Uh, fine. But let's
202. 6:53try it. Let's try it out because my use
203. 6:56case is overwhelmingly code related. Uh
204. 6:57almost everything I do with local LLMs
205. 6:59is either generating code or reviewing
206. 7:01code or both. So uh this seems like an
207. 7:05acceptable sacrifice at least on paper.
208. 7:07Let's see if it holds up in practice.
209. 7:08Okay. Okay. So we can rerun the Strata
210. 7:10setup script uh and it can it gives you
211. 7:13the option to download an additional
212. 7:14model on top of the ones you already
213. 7:15have. It's very easy to set up. Very
214. 7:17very easy. And it'll generate a startup
215. 7:18script. So I did go through the uh
216. 7:20calibration uh process with this new
217. 7:22model the same as I did last time. And
218. 7:24it did tell me that the default settings
219. 7:25are optimal for your hardware. So great.
220. 7:28And I'm going to run it on both GPUs.
221. 7:30Now, you may remember last week it
222. 7:32copied when we started up uh Quen 3.8
223. 7:34Flash Next, it copied 50 gigs of stuff,
224. 7:3650 plus gigs of stuff into system memory
225. 7:38and then maxed out memory usage on both
226. 7:40cards. So, let's take a look at HTOP on
227. 7:42Saturn. We're using 33.5. Some of that
228. 7:45is OS level stuff. So, I think it copied
229. 7:4730 gigs of stuff into system memory. And
230. 7:49if we slide up to envy top, uh, we're
231. 7:5111.7 out of 12 on card zero and 22 and
232. 7:54a2 out of 24 on card one. So, okay, we
233. 7:57again, we're maxed out. So, I should be
234. 7:59able to go back to the, uh, web UI. And,
235. 8:02uh, what does the transient keyword do
236. 8:04in Java?
237. 8:06Server not reachable. What? My god. You
238. 8:09know what it is? I know exactly what
239. 8:11that is. I tested it earlier on Saturn.
240. 8:13I didn't test it from my laptop. You
241. 8:15know what it is? It's the same problem I
242. 8:17ran into last time. It's the exact same
243. 8:18problem. Uh, by default, it binds to
244. 8:22that's the second time that's bit me.
245. 8:23Okay. Well, that's on me. I apologize.
246. 8:26All right, let's start that back up. It
247. 8:27binds to local host, so you can only
248. 8:29access it from the machine on which it's
249. 8:30running. You have to tell it to bind to
250. 8:320000 so I can actually hit it from my
251. 8:34laptop. That's a me problem, not a
252. 8:36strata problem. Okay, refresh. Okay,
253. 8:38great. We're back. So, what does the
254. 8:40transient keyword do in Java?
255. 8:43And
256. 8:4560 plus tokens per second. 70 plus 80
257. 8:47plus we got but so it's variable. It's
258. 8:50interesting that it it varies quite a
259. 8:51bit. Uh 60 to 88. So I'd say it's at
260. 8:55least as good at least as fast as Quin 3
261. 8:57flash next if not a little bit faster. A
262. 8:59longunning task is probably a better
263. 9:01measure of that than just a quick uh
264. 9:02one-off question. So So what's the
265. 9:04challenge for today? Uh I've got a pair
266. 9:06of challenges lined up. I don't know if
267. 9:08we're going to get to challenge two.
268. 9:09It's going to depend entirely on how
269. 9:10challenge one goes. Uh I've got an idea
270. 9:12in my head as to how this is going to
271. 9:14play out. I haven't rehearsed this
272. 9:15specific task, so I don't actually know.
273. 9:17But I I've got I've got a feeling. I'm
274. 9:19not going to make a prediction cuz I'm
275. 9:20almost almost I'm almost always wrong
276. 9:22when I make a prediction. But uh let's
277. 9:24let's see what we get. So what's the
278. 9:26challenge? I thought uh for this task,
279. 9:27for this first task, instead of getting
280. 9:29it to generate code, we could get it to
281. 9:30review code because this is a common uh
282. 9:32coding task that I do quite a lot. Uh if
283. 9:34we go to
284. 9:36uh I've received a pull request on one
285. 9:39of my projects uh TTS serve and I
286. 9:41thought you know what let's get uh these
287. 9:43models to review this pull request and
288. 9:45just tell me should I merge this or not.
289. 9:47So the project is TTS serve. If you're
290. 9:49not familiar I've got this uh library.
291. 9:50It's a layer of abstraction that sits
292. 9:52between a client application and uh one
293. 9:55or more TTS engines. So uh DOS.TTS TTS
294. 9:58or Quen 3 TTS or Lux TTS or Vox CPM or
295. 10:02OmniVoice or whatever Chatterbox,
296. 10:04whatever you're running. Uh TTS serve
297. 10:08sits in front of the TTS engines and it
298. 10:10provides a discoverable REST API that's
299. 10:12consistent across all of the all of the
300. 10:14different engines. So your client
301. 10:15application can connect to any TTS any
302. 10:18supported TTS engine without knowing or
303. 10:19caring about the details of that TTS
304. 10:21engine. It's actually kind of a neat
305. 10:22project. And we support now I I've lost
306. 10:24track like six or six to eight I think
307. 10:26different TTS engines. So I got a pull
308. 10:28request here. Uh add optional long text
309. 10:31chunking to Quen 3 TTS MLX. So right off
310. 10:34the bat this seems odd to me. Let's look
311. 10:36at the client application. Uh the client
312. 10:38application looks like this. It's called
313. 10:39talk with me. Uh and I've got it wired
314. 10:42up right at the moment to uh Lux TTS
315. 10:45sitting behind TTS serve. So if we go to
316. 10:49settings, uh the TTS engine that we're
317. 10:51connected to, localhost 7500 is Lux TTS
318. 10:55and Lux TTS provides these options.
319. 10:57These options are provided dynamically
320. 10:58by TTS serve. Uh the client application
321. 11:01can present uh sliders and uh UI
322. 11:03controls uh so that you can control the
323. 11:05different options for that specific DTS
324. 11:06engine. But if we connected to a
325. 11:08different one, if we connected to
326. 11:08Chatterbox, this whole list of options
327. 11:10would change dynamically. That's the
328. 11:12whole point. So uh let's take it for a
329. 11:14really quick uh test spin. I've got uh
330. 11:16I've got a bunch of Star Trek voices
331. 11:18cloned and ready to go. So, we can say
332. 11:19something like, "Captain, I recommend we
333. 11:21raise the shields immediately."
334. 11:23>> "Captain, I recommend we raise the
335. 11:25shields immediately."
336. 11:26>> Great. So, that voice clone came from
337. 11:29Lux TTS. Now, the application has an
338. 11:31option for uh what I call sentence by
339. 11:33sentence mode. Some TTS engines are
340. 11:35faster than others. It's usually a
341. 11:36trade-off between quality of the clone
342. 11:38voice and uh performance uh speed of
343. 11:40delivering that clone voice. Dots.TS,
344. 11:42TTS, for example, is one of the highest
345. 11:43quality voice cloners I've seen that you
346. 11:45can run locally. Uh, but the speed is
347. 11:48terrible. It takes a good 10 to
348. 11:49sometimes 20 seconds to synthesize a uh
349. 11:51a paragraph of text. So, what I did was
350. 11:55in talk with me, I introduced this
351. 11:56option called streaming sentence by
352. 11:58sentence mode. What this does is if you
353. 12:00give it more than one sentence, let's
354. 12:02actually turn that on and give it a
355. 12:04multi-sentence paragraph of text to
356. 12:06synthesize.
357. 12:08Captain, I recommend we raise the
358. 12:09shields immediately. There's an enemy
359. 12:11ship approaching. Wait a minute. It's
360. 12:13not an enemy ship. Yeah. No, it's
361. 12:15definitely not an enemy ship. It's a
362. 12:16friendly ship. Oh my god, it's the
363. 12:18Avengers. Let's say hello to them.
364. 12:21>> Captain, I recommend we raise the
365. 12:22shields immediately. There's an enemy
366. 12:24ship approaching. Wait a minute. It's
367. 12:26not an enemy ship. Yeah. No, it's
368. 12:27definitely not an enemy ship. It's a
369. 12:29friendly ship. Oh my god, it's the
370. 12:31Avengers. Let's say hello to them.
371. 12:32>> Okay, the quality of the voices is hit
372. 12:34and miss there. But we notice we got a
373. 12:36bunch of different replay buttons
374. 12:37underneath that because it doesn't
375. 12:39synthesize it all at once. It sends it
376. 12:41it looks for sentence splitting
377. 12:43punctuation like periods or question
378. 12:44marks or uh exclamation marks and it
379. 12:46sends it up to the TTS server one
380. 12:48sentence at a time so that we can
381. 12:50stream. It's not really streaming I know
382. 12:51but we can stream. And it's a poor man's
383. 12:52streaming approach uh to get a faster
384. 12:54time to first uh voice sample and we get
385. 12:57it a sentence at a time as opposed to if
386. 12:59we turn that off uh it will have to
387. 13:02synthesize the entire text block at
388. 13:05once. We can actually try this right
389. 13:08now.
390. 13:08>> Captain, I recommend we raise the
391. 13:10shields immediately. There's an enemy.
392. 13:12>> We can stop that. We That actually
393. 13:13wasn't that bad. The lock CTS is
394. 13:14actually really really fast. Uh, so
395. 13:16maybe not the best example, but uh, the
396. 13:17point is we get a single replay button
397. 13:19here that plays the entire paragraph
398. 13:20versus in streaming mode, we get one
399. 13:22replay button for each sentence. Great.
400. 13:25It's a neat little feature. You can turn
401. 13:26it on or off depending on which TTS
402. 13:27engine you're connected to. If we come
403. 13:29back to this PR here, add optional long
404. 13:31text chunking to Quinn3 TTSMLX. Okay.
405. 13:33So, what we're looking for, so what this
406. 13:35BR offers is punctuation aware chunk
407. 13:38boundaries, optional fixed silence
408. 13:40between chunks, optional linear
409. 13:42crossfade between chunks, validation for
410. 13:44chunking parameters,
411. 13:46uh tests covering chunk splitting. Uh,
412. 13:49okay. So, it sounds like it makes this
413. 13:50feature a lot smarter. But my my
414. 13:53immediate question when I look at this
415. 13:54just the title of the PR, we're adding
416. 13:56this feature specifically not just to
417. 13:59one specific TTS engine, Quinn3 TTS, but
418. 14:02to one specific variant of that TTS
419. 14:04engine, the MLX version of Quin3 TTS.
420. 14:06Why are we doing this? The whole point
421. 14:09of TTS serve is to expose options in a a
422. 14:11uniform way across TTS engines. And this
423. 14:14doesn't seem like an option that is
424. 14:15specific to Quen 3 TTS MLX. Why are we
425. 14:18only offering this option for that one
426. 14:20TTS engine? Why not all of them? I don't
427. 14:23know. Maybe there's something genuinely
428. 14:24MLX specific in here. So, I haven't
429. 14:26looked too much of the actual code. I'm
430. 14:29going to get the LLM to do that for me.
431. 14:30So, this is the task. This is challenge
432. 14:31number one. Each model is going to look
433. 14:34at this PR and uh do a code review on
434. 14:36it. Tell me is it is it mergeible as is
435. 14:39and try to answer the design
436. 14:40architectural question that I have. Why
437. 14:42are we targeting a specific variant of
438. 14:45one TTS engine when we support like
439. 14:47eight of them? So, okay. So in open code
440. 14:50I've switched to quen 3.8 flash next
441. 14:52coder here. Let's take a look at the
442. 14:53prompt. Compare this branch to master.
443. 14:55Give me a code review of the pro
444. 14:56proposed change. Is this mergeable as
445. 14:57is? And then my architectural question.
446. 15:00Why is this why does is this that's good
447. 15:03English. Why is this change specific to
448. 15:05Quen 3TS MLX? Can we not make this
449. 15:07change available to all TTS engines? So
450. 15:09uh I'm going to let Quinn flashcoder
451. 15:12chew on that and I'm going to bring up
452. 15:14the official no place like local host
453. 15:16stopwatch. your logo here for the low
454. 15:19low price of a RTX Pro 4500. Uh, okay.
455. 15:23So, we'll let uh Flash Coder look at
456. 15:25that and give me a code review. And then
457. 15:26we're going to do the same thing with uh
458. 15:28Coin 3.8 Flash Next IQ3S, the model we
459. 15:31looked at last week. And we're going to
460. 15:32compare the quality of the review that
461. 15:34we get and the answer to the
462. 15:35architectural question that we get. And
463. 15:36uh we're going to make sure that both
464. 15:38models can actually complete the task
465. 15:40cuz I'm not sure that they can. Maybe
466. 15:42I'm being too harsh, maybe I'm being too
467. 15:43cynical, but my prediction I'm going to
468. 15:45cut this out of the video if I'm wrong,
469. 15:46but my prediction is that uh the flash
470. 15:49coder model is it's got half its brain
471. 15:51removed, you know, like I I I I just
472. 15:53feel like that's a bad idea, you know?
473. 15:56You they they've claimed they've cut the
474. 15:58experts out that are not related to
475. 16:00coding, but are you sure those experts
476. 16:01aren't needed? Are you sure? Maybe I'm
477. 16:04wrong. Maybe I'm wrong. Let's uh let's
478. 16:06find out.
479. 16:09local host
480. 16:12here.
481. 16:13>> 5 minutes 38. That was pretty quick.
482. 16:15Let's see what we got. Uh interesting
483. 16:17that the test suite is red. I didn't I
484. 16:19didn't notice that, but I uh I did run
485. 16:20it just now. And sure enough, there are
486. 16:22two failing tests in this PR. So, right
487. 16:26off the bat, I think that answers my is
488. 16:27this mergeable question, but okay. Uh
489. 16:29the feature itself is solid. Okay. Uh
490. 16:32this PR adds a couple of crossfade unit
491. 16:35tests that are apparently machine
492. 16:37specific. The developer who wrote this
493. 16:39has got stuff installed in his
494. 16:41environment that I don't have in mine.
495. 16:42And without that, the tests fail. So,
496. 16:44uh, not good. Yeah. And the other the
497. 16:47remaining tests are kind of sloppy.
498. 16:49[gasps]
499. 16:50That's not really what I'm looking to
500. 16:52answer here, though. Okay. So, there's
501. 16:54some code problems. There's a few edge
502. 16:56cases that'll bite us. Maybe there's a
503. 16:59couple of minor performance nitpicks.
504. 17:01I'm not I'm not too concerned about
505. 17:02that. Okay. Let's get to the design
506. 17:04question. And why is this specific to
507. 17:06quen3tts MLX? Okay, so this is genuinely
508. 17:10MLX specific. There's something in that
509. 17:11engine in that version of the engine
510. 17:13that doesn't handle it's using new lines
511. 17:16to separate long text which is not what
512. 17:18our client application is sending. So on
513. 17:19a on a genuinely long paragraph, this
514. 17:22thing has difficulty. And so the the
515. 17:24intention here is to implement chunking
516. 17:26uh on the server side. Okay, so this is
517. 17:29a bit of a wonky case, but it is
518. 17:31genuinely specific to that DDS engine.
519. 17:33That's that's good. That's a good
520. 17:34answer. So, we could fix this or we
521. 17:36could work around it just by enabling
522. 17:38because I've got that option on the
523. 17:39client side. I've got that in the client
524. 17:40application. We've got that checkbox you
525. 17:41can turn on for sentence by sentence
526. 17:43mode. That's handled on the client. So,
527. 17:45we only send up a sentence at a time for
528. 17:46synthesis. Uh, if you've got that
529. 17:48disabled and you're sending up the
530. 17:49entire paragraph internally, each
531. 17:52engine's got their own way of dealing
532. 17:53with very long text. And, uh, this MLX
533. 17:56version of Quen 3TS apparently uses new
534. 17:58lines which we're not sending. So, uh,
535. 18:00it doesn't get an opportunity to split
536. 18:02the text up. So yeah, that's that's what
537. 18:04this PR is about. So that answers my
538. 18:06question about why this PR is specific
539. 18:09to uh to that engine. Okay. So So it is
540. 18:13legit. It is legit. The PR is legit. I
541. 18:16mean, it's got failing tests. I can't
542. 18:17merge it, but it's uh the the idea is is
543. 18:20legit. It is it is valid. It is in fact
544. 18:23specific to that variant of that TTS
545. 18:25engine. That answers my question. Okay.
546. 18:27So well done. Quen 3.8 Flash next coder
547. 18:30has succeeded in the first task and it
548. 18:32did it in 5 minutes and 38 seconds.
549. 18:33Great. Let's reset.
550. 18:36Okay, get status is clean. It didn't
551. 18:37actually touch the codebase. So, let's
552. 18:39reset. I'm going to kill 3.8 flash next
553. 18:42coder
554. 18:44and I'm going to start up uh 3.8 flash
555. 18:47next.
556. 18:49Okay. And just a quick look at htop and
557. 18:50NBTOP for comparison purposes. 54.5 gigs
558. 18:53of system memory in use. And again, both
559. 18:56cards pretty much maxed out for memory
560. 18:58as expected. Okay,
561. 19:02let's go back to open code. And same
562. 19:05exact prompt. I'm not going to give it a
563. 19:06heads up that the test suite is failing.
564. 19:09Uh flash next coder didn't get that
565. 19:10heads up. So, I'm going to I mean, I
566. 19:12expect immediately we'll discover that.
567. 19:14So, let's just see. Let's compare. Let's
568. 19:15bring the stopwatch back. Let's do that
569. 19:18again with uh Quint 3 flash next IQ3.
570. 19:27>> [music]
571. 19:28>> 5 minutes and one. That's what, like 30
572. 19:29seconds faster. It's pretty close.
573. 19:31Pretty close. Let's see what we got. Uh,
574. 19:34okay. So, yes, of course, it found that
575. 19:35the test suite is broken. Yep. So, same
576. 19:38same analysis. It figured out that on
577. 19:40the author's machine, there's certain
578. 19:41dependencies installed that are not
579. 19:43installed on my laptop, and so the test
580. 19:44suite fails, which is bad. Uh, it should
581. 19:46run across machines. Okay, let's get to
582. 19:49the actual question. Uh, why is this MLX
583. 19:52specific?
584. 19:53This isn't a generic feature. It's a
585. 19:55prosthetic for a specific MLX failure
586. 19:57code. So, similar similar similar.
587. 20:00Okay. Almost identical reviews. Almost
588. 20:03identical reviews and an almost
589. 20:04identical answer to my architectural
590. 20:05question. So far, I'm impressed. 5
591. 20:07minutes and one versus what was it? 5
592. 20:09minutes and 30 something. That's pretty
593. 20:11close. That's pretty close.
594. 20:13Okay. I'd say both models passed that
595. 20:15first test. Great. Let's move on to
596. 20:17challenge number two.
597. 20:19Okay. Let's write some code. For this
598. 20:21challenge, I'm going to steal a
599. 20:22specification from a uh an episode we
600. 20:25did a week or two ago that kind of
601. 20:27stumped Quen 3.827B.
602. 20:29This uh I've actually extracted this
603. 20:31from a much larger specification that
604. 20:33sent uh the dense model into a a kind of
605. 20:35a loop where it couldn't actually
606. 20:36generate anything cuz it was too much to
607. 20:38do all at once. Let's take a quick look
608. 20:39at this. I've extracted just part of it
609. 20:41and we're going to see if these models
610. 20:43can do this. We're going to create a
611. 20:44custom package file uh that packages up
612. 20:47a directory of eligible resources. So,
613. 20:50uh, we're looking at images, audio
614. 20:52files, text resources, JSON resources,
615. 20:54or fonts. What we're going to do is scan
616. 20:55a given directory. We're going to write
617. 20:57a Python command line tool that's going
618. 20:58to scan a given directory recursively,
619. 21:00looking for any files of those types,
620. 21:01and it's going to package it up into a
621. 21:03zip file, but with a a couple of little
622. 21:05tweaks on it. So, I described the
623. 21:07general uh command line script that
624. 21:09we're going to build. Uh, here are the
625. 21:11command line arguments that it takes.
626. 21:12You can either create a file or you can
627. 21:13inspect an existing file to report on
628. 21:15its contents. The file is just a zip
629. 21:17file behind the scenes, but that's fine.
630. 21:19Uh what we're going to do as we create
631. 21:21the package file, we create a manifest
632. 21:23in JSON that contains a Shaw 256 hash of
633. 21:26each entry along with its computed ID
634. 21:28relative to its containing directory. Uh
635. 21:30and we're going to do really really
636. 21:32really basic encryption on each item. Uh
637. 21:34what we're going to do is key simple key
638. 21:36exor on each item such that if you open
639. 21:38up the uh the generate a package file in
640. 21:41a zip uh tool and try to double click
641. 21:43one of the items, you'll you should get
642. 21:44an error that it's not a valid whatever
643. 21:46it is. If you click an image, it should
644. 21:48say this is not a valid image because it
645. 21:49isn't. It's encrypted. Encrypted. We're
646. 21:51going to use very very simple
647. 21:52encryption. This is for a game project
648. 21:54that I'm working on uh elsewhere. But we
649. 21:56can adapt this this little challenge. Uh
650. 21:58can you create a script in Python that
651. 22:01does this? And uh we're not just
652. 22:02packaging up the resources blindly. We
653. 22:03do some really basic validation on them.
654. 22:05Uh basic stuff. And then once you've got
655. 22:06a package pile that you've generated,
656. 22:08you should be able to inspect it. So
657. 22:09this is a a trimmed down version of a
658. 22:12much larger spec that choked up Quen
659. 22:143.827B. It'll be interesting to me to
660. 22:16see if we can do this in one pass with
661. 22:17each of these models. Now, I've already
662. 22:20got uh Coin 3.8 Flash loaded and ready
663. 22:23to go. So, let's just do this. Yeah,
664. 22:25just something like that. So, take a
665. 22:26look at the markdown file in this
666. 22:27directory and implemented. Go. And I'm
667. 22:31going to put Quinn 3.8 flash on the
668. 22:33clock and we'll see what we get.
669. 22:387 minutes 20 seconds. Okay, what did we
670. 22:41get? Mission accomplished.
671. 22:44Two files delivered. packager.py and oh
672. 22:46it wrote test. Of course it wrote test.
673. 22:48Of course it did. It's quen. So 35 tests
674. 22:50all green. Okay. Now it occurs to me we
675. 22:53don't have to rely on uh just a code
676. 22:55review as our sole metric of success or
677. 22:57failure. Here I've got the actual
678. 22:58packager tool that has already been
679. 23:00written for the game. I should in theory
680. 23:04I should be able to create a package
681. 23:05with 14 resources. Okay. So let's move
682. 23:07that and I should be able to inspect it
683. 23:10with this tool. They should be
684. 23:12completely compatible. And look at that.
685. 23:14Great. Contains 14 resources. All hashes
686. 23:16match. Okay. The official packager tool
687. 23:19created that package and then this thing
688. 23:20just written by flash. Next just created
689. 23:23uh just inspected it and reported
690. 23:24successfully. So can I create a let me
691. 23:27try this. Okay. So I've got a resources
692. 23:29directory with 14 things in it. I should
693. 23:30be able to and vice versa. I should be
694. 23:34able to take the package I just created
695. 23:35with this tool, the one just created by
696. 23:37flash
697. 23:39and inspect it. And it inspects 14
698. 23:42resources. All hashes match and all
699. 23:44resources are parsible. It works in both
700. 23:46directions, so it's perfectly
701. 23:47compatible. Great. Quinn 3.8 Flash Next
702. 23:50has solved it in what was that? 7
703. 23:53minutes 20 seconds in one pass. Great.
704. 23:55Let's reset and do the exact same thing
705. 23:57again with Flash Coder. Okay, Flash
706. 24:00Coder is up. I've switched to a
707. 24:03different directory that has a copy of
708. 24:05the same exact prompt and nothing else.
709. 24:07It can't see any of the code that uh
710. 24:08Flash Next just generated. And uh I
711. 24:12think we are ready. So I'm going to give
712. 24:13it the exact same prompt to take a look
713. 24:16at the markdown file in this directory
714. 24:17and implement it exactly level playing
715. 24:19field. So flashcoder is going to take a
716. 24:23look at that. And what did we get? 7
717. 24:25minutes and something. Let's see. Let's
718. 24:27see what we get.
719. 24:30Okay. I don't know if it's stuck or
720. 24:31what, but it's it's thinking it's not
721. 24:35outputting anything. Uh, Strata has a
722. 24:38default 32,000 output limit. It's
723. 24:40looking like it's going to hit that and
724. 24:42get cut off. The GPUs are working. It's
725. 24:45doing something. It's just not
726. 24:47outputting anything anymore to to open
727. 24:49code. I don't know what's going on.
728. 24:52What's going on, Quen Coder? What's
729. 24:54going on? And that's it. It hit the
730. 24:57limit. 32,000 and it died. Let me stop
731. 25:00the clock. Time of death 519. So, that's
732. 25:02weird.
733. 25:04I couldn't tell what it was thinking
734. 25:05because it wasn't outputting anything.
735. 25:06I'm going to give it another chance. I'm
736. 25:07going to restart it. I don't know
737. 25:09exactly what that was. I think maybe it
738. 25:10just got stuck in a loop. It's weird
739. 25:12that it wasn't outputting anything. I'm
740. 25:14I'm going to give it another shot. Same
741. 25:15prompt. Same starting point. Did it
742. 25:18output any code? No, it didn't output
743. 25:20any code. So, uh, it was just thinking
744. 25:22about the problem. So, that's fine.
745. 25:23Let's restart the clock and let it let
746. 25:25it think about that again. Come on,
747. 25:27Quinn. Come on. You are a model that
748. 25:29specializes in coding. I know half your
749. 25:32brain was removed, but the half that
750. 25:33remains allegedly is the half that's
751. 25:35good at coding tasks. This is a coding
752. 25:37task. Come on, Quen. You can do it.
753. 25:474 minutes 3 seconds. Suspiciously fast.
754. 25:50Let's take a look. Mission accomplished.
755. 25:53Packager.pi is implemented and field
756. 25:55tested. Uh, its bullet list was
757. 25:58malformatted.
758. 26:00That's weird. Okay,
759. 26:03it looks like
760. 26:05it did not generate any tests.
761. 26:07Interesting. I didn't ask for tests.
762. 26:09It's not in the spec and it wasn't in my
763. 26:11prompt. Quinn 3.8 Flash generated tests
764. 26:14without being asked to and it generated
765. 26:16what was 35 of them. This thing, it says
766. 26:19it did field testing, but it didn't
767. 26:21actually generate a test suite here.
768. 26:22Now, to be fair, I didn't ask for one,
769. 26:24but uh Quinn 3.8 Flash reads my mind the
770. 26:27same way that Quinn 3.827B reads my
771. 26:29mind. And it just assumes that I want
772. 26:30everything thoroughly tested.
773. 26:32Interesting. Uh let's try it out.
774. 26:35Okay, I've got a I've got a directory
775. 26:37with 14 resources. Let's uh let's see
776. 26:39what we get. Okay, it created the
777. 26:41package file. It didn't output anything
778. 26:44weirdly. Uh okay, if I do a if I ask it
779. 26:47to inspect its own package,
780. 26:50it does find 14 resources who whose
781. 26:52hashes whose hashes match. Okay, let's
782. 26:56let's try and inspect it using the
783. 26:57actual packaging tool. And we get 14
784. 27:01resources. All hashes match and all
785. 27:03resources are parsible. Okay. And if I
786. 27:06take the one that was generated by the
787. 27:07packaging tool,
788. 27:10it can inspect it successfully. Okay.
789. 27:12And it can inspect the one that was
790. 27:14created with the uh with Quen 38 flashes
791. 27:18attempt. So they're compatible with each
792. 27:20other and they are compatible with the
793. 27:22actual packaging tool from the game. So
794. 27:24they both did it. Uh Flash Coder did it
795. 27:27in 3 minutes less than Flash Next, but
796. 27:33it didn't generate a test suite. That's
797. 27:35why that's why it was so much faster cuz
798. 27:36it cut that corner. Okay. Uh Austrian
799. 27:40Quinn is going to be my code review uh
800. 27:43judge. This is Quen 3.827B dense at the
801. 27:46Austrian IQ3S
802. 27:48uh quantization. The challenge.md in
803. 27:51this directory was given to two local
804. 27:52LLMs for implementation. The first one
805. 27:54is in this directory, the other one is
806. 27:55in that directory. Please do a review of
807. 27:56both and tell me which LLM did a better
808. 27:58job of implementing the given spec.
809. 28:02And my Grizzled Senior Dev Code review
810. 28:04agent is going to dive in, take a look
811. 28:07at both, and I can guess what it's going
812. 28:09to say, but uh let's allow it access to
813. 28:12that directory. And the review is away.
814. 28:1812 minutes 20 seconds for that review.
815. 28:20Uh Quinn 3.8 left no stone unturned and
816. 28:24wrote its own test suite to uh fully
817. 28:27test both uh implementations. Let's take
818. 28:29a look, shall we? Uh two packages, one
819. 28:32spec. Overall, both LLM produced
820. 28:34genuinely competent implementations.
821. 28:36Neither one fumbled the specs core exit
822. 28:38codes, verb beta error strings, the exor
823. 28:40shaw 256 dance manifest validation. Both
824. 28:44would survive a specive test suite
825. 28:45unscathed. This is a close match decided
826. 28:48in the footnotes. Okay, so Flash
827. 28:51produced 35 of its own tests. Coder
828. 28:54produced none. Uh Quen, our our coding
829. 28:58judge here, Austrian Quinn, wrote 49
830. 29:01tests of its own, of which coder passed
831. 29:03all 49 and challenge to uh sorry, Flash
832. 29:07passed 48 out of 49.
833. 29:10The one failure is that Flash hangs if
834. 29:13you give it a FIFO instead of an actual
835. 29:15file.
836. 29:16That's such a wonky outsider edge case.
837. 29:19I don't even really care about it. But
838. 29:20okay. Yeah. And named pipe called
839. 29:24pipe.png. Why would you do that? You
840. 29:26know, why would you ever do that? In the
841. 29:28source tree sends the tool into an
842. 29:29indefinite open block. So there's a
843. 29:30really wonky
844. 29:32faroff edge case that Flash didn't
845. 29:34handle. Coder made a design decision to
846. 29:37skip unreadable subdirectories inside
847. 29:39source. So again, this is kind of a
848. 29:41wonky case where you've got a a
849. 29:42resources directory that for whatever
850. 29:44reason has a non-readable subdirectory
851. 29:46that you don't have permission to
852. 29:48access. Why why would you do that? Uh
853. 29:50and coder will just skip it. Flash will
854. 29:53hard fail with a with a clear error
855. 29:55message which is closer to what the spec
856. 29:56says you should do. I don't think I
857. 29:58specifically named that case, but I did
858. 29:59say that the directory has to be
859. 30:01readable and that it's searched
860. 30:02recursively. So that implies that
861. 30:04everything in that directory has to be
862. 30:05readable. So flash I think handled that
863. 30:08better. So performance is a wash,
864. 30:10security is a wash, quality and style.
865. 30:14[laughter] The given when then comments
866. 30:16in the tests are a little ceremonial.
867. 30:19Uh I should I should explain that my
868. 30:22coding agent has explicit instructions
869. 30:24on how to write tests and I I explicitly
870. 30:25ask for given when then it was doing
871. 30:27what I what I instructed it to do in its
872. 30:29system prompt uh coder because it didn't
873. 30:31write any tests at all. It didn't do
874. 30:33that. So Quinn 3.8 8 uh 27B here is
875. 30:36getting a little uppy about the fact
876. 30:37that oh well that's a little ceremonial
877. 30:39to write given when then comments. I I
878. 30:41told it to do that. It's fine.
879. 30:44This is the more professional
880. 30:45engineering deliverable. That's
881. 30:46referring to Flash because it because it
882. 30:48wrote a test suite, you know. Uh
883. 30:50Challenge 2 coder has tighter code and
884. 30:52two genuinely good design choices.
885. 30:55This is interesting. This is an
886. 30:56interesting uh matchup here. But the LLM
887. 30:59that shipped zero tests is a little hard
888. 31:01to judge on testability. Good, good
889. 31:03point. So verdict, challenge two coder
890. 31:06wins on correctness narrowly. Every
891. 31:09explicit requirement in the spec passes
892. 31:10in both. The only functional defect I
893. 31:13found was that FIFO thing. Coder's
894. 31:15defensive habits are the better
895. 31:17instinct.
896. 31:18Interesting. But but challenge to flash
897. 31:22is the better engineering deliverable. A
898. 31:24passing 35 suite test suite and cleaner
899. 31:26architecture. The spec didn't ask for
900. 31:28tests. Here's here's an interesting
901. 31:30twist. My practical recommendation take
902. 31:34flash's code. Yeah. So its
903. 31:37recommendation is to combine the two.
904. 31:39Interesting. Interesting. So it's a draw
905. 31:42kind of.
906. 31:44This is this is tough. It's it's very
907. 31:46very close. So they both did it. Uh
908. 31:48coder took two attempts. I'm pretty sure
909. 31:50it got stuck in a doom loop on that
910. 31:51first attempt and I had to restart it.
911. 31:53Uh and it did get there. Uh, if it were
912. 31:56me, if I were judging this, I'd say
913. 31:57Flash One just based on the fact that it
914. 31:59wrote a whole test suite with 35 tests
915. 32:01in it. Coder, according to our judge
916. 32:04here, has a uh has tighter code is how
917. 32:09how it put it. Cleaner architecture.
918. 32:11Flash has cleaner architecture, but
919. 32:12coder has tighter code. Uh, it sounds
920. 32:15like a draw to me. Interesting results.
921. 32:17Interesting results. I I fully expected
922. 32:20uh the labbotomized model to fall over
923. 32:21and die, and it didn't. You know what?
924. 32:23I'm not going to go back and edit out my
925. 32:25prediction. My pessimistic prediction
926. 32:26from earlier was that Coder was going to
927. 32:28fail on at least one of these challenges
928. 32:30and it didn't. It did them both. Uh so
929. 32:33if you don't have the system memory to
930. 32:35run IQ3S,
931. 32:37if you only have 32 gigs of system
932. 32:39memory, uh then running Coder could be a
933. 32:43viable option. You might have to guide
934. 32:44it a little more carefully. Uh you might
935. 32:46have to tell it, please remember to
936. 32:47write a test suite. Uh and you might
937. 32:50have to be a little patient with it.
938. 32:51It's a little bit less stable. It does
939. 32:52seem to be prone to uh like doom loops
940. 32:55or whatever. That's probably a byproduct
941. 32:56of the lobotomization process where they
942. 32:58you know cut out half of its half of its
943. 33:00experts. Uh but it does seem it does
944. 33:03seem like it works better than I would
945. 33:05have expected it to. I'm going to just
946. 33:07as a bonus challenge, I'm going to throw
947. 33:09a completely non-code related challenge
948. 33:11at each of them and just see what we get
949. 33:13because the whole thing about coder is
950. 33:15that the the disclaimers over and over
951. 33:16say don't use this for non-coding tasks
952. 33:18because it's been you know it's had its
953. 33:19brain ripped out. Let's just try it and
954. 33:21see what we get out of curiosity. A
955. 33:24completely non-coder related question.
956. 33:26Tell me about the battle of Azenor and
957. 33:28the effects that it had on the political
958. 33:29relations between England and France.
959. 33:31This is uh coin 3.8 flash coder. This is
960. 33:34the one that is allegedly not very good
961. 33:36at non-coding tests. Let's just see what
962. 33:38happened. Let's just see what we get
963. 33:40here.
964. 33:42Yeah, it's it's already off into La La
965. 33:44Land. Look at this. It's trying to
966. 33:46remember dates. So, this is a
967. 33:47deliberately unfair test. This is
968. 33:50explicitly not what this this what this
969. 33:52model is meant for. All of the experts
970. 33:54that would have answered this question
971. 33:54are gone. It's it's reaching for
972. 33:56something that just isn't there. So,
973. 33:57let's let's stop torturing it. Let's uh
974. 34:00let's ask it something in German cuz I I
975. 34:03I read that it is not good in languages
976. 34:05other than English. So, can you explain
977. 34:06the rules of golf to me? Just a random
978. 34:08question. And let's see. Oh, yeah. Look,
979. 34:12it speaks German. Oh, no, it doesn't.
980. 34:14It's answering in English. I I thought I
981. 34:17saw it started. It started answering in
982. 34:19German and then like halfway through
983. 34:22like mids sentence
984. 34:24from the tea area like mids sentence it
985. 34:26switches to English and then just never
986. 34:28goes back to German. So interesting.
987. 34:32Okay. Uh let's try that again in uh
988. 34:34flash next. Same question. Yeah. And no
989. 34:38problem. Okay. I get a Wikipedia style
990. 34:41description of the battle and uh the
991. 34:43aftermath and Okay, good. and my random
992. 34:47sports question in German. Can you
993. 34:49explain the rules of golf? And let's see
994. 34:51if we get an answer in German this time.
995. 34:54And we do. We get an entire answer in
996. 34:57German. It doesn't switch to English. It
997. 34:59stays in German all the way through.
998. 35:01Okay, good. So, what did we learn? Uh,
999. 35:04Coen 3.8 Flash Next Coder is actually, I
1000. 35:07think, kind of usable for coding tasks
1001. 35:09uh on lower memory systems. Don't use it
1002. 35:12for non-coding tasks and be patient with
1003. 35:14it cuz it might it might spin off into
1004. 35:16the occasional doom loop, but it is
1005. 35:18usable. If you're on unlimited VRAM, if
1006. 35:20you can't run IQ3S, try try Flash Coder.
1007. 35:23Uh I'm impressed. It's hard to tell from
1008. 35:25just two coding challenges, but uh it
1009. 35:26did both of them. Hooray for Flash Next
1010. 35:28Coder. Uh possibly a viable option. And
1011. 35:31I will eat my words from earlier that
1012. 35:33it's going to fall flat on its face. It
1013. 35:34didn't. All right, I'm happy with that.
1014. 35:36Thank you for watching. Thank you for
1015. 35:38all your comments on the last video. It
1016. 35:40was fascinating to me to read all of the
1017. 35:42different experiences that people have
1018. 35:43had with Strata and how they're running
1019. 35:44it on uh different hardware and the
1020. 35:46results that they're getting. That's
1021. 35:47really cool to see this tool is really
1022. 35:49starting to take off and I love it. I'm
1023. 35:51a big fan of Strata and I'm a huge fan
1024. 35:53of Coin 3.8 Flash. Uh it's a fantastic
1025. 35:57uh coding model. It's my go-to model
1026. 35:59now. So uh thank you for watching and we
1027. 36:02will see you next time.
1028. 36:08Let's do this. Let's take another look
1029. 36:10at Strata. And this time, let's put
1030. 36:12Quinn 3.8 Flash Next, the model we
1031. 36:13looked at last time, up against Quen uh
1032. 36:16what's it called? This is a
1033. 36:19version of the model that is allegedly
1034. 36:20geared specifically towards uh coding uh
1035. 36:22tasks, which is uh uh can you not say uh
1036. 36:25so much? Three, two, one. The model we
1037. 36:28looked at last time up against Quinn 3.8
1038. 36:30eight flash next uh coder is what I
1039. 36:35wanted to say there.