Qwen3.8-Flash-Next + MTPLX 2.10: INSANELY Fast Local Coding Agents! — Transcript

Full transcript

1. 0:00Hi guys, welcome back. In this video
2. 0:02we'll be talking about how you can
3. 0:03achieve almost 70 tokens per second with
4. 0:06Qwen 3.8 Flash Next Imbo e model. We
5. 0:10know that it is a 125 billion parameter
6. 0:12model, but achieving a 70 tokens per
7. 0:15second is kind of impossible these days,
8. 0:17especially if you're running a model in
9. 0:20the Apple M5 Max 128 GB RAM machine like
10. 0:24what I have got. And I'm actually
11. 0:26getting almost 70 tokens per second, but
12. 0:28if I'm going to be using with coding
13. 0:29agent I'm getting around 50 to 60 tokens
14. 0:31per second. And in this video I'll be
15. 0:33talking about the MTP LX which has
16. 0:35released a brand new version which has
17. 0:37got the powerful features which makes
18. 0:40the inferencing way more faster within
19. 0:42your local machine. And we'll be talking
20. 0:44about the top five features of the MTP
21. 0:47LX version 2.10 in this particular
22. 0:49video. So, the first feature is the Qwen
23. 0:513.8 Flash Next support. So, they have
24. 0:54got the Qwen 3.8 Flash Next support
25. 0:56which has got the draft token along with
26. 0:58the MTP support as well which actually
27. 1:01makes things more faster while you're
28. 1:03going to be doing inferencing. The
29. 1:04second feature they have got is the NVMe
30. 1:07SSD with N-gram support as well. So,
31. 1:09essentially you can now load the entire
32. 1:12model within your SSD instead of loading
33. 1:15everything on the virtual memory like
34. 1:17how we did with the OMLX before. The
35. 1:19same idea goes in here as well, they
36. 1:20support it. The third feature is the
37. 1:23long context optimization that the MTP
38. 1:26LX has got. So, it now supports almost
39. 1:28147,000
40. 1:30plus token with zero collapse, which
41. 1:32means now you can work with the longer
42. 1:34context without any problem. That's what
43. 1:36exactly this particular feature is. And
44. 1:38the fourth feature that they have got is
45. 1:39the memory aware KV caching. So, if
46. 1:42you're going to be working with the
47. 1:43traditional tools, you'll always have
48. 1:46the RAM full pretty quickly and you'll
49. 1:49have the swap chaos and slow down of
50. 1:51your entire inferencing, but with the
51. 1:54memory aware KV option, you can now do a
52. 1:58memory budget and controlled SSD spill
53. 2:02in that particular category. That's what
54. 2:04the the author has told over there. And
55. 2:06the last and the final feature that the
56. 2:08author has told is the agent loop
57. 2:10support. Essentially, it will actually
58. 2:12help do the inferencing with the coding
59. 2:16agent way more faster. So, if you're
60. 2:18using PyCodingAgent or Open Code,
61. 2:20whatever that you like, you can see that
62. 2:23without MTPLX the actual coding will be
63. 2:26way more slower because it's going to do
64. 2:27a lot of thinking before it answers for
65. 2:29any of the output, but with the MTPLX
66. 2:33support, you can actually see the same
67. 2:35exact Qwen 3.8 Flash Next model, you are
68. 2:37going to see a faster inferencing at the
69. 2:40same time faster tool calling and
70. 2:42everything goes quite fast in here. So,
71. 2:44the the agent loops gets more and more
72. 2:47responsive. And also, the Flash Next
73. 2:49coding agent also has got a default
74. 2:52medium setting by the time you're going
75. 2:54to be launching the particular model,
76. 2:55but if you're going to be running the X
77. 2:56high settings, things will be bit more
78. 2:59slower, but it will have a quality
79. 3:01output as well. That's what the author
80. 3:02actually says in here. Well, as they
81. 3:04said, in the MTPLX, the author has
82. 3:07released two optimized model of the same
83. 3:10Qwen 3.8 Flash Next. The first model
84. 3:12that they have got is the 4-bit
85. 3:16quantization, which has got faster
86. 3:18inferencing, so it is a speed-oriented
87. 3:20model, and there is a 8-bit
88. 3:22quality-oriented model, which has got
89. 3:25slower inferencing speed, obviously, but
90. 3:27the precision will be higher. So, with
91. 3:29the 4-bit quantization, the speed will
92. 3:31be more faster, but with the 8-bit
93. 3:34quantization, you're going to see the
94. 3:36quality will be better over there. So, I
95. 3:39have downloaded the 4-bit quantization
96. 3:41in this particular demonstration, and I
97. 3:43will show you how the 4-bit quantization
98. 3:46model of the MTPLX is actually related
99. 3:50with the actual 4-bit quantization that
100. 3:53I will be running with the OMLX as well.
101. 3:55So, I'll be comparing both of them this
102. 3:56particular video and I'll show you which
103. 3:58one actually does a better job and which
104. 4:00one actually is doing
105. 4:03a better coding as well as better image
106. 4:05generation. So, I'll be doing the same
107. 4:07exact two tests that I was doing in my
108. 4:08earlier video as well with the OMLX
109. 4:11running the Qwen 3.8 27 billion
110. 4:14parameter model and the Flash Next
111. 4:15model. So, the same exact kind of test
112. 4:17which I've been doing over here as well
113. 4:19with the MP PLX and the OMLX to do a
114. 4:21comparison and let's see which one
115. 4:23actually wins. Well, as that said, this
116. 4:25is the MP PLX website as you can see
117. 4:27over here and they have got the releases
118. 4:29page. So, if you're going to go there,
119. 4:31so this is the latest version that I'm
120. 4:32talking about. You can see that the 2.10
121. 4:36dot zero, it has got faster everywhere
122. 4:39features, memory planned on your Mac,
123. 4:41Qwen Flash Next support, agent finished
124. 4:44faster and stopped dying, and long work
125. 4:47hold up. These are the features that I
126. 4:48was talking about. And this is the full
127. 4:50release notes that you can go and read
128. 4:52if you wanted to. And in the meantime,
129. 4:53the author also released a new feature
130. 4:56just right now and this is the feature
131. 4:58that they have got over here as well.
132. 5:00So, this is pretty amazing and I wanted
133. 5:03to quickly try out and see how things
134. 5:04work. So, I have already downloaded the
135. 5:06model over here, the Qwen 3.8 Flash Next
136. 5:09bare speed model which is the 4-bit
137. 5:11quantization that I was talking about
138. 5:13and I will be running it in the chat
139. 5:15window. I'll show you how faster it is
140. 5:17going to be doing the inferencing there.
141. 5:20So, the moment I'm going to be opening
142. 5:21the chat over here and I'm going to be
143. 5:24creating a new chat
144. 5:26here. So, the model is currently loading
145. 5:28and if I'm going to go and open an
146. 5:31activity monitor over here, you see that
147. 5:34it's only using the 92 GB of the memory
148. 5:36there because it's doing an N-gram SSD
149. 5:39offloading. So, it's not going to be
150. 5:41filling up the entire VRAM that you have
151. 5:44got. That is one feature that the author
152. 5:45has told. Amazing. And now what I'm
153. 5:48going to do is I'm just going to say
154. 5:50"Hi" over here. You see that it's pretty
155. 5:53faster, 63 tokens per second. And if I'm
156. 5:55going to say "Hi" again,
157. 5:58and it's going to be 61 tokens per
158. 6:00second. But you can actually change
159. 6:02these settings as well. So, you see that
160. 6:04the reasoning I have set this to medium
161. 6:06by default over here. And the fan mode
162. 6:09is smart. So, you see that sometime you
163. 6:10hear the fan noise pretty faster
164. 6:13cranking. So, yeah, that's what the
165. 6:15context window that window that I have
166. 6:16set over here. So, let's say I'm going
167. 6:17to be setting a context window of 128.
168. 6:20And the reasoning as X high.
169. 6:23And yeah, that's all the setting that I
170. 6:25have to do. And if I'm going to run the
171. 6:27same exact thing right now as well,
172. 6:30you'll see that there is not a lot of
173. 6:31performance drop or anything like that.
174. 6:33You see that it's 73 tokens per second.
175. 6:35And if I'm going to say "Hi" again, it's
176. 6:37going to do a bit of a thinking at 53
177. 6:39tokens per second. "Hi, hi" again.
178. 6:42Something like that. And it's just going
179. 6:44to be working as well. 50 50 tokens per
180. 6:47second. So, 73, 53, 50.
181. 6:51And is it keep on dropping like 43 or
182. 6:53something like that? No, it's just going
183. 6:54to keep on improving, right? So, that's
184. 6:56what this particular model is doing at
185. 6:58the moment. So, what I'm going to do
186. 6:59right now is I'm going to be doing a
187. 7:00test with the PyCoding agent and see how
188. 7:04things are going to be working. So, you
189. 7:05can see that this is my activity
190. 7:07monitor. Total GB that I have got. I'm
191. 7:09going to again load the PyCoding agent
192. 7:13over here. So, the moment you're going
193. 7:15to be selecting from there, it's going
194. 7:17to open a terminal for me. And you see
195. 7:19that the memory is starting to spike up
196. 7:21there. So, just filling up 93 gig this
197. 7:25time. So, 94. There we go. I have got
198. 7:28the terminal opened. And let me close
199. 7:31this. And I'm going to be using the
200. 7:34PyCoding agent from here in my other
201. 7:36terminal. And you see that the model
202. 7:38that I have chosen is the MTPLX flash
203. 7:40next to bear speed model.
204. 7:42And I'm going to be doing a migration
205. 7:44test this time with a product API to dot
206. 7:46net 10 and dot net 8 the same kind of
207. 7:48test that you might have seen before.
208. 7:50This is the product that I have got
209. 7:52and I'm going to be doing the migration.
210. 7:54There is a breaking change in the code.
211. 7:56We already know that.
212. 7:57So I just wanted to see if this is going
213. 7:59to happen. Actually in this particular
214. 8:01test the one thing which I realized is
215. 8:02that the entire test took just 5 minutes
216. 8:06to do the migration.
217. 8:08And this is pretty interesting to be
218. 8:11honest. The same exact model running in
219. 8:14the MTP LX it's not very same exact
220. 8:16model is a bit more change there. But it
221. 8:19is actually running pretty fast like 5
222. 8:21minutes. It has completed the entire
223. 8:23migration for me over here to dot net 10
224. 8:26as you can see over here.
225. 8:28And the code is also building
226. 8:30successfully for me over here. And you
227. 8:32can see that the total number of cash as
228. 8:34well as the memory usage in the context
229. 8:36is also not quite bad at 567 tokens per
230. 8:39second all the time which is quite
231. 8:40amazing. And I'm going to do the exact
232. 8:43same kind of test with the Quinn 3.8
233. 8:45Flash next model
234. 8:47with 4-bit quantization in the OMLX that
235. 8:49that that I have got. And when I run the
236. 8:52same exact test over there
237. 9:35>> exact kind of image generation test over
238. 9:38here. I'll be doing two tests this time.
239. 9:40One is the the pelican riding the
240. 9:41bicycle in the New York City. I'll be
241. 9:43running the test with two settings. One
242. 9:45is the medium
243. 9:47thinking as well as in the X high
244. 9:50thinking. I will not show you how the
245. 9:52output is unless until I'm going to be
246. 9:54fully completing the test there. So, the
247. 9:57first test is completed with the medium
248. 9:59thinking level. Now, I'm going to be
249. 10:01changing this to X high thinking over
250. 10:03here. I'm going to be running it. So,
251. 10:06last time it took around 2 minutes and
252. 10:08I'm going to run the same exact test
253. 10:09over here with the SVG and I'm going to
254. 10:12be running it as well. It's It took less
255. 10:15than
256. 10:162 minutes this time, like 1 minute
257. 10:1849 seconds or 50 seconds as you can see
258. 10:21over here. It has completed the image
259. 10:24generation for me right now. And now,
260. 10:26I'm going to be running the same exact
261. 10:29test with OMLX over here with the 4-bit
262. 10:32quantization model.
263. 10:33And I'm going to be running the test
264. 10:37right now and you can see that it is
265. 10:40going to be running almost like 21
266. 10:42minutes and it has completed the test
267. 10:45for me right now. And now, it's a
268. 10:47showtime for me. So, these are the
269. 10:48images which are actually generated by
270. 10:51the MTPLX flash over here. You see that
271. 10:55this is what is the image generated
272. 10:58in a highly traffic New York City. Two
273. 11:01pelicans riding the bicycle. I really
274. 11:03like the fact that now the model has
275. 11:06taught quite well within like 2 minutes.
276. 11:09It has created so much a rich image over
277. 11:11here, like two pelicans are riding the
278. 11:13same bicycle like and the the pelicans
279. 11:16are riding it. And you see that there is
280. 11:17a there is a truck kind of
281. 11:21and there is a taxi kind of and there
282. 11:22are some
283. 11:24like buildings there which looks okay,
284. 11:27to be honest. This is what the medium
285. 11:28settings of the Quen
286. 11:31running the MTPLX look like. And if I'm
287. 11:33going to show you how it's going to look
288. 11:35like for the XHigh setting, you see that
289. 11:38with XHigh, it has got a bit of a change
290. 11:42in terms of the cosmetic there. But I
291. 11:44don't really see there is a lot of
292. 11:46change. But I mean, to be honest, it
293. 11:48doesn't look to be like quite amazing to
294. 11:50me over here. But then, I'm going to
295. 11:53show you the same image which is
296. 11:54generated with the OMLX for me, the same
297. 11:59exact model and same exact quantization,
298. 12:02the image is going to be super rich. You
299. 12:04see that? This looks super awesome
300. 12:07comparing to what the
301. 12:10MTP-LX has got over there. Do you see
302. 12:12that? This is XHigh which is pretty
303. 12:14close to what this guy is doing over
304. 12:16here. But I can't really imagine like
305. 12:19how the images are quite bad in here.
306. 12:22All right? For the XHigh with the MTP-LX
307. 12:25as opposed to the OMLX over here. I know
308. 12:28the OMLX took like 21 minutes to
309. 12:30generate, whereas this guy took only 1.2
310. 12:33minutes to generate this particular
311. 12:34image, but the quality of the output is
312. 12:37pretty close to the dense model which is
313. 12:39the Qwen 3.8 27 billion parameter model
314. 12:42which was generating, as you can see
315. 12:44over here. So, the difference with the
316. 12:47the Flash Next model with the dense
317. 12:50model is not a lot over here. It's
318. 12:53pretty much exactly similar as I can see
319. 12:56over there. All right? But the only
320. 12:58thing is the OMLX is actually doing
321. 13:00quite good in terms of using the entire
322. 13:02power of the model. I mean, it's not
323. 13:03really doing a fast inferencing at 21
324. 13:05minutes it did to generate this
325. 13:07particular image, but it has got that
326. 13:09particular image quite neat
327. 13:10comparatively. But long story short, if
328. 13:12you're using any coding agent and if you
329. 13:14want to do a quick migration or write a
330. 13:17code or maybe fix a code in a quick way,
331. 13:20probably I see MTP-LX inferencing is
332. 13:23quite faster, especially with the coding
333. 13:25agent like PyCoding agent or open code.
334. 13:27It looks quite faster. Whereas in the
335. 13:30OMLX case, it's entirely different. It
336. 13:32takes a long time to do the migration or
337. 13:34the image generation, but it actually
338. 13:37helps you to a quality output there.
339. 13:41That's what I think between OMLX versus
340. 13:43the MTP LX. What do you guys think and
341. 13:46what do you see the output in your
342. 13:48machine while you're running the same
343. 13:49exact kind of model in your [music]
344. 13:51machine, maybe Apple's M1 Max or some
345. 13:54other CUDA base machine. Once again,
346. 13:56thank you so much for watching this
347. 13:57video. Catch you in the next one.