【ローカルLLM】32GBメモリで今どのモデルを使う?Qwen4を待つのをやめて Flash Next にした話 — Transcript
Full transcript
- 0:01No way, I’ve been so anxious all week,
- 0:03but I think everyone’s starting to
- 0:06realize it: Qwen 4 isn't coming out, is
- 0:08it?
- 0:11It's like, "I guess Qwen 4 isn't
- 0:13happening."
- 0:18I mean, since last week, everyone kept
- 0:20saying it was going to drop, kept going
- 0:22on and on about it.
- 0:26I really thought it would, but the 27-
- 0:28billion parameter one didn't show up,
- 0:31and I was just, like, totally crushed.
- 0:37Just completely gutted.
- 0:40Like, uh, Kodak or something.
- 0:45You know, it had been kept under wraps
- 0:47like some sort of hidden model, but it
- 0:50turns out the one everyone thought was
- 0:5227 billion was actually just Qwen 4 Max
- 0:54.
- 0:59And Qwen 4 Max is available now as a
- 1:01preview model for, like, two weeks at a
- 1:04super cheap price.
- 1:08But honestly, every time I use Qwen 4
- 1:11Max, I’m just like, "Eh, it's not that
- 1:13great," and keep judging it as
- 1:15underwhelming.
- 1:21Since Qwen 4 didn't drop after all, I’ve
- 1:24realized it’s a lost cause, and now I’m
- 1:26thinking there’s a real chance it won't
- 1:28be released this month.
- 1:34It's looking like it might not come out
- 1:36this month after all.
- 1:39At the very least, I don't think the
- 1:40next step will happen until this
- 1:42two-week preview ends.
- 1:47So, I’ve decided to stop holding my
- 1:49breath for Qwen and start looking into
- 1:51something new.
- 2:00On the other hand, there are actually
- 2:01some new movements happening outside of
- 2:03Qwen.
- 2:07Where was it again?
- 2:11There’s talk about a model called Ernie
- 2:13, or something like that, from Baidu
- 2:15coming out late this month, so things
- 2:17outside of Qwen are starting to look
- 2:19active again.
- 2:23Yeah.
- 2:26I think a model based on Qwen 3.8 Flash
- 2:28Next that’s been further fine-tuned
- 2:30might have even come out today.
- 2:34Wait, what was that model called? What
- 2:36was it again?
- 2:42There are models like the Edge 145B and
- 2:44120B, which seem to be built by applying
- 2:47the structure of Flash-Next directly, so
- 2:50I think this might become a new trend.
- 2:59However, when you look at the licensing
- 3:01, it’s quite complicated, and, honestly
- 3:03, you can't really use them for
- 3:05commercial purposes.
- 3:08So, when it comes to business use, it’s
- 3:10a model that’s a bit hard to choose.
- 3:18Right now, I'm focusing on a slightly
- 3:20older model, DeepSeek version 4, the V4
- 3:23Flash, and I've been wondering if I
- 3:25could run it locally.
- 3:30Well, not just wondering, I was actually
- 3:32thinking about that. The V4 model is
- 3:34actually a bit smaller in size. What was
- 3:37it again? It was around 280, something
- 3:40like that.
- 3:42The active parameters are around 12
- 3:44billion, so, in other words, since the
- 3:47active size isn't that large and the
- 3:49total size isn't, you know, 500 or
- 3:51something, I thought it might be
- 3:53interesting if I could get it running,
- 3:55so I tried it out.
- 3:59It really is satisfying when it works.
- 4:03Surprisingly, the speed was 2.6 tokens
- 4:06per second.
- 4:08On an M6 Mac.
- 4:10"Wow, that's slow," I thought to myself
- 4:12. Yeah.
- 4:15But even so, the fact that it actually
- 4:18runs was a bit fresh for me; it was like
- 4:21, "Wow, DeepSeek V4 is actually running
- 4:23," so I was quite moved.
- 4:30Still, after trying it out, I realized
- 4:32it is slow; 2.5 tokens per second is
- 4:35just damn slow, and it ended up being
- 4:38just a romantic experiment.
- 4:42Yeah. I definitely don't think I'll be
- 4:44using this for practical work.
- 4:47Yeah.
- 4:52When you think about it that way, Qwen
- 4:543.8 Flash-Next reaching 17 tokens per
- 4:57second is really something quite
- 4:59impressive.
- 5:07I wonder, Qwen is probably strong when
- 5:09it comes to smaller models, I guess.
- 5:14I'm actually using the 27-billion
- 5:16parameter Qwen 3.8 Flash-Next model
- 5:19again lately.
- 5:23The other day, Anthropic announced a new
- 5:25quantization method, maybe it was
- 5:27Dynamic 3.1 or something.
- 5:32They released something like that, which
- 5:34apparently improves the accuracy even
- 5:36further.
- 5:40So, even with the same quantized model,
- 5:43the size goes up just a little bit, but
- 5:45the performance improves by about 10%,
- 5:47and stuff like that is popping up here
- 5:50and there.
- 5:54Even when I try playing around with it,
- 5:56I feel like Anthropic's model is more
- 5:58stable.
- 6:03It feels like there's less garbled text
- 6:05, so I've been using Anthropic Dynamic's
- 6:08Qwen 3.8 Flash Next, the one they call
- 6:11the 2-bit model, IQ2XS.
- 6:19But in reality, if you add up all the
- 6:21quantization settings and divide them,
- 6:24it apparently comes out to 3.5
- 6:25quantization.
- 6:30It’s like 3.50 BPW, so it’s actually
- 6:33quite strong.
- 6:35Yeah.
- 6:38So, the number feels a bit like a lie,
- 6:41but that’s why I use that Anthropic Qwen
- 6:443.8 Flash Next model quite regularly.
- 6:52I run that using the Splash engine, and
- 6:54as for the tool, I use an agent tool
- 6:57called Hermes Agent to interface with it
- 7:00—so, Qwen 3.8 Flash Next via Splash.
- 7:07And the interface is the Hermes Agent
- 7:09that I have installed on Windows.
- 7:12Splash is already at 1.6.1, they say.
- 7:17Yeah, Splash has been updated to 1.6.1.
- 7:24If you haven't updated yet, please make
- 7:26sure you do.
- 7:28Memory efficiency has improved
- 7:30significantly, so it’s definitely better
- 7:32to update.
- 7:37It’s gotten a lot better with memory
- 7:39efficiency and how it handles SSD
- 7:41caching, so it can process things more
- 7:44stably.
- 7:50Yes. And sorry to keep asking. How much
- 7:53speed do you get when running Qwen 3.8
- 7:55Flash Next with 32GB of memory? In my
- 7:58current setup, it runs at 16.7 tokens
- 8:01per second. Yes. About 40 watts.
- 8:05That is, the power consumption.
- 8:09This fluctuates quite a bit depending on
- 8:11what kind of questions or input you
- 8:13throw at it.
- 8:15>> Yeah.
- 8:19>> So, well, during standard agent
- 8:22operations, it sometimes hits 20 tokens
- 8:24, but occasionally, in turns where it
- 8:26needs to think things through, it might
- 8:29drop to 11 or 12, so the token speed is
- 8:31actually quite volatile.
- 8:37Well, I think that kind of fluctuation
- 8:39is probably the same for any model.
- 8:41Still, you can definitely have plenty of
- 8:43fun with it.
- 8:46Yeah.
- 8:48Yeah.
- 8:50Lately, though, it feels like things
- 8:53have come full circle, and running AI—
- 8:55or rather, running local LLMs—has become
- 8:58quite stable with Splash.
- 9:01Though, I do sometimes find myself
- 9:03wondering what exactly I should be
- 9:05running on Splash.
- 9:10I go back and forth between running a 27
- 9:12-billion parameter 4-bit quantized model
- 9:15or something like Flash-Next 3.5, or a
- 9:183.5 B model, but when I really need
- 9:20processing speed, I know I should stick
- 9:23with the 27B.
- 9:31However, when I'm unsure about a
- 9:33decision or really want it to think
- 9:35things through, Flash-Next is clearly
- 9:37the stronger choice.
About this transcript
This page contains the full transcript of 【ローカルLLM】32GBメモリで今どのモデルを使う?Qwen4を待つのをやめて Flash Next にした話 by ふじもったチャンネル, generated from the public captions YouTube serves with the video. The transcript has 1,100 words across 189 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.