Is Qwen 3.8 27b The New King of the Local LLMs? — Transcript

Full transcript

1. 0:00So Quen 3.8 27 billion parameter local
2. 0:03AI model has been out for a few days now
3. 0:06and I've been testing it to see what it
4. 0:08is capable of and I've been surprised on
5. 0:10how well it is performing especially
6. 0:12compared to the models that we had what
7. 0:14just a few weeks ago. This is another
8. 0:17major step change in what you can
9. 0:19achieve locally on local hardware
10. 0:22without having to send everything up
11. 0:24into the cloud. That's the topic of
12. 0:26today's video. So if you want to find
13. 0:27out more, please let me explain.
14. 0:32So Quen 3.8 is a 27 billion parameter
15. 0:35model. Now it's full size at 16 bit
16. 0:38floating point numbers. It's a 55 GB
17. 0:41download and you're probably going to
18. 0:43need 128 GB of RAM to run it, which is
19. 0:46beyond what most people have. That's
20. 0:48VRAM. Beyond what most people have kind
21. 0:51of at home, even with a very good gaming
22. 0:53setup. However, if you do have something
23. 0:56like an RTX 5090 or something else with
24. 0:5932 GB of VRAM, then the [snorts] 4bit
25. 1:03quantized version of Quen 3.827 billion
26. 1:06is runnable on your local PC and
27. 1:10actually at a fairly decent rate. I was
28. 1:12getting 96 tokens a second uh running it
29. 1:15on an RTX5090. That means you can run it
30. 1:17locally on your PC and do everything
31. 1:20you'd want to do with a normal large
32. 1:22language model out in the cloud, but now
33. 1:24you're running it locally. And I've been
34. 1:26testing it specifically for how good is
35. 1:28it at you doing coding using open code
36. 1:31as the coding harness. But first, let's
37. 1:34start with a few simple things. Does it
38. 1:36answer the Alice question? Yes, it does.
39. 1:39No problem, as you would expect. Does it
40. 1:41handle the 7 minute egg timer and the 11
41. 1:45minute egg timer question? Yes, it does.
42. 1:48No problem. So, now those are out the
43. 1:50way, we can kind of get a feel for the
44. 1:52kind of capabilities of this model.
45. 1:54Let's push it a little further. The
46. 1:56first thing I did cuz I wanted a bit of
47. 1:58fun was to get it to create a Space
48. 2:00Invaders game using a single HTML file
49. 2:03using inline JavaScript and no external
50. 2:06libraries. So this is just had to create
51. 2:08everything itself inside of the HTML
52. 2:10file. It did that. It produced me
53. 2:13index.html. You can doubleclick it and
54. 2:16you get a nice space invaders game. So I
55. 2:18wanted to push it a bit more. So I use
56. 2:19my protocol encoding type length value
57. 2:22encoding question. Asked it to create a
58. 2:25specification similar to ASN1 similar to
59. 2:29Google's Protobuff. Now it created that
60. 2:31specification. No trouble. There were a
61. 2:33few tiny mistakes in it. It did all the
62. 2:35specification defining how the protocol
63. 2:37should be laid out. But then at the very
64. 2:39end, it gives some working examples.
65. 2:40This is what happens if you want to
66. 2:41encode an integer. This is what happens
67. 2:43if you want to encode a string. And
68. 2:45actually one of those or two of those
69. 2:46had a minor error in the numbers it was
70. 2:49telling you that you should use. So the
71. 2:51examples it was giving uh didn't
72. 2:53actually quite work out. But if you fix
73. 2:54those, no problem whatsoever. So I then
74. 2:56went on to say, well now implemented
75. 2:58including testing, including end toend
76. 3:00testing, unit testing, and so on. And I
77. 3:02did open code, let it run. A few times I
78. 3:05had to interact with it, telling it, you
79. 3:07know, here's a plan. This is what you
80. 3:09should do now. Go ahead and do it. And
81. 3:12just as you would do with normal coding
82. 3:14with a harness and it worked perfectly.
83. 3:16It gave me a nice little library that
84. 3:19could implement the specification,
85. 3:20including some end to-end testing,
86. 3:21including some unit tests. Worked
87. 3:24absolutely fine. So my big test after
88. 3:26that as I've done in several other
89. 3:27videos here on this channel is my new
90. 3:30scrippy uh programming language. So the
91. 3:32idea is I just give it a example of the
92. 3:35of the new scripping language and even
93. 3:37define it formally. I tell it a little
94. 3:39bit about it like it doesn't use
95. 3:41semicolons. I tell it it's an
96. 3:43interpreter and I tell it it's written
97. 3:45in C and then I say go ahead knock
98. 3:47yourself out go and write the
99. 3:49interpreter for this language. Now
100. 3:51that's quite a tall order because
101. 3:52there's no formal specification for the
102. 3:54language just an example code and a lot
103. 3:57of the stuff it has to kind of infer
104. 3:59from the example code infer from what
105. 4:01I've said uh in the bit of text. Now
106. 4:04I've tried this before in other videos
107. 4:05and really local models weren't able to
108. 4:08do it even when uh 3.6 they weren't able
109. 4:11to do it. They kind of either got stuck
110. 4:13in a loop just not being able to finish
111. 4:15it or what they produced crashed all the
112. 4:17time had severe memory leaks. It was bit
113. 4:20of a disaster. However, the amazing news
114. 4:22is here is that in a relatively short
115. 4:25amount of time certainly wasn't even
116. 4:27overnight just a few hours again few
117. 4:30prompting there was a plan and then we
118. 4:33went through the different steps on how
119. 4:34to implement it and then you know I said
120. 4:36also write some documentation and I told
121. 4:38it all the stuff back and forth as you
122. 4:40would with a coding harness. It worked
123. 4:43absolutely worked. You can run new
124. 4:45scrippy. You It created its own set of
125. 4:47test scripts to run as well. Uh and the
126. 4:51example script that I gave it for the
127. 4:53readme file that runs. And so there you
128. 4:56go. This local LLM running on a PC.
129. 5:00Quinn 3.8 was able to do all of that
130. 5:02coding using open code. You could use
131. 5:04other harnesses as well, I'm sure.
132. 5:06Haven't tested them specifically, but
133. 5:07I'm sure it'll be just as good. And it
134. 5:09did it all for me. And it actually
135. 5:11works. So, this is really amazing that
136. 5:12we've reached this point. Uh, and I'm
137. 5:15sure things are going to get better, but
138. 5:17this is amazing that we've reached this
139. 5:18point for local LLMs, just the power and
140. 5:21the capabilities that they have. Okay,
141. 5:23I'd love to hear your thoughts in the
142. 5:24comments. Have you tried Quen 3.8? Do
143. 5:27you use uh local uh AI for coding or do
144. 5:31you only rely on the big models? Do you
145. 5:33not do any coding on it? Love to hear
146. 5:35your thoughts in the comments below.
147. 5:37Okay, that's it. I'll see you in the
148. 5:39next one.
149. 5:43>> [music]