Ternary Bonsai 2 27B in 7 GB: 43 tok/s on One Nvidia P100 — Transcript

Full transcript

1. 0:00One NVIDIA Tesla P200 from 2016.
2. 0:04Ternary bonsai two, 27 B in a seven gig
3. 0:08file. Out of the box, 26.6 tokens a
4. 0:12second. And now 43.4
5. 0:15with MTP, a little guesser head added
6. 0:17onto the model. Same card. And on my
7. 0:20test prompts, the answers didn't change.
8. 0:22Not one word. Nothing bought. My wallet
9. 0:25wasn't even invited. I'll show you where
10. 0:27the speed was hiding and all the stuff
11. 0:29that made it slower. Oh, and there's a
12. 0:32catch.
13. 0:33First run, stock. Prism ML's fork of
14. 0:36llama CPP, one card, 26.6.
15. 0:41One of you guys asked for this one. The
16. 0:43model is Ternary bonsai two, 27 B by
17. 0:46Prism ML. Prism ML says nearly every
18. 0:49weight is just minus one, zero, or plus
19. 0:51one. That's why it's called bonsai. It's
20. 0:54still the whole tree. It just lives in a
21. 0:55pot now. Very Mr. Miyagi.
22. 0:59We had a theory. Our notes said about 20
23. 1:02milliseconds of every token was the card
24. 1:04waiting for orders. A formula even said
25. 1:0626.7
26. 1:08and the card did 26.6. I was like, holy
27. 1:11cow, nailed it. Nope. Right number,
28. 1:14wrong reason. The profiler said busy 94%
29. 1:18of the time. Busy doing what? It's like
30. 1:21the guy at the DMV. Looks super busy,
31. 1:23line doesn't move. One job ate most of
32. 1:26every token, reading the weights.
33. 1:28[snorts] This card reads its memory at
34. 1:30605 gigabytes a second. We measured it.
35. 1:33That job did 264.
36. 1:36Okay, so it's like a warehouse. Every
37. 1:39token, a forklift hauls out the whole
38. 1:41model. Seven gigs of tiny boxes. One box
39. 1:44is one byte, four weights.
40. 1:47Problem one. The old code carried two
41. 1:49boxes per trip. Even with no math at
42. 1:51all, that tops out around 370 gigabytes
43. 1:54a second.
44. 1:56Problem two, the math. Newer NVIDIA
45. 1:58cards have a fast
46. 2:00for multiplying small whole numbers. The
47. 2:02P100 doesn't. Our first patch swapped in
48. 2:05another one, 28.9.
49. 2:07Then we measured it.
50. 2:09That instruction runs at half speed on
51. 2:11this chip. Half.
52. 2:14So my AI agents and I wrote a new
53. 2:16version of that one job. Programmers
54. 2:18call it a kernel. 16 boxes per trip. The
55. 2:21math moves to the half precision
56. 2:22calculator, full speed on this card.
57. 2:25Normally that one rounds off, but these
58. 2:27numbers are so tiny every sum stays
59. 2:29under 2048 and down there nothing
60. 2:32rounds. So the answers come out exactly
61. 2:34the same, 36.2.
62. 2:37Then we switched on a shortcut the fork
63. 2:39already had for newer cards. 401 little
64. 2:41jobs per token gone. 37.3.
65. 2:45One change. Test. Check the answers
66. 2:48again and again. Wax on, wax off.
67. 2:52Then speculative decoding. In my first
68. 2:55video the guesses came from a little
69. 2:57head built into the model. Bonsai's file
70. 2:59doesn't have one. So first we try a
71. 3:02separate draft model. It guesses ahead,
72. 3:04the big one checks.
73. 3:0637 went to 20. Great. Faster at going
74. 3:10backwards. Thanks buddy. Checking five
75. 3:12tokens at once cost four and a half
76. 3:14normal tokens because this card is
77. 3:16already busy with the math.
78. 3:19But the numbers did say one thing. The
79. 3:21guessing was cheap. The checking was
80. 3:23slow, still on the old kernel. Our
81. 3:26estimate, a new checking kernel plus a
82. 3:28little head guessing just one token
83. 3:30ahead, maybe 42.
84. 3:33So the new kernel learned to check two
85. 3:34or three tokens at once. [snorts] One
86. 3:36forklift trip, boxes unpacked once for
87. 3:39all of them. And this time not a second
88. 3:41model. A tiny guesser head from Pro
89. 3:43Creations added onto the model. One
90. 3:46guess ahead.
91. 3:4743.4.
92. 3:4943.4 tokens a second on one freaking
93. 3:52card from 2016. That's with MTP. The
94. 3:56kernel on its own does 37.3 and the head
95. 3:59guessed right about seven to nine times
96. 4:01out of 10.
97. 4:02Okay,
98. 4:03but did it get dummer? Because fast and
99. 4:06wrong I can do myself. Every step we ran
100. 4:09the same three test prompts, some pros,
101. 4:12some code, a list. At temperature zero,
102. 4:15so the model always picks its top word,
103. 4:17no dice rolls. Then we compared the full
104. 4:19text with the step before, word for word
105. 4:22the same all the way back to stock. Five
106. 4:24out of five on the reasoning questions,
107. 4:26every step. And the perplexity, that's a
108. 4:29score for how surprised the model is by
109. 4:31a test text. Lower is better. Stock
110. 4:33build, 6.0368.
111. 4:36New kernel before the guesser head,
112. 4:386.0368.
113. 4:40No, that's not a typo. Same in every
114. 4:43chunk, too. So no, it didn't get dummer,
115. 4:46at least not on what I tested.
116. 4:49Long chats got faster, too. 30,000
117. 4:51tokens deep, that's a long report pasted
118. 4:54in. And about 20 tokens a second became
119. 4:56about 30.
120. 4:58And there's one setting, batch size,
121. 5:00that reads long prompts faster from
122. 5:02about 116 to 160 tokens a second. No new
123. 5:06kernel needed. But it can change a word
124. 5:08here and there, so it's not in my same
125. 5:10answers test. Details are in the
126. 5:12description. The guesser head adds half
127. 5:14a gig, and all of it still fits in about
128. 5:1710 GB of the 16. I mean, on one card.
129. 5:21Is there more? A bit. The new kernel is
130. 5:24still about 4 milliseconds per token
131. 5:26slower than the same reads with no math
132. 5:28at all.
133. 5:29Close that, and our estimate is about 45
134. 5:32without the guesser. I didn't test it.
135. 5:35So of course I got greedy. One guess
136. 5:38ahead gave 43. Two guesses ahead?
137. 5:4139.2
138. 5:43slower. Classic. Up at the casino and
139. 5:46you just have to go one more round.
140. 5:49Every extra guess has to be checked, and
141. 5:51on this card checking isn't cheap. Oh,
142. 5:54and CUDA graphs, supposed to cut the
143. 5:56waiting. The fork turns them off on this
144. 5:58card, so we turn them on. And they
145. 6:00really switched on.
146. 6:011% slower. We tried a bunch of other
147. 6:04kernel ideas too. Everyone exact and not
148. 6:07one of them faster. So one guess it is.
149. 6:11That's the actual card by the way, the
150. 6:13top one. Okay, the catch I promised. In
151. 6:16my first video, two of these cards ran
152. 6:18the regular 27B Quen 3.8 at 67 tokens a
153. 6:22second. Bonsai on one card does 43.4
154. 6:26with MTP. Different model, different
155. 6:28test, you know, but two cards are still
156. 6:30faster. So Bonsai on one card, it fits
157. 6:34and it got a lot faster. It just doesn't
158. 6:36beat two. And you know what I like about
159. 6:39it? It only needs one card. So the
160. 6:41second card is yours.
161. 6:43Run two Bonsais at the same time working
162. 6:46on different things or one is the main
163. 6:48brain and the other one is an agent or
164. 6:50put a model there that always has to be
165. 6:52on doing its own job. Whatever you need.
166. 6:55That card is free now. And to be fair, I
167. 6:58checked that Bonsai gives the same
168. 7:00answers on my test prompts as before our
169. 7:02changes. I didn't test if it's as smart
170. 7:04as the big model on real work or measure
171. 7:06the power. That's for another day.
172. 7:08Patience, Daniel San.
173. 7:10Big thanks to Prism ML for the model and
174. 7:13to Pro Creations for the guesser head.
175. 7:15Their names are in the description. So
176. 7:17yeah, one card from 2016, a 7 gig model,
177. 7:2143 tokens a second with MTP and the same
178. 7:24answers as stock on my tests. And my
179. 7:26wallet, still not invited. So tell me,
180. 7:30what's on your one card and what should
181. 7:32I squeeze next?