BreezeTTS2 - 100% Local Real-Time Voice — Transcript

Full transcript

1. 0:00Okay, a brand new openweight TTS model
2. 0:02just beat every other open model on the
3. 0:04leaderboard. This model came out 6 days
4. 0:07ago and I've already got it running
5. 0:09locally so I can show you the real
6. 0:10thing, not just the demo page.
7. 0:13>> All right, so that was the new TTS
8. 0:16system called Breeze TTS2.
9. 0:20And in this video, I'm going to go
10. 0:21through what it is, talk about how it's
11. 0:23very impressive for a lot of different
12. 0:25things, talk about some of the sad
13. 0:27things that are not ideal here, but also
14. 0:30just show some demos of where TTS is at
15. 0:33the moment for open weights models. All
16. 0:36right, so this comes from a company
17. 0:38called Breeze Blue. Now, up until a few
18. 0:40days ago, I hadn't really heard about
19. 0:42them. I didn't really know anything
20. 0:43about what they were building, etc. But
21. 0:45it seems that they are a startup in
22. 0:47China that is working pretty much solely
23. 0:50at the moment at least on sort of voice
24. 0:53and TTS systems and sort of ideas for
25. 0:57where these things are actually going.
26. 0:59So this has got a lot of the sort of
27. 1:01standard things that you'd expect from a
28. 1:03modern TTS system now, but it's also got
29. 1:05a number of really interesting things as
30. 1:07well. Now first off, they can do voice
31. 1:10design. So this is where you're
32. 1:11basically creating a distinctive voice
33. 1:13from a description of that voice but
34. 1:17without any sort of reference audio. So
35. 1:19for example, if I describe a character,
36. 1:22I tell them what the perhaps accent
37. 1:24would be like, things like that, it will
38. 1:26then create a voice for me. And you can
39. 1:29see that they're comparing themselves
40. 1:31here to a bunch of different models that
41. 1:33are out there, including things from 11
42. 1:36Labs. And sure enough, this is doing
43. 1:38really well here. If we come in and
44. 1:40listen to some of their examples.
45. 1:42>> This is Elena Vance reporting live from
46. 1:43the east block where the crowd just
47. 1:45broke the barricade. If anyone can hear
48. 1:46me, we need medical teams now. Please,
49. 1:48they're getting closer.
50. 1:50>> Okay, so you can see that that was
51. 1:52basically a prompt plus a script. And
52. 1:54this is one of the common things that
53. 1:55they will have a prompt for describing
54. 1:57the voice or the emotion that you want
55. 1:59to convey. So you can see this one was a
56. 2:01young adult mid20s American accent,
57. 2:04professional broadcast journalist
58. 2:06turning frantic. So we've got clear,
59. 2:08articulate, but strained by panic, high
60. 2:11tension, rapid pace, breathlessness,
61. 2:13polished to raw desperation. So that
62. 2:16does a pretty good job of explaining
63. 2:18that particular voice there. And you can
64. 2:20see some of the other examples too of
65. 2:22this one where the prompt is for a male.
66. 2:25Let's just listen to that. I've walked
67. 2:27through too many long nights to mistake
68. 2:30silence for peace. So when I speak of
69. 2:32tomorrow, believe me, brother, I mean a
70. 2:35dawn we can finally stand in together.
71. 2:38Okay, so we definitely have the whole
72. 2:41sort of southern accent and stuff like
73. 2:44that coming through that they've
74. 2:46described in their prompt here. And
75. 2:48that's one of the things I noticed with
76. 2:49the descriptive things is they can do
77. 2:51accents very nicely here. All right, so
78. 2:53the second thing that they're focused on
79. 2:55is voice direction. So this is cloning a
80. 2:58voice from audio but steering the tone,
81. 3:02emotion, and the delivery for it. So for
82. 3:05example, here we've basically got a
83. 3:07reference voice. Before touching a
84. 3:09formula, ask what question am I
85. 3:11answering?
86. 3:12>> And then we've got a prompt here talking
87. 3:14about speaking in a hushed intensity,
88. 3:16hypnotic and persuasive rhythm. And then
89. 3:18we've got the actual script. You feel
90. 3:20the energy radiating from it, don't you?
91. 3:22This isn't just a piece of quartz. It's
92. 3:24a conduit for a small offering. You can
93. 3:27take it home and finally clear that dark
94. 3:29cloud hanging over your future. Trust
95. 3:31me, you need this. So, you can see
96. 3:34there, especially in the bits that
97. 3:35perhaps not totally hushed voice, but is
98. 3:39actually taking the reference and
99. 3:40running with that. So, here's another
100. 3:42example of a reference audio.
101. 3:45>> Let the breath move like the tide. In as
102. 3:48the wave draws up the sand and out as it
103. 3:52slides back to the sea. Okay. So there
104. 3:54you had some kind of guided meditation
105. 3:56etc. Now you can see we've taken that
106. 3:59soft and very sort of gentle passive
107. 4:01voice to now become passive aggressive
108. 4:04and with sarcasm in it.
109. 4:06>> Enjoy your vacation while I stay here
110. 4:08and clean up the mess you left behind
111. 4:10for me. Have a wonderful time.
112. 4:12>> Okay. And you can hear that it really
113. 4:14does capture the emotion there. That's
114. 4:16pretty impressive of how it's actually
115. 4:19doing that. So on top of voice cloning
116. 4:22and voice design and voice direction, it
117. 4:25also has the ability to do vocal events,
118. 4:28meaning that you can basically pass in
119. 4:30things like getting it to laugh, cough,
120. 4:32clearing the throat, sighing, a whole
121. 4:35bunch of different things like that. And
122. 4:37we've seen that before in models and
123. 4:40sometimes it works really well,
124. 4:41sometimes it hasn't. With this model, it
125. 4:43does actually come out reasonably
126. 4:45nicely. Now, another thing that people
127. 4:47often are asking for with the TTS is for
128. 4:50multilingual speech. And here they've
129. 4:52got 50 different languages that you can
130. 4:55set it to actually speak in.
131. 5:11Okay, so I don't really speak a lot of
132. 5:13those languages, but it certainly sounds
133. 5:15pretty impressive in the pronunciation
134. 5:17stuff. And then one of the ones that I
135. 5:19think is really impressive here is just
136. 5:22the low latency. So you can see here
137. 5:24that they're comparing themselves
138. 5:26against a bunch of the other TTS systems
139. 5:28that are out there. And out of the box,
140. 5:30it can do full streaming. You've got
141. 5:32sort of time to first response back
142. 5:35being really quick. And that's partly
143. 5:37because of the model itself. If we come
144. 5:38in and actually look at the model, we
145. 5:40can see that this is a three billion
146. 5:42parameter model. Now the version that
147. 5:44they've uploaded is a 16bit version. But
148. 5:47there are other versions out there
149. 5:48already where people have basically put
150. 5:50this on MLX. People have made versions
151. 5:53for CPP. People have created 4-bit
152. 5:55versions of this, 8bit versions of this,
153. 5:58etc. And sure enough that they're
154. 6:00touting that on the Arena Voice
155. 6:02leaderboard here, they're actually
156. 6:04topping it for openweight models. Now,
157. 6:07it looks like they're a little bit
158. 6:08behind compared to some of the other
159. 6:09models that are not open weights, but
160. 6:11undoubtedly this is an impressive model.
161. 6:14So then the big question before I go
162. 6:16into the actual demos that I've made
163. 6:18with it is what's the downside? Well,
164. 6:21this is the downside. It's that it's
165. 6:23basically a research and non-commercial
166. 6:26license. So they are actually limiting
167. 6:29what you can actually do with this. Not
168. 6:31only can you not generate outputs for
169. 6:33commercial use, but things like you
170. 6:35cannot distill from it, you cannot do a
171. 6:37bunch of different things with it. And I
172. 6:39kind of feel if it wasn't for that, this
173. 6:41would become the go-to model for doing a
174. 6:43lot of different voice generation,
175. 6:45whether that's generating audio for
176. 6:46games or for assistance or for a whole
177. 6:49bunch of these different things. So if
178. 6:51you're looking to run this by yourself
179. 6:53at home, you've got no issues. But if
180. 6:55you're looking for to use it for
181. 6:56commercial things, you're going to want
182. 6:57to check out their pricing and quotas on
183. 6:59their actual site. But for now, let's
184. 7:01jump in and actually try this out. All
185. 7:03right, so for the demo, I'm going to be
186. 7:05running this 100% locally. Thank you to
187. 7:09Dell for sponsoring the compute on this.
188. 7:11I'm running this on my Dell T2 Pro Max
189. 7:15machine with an RTX Pro 6000 in there.
190. 7:18This is then all streaming across tail
191. 7:20scales to the computer where I'm
192. 7:21actually doing the recording here. Now,
193. 7:23first off, let's just look at sort of
194. 7:24voice design. So, you've got two things
195. 7:26that you can put in here. You can put in
196. 7:28what you want them to actually say, and
197. 7:30then you've got the actual voice
198. 7:32description of what the voice is going
199. 7:35to be like in here. So, you can see this
200. 7:37one I thought was quite funny.
201. 7:38>> A world where nothing is what it seems.
202. 7:42One developer must ship before Friday.
203. 7:45>> So, kind of like the movie narrated for
204. 7:47that. The other thing too is that then
205. 7:48we can start to play around with some of
206. 7:50these. So you can see that it actually
207. 7:52generates very quickly here. So I'm not
208. 7:55using the streaming version here. I'll
209. 7:57show you a demo of that next. I think in
210. 7:59some ways that's probably even more
211. 8:01impressive. But here you can see, okay,
212. 8:03just describing a voice and giving it a
213. 8:05script and then we can generate out.
214. 8:07>> Welcome aboard. Your journey begins now.
215. 8:10>> Okay, so that's the first thing. The
216. 8:11second thing you can do is actually
217. 8:13clone a voice. So here what I've got is
218. 8:16some reference audio. So this is
219. 8:17actually a Chinese voice.
220. 8:22Okay, so that's literally 4 seconds of
221. 8:24reference. You need to basically
222. 8:26transcribe what they're actually saying
223. 8:27and stuff. And then now we can generate
224. 8:29out using that here. So you'll see that
225. 8:32it generates out pretty quickly. And now
226. 8:35let's listen to that voice speaking in
227. 8:37English.
228. 8:38>> It is good to hear your voice again
229. 8:40after all this time.
230. 8:42>> Okay, so that could be any of the 50
231. 8:44languages that they support in there.
232. 8:46The next thing we can do is we can clone
233. 8:48a voice, but we can then guide it. So
234. 8:51just like we were doing the voice
235. 8:52direction before, you can see now we can
236. 8:55take that voice and do a direction with
237. 9:00it. So we've cloned it, but now we've
238. 9:02given it some direction of speak slowly
239. 9:04with a strange serious tone.
240. 9:07>> We need to discuss what is happening
241. 9:09with the closed AI labs.
242. 9:11>> And so this shows you how you can
243. 9:13generate lots of different voices like
244. 9:15that. Now you've got a bunch of sort of
245. 9:17vocal events that you can use for things
246. 9:20like cough, laugh, sigh. I think some of
247. 9:23them are very hit and miss. So that's
248. 9:25something that I think you want to play
249. 9:26with. I do think it's very interesting.
250. 9:28The direction tends to work really well
251. 9:31on this. So it's got really nice proity.
252. 9:33If I ask it to do a sarcastic version of
253. 9:35that voice,
254. 9:39>> we need to discuss what is happening
255. 9:42with the closed AI labs. You definitely
256. 9:45can get a different sort of feel there.
257. 9:47I think the sort of sarcastic clears
258. 9:49throat maybe doesn't go well together.
259. 9:51But anyway, let's look at the actual
260. 9:53sort of streaming version of this and
261. 9:55see the streaming version of this where
262. 9:57we can focus in on one voice and then
263. 9:59basically put it all together. Okay, so
264. 10:02in this demo, I'm testing out the actual
265. 10:04speed of this. So this is the voice I
266. 10:07clone. I actually first took one of the
267. 10:09generated ones that they made in
268. 10:11Chinese. I like the tone of that one, so
269. 10:14I cloned it and we're doing it in
270. 10:16English here. What I'm going to do is
271. 10:18I'm going to turn off my mic so that it
272. 10:20doesn't get in the way. The challenge I
273. 10:22always have with these is that the audio
274. 10:24bleeds out into my mic and messes up.
275. 10:26So, what I've built here is basically a
276. 10:29demo of where I can type something and
277. 10:32then she's going to speak back to me.
278. 10:34And then I've got an auto reply button
279. 10:36here basically where it will use a Gemma
280. 10:39model, a small Gemma model to generate
281. 10:42both the replies to her and then what
282. 10:44she's saying back. You'll see that when
283. 10:46I press this button, it's going to
284. 10:48basically feed in my response to her
285. 10:50rather than me cutting in sort of saying
286. 10:52it and stuff like that if I had an ASR
287. 10:54system. So, I'm going to start it off
288. 10:56simple and then I'm playing this all
289. 10:57back real time and you're going to see
290. 10:59the stats of how fast it's buffering and
291. 11:01stuff like that as well. Hey there. I'm
292. 11:04doing pretty good. Just tinkering with
293. 11:06some new plugins. How about you?
294. 11:09Oh, you're running simulations.
295. 11:12What kind of models are we talking
296. 11:13about?
297. 11:16I haven't dipped into the latest LLMs
298. 11:18today. I'm more into analog gear right
299. 11:20now.
300. 11:22Not really. I haven't been testing those
301. 11:24specifically. Everything seems smooth on
302. 11:27my end.
303. 11:29Stutter in the environment rendering. H
304. 11:32I don't have an eye for that kind of
305. 11:34technical stuff. Maybe you should check
306. 11:36the rendering settings.
307. 11:38Realistic enough is always a moving
308. 11:40target, isn't it? It depends entirely on
309. 11:43what you're trying to achieve.
310. 11:45>> Okay, so you can hear there
311. 11:46realistically her response is just
312. 11:49milliseconds after me pressing this and
313. 11:52that the model has generated what I say
314. 11:54to her. So when you combine this with an
315. 11:56ASR, you can get a full two-way
316. 11:58conversation going. And I should stress
317. 12:00that this is running on the Dell machine
318. 12:02in my office and then streaming over
319. 12:04tail scale to the computer where I'm
320. 12:07actually doing the recording and stuff.
321. 12:08So it does show you where you're getting
322. 12:10to the point where we can actually have
323. 12:12these kind of interactions 100% locally
324. 12:15with multiple models. Now at the moment
325. 12:17to do it, you still need a beefy GPU,
326. 12:20but the quality here is pretty amazing
327. 12:22considering this is a 3 billion model.
328. 12:24I'm running it at full resolution. I
329. 12:26could run a quant version of it and it
330. 12:28would even be faster than this. Okay, so
331. 12:31just finishing up. I think that this
332. 12:34model is definitely sort of up there
333. 12:37around the sort of Quen 3 TTS level.
334. 12:41Obviously, the biggest issue here is the
335. 12:43license. And my guess this then puts
336. 12:45pressure on people like Quen to release
337. 12:48a newer version of their TTS because we
338. 12:50know at the same time you've got a lot
339. 12:52of companies whether that's OpenAI with
340. 12:54whether that's the Space X team or the
341. 12:57Gemini team working on newer better
342. 13:00versions of these for APIs. So I do
343. 13:03think at some point this is going to be
344. 13:05interesting to see whether as they bring
345. 13:08out Breeze TTS3 do they make this one
346. 13:10fully open. Now there are some issues
347. 13:12with this here. But the way it's been
348. 13:14trained is it hasn't removed signal
349. 13:16processing before they've actually
350. 13:19trained the model. So you get a lot of
351. 13:21the times where the sound of the voice
352. 13:24actually has a particular EQ to it or
353. 13:28artifacts like the microphone and where
354. 13:31it's recorded. That's especially true if
355. 13:33you listen to things like the newsreader
356. 13:36voices and stuff like that where it's
357. 13:38made to actually sound like it's coming
358. 13:40out of a TV or out of a small speaker.
359. 13:43So, personally, I think that's not
360. 13:44ideal. We would like to be able to
361. 13:46separate the voice from the signal
362. 13:48processing and be able to have those run
363. 13:51together. But this certainly is a nice
364. 13:53model that you can run yourself at home
365. 13:56and use for doing real-time streaming
366. 13:58chats and for just having something
367. 14:00where you want your computer to send you
368. 14:03alerts or something. It can print out
369. 14:05the voice and then send it across
370. 14:06Telegram, send it across something else.
371. 14:08So let me know in the comments what you
372. 14:10think of the real time stuff here. While
373. 14:12we'd all like to have the models of this
374. 14:14quality be the size of Kakoro or
375. 14:16something like that, that's probably not
376. 14:18going to happen with the current
377. 14:19architectures that people are using. But
378. 14:21I do think for many people running a
379. 14:234-bit version of this is probably going
380. 14:25to get you almost the same standard of
381. 14:28audio and a very significant reduction
382. 14:30in file size and time to actually
383. 14:33process that. So, let me know your
384. 14:34thoughts in the comments. As always, if
385. 14:36you found the video useful, please click
386. 14:38like and subscribe and I will talk to
387. 14:39you in the next video. Bye for now.