Gemini 3.8 Flash TTS with Voice Cloning — Transcript

Full transcript

1. 0:00Okay, so I think it's fair to say that
2. 0:02Google has kind of dropped the ball on
3. 0:04LLM's lately. The proposed Gemini 3.5
4. 0:08Pro never came, and hopefully all of
5. 0:10that's going to change very soon with
6. 0:11the Gemini 4 release. But on the
7. 0:13multimodal side, it's a very different
8. 0:16story. The DeepMind multimodal team have
9. 0:18been doing really well with the
10. 0:20multimodal releases that have been based
11. 0:22on the Gemini Flash models. So we've had
12. 0:25Omni 1.1 Flash update. On the voice
13. 0:28side, we've seen the live models with
14. 0:29things like live translate and
15. 0:31subscribe. And now there are two new
16. 0:33text-to-speech models, Gemini 3.8 Flash
17. 0:36TTS and Gemini 3.8 Flash Light TTS. Now,
18. 0:40Google's pitches that these are the most
19. 0:42expressive voice models yet, and they're
20. 0:44a number one on a bunch of the
21. 0:46benchmarks. I certainly some of that
22. 0:48holds up, but when you look at some of
23. 0:49the independent numbers, it's perhaps a
24. 0:51bit more mixed than the blog post makes
25. 0:53it sound. Okay, so first off, there are
26. 0:55two models. They're aimed at different
27. 0:57jobs. Flash TTS is the creative one.
28. 1:00This is for things like character
29. 1:01voices, audiobooks, podcasts, games, all
30. 1:05those kinds of things. Flash Light TTS
31. 1:08on the other hand is the volume one.
32. 1:10It's built for dubbing, bulk audio
33. 1:13generation, voice agents, and anything
34. 1:15where you're generating a lot of speech
35. 1:18and cost matters more than getting every
36. 1:20single nuance perfect in there. So the
37. 1:22headline feature here is all about voice
38. 1:25design. Instead of just picking from a
39. 1:27list of preset voices, you describe the
40. 1:30voice that you want in plain language,
41. 1:32you give it a role, and accents, the
42. 1:34characteristics of the voice, and it
43. 1:36builds that voice for you across over a
44. 1:38hundred different languages. And this is
45. 1:41something that we've seen both from
46. 1:42other TTS providers, but also from open
47. 1:45models. But certainly does seem that
48. 1:47Google's improved the whole concept of
49. 1:49this year. This is one of the things
50. 1:50that I found often to be the most
51. 1:52frustrating with TTS models. Is that
52. 1:55while things have been getting
53. 1:56fantastically better and we're starting
54. 1:58to see really good prosody in the
55. 2:00models, we're starting to see really
56. 2:02good human-likeness, etc. Often the
57. 2:04challenge becomes is that when you go
58. 2:06through the test voices that they've
59. 2:08got, none of them really fit what you're
60. 2:10trying to do. And so you end up having
61. 2:12to try and do some kind of voice
62. 2:14cloning. And obviously the challenge is
63. 2:16there can be a lot of legal issues. So
64. 2:18we have seen some really nice attempts
65. 2:19at this from the Quen TTS people, but
66. 2:22some of the more recent ones that I've
67. 2:23looked at, no matter how hard I try to
68. 2:26describe the voice, it ends up going in
69. 2:29a slightly different direction. In fact,
70. 2:31what I've often ended up doing is
71. 2:33basically generating a lot of different
72. 2:36voices with slightly different
73. 2:37descriptions, going through and curating
74. 2:40the best one, and then actually cloning
75. 2:42that artificially made voice. Now,
76. 2:44alongside voice designing here, there's
77. 2:46a library of over 2,000 different voices
78. 2:48in here. They've got a whole bunch of
79. 2:50different regional accents, and you can
80. 2:52actually sort of take a voice,
81. 2:53manipulate it the way you want, and then
82. 2:55actually save that voice so that you can
83. 2:57use it going forward. And I guess this
84. 2:59is really no surprise. Google's had this
85. 3:01feature in testing for quite a while,
86. 3:03but they just haven't rolled it out. And
87. 3:05that's the whole feature of voice
88. 3:06cloning. So you give it a 30-second
89. 3:09sample, and it creates that voice. Now
90. 3:11obviously compared to other startups,
91. 3:13when Google does something like this,
92. 3:15they get in way more trouble if it goes
93. 3:16wrong. So they clearly have taken their
94. 3:18time with doing this, and they've really
95. 3:20focused on putting some guardrails on
96. 3:22this. So the person whose voice it's
97. 3:24going to actually create has to go
98. 3:26through a spoken consent clip. And that
99. 3:29clip has to match the reference speaker
100. 3:31before the clone ever gets created. On
101. 3:33top of this, of course, they've also got
102. 3:35the whole set of safety features in
103. 3:37there that every clip these models
104. 3:39generate is watermarked with SynthID,
105. 3:41and there are C2PA credentials on top of
106. 3:43that. The other challenge that a lot of
107. 3:45people are going to face is that because
108. 3:47of legal requirements, that voice
109. 3:49cloning in AI Studio isn't available in
110. 3:52a lot of different countries and even in
111. 3:54some states in the United States. So, it
112. 3:57does show just how touchy of a subject
113. 3:59that can be, especially for a big
114. 4:01company like Google. Now, once you've
115. 4:03actually got your voice, the other big
116. 4:04thing you can do here is you can give
117. 4:06stage directions line by line. So, you
118. 4:07can say that this one should be
119. 4:09whispered, this one should be rushed,
120. 4:11this one should be sarcastic, etc. And
121. 4:13you can drop in sort of non-verbal tags,
122. 4:16like I've shown you in some of the other
123. 4:17TTS systems, for things like sighs,
124. 4:20laughs, gasps, all those kinds of
125. 4:23things. Another nice feature that you've
126. 4:25got in here is you can actually do two
127. 4:26speaker scenes from a single script. And
128. 4:29Google claims with that that it can
129. 4:31actually hold the voices for hours of
130. 4:34long-form audio without the voice sort
131. 4:36of actually drifting. That's one of the
132. 4:38issues that I've run into with some of
133. 4:40the open TTS systems recently is that
134. 4:42once you get that voice right and you
135. 4:45start generating anything of any length,
136. 4:46it starts to end up drifting off. All
137. 4:48right, so let's talk a little bit about
138. 4:50the benchmarks because this is where it
139. 4:51gets interesting, but also a bit
140. 4:53questionable in some of the things that
141. 4:54they've done. So, Google's numbers
142. 4:57mostly come from Hume AI. They talk
143. 4:59about that the Flash TTS is number one
144. 5:02on Hume's voice design benchmark and it
145. 5:04leads on accents. And you can see people
146. 5:06like Logan actually tweeting this out.
147. 5:08But here's the thing, one of the two
148. 5:10people that wrote the blog post is Alan
149. 5:12Cohen and Alan actually was the CEO and
150. 5:15chief scientist at Hume until January
151. 5:18this year when Google DeepMind licensed
152. 5:20their tech and hired a bunch of the
153. 5:22people from that startup over. So, I'm
154. 5:24certainly not saying that their numbers
155. 5:25are wrong, but I'm not sure if quoting
156. 5:27benchmark numbers from a benchmark that
157. 5:29you designed yourself is the best way to
158. 5:32judge this. Now, the good thing here is
159. 5:34that artificial analysis have also done
160. 5:36their own tests on that and I think
161. 5:38those are reasonably independent. If you
162. 5:40look at the sort of preference Elo,
163. 5:41which is the common way that people
164. 5:43actually test these kind of models, you
165. 5:44can see that the Gemini 3.8 Flash TTS is
166. 5:48coming second in here and the Flash
167. 5:50Light TTS is coming in sixth in here.
168. 5:53And we see that the Flash model is just
169. 5:55ahead of the new Quan Audio 3 TTS Plus.
170. 5:59Now, at the time of me recording this,
171. 6:01that one is still not an open model, but
172. 6:03if Quan goes on to release the weights
173. 6:05for that as an open model, like they did
174. 6:07with the Quan TTS before, that's going
175. 6:09to be really interesting to see as well.
176. 6:11Where the Gemini models really do seem
177. 6:13to take the lead and some of the
178. 6:14benchmarks around pronunciations, etc.
179. 6:16That's especially useful if you're using
180. 6:18the Flash Light model to bulk generate
181. 6:21audio of where you're saying things like
182. 6:23numbers, IDs, things that people really
183. 6:25want to be able to hear correct the
184. 6:26first time. That's something that really
185. 6:28does matter if you're building a voice
186. 6:30agent with this kind of model. If we
187. 6:32look at how the models actually do on
188. 6:33other languages, we can see this is, and
189. 6:36I would argue that this is Google's
190. 6:38strength, right? Google has a whole
191. 6:40bunch of different multilingual data and
192. 6:43you can see this clearly in here of
193. 6:45looking at how they compare to other
194. 6:46providers. That's say Cartesia and
195. 6:48ElevenLabs here are really making a
196. 6:50strong move at the multilingual TTS
197. 6:53systems. Probably the other big demo
198. 6:54here is actually price. So, Google's
199. 6:57kind of sitting in the middle of the
200. 6:59pack here. They're obviously a lot
201. 7:01better than some of the offerings from
202. 7:03Cartesia and ElevenLabs and certainly
203. 7:05from MiniMax, etc., but they're by far
204. 7:07from being the cheapest in here. And I
205. 7:09think in some ways more importantly, the
206. 7:11Flash Light model isn't hugely cheaper
207. 7:14than the Flash model itself. Which is
208. 7:16definitely a difference than when we
209. 7:17compare the text versions of these
210. 7:19models. All right, so let's go up to the
211. 7:20AI studio, have a play with this, and
212. 7:22then we'll try out the two models, and
213. 7:24then we can try all this out in Colab as
214. 7:25well. All right, so if you want to check
215. 7:28out the voices, you can actually come
216. 7:30into AI studio and just pick one of the
217. 7:33voice models. You can see here I've
218. 7:34picked the Gemini 3.8 Flash TTS and then
219. 7:37you'll be asked to basically hook up an
220. 7:39API key to it. But from there, you can
221. 7:41basically go through and you can explore
222. 7:43a whole bunch of different models in
223. 7:45here. So if I just come in here, I can
224. 7:48basically take the model and then I can
225. 7:49make a description in here. And the cool
226. 7:51thing is if you want to, they've
227. 7:52actually got voice rewriting in here. So
228. 7:55you can see that if I click on that,
229. 7:57that will basically just use Gemini to
230. 7:59flesh out the actual voice prompt in
231. 8:02here of what I want the voice to be
232. 8:04like. And then you'll see from there, it
233. 8:06will give you three voice options that
234. 8:09you can use. Everything from describing
235. 8:11the person themselves through to the
236. 8:14position that they are on a microphone
237. 8:17and that kind of thing. So that's really
238. 8:19kind of cool. Unfortunately, as I'm
239. 8:20recording it now, it seems like the
240. 8:22voices are failing in here. But normally
241. 8:24what you would do is you would get
242. 8:26basically three voices and those voices
243. 8:29then you can customize it, you can then
244. 8:30refine it, etc. All right, after
245. 8:32multiple attempts to try and generate
246. 8:34this, it doesn't seem like this part is
247. 8:36working now. Let's just jump into the
248. 8:37code and have a play with the actual
249. 8:39Colab file for this. All right, so let's
250. 8:41take a listen to it.
251. 8:42>> Hi, I'm Sam and today we're looking at
252. 8:45the new Gemini 3.8 text-to-speech
253. 8:47models.
254. 8:48>> So as you can see that the basic voice
255. 8:50just works pretty well. And the cool
256. 8:52thing is you've got lots of choices if
257. 8:54you want to do just sort of standard
258. 8:56voices and stuff in here. Let's come
259. 8:58down and listen to the comparison
260. 9:00between the flash model and the
261. 9:02flashlight model.
262. 9:04>> The quarterly numbers are in. Revenue is
263. 9:06up 11% and churn is the lowest it has
264. 9:09been in 2 years.
265. 9:10>> Okay, so that was the normal flash TTS.
266. 9:13Now let's listen to the flashlight.
267. 9:15>> The quarterly numbers are in. Revenue is
268. 9:18up 11%
269. 9:20and churn is the lowest it has been in 2
270. 9:23years.
271. 9:24>> Okay, so the common thing I notice with
272. 9:26the flashlight TTS is it tends to be a
273. 9:29bit more lazy, it might go on for
274. 9:32longer, etc. And honestly, I feel that
275. 9:34the The difference is not huge. So
276. 9:37you've got the Flash TTS being 50% more.
277. 9:40Then if I was doing something that
278. 9:41wasn't too long, I'm probably just going
279. 9:43to go for the good voices. Okay, so you
280. 9:45can see now we've got the style control.
281. 9:47This is basically where we can give it
282. 9:49direction on how we want what we're
283. 9:53asking it to speak, how we want it
284. 9:54pronounced, etc.
285. 9:56>> We just got the results back and the
286. 9:59prototype actually works.
287. 10:02>> Okay, so that one certainly is flat and
288. 10:04bored. Let's try thrilled and
289. 10:06breathless. We just got the results back
290. 10:09and the prototype actually works. Okay,
291. 10:11and then now nervous almost whispering.
292. 10:14>> We just got the results back and the
293. 10:16prototype actually works.
294. 10:18>> Okay, so you can hear that all three of
295. 10:20them definitely sound very different
296. 10:22even though they're the exact same
297. 10:23words. So one of the things that I do
298. 10:25like about this is that this affects the
299. 10:29length of the audio. So with some of the
300. 10:32open models, one of the challenges that
301. 10:33you have is it would try and say
302. 10:37something in the style that you're
303. 10:38asking, but it's also trying to hit a
304. 10:40certain time length in the way that it
305. 10:43returns back. Here we're basically
306. 10:45letting the time length be dictated by
307. 10:48the style that it's saying. All right,
308. 10:50the other thing that you can do in here
309. 10:51is that in the actual script that you
310. 10:53want them to say, you can actually put
311. 10:55in these inline tags. And so these are
312. 10:58things like whisper, laugh, sigh, that
313. 11:01kind of thing.
314. 11:01>> Wait. Shh. Did you hear that in the
315. 11:04hallway? Never mind. It was just a cat.
316. 11:06>> You should have seen your face.
317. 11:08>> Okay, so you could hear there that the
318. 11:10inline tags, it certainly does what
319. 11:12they're basically there for. In the case
320. 11:14of whisper, sigh, laugh, the laughs will
321. 11:18change also depending on the style
322. 11:20directions you've given it as well. All
323. 11:22right, so next up is the whole idea of
324. 11:25voice design. So now what we're doing is
325. 11:28we're not just passing in a script and
326. 11:30sort of a guidance of how to speak it,
327. 11:33we're actually giving a description of
328. 11:36the voice itself. So, the idea here is
329. 11:39that this voice should then carry over
330. 11:42to everything that it's going to speak,
331. 11:44even if we give it other sort of
332. 11:45descriptions of how to speak it going
333. 11:48through. So, this is where you can put
334. 11:49in things like where the person comes
335. 11:51from, the accent that you want, if you
336. 11:54want to find something about age, that
337. 11:56kind of thing in here. So, you can see
338. 11:57in here I've got a character called the
339. 12:00astronomer that's in his late 60s with a
340. 12:02gentle British accent.
341. 12:05>> Look out past the rings of Saturn.
342. 12:08That faint light left its source long
343. 12:11before humans walked the Earth.
344. 12:14>> Okay, so you can see there it's got some
345. 12:17vibes of sort of a David Attenborough
346. 12:20voice there, but without actually
347. 12:22mentioning him. Personally, I feel the
348. 12:24voice sounds a lot younger than late
349. 12:2660s, but let me know in the comments
350. 12:28what you think if you try that out. All
351. 12:30right, next up is the fun one where you
352. 12:32can basically have multi-speaker
353. 12:34dialogues going on in here. So, here
354. 12:36we're doing one API call with two
355. 12:39voices. And so, we've got this idea of
356. 12:42per turn style, meaning that we can
357. 12:44basically set what each character says
358. 12:46in here, and you can see we can define
359. 12:48the actual speakers, who they are. But,
360. 12:50we can also define the style for each
361. 12:52line of dialogue that they've actually
362. 12:54got. So, I'm not going to play the whole
363. 12:56thing here. I've put the notebook up.
364. 12:58You can have a play with this yourself
365. 13:00and try it out, and just see how it
366. 13:02goes. But, let's take a little listen to
367. 13:03this.
368. 13:05>> Welcome back to the show. Today, we're
369. 13:06testing the new Gemini voices.
370. 13:09>> Thanks for having me. The turn taking is
371. 13:11a lot more natural than the last
372. 13:12version.
373. 13:13>> So, is it good enough for real-time
374. 13:15agents?
375. 13:16>> For a lot of use cases, I think it is.
376. 13:18>> A lot of use cases. That's doing some
377. 13:21>> Okay, you can hear there this is
378. 13:23definitely got the vibes of sort of a
379. 13:25notebook LM cast kind of thing that gets
380. 13:28generated. Now, for me, I don't know if
381. 13:30it's just that I'm so used to the
382. 13:32hearing these voices now that they kind
383. 13:34of sound fake to me, but the cool thing
384. 13:37is you can swap out other voices, you
385. 13:38could make your own, etc. as you go
386. 13:40through this. All right, lastly, we've
387. 13:42got the voice cloning in here. And the
388. 13:44voice cloning can be done just by code,
389. 13:47right? So, you can see the code here is
390. 13:49basically just making an HTML interface
391. 13:52where you can record the reference
392. 13:55audio, so which is going to be what you
393. 13:57want the audio to sound like, and then
394. 14:00second, where you can record the consent
395. 14:02audio. I do think that Google's in a
396. 14:04very hard spot here. We know that this
397. 14:06is something that their TTS systems have
398. 14:08been able to do for quite a number of
399. 14:11years now. And this goes back to a paper
400. 14:13called Sound Storm, which at the time
401. 14:16was absolutely amazing. This is more
402. 14:17than 3 years ago now. And even then,
403. 14:20they were able to basically clone voices
404. 14:22so well. And at the time, I was actually
405. 14:24given a demo of this, and it was amazing
406. 14:26just how well it could clone someone's
407. 14:28voice back then. Jump forward to today,
408. 14:31many of the open models now have got
409. 14:33this worked out. And yet, this is the
410. 14:35first time that we've seen Google
411. 14:37actually make this available to people.
412. 14:39And I think it's been very clear that
413. 14:41Google's very conscious of the
414. 14:42ramifications of this, and that people
415. 14:45can use this kind of tech for doing a
416. 14:47lot of nefarious kind of things. But, I
417. 14:49got to say, in some ways, I wonder
418. 14:51whether Google's version here has
419. 14:53actually been dumbed down a little bit
420. 14:56to be not quite as good as some of the
421. 14:59open versions that are out there. So,
422. 15:01basically, what you do here with this
423. 15:02HTML interface is that you can record in
424. 15:06your reference audio, so this is the
425. 15:08audio to replicate, and then you can
426. 15:10record in the consent audio. And this is
427. 15:13the really important bit where you're
428. 15:15basically giving Google consent to make
429. 15:17a synthetic voice based on your voice.
430. 15:20And if we come down and actually listen
431. 15:22to these, first I'll play you the
432. 15:23original bit that I recorded in here.
433. 15:26The cosmos is within us.
434. 15:28We are made of star stuff. We are a way
435. 15:31for the cosmos to know itself. Okay, so
436. 15:34that was the sort of original recording
437. 15:36I did. And now let's listen to the clone
438. 15:38of my voice done with this. This was
439. 15:41generated from a 30-second sample of my
440. 15:43voice. Now, I'll leave it to you guys to
441. 15:45decide. For me, it sounds quite
442. 15:47different. I think possibly had I
443. 15:49uploaded a longer reference audio, it
444. 15:52might be able to get a better version of
445. 15:53my voice. Probably the best model that
446. 15:55I've seen doing this open wise was the
447. 15:57Quan TTS. I made a whole video about
448. 16:00that when it came out. And I still think
449. 16:02that one is really amazing at how well
450. 16:05it can clone voices. And in some way, I
451. 16:07would say that was their strength, at
452. 16:09least back then. And yet, they were
453. 16:11probably weaker with the descriptions
454. 16:13for creating a voice. And that's the
455. 16:15Gemini's strength in here now. So, just
456. 16:17to finish up, I'd say this is definitely
457. 16:19worth checking out. It is really cool
458. 16:20how easy it is nowadays to basically do
459. 16:24voice things. Things that just used to
460. 16:26be so difficult in the past are now a
461. 16:29very simple API call. And I think for
462. 16:32people who perhaps don't have hardware
463. 16:34to run their own TTS system, to run
464. 16:37something locally, this is going to be
465. 16:39quite nice for just giving you an API
466. 16:41that you can ping and get high-quality
467. 16:44voices out with really nice prosody and
468. 16:47with a custom style that you can define
469. 16:49all by yourself. So, anyway, let me know
470. 16:51in the comments. I know there are TTS
471. 16:53models coming out all the time. I try to
472. 16:55keep up with them. I've missed some. If
473. 16:57you've got any recommendations that you
474. 16:59would like me to cover, please put them
475. 17:01in the comments below. I'd love to hear
476. 17:02that from everyone. Anyway, as always,
477. 17:04if you found the video useful, please
478. 17:06click like and subscribe, and I will
479. 17:08talk to you in the next video. Bye for
480. 17:09now.