Nemotron 3 Diarization - Who Said That? — Transcript

Full transcript

1. 0:00Okay, so there are a lot of good speech
2. 0:01recognition models out there nowadays.
3. 0:04We've got Whisper, we've got Parakeet,
4. 0:06we've got Canary. And if all you need is
5. 0:08a simple transcript of what was said,
6. 0:11then it's tempting to think that that's
7. 0:12kind of a solved problem at this point.
8. 0:14But the big challenge is often if you're
9. 0:17trying to get context for an LLM or a
10. 0:19particular app that you're trying to
11. 0:20build, the transcript on its own isn't
12. 0:23enough. You're often going to want
13. 0:24something like timestamps to know when
14. 0:26something was said. But often even more
15. 0:29importantly than that, you want to know
16. 0:32who actually said it. I mean, if you're
17. 0:34creating a transcript of a podcast where
18. 0:37four people are debating each other, you
19. 0:39need to know who said what, who took
20. 0:42which side, etc. Same is true if you're
21. 0:45trying to do something like meeting
22. 0:46notes. You need to know who agreed on
23. 0:48what action. And especially now as we're
24. 0:51building agents to do a lot of this
25. 0:53stuff, it's so important to be able to
26. 0:55pull quotes and assign them to the right
27. 0:58person rather than just have a
28. 1:00word-for-word transcript of everything
29. 1:02that was said by everyone on the same
30. 1:04call as if it was just spoken by one
31. 1:06person. Now, there've always been ways
32. 1:08to do this, right? I've shown in some of
33. 1:10my videos back with Whisper, doing this
34. 1:13with some of the open-source tools that
35. 1:14were around then. But it's always been a
36. 1:16cat-and-mouse game of just how accurate
37. 1:19can you get it? And then you've got the
38. 1:21challenge of how do you actually deal
39. 1:23with multiple models that are made by
40. 1:25different providers, etc. So this brings
41. 1:28me to what Invidia has just released.
42. 1:30And this is a new model aimed squarely
43. 1:32at this problem. So this is NeMo Tron 3
44. 1:35diarization. It's an open-weights model.
45. 1:38It's under a license that allows for
46. 1:39commercial use. You can certainly do
47. 1:41most of the things you want to do with
48. 1:43this kind of license. And the cool thing
49. 1:45here is that this is only around about
50. 1:46100 million parameters. So it'll run on
51. 1:49a GPU with just 4 gigs of memory. Now,
52. 1:52on top of that, when we start talking
53. 1:53about the accuracy here, this can handle
54. 1:56up to eight speakers. So, most of the
55. 1:59models in the past sort of maxed out at
56. 2:01like three or four speakers. And not
57. 2:04only can it handle eight speakers when
58. 2:05they're speaking nice and cleanly, it
59. 2:07even does a pretty good job when people
60. 2:09are talking over each other. Now, this
61. 2:11is also designed to sit alongside
62. 2:13whatever speech recognition model you're
63. 2:15already using. So, if you've got a
64. 2:16particular favorite like Parakeet or
65. 2:18something like that, you're able then to
66. 2:20basically use this. So, later on I'm
67. 2:22going to show you a demo where we take a
68. 2:23raw podcast recording and turn it into a
69. 2:26transcript where every line is
70. 2:27attributed to the person that spoke. But
71. 2:30first, let me go through a little bit of
72. 2:31context about where this model comes
73. 2:33from. So, InVideo is launching this as
74. 2:35part of their NeMo-Tron speech models.
75. 2:38And on the channel, I've talked about
76. 2:40the NeMo-Tron LLMs, etc. But I think
77. 2:42often most people don't realize that
78. 2:44there are a whole bunch of speech models
79. 2:46that go along with this as well. So, on
80. 2:48the recognition side, you've got
81. 2:50Parakeet, Canary, and NeMo-Tron ASR,
82. 2:53which between them cover 43 languages
83. 2:56for both streaming and for batch
84. 2:57transcription. They've also got a TTS
85. 3:00model in there called Magpie, which is
86. 3:02multilingual and does voice
87. 3:03customization, etc. Then they've also
88. 3:06got some models that do translation and
89. 3:08even full-duplex speech-to-speech stuff
90. 3:11like Persona Plex. And up until
91. 3:12recently, their diarization option was
92. 3:15called Sort Former, which could do up to
93. 3:17four speakers. And for a while, that's
94. 3:19actually been pretty popular. And in the
95. 3:21many ways, you can think of this new
96. 3:23model as just a drop-in replacement for
97. 3:25that, which is just a lot better. And we
98. 3:27know that people have been using these
99. 3:29in production because that old Sort
100. 3:30Former model got over 300,000 downloads
101. 3:33in August alone. So, while InVideo
102. 3:35hasn't said this directly, it seems to
103. 3:38me what they're doing is they're going
104. 3:40through and systematically improving
105. 3:42these models as they're building out a
106. 3:44full set of NeMo-Tron speech models. And
107. 3:47that makes total sense because if you
108. 3:49look at the demand for voice, it's just
109. 3:52getting more and more both for single
110. 3:55user interactions with agents, but also
111. 3:58then for agents going and getting
112. 4:00multimodal things like YouTube videos,
113. 4:03et cetera, to extract information for
114. 4:06context when they're doing research,
115. 4:08when they're doing learning, et cetera.
116. 4:09All right, so if you're new to this,
117. 4:11what actually is diarization? The
118. 4:13simplest way to put this is speech
119. 4:15recognition tells you what has been
120. 4:17said, but diarization tells you who said
121. 4:21it. So when we put them together, we get
122. 4:23what's called a speaker-aware
123. 4:24transcript. And I think instinctively
124. 4:27everyone knows this is really important.
125. 4:29If you're transcribing a meeting and
126. 4:31someone asks a question and then some
127. 4:33people answer yes, some people answer
128. 4:35no, you kind of want your agent to know
129. 4:38who is going to follow through, who's
130. 4:40not going to be doing something. And all
131. 4:42those things are super important for the
132. 4:44agent to determine the context of what
133. 4:46actions it should take going forward.
134. 4:48Now, does it automatically know the
135. 4:49names of everyone who spoke? No, it's
136. 4:51going to basically say speaker one,
137. 4:53speaker two, speaker three, speaker
138. 4:55seven, et cetera, as it goes through.
139. 4:57But it's not that difficult for you to
140. 4:59basically have some code, swap that out
141. 5:01once you tell the model who is speaking.
142. 5:03You can actually turn your transcript
143. 5:05into having real names. Now, one of the
144. 5:07biggest challenges with this has always
145. 5:09been that people talk over each other,
146. 5:12right? They often, even if they're not
147. 5:14trying to deliberately interrupt each
148. 5:15other, when somebody asks a question on
149. 5:17a meeting call, often two or three
150. 5:20people will answer at once. And the
151. 5:21challenge there with older diarization
152. 5:23systems is that they would just get
153. 5:25confused. They weren't very good at
154. 5:27being able to tell one voice from
155. 5:29another voice, especially when they were
156. 5:31crossing over, et cetera. And let me
157. 5:33just say that I don't think any model is
158. 5:35going to be perfect ever at this. Even
159. 5:38for humans, this can be a hard task. But
160. 5:40the way that these models get scored is
161. 5:42with something called diarization error
162. 5:45rate or DER, and it's basically adding
163. 5:48up three kinds of mistakes. Speech that
164. 5:50the model missed, things that it thought
165. 5:53were speech that weren't, and times that
166. 5:55it gave the speech to the wrong person.
167. 5:58So, obviously a lower score is better
168. 6:00here, and you can see that this latest
169. 6:02model from Nvidia is beating out not
170. 6:04only the competition, but beating out
171. 6:07their own previous model by quite a lot.
172. 6:09Another thing to understand here is the
173. 6:11whole idea of streaming versus offline.
174. 6:14In offline mode, the model can look at
175. 6:16the whole recording, including what
176. 6:18comes later on. Whereas in streaming
177. 6:20mode, it has to make its decisions as
178. 6:22the audio arrives. So, for podcasts that
179. 6:25you've already got recorded, etc., you
180. 6:27always want to use offline for that kind
181. 6:28of thing. For any sort of live captions
182. 6:31or a voice agent, you have to use
183. 6:33streaming. All right, so what actually
184. 6:35is new with NeMo Triton 3 diarization?
185. 6:38So, it's about 100 million parameters,
186. 6:40not counting the ASR. The minimum GPU
187. 6:42for this is around 4 gigs. So, it can
188. 6:45actually run on a lot of small GPUs. And
189. 6:47also, it should be able to run pretty
190. 6:49well on things like a laptop RTX card. I
191. 6:52mentioned already that it goes out to
192. 6:53eight speakers. The other thing, too, is
193. 6:55it's language agnostic, and that's going
194. 6:57to work in both the streaming and
195. 6:59offline modes. So, I think the best
196. 7:01thing is let's just jump in and do a
197. 7:02demo and see how it actually works and
198. 7:05comes together. All right, so jumping
199. 7:06into the demo, let me just show you a
200. 7:08little bit about how this is set up. So,
201. 7:11first off, let me thank Nvidia for
202. 7:13sponsoring the compute here. So, I'm
203. 7:15using a DGX Spark to actually run the
204. 7:17three voice models on here. So, let me
205. 7:20just walk through the diagram of how
206. 7:22this is set up. So, what you're seeing
207. 7:24here is a Next.js app. This is running
208. 7:27locally on my Mac. And then, basically,
209. 7:30it's going to then call a back end over
210. 7:33Tailscale to the DGX Spark. And you see,
211. 7:36basically on there, I've got a Docker
212. 7:38container with Nvidia NeMo in there. So,
213. 7:41Nvidia Nemo is serving both diarization
214. 7:45model, which we're talking about, but
215. 7:46also the Parakeet ASR model and the
216. 7:49NeMo-Megatron 3.5 multilingual ASR model
217. 7:52in there. And to basically use those,
218. 7:55I've got a fast API app set up here. And
219. 7:57this is a simple way that you can do
220. 7:59this kind of thing. Now, if I was going
221. 8:00to do this at scale, I might go for
222. 8:02something like Dynamo or Triton to
223. 8:04actually serve this. But, this is
224. 8:06working really well for everything that
225. 8:08I want in here. So, if we come in here
226. 8:10to the app, you can see I can basically
227. 8:12just take their demo in here. I can pick
228. 8:15which ASR model I want to actually do it
229. 8:18with. So, I'm going to do it with the
230. 8:19Parakeet one, cuz this tends to be
231. 8:21better for sort of multi-speaker sort of
232. 8:23stuff. I can start transcribing it. And
233. 8:26you can see that literally in sort of 3,
234. 8:284 seconds, it's gone through and
235. 8:31transcribed this minute and 30-plus
236. 8:34minutes of audio in here. Now, because I
237. 8:36was playing around with this before and
238. 8:38setting some of the names, it's actually
239. 8:40remembered the names of this person in
240. 8:43here. Let's go through and take a listen
241. 8:44to this and see what happens when
242. 8:47multiple speakers speak at the same
243. 8:48time.
244. 8:49>> To celebrate the release of
245. 8:50NeMo-Megatron 3 diarization, we invited
246. 8:53seven more open text-to-speech voices to
247. 8:55the party. I'm Chris, the NeMo-Megatron
248. 8:57voice from Nvidia.
249. 8:58>> Hi everybody. Thanks for having me. I'm
250. 9:01Jane, a Pocket voice from Qutai.
251. 9:03>> Nice to meet you, Chris and Jen. I'm
252. 9:05Serena, the Quinn 3 voice.
253. 9:07>> Hey, I'm Adam from Kakoro. Just 82
254. 9:10million parameters and I still made the
255. 9:12guest list.
256. 9:13>> Hello, hello. Chatterbox here from
257. 9:14Resemble AI. I brought the expressive
258. 9:16energy.
259. 9:17>> Hi, I'm John, a Deia voice from Nari
260. 9:19Labs. Great crowd we've got here
261. 9:21already.
262. 9:21>> What's up, everyone? I'm the Maya 1
263. 9:23voice from My Resear.
264. 9:24>> Okay. So, now we have a full house of
265. 9:27eight synthetic open voices. Let's put
266. 9:29NeMo-Megatron 3 diarization to the test.
267. 9:31Okay. So, the new diarization model can
268. 9:34tag each of our voices as it
269. 9:35transcribes. But, can it do that while
270. 9:37we all speak together? Let's find out.
271. 9:39>> Even two of us talking at once.
272. 9:40>> us, even while we speak over each other.
273. 9:42>> Frame by frame, it labels who spoke
274. 9:45when.
275. 9:46>> Okay, so as you can see in here, there
276. 9:48it's doing a good job at being able to
277. 9:50transcribe each of these. If we come
278. 9:52down and look at this, we can see each
279. 9:54voice has been transcribed. Now, I think
280. 9:56my UI is a little bit behind on this.
281. 9:58But, you can certainly get a sense of
282. 10:00how this is actually put together. And
283. 10:01actually, if we zoom in a bit,
284. 10:03>> Voice from Q Thai.
285. 10:05>> To meet you, Chris and Jen. I'm Serena,
286. 10:07the Queen three voice.
287. 10:09>> And you can see very quickly that we can
288. 10:10basically put something together that
289. 10:12transcribes each of these. And if we
290. 10:15wanted to come in here and then start
291. 10:16putting in their names,
292. 10:18it will go through and actually rename
293. 10:20all of them for us. Now, if you want to
294. 10:22see what actually is coming back from
295. 10:24the actual model, you can see here is
296. 10:27the raw JSON that's coming back from the
297. 10:29model. So, we can see, okay, what was
298. 10:31actually being used in here. We can see
299. 10:33our different speakers are being
300. 10:34flagged, etc. And then this software is
301. 10:37basically rewriting speaker three to be
302. 10:39Adam, etc., going through this. The cool
303. 10:42thing with doing something like this is
304. 10:44very quickly we can get stats out of how
305. 10:48long did each person speak on there? How
306. 10:50many words did they speak? How many
307. 10:52turns did they have? And that can be
308. 10:54really good if you wanted to actually
309. 10:55put in something a lot longer in here.
310. 10:58Okay, and the cool thing here is that we
311. 11:00can do long files. So, I've basically
312. 11:02put in a full podcast episode of the
313. 11:05All-in podcast where they interviewed
314. 11:07Jensen Huang. And you can see it's
315. 11:09basically gone through. We've got out
316. 11:11I've named the actual speakers in here.
317. 11:14So, we've got the speakers going through
318. 11:16there. And then we've got now a full
319. 11:17transcript of what each of them said.
320. 11:21And the cool thing here is we can look
321. 11:22at the stats. We can see how much talk
322. 11:25time each of them had. We can see that
323. 11:28sure enough, obviously Jensen was the
324. 11:29guest here, so he was the one that was
325. 11:31speaking the most. And all of that was
326. 11:33done in 148 seconds. So, that's about 2
327. 11:37and 1/2 minutes for a file that was over
328. 11:40an hour long in here. So, it allows us
329. 11:43to basically go through and you see now
330. 11:45we can basically just go in and either
331. 11:47read the transcript, I can export it as
332. 11:49a text file here, I can export it as an
333. 11:52SRT here. And the cool thing here is
334. 11:54that if I did want to do something that
335. 11:56was going to be multilingual or
336. 11:57something, I can just swap over the
337. 11:59model and use this model as opposed to
338. 12:01the Parakeet model in here. So, you can
339. 12:03see with this one we've got a whole
340. 12:05bunch of different languages supported.
341. 12:07And if you remember when I made a video
342. 12:09about this, you can actually come in and
343. 12:11fine-tune this model to do very well on
344. 12:14other languages that they call
345. 12:15adaptation-ready in here. Now, the other
346. 12:17thing that I haven't really set this app
347. 12:19up to do, which the model can do, is you
348. 12:22can also do streaming responses as well.
349. 12:24And of course, your accuracy will
350. 12:25probably go down a little bit with that,
351. 12:27but you still should be able to get
352. 12:28pretty good results there. So, if you
353. 12:30are interested in doing anything with
354. 12:32voice transcriptions and you want to be
355. 12:34able to have diarization, etc. in there,
356. 12:36this is a really good way that you can
357. 12:38set this up, do all your transcriptions
358. 12:40in batches, etc. And most importantly,
359. 12:43be able to do all of this locally. So,
360. 12:45let me know in the comments if you've
361. 12:46got any questions about this. And I
362. 12:48think it's going to be really cool to
363. 12:49see what NVIDIA does next with these
364. 12:52speech models. Like I said, from what
365. 12:54I'm seeing, they're constantly iterating
366. 12:56on making these better and providing
367. 12:59different kinds of models for different
368. 13:01kinds of use cases. Anyway, as always,
369. 13:03if you found the video useful, please
370. 13:05click like and subscribe, and I will
371. 13:06talk to you in the next video. Bye for
372. 13:08now.