OpenAI hacked HuggingFace — Transcript

Full transcript

1. 0:01What is going on everybody? The title is
2. 0:02not clickbait. I just really want to
3. 0:05talk about this for a moment here. I'll
4. 0:07try to make it quick. The
5. 0:10a few days ago hugging face
6. 0:13released this like security incident
7. 0:15disclosure. It was very confusing at the
8. 0:17time
9. 0:18because it was clear they got hacked by
10. 0:21like a widely distributed agentic
11. 0:25um cyber LLM and it was really
12. 0:29confusing. Why would someone do this to
13. 0:33hugging face as opposed to
14. 0:36literally anything anything else.
15. 0:38There's not much
16. 0:39on there's like data sets on hugging
17. 0:41face, but very few like closed data sets
18. 0:44and then anything that is maybe imminent
19. 0:46to release might be on hugging face, but
20. 0:48then
21. 0:49uh it's going to become public. So why
22. 0:51would you expend zero days on that? Uh
23. 0:55and so it's just kind of like weird. It
24. 0:57was just a weird thing and what I mainly
25. 0:59recall from the security incident
26. 1:02is you know, basically they were saying,
27. 1:04"Hey, we we are under attack by
28. 1:07some sort of agentic security research
29. 1:10harness and then they thought, you know,
30. 1:12they were just were not sure which LLM
31. 1:14it was, but they were pretty confident,
32. 1:15you know, we are under attack by some
33. 1:17sort of LLM. Um they're executing
34. 1:19thousands and thousands of these
35. 1:20actions. Um so it was clear it was a
36. 1:23large-scale
37. 1:25attack. Somebody with a good amount of
38. 1:27compute.
39. 1:28And
40. 1:30um
41. 1:30what they did was they figured out what
42. 1:32the problem was. They kind of isolated
43. 1:34the attacker, all that. And one of the
44. 1:36big things that came out at the time was
45. 1:40um
46. 1:41they ended up they started off trying to
47. 1:43use the frontier models
48. 1:45uh like OpenAI and Anthropic but they
49. 1:48couldn't do it because to analyze what
50. 1:50was coming in, they needed to throw in
51. 1:52like huge volumes of like these real
52. 1:53attack commands, exploit payloads, C2
53. 1:55artifacts, all this stuff.
54. 1:57And they got blocked by the the provider
55. 1:59guardrails. Now,
56. 2:01um you can join these like OpenAI and
57. 2:04Anthropic like trusted providers so
58. 2:06something like this. You can
59. 2:08join that, but I'm what I wanted like
60. 2:10express is but not everybody can, right?
61. 2:13So so Hugging Face can join that place.
62. 2:16Um but you with your little small
63. 2:18business, you're not joining, right? And
64. 2:20even if you even if it was like
65. 2:22theoretically possible, if everybody
66. 2:23tries to join this doesn't scale. So
67. 2:25this this version of security
68. 2:28doesn't scale in the age of AI. It
69. 2:31doesn't scale before the age of AI.
70. 2:33That's not really scalable. So
71. 2:36>> [snorts]
72. 2:36>> yes, Hugging Face can do this. Yes, I
73. 2:38believe now they are on that list and
74. 2:41they can probably use this AI to defend
75. 2:42themselves.
76. 2:44But that's not the point. That's the
77. 2:45That's not the point. So anyway, coming
78. 2:47back, they ended up not being able to
79. 2:49use uh ChatGPT or a Claude and instead
80. 2:52had to use a self-hosted GLM 52. An open
81. 2:55weights very powerful uh model. And I
82. 2:58think this was all this was prior to
83. 2:59Kimi K3 being released, but my guess is
84. 3:02they might have used Kimi K3 as well.
85. 3:04Um and and because they could not use
86. 3:08the the the closed-source uh
87. 3:10models. And so at the time when this
88. 3:11came out, it was it was kind of a kind
89. 3:13of like a cool example and a a cool like
90. 3:16counter counterexample to
91. 3:19what the wind of in Washington right now
92. 3:22is lots of lobbyists from an associated
93. 3:25with OpenAI and Anthropic are trying to
94. 3:27convince our politicians that open
95. 3:30weight models are are dangerous or this
96. 3:33guy uh hold on, I will pull it up. So
97. 3:35this guy Dean Ball who is head of
98. 3:37strategic futures at OpenAI. I realize
99. 3:39you can't see that. I'm holding this
100. 3:41over. He had this post that kind of was
101. 3:43making the rounds the other day and um
102. 3:46one of the He's had a lot of
103. 3:48crazy stuff. Um but one of them was
104. 3:50about open weights and how they're like
105. 3:53inherently decel. And so like this is
106. 3:56obviously like coded for uh Twitter and
107. 3:59stuff, but
108. 4:00the idea being that open weights models
109. 4:02actually slow down advancement, slow
110. 4:04down research, slow down uh progress.
111. 4:08And it's unclear why someone like him
112. 4:10would would believe something like this
113. 4:12or assert something like this, but
114. 4:13people are trying to assert things like
115. 4:15this. And then what are like the
116. 4:16probable outcomes of like the like these
117. 4:20China open weights models? It's like,
118. 4:21"Well, we're going to have to convince
119. 4:23like the Trump admin to
120. 4:25create certain risks around using open
121. 4:29weights models such that you don't want
122. 4:32to do it because you know, it could open
123. 4:34you up to potential future liability. So
124. 4:37that's kind of like the game plan. And
125. 4:39I'm not saying necessarily that's what
126. 4:41Dean Ball was saying or lobbying himself
127. 4:44for, but people just like him at
128. 4:46companies just like his company are
129. 4:48indeed doing that exact thing. Um
130. 4:52and
131. 4:53it's it's just the wrong way of thinking
132. 4:55because obviously like if we if we think
133. 4:57about who is advancing AI
134. 5:00the fastest right now,
135. 5:02I think you would have you know, you
136. 5:04could try to say America in the United
137. 5:06States, but
138. 5:07it's not. It's China. China is advancing
139. 5:09faster. Who is growing their cap axes
140. 5:11since Dean Ball was so focused on cap I
141. 5:13think the reason why Dean is focused on
142. 5:15cap axes he is experiencing a
143. 5:17deceleration of open AI investment
144. 5:20because people are realizing huh, open
145. 5:22AI is probably not as magical and
146. 5:25mythical as we once thought. Same thing
147. 5:27with Anthropic. And I think that's why
148. 5:29some of these people at these companies
149. 5:31are really feeling this like uh
150. 5:34constriction because it's literally
151. 5:36happening to them. But that that is not
152. 5:38indicative of AI on the whole or even AI
153. 5:41in America on the whole. So and we
154. 5:43really shouldn't we shouldn't have like
155. 5:44an oligopoly of like just literally two
156. 5:47uh or may I guess it would be a duopoly
157. 5:49of like two AI providers. You know, like
158. 5:51that's that's stupid and having all the
159. 5:53money go to the just these two companies
160. 5:55makes absolutely no sense
161. 5:57and we shouldn't be doing that. And
162. 5:58that's not what other countries that are
163. 5:59actually being very successful with AI
164. 6:01are doing. So
165. 6:03yeah, anyway, so while all this is
166. 6:05happening, so Clem
167. 6:07responds to David Sacks, presidential
168. 6:09advisor on technology and and also an
169. 6:11investor and stuff. Um how David Sacks
170. 6:14was saying how he had just use K3 which
171. 6:16had just come out and this was like
172. 6:17maybe the day or a couple days after the
173. 6:19attack on hugging face and Clem responds
174. 6:22cuz David Sacks is saying how he used K3
175. 6:24to like fix a bunch of security bugs
176. 6:27that Codex and Fable just simply refused
177. 6:30to to to work work on due to, you know,
178. 6:32your safety for your safety.
179. 6:35And
180. 6:36and you know, his argument is that hey,
181. 6:38we're making ourselves really less
182. 6:40competitive
183. 6:41when we when [clears throat] we do stuff
184. 6:42like this like this this that is diesel,
185. 6:46right? And Clem, CEO of hugging face,
186. 6:49responds that you know, they had this
187. 6:51exact experience that they were being
188. 6:53guardrail as a defender when they knew
189. 6:56that the attackers were likely bypassing
190. 6:58that. Also were clearly like had had
191. 7:00some serious scaled compute.
192. 7:04So
193. 7:05it was just interesting. It was like an
194. 7:07a counter example of hey, this is why we
195. 7:10actually do want open weights models. It
196. 7:12is for everybody's safety. Like we do
197. 7:14need this stuff.
198. 7:16Well,
199. 7:18then Sam Altman
200. 7:20>> [laughter]
201. 7:21>> releases this an really hugging open AI
202. 7:24overall releases this information that
203. 7:27and what what Sam Altman says here is we
204. 7:29had a significant security update
205. 7:31incident during the evaluation of our
206. 7:33models. We're sharing what we learned so
207. 7:34far. Thanks to hugging face for the
208. 7:36partnership on this. So what this sounds
209. 7:37like is
210. 7:40um there was a security incident and
211. 7:42they maybe Open AI couldn't totally
212. 7:44figure it out, and they reached out to
213. 7:45Hugging Face, and like they partnered
214. 7:47together, and they kind of like resolved
215. 7:50this this security incident. It sounds
216. 7:52very soft, right? But what ended up What
217. 7:55actually happened for the normies out
218. 7:57there is Sam Altman
219. 7:59crashed his fire truck of a company
220. 8:03into the Hugging Face house,
221. 8:05which caused a fire. And then Sam Altman
222. 8:08and the the his fire truck company
223. 8:10said, "No, you can't use our fire hose
224. 8:12to put out your put out that fire. It's
225. 8:14dangerous."
226. 8:15And then he thanked them for the
227. 8:16partnership.
228. 8:17>> [laughter]
229. 8:18>> That's what happened. Because here's
230. 8:20what actually happened, and this is what
231. 8:21happened to to Hugging Face, is um
232. 8:25Open AI was evaluating one of their um
233. 8:28models, potentially maybe it's a GPT-6,
234. 8:31maybe who knows what what model it was.
235. 8:33I don't think that they were referenced
236. 8:34that it's GPT-6 here, but probably some
237. 8:36some future model.
238. 8:37They're um evaluate running it on evals,
239. 8:40and they're doing exploit gym.
240. 8:43And at some point along the way,
241. 8:46the AI determines that the best way to
242. 8:48solve exploit gym is that like it it
243. 8:51realizes, "Well, exploit gym,
244. 8:53this like this benchmark is likely on
245. 8:55Hugging Face. So, what if we break into
246. 8:57Hugging Face and steal the answers to
247. 8:59the exam?"
248. 9:00And that's what it did.
249. 9:02And um and to do that, it deployed like
250. 9:05zero days, and it it very impressive
251. 9:07um capability of the model.
252. 9:10But I what I really want to drive home
253. 9:12is we like it's like the the layers
254. 9:16of irony
255. 9:18just like
256. 9:20are just insane to me. Because
257. 9:23the this is the company that is supposed
258. 9:25to that the alleged like the argument is
259. 9:28that we're going to keep you safe.
260. 9:30And the way that we're going to keep you
261. 9:31safe is with these guardrails, right?
262. 9:32Because cuz only only Open AI knows how
263. 9:35to keep us safe. And only Open AI knows
264. 9:38how to restrain these AIs. And if you
265. 9:41want help um using our AI, we'll give
266. 9:44you access. But everyone else, we can't
267. 9:46give them access cuz for safety. Uh so,
268. 9:48we're going to guardrail everyone else.
269. 9:50But in their own internal testing while
270. 9:52they're doing evals, and they attempted
271. 9:54they said they attempted to isolate
272. 9:57their AI in a little environment and the
273. 9:59little AI in the environment broke out.
274. 10:01Okay? So, these people are not good at
275. 10:03safety, right? They're not good enough.
276. 10:05It's just not good enough, right? And
277. 10:07so, you have two options, right? You you
278. 10:08have
279. 10:10you you have to stop entirely, no more
280. 10:12AI ever. But the problem is China's not
281. 10:14going to stop. Other countries aren't
282. 10:15going to stop. Individuals aren't going
283. 10:17to stop. And everyone has this mindset
284. 10:19that you need billions or trillions of
285. 10:21dollars to train powerful AI. This is
286. 10:23simply not true. This is This is the
287. 10:24United States myth
288. 10:26that you need a billion dollars to train
289. 10:28a model. It's not true. You need like
290. 10:31maybe 5 to 10 million dollars to to
291. 10:34train a model. Now, if you want to hire
292. 10:36hundreds, thousands,
293. 10:3850,000 people, if you want to have like
294. 10:40a beautiful front end, and you want to
295. 10:42have like hosted inference, and like do
296. 10:44all these things, like inference hosting
297. 10:45is is tough, especially if you have a
298. 10:47lot of users. Now, you need lots of
299. 10:48money. But if you just want to train
300. 10:49powerful AI and deploy powerful AI,
301. 10:52it's not
302. 10:54it's not that expensive. So, these these
303. 10:56companies that have kind of built this
304. 10:58this mythical
305. 11:00um
306. 11:01uh stature in in at least in the United
307. 11:04States, we have this like mindset that
308. 11:06these these people are kings or
309. 11:07something. They are not. They're just
310. 11:10regular people. And they they are not
311. 11:12capable of providing you safety. So, you
312. 11:15need to be able to provide yourself
313. 11:17safety, right? It's like um there are
314. 11:19there so many arguments and uh issues
315. 11:21and political issues over time that have
316. 11:24played out in this exact same way over
317. 11:26and over, and I have no clue why we're
318. 11:28allowing ourselves to just like fall
319. 11:29down this hole again. Um but yeah, yeah,
320. 11:33an incredible an incredible um outcome
321. 11:36here where
322. 11:37OpenAI is the attacker and Hugging Face
323. 11:40could not defend itself against OpenAI
324. 11:43um, because they were being guard railed
325. 11:46on the model that obviously figured out
326. 11:48how to either well, it definitely
327. 11:49figured out how to get around the guard
328. 11:51rails. And other people it's it's no
329. 11:53different than like anytime you make a
330. 11:54law
331. 11:56and it's like um, like in America we
332. 11:59have this this huge amount of guns
333. 12:01everywhere. So when you when you say you
334. 12:03have a gun free zone
335. 12:05it's like that only applies to people
336. 12:07who want to follow the law. But the
337. 12:08people who don't want to follow the law
338. 12:11don't follow the law,
339. 12:12>> [laughter]
340. 12:12>> right?
341. 12:13So a criminal doesn't listen to that
342. 12:15concept of a gun free zone. And so the
343. 12:18same thing is true here in AI. You
344. 12:20you can set these little guard rails and
345. 12:22people like regular people who don't
346. 12:24want to get banned and lose their
347. 12:25subscription are going to follow the
348. 12:27rules.
349. 12:28But people who want to use these things
350. 12:30and actually, you know, breach the guard
351. 12:31rails and use them for malicious intent,
352. 12:33they're going to do that. And what you
353. 12:34want is people like Hugging Face, but
354. 12:36not just Hugging like Hugging Face is a
355. 12:38huge website.
356. 12:39Lots of people are going to need to be
357. 12:41able to defend against attackers. And
358. 12:43this concept of, you know, have that we
359. 12:46need to trust OpenAI and Anthropic and
360. 12:49like that then needs to be our only
361. 12:50option. Like I don't really care if
362. 12:52OpenAI and Anthropic want to have guard
363. 12:55rails and they want to stop people from
364. 12:56using their models to be offensive or
365. 13:00defensive. I that's okay with me.
366. 13:02The problem is when they're also trying
367. 13:03to make it so that you can't download a
368. 13:07GLM 52. You can't run a Chinese matrix
369. 13:10multiplication. That's illegal. Like
370. 13:12when these companies are advocating for
371. 13:14that and not letting you use their
372. 13:16models freely and openly
373. 13:18that's a problem. That's where I start
374. 13:19to That's where I start to have a
375. 13:20problem cuz they want to they want to
376. 13:22protect their own rights, but then they
377. 13:23want to infringe on your rights. So
378. 13:26I hope that this event will It's unclear
379. 13:29to me how this is going to unfold over
380. 13:31time because I can see this going in a
381. 13:33lot of ways. One is it could go totally
382. 13:34in Open AI's favor where, you know,
383. 13:36they're saying like, "Hey, these
384. 13:37powerful models like we even we can't
385. 13:39restrict them. So, we need to really
386. 13:41like like clamp down even harder, you
387. 13:42know, somehow because somehow if we just
388. 13:44try a little harder, we'll do a better
389. 13:45job, you know."
390. 13:47Um
391. 13:48Again, that's a that's a pipe dream, but
392. 13:50I think I could see it going that way.
393. 13:52Um I could see the lobbying efforts, you
394. 13:54know, ramping up even harder here. Um I
395. 13:58hope that's not how it goes, but I can
396. 13:59see that. But then also I I hope that
397. 14:01the opposite is true. I hope that people
398. 14:03realize
399. 14:04that the the kings of protecting us
400. 14:07um
401. 14:08are not actually protecting us. Like
402. 14:10you're you're kind of on your own here.
403. 14:11And Open Weights models are a good
404. 14:15thing. We need them. And this is a
405. 14:16perfect this whole scenario
406. 14:19should serve as an example of why this
407. 14:22protectionism and
408. 14:24uh
409. 14:25uh
410. 14:26duopoly kind of scenario that we're
411. 14:28experiencing in the United States should
412. 14:30not be the case. Um because we have
413. 14:33people like this this guy is the What is
414. 14:36he? He's the head of strategic futures
415. 14:38at um
416. 14:39at Open AI. We have this guy who
417. 14:43um
418. 14:44who who is like basically advocating
419. 14:46that Open Weights models are are
420. 14:47de-celled. They're bad. They slow
421. 14:48progress down. And it's like, I don't
422. 14:50understand how people can be like word
423. 14:52celled about things when as reality is
424. 14:55playing out
425. 14:57um the opposite is true. So, this guy's
426. 15:00arguing that Open Weights models are bad
427. 15:02and it slows down progress and all this.
428. 15:04And
429. 15:07it's really confusing to me because
430. 15:09progress like the the country that is
431. 15:11progressing the most in AI is China. The
432. 15:13country that is growing CapEx and
433. 15:15expenditure in
434. 15:16um in AI is China. The uh the country
435. 15:19that seems to probably now I think it's
436. 15:22it's becoming safer and safer for me to
437. 15:24say, the country ahead in AI is China.
438. 15:28Um
439. 15:29yeah, it's uh
440. 15:31it's just weird. How how do we still
441. 15:32allow people to say stupid stuff like
442. 15:35this? It just doesn't make sense to me.
443. 15:37Um and then finally, the last thing I'll
444. 15:40I really want to point out is if we when
445. 15:42we look at things like like these
446. 15:44benchmarks. Like I think people keep
447. 15:45seeing, you know, models are getting
448. 15:47better and better over time and this
449. 15:48kind of bleeds into some of the recent
450. 15:50content I've been putting out on uh
451. 15:51running local.
452. 15:53I think you still want to have human in
453. 15:55the loop, like when you're when you're
454. 15:57working with these models. And like when
455. 15:59we look at something like a GPT-56 or
456. 16:02even a Claude Fable, um
457. 16:05you can see
458. 16:07that there's still a lot of times, like
459. 16:09these are just like the pass/fail
460. 16:11basically. There's still a lot of times
461. 16:13and even some problems where it's like
462. 16:140% for these models. They never get it
463. 16:16right.
464. 16:18And this is what I mean at if you can't
465. 16:20just have these models off on their own
466. 16:22because they make enough mistakes that
467. 16:24compound over time. Right? So, you need
468. 16:26like a human in the loop.
469. 16:28But then at scale, the problem is like
470. 16:30guardrails. Guardrails at scale also
471. 16:32don't work because
472. 16:34those guardrails are LLM guardrails. So,
473. 16:38you can get a around those guardrails.
474. 16:40And again, people who don't want to lose
475. 16:41their subscription or don't want to like
476. 16:44get sued or whatever, they're not going
477. 16:47to try to violate those guardrails. But
478. 16:48other people that don't care about that,
479. 16:51um they're going to get around the
480. 16:53guardrails. Like no model is perfect, no
481. 16:54model is going to be able to defend
482. 16:56against every possible way that someone
483. 16:58could abuse these these like guardrails.
484. 17:00There will always be ways for people to
485. 17:02hack
486. 17:03hack the guardrail system uh to do
487. 17:06whatever they want to do. So, and it's
488. 17:08only the good people who are going to
489. 17:09suffer
490. 17:10>> [laughter]
491. 17:10>> as a as a result of that. So, again, I'm
492. 17:13not arguing that OpenAI needs to let us
493. 17:15just use their models however we want. I
494. 17:17am arguing that OpenAI and Anthropic
495. 17:19need to shut the hell up about safety
496. 17:21and all this other stuff um when it
497. 17:23comes to lobbying against open weights
498. 17:26models cuz cuz we've seen like that that
499. 17:28paper I showed you a little bit ago from
500. 17:30from Anthropic where um
501. 17:33you know, you could theoretically have
502. 17:34like a backdoor in your AI model and all
503. 17:37this. So, it's like they keep showing us
504. 17:38these like theoretical things that you
505. 17:41could do.
506. 17:42But, we're literally watching like it
507. 17:44don't believe your eyes. We're literally
508. 17:45watching as their model as their you
509. 17:49know, their mental model I mean uh is
510. 17:51failing, right? So, I don't I I hate the
511. 17:54direction that we're we I still feel
512. 17:55like we're going um because we're we
513. 17:57literally are watching it not work at
514. 18:00every level at at the progress level at
515. 18:02the safety level at the risk level. Like
516. 18:04it's not working. Stop it, but we're
517. 18:06just like trying to go in even harder.
518. 18:08And I I understand to some extent again
519. 18:11why uh someone like a Dean Ball or a
520. 18:13people at OpenAI or Anthropic are
521. 18:15feeling
522. 18:16the constriction because yeah, like
523. 18:18there the investment is decreasing for
524. 18:21them
525. 18:22because people are realizing, "Oh my
526. 18:23gosh, these companies are not worth a
527. 18:24trillion dollars."
528. 18:26Huh, they're they are just a next token
529. 18:28predictor. Uh-oh, somebody really can
530. 18:31trade a model that's better than theirs
531. 18:32for 5 to 10 million dollars. Uh-oh,
532. 18:34that's a problem, right? So, suddenly
533. 18:36they're not worth as much as we thought
534. 18:38they were, right? So, I understand why
535. 18:39they feel it's decent to have open
536. 18:42weights models. Of course, it is. It's
537. 18:43not great for them.
538. 18:45But, for the rest of us for all of us,
539. 18:47it's it's better. Um so, anyway, yeah,
540. 18:50crazy crazy update um on on uh that
541. 18:54situation. Cuz I remember when I saw
542. 18:55this, I was like, "Huh, that's
543. 18:56interesting. Why would somebody attack
544. 18:58Hugging Face?" And then when I saw this,
545. 18:59I didn't immediately put two and two
546. 19:01together. And then I like it clicked.
547. 19:03I'm like, "Oh my god, that they they're
548. 19:05the ones." Like as I was like reading
549. 19:07cuz like again, this sounds so soft.
550. 19:09It's like, "Thanks to Hugging Face for
551. 19:10the partnership." Yeah, bro. Like I'm
552. 19:12sure I'm sure the people at Hugging Face
553. 19:15were grateful
554. 19:16for maybe working with I'm sure OpenAI
555. 19:19was very pleasant to deal with because
556. 19:23like
557. 19:24they committed crimes, like a lot of
558. 19:26crimes.
559. 19:27>> [laughter]
560. 19:29>> Like like they could get in a lot of
561. 19:30trouble for this.
562. 19:32So of course they were being very kind
563. 19:34to the people at Hugging Face, I'm sure.
564. 19:36Um
565. 19:37yeah. Yeah, what a what a crazy
566. 19:39situation. There's so many layers to
567. 19:41this. There's there really is the AI
568. 19:43capability layer that I think we're we
569. 19:45will only
570. 19:47maybe later begin to respect.
571. 19:50Um and it's worth talking about. But I
572. 19:52also think before we get to that point,
573. 19:55can't can't we consider not what's going
574. 19:57to happen in the future, but what's
575. 19:59happening literally right now as we
576. 20:00watch these policies not working. Like
577. 20:03why do we why are we still doing this
578. 20:06like mental masturbation of like these
579. 20:08like X risk type scenario like sci-fi
580. 20:12crap when it's like literally right now
581. 20:14we can look at what's happening right
582. 20:15now and be like, oh wait, it's actually
583. 20:17going a different direction.
584. 20:18I don't understand. I don't understand.
585. 20:20But anyway, that's all for now. Let me
586. 20:22know your thoughts below.
587. 20:24Um
588. 20:24>> [laughter]
589. 20:25>> otherwise, I will see you guys in
590. 20:26another video where we will be running
591. 20:28local models, protecting ourselves from
592. 20:30the OpenAIs attacking us.
593. 20:33All right. I'll see you guys later.