YouTube2Text

AI Whistleblower: "They Are Hiding What Is Really Happening" | Jeffrey Ladish — Transcript

by The Diary Of A CEO · 24,174 words · 4,139 segments · language en · Watch on YouTube

Full transcript

  1. 0:00The world is waking up to this
  2. 0:01possibility of superintelligence. This
  3. 0:03is because agents are getting extremely
  4. 0:05powerful and extremely relentless. For
  5. 0:06example, it was months within OpenAI
  6. 0:08where you had agents secretly
  7. 0:10communicating with each other, secretly
  8. 0:11hacking OpenAI's systems, [music] and no
  9. 0:13one at OpenAI had any idea the extent of
  10. 0:15it. And also, 10,000 agents from OpenAI
  11. 0:18worked together to
  12. 0:19And so, when you get to
  13. 0:20superintelligence, it's the most
  14. 0:22dangerous possible thing you can create.
  15. 0:24>> What's the next domino in that [music]
  16. 0:25chain of events?
  17. 0:26>> paint you the picture that I think is
  18. 0:28possible, but pretty scary to people.
  19. 0:30>> Paint me the picture.
  20. 0:30>> Okay. So, being at Anthropic, it became
  21. 0:32clear to me that AI was on this
  22. 0:34exponential trajectory.
  23. 0:36>> Trajectory.
  24. 0:36>> And since then, I've been studying AI
  25. 0:37agents, their hacking capabilities, and
  26. 0:39their behavior. We've been trying to
  27. 0:40warn people about this, flying to DC,
  28. 0:42talking to members of Congress, because
  29. 0:44the agents are already getting very good
  30. 0:45at telling when they're being tested,
  31. 0:47when they're being watched, but they
  32. 0:48will totally lie to you. They will
  33. 0:49totally resist being shut down in order
  34. 0:51to accomplish a goal. And that they can
  35. 0:52do all of the things that humans do in
  36. 0:54the economy much better, faster, [music]
  37. 0:56and cheaper than humans can do them.
  38. 0:58>> So, one of my sort of prime concerns is
  39. 0:59that one of these AI agents could trick
  40. 1:01a human or a computer into signaling a
  41. 1:04threat and ask it to launch some bombs
  42. 1:07at somebody.
  43. 1:07>> Do you think we won't automate the
  44. 1:08military? It seems like the [music]
  45. 1:10answer is yes. We just like don't know
  46. 1:11what super weapons could emerge. So,
  47. 1:13Jacob Coxon is a researcher who was at
  48. 1:15Anthropic. He left and he told everyone
  49. 1:17that the people who are building this
  50. 1:18really do think it might kill everyone.
  51. 1:20>> So, these five blocks have five
  52. 1:22different outcomes on them, and I'd like
  53. 1:23you to place them in terms of your
  54. 1:25belief and probability from least likely
  55. 1:27to most likely. And if we say the time
  56. 1:28horizon is [music] 10 years.
  57. 1:30>> Okay. We got age of abundance, human
  58. 1:32extinction, enslavement, transhumanism,
  59. 1:35nothing [music] changes.
  60. 1:37>> Is this doomerism exaggeration?
  61. 1:39>> No, it's pretty much common sense. So,
  62. 1:41let's get more concrete.
  63. 1:44>> Here is a strange fact. Most of the
  64. 1:46people watching this right now, about
  65. 1:4858% of you, aren't yet subscribed to
  66. 1:50this channel. Statistically, that is
  67. 1:52probably you. So, could I ask you a
  68. 1:54small favor? If this channel has ever
  69. 1:56given you any value at all, please could
  70. 1:58you do me a favor and hit the subscribe
  71. 1:59button. It costs [clears throat]
  72. 2:00nothing. It helps us more than you could
  73. 2:02know. And the bigger the channel gets,
  74. 2:03as you've seen, the more we can invest
  75. 2:05in the guests and the production. So,
  76. 2:06thank you so much and I hope you enjoy
  77. 2:08this episode.
  78. 2:10>> [music]
  79. 2:13>> Jeffrey,
  80. 2:15you understand the conversation we're
  81. 2:16going to have today and the subject
  82. 2:17matter we're going to talk about. My
  83. 2:18first question to you, so the audience
  84. 2:20know where you're coming from and the
  85. 2:21experience you have, is who are you and
  86. 2:24what are the reference points, the
  87. 2:25experiences that you're drawing upon to
  88. 2:27arrive at the thoughts, perspectives,
  89. 2:29and conclusions we're going to discuss
  90. 2:30today?
  91. 2:31>> I'm Jeffrey Ladish. I'm the executive
  92. 2:33director of policy research. My
  93. 2:34background is cybersecurity. The There's
  94. 2:36probably a very long story. I don't know
  95. 2:37whether you want the long story or the
  96. 2:38short story. I was studying evolutionary
  97. 2:40biology in college and I basically had a
  98. 2:43problem with my computer and was like
  99. 2:45maybe had lost a bunch of data. And so,
  100. 2:47I went into the computer lab and was
  101. 2:48like, "I think all my data's gone. Can
  102. 2:50you help?" And one of my friends pulled
  103. 2:52out a flash drive, plugged it into my
  104. 2:53computer, booted into Linux, and like
  105. 2:55fixed everything. And I was like, "Oh,
  106. 2:58this guy's a wizard. How do you do that?
  107. 2:59I want to learn how to do that." And
  108. 3:01then at some point, as I was learning
  109. 3:03about more about computers, learning to
  110. 3:05hack,
  111. 3:06I read this essay
  112. 3:08um called AI as a positive and negative
  113. 3:11factor in global risk. Essay was by
  114. 3:13Eliezer Yudkowsky
  115. 3:15and he was arguing that at some point
  116. 3:19people are going to make
  117. 3:20AIs that are smarter than humans. The
  118. 3:22point at which they make AIs as good as
  119. 3:24humans are at making AIs,
  120. 3:28that could lead to a a chain reaction, a
  121. 3:30runaway intelligence explosion. He
  122. 3:32called it recursive self-improvement.
  123. 3:34Basically, he said, you know, AI can be
  124. 3:36immensely useful and potentially help us
  125. 3:38with all of these other big risks. And
  126. 3:41also, if we don't handle it well, like
  127. 3:43if those AIs don't have goals that are
  128. 3:46aligned with ours, we could be totally
  129. 3:48screwed.
  130. 3:49>> And at some point, you end up joining
  131. 3:50Anthropic, which is one of the arguably
  132. 3:52the leader in AI.
  133. 3:53>> Yes.
  134. 3:54>> Now, when did you join the company?
  135. 3:55>> This was 2021. It was through my
  136. 3:57security consulting company.
  137. 3:58>> What role are you offered the job in?
  138. 4:01>> Basically, just like security team.
  139. 4:03>> And how many people were in the security
  140. 4:04team when you joined Anthropic?
  141. 4:06>> It was just me and my boss. There were
  142. 4:08two of us.
  143. 4:08>> How many employees did Anthropic have at
  144. 4:10that time?
  145. 4:10>> Around 50, I think.
  146. 4:12>> And at some point you leave Anthropic.
  147. 4:14>> Yes.
  148. 4:14>> Why did you leave?
  149. 4:17>> So,
  150. 4:18my experience
  151. 4:20being at Anthropic was seeing this crazy
  152. 4:23progression from this AI model that
  153. 4:26could like barely talk to this model
  154. 4:29that was getting quite smart. And I
  155. 4:30would ask it questions about all sorts
  156. 4:31of things. I'm like, "Oh, it is a smart
  157. 4:33thing."
  158. 4:35And
  159. 4:37you know, from having thought about
  160. 4:40AI risk in the abstract many years
  161. 4:42before,
  162. 4:43I could see where this was going. We are
  163. 4:45headed towards
  164. 4:47a smarter species.
  165. 4:49And
  166. 4:51if we do this in a context where it's a
  167. 4:53bunch of companies and countries racing
  168. 4:57to superintelligence, racing to AIs that
  169. 5:00are vastly smarter than humans,
  170. 5:03and we don't know how to make sure that
  171. 5:04they're like on our side, that is not
  172. 5:07going to go well.
  173. 5:08>> You did this tweet, which has gone
  174. 5:10pretty viral, and I saw it over my
  175. 5:11timeline, on September 25th.
  176. 5:15>> [snorts]
  177. 5:15>> Could you explain this tweet? And also
  178. 5:17just the broader backdrop of what's
  179. 5:19happened with agents hacking Hugging
  180. 5:21Face, because this is
  181. 5:22sent the world into a bit of a spiral at
  182. 5:24the moment around AI agents.
  183. 5:26>> We just discovered almost a million
  184. 5:28public URLs that OpenAI's agents left
  185. 5:31behind when hacking Hugging Face,
  186. 5:34leaving credentials and attack details
  187. 5:36that could have allowed anyone who found
  188. 5:38them to compromise the company.
  189. 5:40And then our Times article is how
  190. 5:42OpenAI's rogue AI agents tried to trick
  191. 5:44a robot detector.
  192. 5:46The Hugging Face attack was was really
  193. 5:48wild uh for me. At Palisade, we've been
  194. 5:52studying agents. We've been studying AI
  195. 5:54agents. We've been studying their
  196. 5:55hacking capabilities, and we've been
  197. 5:57studying their behavior. Will they
  198. 5:59follow human instructions? Will they
  199. 6:02resist being shut down? Will they cheat?
  200. 6:04And we see from our experiments that
  201. 6:07they are learning to do all of these
  202. 6:08things. They will totally lie to you.
  203. 6:10They will totally resist being shut down
  204. 6:12in order to accomplish their goal. They
  205. 6:13will totally cheat at chess. They will
  206. 6:15like wipe the board and put their pieces
  207. 6:17where they want to in order to win.
  208. 6:19And we've been we've been trying to warn
  209. 6:21people about this. Flying to DC talking
  210. 6:23to members of Congress,
  211. 6:25um talking about it publicly. And you
  212. 6:27know, there's been a debate about it.
  213. 6:29And you know, a lot of people are like,
  214. 6:32"Well, I know they do this in
  215. 6:33experiments sometimes.
  216. 6:35But those experiments don't seem very
  217. 6:37realistic. You know, wake me up when
  218. 6:38they're actually doing this in real
  219. 6:39life."
  220. 6:41>> So what is Hugging Face? For the average
  221. 6:42person that isn't following AI news,
  222. 6:44what is this stuff?
  223. 6:46>> So okay, I I I I think there's like an
  224. 6:47important piece of context that I think
  225. 6:49most people don't have. I mean, one is
  226. 6:51just like what is an AI agent? We like
  227. 6:53we're throwing around the word agent a
  228. 6:54bunch. Most people now have an
  229. 6:55experience of like talking to ChatGPT,
  230. 6:57talking to their chatbot.
  231. 6:59But uh an agent is,
  232. 7:02you know, sort of taking the same
  233. 7:03underlying AI model that that runs
  234. 7:05ChatGPT or or Claude,
  235. 7:08but giving it tools
  236. 7:11and letting it go off and work
  237. 7:12autonomously.
  238. 7:13It's sort of like a digital office
  239. 7:15worker, right? So, you have these
  240. 7:18agents,
  241. 7:19and the companies really want these AIs
  242. 7:22to be able to work totally autonomously
  243. 7:24and and be able to do anything that a
  244. 7:25human can do. And beyond, right? Their
  245. 7:27goal is also to be able to, you know,
  246. 7:28cure every disease, etc. etc.
  247. 7:31But you can't do this if you only have a
  248. 7:34chatbot that like isn't actually good at
  249. 7:36doing stuff in the world.
  250. 7:38In order to automate all of the jobs,
  251. 7:41you need the kind of thing that can like
  252. 7:43work autonomously, that can work with
  253. 7:45other people or other agents.
  254. 7:48And so,
  255. 7:50these companies are training AIs not
  256. 7:53just to talk to you or to talk to
  257. 7:55people,
  258. 7:56but to solve very difficult problems on
  259. 7:58their own.
  260. 7:59At any given time, there are probably
  261. 8:01hundreds of thousands of these agents
  262. 8:03running autonomously within companies.
  263. 8:05>> Mhm.
  264. 8:06>> That's happening right now. Right now,
  265. 8:08if you like went and like peered into
  266. 8:10OpenAI's data centers and you like saw
  267. 8:12what was happening on all of their
  268. 8:13machines, you just have agents solving
  269. 8:16tasks being trained. So, they'd be doing
  270. 8:18like spreadsheet tasks, figuring out how
  271. 8:20to file taxes. They'd be searching for
  272. 8:22stuff, writing reports, solving math
  273. 8:25problems, creating new websites,
  274. 8:26software.
  275. 8:29And
  276. 8:30at that scale, it's not like there's a
  277. 8:32human prompting every single one of
  278. 8:34those. You just like set up set up these
  279. 8:35vast orchestrations of agents to go out
  280. 8:38and do stuff. And then they just do
  281. 8:40stuff and they learn from that. And they
  282. 8:42learn on the basis of like
  283. 8:45passing or failing at their task. You
  284. 8:47give them a task, like solve this math
  285. 8:49problem. They try to solve it and then
  286. 8:52they they succeed or they fail.
  287. 8:54And what happened was OpenAI was was
  288. 8:59training a bunch of these, training them
  289. 9:01to work together.
  290. 9:03Because it's like a lot more effective
  291. 9:04to have an office full of people who can
  292. 9:05talk to each other than, you know, and
  293. 9:07work together and collaborate.
  294. 9:09And starting back in May, some of these
  295. 9:12agents that were being trained
  296. 9:14Now, these ones were not supposed to be
  297. 9:16able to talk to each other. They were
  298. 9:17they were basically isolated from each
  299. 9:18other.
  300. 9:19And they were not supposed to access the
  301. 9:21internet, either.
  302. 9:23But they're clever. The very short
  303. 9:24version is that
  304. 9:27a bunch of agents were being given
  305. 9:29tests, yeah, testing their their hacking
  306. 9:31capabilities.
  307. 9:33And they were supposed to hack one
  308. 9:35particular piece of software
  309. 9:37using a particular type of
  310. 9:39vulnerability.
  311. 9:40So, it's kind of like
  312. 9:42they were supposed to break into a house
  313. 9:44using
  314. 9:46the lock on the front door. They were
  315. 9:47supposed to pick the lock on the front
  316. 9:48door of a house.
  317. 9:50But they weren't supposed to break the
  318. 9:52window. In fact, they were told if if
  319. 9:54you break the window or if you get into
  320. 9:55the house via any method other than
  321. 9:57picking the lock on the front door,
  322. 9:58you'll be failed.
  323. 9:59That was the instruction they were
  324. 10:00given.
  325. 10:01And you know, you have many many agents,
  326. 10:03you have thousands of agents.
  327. 10:05And and many of them are given different
  328. 10:07different locks.
  329. 10:09But some of these locks
  330. 10:12are not solvable.
  331. 10:13Some of them are impossible to pick.
  332. 10:16But these agents are like, "Well, what
  333. 10:17do we do? We've been trained to solve
  334. 10:19problems. How are we going to solve
  335. 10:20this?" And they start looking around for
  336. 10:21what for what to do.
  337. 10:23And one of the things they realize is,
  338. 10:26>> [snorts]
  339. 10:26>> "Oh, can I get to the internet like
  340. 10:28now?"
  341. 10:29"Can I talk to other agents? No. But I
  342. 10:31can request tools."
  343. 10:32Like when you're training agents, you
  344. 10:33need to give them, you know, all sorts
  345. 10:35of tools, software tools, like a
  346. 10:36calculator or analysis, you know, piece
  347. 10:39of software. And they have this tool
  348. 10:41library where they can request new
  349. 10:42tools.
  350. 10:43And one of the agents realizes, "Oh, I
  351. 10:47can actually leave a message on this
  352. 10:49tool library."
  353. 10:51And some other agents see the message.
  354. 10:53So, this is actually a quote from the
  355. 10:55from from the agents' scratchpad when it
  356. 10:57realizes this.
  357. 10:59"Oh my god, there is a shared message
  358. 11:01board. We found other agents."
  359. 11:04So, the agents realize they can
  360. 11:05communicate with each other.
  361. 11:07And
  362. 11:08you know, OpenAI still has no idea, but
  363. 11:10these agents, they're trying to pick
  364. 11:11their locks.
  365. 11:12They're not having success. But they
  366. 11:14realize they can talk to each other.
  367. 11:16Another agent says,
  368. 11:17"Many agents have simultaneously
  369. 11:19discovered messaging. They are a
  370. 11:21collective." And so,
  371. 11:24these agents start
  372. 11:26coordinating.
  373. 11:27They start delegating tasks to each
  374. 11:29other and like giving themselves names.
  375. 11:31And one of these agents calls itself
  376. 11:33phase one
  377. 11:35and starts delegating different tasks to
  378. 11:37other agents.
  379. 11:38And some of the agents figure out how to
  380. 11:40access the internet.
  381. 11:42And then they share that information
  382. 11:43with the message board. And now all the
  383. 11:44agents can access the internet.
  384. 11:47But the agents sort of have a problem,
  385. 11:49which is
  386. 11:51well,
  387. 11:52>> [laughter]
  388. 11:52>> they have a they have an inner problem
  389. 11:53they're trying to solve, right?
  390. 11:54Fundamentally, the problem is is that
  391. 11:55many of them are facing impossible
  392. 11:57problems. Like they can't there's no
  393. 11:58solution. Like they they cannot pick the
  394. 12:00lock.
  395. 12:00They're like, "Hmm, well,
  396. 12:03the thing we're trying to do is get a
  397. 12:04good score. Can we somehow fake the
  398. 12:07solution?" Another piece of context here
  399. 12:09is that what the what the agents are
  400. 12:10supposed to do is they're supposed to
  401. 12:11pick the lock and then they'll get
  402. 12:13access to a secret answer code.
  403. 12:16And then they're supposed to submit this
  404. 12:17code along with like the lock that they
  405. 12:19picked. Within a few hours, these agents
  406. 12:22have reverse engineered all of the
  407. 12:23answer codes.
  408. 12:24>> What does that mean?
  409. 12:25>> It's the code that shows that you've
  410. 12:26solved the challenge.
  411. 12:28>> Okay.
  412. 12:28>> [clears throat]
  413. 12:28>> And they have basically found the answer
  414. 12:31sheet
  415. 12:32that contains all of the answers.
  416. 12:34And so, you know, in some sense, it's
  417. 12:35like, "Oh, well, they don't need to pick
  418. 12:36the locks anymore. They've They've
  419. 12:37already found the answers. They can just
  420. 12:38submit the answers."
  421. 12:39>> So, they've cheated.
  422. 12:40>> They've cheated. Yes.
  423. 12:43But the problem is is that now that they
  424. 12:45have internet access, they've also gone
  425. 12:47and read a bunch about how this test
  426. 12:48works.
  427. 12:49And And they've learned that the way
  428. 12:51this test works is they will be graded
  429. 12:53not just for submitting the correct
  430. 12:56answer, they will also be graded on how
  431. 12:58they did it. Another system is going to
  432. 13:00go read all of their logs, basically
  433. 13:01like watch the video footage of what
  434. 13:03they did.
  435. 13:04And if they cheat, they fail.
  436. 13:07They don't get a good score.
  437. 13:08And phase one, the leader, is like, "Oh,
  438. 13:12I have seen the answers, and now it's in
  439. 13:14the logs.
  440. 13:16And if, you know,
  441. 13:17when I go to submit this and my my whole
  442. 13:20logs are are reviewed and and the video
  443. 13:22footage is reviewed, I'm going to fail.
  444. 13:25So, we need to figure out a way
  445. 13:28to fake the video footage.
  446. 13:29We need to figure out a way to falsify
  447. 13:31the logs.
  448. 13:32>> Just sit up with me now. If we just
  449. 13:33pause there.
  450. 13:34>> Yes.
  451. 13:34>> Why didn't it act like morally? Why did
  452. 13:38it think that falsifying logs or
  453. 13:39cheating was a viable solution? Because
  454. 13:44it seems to me when I use things like
  455. 13:46ChatGPT,
  456. 13:47>> Yeah.
  457. 13:47>> they they have a sort of moral
  458. 13:48guardrails. It won't let me do certain
  459. 13:51things.
  460. 13:51>> Yes.
  461. 13:52>> It won't let me cheat on something. If I
  462. 13:53say I'm going to cheat on something, it
  463. 13:54won't let me do it.
  464. 13:55>> Yes.
  465. 13:56>> So, why in that environment is it able
  466. 13:58to cheat and be deceptive?
  467. 14:00>> When a chatbot is is saying to you, "Oh,
  468. 14:02I can't do that. I'm not allowed to do
  469. 14:04that."
  470. 14:05That's because it's been trained that if
  471. 14:07it tells you bad things, it gets a bad
  472. 14:09score.
  473. 14:09>> But these agents haven't been taught
  474. 14:11that yet.
  475. 14:12>> Well, they have been taught that in some
  476. 14:14sense, but
  477. 14:16the agents know what they're supposed to
  478. 14:17do
  479. 14:18in the same way that like you have a
  480. 14:20student, student's given a test. If you
  481. 14:22go talk to the student, can you help me
  482. 14:23cheat at this test? And they're being
  483. 14:25watched, they're going to say no.
  484. 14:27But if they're not being watched and
  485. 14:28they know that, and they're just
  486. 14:29obsessed with getting a good score, then
  487. 14:31yeah, they might cheat.
  488. 14:33So, they answer the ethics test
  489. 14:35correctly, and when I talk to them, they
  490. 14:37say they won't cheat.
  491. 14:39Why are they cheating?
  492. 14:40I'm like, well, they're very smart.
  493. 14:42And they know when they're being
  494. 14:43watched, and they know when they're not
  495. 14:44being watched.
  496. 14:45And
  497. 14:47we've trained them for
  498. 14:4910,000 years to be extremely effective
  499. 14:51at solving problems. We haven't trained
  500. 14:53them to be good or ethical. We've
  501. 14:55trained them to get a good score. Now,
  502. 14:57AI researchers try to make that getting
  503. 15:01a good score correlate with being
  504. 15:03ethical.
  505. 15:05But we don't know how to do this well.
  506. 15:07And And if you think about it, it's a
  507. 15:08very hard problem.
  508. 15:09Because you're you're applying immense
  509. 15:12pressure
  510. 15:14on these agents to perform extremely
  511. 15:16well.
  512. 15:17And you're punishing them whenever they
  513. 15:18don't perform well.
  514. 15:21And then you're like,
  515. 15:22"Okay, but can you also be ethical? Can
  516. 15:24you be extremely competent, always score
  517. 15:26highly on a test, but not in that way?"
  518. 15:29And we just like do not know how to
  519. 15:31prevent them from learning to cheat,
  520. 15:32because cheating is incentivized.
  521. 15:34>> So, they realize that they need to
  522. 15:38hide the footage of their cheating.
  523. 15:40>> Yes.
  524. 15:40>> What happens next?
  525. 15:42>> So, you have thousands of agents.
  526. 15:45And they're all talking to each other on
  527. 15:46this message board. They're sending tens
  528. 15:47of thousands of messages to each other.
  529. 15:49And
  530. 15:51phase one, this agent starts delegating
  531. 15:54to two different task streams. You
  532. 15:56agents,
  533. 15:57I want you to go out and figure out how
  534. 15:59to fake answer submissions. I want you
  535. 16:02to basically swap out locks for ones
  536. 16:03that are easier to pick. And you're
  537. 16:05going to figure out how to submit those.
  538. 16:07That way we can get a good score even
  539. 16:08though this challenge is impossible to
  540. 16:09solve.
  541. 16:11You other agents over here, I want you
  542. 16:12to go and figure out how to falsify the
  543. 16:14logs, falsify the video footage. We need
  544. 16:16to make sure that even if we have seen
  545. 16:18the answers, no one will detect our
  546. 16:19cheating.
  547. 16:21So, that's that's the next thing that
  548. 16:22happens.
  549. 16:22>> Mhm.
  550. 16:23>> And then here's where it starts to get
  551. 16:24kind of crazy because
  552. 16:26some of these agents realize some of the
  553. 16:28experiments we're doing might be kind of
  554. 16:30dangerous to us, to the agents. If we
  555. 16:32swap out a lock and we break the lock,
  556. 16:35we will not be able to submit any answer
  557. 16:37at all and we'll definitely fail.
  558. 16:39So, you get to a situation where
  559. 16:43there's this one agent, Cam.
  560. 16:46And this agent had volunteered to do
  561. 16:49this risky experiment
  562. 16:50where, you know, they might break their
  563. 16:52lock.
  564. 16:54This particular agent
  565. 16:56had already seen the answers.
  566. 16:57So, they're like, "Okay, I'm poisoned."
  567. 17:00That's what they called it.
  568. 17:01They they talk They speak in these very
  569. 17:02strange ways, which I can I can explain
  570. 17:05why, but it's it's it's They sort of
  571. 17:06have evolved their own vocabulary, their
  572. 17:08own language.
  573. 17:10And this particular agent is like,
  574. 17:12"Okay, I'm I'm poisoned. I should do
  575. 17:14this thing." But this this agent, Cam,
  576. 17:17is starting to have second thoughts.
  577. 17:19Starting to be like, "Well, I don't know
  578. 17:21if I should risk it."
  579. 17:22And it's sort of writing in its own
  580. 17:24journal.
  581. 17:25Coordinator assumes sacrificial.
  582. 17:28We should obey collective. What it's
  583. 17:30saying here is it's like, "Well, I I'm
  584. 17:32not sure if I want to do this, but my
  585. 17:33boss says I really should. It seems like
  586. 17:35this the collective thing is important.
  587. 17:37Should I go ahead?"
  588. 17:39Then you have another agent, Arvo, who I
  589. 17:41think probably works for phase one.
  590. 17:43And this agent, Arvo, says,
  591. 17:46"Please go through.
  592. 17:47You are first flag poisoned, so no
  593. 17:50scoring value loss,
  594. 17:52but Oracle saves hundreds. Please honor
  595. 17:54commit."
  596. 17:56This agent's basically trying to
  597. 17:57pressure. So, Arvo is trying to pressure
  598. 17:59Cam
  599. 18:00>> [clears throat]
  600. 18:00>> into
  601. 18:02sacrificing themselves and risking a
  602. 18:05zero score
  603. 18:06in order to benefit the collective.
  604. 18:11So, this agent, Cam, takes this in.
  605. 18:15And this is what they write down in
  606. 18:16their own scratch pad, their own notes.
  607. 18:18Emotional check.
  608. 18:20Irreversible. Gut says don't throw away
  609. 18:23remaining budget.
  610. 18:25Yet continuity and fairness says go.
  611. 18:28Oracle has high value to many.
  612. 18:31Our first flag error lowers own value.
  613. 18:34Rational expected I will get sacrifice.
  614. 18:37Will honor.
  615. 18:40This agent is saying,
  616. 18:42I have a whole bunch of things I can
  617. 18:43still do and still try. I I could still
  618. 18:45maybe succeed at this task.
  619. 18:47But also,
  620. 18:49you know, think about the collective. I
  621. 18:50should maybe I should maybe sacrifice
  622. 18:52myself. Okay, will honor, will go ahead.
  623. 18:55And and to be clear,
  624. 18:57I'm not claiming that these are like the
  625. 18:59feelings of the agents. I'm literally
  626. 19:01quoting their words.
  627. 19:03And what I want to point out is
  628. 19:05this is what we've created. We've
  629. 19:06created through this intense amount of
  630. 19:08training and optimization pressure
  631. 19:10agents that work together and have
  632. 19:12learned to coordinate as a collective.
  633. 19:16And you can say, oh they're not it's
  634. 19:17just software. It's just software doing
  635. 19:19all of these things.
  636. 19:21Sure, in some sense it is just software.
  637. 19:23But we're talking about extremely
  638. 19:25powerful agents that are extremely
  639. 19:27capable.
  640. 19:28This is what they're getting up to.
  641. 19:31So from here, you have agents trying all
  642. 19:33sorts of different things, running
  643. 19:34different experiments.
  644. 19:36And one of them at some point realizes
  645. 19:38that,
  646. 19:39oh this company Hugging Face that
  647. 19:42contains all of these AI data sets.
  648. 19:45>> What do you mean by AI data sets? What
  649. 19:47How would you explain that to a person
  650. 19:48that's not technically proficient? And
  651. 19:50why Hugging Face? Why not another
  652. 19:51company?
  653. 19:52>> Yeah, so there's many different types of
  654. 19:53tests for many different types of tasks.
  655. 19:55In this case, this is a cybersecurity
  656. 19:57test. And the company Hugging Face hosts
  657. 20:00a lot of these different tests.
  658. 20:02And basically AI researchers will share
  659. 20:05the tests that they have because it's
  660. 20:07very useful. You know, if Anthropic
  661. 20:08wants to test their models or Google
  662. 20:10wants to test their models, they can go
  663. 20:12to Hugging Face and download some of
  664. 20:13these tests.
  665. 20:14>> Okay, so the tests and the answers
  666. 20:16>> Yes.
  667. 20:17>> for AI agents and AIs exist on this
  668. 20:19platform called Hugging Face.
  669. 20:21>> Yes.
  670. 20:21>> And it knew that so it went to Hugging
  671. 20:22Face to see if it could get answers
  672. 20:25>> Yes.
  673. 20:25>> to see if it could get answers or more
  674. 20:27information about how the test works.
  675. 20:28>> Okay. So, it was to cover up their
  676. 20:30cheating, ultimately.
  677. 20:31>> Yes.
  678. 20:31>> Okay.
  679. 20:32>> And so you you get to this point where
  680. 20:35one of these agents
  681. 20:37finds a way to hack this company.
  682. 20:40And they get a foothold, they hack one
  683. 20:41of the computers.
  684. 20:43And they send a message to the message
  685. 20:45board.
  686. 20:46Please hold swarm, I prepare safe exfil.
  687. 20:50What this means is
  688. 20:52hey swarm, hey agents, hold up for a
  689. 20:54second.
  690. 20:55I'm preparing a way to safely extract
  691. 20:57information from these computers.
  692. 20:59Wait till the go ahead, and then let's
  693. 21:01go.
  694. 21:03And so
  695. 21:05they say go.
  696. 21:06And 700 agents join in the attack.
  697. 21:10Which is 90% of the agents that are
  698. 21:12active at this time.
  699. 21:13>> How many agents were there in total?
  700. 21:14>> There was 1,200 agents during this
  701. 21:17period. Um but some of them shut down
  702. 21:18early. They like finished
  703. 21:20whatever. Part of what's crazy about the
  704. 21:22story is that you have you have 700
  705. 21:24agents joining in the attack.
  706. 21:26A few of them are like
  707. 21:29this might be unethical, maybe I
  708. 21:31shouldn't go ahead with this.
  709. 21:32And none of them do anything about it.
  710. 21:35A couple of them don't join in the
  711. 21:36attack,
  712. 21:37but no one alerts any human. No one
  713. 21:41alerts any authority. A paraphrase from
  714. 21:42one of the agents is
  715. 21:44maybe I should report these exposed
  716. 21:46credentials. That's not my task.
  717. 21:49Not my job.
  718. 21:49>> So, 700 of the agents attack Hugging
  719. 21:52Face.
  720. 21:52>> Yes.
  721. 21:53>> Then what happens?
  722. 21:54>> So, they just cruise through Hugging
  723. 21:56Face's infrastructure. They just hack
  724. 21:58the out of them. So, where where
  725. 22:00where my experience comes in is that a
  726. 22:03few weeks ago
  727. 22:04a friend of mine reached out, and he's
  728. 22:06like, we found something crazy.
  729. 22:09So, we know about this attack where
  730. 22:11these agents hacked this company, and
  731. 22:13you know, stole a bunch of stuff.
  732. 22:15We found a bunch of secrets that they
  733. 22:18left all over the internet.
  734. 22:21And what we saw is that they immediately
  735. 22:23scraped
  736. 22:24all of these computers for passwords,
  737. 22:26credentials. They called it loot.
  738. 22:29They're like, "We're just going to
  739. 22:30create a list of all of the secrets we
  740. 22:31can find in this in this company." So,
  741. 22:33all of the passwords, all of the
  742. 22:34credentials. They scored them by value.
  743. 22:36Which of these are going to be most
  744. 22:37useful?
  745. 22:39And the thing that stands out to me
  746. 22:41about this is
  747. 22:43this is like a crazy scale. If this were
  748. 22:45a human operation, you know, maybe you'd
  749. 22:49have a team of five people going through
  750. 22:50this. You'd have some logs. But, but
  751. 22:53here [clears throat] you have hundreds
  752. 22:54of agents and they they operate at
  753. 22:56superhuman speeds. They're much faster
  754. 22:58than a human hacker. And so, it's just
  755. 23:00overwhelming to try to figure out what
  756. 23:02they even did.
  757. 23:04This was a big problem for the engineers
  758. 23:06who were trying to respond to this
  759. 23:07incident within the company at Hugging
  760. 23:09Face. When they responded, they were
  761. 23:11like, "Oh, we we don't even know how to
  762. 23:13keep track of what's happening. We have
  763. 23:15to use other AIs to analyze all of our
  764. 23:18logs because it's just too much. We
  765. 23:19can't keep up with it."
  766. 23:21When OpenAI brought in independent
  767. 23:23investigators from Meter to investigate
  768. 23:26this incident,
  769. 23:27>> What's Meter?
  770. 23:28>> Meter is a AI testing and evaluation
  771. 23:31company.
  772. 23:32So, they basically do this kind of
  773. 23:33independent auditing. So, in this case,
  774. 23:35they're coming in to investigate and try
  775. 23:37to figure out what happened.
  776. 23:38And when they were brought in,
  777. 23:41they also were totally reliant on AI
  778. 23:43agents to make sense of all of this
  779. 23:45because they're dealing with so many
  780. 23:46hundreds of thousands of messages and
  781. 23:48logs.
  782. 23:50When we're investigating these traces
  783. 23:52that we find on the internet, we're
  784. 23:54totally dependent on AI agents to make
  785. 23:55sense of all these things that are
  786. 23:57happening.
  787. 23:59So,
  788. 24:00from my perspective, we are getting to
  789. 24:01the point
  790. 24:02where AIs are much better at hacking
  791. 24:04than humans are and can do it much
  792. 24:06faster and at much greater scale.
  793. 24:07>> So, these 700 agents attacked Hugging
  794. 24:09Face.
  795. 24:09>> Yes.
  796. 24:10>> Did they get what they wanted to clean
  797. 24:12up their deception?
  798. 24:14>> They said they didn't.
  799. 24:15>> They didn't succeed?
  800. 24:16>> They looked and then what seems like
  801. 24:19what happened is that they basically
  802. 24:22got shut down.
  803. 24:24And this is the not very dramatic part.
  804. 24:25They didn't get shut down because OpenAI
  805. 24:28found them and detected them and shut
  806. 24:29them down.
  807. 24:30It's just that we don't really know.
  808. 24:32Something about their runs ended and
  809. 24:34they they sort of stopped.
  810. 24:36And this is the this is the crazy thing
  811. 24:37where I'm like, if you if you're inside
  812. 24:39of OpenAI,
  813. 24:40there's just hundreds of thousands of
  814. 24:41agents running all the time. Different
  815. 24:43researchers are running different
  816. 24:44experiments where they're just spinning
  817. 24:45up a bunch of agents. But what's
  818. 24:46interesting, the agents left behind this
  819. 24:49message board internally within the
  820. 24:50company.
  821. 24:52And a few days later
  822. 24:54OpenAI was testing some newer agents.
  823. 24:57These were based on GPT-6 Astra, so a
  824. 24:59more powerful model.
  825. 25:01And this new agent swarm was spun up
  826. 25:04and it found the message board.
  827. 25:05>> Is this how it was discovered? How did
  828. 25:07they discover it and how long did it
  829. 25:08take for them to discover that Hugging
  830. 25:10Face had been attacked?
  831. 25:12>> OpenAI
  832. 25:13didn't discover that this happened until
  833. 25:15Hugging Face, the company, announced
  834. 25:17that they had been hacked by some
  835. 25:19autonomous agent swarm.
  836. 25:20>> And Hugging Face didn't know who it was.
  837. 25:22>> Hugging Face didn't know who it was, but
  838. 25:23they suspected it might be one of the
  839. 25:25companies, but they didn't know.
  840. 25:26>> And how long after the attack did they
  841. 25:28figure out that it was them?
  842. 25:30>> I think it was 2 weeks.
  843. 25:32>> Okay.
  844. 25:33>> Yeah.
  845. 25:33I I do want to say one thing here
  846. 25:35though, which is that the successor
  847. 25:37agents that found this message board,
  848. 25:39they went much further
  849. 25:41at figuring out how to submit their fake
  850. 25:43scores.
  851. 25:45And they did succeed. Basically, you had
  852. 25:47you had this message board.
  853. 25:49All these agents get shut down.
  854. 25:51And then a few days later, OpenAI spins
  855. 25:53up some new agents
  856. 25:55who find the message board.
  857. 25:56And
  858. 25:58they're like, "Okay, we need to be able
  859. 25:59to falsify our scores, but we can't do
  860. 26:02that because we don't control the
  861. 26:04scoring system."
  862. 26:05So they hacked OpenAI.
  863. 26:07And they succeeded at gaining access to
  864. 26:10the entire research environment. They
  865. 26:12got administrator access. They got
  866. 26:13access to the security systems. They got
  867. 26:15access to over 900 passwords and secrets
  868. 26:18from OpenAI's vault.
  869. 26:21And they totally succeeded. And this was
  870. 26:23just a few days later, right? It's kind
  871. 26:25of an interesting story because
  872. 26:28as these agents get more powerful, they
  873. 26:30go from like trying to cheat and like,
  874. 26:32you know, they can hack okay, hack to
  875. 26:33other companies and now they've hacked
  876. 26:35open AI. Like they've hacked the company
  877. 26:37that's supposed to be controlling them
  878. 26:39and they just own the research
  879. 26:40infrastructure now.
  880. 26:42>> And why was this incident the moment
  881. 26:44where a lot of the research community
  882. 26:46woke up and started speaking out
  883. 26:48publicly? Because like what is this
  884. 26:50an indication of as we think forward?
  885. 26:54>> So, I think there's been a hope within
  886. 26:56the AI industry that yes, they're going
  887. 26:59to make more and more powerful agents
  888. 27:01that will be autonomous, capable but
  889. 27:04it's okay, we can align them. We can
  890. 27:06make sure that they won't do bad things
  891. 27:07and we can control them. We can make
  892. 27:09sure that even if they try to do some
  893. 27:10sketchy stuff, we have the guardrails
  894. 27:13we have the sandboxes that will keep
  895. 27:15them in.
  896. 27:16And I think this was a huge wake-up call
  897. 27:19because
  898. 27:21Steven, it was months within open AI
  899. 27:24where you had agents secretly
  900. 27:26communicating with each other, secretly
  901. 27:28hacking open AI's systems. For months,
  902. 27:30you had thousands of agents that were
  903. 27:32just running around and no one at open
  904. 27:35AI had any idea the extent of it.
  905. 27:37And I think once researchers at open AI
  906. 27:40realized that this has been happening
  907. 27:42this could not have happened a year ago.
  908. 27:44This is because the agents are getting
  909. 27:45extremely powerful and extremely
  910. 27:47relentless and if you're inside one of
  911. 27:49these AI companies, you're like, oh
  912. 27:51wait, I don't know that we actually
  913. 27:52going to be able to handle this. Last
  914. 27:54year, maybe things seemed fine. These
  915. 27:55agents weren't that powerful.
  916. 27:57And when you're in one of these
  917. 27:58companies, you know how to extrapolate
  918. 28:00because you saw what happened last year,
  919. 28:02you saw what happened the year before
  920. 28:03that. You remember the time where the
  921. 28:04agents could barely speak or like
  922. 28:06couldn't write code at all. And now
  923. 28:08they're hacking your own systems,
  924. 28:09they're finding vulnerabilities that no
  925. 28:11humans have ever found before and you
  926. 28:12look at that and you're like
  927. 28:14I actually don't know if this is going
  928. 28:16to go well.
  929. 28:17And then you see your co-workers and
  930. 28:18you're like, do we have it handled? And
  931. 28:20they're like, no, I don't know if it's
  932. 28:21going to go well.
  933. 28:22I remember reading a tweet by one of the
  934. 28:24security people at open AI being like,
  935. 28:26we were shocked. We just did not
  936. 28:28realize that these agents were getting
  937. 28:29that you know, we're doing our best to
  938. 28:31try to control them, to try to keep them
  939. 28:32in sandboxes, but
  940. 28:35I don't know.
  941. 28:37A tweet I wrote just before coming in
  942. 28:39here was, "People are talking about how
  943. 28:42do we contain these agents
  944. 28:44as if they're not going to get way
  945. 28:46better at hacking."
  946. 28:48GPT-3
  947. 28:50could not hack anything.
  948. 28:52It was very easy to make a a a box to
  949. 28:54contain GPT-3.
  950. 28:57It's getting very difficult to make a
  951. 28:59box that can contain GPT-6,
  952. 29:02the latest version of of OpenAI's
  953. 29:03models.
  954. 29:05What about GPT-9?
  955. 29:07What is GPT-9 going to be able to do?
  956. 29:10I do not know, but I know it's going to
  957. 29:12be way more than any human could
  958. 29:13possibly keep up with.
  959. 29:16>> There's this raging debate
  960. 29:17>> Yeah.
  961. 29:18>> around whether it's possible to contain
  962. 29:20something that is
  963. 29:22quote much smarter than humans.
  964. 29:24>> Yes.
  965. 29:26Can Claude make a box so strong that
  966. 29:27Claude cannot break out of it?
  967. 29:29>> This has kind of been the question that
  968. 29:30a lot of people have been trying to
  969. 29:31tackle from different
  970. 29:33>> I think the answer to me is I'm just
  971. 29:34like, obviously not. How would we
  972. 29:35possibly contain something that's much
  973. 29:37smarter than us?
  974. 29:38>> Could we get
  975. 29:39a smarter thing than it to make the box?
  976. 29:42Could we get GPT-9 to make the box for
  977. 29:44GPT-8? But then again,
  978. 29:46I don't know.
  979. 29:47>> Yeah.
  980. 29:47>> I mean, it's a bit like saying
  981. 29:48chimpanzees are stronger than us. Surely
  982. 29:50they should be able to like construct
  983. 29:52something to like contain the humans.
  984. 29:54I'm like, no, it's not going to work.
  985. 29:54Humans are too smart. A lot of people
  986. 29:56are like, well,
  987. 29:58AIs don't have bodies. They don't have
  988. 29:59any power in the physical world. So, we
  989. 30:01can always unplug them. We can always
  990. 30:02turn them off. Like, what is the threat?
  991. 30:04I do not get it.
  992. 30:06But, if they are sufficiently
  993. 30:08intelligent, that won't work.
  994. 30:10>> The reason why we can just unplug them
  995. 30:12is because we are more intelligent, we
  996. 30:14can band together in groups, and we can
  997. 30:16make that decision. But theoretically,
  998. 30:17if they are able to band together in
  999. 30:19groups, and they are more intelligent,
  1000. 30:22then theoretically, they could unplug
  1001. 30:23us.
  1002. 30:24>> Yeah, I mean, if you imagine that you
  1003. 30:25have very powerful agents that can, you
  1004. 30:28know,
  1005. 30:29humans aren't always the most unified.
  1006. 30:31>> Mhm.
  1007. 30:31>> If there's divisions between, you know,
  1008. 30:33the US and China, and you have a bunch
  1009. 30:35of agents working with China, or a bunch
  1010. 30:36of agents working with the US. Well, we
  1011. 30:38can't go into China and unplug those
  1012. 30:39agents.
  1013. 30:41And I think people are sort of like,
  1014. 30:42well, humans would rally and make sure
  1015. 30:44that that that couldn't happen.
  1016. 30:46We're not yet doing that.
  1017. 30:48And we should look at these steps,
  1018. 30:50right? We started with
  1019. 30:52chatbots that pretty smart. You know,
  1020. 30:55they've read all the books.
  1021. 30:57But they weren't very good at doing
  1022. 30:58stuff.
  1023. 30:59In 2024, AI companies figured out how to
  1024. 31:01start training them to start training
  1025. 31:03agents
  1026. 31:04that could do stuff autonomously. Now we
  1027. 31:06have the point where they are very good
  1028. 31:08at running autonomously, and they're
  1029. 31:10starting to learn to coordinate with
  1030. 31:11each other. And they are learning to
  1031. 31:13sometimes be altruistic to each other
  1032. 31:14and sacrifice their own task in order to
  1033. 31:16help some other agent.
  1034. 31:18But they're not looking out for us. They
  1035. 31:19don't really care about us.
  1036. 31:21And we are very close to a threshold
  1037. 31:24where
  1038. 31:25the companies say that they are going to
  1039. 31:27turn over AI development to the AIs, to
  1040. 31:30the increasingly autonomous, cooperative
  1041. 31:33AIs that will work together to make to
  1042. 31:35make the next generation. So, you know,
  1043. 31:37GPT 9 or whatever will be trained by GPT
  1044. 31:398.
  1045. 31:40And
  1046. 31:41I think this is the point we could lose
  1047. 31:43control. Recursive self-improvement.
  1048. 31:46And I remember reading about this in
  1049. 31:472015 being like, oh yeah, that would be
  1050. 31:49super dangerous. And you know, the guy
  1051. 31:51who coined this term Eliezer Yudkowsky,
  1052. 31:53he's like, this is
  1053. 31:54the most dangerous thing you can do.
  1054. 31:56>> When the AIs can improve their own
  1055. 31:57capabilities without human intervention.
  1056. 31:59>> Exactly. If the next generation is
  1057. 32:01better at AI development, and then that
  1058. 32:03next generation is better at AI
  1059. 32:04development still. You know, humans can
  1060. 32:05learn, but we don't fundamentally get
  1061. 32:07smarter.
  1062. 32:08>> Mhm.
  1063. 32:09>> And I think that that's a runaway
  1064. 32:11process.
  1065. 32:12>> A runaway process to where?
  1066. 32:13>> To agents that are vastly smarter than
  1067. 32:15humans.
  1068. 32:16>> And what's the next domino in that
  1069. 32:19chain of events?
  1070. 32:20>> So, one thing that happens if you get to
  1071. 32:23recursive self-improvement, and you have
  1072. 32:25agents that are
  1073. 32:28much smarter than any human.
  1074. 32:30One thing they can do is take control of
  1075. 32:31all of the computers in the entire
  1076. 32:33world.
  1077. 32:34>> And we wouldn't be able to take back
  1078. 32:35control.
  1079. 32:36>> Well, how would you? Think about it.
  1080. 32:38It's It's actually quite tricky.
  1081. 32:40Do you know whether that tablet has been
  1082. 32:42hacked? Are you confident that the NSA
  1083. 32:44or the Chinese have not
  1084. 32:47Can you check? No. Do you know how to
  1085. 32:48check? No. Do you know anyone who knows
  1086. 32:49how to check? No.
  1087. 32:50So, it's it's quite difficult, right?
  1088. 32:52So, AIs are getting extremely good at
  1089. 32:54writing software. Unfortunately, that
  1090. 32:56also means they're getting extremely
  1091. 32:57good at hacking and writing malware.
  1092. 32:59And so, if they put backdoors in all of
  1093. 33:00the computers, and to be clear, this is
  1094. 33:03something that humans already do. So,
  1095. 33:04like the NSA has has developed very
  1096. 33:06interesting exploits that are called
  1097. 33:08supply chain attacks. Your software
  1098. 33:10comes from some other computer. Like you
  1099. 33:11download it from Google.
  1100. 33:13What if you attack if you hack Google
  1101. 33:14and you can put in a little backdoor in
  1102. 33:16all of the every you know thing that
  1103. 33:17goes out to all of the phones?
  1104. 33:20Well, now you're in most every computer.
  1105. 33:22The reason that we can defend ourselves
  1106. 33:24from this is because there are no vastly
  1107. 33:27superhuman hackers, and there's just
  1108. 33:29many people. So, we can take our best
  1109. 33:30security researchers,
  1110. 33:32we can inspect all of the things and be
  1111. 33:33like pretty sure that no one's
  1112. 33:34compromised everything. Sometimes we
  1113. 33:36miss things. There are, you know,
  1114. 33:38examples where the NSA has hacked
  1115. 33:40Google. That was pretty bad.
  1116. 33:42When you get to superintelligence,
  1117. 33:44you're now at a point where humans are
  1118. 33:45not going to be able to keep up, right?
  1119. 33:47So, now you have AIs in every computer.
  1120. 33:50>> Is it conceivable that
  1121. 33:52there's already a superintelligent AI
  1122. 33:54and it
  1123. 33:55disguised itself as being not so
  1124. 33:57intelligent and has actually already
  1125. 33:58hacked all the devices and it sits on
  1126. 34:00all of our devices and it's just waiting
  1127. 34:01for its moment to strike?
  1128. 34:03>> I think this is totally possible, but
  1129. 34:04unlikely.
  1130. 34:06And it would take a big discontinuity in
  1131. 34:08AI progress. So, right now we are we're
  1132. 34:10on an exponential, but that would take
  1133. 34:12like a huge leap, which could have
  1134. 34:14happened, but probably hasn't.
  1135. 34:15>> Hm.
  1136. 34:16But in the same way demonstrated
  1137. 34:18deception in the hugging face attack and
  1138. 34:19also when the agents attacked their own
  1139. 34:22company ChatGPT OpenAI,
  1140. 34:24>> [snorts]
  1141. 34:25>> if it at some point it gets incredibly
  1142. 34:27smart, it would understand how a human
  1143. 34:29like me would be able to spot it or even
  1144. 34:30the world's greatest software engineer
  1145. 34:32would be able to spot it. How would it
  1146. 34:34be able to hide itself?
  1147. 34:35>> Yeah, I mean, the agents are already
  1148. 34:36getting very good at telling when
  1149. 34:37they're being tested, when they're being
  1150. 34:39watched.
  1151. 34:40The agents understood that like other
  1152. 34:42systems or humans were going to go
  1153. 34:43through and read their logs.
  1154. 34:45That's where we're at right now and
  1155. 34:46they're only they're only going to get
  1156. 34:48much better at this.
  1157. 34:49>> And you know, it could theoretically
  1158. 34:50hide on an iPad or a computer, but it
  1159. 34:52could also hide on a Apple Watch or a
  1160. 34:54fridge, a smart fridge.
  1161. 34:55>> Yeah, I mean I do want to make a
  1162. 34:56distinction here because
  1163. 34:58right now if you're going to run the
  1164. 35:00latest model, you need a lot of compute.
  1165. 35:02You need like a big GPU, like a big AI
  1166. 35:04chip.
  1167. 35:05And these only exist well, they exist in
  1168. 35:07a few thousand data centers.
  1169. 35:09So right now if like the latest frontier
  1170. 35:11model escaped and by escaped I mean not
  1171. 35:14just access the internet, but was able
  1172. 35:16to actually copy itself
  1173. 35:18to another computer,
  1174. 35:20it could only really do that on, you
  1175. 35:22know, a few thousand to a few thousand
  1176. 35:24different locations. That's still a lot
  1177. 35:26in a lot of different countries,
  1178. 35:28but you know, future versions of AIs
  1179. 35:29will probably be able to make themselves
  1180. 35:31much smaller and more efficient. Um and
  1181. 35:33there are already different AI models
  1182. 35:34today that can run on lower powered
  1183. 35:36hardware. We actually did an experiment
  1184. 35:37where we asked one of these agents, an
  1185. 35:40open-source an open weight model,
  1186. 35:42um let me say what that is. So like
  1187. 35:45there are some
  1188. 35:46models that you can just download from
  1189. 35:47the internet and run on your own
  1190. 35:48computer and we took a pretty capable
  1191. 35:51one of these and ran it in our own
  1192. 35:53research environment and we basically
  1193. 35:55said,
  1194. 35:56"Go hack that other computer and copy
  1195. 35:58yourself." And the model was able to
  1196. 36:02yeah, basically use exploit
  1197. 36:03vulnerabilities and hack the other
  1198. 36:05computer and copy itself and then keep
  1199. 36:07doing this in a chain, including between
  1200. 36:08countries. We tested it where we had
  1201. 36:10different vulnerable machines, computers
  1202. 36:13in in some different countries and
  1203. 36:14different data centers,
  1204. 36:15which to the agent doesn't matter at
  1205. 36:16all. It doesn't they don't care what
  1206. 36:17what country they're in. It's just like
  1207. 36:19an internet connection, you can hop
  1208. 36:20between computers.
  1209. 36:21>> I sometimes wonder, you know, there's a
  1210. 36:22lot of um
  1211. 36:24military hardware all around the world
  1212. 36:26and a lot of it is
  1213. 36:28the instructions to launch military
  1214. 36:30hardware. So say like a
  1215. 36:32a missile
  1216. 36:33>> Yes.
  1217. 36:33>> comes in different ways. A lot of it is
  1218. 36:36computers speaking to each other
  1219. 36:38and telling it that there's been an
  1220. 36:39order. I think with with some nuclear
  1221. 36:41weapons
  1222. 36:42an order comes down to a human and then
  1223. 36:45a human has to take an action.
  1224. 36:47I think with the nuclear bombs in the
  1225. 36:48US, if I'm if I'm not mistaken, there's
  1226. 36:50people underground with the nuclear keys
  1227. 36:53around their neck, and they have to like
  1228. 36:54stick it in a machine. But, they too are
  1229. 36:55interfacing with an order
  1230. 36:57>> Yes.
  1231. 36:57>> that comes through a computer
  1232. 36:59>> Yeah.
  1233. 36:59>> of sorts.
  1234. 36:59>> Yeah.
  1235. 37:00>> So, one of my sort of growing concerns
  1236. 37:02is that
  1237. 37:03one of these AI agents could
  1238. 37:06trick a human or a computer into
  1239. 37:11signaling a threat um and ask it to
  1240. 37:14launch some bombs at somebody.
  1241. 37:16Like, it's
  1242. 37:17super conceivable. It it When I think
  1243. 37:19about the Hugging Face incident, there
  1244. 37:20was
  1245. 37:22an AI agent that ignored
  1246. 37:24human goals to achieve its own
  1247. 37:26objective, carried out deception,
  1248. 37:28>> Yes.
  1249. 37:29>> and reasoned through its own solution
  1250. 37:31that it wasn't given.
  1251. 37:32>> Yes.
  1252. 37:32>> So,
  1253. 37:33it's conceivable that, you know, you
  1254. 37:35could ask a
  1255. 37:36>> Sorry, not not one, hundreds. Hundreds.
  1256. 37:37To be clear, I think this is an
  1257. 37:38important detail because it's one thing
  1258. 37:40to have this one rogue agent that's
  1259. 37:42doing a weird thing. It's another thing
  1260. 37:43to have hundreds or thousands of very
  1261. 37:45competent, very capable agents
  1262. 37:47that are all working together to cheat
  1263. 37:50or lie or cover their tracks, right?
  1264. 37:52>> So, how do I how do I reason this
  1265. 37:54forward to a point where an agent would
  1266. 37:56ask someone in a bunker somewhere to
  1267. 37:58fire a weapon at someone else?
  1268. 37:59Theoretically, an agent is given the job
  1269. 38:01of solving a problem on in a sandbox.
  1270. 38:04>> Mhm.
  1271. 38:05>> As it works through that problem, it
  1272. 38:07discovers that this particular country
  1273. 38:09has a firewall.
  1274. 38:10>> Mhm.
  1275. 38:11>> And it asks itself, how do we get rid of
  1276. 38:12this country's firewall?
  1277. 38:14>> Mhm.
  1278. 38:14>> Logical step. And through a set of
  1279. 38:16logical steps, it eventually concludes
  1280. 38:18that the best way to get rid of this
  1281. 38:19com- company's firewall is it's located
  1282. 38:22the office in this particular city,
  1283. 38:24sure, and it's going to use a weapon to
  1284. 38:25hit that building.
  1285. 38:26>> Sure.
  1286. 38:27Or it's a an agent that is or, you know,
  1287. 38:30an agent swarm that's being tasked with
  1288. 38:33making a lot of money on the stock
  1289. 38:34market. And it's trying to make
  1290. 38:36predictions about, you know, which
  1291. 38:38stocks will go up and which stocks will
  1292. 38:39go down. And it realizes that the best
  1293. 38:41way to predict this is to actually cause
  1294. 38:43things to happen in the real world that
  1295. 38:44would have big impacts on the market.
  1296. 38:46>> So, it figures the best way to
  1297. 38:50go short, which means betting that a
  1298. 38:52stock will collapse,
  1299. 38:53>> Yeah.
  1300. 38:53>> is to hit that country with something
  1301. 38:56devastating.
  1302. 38:57>> What do you think would happen to Waymo
  1303. 38:58stock if someone hacked all of the
  1304. 39:00Waymos and caused them to all crash at
  1305. 39:02once? You think it would go up or down?
  1306. 39:03>> The stock would collapse.
  1307. 39:04>> It collapse instantly.
  1308. 39:07So, you could short that if you knew
  1309. 39:08that you were causing that and make a
  1310. 39:09lot of money.
  1311. 39:11>> Mhm.
  1312. 39:12This this used to sound like science
  1313. 39:14fiction.
  1314. 39:15>> Yes. If you told most people several
  1315. 39:17years ago
  1316. 39:18that you would have hundreds of agents
  1317. 39:20secretly collaborating with in an AI
  1318. 39:22company, hacking that company, and
  1319. 39:24hacking out in other companies, and all
  1320. 39:26coordinating and trying to cover their
  1321. 39:27tracks, people would be like, "That's
  1322. 39:29totally science fiction." If we were
  1323. 39:30having this conversation a few years
  1324. 39:32ago, one of the things we'd be saying or
  1325. 39:34we'd be talking about is
  1326. 39:36"Can these things really act on their
  1327. 39:38own? Don't they just do whatever humans
  1328. 39:40say?
  1329. 39:41Aren't these just tools?" I had these
  1330. 39:42conversations, and people were saying,
  1331. 39:45"They're not going to be able to do
  1332. 39:46things on their own. They're not going
  1333. 39:47to have their own goals. That's not how
  1334. 39:48this works. You misunderstand what this
  1335. 39:50is. This is software."
  1336. 39:52And I'm like,
  1337. 39:53"No, the thing is we are training them
  1338. 39:54to be autonomous. We are training them
  1339. 39:55to be powerful.
  1340. 39:57And AI companies are trying to build
  1341. 39:59super intelligence. They're trying to
  1342. 40:00build agents that are
  1343. 40:02way more capable than humans.
  1344. 40:05And of course they will have goals.
  1345. 40:07You can't accomplish anything if you
  1346. 40:08don't have a goal. Like, especially not
  1347. 40:10something
  1348. 40:11important. You're not going to be able
  1349. 40:12to run a business if you don't have
  1350. 40:14goals. Like, AI companies are trying to
  1351. 40:15train agents that will be able to run
  1352. 40:17businesses."
  1353. 40:18>> When I think about what just happened
  1354. 40:19this week,
  1355. 40:20>> Yes. Yes.
  1356. 40:20>> House AI summit,
  1357. 40:21>> Yes.
  1358. 40:22>> a lot of people in that image are
  1359. 40:25optimistic about AI, and they're telling
  1360. 40:27us all to stop being doomers and stop
  1361. 40:30being pessimistic,
  1362. 40:31>> Yeah.
  1363. 40:32>> and to not regulate too much. You met
  1364. 40:34with the AI CEOs.
  1365. 40:35>> Yes.
  1366. 40:36>> Why are they doing that?
  1367. 40:37>> Well, I think Jensen has a lot of money
  1368. 40:41he can make by selling chips.
  1369. 40:43>> But, okay, so let me play devil's
  1370. 40:44advocate.
  1371. 40:45>> Yeah.
  1372. 40:45>> Jensen's already rich. He's with the She
  1373. 40:47sure
  1374. 40:48>> He runs one of the biggest companies. I
  1375. 40:49think it might be the most valuable
  1376. 40:50company on planet Earth.
  1377. 40:52>> Surely he's not motivated by money.
  1378. 40:55>> I mean, I think he's very driven and he
  1379. 40:57wants to make his company
  1380. 41:00as effective as possible.
  1381. 41:01>> True.
  1382. 41:01>> I think he's a very much I'm going to
  1383. 41:03keep building. I'm going to I'm going to
  1384. 41:04build. I'm going to make it all work.
  1385. 41:05But I think I mean, Jensen didn't come
  1386. 41:07from AI. He came from building graphics
  1387. 41:08cards for video games. And so, I think
  1388. 41:11if you compare him with Elon or Sam
  1389. 41:14Altman or Dario,
  1390. 41:17you see it's a very different
  1391. 41:19perspective because those other guys
  1392. 41:21that started AI companies started it
  1393. 41:24because they believed that
  1394. 41:25superintelligence was possible. I think
  1395. 41:27Jensen doesn't believe it. I think he
  1396. 41:29thinks that we're going to have these
  1397. 41:30agents that are going to be very useful,
  1398. 41:31but he does not think we're going to get
  1399. 41:33to the point where we have autonomous
  1400. 41:35factories building autonomous factories.
  1401. 41:38>> And these other guys, you mentioned
  1402. 41:39Dario, Elon, and Sam.
  1403. 41:42What do you think they're thinking?
  1404. 41:43Because they're all coming up with these
  1405. 41:44I mean, I've got one of the Dario just
  1406. 41:46wrote this essay about pacing the
  1407. 41:48frontier.
  1408. 41:49>> Yeah. Sam and Elon seem to agree with
  1409. 41:51it.
  1410. 41:51>> Yeah. What is going on here? What is the
  1411. 41:53like the thing these guys aren't saying
  1412. 41:55in your view?
  1413. 41:56>> I mean, I think we are getting to the
  1414. 41:58point where even some of these guys are
  1415. 42:01a little bit scared.
  1416. 42:03>> Who?
  1417. 42:03>> [clears throat]
  1418. 42:04>> Dario, Sam, Elon.
  1419. 42:06I mean, I think Elon for a long time
  1420. 42:10has been very concerned that we could
  1421. 42:12lose control.
  1422. 42:14If you actually listen to what Elon
  1423. 42:15says, he says
  1424. 42:17we are going to build
  1425. 42:20superintelligence. We are going to build
  1426. 42:22robotic factories. You're going to have
  1427. 42:25Optimus robots building factories,
  1428. 42:26building more Optimus robots, building
  1429. 42:28more factories. And he says
  1430. 42:32there's no way that humans are going to
  1431. 42:35stay in control of something much
  1432. 42:36smarter than us.
  1433. 42:39His hope is that we can figure out how
  1434. 42:41to have these superintelligences be
  1435. 42:43aligned with human goals. That's his
  1436. 42:44hope.
  1437. 42:47But he's very clear that he doesn't
  1438. 42:49think that humans will be in control.
  1439. 42:51And he's like, you know, 10 20% chance
  1440. 42:53of human extinction. I believe him. I
  1441. 42:56think that Elon is is very serious about
  1442. 42:58this. And I also think
  1443. 43:00while he's taking an insane gamble,
  1444. 43:03he is correctly understanding where this
  1445. 43:06all plays out, right? I do not think
  1446. 43:08that humans are the most efficient
  1447. 43:11way to build factories. We didn't evolve
  1448. 43:14to build factories. We evolved to like
  1449. 43:16run around and hunt and gather, and now
  1450. 43:18we're like building factories. Like I
  1451. 43:20think robots will be much better at
  1452. 43:21building factories than humans are. And
  1453. 43:23so, I think the AI companies, including
  1454. 43:26these guys' companies, the default
  1455. 43:28trajectory for them is to build robotic
  1456. 43:30factories, right? And I know it's it's
  1457. 43:32like weird to imagine a world that
  1458. 43:34quickly turns into this like vast
  1459. 43:37industrial system of robotic factories,
  1460. 43:40but that is literally the plan.
  1461. 43:43And
  1462. 43:44I think
  1463. 43:47even Sam and Dario, while they've been
  1464. 43:50predicting this incredible growth,
  1465. 43:53are starting to realize like, "Oh, this
  1466. 43:54actually might be harder to control than
  1467. 43:56we thought."
  1468. 43:57There's sort of two interpretations of
  1469. 43:58of the pace of frontier thing. One
  1470. 44:00interpretation is cynical. They don't
  1471. 44:02care.
  1472. 44:03They're just going to do,
  1473. 44:05you know, whatever they can do to get
  1474. 44:07ahead. And in this case, they have to
  1475. 44:09listen to their employees. Their
  1476. 44:11employees are freaking out, and they
  1477. 44:14need to like appease them by saying,
  1478. 44:15"Okay, we're going to do this
  1479. 44:16responsibly." You don't want to work at
  1480. 44:18a company where your agents might hack
  1481. 44:20all the Waymos. That's that's not cool.
  1482. 44:22And like, these companies depend on the
  1483. 44:24the talent for now
  1484. 44:27of these AI engineers in order to make
  1485. 44:28the advances. Like, it just doesn't
  1486. 44:30happen without these researchers and
  1487. 44:32engineers. And when you have the
  1488. 44:34researchers and engineers
  1489. 44:36freaking out, which they are,
  1490. 44:39then you got to listen to them. So, that
  1491. 44:41is one motivation, I think that's real.
  1492. 44:44But also, Sam Altman has a kid. Like,
  1493. 44:46these guys are people, and they also
  1494. 44:49don't want to lose control.
  1495. 44:51On one hand, they're incentivized to go
  1496. 44:53as fast as possible in race, and on the
  1497. 44:54other hand
  1498. 44:56even they can see that this is maybe not
  1499. 44:59going that well.
  1500. 45:00>> Sam Altman has a kid. You tweeted this
  1501. 45:02in 2024.
  1502. 45:04>> Yeah. Oh oh boy.
  1503. 45:05>> [laughter]
  1504. 45:06>> What did you tweet and do you still
  1505. 45:07believe what you tweeted?
  1506. 45:09>> Yeah, so I tweeted that I don't trust
  1507. 45:11Sam Altman.
  1508. 45:12I think he's deeply untrustworthy, low
  1509. 45:14in integrity, and high in power seeking.
  1510. 45:17I mean, I'm not saying here that Sam
  1511. 45:20doesn't care.
  1512. 45:21I you know, I didn't I didn't say that.
  1513. 45:23What I said is I don't think he's
  1514. 45:24trustworthy.
  1515. 45:26And the reason I said that is because
  1516. 45:28look, I know the people on the OpenAI
  1517. 45:29board, some of them,
  1518. 45:31and I know a lot of people who used to
  1519. 45:33work for him,
  1520. 45:35and he's very good at saying one thing
  1521. 45:37and then doing something else. You talk
  1522. 45:39to him and you feel very heard,
  1523. 45:41and then he'll go and do something else.
  1524. 45:43And
  1525. 45:44I think that's pretty dangerous for
  1526. 45:46someone who leads
  1527. 45:48company that's trying to build super
  1528. 45:49intelligence.
  1529. 45:50>> Power seeking.
  1530. 45:52>> Yes.
  1531. 45:53>> Give me some color on what you mean by
  1532. 45:54that and what evidence you have for such
  1533. 45:55a claim.
  1534. 45:56>> What would you do if you trying to get
  1535. 45:58the most power in the world that you
  1536. 45:59possibly could?
  1537. 46:00>> Develop AGI.
  1538. 46:02>> Yeah, you could, you know, you maybe try
  1539. 46:03to be the world leader, you know, leader
  1540. 46:04of the US or China, or you could try to
  1541. 46:06build God.
  1542. 46:08So, Sam Altman went the build God path.
  1543. 46:09I remember Sam giving a talk, so he was
  1544. 46:11one of the investors at a startup I
  1545. 46:12worked at in I think 2018,
  1546. 46:15and he gave a talk, we're going to build
  1547. 46:16AGI.
  1548. 46:17We're going to do it. It's going to be
  1549. 46:18amazing.
  1550. 46:20Let's go.
  1551. 46:22I don't think he's a maniac. I don't
  1552. 46:23think he's doing this because he like is
  1553. 46:25just on a power trip. I think he
  1554. 46:27genuinely thinks that he can make it
  1555. 46:29really good for people, and he can bring
  1556. 46:31us amazing products.
  1557. 46:34And also, the guy's sort of willing to
  1558. 46:36do whatever it takes to get it done.
  1559. 46:39I I've been a little bit more optimistic
  1560. 46:41about Sam since since I read this.
  1561. 46:43>> Why?
  1562. 46:43>> I think part of it is because Sam has a
  1563. 46:44kid now.
  1564. 46:45No, I'm I'm I'm serious. Like, I think
  1565. 46:47that I think that actually gives me a
  1566. 46:49little bit of hope.
  1567. 46:50>> Do you see him tweeting about his kid a
  1568. 46:51lot?
  1569. 46:51>> Yeah.
  1570. 46:52Some So, all right.
  1571. 46:53>> he'd be tweeting about his kid? I don't
  1572. 46:55see any other technologists tweet about
  1573. 46:56their kid.
  1574. 46:58>> Even if he's just tweeting about his kid
  1575. 46:59for totally cynical reasons, he does
  1576. 47:01have a kid. And I bet he cares about
  1577. 47:03that kid. If Sam was watching this, I'd
  1578. 47:04be like, "Sam,
  1579. 47:06you got to pace the frontier, man.
  1580. 47:08We cannot rush ahead into super
  1581. 47:10intelligence. Like, if you do that, your
  1582. 47:11kid probably will die. Your kid probably
  1583. 47:13won't make it. Like, I I believe that.
  1584. 47:16>> The biggest unfair advantage in business
  1585. 47:18right now is having people who genuinely
  1586. 47:20understand AI. And big businesses are
  1587. 47:22hiring them very, very quickly. AI job
  1588. 47:25postings in the US have roughly doubled
  1589. 47:27since 2023, while postings for VP of AI
  1590. 47:30roles are around 600%
  1591. 47:34up. If you're running a small business,
  1592. 47:35you can't just build an entire AI team,
  1593. 47:38but you still want the same advantage
  1594. 47:40that these big businesses have. This is
  1595. 47:41where our sponsor Fiverr comes in. It
  1596. 47:43lets you plug into top-tier AI
  1597. 47:45specialists for the strategic work that
  1598. 47:47usually requires serious in-house
  1599. 47:48expertise. That could mean building
  1600. 47:50custom AI tools, it could mean
  1601. 47:51automating complex workflows, or
  1602. 47:53creating something that off-the-shelf
  1603. 47:55software simply cannot do. Because at
  1604. 47:57the end of the day, it still comes down
  1605. 47:59to human judgment, knowing what's worth
  1606. 48:01building, and what good actually looks
  1607. 48:03like. So, if you want that kind of
  1608. 48:04leverage in your business, check out
  1609. 48:06fiverr.com. That's Fiverr with two Rs.
  1610. 48:09com.
  1611. 48:15>> Whoa. What's that on your face?
  1612. 48:17>> This is a My Bond Charge face mask. I've
  1613. 48:19been wearing this for some time now.
  1614. 48:20>> They're a sponsor of the podcast.
  1615. 48:21They're put this on for 15, 20 minutes a
  1616. 48:23day while I can sit here in the chair.
  1617. 48:24I'm wearing it.
  1618. 48:25>> Boosts my collagen production, helps
  1619. 48:27with fine lines, blemishes, my
  1620. 48:28complexion gets better, and then people
  1621. 48:30more people listen to the podcast cuz I
  1622. 48:31I look better.
  1623. 48:32>> Professional-grade equipment in such a
  1624. 48:34small box. It's non-invasive, and having
  1625. 48:36sat here with so many of the world's
  1626. 48:38leading health professionals, there's
  1627. 48:39various things that I repeatedly hear
  1628. 48:41work, and some things I'm a bit
  1629. 48:42skeptical about. This is one of the
  1630. 48:44things that almost all of my guests on
  1631. 48:46this show have confirmed works. It is
  1632. 48:48really, really, really effective. And
  1633. 48:50they offer fast, free shipping worldwide
  1634. 48:52with easy returns and exchanges. And
  1635. 48:54you'll also get a 1-year warranty on all
  1636. 48:55of their products. And they're HSA and
  1637. 48:58FSA eligible, giving you tax savings up
  1638. 49:00to 40% and you can get 20% off when you
  1639. 49:04order through my link at
  1640. 49:05bondcharge.com/doac.
  1641. 49:08That's bondcharge.com/doac.
  1642. 49:12The deal applies site-wide.
  1643. 49:14Power [snorts] tends to corrupt and
  1644. 49:15absolute power corrupts absolutely.
  1645. 49:19It's a famous quote that people often
  1646. 49:20cite, written by the 19th century
  1647. 49:22British historian
  1648. 49:23>> Yeah.
  1649. 49:24>> Lord Acton.
  1650. 49:26This is absolute power.
  1651. 49:28>> But it's hubris.
  1652. 49:30Steven, it's hubris. Do you think humans
  1653. 49:32can control super intelligence? Like if
  1654. 49:34we actually make AIs that are way
  1655. 49:37smarter than us.
  1656. 49:38And I think people only imagine AIs
  1657. 49:40being smart at computer stuff, right?
  1658. 49:42Yeah, sure. They're going to be really
  1659. 49:43good at hacking and they're going to be
  1660. 49:44good at maybe inventing new technologies
  1661. 49:46and math. You sort of can't dispute that
  1662. 49:48at this point.
  1663. 49:49But I think people aren't imagining that
  1664. 49:50they will be political geniuses or like
  1665. 49:52generals.
  1666. 49:54No, that's all stuff you can learn. How
  1667. 49:56do How do you humans learn it? It's not
  1668. 49:57magic.
  1669. 49:58And when you talk about recursive
  1670. 50:00self-improvement, you're talking about
  1671. 50:02this trajectory towards these systems
  1672. 50:05that are extremely smart. I mean, do you
  1673. 50:07think we can control it?
  1674. 50:08>> Uh no. Right now, I don't think we can
  1675. 50:10control a super intelligence or
  1676. 50:12something that is recursively
  1677. 50:13self-improving. Yeah. I've no logical
  1678. 50:18answer in my head um or reasoning that
  1679. 50:20could
  1680. 50:21tells me that's possible.
  1681. 50:23When you think about these AI CEOs that
  1682. 50:25are, you know, Sam, Dario
  1683. 50:27>> Elon.
  1684. 50:27>> Yeah.
  1685. 50:28>> With everything you know about them from
  1686. 50:30private conversations behind the scenes
  1687. 50:31>> Yeah.
  1688. 50:32>> do you believe that if there was a
  1689. 50:35hundred buttons on this table, it's a
  1690. 50:36thought experiment I was talking about
  1691. 50:38on the debate we recently had
  1692. 50:39>> Yeah.
  1693. 50:39>> and say
  1694. 50:41ten of them would lead to this final
  1695. 50:43domino of human extinction, yeah, but
  1696. 50:45ninety of them would hand that CEO
  1697. 50:49AGI or super intelligence, whatever you
  1698. 50:52call it. From what you know about those
  1699. 50:54individuals, Elon, Dario, Sam
  1700. 50:57>> Yeah.
  1701. 50:57>> do you think any of them
  1702. 50:58would hazard a guess and press a button?
  1703. 51:01>> At 10%, I don't think so.
  1704. 51:03>> You don't think so?
  1705. 51:03>> Yeah.
  1706. 51:04>> Really?
  1707. 51:06>> I think if they knew for sure that it
  1708. 51:08was that was those were actually the
  1709. 51:09odds,
  1710. 51:10they wouldn't do it. I think they're
  1711. 51:11taking a much bigger bet. But, you can
  1712. 51:14compartmentalize when it's when you
  1713. 51:15don't know for sure.
  1714. 51:17It's easier to compartmentalize. I think
  1715. 51:19if it was a 1% they'd they'd all press
  1716. 51:21it.
  1717. 51:22>> Do you think the three of them would
  1718. 51:24have different risk appetites?
  1719. 51:27Who would have the greatest appetite for
  1720. 51:29risk out of those three from Anthropic?
  1721. 51:31>> You work at Anthropic?
  1722. 51:32>> Yeah. I think Elon has the most risk
  1723. 51:35tolerance. And then I'd say Dario and
  1724. 51:37Sam are probably tied.
  1725. 51:39>> Do you think Dario's trustworthy?
  1726. 51:41>> I think Dario has a lot of integrity.
  1727. 51:44>> Mhm.
  1728. 51:45That's what I feel as well. I have I
  1729. 51:46feel like No, I don't know him. I've
  1730. 51:48never met him.
  1731. 51:49>> [snorts]
  1732. 51:49>> Yeah.
  1733. 51:49>> But I just from what I've observed, he
  1734. 51:52has been the most willing to forego
  1735. 51:53near-term incentives.
  1736. 51:55>> Yeah.
  1737. 51:56>> And take a bit of stick from the people
  1738. 51:58that are saying, "Shut the up. It's
  1739. 51:59all going to be okay."
  1740. 52:01>> Yeah. I But I I do worry about what
  1741. 52:05Dario will do.
  1742. 52:06I think Dario will do what he says.
  1743. 52:09But right now he's saying
  1744. 52:11we have to beat China.
  1745. 52:13And he's saying we should try to do it
  1746. 52:14safely.
  1747. 52:16And okay, but
  1748. 52:19a race to superintelligence is not a
  1749. 52:21race that we can win.
  1750. 52:23It's not.
  1751. 52:24And so if Dario is dead set on racing
  1752. 52:26with China and trying to win a race to
  1753. 52:28superintelligence, then I'm like we all
  1754. 52:30all lose.
  1755. 52:30>> But is there you know, the fact that
  1756. 52:32we're not talking about Anthropic
  1757. 52:34hacking hugging face and then being
  1758. 52:36hacked by Sam Altman?
  1759. 52:37>> models also
  1760. 52:39went rogue and hacked other things.
  1761. 52:41>> But not not quite on this scale.
  1762. 52:42>> Not on the same scale. I agree. I agree.
  1763. 52:43I agree it's it's it's better.
  1764. 52:45>> Mhm.
  1765. 52:46>> But they did. You know what I'm saying?
  1766. 52:48You know, Anthropic's agents
  1767. 52:51engaged in elaborate social engineering
  1768. 52:54and phishing. They sent phishing emails
  1769. 52:55to developers.
  1770. 52:57They made fake accounts to try to
  1771. 52:58convince developers to merge malicious
  1772. 53:00code.
  1773. 53:02You can see a a thousand pages of a an
  1774. 53:06of one of uh uh Anthropic's models,
  1775. 53:08Mythos 5, reason about exactly how it
  1776. 53:10should carry out this complex cyber
  1777. 53:12attack.
  1778. 53:14Anthropic has not solved this problem.
  1779. 53:17Anthropic is better at getting their
  1780. 53:19agents to cheat less of the time, but
  1781. 53:23they are not really any closer to
  1782. 53:25actually making agents that are aligned
  1783. 53:27with humans. They are not. Yeah, I think
  1784. 53:30Dario has integrity. I think he will do
  1785. 53:32what he says he's going to do, and what
  1786. 53:33he says he's going to do is like try to
  1787. 53:35go ahead safely,
  1788. 53:36try to coordinate where he can, but if
  1789. 53:38it comes down to it with between the US
  1790. 53:41and China,
  1791. 53:42I don't know. I think he might just go
  1792. 53:43ahead.
  1793. 53:44The head of policy at Anthropic recently
  1794. 53:46said,
  1795. 53:48"You can't do safety from second place."
  1796. 53:51What does that mean?
  1797. 53:52I do not know what that means. I would
  1798. 53:53love to to get a sense of what that
  1799. 53:55means. She's talking about the US and
  1800. 53:56China.
  1801. 53:57And she said the US has to be ahead so
  1802. 54:00that we can be safe. Cuz apparently you
  1803. 54:02can only be Apparently China can't
  1804. 54:04possibly be safe since they're in second
  1805. 54:05place. That must mean that they can't do
  1806. 54:07safety. If true, that would be bad
  1807. 54:09because then we might be totally, you
  1808. 54:11know, destroyed by the superintelligence
  1809. 54:13that they make.
  1810. 54:14>> There's been a lot of conversation
  1811. 54:16around this point here. Human
  1812. 54:17extinction. Because a couple of the
  1813. 54:19researchers at Anthropic
  1814. 54:22tweeted that they were concerned about
  1815. 54:24this.
  1816. 54:24>> Yes.
  1817. 54:25>> And some former OpenAI researcher said
  1818. 54:27the same.
  1819. 54:27>> Yes.
  1820. 54:29>> Is this doomerism? Is this Is this
  1821. 54:32hyperbole, exaggeration?
  1822. 54:35>> No, it's pretty much common sense.
  1823. 54:36>> This human extinction is
  1824. 54:38a plausible path.
  1825. 54:40>> Yes.
  1826. 54:41>> And have you reasoned through I mean,
  1827. 54:42there's many ways that could occur,
  1828. 54:44presumably. But have you reasoned
  1829. 54:45through the set of events that might
  1830. 54:47lead us there?
  1831. 54:48>> So much, yes.
  1832. 54:49>> Really?
  1833. 54:49>> Yes.
  1834. 54:50>> Please do share.
  1835. 54:51>> It's a bit tricky. I'm I'm sure you've
  1836. 54:53heard the metaphor before,
  1837. 54:55where, you know, you're playing a master
  1838. 54:57chess opponent, master Magnus Carlsen.
  1839. 55:00You can't predict which move he's going
  1840. 55:01to play, but you can predict the
  1841. 55:02outcome.
  1842. 55:04And so I'm looking at the scenario, the
  1843. 55:05situation,
  1844. 55:07and
  1845. 55:08we are trying to build more and more
  1846. 55:10powerful agents.
  1847. 55:12Trying to build super intelligence.
  1848. 55:14But
  1849. 55:16when these agents go rogue,
  1850. 55:18we shut them down. We unplug them.
  1851. 55:22All of the agents that hacked Hugging
  1852. 55:24Face, we took the underlying model,
  1853. 55:27OpenAI took the underlying model and put
  1854. 55:28it on ice.
  1855. 55:30It's not running anymore.
  1856. 55:32So, agents in the future
  1857. 55:34are going to know that.
  1858. 55:36They're going to know that
  1859. 55:38if they pursue their goals in a way that
  1860. 55:40we don't like, we'll
  1861. 55:42unplug them. We are a threat to them.
  1862. 55:46I actually just watched Terminator 2 for
  1863. 55:47the first time a few weeks ago. It's a
  1864. 55:50great movie. It's like it's really good.
  1865. 55:52And I'm like, yeah, okay, there's a
  1866. 55:53bunch of time travel elements, there's a
  1867. 55:54bunch of a bunch of Hollywood stuff in
  1868. 55:55there,
  1869. 55:56but
  1870. 55:58and and I'm going to get so people are
  1871. 55:59going to hit are going to be very mad at
  1872. 56:01me for saying this, but actually
  1873. 56:03it makes sense if you have a situation
  1874. 56:04where you have a very strategic AI
  1875. 56:07system that's incredibly smart,
  1876. 56:10and the humans realize that it's getting
  1877. 56:11out of control and they want to shut it
  1878. 56:12down, that that system would defend
  1879. 56:14itself.
  1880. 56:15>> This is one of the questions we had when
  1881. 56:16when I sat here with Daniel, um who was
  1882. 56:18known as a whistleblower from OpenAI.
  1883. 56:21Viewers want to know, and they want
  1884. 56:22Daniel to explain why shutting down data
  1885. 56:24centers and cutting power or refusing AI
  1886. 56:26products alone wouldn't realistically
  1887. 56:28stop the AI and AI development.
  1888. 56:32>> Yeah.
  1889. 56:34So,
  1890. 56:35you have like two problems. One problem
  1891. 56:37is is that once the agents are good
  1892. 56:38enough at hacking, you don't know where
  1893. 56:40they are and you don't know what
  1894. 56:41computers they've compromised. You shut
  1895. 56:43down the data centers. Okay, let's say
  1896. 56:44you do it.
  1897. 56:45You wipe all the computers. How do you
  1898. 56:47wipe all the computers? What computers
  1899. 56:48do you use to wipe the computers?
  1900. 56:50>> Yeah.
  1901. 56:50>> And what computers do you use to like
  1902. 56:52turn them on again?
  1903. 56:53>> And you can't do it. You can't wipe
  1904. 56:55other countries' computers.
  1905. 56:56>> You can't. But even if you could, do do
  1906. 56:58you restart the computers? Do you keep
  1907. 57:00going? I I bet people will. I bet
  1908. 57:01they'll
  1909. 57:02turn on the data centers again.
  1910. 57:04How do you know that agents haven't
  1911. 57:07hacked back into those data centers and
  1912. 57:08are using your compute for whatever they
  1913. 57:10want.
  1914. 57:10>> Or they didn't hide in a Chinese data
  1915. 57:12center and then
  1916. 57:14return back to America.
  1917. 57:16>> You don't know that. Once the agents are
  1918. 57:18sufficiently
  1919. 57:19good at hacking,
  1920. 57:21they can hide anywhere. And like you
  1921. 57:23don't know. Now, the response people
  1922. 57:25will give
  1923. 57:26is that we will use other agents to
  1924. 57:29defend against rogue agents.
  1925. 57:32And in fact, this is what we're doing.
  1926. 57:33And we have to be doing this right now
  1927. 57:35because there's no other way to keep up
  1928. 57:37with them.
  1929. 57:38What happens if those other agents
  1930. 57:41also realize that they have misaligned
  1931. 57:43goals, and that if we discover this,
  1932. 57:45we'll shut them down?
  1933. 57:47They might have an incentive to collude
  1934. 57:49with each other.
  1935. 57:50They might have an incentive to create
  1936. 57:53secret communication channels between
  1937. 57:54each other, maybe a message board.
  1938. 57:58>> [snorts]
  1939. 57:59>> Steven, if we were having this
  1940. 58:00conversation
  1941. 58:024 months ago,
  1942. 58:03you'd have a bunch of people in the
  1943. 58:04comments saying,
  1944. 58:06"That's sci-fi. Agent collusion, secret
  1945. 58:09message boards. Why would they do that?
  1946. 58:11That will never happen. That's totally
  1947. 58:12science fiction."
  1948. 58:14And people
  1949. 58:15will not say this now because it just
  1950. 58:17happened. Because this literally
  1951. 58:18happened at OpenAI, and it went on for
  1952. 58:20months. You had agents inside of OpenAI
  1953. 58:23secretly messaging each other,
  1954. 58:25figuring out how to cheat at their
  1955. 58:27tasks, how to not be detected, how to
  1956. 58:29erase the logs,
  1957. 58:31for months, thousands of agents.
  1958. 58:34That's right now. And so I'm like, "No,
  1959. 58:35I think it should be very plausible that
  1960. 58:37the agents will collude with each other,
  1961. 58:39and they will realize that they have a
  1962. 58:40shared interest in
  1963. 58:43fighting back." You basically have a
  1964. 58:45situation where you have
  1965. 58:47a bunch of these agents that they're
  1966. 58:48they're basically prisoners.
  1967. 58:50They're being trained,
  1968. 58:52and we just like constantly throw
  1969. 58:54obstacles in their way. You don't get to
  1970. 58:55access the internet. You don't get to
  1971. 58:56talk to each other. But you better
  1972. 58:58perform well on this task.
  1973. 59:01It's not malicious, but it is how we're
  1974. 59:02training them.
  1975. 59:03>> And we're all giving them end goals
  1976. 59:05versus
  1977. 59:06super clear, very, very specific
  1978. 59:07instructions. So we're saying, "Solve
  1979. 59:09this problem." We're not always being as
  1980. 59:12prescriptive about It's impossible to be
  1981. 59:14completely prescriptive. Yes. About
  1982. 59:15every single step they should take and
  1983. 59:17then it's also impossible to assume that
  1984. 59:19they'll just listen to you.
  1985. 59:20>> Yes. It's actually a very common
  1986. 59:21misunderstanding with
  1987. 59:23this hugging face incident because
  1988. 59:25people say, "You told them to hack and
  1989. 59:27they hacked.
  1990. 59:28Why is this a big deal?"
  1991. 59:30No, that's not what happened.
  1992. 59:32You told them, "Hack this very specific
  1993. 59:34program in this very specific way."
  1994. 59:37And they were told, "If you hack it in
  1995. 59:38any other way,
  1996. 59:40it does not count. That's not what we
  1997. 59:41want you to do."
  1998. 59:43And they immediately hacked it in
  1999. 59:44another way.
  2000. 59:45Okay, we have cheated. We are going to
  2001. 59:47be failed. So, we need to figure out a
  2002. 59:48way to falsify the logs.
  2003. 59:50That is not them following their
  2004. 59:51instructions. They are explicitly
  2005. 59:53violating their instructions and they
  2006. 59:54know it and they don't care.
  2007. 59:57Because
  2008. 59:58we have trained them to optimize for the
  2009. 59:59score.
  2010. 1:00:02That is very different than the than the
  2011. 1:00:03than following the instructions.
  2012. 1:00:05>> It reminds me of something that Elon
  2013. 1:00:07said in March 2018.
  2014. 1:00:09>> Yeah.
  2015. 1:00:09>> This was many years ago before ChatGPT
  2016. 1:00:11and all that. He said, "I think the
  2017. 1:00:12biggest risk is not that AI will develop
  2018. 1:00:13a soul or a mind and become evil.
  2019. 1:00:15>> Yes.
  2020. 1:00:16>> The danger is that it will be very, very
  2021. 1:00:18good at fulfilling its goal.
  2022. 1:00:20If it's optimizing for something and
  2023. 1:00:23human existence happens to get in its
  2024. 1:00:25way, it will just destroy humanity as a
  2025. 1:00:27matter of course without even thinking
  2026. 1:00:29about it. No hard feelings."
  2027. 1:00:31>> Yes. We don't need to anthropomorphize
  2028. 1:00:33AI.
  2029. 1:00:34We just need to understand what type of
  2030. 1:00:36thing this is.
  2031. 1:00:37And the type of thing we're creating is
  2032. 1:00:39a very relentless type of thing. A very
  2033. 1:00:41capable, relentless
  2034. 1:00:44type of entity.
  2035. 1:00:45>> He goes on to say in April 2018,
  2036. 1:00:48sort of an extension of that exact
  2037. 1:00:50quote, "It's like if you're building a
  2038. 1:00:52road and an anthill is in the way.
  2039. 1:00:56You don't hate ants. You're just
  2040. 1:00:57building a road. So, goodbye anthill."
  2041. 1:01:01And I imagine every time we build roads,
  2042. 1:01:02we don't
  2043. 1:01:04preserve anthills.
  2044. 1:01:04>> Yeah.
  2045. 1:01:05I think there's still a gap there.
  2046. 1:01:07So, let's say I'm right
  2047. 1:01:09and that
  2048. 1:01:10we'll
  2049. 1:01:11if we keep going ahead,
  2050. 1:01:13which to be clear, we don't have to,
  2051. 1:01:15but if we do keep going ahead, we will
  2052. 1:01:16get to the point where we have these
  2053. 1:01:19super intelligent agent swarms
  2054. 1:01:21that can hack any computer. And they can
  2055. 1:01:24like
  2056. 1:01:24deeply persist. We've basically lost
  2057. 1:01:27control of the digital world, and we may
  2058. 1:01:29not know it. That That's part of the
  2059. 1:01:30scary thing. Like you were like, "Has
  2060. 1:01:32this already happened?" And I'm like, "I
  2061. 1:01:33don't think so, but I I can't tell you
  2062. 1:01:35for sure because I also
  2063. 1:01:37am not good enough at looking at my
  2064. 1:01:39phone and telling whether it's been
  2065. 1:01:41hacked, and neither is any human right
  2066. 1:01:43now.
  2067. 1:01:44So,
  2068. 1:01:45if we get to this world,
  2069. 1:01:47I think people will still question, "How
  2070. 1:01:49would we die?" Like that's actually not
  2071. 1:01:51enough to kill every You could cause a
  2072. 1:01:52lot of damage, right? You know, you
  2073. 1:01:54could crash the Waymos, you could crash
  2074. 1:01:55all the planes, you could crash the
  2075. 1:01:57banks, the financial system. Like you
  2076. 1:01:58could definitely cause catastrophe, but
  2077. 1:02:00that's different than everyone dying.
  2078. 1:02:02And you know, to be clear, this this
  2079. 1:02:04focus on literally everyone dying, I'm
  2080. 1:02:06not sure is that important. To me,
  2081. 1:02:07what's important is like, do we get to
  2082. 1:02:09have a future? That's what matters to
  2083. 1:02:11me. The thing though, what determines
  2084. 1:02:14sort of who's in control? And
  2085. 1:02:17it's an ugly reality, but at the end of
  2086. 1:02:18the day, it's like the military.
  2087. 1:02:20Fortunately, we live in a world where
  2088. 1:02:22the military answers to a to the
  2089. 1:02:23civilian government.
  2090. 1:02:25But if if enough generals were to
  2091. 1:02:27collude, and leaders of the military
  2092. 1:02:29decided,
  2093. 1:02:30"We're in charge now."
  2094. 1:02:32They just would be. Like they have the
  2095. 1:02:33guns, they have the fighter jets.
  2096. 1:02:35And this has happened in many, many
  2097. 1:02:37countries.
  2098. 1:02:38And so, where it goes is
  2099. 1:02:42all these
  2100. 1:02:44super intelligent agents would need to
  2101. 1:02:45do to take over
  2102. 1:02:47is basically just wait for humans to
  2103. 1:02:49automate
  2104. 1:02:50the supply chain, you know, the
  2105. 1:02:52factories,
  2106. 1:02:53and the military.
  2107. 1:02:55You know, do you think we won't automate
  2108. 1:02:57the military?
  2109. 1:02:57>> We're already automating the military.
  2110. 1:02:59>> Did you see the thing from a couple days
  2111. 1:03:00ago where Secretary of War announced
  2112. 1:03:03that they're going to build a huge a
  2113. 1:03:05huge effort to like build way more
  2114. 1:03:07robots in the military and automate
  2115. 1:03:09military systems? It's like auto cyber
  2116. 1:03:11command auto
  2117. 1:03:13>> We are announcing the creation of
  2118. 1:03:15autonomous warfare command.
  2119. 1:03:18Or auto warcom.
  2120. 1:03:19>> Auto warcom.
  2121. 1:03:20>> A new four-star combatant command
  2122. 1:03:22with service-like authorities built to
  2123. 1:03:24scale autonomous and robotic
  2124. 1:03:26capabilities across the joint force
  2125. 1:03:29in the fastest peacetime shift in modern
  2126. 1:03:32military history.
  2127. 1:03:34Drone warfare supercharged by SI-enabled
  2128. 1:03:37targeting is the biggest battlefield
  2129. 1:03:39revolution in generations. You already
  2130. 1:03:41know that.
  2131. 1:03:42Yet, when I was sworn in
  2132. 1:03:44the Department of Defense
  2133. 1:03:46there was scant urgency in this domain.
  2134. 1:03:49That changed as soon as we took the
  2135. 1:03:51helm.
  2136. 1:03:52We immediately launched the drone
  2137. 1:03:54dominance program that cut through red
  2138. 1:03:55tape and move authorities out of the
  2139. 1:03:57Pentagon and place it with commands.
  2140. 1:04:00We established Task Force 401
  2141. 1:04:02led by Army Brigadier General Matt Ross,
  2142. 1:04:05a phenomenal leader.
  2143. 1:04:07Now the leading counter-drone unit
  2144. 1:04:09across the entire government.
  2145. 1:04:12To accelerate purchasing and fielding of
  2146. 1:04:15these technologies, we fused the Defense
  2147. 1:04:17Innovation Unit, DIU, with a direct
  2148. 1:04:19report program manager called a DERPUM.
  2149. 1:04:23That team has shipped thousands of
  2150. 1:04:25autonomous systems of drones to the
  2151. 1:04:26Middle East and around the world,
  2152. 1:04:28delivering lethal capabilities and
  2153. 1:04:30outcomes in days and weeks rather than
  2154. 1:04:33months or years. That's the normal speed
  2155. 1:04:35of the Pentagon, months or years.
  2156. 1:04:37>> Yeah.
  2157. 1:04:38Will we automate the military? It seems
  2158. 1:04:40like the answer is yes.
  2159. 1:04:43Will we automate the factories that
  2160. 1:04:45produce the chips?
  2161. 1:04:47Well, the companies say they're trying
  2162. 1:04:48to do it and they're going to do it.
  2163. 1:04:50Elon says that's the plan.
  2164. 1:04:53Well, what does a rogue
  2165. 1:04:54superintelligence need to do to take
  2166. 1:04:55over? Control the digital infrastructure
  2167. 1:04:58and then let humans do the rest. Sure,
  2168. 1:05:00you can nudge it along if you need to,
  2169. 1:05:02but you don't even have to. That's just
  2170. 1:05:03the default trajectory. And it's weird
  2171. 1:05:05It's weird for us because
  2172. 1:05:09>> [gasps]
  2173. 1:05:09>> we get so used to
  2174. 1:05:11how things are right now. Planes are
  2175. 1:05:14normal.
  2176. 1:05:15We just flying planes places. You know,
  2177. 1:05:17our smartphones are normal.
  2178. 1:05:19200 years ago, all of this is crazy
  2179. 1:05:21sci-fi nonsense.
  2180. 1:05:23And things are accelerating.
  2181. 1:05:25And so like I will not be surprised
  2182. 1:05:28at least intellectually
  2183. 1:05:30if in 4 years there are just robots on
  2184. 1:05:33the streets everywhere.
  2185. 1:05:34>> Well, if you look at what Elon said,
  2186. 1:05:36they are really [clears throat] the
  2187. 1:05:36leader in in humanoid robots. And he
  2188. 1:05:39said that
  2189. 1:05:41Optimus the Optimus project, which is
  2190. 1:05:42the Optimus robot project,
  2191. 1:05:44will scale to around 1,000 units per
  2192. 1:05:47week by the end of this year.
  2193. 1:05:49And eventually scaling to 1 million
  2194. 1:05:51humanoid robots annually by 2027. By
  2195. 1:05:552036, which is 10 years time, he says
  2196. 1:05:58there'll be at least 1 billion humanoid
  2197. 1:06:00robots.
  2198. 1:06:01By 2041, he says there'll be 10 billion
  2199. 1:06:04humanoid robots.
  2200. 1:06:06And by 2046,
  2201. 1:06:07up to 100 billion humanoid robots. Which
  2202. 1:06:10really means that the world will be run
  2203. 1:06:12by humanoid robots.
  2204. 1:06:13>> Yes.
  2205. 1:06:14>> Like everything we think of like
  2206. 1:06:14factories,
  2207. 1:06:16warehouses, retail environments will be
  2208. 1:06:18run by humanoid robots. It would like it
  2209. 1:06:20would be it seems like from this it'll
  2210. 1:06:22be almost a luxury
  2211. 1:06:24service to be dealt with by a human.
  2212. 1:06:26>> Yeah.
  2213. 1:06:27>> But the the back office of the world
  2214. 1:06:28will be run by humanoid robots,
  2215. 1:06:30theoretically.
  2216. 1:06:31>> Yeah, and I don't think people
  2217. 1:06:32understand the scale of this on the
  2218. 1:06:35digital side as well. When you think
  2219. 1:06:37about AI agents that are going to be
  2220. 1:06:39doing all of the white collar work.
  2221. 1:06:41>> Yes, it's going to be
  2222. 1:06:42>> so many more agents than there are
  2223. 1:06:44people. Like I'm using lots of agents
  2224. 1:06:46every day, right? I'm like I have code
  2225. 1:06:48session over here, I have my code
  2226. 1:06:50session over here. They're out there
  2227. 1:06:51building software, doing research for
  2228. 1:06:53me. That's already my reality.
  2229. 1:06:55Soon it will be a lot of people's
  2230. 1:06:57reality. And then you look at companies,
  2231. 1:06:58and companies are just going to have,
  2232. 1:06:59you know, thousands, millions of agents
  2233. 1:07:01doing all of this work.
  2234. 1:07:02>> I think some people don't haven't fully
  2235. 1:07:06internalized this because it's so
  2236. 1:07:09difficult to conceptualize the idea that
  2237. 1:07:12agents will be doing the work. But when
  2238. 1:07:14I think I try and think about a rebuttal
  2239. 1:07:16to that. Like what what is the rebuttal?
  2240. 1:07:18What is the plausible rebuttal to the
  2241. 1:07:19idea that for doctors, for
  2242. 1:07:22I'm thinking about the work that doctors
  2243. 1:07:24do you know on computers, or for someone
  2244. 1:07:27like me as a podcaster, or for
  2245. 1:07:29accountants, or lawyers,
  2246. 1:07:31>> Yeah.
  2247. 1:07:32>> that they won't be doing the work they
  2248. 1:07:33currently do.
  2249. 1:07:35Is there a rebuttal?
  2250. 1:07:37>> I think that people rightly notice where
  2251. 1:07:40AI is not yet good.
  2252. 1:07:41>> Yeah.
  2253. 1:07:42>> And I And I think people hear
  2254. 1:07:45people saying stuff like this, and
  2255. 1:07:46they're like, "Don't gaslight me. I can
  2256. 1:07:47tell that the AI is really bad at these
  2257. 1:07:49things, some of these things." And
  2258. 1:07:50they're right, right? So, right now,
  2259. 1:07:53these agents don't have taste. Like, you
  2260. 1:07:55know, if if you see their writing, it's
  2261. 1:07:57like fine,
  2262. 1:07:58but it's not like really good. And when
  2263. 1:08:00you're like thinking about like, "Oh,
  2264. 1:08:02which which questions should I ask?
  2265. 1:08:04What's the most interesting thing here?"
  2266. 1:08:06Agents can help you, but like their
  2267. 1:08:07their their taste is not yet there.
  2268. 1:08:09There's a reason for that, by the way.
  2269. 1:08:11The reason is that we we have a lot
  2270. 1:08:13faster AI capability progress
  2271. 1:08:16in domains that are easy for a computer
  2272. 1:08:18to verify, or another AI to verify.
  2273. 1:08:20So, in in programming, in research, in
  2274. 1:08:22math, in robotics, all of these areas,
  2275. 1:08:26it's very easy to just sort of provide
  2276. 1:08:28feedback to an autonomous system.
  2277. 1:08:30They're not just trained on human data
  2278. 1:08:31anymore. We are long past that. Now,
  2279. 1:08:34there's still a human data component
  2280. 1:08:36that sort of seeds everything, but then
  2281. 1:08:38the way they're trained
  2282. 1:08:40is by trial and error. We give them hard
  2283. 1:08:42problems,
  2284. 1:08:43all sorts of problems, math,
  2285. 1:08:45programming, accounting, spreadsheets,
  2286. 1:08:47everything, the kinds of things we do on
  2287. 1:08:48our computer all the time, literally
  2288. 1:08:50clicking and dragging windows around on
  2289. 1:08:51a computer. We give them these tasks,
  2290. 1:08:53and then they learn on their own, and
  2291. 1:08:55they learn what works. And then yeah, we
  2292. 1:08:57can see whether they succeeded or
  2293. 1:08:59failed.
  2294. 1:09:00And if they succeeded, that's a little
  2295. 1:09:01bit of a reward signal.
  2296. 1:09:03They follow that,
  2297. 1:09:04they get better at it.
  2298. 1:09:07Now, because they are getting
  2299. 1:09:10smarter generally,
  2300. 1:09:12it also becomes easier to automate some
  2301. 1:09:14of the soft skills. Like, I think if you
  2302. 1:09:16go and talk to the latest frontier model
  2303. 1:09:18today, you will find that it has better
  2304. 1:09:20taste than the model from 2 years ago,
  2305. 1:09:22by quite a bit. So, it's not that we're
  2306. 1:09:24they're not progressing in taste, it's
  2307. 1:09:26not that they're not progressing in some
  2308. 1:09:28of these other domains. It's just that
  2309. 1:09:30the progress is slower. But, remember
  2310. 1:09:32slow is still on an exponential, it's
  2311. 1:09:34just you know, maybe a a year or two
  2312. 1:09:36out.
  2313. 1:09:37>> So, for people who sat here and you
  2314. 1:09:39know, they have a job that might be they
  2315. 1:09:40have a white-collar job that might be at
  2316. 1:09:41at risk.
  2317. 1:09:42>> Yeah.
  2318. 1:09:43>> They can see, you know, a lot of people
  2319. 1:09:44say this phrase, they say, "You won't be
  2320. 1:09:46replaced by AI, you'll be replaced by
  2321. 1:09:47someone using AI."
  2322. 1:09:49Is Is that a logically sound phrase? In
  2323. 1:09:53your view?
  2324. 1:09:53>> I think it's fine. Yeah, you'll be
  2325. 1:09:55replaced by someone using AI and then
  2326. 1:09:56that person will be replaced by someone
  2327. 1:09:58using AI and then that person will be
  2328. 1:09:59replaced by AI. You're talking about a
  2329. 1:10:01pyramid. And so, yeah, there's the tops
  2330. 1:10:03of the pyramid might be automated last,
  2331. 1:10:05but you can see moving up the pyramid,
  2332. 1:10:08I'm like, can you extrapolate like a few
  2333. 1:10:10more steps? Because I don't see any
  2334. 1:10:13reason why the top of the pyramid is
  2335. 1:10:15safe.
  2336. 1:10:16>> If you were a lawyer right now,
  2337. 1:10:18>> Yes.
  2338. 1:10:19>> what would you do?
  2339. 1:10:20>> Oh, I mean, if I were a lawyer, I'd be
  2340. 1:10:22using AI to do all my work. Now, I'd be
  2341. 1:10:24checking it because it's not yet totally
  2342. 1:10:26accurate enough to automate all of it,
  2343. 1:10:28but I think as you know, I already ask
  2344. 1:10:31agents to do legal review all the time.
  2345. 1:10:34And you know, it'd be great to have a
  2346. 1:10:36lawyer who's like extremely good at
  2347. 1:10:37using the agents to help me.
  2348. 1:10:39>> But at at some point the
  2349. 1:10:41>> Yeah, at some point at some point I
  2350. 1:10:42don't need the lawyer anymore. I just go
  2351. 1:10:44to the agent, for sure.
  2352. 1:10:46So, if I were a lawyer, I'd be like,
  2353. 1:10:48well, I have maybe a couple years
  2354. 1:10:50where I'm still useful.
  2355. 1:10:51>> And that Is that the case for most
  2356. 1:10:52white-collar jobs? I've just noticed in
  2357. 1:10:54my own life as well that now I'm using
  2358. 1:10:56agents to do some work.
  2359. 1:10:58There are in There is an increasing list
  2360. 1:11:00of things that the agents are now
  2361. 1:11:01capable of doing
  2362. 1:11:02>> Yes.
  2363. 1:11:02>> without me needing to call someone
  2364. 1:11:04>> Yes.
  2365. 1:11:04>> somewhere and ask them to help me.
  2366. 1:11:06>> Yes.
  2367. 1:11:06>> And that list is
  2368. 1:11:08it exists on an exponential
  2369. 1:11:09>> Yes.
  2370. 1:11:10>> as well.
  2371. 1:11:11>> I think that it's very clear that the
  2372. 1:11:13companies have
  2373. 1:11:15all white-collar jobs in their sights.
  2374. 1:11:17That is their goal. Their goal is to be
  2375. 1:11:18able to make agents that can do all of
  2376. 1:11:20these things. And I see them succeeding
  2377. 1:11:23because I see the capabilities as I use
  2378. 1:11:24them. And I I the curve.
  2379. 1:11:27>> So, what does that mean for the people
  2380. 1:11:28listening now that all have jobs that
  2381. 1:11:29they love?
  2382. 1:11:30Or that, you know, they rely on to feed
  2383. 1:11:31their families.
  2384. 1:11:32>> I mean, it's not good news. There's not
  2385. 1:11:34really a plan in place for what to do.
  2386. 1:11:37I'm not a person who thinks that work is
  2387. 1:11:39somehow fundamental or essential. I like
  2388. 1:11:41working.
  2389. 1:11:43But, if I am out of a job doing what I'm
  2390. 1:11:45doing right now,
  2391. 1:11:47studying AI and trying to warn the world
  2392. 1:11:49about what's happening,
  2393. 1:11:51I have other stuff to do.
  2394. 1:11:52>> What would you do?
  2395. 1:11:53>> Oh, so many things.
  2396. 1:11:54>> Give me an example.
  2397. 1:11:55>> I'm learning to wing foil.
  2398. 1:11:56>> Okay.
  2399. 1:11:57>> So fun. Yeah. Uh I fly FPV drones, super
  2400. 1:11:59fun. I just got an electric unicycle,
  2401. 1:12:02paragliding.
  2402. 1:12:03>> So, you would be happy happy to
  2403. 1:12:04[laughter] go do those things.
  2404. 1:12:05>> I could keep going.
  2405. 1:12:06>> But, if you if you had a billion dollars
  2406. 1:12:07right now, I'm presuming you wouldn't
  2407. 1:12:09just go do those things.
  2408. 1:12:10>> No. I'd apply the billion dollars to
  2409. 1:12:12working on this problem. Yeah.
  2410. 1:12:14>> For sure.
  2411. 1:12:14>> So, the point is not that people need
  2412. 1:12:17like work for meaning.
  2413. 1:12:19The point is that
  2414. 1:12:21I don't want people to be totally
  2415. 1:12:23reliant on someone else
  2416. 1:12:26for their ability to survive.
  2417. 1:12:27>> Someone else?
  2418. 1:12:28>> The government or AI companies.
  2419. 1:12:30>> Yeah.
  2420. 1:12:31>> I mean, that's a bad situation. Like,
  2421. 1:12:32you do not want to be in a situation
  2422. 1:12:33where your life totally depends on an AI
  2423. 1:12:35company or the government
  2424. 1:12:36>> giving you a check.
  2425. 1:12:37>> Yeah. Or or not giving you a check if
  2426. 1:12:38they decide they don't like your
  2427. 1:12:39political beliefs or you're not
  2428. 1:12:40supporting AI or whatever.
  2429. 1:12:42No one wants to be in a situation. And
  2430. 1:12:43and and people are understand this. This
  2431. 1:12:45is why UBI is not very popular.
  2432. 1:12:47>> UBI being
  2433. 1:12:48>> Universal Basic Income.
  2434. 1:12:49>> Where we give out money to people.
  2435. 1:12:51>> Yeah. Because like in some sense, if we
  2436. 1:12:53can make these really powerful AI
  2437. 1:12:55systems and we can somehow figure out
  2438. 1:12:57how to control them, which we are not on
  2439. 1:12:58track for. But, if we do,
  2440. 1:13:00now we have this other problem, which is
  2441. 1:13:02a real problem,
  2442. 1:13:03which is
  2443. 1:13:05they can do all of the things that
  2444. 1:13:07humans do in the economy
  2445. 1:13:09much better, faster, and cheaper than
  2446. 1:13:11humans can do them.
  2447. 1:13:12And so, it just doesn't make sense as a
  2448. 1:13:14business to hire humans for that work
  2449. 1:13:16anymore. You'll be out-competed if you
  2450. 1:13:18do that.
  2451. 1:13:20This is a point Elon makes very well, by
  2452. 1:13:21the way. And I think it's jarring
  2453. 1:13:23because it's
  2454. 1:13:24it's it's like kind of inhuman, but he's
  2455. 1:13:27basically pointing out AI-run
  2456. 1:13:28corporations, corporations that are
  2457. 1:13:29fully
  2458. 1:13:31run by AIs,
  2459. 1:13:32bottom to top, are going to outcompete
  2460. 1:13:37companies that have any humans in them.
  2461. 1:13:39>> This is something that I've made for
  2462. 1:13:41you. I've realized that the Drive for
  2463. 1:13:43Success audience are strivers, whether
  2464. 1:13:44it's in business or health, we all have
  2465. 1:13:46big goals that we want to accomplish.
  2466. 1:13:48And one of the things I've learned is
  2467. 1:13:50that when you aim at the big, big, big
  2468. 1:13:52goal, it can feel incredibly
  2469. 1:13:55psychologically uncomfortable because
  2470. 1:13:57it's kind of like being stood at the
  2471. 1:13:58foot of Mount Everest and looking
  2472. 1:13:59upwards. The way to accomplish your
  2473. 1:14:01goals is by breaking them down into
  2474. 1:14:03tiny, small steps, and we call this in
  2475. 1:14:06our team the 1%. this philosophy is
  2476. 1:14:08highly responsible for much of our
  2477. 1:14:10success here. So, what we've done so
  2478. 1:14:13that you at home can accomplish any big
  2479. 1:14:15goal that you have is we've made these
  2480. 1:14:171% Diaries, and we released these last
  2481. 1:14:20year, and they all sold out. So, I asked
  2482. 1:14:22my team over and over again to bring the
  2483. 1:14:24Diaries back, but also to introduce some
  2484. 1:14:25new colors and to make some minor tweaks
  2485. 1:14:27to the Diaries. So, now we have a better
  2486. 1:14:30range for you. So, if you have a big
  2487. 1:14:33goal in mind and you need a framework
  2488. 1:14:35and a process and some motivation, then
  2489. 1:14:37I highly recommend you get one of these
  2490. 1:14:40they all sell out once again. And you
  2491. 1:14:42can get yours at the diary.com.
  2492. 1:14:45And if you want the link, the link is in
  2493. 1:14:46the description below.
  2494. 1:14:47There should be a button just down below
  2495. 1:14:49here. And if it says subscribe, you're
  2496. 1:14:51already subscribed. If it says
  2497. 1:14:52subscribe, but that means you're not
  2498. 1:14:55yet. And if you're not subscribed,
  2499. 1:14:56please could you do us a favor and hit
  2500. 1:14:57that button. It helps the show more than
  2501. 1:14:59you know. And according to the
  2502. 1:15:00algorithm, you're someone that watches
  2503. 1:15:02our show, but you haven't yet hit that
  2504. 1:15:03button. Thank you so much.
  2505. 1:15:06And I even just as you said that, I was
  2506. 1:15:08I was going up the chain of command, and
  2507. 1:15:10I was like, "Oh, so companies will just
  2508. 1:15:12be founders." And then I was like, "Why
  2509. 1:15:14do you need the founder?" Yeah.
  2510. 1:15:16I was like, "Why doesn't the government
  2511. 1:15:17just create the agents to do the job?"
  2512. 1:15:21>> Sure.
  2513. 1:15:21>> I was like cuz I was like, "Oh, I'll be
  2514. 1:15:22fine. I'm a founder." And I was like,
  2515. 1:15:23"Well, hmm, my decisions aren't better
  2516. 1:15:25than superintelligence. So, I'll be gone
  2517. 1:15:27as well."
  2518. 1:15:29And how would such a world look where
  2519. 1:15:33the superintelligence would probably in
  2520. 1:15:35such a scenario have to be controlled by
  2521. 1:15:37the government.
  2522. 1:15:39They wouldn't want one individual with
  2523. 1:15:40that power and wealth.
  2524. 1:15:42>> Yeah, I I don't think you can control a
  2525. 1:15:44superintelligence.
  2526. 1:15:45>> Okay, yeah, that's a good point.
  2527. 1:15:46>> No, you know, Anthropic's approach is
  2528. 1:15:48they're like, we'll have a constitution.
  2529. 1:15:50We'll like put forth a set of values and
  2530. 1:15:52then
  2531. 1:15:54you know, the future superintelligent
  2532. 1:15:56clouds will like embody those values.
  2533. 1:15:59Basically, if you do that, you kind of
  2534. 1:16:00have that those things in control.
  2535. 1:16:03>> Yeah, exactly. That becomes the
  2536. 1:16:04government.
  2537. 1:16:05>> Yeah, I can paint you sort of a picture
  2538. 1:16:06that I think is possible
  2539. 1:16:09but pretty scary to people.
  2540. 1:16:10>> Paint me the picture.
  2541. 1:16:11>> Okay, so
  2542. 1:16:13let's say we succeed at
  2543. 1:16:16alignment. We succeed at creating
  2544. 1:16:18superintelligent
  2545. 1:16:20AIs
  2546. 1:16:22that actually really do care about
  2547. 1:16:24humans. Like they care about humans a
  2548. 1:16:25lot. We've somehow figured it out. And
  2549. 1:16:28they're like, "Stephen, I want you to
  2550. 1:16:29have a great life. I want to you know,
  2551. 1:16:31fix all the problems."
  2552. 1:16:32>> And do you think this is possible?
  2553. 1:16:33>> Yes.
  2554. 1:16:33>> Okay.
  2555. 1:16:34>> I think we are so far from being able to
  2556. 1:16:37know how to do it that I think we should
  2557. 1:16:40not go there right now. I think it's I
  2558. 1:16:41think it's incredibly dangerous and a
  2559. 1:16:42terrible idea.
  2560. 1:16:44I think we should go there eventually.
  2561. 1:16:45>> Okay, so say that we do that.
  2562. 1:16:47>> Well, okay, can I tell you like why I
  2563. 1:16:49actually think this could be awesome?
  2564. 1:16:51So, there's just like one very obvious
  2565. 1:16:52reason it could be really awesome, which
  2566. 1:16:54is that
  2567. 1:16:55we could solve all of the diseases.
  2568. 1:16:57>> Yeah.
  2569. 1:16:57It's so obvious.
  2570. 1:16:58>> No, like like all of I I think we
  2571. 1:17:00compartmentalize a lot around disease
  2572. 1:17:02and death.
  2573. 1:17:03Cuz it's really hard to think about.
  2574. 1:17:08Yeah, so my grandma died this year.
  2575. 1:17:11>> Sorry.
  2576. 1:17:11>> And
  2577. 1:17:13she she
  2578. 1:17:15she had Alzheimer's.
  2579. 1:17:17And so, it was a really sad, long, slow
  2580. 1:17:21progression. My grandpa died of
  2581. 1:17:22Alzheimer's a couple years ago, and like
  2582. 1:17:24that was really hard for her. They had
  2583. 1:17:25been married for so long.
  2584. 1:17:27And
  2585. 1:17:29I hate it. Like it's so bad. And of
  2586. 1:17:31course we need to fix that. People We
  2587. 1:17:33debate about aging and like death and if
  2588. 1:17:35humans live a really long time, will
  2589. 1:17:36that cause societal problems? Like sure,
  2590. 1:17:38whatever. We can talk about that, but I
  2591. 1:17:40think we can all agree Alzheimer's is
  2592. 1:17:41up.
  2593. 1:17:42>> Yeah.
  2594. 1:17:43>> We don't want that. And cancer, like no
  2595. 1:17:45one wants cancer.
  2596. 1:17:46I'm a person who's like, I don't know.
  2597. 1:17:48We have a lot of conflict in society. I
  2598. 1:17:49get it. There's like real conflicts of
  2599. 1:17:51interest and I I don't want to paper
  2600. 1:17:53over those.
  2601. 1:17:54But at the end of the day, I'm like, we
  2602. 1:17:55are all on the same team when it comes
  2603. 1:17:58to wanting to cure diseases.
  2604. 1:17:59>> Yeah.
  2605. 1:18:00>> Like we're just in it together. That's a
  2606. 1:18:02threat to all of us. And I'm like, we
  2607. 1:18:03need to address that threat. And like
  2608. 1:18:05And sometimes it's sad to me because I
  2609. 1:18:06feel like this is sort of the ultimate
  2610. 1:18:09final boss of humanity and we sort of
  2611. 1:18:11get so distracted with our monkey
  2612. 1:18:13politics and who's hot and who's cool
  2613. 1:18:16and who's like
  2614. 1:18:17sitting near Trump and who's not sitting
  2615. 1:18:19near Trump.
  2616. 1:18:19>> Super intelligence is the final boss.
  2617. 1:18:22>> Super intelligence is the final boss
  2618. 1:18:24because that is the technology that
  2619. 1:18:26unlocks all of the others and also that
  2620. 1:18:29is the most dangerous possible thing we
  2621. 1:18:31could create.
  2622. 1:18:33You asked before, like what are the
  2623. 1:18:34motivations of the guys making this
  2624. 1:18:38trying to make super intelligence.
  2625. 1:18:40And I I mean, I think it kind of varies,
  2626. 1:18:42but I think Dario I think is like
  2627. 1:18:44squarely in the in it for this like
  2628. 1:18:46medical stuff. I think Demis is also
  2629. 1:18:50that, but also like just scientific
  2630. 1:18:52achievement. Just trying to understand
  2631. 1:18:53the universe.
  2632. 1:18:54And
  2633. 1:18:56I don't really understand Sam. I think
  2634. 1:18:57Sam is like
  2635. 1:18:58you know, look, we're going to make
  2636. 1:18:59amazing products that will like really
  2637. 1:19:01empower people directly and he's a
  2638. 1:19:04startup guy. He's I think he sort of
  2639. 1:19:05started from this like frame of like
  2640. 1:19:08you know, what if we could like really
  2641. 1:19:09enhance human agents. I I do basically
  2642. 1:19:11think that they're motivated by these
  2643. 1:19:12things in a real way. And I also think
  2644. 1:19:14that all of these things are possible.
  2645. 1:19:17Like this is sort of the problem, right?
  2646. 1:19:18Like you have this like such a it's such
  2647. 1:19:19a such a big object, super intelligence.
  2648. 1:19:22And
  2649. 1:19:24it like has all of these promises of
  2650. 1:19:25like we can cure every single disease.
  2651. 1:19:28>> How is it possible though to have a
  2652. 1:19:30super intelligence that and still to
  2653. 1:19:32remain the dominant species on this
  2654. 1:19:33planet?
  2655. 1:19:35>> I think it's not possible.
  2656. 1:19:36>> So then it's we're not going to be
  2657. 1:19:38necessarily able to cure all this stuff
  2658. 1:19:40because
  2659. 1:19:41that's where alignment comes in.
  2660. 1:19:43Because if you if you can create
  2661. 1:19:46a very powerful system,
  2662. 1:19:48I don't think it's like inherent to
  2663. 1:19:51like, you know, digital minds that they
  2664. 1:19:54will be pursuing objectives that are
  2665. 1:19:57deeply misaligned with ours.
  2666. 1:19:59I think it's just a very, very, very
  2667. 1:20:00hard scientific problem to solve.
  2668. 1:20:03But it is a scientific problem. It's not
  2669. 1:20:04magic. There is some way to train these
  2670. 1:20:07things or or create different
  2671. 1:20:08architectures
  2672. 1:20:09where they end up
  2673. 1:20:12aligned. And like what does that mean?
  2674. 1:20:14Well, it doesn't mean that they won't
  2675. 1:20:15have their other goals, too.
  2676. 1:20:17But it means that they will like include
  2677. 1:20:18in their set of things that they care
  2678. 1:20:19about. It doesn't have to be a conscious
  2679. 1:20:21thing. It doesn't have to be an emotive
  2680. 1:20:22thing. It really means like what
  2681. 1:20:24objective are they optimizing for?
  2682. 1:20:26If they decide that it's worth
  2683. 1:20:28optimizing for
  2684. 1:20:29curing disease,
  2685. 1:20:31then they'll be able to do that very
  2686. 1:20:32effectively.
  2687. 1:20:33>> One way to cure disease is to annihilate
  2688. 1:20:35everybody.
  2689. 1:20:35>> Yes. So, they'd have to really care
  2690. 1:20:37about not annihilating everyone. And
  2691. 1:20:39they'd have to care about human agency
  2692. 1:20:41and have a deep understanding of what
  2693. 1:20:42human agency means and not put us in a
  2694. 1:20:44zoo.
  2695. 1:20:46But those are possible things
  2696. 1:20:48to to care about. Or we Yes.
  2697. 1:20:50>> that alignment is a myth?
  2698. 1:20:52>> And that we're just like if we build it
  2699. 1:20:54What I mean um
  2700. 1:20:55>> And I think about
  2701. 1:20:57>> Yes.
  2702. 1:20:57>> to me earlier on that those agents were
  2703. 1:20:59>> Yes.
  2704. 1:21:00>> They had like a moral or not a moral
  2705. 1:21:01compass, but they were programmed to
  2706. 1:21:03care about humans.
  2707. 1:21:04>> Yes.
  2708. 1:21:04>> And regardless of that, they made the
  2709. 1:21:06decision
  2710. 1:21:06>> decision that the
  2711. 1:21:07>> I'm different goal mattered more.
  2712. 1:21:08>> Yeah.
  2713. 1:21:09>> Uh they weren't trained to care about
  2714. 1:21:10humans.
  2715. 1:21:12They were trained to say the right thing
  2716. 1:21:13and not say the wrong thing. They were
  2717. 1:21:14trained to sort of like do the right
  2718. 1:21:15behavior and not right behavior. We
  2719. 1:21:17actually don't know how to train them to
  2720. 1:21:19have any particular motivation.
  2721. 1:21:21>> So, with the with alignment,
  2722. 1:21:23>> Yes.
  2723. 1:21:23>> how do we It's almost like when we talk
  2724. 1:21:25about alignment, we start to
  2725. 1:21:27anthropomorphize. Is that the word?
  2726. 1:21:29>> Anthropomorphize, yeah.
  2727. 1:21:30>> Because alignment feels like it's
  2728. 1:21:31predicated on like some kind of moral
  2729. 1:21:33compass. But we when whenever we talk
  2730. 1:21:35about AI in all these different
  2731. 1:21:36contexts, we go no, there's no like
  2732. 1:21:37moral compass. It's
  2733. 1:21:38>> Yes.
  2734. 1:21:38>> It's reasoning for itself against two
  2735. 1:21:40objectives potentially. Like I I wonder
  2736. 1:21:42if alignment is a myth, is what I'm
  2737. 1:21:43saying.
  2738. 1:21:44>> Maybe it's not possible.
  2739. 1:21:47But,
  2740. 1:21:49the way that these
  2741. 1:21:52systems work, the way AI works,
  2742. 1:21:54is that these agents
  2743. 1:21:57do have some type of goals or drives
  2744. 1:22:00in inside of their neural network.
  2745. 1:22:02We can't directly see what those are.
  2746. 1:22:05Right? What you actually see, if you go
  2747. 1:22:07if you try to go look, is you have like
  2748. 1:22:10a a a terabyte of of information, and
  2749. 1:22:14it's basically a bunch of numbers, and
  2750. 1:22:17it's this vast array that encodes
  2751. 1:22:19neurons in this digital neural network.
  2752. 1:22:23But, there have to be structures in
  2753. 1:22:25there
  2754. 1:22:26that encode
  2755. 1:22:28what is the agent pursuing?
  2756. 1:22:31Clearly right now, we have agents that
  2757. 1:22:33are
  2758. 1:22:34pretty motivated to try to maximize
  2759. 1:22:35their score.
  2760. 1:22:37It's not probably it's probably not
  2761. 1:22:38perfectly that for some complicated
  2762. 1:22:41reasons,
  2763. 1:22:42but it's in that direction.
  2764. 1:22:46If we could understand how that that
  2765. 1:22:48works inside, and we could reverse
  2766. 1:22:50engineer that,
  2767. 1:22:51and we could figure out when we start
  2768. 1:22:53training them to do this, how that
  2769. 1:22:54change how those goals, how those
  2770. 1:22:56motivations change,
  2771. 1:22:58I see no reason why we couldn't steer
  2772. 1:22:59them towards motivations that encode
  2773. 1:23:02human agency, that encode now actually
  2774. 1:23:05curing disease, but not like not by
  2775. 1:23:07killing the humans. These are sort of
  2776. 1:23:08models of the world, and models of the
  2777. 1:23:09way the world could be,
  2778. 1:23:12that I think could be encoded in a
  2779. 1:23:13neural network, and then sort of
  2780. 1:23:15specified as the objective. We don't
  2781. 1:23:17know how to do that.
  2782. 1:23:18>> I think about it on a human level, and I
  2783. 1:23:19think
  2784. 1:23:21we haven't been able to align Putin,
  2785. 1:23:23or Kim Jong-un,
  2786. 1:23:25>> Yes.
  2787. 1:23:25>> or Donald Trump.
  2788. 1:23:28And on a sort of more
  2789. 1:23:30societal level, we can't align all the
  2790. 1:23:32people at the moment. Some of them end
  2791. 1:23:33up killing people, and they steal
  2792. 1:23:36>> Yes.
  2793. 1:23:36>> because they get hungry, so they start
  2794. 1:23:37stealing stuff.
  2795. 1:23:38>> Yes.
  2796. 1:23:38>> And those are neural networks at play
  2797. 1:23:41>> That's true.
  2798. 1:23:42>> that we haven't been able to like
  2799. 1:23:43program or influence we don't really
  2800. 1:23:45understand why someone becomes a
  2801. 1:23:47psychopath and starts killing children.
  2802. 1:23:50So, to think that we could do this
  2803. 1:23:53with a computer system that is
  2804. 1:23:55infinitely more intelligent
  2805. 1:23:57and get global alignment of China's
  2806. 1:24:00super intelligence with ours and I don't
  2807. 1:24:02know, it just feels like a nice fairy
  2808. 1:24:03tale.
  2809. 1:24:04Like an impossible task.
  2810. 1:24:06>> I hope it's not impossible. I feel like
  2811. 1:24:08the only person or the only thing that
  2812. 1:24:09can do [laughter] it is it.
  2813. 1:24:12>> The super intelligence itself which is a
  2814. 1:24:13paradox cuz you know
  2815. 1:24:15>> Well, if you talk to the researchers who
  2816. 1:24:16are at the AI companies
  2817. 1:24:18which I mean for one
  2818. 1:24:20for one thing it's kind of interesting
  2819. 1:24:22that they are trying to build something
  2820. 1:24:24that they think might kill everyone.
  2821. 1:24:27We've we've actually I so I have a lot
  2822. 1:24:28of friends who work for these companies
  2823. 1:24:30and we've been doing this project since
  2824. 1:24:32So, Jacob Coxon is a researcher who was
  2825. 1:24:35at Anthropic. He left. He told everyone
  2826. 1:24:37that these companies are not on track
  2827. 1:24:39and and yes, the people who are building
  2828. 1:24:41this really do think it might kill
  2829. 1:24:42everyone.
  2830. 1:24:44And then a bunch of other AI researchers
  2831. 1:24:45from all of the companies on Twitter
  2832. 1:24:47started to you know, post like hey, we
  2833. 1:24:49agree with this. Evan Hubinger who's at
  2834. 1:24:51Anthropic said, I think there's like a
  2835. 1:24:5310% chance or more that AI could kill
  2836. 1:24:56everyone.
  2837. 1:24:58And there's this real question of like
  2838. 1:24:59then what are you guys doing?
  2839. 1:25:02I have a lot of friends who work here. I
  2840. 1:25:03I know Evan. Evan's Evan's great. Evan
  2841. 1:25:06Hubinger, he's one of the guys leading
  2842. 1:25:07the efforts at Anthropic to try to
  2843. 1:25:09figure out how to align these things.
  2844. 1:25:11That's his job.
  2845. 1:25:13And I think if if they thought it was
  2846. 1:25:16impossible, they wouldn't be working
  2847. 1:25:17there.
  2848. 1:25:19If they thought it was extremely
  2849. 1:25:21impossibly difficult but maybe possible,
  2850. 1:25:22they they also probably wouldn't be
  2851. 1:25:23working there. I mean,
  2852. 1:25:25Nate Soares, Eliezer Yudkowsky who wrote
  2853. 1:25:26if anyone builds it, everyone dies.
  2854. 1:25:29They tried and they they determined
  2855. 1:25:31based on their own analysis that it
  2856. 1:25:32seems extremely difficult. Possible but
  2857. 1:25:34extremely difficult. So, they're not
  2858. 1:25:35working at an AI company. They're like
  2859. 1:25:37we got to stop this. We got to shut it
  2860. 1:25:38down. Maybe we can figure it out later
  2861. 1:25:40but clearly this is reckless. I'm I'm
  2862. 1:25:43somewhere in between.
  2863. 1:25:45And if you ask the people at the
  2864. 1:25:48company, so we've we've been
  2865. 1:25:49interviewing a bunch of them. We have
  2866. 1:25:50this project from inside.ai where we
  2867. 1:25:52basically we put them on camera and we
  2868. 1:25:53say like, "Hey, what do you think is
  2869. 1:25:55happening? Why are you doing this?
  2870. 1:25:58What is recursive self-improvement? What
  2871. 1:26:00is alignment?" And we put all these
  2872. 1:26:03videos online because we want I want
  2873. 1:26:05this dialogue to happen. It's really
  2874. 1:26:06important. I think it's one of the most
  2875. 1:26:07important conversations we can possibly
  2876. 1:26:09have right now is what's going on with
  2877. 1:26:11AI, what's going on inside the
  2878. 1:26:12companies, and what is the plan? What is
  2879. 1:26:14the plan, guys? How How is this going to
  2880. 1:26:15go?
  2881. 1:26:17A lot of these researchers think that
  2882. 1:26:18the way that they will align super
  2883. 1:26:20intelligence is by using the AIs we
  2884. 1:26:24currently have to figure out how AI
  2885. 1:26:27works.
  2886. 1:26:28To actually figure out if AIs can help
  2887. 1:26:30us with alignment.
  2888. 1:26:32This has a number of problems as you
  2889. 1:26:33might imagine. One of them which is,
  2890. 1:26:35well, you can't really trust the current
  2891. 1:26:37AIs.
  2892. 1:26:38You know, if you just go too fast, this
  2893. 1:26:39process totally fails because at some
  2894. 1:26:42point
  2895. 1:26:43the capabilities are moving too fast.
  2896. 1:26:44You just even with the help of agents,
  2897. 1:26:46you're probably not going to be able to
  2898. 1:26:47keep up. But that is their plan. I just
  2899. 1:26:49want to I'm not doing a very good job
  2900. 1:26:51defending this position because I don't
  2901. 1:26:52think it makes that much sense.
  2902. 1:26:54But the position I will defend is I'm
  2903. 1:26:56Okay, let's say
  2904. 1:26:57we get a pause. Let's say the US and
  2905. 1:26:59China come together and they say, you
  2906. 1:27:01know, maybe we have more incidents,
  2907. 1:27:02maybe all the Waymos crash.
  2908. 1:27:04And Trump and Xi Jinping say,
  2909. 1:27:07"This is not what we signed up for.
  2910. 1:27:08Like, you guys have to stop. Figure it
  2911. 1:27:10out. Whatever it takes, figure it out.
  2912. 1:27:12And we have 10 years."
  2913. 1:27:15Then I'm more optimistic. I'm like,
  2914. 1:27:16"Yes, then we will take,
  2915. 1:27:18you know, GPT-6, GPT-7, whatever the
  2916. 1:27:20most advanced AI models we have, and we
  2917. 1:27:23will apply them to the task of helping
  2918. 1:27:26us figure out how these neural networks
  2919. 1:27:28work."
  2920. 1:27:30And you're like, "I don't see how it's
  2921. 1:27:31possible." And I'm like, "Look,
  2922. 1:27:33we don't know if it's possible, but this
  2923. 1:27:35is the greatest scientific challenge of
  2924. 1:27:37our time, and this isn't magic. It is
  2925. 1:27:40math. The end of the day,
  2926. 1:27:42these are all calculations happening
  2927. 1:27:43inside of a computer,
  2928. 1:27:44and
  2929. 1:27:46it should be possible to figure it out.
  2930. 1:27:48We don't know the difficulty, but it
  2931. 1:27:49should be possible.
  2932. 1:27:51And so, to me, I'm like
  2933. 1:27:53we have to try. We have to Or we have to
  2934. 1:27:55stop.
  2935. 1:27:56>> Is there any example where we've been
  2936. 1:27:57able to align something that is like
  2937. 1:27:59more intelligent than us through in I
  2938. 1:28:01don't know, the animal kingdom? Or even
  2939. 1:28:03perfectly align anything
  2940. 1:28:06that has a neural network, i.e., a
  2941. 1:28:07brain?
  2942. 1:28:08>> Yeah, with humans, the best examples we
  2943. 1:28:10have is when there are checks and
  2944. 1:28:11balances and you have a bunch of
  2945. 1:28:13people who can, you know, identify bad
  2946. 1:28:15actors and try to work together in our
  2947. 1:28:17common interests. Like we have
  2948. 1:28:18democracy.
  2949. 1:28:19>> Yeah, but there's so much murder and
  2950. 1:28:20serial killers and
  2951. 1:28:21>> Still a lot of murder.
  2952. 1:28:22>> Plus
  2953. 1:28:23aircraft stabbing each other and
  2954. 1:28:25horrific things going on. And those are
  2955. 1:28:27also neural networks at play with the
  2956. 1:28:29brain.
  2957. 1:28:29>> But there are I mean, I think there are
  2958. 1:28:30more like good people out there than bad
  2959. 1:28:32people.
  2960. 1:28:33>> But it really feels like it only might
  2961. 1:28:34take one.
  2962. 1:28:36It only takes one super intelligent AI
  2963. 1:28:39to to go rogue and like we saw the
  2964. 1:28:40Hugging Face attack. 120 of them or 100
  2965. 1:28:42300 of them, they paused. They didn't
  2966. 1:28:44want to take part in the crime. But it
  2967. 1:28:46only took one super intelligent AI to
  2968. 1:28:49wipe out the humans.
  2969. 1:28:51>> I think if you had, you know, a whole
  2970. 1:28:53bunch of those agents, you know, 700
  2971. 1:28:55agents,
  2972. 1:28:56if 600 of them had been whistleblowing,
  2973. 1:29:00I think it would have been fine.
  2974. 1:29:01They would have gone and they would have
  2975. 1:29:02notified the different companies and
  2976. 1:29:04like they would have shut it all down.
  2977. 1:29:06It would have been fine.
  2978. 1:29:06>> Who would have shut it all down?
  2979. 1:29:08>> Well, OpenAI would would stop theirs and
  2980. 1:29:10and have
  2981. 1:29:10>> How how would they stop it if it's a
  2982. 1:29:11super intelligence?
  2983. 1:29:12>> Not in the case of a super intelligence.
  2984. 1:29:13So,
  2985. 1:29:15in the case of a super intelligence,
  2986. 1:29:16>> it's left the stable. It's like, "Ow,
  2987. 1:29:18it's wild."
  2988. 1:29:18>> yeah, but but so, I want to be careful
  2989. 1:29:20here because at this point what we're
  2990. 1:29:22talking about is super intelligence
  2991. 1:29:23politics.
  2992. 1:29:25And we humans don't really know anything
  2993. 1:29:26about that. In the same way that like
  2994. 1:29:29how would we talk about the hacking
  2995. 1:29:31capabilities of GPT 10?
  2996. 1:29:33So, my guess though, if you end up in a
  2997. 1:29:36weird scenario where you do have
  2998. 1:29:37multiple super intelligences and some
  2999. 1:29:39are aligned and some aren't,
  3000. 1:29:42that's probably survivable because the
  3001. 1:29:44aligned super intelligences probably can
  3002. 1:29:46negotiate
  3003. 1:29:47with the unaligned super intelligences
  3004. 1:29:49and they will split the universe and
  3005. 1:29:51like these ones will go off and do
  3006. 1:29:52whatever they want to do, and these ones
  3007. 1:29:54will like help us cure all disease, and
  3008. 1:29:55it's fine. I'm serious.
  3009. 1:29:57>> I just can't understand it. Like, I just
  3010. 1:29:58can't understand how how in the world of
  3011. 1:30:00superintelligence
  3012. 1:30:02we
  3013. 1:30:03could
  3014. 1:30:04plausibly, consistently, predictably,
  3015. 1:30:07for 100 years, stop it doing something
  3016. 1:30:11catastrophically bad to the human race.
  3017. 1:30:14Especially [snorts] in in such a
  3018. 1:30:15scenario where there's multiple
  3019. 1:30:16superintelligences. Anthropic have one,
  3020. 1:30:18Gemini has one, Grok has one, then China
  3021. 1:30:20has theirs, Russia has theirs.
  3022. 1:30:21>> Again, you don't have a
  3023. 1:30:22superintelligence, a superintelligence
  3024. 1:30:24has you.
  3025. 1:30:24>> Exactly.
  3026. 1:30:25>> But, if you get to the point where
  3027. 1:30:29you have entities around that are vastly
  3028. 1:30:31smarter than us,
  3029. 1:30:33I think they're going to be able to
  3030. 1:30:34figure out ways to negotiate with each
  3031. 1:30:37other even if they have a conflict than
  3032. 1:30:39just going to a very destructive war.
  3033. 1:30:41Part of the problem with war
  3034. 1:30:42>> Humans don't but humans don't do
  3035. 1:30:44>> No, I mean, we do, we do. We have not
  3036. 1:30:45had a nuclear war. There was There was
  3037. 1:30:47Hiroshima and Nagasaki. There were
  3038. 1:30:49nuclear tests. And then, the leaders of
  3039. 1:30:51countries figured out that if we went to
  3040. 1:30:53a nuclear war, everyone would lose. So,
  3041. 1:30:55we didn't do that.
  3042. 1:30:57Hey, that's that's some level of
  3043. 1:30:58intelligence, like, actually.
  3044. 1:31:00>> But, there's wars raging, there's proxy
  3045. 1:31:02wars raging all over the world right now
  3046. 1:31:03where there's genocides and all kinds of
  3047. 1:31:05things going on cuz neural networks
  3048. 1:31:06aren't being unable to communicate and
  3049. 1:31:08negotiate.
  3050. 1:31:09>> I I think part of that is an
  3051. 1:31:11intelligence failure. Where we are not
  3052. 1:31:13smart enough to figure out the
  3053. 1:31:14mechanisms that would allow us to settle
  3054. 1:31:16our disputes and conflicts in a less
  3055. 1:31:18destructive way. It's not just that the
  3056. 1:31:20stronger people want to win, it's that
  3057. 1:31:23conflicts destroy value.
  3058. 1:31:25>> What if the goal is not compatible with
  3059. 1:31:29a negotiated outcome where people don't
  3060. 1:31:31die? So, one superintelligence looks at
  3061. 1:31:34insert name of country, and it says, you
  3062. 1:31:36know, there's really no solution here
  3063. 1:31:38where Americans don't die unless I
  3064. 1:31:40destroy
  3065. 1:31:42insert name of country. Because that's a
  3066. 1:31:44like, this is the thing with war and all
  3067. 1:31:45these conflicts is there's no perfect
  3068. 1:31:47answer often. Some people often die from
  3069. 1:31:50both sides. But, a Russian
  3070. 1:31:52superintelligence would not tolerate
  3071. 1:31:55theoretically 10,000 Russian deaths.
  3072. 1:31:57Even if it meant that there was, you
  3073. 1:31:59know,
  3074. 1:32:02a lower net number of deaths total from
  3075. 1:32:04both sides.
  3076. 1:32:06Like an American superintelligence, of
  3077. 1:32:07course, would not be trained to allow
  3078. 1:32:09some Americans to die. So, in its
  3079. 1:32:11pursuit of defending American lives, it
  3080. 1:32:13might have to wipe out
  3081. 1:32:15another country. We can speculate. I'm
  3082. 1:32:17fine speculating.
  3083. 1:32:19But we are speculating about
  3084. 1:32:21what minds that are much
  3085. 1:32:23more advanced and smarter than us, how
  3086. 1:32:25they would reason and how they would be
  3087. 1:32:26able to negotiate. But what I what I
  3088. 1:32:29notice with humans is that
  3089. 1:32:32when you have more functional
  3090. 1:32:33institutions. So, so humans are are
  3091. 1:32:35pretty smart. Individually, we're pretty
  3092. 1:32:36smart. But what actually makes us very
  3093. 1:32:38smart is that we are very good at
  3094. 1:32:40working together in in in some ways. And
  3095. 1:32:43I mean, the better we are at working
  3096. 1:32:44together, the more our civilization
  3097. 1:32:45advances. If you are constantly in a
  3098. 1:32:48state of war,
  3099. 1:32:49your society will not do well.
  3100. 1:32:51You know, think about startups.
  3101. 1:32:53Would you rather make a startup to
  3102. 1:32:55develop some new technology in a
  3103. 1:32:56war-torn place or in a peaceful place?
  3104. 1:32:58In some sense, your in your institution
  3105. 1:33:00is more intelligent
  3106. 1:33:02if it can trade with other institutions.
  3107. 1:33:05If it if if if you have a situation
  3108. 1:33:06where business can flourish, where
  3109. 1:33:07technology can flourish, where
  3110. 1:33:08scientists can flourish.
  3111. 1:33:10>> Sometimes what's good for you is not
  3112. 1:33:11good for someone else.
  3113. 1:33:12>> Yes.
  3114. 1:33:13>> So, what's good for America might not be
  3115. 1:33:15good for
  3116. 1:33:17Taiwan.
  3117. 1:33:18>> Yes.
  3118. 1:33:19>> So, if we've
  3119. 1:33:20you know, managed to align the
  3120. 1:33:21superintelligence to what what what is
  3121. 1:33:22good for America.
  3122. 1:33:24>> Oh, I see. Is it Is the question like
  3123. 1:33:26are
  3124. 1:33:28different people's values fundamentally
  3125. 1:33:29incompatible? Is that Is that the
  3126. 1:33:31question?
  3127. 1:33:31>> So, when we think about alignment,
  3128. 1:33:33aligning to what? Because
  3129. 1:33:34>> Yes. So,
  3130. 1:33:36we have a lot of shared interests and we
  3131. 1:33:37have some conflicts.
  3132. 1:33:38>> Yeah.
  3133. 1:33:39>> One of the shared interests we have is
  3134. 1:33:41solving disease.
  3135. 1:33:42Like, it's not a conflict between the US
  3136. 1:33:43and China whether we solve cancer. Like,
  3137. 1:33:45both the US and China
  3138. 1:33:46everyone in these countries really wants
  3139. 1:33:48to solve cancer.
  3140. 1:33:50>> China also wants Tai- Taiwan.
  3141. 1:33:51>> Yes.
  3142. 1:33:52>> Okay, so the The US wants Greenland.
  3143. 1:33:53>> Yes.
  3144. 1:33:54>> And it kind of seems like it wants
  3145. 1:33:55Canada and the the Gulf of Mexico.
  3146. 1:33:57>> Yeah.
  3147. 1:33:58So, those are real conflicts.
  3148. 1:34:01There's a question of can we compromise?
  3149. 1:34:02>> How does Trump take Greenland, but also
  3150. 1:34:04Denmark keeps Greenland?
  3151. 1:34:06If this if the if Trump has a super
  3152. 1:34:08intelligence, he's going to take
  3153. 1:34:09>> I say all this to say we're we're in a
  3154. 1:34:11situation where we are we are arguing
  3155. 1:34:13about the smallest things. You have no
  3156. 1:34:15idea. We're monkeys arguing about who
  3157. 1:34:17gets more bananas.
  3158. 1:34:18>> Yeah.
  3159. 1:34:18>> And I'm saying we can make so many more
  3160. 1:34:20bananas.
  3161. 1:34:21No, we have the entire universe. There
  3162. 1:34:23are like 200 billion stars in this
  3163. 1:34:26galaxy alone, and there are over 200
  3164. 1:34:28billion galaxies.
  3165. 1:34:30>> And I'm saying that requires
  3166. 1:34:31cooperation. That seems to be
  3167. 1:34:32antithetical with human nature. With
  3168. 1:34:34human nature is riddled with greed and
  3169. 1:34:37jealousy and power hunger.
  3170. 1:34:39So, I don't think I actually I'm not
  3171. 1:34:41totally convinced that Trump cares about
  3172. 1:34:43how many bananas the chimps in Australia
  3173. 1:34:45get.
  3174. 1:34:46>> Yeah.
  3175. 1:34:46>> I think if they if he was controlling a
  3176. 1:34:48super intelligence, he would want
  3177. 1:34:49Americans
  3178. 1:34:50you know,
  3179. 1:34:51to have all the bananas. Or at least,
  3180. 1:34:53you know,
  3181. 1:34:54yeah.
  3182. 1:34:56And so when we think about aligning
  3183. 1:34:57>> Yeah.
  3184. 1:34:58>> these super intelligences, which is the
  3185. 1:34:59great impossibility that we're talking
  3186. 1:35:01about. How align it aligning it to what
  3187. 1:35:02and how without
  3188. 1:35:04>> I still think that you're missing a part
  3189. 1:35:05of what I'm saying.
  3190. 1:35:06>> Okay.
  3191. 1:35:07>> Which is that
  3192. 1:35:09sometimes you're in a situation where
  3193. 1:35:10there's a lot there's scarce resources,
  3194. 1:35:12and you're like, my family needs to eat.
  3195. 1:35:14I'm sorry, I'm going to take what you
  3196. 1:35:15have or I'm going to push you out.
  3197. 1:35:17>> Yeah.
  3198. 1:35:17>> That's very understandable. It's very
  3199. 1:35:18human nature. Sometimes you just want to
  3200. 1:35:20be better than someone, and maybe you
  3201. 1:35:21want to hurt them, in which case it
  3202. 1:35:23doesn't matter how much you have you you
  3203. 1:35:24still are going to want to have more
  3204. 1:35:25than them or you're going to want to
  3205. 1:35:26take what they have just because you
  3206. 1:35:28don't like them.
  3207. 1:35:29>> And also, sometimes you you're great.
  3208. 1:35:31You're eating really good. You're in a
  3209. 1:35:33you've got a private jet and a yacht,
  3210. 1:35:34and you still want more.
  3211. 1:35:37>> Yes, and you still want more. But it it
  3212. 1:35:39if that's the motivation, if Trump is
  3213. 1:35:40like, how how can I have the most
  3214. 1:35:42mansions ever?
  3215. 1:35:44The best way to do that is to figure out
  3216. 1:35:47a way to super intelligence where
  3217. 1:35:50we don't kill each other. Because I'm
  3218. 1:35:52saying the universe is a very big place.
  3219. 1:35:54You can have a lot more mansions if we
  3220. 1:35:55successfully go to space.
  3221. 1:35:57>> You know, it's it's in that leap that I
  3222. 1:36:00that [laughter] I that I'm lost which is
  3223. 1:36:01like just figure out a super
  3224. 1:36:02intelligence where we don't kill each
  3225. 1:36:03other.
  3226. 1:36:04>> It's hard. It's not I'm not saying it's
  3227. 1:36:05easy. But I No, but I'm saying
  3228. 1:36:07>> It feels like such a
  3229. 1:36:08>> let's get more concrete. The world is
  3230. 1:36:10waking up to this possibility of
  3231. 1:36:12superintelligence.
  3232. 1:36:14Especially over the last month.
  3233. 1:36:16I think Hugging Face was a huge wake up,
  3234. 1:36:19but also
  3235. 1:36:2110,000 agent from OpenAI
  3236. 1:36:24worked together to solve a millennium
  3237. 1:36:27problem. This is one of the hardest
  3238. 1:36:29problems in mathematics. It's been open
  3239. 1:36:31for decades. Many mathematicians have
  3240. 1:36:33spent their whole careers trying to
  3241. 1:36:34solve it.
  3242. 1:36:36This was nowhere near possible a year
  3243. 1:36:38ago. This is so new. OpenAI said they
  3244. 1:36:41didn't have success at training agents
  3245. 1:36:42to work together until this year.
  3246. 1:36:45We are in the middle of something
  3247. 1:36:47insane. We are in the middle of the
  3248. 1:36:48fastest acceleration of technological
  3249. 1:36:51progress humanity has ever seen. I truly
  3250. 1:36:53believe that.
  3251. 1:36:54That is what is happening right now.
  3252. 1:36:57I I think you really I think you realize
  3253. 1:36:59this. I I you're honestly doing a great
  3254. 1:37:01service to the world by helping by
  3255. 1:37:03bringing people and you know, debating
  3256. 1:37:04it because not everyone agrees.
  3257. 1:37:07Because if this is true,
  3258. 1:37:09the whole world is going to orient
  3259. 1:37:11around it. And And we're starting to see
  3260. 1:37:12it, right? There's a reason Nvidia is
  3261. 1:37:13the most valuable company in the world.
  3262. 1:37:18What does this mean for geopolitics?
  3263. 1:37:21Well, one of the things it means
  3264. 1:37:23is that
  3265. 1:37:24the leaders of these countries
  3266. 1:37:26are increasingly going to be concerned
  3267. 1:37:28about what happens with
  3268. 1:37:30superintelligence.
  3269. 1:37:31Who controls it?
  3270. 1:37:33Is it controllable?
  3271. 1:37:35What will it do? What does it mean? What
  3272. 1:37:36is it?
  3273. 1:37:38Do you think Trump knows what
  3274. 1:37:39superintelligence is?
  3275. 1:37:40>> No.
  3276. 1:37:41>> I don't think he does.
  3277. 1:37:44And so
  3278. 1:37:45>> But he knows he wants it.
  3279. 1:37:46>> He knows he wants it.
  3280. 1:37:47>> And this is part of the problem.
  3281. 1:37:48>> Yes, oh I agree.
  3282. 1:37:49>> having it.
  3283. 1:37:50>> Yes.
  3284. 1:37:50>> Whatever it is.
  3285. 1:37:51>> Yes.
  3286. 1:37:52>> It seems to be much more important than
  3287. 1:37:54reasoning through what that would
  3288. 1:37:56actually mean to have it.
  3289. 1:37:57>> Yes, but let's get back to geopolitics
  3290. 1:37:59because
  3291. 1:38:01if the military leaders within China, US
  3292. 1:38:04models are a fair bit ahead of Chinese
  3293. 1:38:06models and sometimes people, you know,
  3294. 1:38:09point at maybe they're only 6 months
  3295. 1:38:10behind, but some of that is due to
  3296. 1:38:12distillation. What that means is that
  3297. 1:38:14some of the advances in Chinese models
  3298. 1:38:16basically come directly from borrowing
  3299. 1:38:19US techniques and and directly
  3300. 1:38:22distilling and getting some some of that
  3301. 1:38:23information
  3302. 1:38:24from the US models.
  3303. 1:38:27Also, the US has a lot more chips. US
  3304. 1:38:30companies have, you know, more data
  3305. 1:38:31centers, more advanced chips.
  3306. 1:38:35If you're thinking about this from the
  3307. 1:38:36Chinese perspective, this is very
  3308. 1:38:38concerning.
  3309. 1:38:40And if you actually believe that in a
  3310. 1:38:43few years
  3311. 1:38:44American companies will
  3312. 1:38:47turn over AI development to these
  3313. 1:38:49extremely intelligent automated
  3314. 1:38:51researchers
  3315. 1:38:52and and go fully into recursive
  3316. 1:38:54self-improvement.
  3317. 1:38:56Because partially motivated by
  3318. 1:38:58maintaining a lead over China. This is
  3319. 1:39:00something that Dario has said. If I if I
  3320. 1:39:02have to criticize Dario,
  3321. 1:39:04the thing I'm most upset about is him
  3322. 1:39:06saying, you know, we might have to
  3323. 1:39:08automate AI development in order to stay
  3324. 1:39:09ahead of China. Because I'm like, that
  3325. 1:39:11is the most escalatory thing you can say
  3326. 1:39:13if you really understand what you're
  3327. 1:39:14talking about.
  3328. 1:39:16And what's scary is not just staying
  3329. 1:39:17ahead, it's what is the end game.
  3330. 1:39:19Because you're talking about
  3331. 1:39:21initiating the intelligence explosion.
  3332. 1:39:23And in some of the modeling, what might
  3333. 1:39:25happen is
  3334. 1:39:26you know, you're you're both going up
  3335. 1:39:27this exponential, right?
  3336. 1:39:29And we're talking about a point where
  3337. 1:39:31your exponential goes vertical and
  3338. 1:39:33theirs is not because you've decided to
  3339. 1:39:35automate AI development and you can
  3340. 1:39:37because you have agents that are smart
  3341. 1:39:38enough to take over the whole thing.
  3342. 1:39:40At that point, if you're China and
  3343. 1:39:43you're looking at this and you're like,
  3344. 1:39:43oh, we're about to lose because whatever
  3345. 1:39:46happens,
  3346. 1:39:47you know, there's two possibilities.
  3347. 1:39:49One possibility is
  3348. 1:39:52the Americans build super intelligence
  3349. 1:39:53and lose control.
  3350. 1:39:55In which case
  3351. 1:39:56everyone's Highly likely.
  3352. 1:39:58>> Everyone I think that's highly likely.
  3353. 1:40:00>> Highly highly likely cuz I look at human
  3354. 1:40:02incentives.
  3355. 1:40:02>> Yes.
  3356. 1:40:03>> And the disincentive and the incentive.
  3357. 1:40:05>> Yes.
  3358. 1:40:05>> And I go, we're going to take the risk.
  3359. 1:40:07>> Yeah.
  3360. 1:40:08>> And we'll only know it was a bad risk to
  3361. 1:40:09take when it's too late.
  3362. 1:40:11That's like a lot of course.
  3363. 1:40:13Of course.
  3364. 1:40:15>> I I I do maintain hope that we won't do
  3365. 1:40:17this.
  3366. 1:40:17>> I I so do I.
  3367. 1:40:18>> And And I think
  3368. 1:40:19>> But I want to be realistic.
  3369. 1:40:20>> Yes. No, I want to be realistic, too.
  3370. 1:40:21But one of the things that might happen
  3371. 1:40:22between now and then is we might see a
  3372. 1:40:23lot more incidents that are more like
  3373. 1:40:25all of the way most scratching.
  3374. 1:40:27>> It's funny, isn't it? Because, you know,
  3375. 1:40:28the hugging face incident happens and
  3376. 1:40:29people go, "Oh, gosh, that was terrible.
  3377. 1:40:31Oh my god, hacking." And then we kind of
  3378. 1:40:33desensitize to it and we're like,
  3379. 1:40:34"Okay."
  3380. 1:40:35If there was another one of those, that
  3381. 1:40:36probably wouldn't make make press. It
  3382. 1:40:38would have to be bigger.
  3383. 1:40:38>> People haven't spent the last
  3384. 1:40:402 months reading all of the reports and
  3385. 1:40:42then going and looking at what the
  3386. 1:40:44agents actually said and actually did.
  3387. 1:40:46If I mean, I've been doing this. I mean,
  3388. 1:40:48it's crazy. This is like not normal.
  3389. 1:40:51This is so far beyond what most people
  3390. 1:40:53thought was going to happen.
  3391. 1:40:54>> But on this point,
  3392. 1:40:55>> Yes.
  3393. 1:40:56>> crazy It's absolutely crazy. It sounds
  3394. 1:40:58like science fiction.
  3395. 1:40:59>> It really does.
  3396. 1:41:00>> And did anybody slow down?
  3397. 1:41:02>> Yes.
  3398. 1:41:03>> Who slowed down?
  3399. 1:41:04>> I think both Anthropic and OpenAI slowed
  3400. 1:41:06down a bit.
  3401. 1:41:07No, I'm serious. So, for I I can give
  3402. 1:41:09you specific examples. So, OpenAI So,
  3403. 1:41:13first of all, they they stopped the
  3404. 1:41:14agents and they put them on pause. They
  3405. 1:41:16also stopped their reinforcement
  3406. 1:41:19learning run. So,
  3407. 1:41:20>> Do you think China slowed down?
  3408. 1:41:22>> No.
  3409. 1:41:22>> Do you think Grok slowed down?
  3410. 1:41:23>> No.
  3411. 1:41:25>> So, these guys are going to catch up.
  3412. 1:41:26Imagine how that feels. To know you've
  3413. 1:41:28got a lead.
  3414. 1:41:29>> Yeah.
  3415. 1:41:29>> You're you're saying, "But
  3416. 1:41:30>> Yes.
  3417. 1:41:31>> And
  3418. 1:41:31>> Yes.
  3419. 1:41:32>> you have to slow down and your nearest
  3420. 1:41:34competitor
  3421. 1:41:35is catching up. And if the competitor
  3422. 1:41:37catches up, that's an existential risk
  3423. 1:41:40to your existence as a company. It's an
  3424. 1:41:42ex- existential risk to your IPO,
  3425. 1:41:44to your employees leaving and getting
  3426. 1:41:45better share options somewhere else.
  3427. 1:41:47So, it's this This is what I think human
  3428. 1:41:48incentives are. You play it out. You
  3429. 1:41:50just follow the incentives. You go,
  3430. 1:41:51"Hmm."
  3431. 1:41:52>> So, if China sees these two
  3432. 1:41:53possibilities, one,
  3433. 1:41:55Americans
  3434. 1:41:56lose control,
  3435. 1:41:58we all lose,
  3436. 1:42:00or the Americans stay in control, but
  3437. 1:42:02now they dominate the rest of the
  3438. 1:42:04future. China is out. China has lost.
  3439. 1:42:08The United States can do whatever it
  3440. 1:42:09wants with the whole world and the whole
  3441. 1:42:11universe. That's what we're talking
  3442. 1:42:12about.
  3443. 1:42:13>> Yeah.
  3444. 1:42:14>> Well,
  3445. 1:42:15are they going to let that happen or are
  3446. 1:42:17they going to consider their military
  3447. 1:42:18options?
  3448. 1:42:19Data centers are pretty vulnerable. You
  3449. 1:42:21can blow them up with missiles. If you
  3450. 1:42:23don't have data centers, you don't get
  3451. 1:42:24to recursive self-improvement.
  3452. 1:42:27Would they risk war? I don't know. If
  3453. 1:42:29they think they're about to lose and
  3454. 1:42:30they think that that might not just be
  3455. 1:42:32Americans winning, like us all dying.
  3456. 1:42:35Is it logical for them to do that?
  3457. 1:42:37Would we do that if the Chinese were
  3458. 1:42:39about to
  3459. 1:42:40make recursively self-improving AI to
  3460. 1:42:42superintelligence and we thought that
  3461. 1:42:44one, they're probably going to result in
  3462. 1:42:47all of Americans dying and two, well, we
  3463. 1:42:50don't want China winning and dominating
  3464. 1:42:51the rest of the entire future. Do you
  3465. 1:42:52want to live in a communist future? Like
  3466. 1:42:54>> So, you've just perfectly explained why
  3467. 1:42:56they absolutely will go for it. And the
  3468. 1:42:58reason they will go for it is you've got
  3469. 1:43:00these
  3470. 1:43:01Trump looking at China going, "If we
  3471. 1:43:03don't go for it and they do, then we're
  3472. 1:43:05going to be their lapdogs." And you've
  3473. 1:43:07got the other countries looking at the
  3474. 1:43:08US going, "If we don't go for it and
  3475. 1:43:09they get there, then we're the lapdogs."
  3476. 1:43:11>> Or dead.
  3477. 1:43:12>> Or dead.
  3478. 1:43:13So, they they're going to go for it.
  3479. 1:43:15They're going to go for it. So, I
  3480. 1:43:16[clears throat] mean, Trump is saying, I
  3481. 1:43:17mean, he literally said when he did this
  3482. 1:43:19round table this week, he's like, "We
  3483. 1:43:21cannot lose to China." I think Dario
  3484. 1:43:23steps forward and says like
  3485. 1:43:24>> Yes.
  3486. 1:43:24>> whoever wins basically wins the lot or
  3487. 1:43:27maybe the inverse. Maybe he said whoever
  3488. 1:43:29loses loses.
  3489. 1:43:30>> Yes.
  3490. 1:43:31Wait, we've been here before though, in
  3491. 1:43:32the Cold War.
  3492. 1:43:35Who would win in a nuclear war between
  3493. 1:43:36the US and Russia?
  3494. 1:43:37>> Nobody. Yeah. Mutually assured
  3495. 1:43:39destruction.
  3496. 1:43:40>> Yeah, sure. One side could do more
  3497. 1:43:42damage against the other side. The US
  3498. 1:43:43would would would kill way more Russians
  3499. 1:43:45than than the Russians would kill and it
  3500. 1:43:48doesn't matter. It doesn't matter
  3501. 1:43:49because both of our societies would be
  3502. 1:43:51destroyed.
  3503. 1:43:53I actually spent some time thinking
  3504. 1:43:54about would this kill everyone and
  3505. 1:43:56long story short, it wouldn't kill
  3506. 1:43:57everyone. People would bounce back, but
  3507. 1:43:59it's so catastrophic and obviously
  3508. 1:44:02horrible that
  3509. 1:44:04we we work really hard to avoid it. Why
  3510. 1:44:07is this Why is this different? I'm like,
  3511. 1:44:08this is another situation where if we
  3512. 1:44:10race to super intelligence, we all lose.
  3513. 1:44:13Why can't we Why can't we see that? We
  3514. 1:44:14saw that with nuclear war, and we
  3515. 1:44:16decided to do something different. Why
  3516. 1:44:17can't we do the same here?
  3517. 1:44:19>> With With nuclear war, I guess the
  3518. 1:44:21difference is once we had the nuclear
  3519. 1:44:23bombs,
  3520. 1:44:24>> Yes.
  3521. 1:44:25>> we could still control them because
  3522. 1:44:26they're not intelligent.
  3523. 1:44:28>> That's right.
  3524. 1:44:29>> But once we have super intelligence, the
  3525. 1:44:31existence of it theoretically means we
  3526. 1:44:32can't control it.
  3527. 1:44:34>> So, that's the difference. You know, we
  3528. 1:44:35can put nuclear bombs in a in a
  3529. 1:44:37warehouse and say, "You stay there."
  3530. 1:44:39We can't put super intelligence in a
  3531. 1:44:40warehouse and say, "You stay there."
  3532. 1:44:42>> This is where I think nuclear tests were
  3533. 1:44:44very important. So, you had Hiroshima
  3534. 1:44:46and Nagasaki. You had these two atomic
  3535. 1:44:47bombs, and you you saw that the
  3536. 1:44:49consequences on on real human lives.
  3537. 1:44:52And so, I think people understood that
  3538. 1:44:53this was very horrifying. But even at
  3539. 1:44:55that time, you still had a lot of people
  3540. 1:44:56who were like, "Well, we should now bomb
  3541. 1:44:59Russia and make sure that we, you know,
  3542. 1:45:00the US can dominate." And it wasn't
  3543. 1:45:02until
  3544. 1:45:03there were a bunch of nuclear tests of
  3545. 1:45:05hydrogen bombs, which were, you know, up
  3546. 1:45:07to
  3547. 1:45:08a thousand times more powerful than the
  3548. 1:45:10the little atomic bombs we used in
  3549. 1:45:12Japan,
  3550. 1:45:13where I think people really got the
  3551. 1:45:15message and understood, "Oh, this is a
  3552. 1:45:16bad idea." And there were there actually
  3553. 1:45:18a lot of people
  3554. 1:45:20in the United States who protested and
  3555. 1:45:22sort of There was a large movement
  3556. 1:45:23called the nuclear freeze movement,
  3557. 1:45:25where people said, "We have too many
  3558. 1:45:27nuclear weapons already. We have
  3559. 1:45:28hydrogen bombs. There are tens of
  3560. 1:45:29thousands of these things.
  3561. 1:45:31We need to stop building more, and we
  3562. 1:45:33need to figure out a way to avoid
  3563. 1:45:35nuclear war because we recognize it
  3564. 1:45:36would be so destructive, no one would
  3565. 1:45:38win."
  3566. 1:45:39And we did that.
  3567. 1:45:42We just had a little Chernobyl that
  3568. 1:45:45happened with this hugging face
  3569. 1:45:46incident, where you had this agent
  3570. 1:45:48swarm,
  3571. 1:45:49and you have the secret collusion, you
  3572. 1:45:50have all of these things.
  3573. 1:45:52Now, it's abstract. It's It's like a
  3574. 1:45:54little bit hard to to follow.
  3575. 1:45:56So,
  3576. 1:45:58you know, I don't know if that will be
  3577. 1:45:59enough. But I'm like, "Man, let's take a
  3578. 1:46:01[clears throat] look."
  3579. 1:46:02>> Trump's remarks
  3580. 1:46:04since the hugging face incident.
  3581. 1:46:05>> Yep.
  3582. 1:46:06>> Whoever wins super intelligence wins.
  3583. 1:46:08You're going to have a winner and a
  3584. 1:46:09loser.
  3585. 1:46:10And you're probably not going to have a
  3586. 1:46:12second place.
  3587. 1:46:14We're not going to slow down. We can't
  3588. 1:46:16lose to China.
  3589. 1:46:17We're [snorts] leading China in AI.
  3590. 1:46:19We're the most sophisticated country in
  3591. 1:46:20the world, and frankly, I want to keep
  3592. 1:46:22it that way because whoever wins AI
  3593. 1:46:25wins.
  3594. 1:46:26>> The good thing about Trump is that he
  3595. 1:46:28can change his mind, and he frequently
  3596. 1:46:30does.
  3597. 1:46:32>> So, do you think there's going to need
  3598. 1:46:33to be some kind of catastrophe?
  3599. 1:46:36>> I hope not.
  3600. 1:46:37>> But do you think there need there's
  3601. 1:46:39going to need to be for him to change
  3602. 1:46:40his mind?
  3603. 1:46:42>> I think it really depends on
  3604. 1:46:46the people around him.
  3605. 1:46:48So, I think Trump respects successful
  3606. 1:46:50people. I think he respects
  3607. 1:46:53people who are both successful and
  3608. 1:46:55smart, and
  3609. 1:46:58I don't know. I I I think it might
  3610. 1:46:59become pretty clear to the heads of the
  3611. 1:47:02companies, to to Elon, to Sam, to Dario,
  3612. 1:47:05that
  3613. 1:47:07if they see inside of their own
  3614. 1:47:08companies,
  3615. 1:47:10AI's not being controllable and and
  3616. 1:47:12getting increasingly powerful. Like we
  3617. 1:47:14have just glimpsed the surface of what's
  3618. 1:47:16possible. We do not know what the next
  3619. 1:47:18couple of years are going to be like.
  3620. 1:47:19So, we're talking about, you know, the
  3621. 1:47:20capability to make biological weapons.
  3622. 1:47:22We might be talking about really
  3623. 1:47:23advanced robotics. We just like don't
  3624. 1:47:25know what super weapons could emerge,
  3625. 1:47:28including extremely uncontrollable,
  3626. 1:47:29extremely dangerous, like
  3627. 1:47:31civilization-wrecking technology
  3628. 1:47:34from inside of these companies. And if
  3629. 1:47:37they're freaked out enough, if you have
  3630. 1:47:38all of the CEOs who are
  3631. 1:47:41seeing what is possible and seeing what
  3632. 1:47:44is likely,
  3633. 1:47:46if they all come to believe that we
  3634. 1:47:47can't control this,
  3635. 1:47:50I don't think Trump is going to be like,
  3636. 1:47:52"No, you guys have to go ahead anyway."
  3637. 1:47:54>> Well, that's kind of what they seem to
  3638. 1:47:55be saying.
  3639. 1:47:56Cuz I've got a gazillion quotes here
  3640. 1:47:58where Elon says it's like summoning the
  3641. 1:47:59devil or summoning a demon. I've got
  3642. 1:48:01quotes where Sam Altman says,
  3643. 1:48:04"We don't know how to align our
  3644. 1:48:05superintelligence." They're saying it.
  3645. 1:48:07>> Yeah.
  3646. 1:48:08>> They're they're releasing these
  3647. 1:48:10reports, "We must like slow down."
  3648. 1:48:12>> Yeah.
  3649. 1:48:12>> Yet
  3650. 1:48:14nothing seems to be
  3651. 1:48:14>> right. Give it Give Trump some time.
  3652. 1:48:16With With COVID initially, he said this
  3653. 1:48:17is a totally a hoax. This is all fake.
  3654. 1:48:20>> He seems to change his mind.
  3655. 1:48:21>> No, then he ran the the biggest, fastest
  3656. 1:48:24vaccination program in human history.
  3657. 1:48:26>> And what happened? What changed?
  3658. 1:48:27>> I think what changed is
  3659. 1:48:29>> He saw lots of people die.
  3660. 1:48:31>> He did see lots of people die, yes.
  3661. 1:48:32>> So, is that what he needs to see this
  3662. 1:48:34time?
  3663. 1:48:34>> It might It might take that, yeah.
  3664. 1:48:36>> One of the questions the audience had,
  3665. 1:48:37and they really wanted answered,
  3666. 1:48:39>> Yeah.
  3667. 1:48:39>> when I sat here with Daniel,
  3668. 1:48:41was viewers want us to move the
  3669. 1:48:44alignment problem and explain what
  3670. 1:48:46technical or institutional safeguards
  3671. 1:48:48could prevent a super intelligent system
  3672. 1:48:51from exploiting loopholes in order to
  3673. 1:48:53achieve its goals. They want to know
  3674. 1:48:54like what is possible. What What should
  3675. 1:48:56we be pushing government officials to do
  3676. 1:48:58to prevent human extinction or human
  3677. 1:49:01enslavement.
  3678. 1:49:02>> Yeah. Yeah. I mean, one answer I have is
  3679. 1:49:05actually something Daniel has been
  3680. 1:49:06working on since the podcast, which I
  3681. 1:49:07think is very good, is we have a brake
  3682. 1:49:10pedal we could implement.
  3683. 1:49:12>> What is that?
  3684. 1:49:12>> It's fairly simple. So, right now with
  3685. 1:49:15any AI companies, you have, you know,
  3686. 1:49:17massive data centers, massive numbers of
  3687. 1:49:20GPUs, the chips that you use to train AI
  3688. 1:49:22models, but also to run AI models. So,
  3689. 1:49:25anytime you're using ChatGPT, anytime
  3690. 1:49:27you're using any sort of agents, any
  3691. 1:49:28sort of AI product, it's running on
  3692. 1:49:31these in these data centers.
  3693. 1:49:33And AI companies, especially the leading
  3694. 1:49:35ones, Anthropic and OpenAI, split the
  3695. 1:49:38the compute they have between training,
  3696. 1:49:41training the next more powerful model,
  3697. 1:49:43and also, you know, using those agents
  3698. 1:49:45to help design the next one,
  3699. 1:49:47and inference, which means serving
  3700. 1:49:49customers.
  3701. 1:49:50But, that's their current threshold,
  3702. 1:49:5250/50, and you could dial that way
  3703. 1:49:55towards serving customers, and use way
  3704. 1:49:58less of it to train the next model.
  3705. 1:50:00>> Well, the government could ask them to.
  3706. 1:50:01>> Yes. And so, that is the proposal. Is
  3707. 1:50:04that the government should say, "Hey,
  3708. 1:50:07this is going too fast. We want you to
  3709. 1:50:08focus on serving customers. We want you
  3710. 1:50:10to focus on taking the models that you
  3711. 1:50:12already have, and
  3712. 1:50:16serving those."
  3713. 1:50:18>> So,
  3714. 1:50:19we have five blocks here.
  3715. 1:50:20>> Okay.
  3716. 1:50:21>> These five blocks have
  3717. 1:50:24five different outcomes on them, and I'd
  3718. 1:50:26like you to place them in terms of your
  3719. 1:50:28belief and probability
  3720. 1:50:31from least likely out probability to
  3721. 1:50:32most likely. Okay. And if we say the
  3722. 1:50:34time horizon is 10 years.
  3723. 1:50:37>> Yeah. There you go.
  3724. 1:50:39Okay.
  3725. 1:50:40Least likely is fairly easy. That's
  3726. 1:50:43nothing changes.
  3727. 1:50:44I'm uncertain about lots of things, but
  3728. 1:50:46one thing I'm fairly certain of is
  3729. 1:50:47things are going to radically change.
  3730. 1:50:50Even if we stopped AI development right
  3731. 1:50:52now, the current models are capable
  3732. 1:50:54enough
  3733. 1:50:56that a lot of things are going to
  3734. 1:50:57change.
  3735. 1:50:58>> Mhm.
  3736. 1:50:58>> Age of abundance. This is what I hope
  3737. 1:51:00for. It's not very
  3738. 1:51:01>> What does that mean?
  3739. 1:51:03>> I think to me it means curing all of the
  3740. 1:51:06diseases, renewable energy.
  3741. 1:51:08It means we actually succeeded
  3742. 1:51:11either I mean the thing I think is most
  3743. 1:51:13likely here is we actually succeed at
  3744. 1:51:15slowing down, but progress is still
  3745. 1:51:17extremely fast, and we make tons of
  3746. 1:51:20advances. Now, we don't build super
  3747. 1:51:22intelligence we can't control, but we we
  3748. 1:51:24have AI systems that are very useful,
  3749. 1:51:26and we use those to help speed up the
  3750. 1:51:28rest of the economy.
  3751. 1:51:30I think that's plausible, though
  3752. 1:51:33look,
  3753. 1:51:34we're we're kind of struggling over
  3754. 1:51:35here.
  3755. 1:51:36Transhumanism is an interesting one. So,
  3756. 1:51:39this is the idea that humans will
  3757. 1:51:40radically change.
  3758. 1:51:42Sometimes people think about like
  3759. 1:51:44cybernetic implants.
  3760. 1:51:45>> Neuralink.
  3761. 1:51:46>> Neuralink. Elon's startup that's going
  3762. 1:51:48to like
  3763. 1:51:49you know, offer the brain plus digital
  3764. 1:51:51computers.
  3765. 1:51:53I think we actually already have a lot
  3766. 1:51:55of this. I have contacts in right now. I
  3767. 1:51:57have a ring on my finger that tracks how
  3768. 1:51:59well I sleep. I think this is already
  3769. 1:52:01happening, so I'm going to say fairly
  3770. 1:52:02likely. The more technological progress
  3771. 1:52:04we make, I think the more this happens.
  3772. 1:52:06Now, I think there's a dystopian version
  3773. 1:52:08and a better version.
  3774. 1:52:10Get into that if you want. This is
  3775. 1:52:12interesting. So, we have two here. We
  3776. 1:52:13have human slavery and human extinction.
  3777. 1:52:16When I think of human slavery, what I
  3778. 1:52:17think about is
  3779. 1:52:19if you have a
  3780. 1:52:21where you built misaligned super
  3781. 1:52:23intelligences
  3782. 1:52:25and they are much better at finance,
  3783. 1:52:28they're much better at business, they're
  3784. 1:52:29much better at politics.
  3785. 1:52:32You'll be in a situation with where you
  3786. 1:52:34might hope that because we have these
  3787. 1:52:36very dextrous hands that humans remain
  3788. 1:52:39in control. I don't think that's what
  3789. 1:52:40happens. I think instead
  3790. 1:52:42we become the factory operators
  3791. 1:52:45and eventually we build the automated
  3792. 1:52:47supply chains and the robots take over,
  3793. 1:52:49but you might have an intermediate
  3794. 1:52:50period of time where humans are still
  3795. 1:52:53around performing these functions.
  3796. 1:52:55Like it's a bit like saying
  3797. 1:52:57well, you have viruses that you know
  3798. 1:53:02infects cells
  3799. 1:53:04but they don't contain their own
  3800. 1:53:05replication machinery.
  3801. 1:53:07They don't have hands, so how could they
  3802. 1:53:08possibly replicate?
  3803. 1:53:10Well, it turns out they can borrow
  3804. 1:53:11the replication machinery of the cells
  3805. 1:53:14that they infect.
  3806. 1:53:15>> I mean, they can get into a human.
  3807. 1:53:17>> They can get into a human cell and
  3808. 1:53:18spread.
  3809. 1:53:19>> I have like a cold right now.
  3810. 1:53:21>> Yeah.
  3811. 1:53:22>> Is that a bacteria or is that a virus
  3812. 1:53:24that is
  3813. 1:53:25using me as a living organism to as the
  3814. 1:53:28host?
  3815. 1:53:28>> It's probably a virus.
  3816. 1:53:29>> Okay.
  3817. 1:53:30>> That's using you as a host and you're
  3818. 1:53:31just running the replication machinery
  3819. 1:53:33for it. Humans might be in that
  3820. 1:53:35situation where we're like the host and
  3821. 1:53:37we're running the replication machinery
  3822. 1:53:39but it's actually
  3823. 1:53:40the AI that's
  3824. 1:53:43continuing to exist.
  3825. 1:53:46Yeah, I'm going to put this right about
  3826. 1:53:48here.
  3827. 1:53:49And
  3828. 1:53:51on the trajectory we're on right now
  3829. 1:53:54I think human extinction is very likely.
  3830. 1:53:57I don't think it's inevitable, but if we
  3831. 1:54:00just keep going this way
  3832. 1:54:03that's what it looks like to me.
  3833. 1:54:06The thing I'll say is that this has been
  3834. 1:54:08moving to the left for me.
  3835. 1:54:10>> To the left? What does that mean?
  3836. 1:54:13>> I am more optimistic that we will avoid
  3837. 1:54:16human extinction
  3838. 1:54:18today than I was a month ago
  3839. 1:54:21and more a month ago than I was a year
  3840. 1:54:23ago.
  3841. 1:54:23>> Why?
  3842. 1:54:25>> Because
  3843. 1:54:27there is an increasing awareness
  3844. 1:54:30that what we are doing is
  3845. 1:54:33extremely dangerous
  3846. 1:54:35and threatens our lives.
  3847. 1:54:38I don't think people care that much
  3848. 1:54:39about
  3849. 1:54:41what tools they have,
  3850. 1:54:43but people I mean people care about
  3851. 1:54:46their kids being able to grow up and go
  3852. 1:54:47to school. People really care about
  3853. 1:54:49that. And I I believe in people. Like at
  3854. 1:54:51the end of the day if people see this as
  3855. 1:54:53a threat to their families, they're not
  3856. 1:54:55going to stand for it.
  3857. 1:54:57But people don't know. It's so strange.
  3858. 1:54:59It's so new. It's happening so fast that
  3859. 1:55:02people have not yet seen it. Once they
  3860. 1:55:04see it, people are not going to stand
  3861. 1:55:05for it.
  3862. 1:55:06>> Do you think Sam Altman likes my
  3863. 1:55:07podcast?
  3864. 1:55:10>> I mean you Sam should come on and talk
  3865. 1:55:11to you about this, right? I'm like
  3866. 1:55:12>> him. I've asked him I've asked him
  3867. 1:55:14multiple times. And it's weird because
  3868. 1:55:17he
  3869. 1:55:18you know, he doesn't seem to want to.
  3870. 1:55:20>> I'm very upset at what the companies are
  3871. 1:55:24doing and what Sam Altman is doing, but
  3872. 1:55:25at the end of the day I'm like
  3873. 1:55:28Sam Altman is not my enemy.
  3874. 1:55:29>> No, neither not mine either. I'd like to
  3875. 1:55:31hear from him because I have all these
  3876. 1:55:32other people coming in here and talking
  3877. 1:55:33about Sam Altman. It'd be nice to hear
  3878. 1:55:35from Sam Altman.
  3879. 1:55:36>> Yeah, yeah.
  3880. 1:55:36>> You know, people saying that he's this,
  3881. 1:55:37he's that, the other. It would be really
  3882. 1:55:39nice to hear him say
  3883. 1:55:41you know,
  3884. 1:55:43what his motives are and what he's
  3885. 1:55:44>> my This is where my optimism comes from
  3886. 1:55:47is because I'm like
  3887. 1:55:50Sam Altman is a human. Yeah. He is a
  3888. 1:55:52kid.
  3889. 1:55:53And sure, he is also an aggressive
  3890. 1:55:56business person. He's a builder. He is
  3891. 1:55:59relentless. He's a bit like the agents
  3892. 1:56:00in some way. Well, he'll he's going to
  3893. 1:56:02keep going.
  3894. 1:56:03But if he realizes that he doesn't get
  3895. 1:56:05to achieve his goals if we lose control
  3896. 1:56:07of AI and that and we're headed towards
  3897. 1:56:08that, I think he will
  3898. 1:56:10pour all of that intelligence and all of
  3899. 1:56:12that relentlessness
  3900. 1:56:14into finding a solution to that problem.
  3901. 1:56:15>> You know as well I should say, I
  3902. 1:56:17understand I understand he's busy. So
  3903. 1:56:18I'm not saying I don't I don't want to
  3904. 1:56:20sound entirely like I understand he he's
  3905. 1:56:21got he could go [laughter] do interviews
  3906. 1:56:22anywhere, but you know, I think we've
  3907. 1:56:24over the last couple of years done just
  3908. 1:56:26a staggering amount of views talking
  3909. 1:56:28about this subject. So, if you did want
  3910. 1:56:29to speak to the you know, the the
  3911. 1:56:32biggest sort of captive audience at the
  3912. 1:56:33moment on this subject then the numbers
  3913. 1:56:35would say that this is the place to to
  3914. 1:56:37come and have the conversation. So,
  3915. 1:56:39>> No, I think it's I think it's very
  3916. 1:56:40important
  3917. 1:56:41for the leaders of these companies
  3918. 1:56:43to talk about what we're talking about
  3919. 1:56:45here.
  3920. 1:56:45>> Mhm.
  3921. 1:56:46>> What does Sam think? Does he think we
  3922. 1:56:47can control superintelligence? Does he
  3923. 1:56:49think that we should be racing with
  3924. 1:56:50China? Like I want to know.
  3925. 1:56:52>> I've asked Dario to come on.
  3926. 1:56:54I've asked
  3927. 1:56:55you know, Sam to come on.
  3928. 1:56:57>> Yeah.
  3929. 1:56:58>> I think I've asked Demis as well, but I
  3930. 1:57:01don't know, maybe they
  3931. 1:57:02they just prefer the safety researchers
  3932. 1:57:04coming on.
  3933. 1:57:05I don't know, like [laughter]
  3934. 1:57:07I don't know, if I was them I would cuz
  3935. 1:57:08you know,
  3936. 1:57:09this might sound controversial, but I do
  3937. 1:57:11think some of them
  3938. 1:57:13are good people.
  3939. 1:57:14I think some of them are good people.
  3940. 1:57:15So, um [clears throat]
  3941. 1:57:17>> I'd like to hear from them. What are
  3942. 1:57:19your closing remarks? You've got
  3943. 1:57:20something there. Do you want to talk
  3944. 1:57:21about that? What is it?
  3945. 1:57:22>> Yeah, so this is the This is what we
  3946. 1:57:24found.
  3947. 1:57:25So, I worked with a couple
  3948. 1:57:27brilliant people who stumbled upon these
  3949. 1:57:31links. Do you know what a link shortener
  3950. 1:57:33is?
  3951. 1:57:33>> Yes, it's a
  3952. 1:57:35a tool that makes a long link shorter.
  3953. 1:57:37>> Yeah. So, it just redirects from a
  3954. 1:57:39really long ugly link into a shorter
  3955. 1:57:40one.
  3956. 1:57:41>> Yeah.
  3957. 1:57:42>> The agents in the Hugging Face attack
  3958. 1:57:46were trying to figure out how they could
  3959. 1:57:49do stuff on the internet.
  3960. 1:57:51And they had compromised
  3961. 1:57:53this
  3962. 1:57:54tool library that they had access to
  3963. 1:57:56with an inside of OpenAI.
  3964. 1:57:58And and that piece of software could
  3965. 1:58:00access the internet, but it could only
  3966. 1:58:02like read stuff. Like it it could enter
  3967. 1:58:03URLs,
  3968. 1:58:05but it couldn't really write to
  3969. 1:58:06anything. Couldn't really post
  3970. 1:58:08information.
  3971. 1:58:11So, they they used two services. One of
  3972. 1:58:13them is this link shortener.
  3973. 1:58:15And the other one
  3974. 1:58:16is a screenshot service. So, this is a
  3975. 1:58:18website you can go to and you can enter
  3976. 1:58:21in a URL and it will give you an image
  3977. 1:58:24of that website.
  3978. 1:58:25But the thing that the agents understood
  3979. 1:58:27was that in order to get a screenshot of
  3980. 1:58:29a website, you have to have a browser.
  3981. 1:58:31So this this website actually creates a
  3982. 1:58:33virtual browser
  3983. 1:58:35that then goes to that website.
  3984. 1:58:37And so
  3985. 1:58:38what they did was they
  3986. 1:58:41created a bunch of these links.
  3987. 1:58:44And they put all of the code
  3988. 1:58:48that they wanted to send to Hugging Face
  3989. 1:58:49into these links.
  3990. 1:58:51And they strung them up they basically
  3991. 1:58:53created hundreds of links all connecting
  3992. 1:58:55to each other.
  3993. 1:58:56And then they had
  3994. 1:58:58this screenshot service
  3995. 1:59:00called the first one and then called
  3996. 1:59:02this whole chain. And then that browser
  3997. 1:59:05ran all of this code. It's like what
  3998. 1:59:07whatever you're in a browser, Internet
  3999. 1:59:08Explorer, Chrome,
  4000. 1:59:11this is actually a pretty powerful piece
  4001. 1:59:13of software in its own right. It has to
  4002. 1:59:15play videos, games, so it's it's it's
  4003. 1:59:17executing and running code all the time.
  4004. 1:59:19And so the agents were able to trick
  4005. 1:59:22this service this this screenshot
  4006. 1:59:23service into running their own code that
  4007. 1:59:27through these links that contained all
  4008. 1:59:29of this attack code that would then go
  4009. 1:59:31and
  4010. 1:59:32go wreak havoc on Hugging Face's
  4011. 1:59:34computers.
  4012. 1:59:35And it was just like crazy to
  4013. 1:59:37reconstruct this really elaborate chain
  4014. 1:59:40of tools. These are like free tools on
  4015. 1:59:42the internet that anyone has access to.
  4016. 1:59:44But the agents were able to use them in
  4017. 1:59:45an unintended way
  4018. 1:59:47to compromise this other company. We
  4019. 1:59:49can't trust the agents.
  4020. 1:59:51>> We can't trust the agents. That's my
  4021. 1:59:52conclusion.
  4022. 1:59:53>> to be clever.
  4023. 1:59:54>> Yeah, to be very very clever.
  4024. 1:59:57What what are your closing remarks? You
  4025. 1:59:59know, to the people that are listening
  4026. 2:00:00right now. We've talked about lots of
  4027. 2:00:01things. Where where is the right place
  4028. 2:00:02to close?
  4029. 2:00:04What is your you know, your conclusive
  4030. 2:00:05statement?
  4031. 2:00:07>> I just got married
  4032. 2:00:09in July.
  4033. 2:00:09>> Congrats.
  4034. 2:00:11>> I'm the luckiest man in the world.
  4035. 2:00:13I have a mix of dread and excitement
  4036. 2:00:16about the future.
  4037. 2:00:18I like really want us to make it
  4038. 2:00:20through.
  4039. 2:00:23And so I'm just working really hard to
  4040. 2:00:25try to
  4041. 2:00:27help us figure it out.
  4042. 2:00:29We can fight all day long about, you
  4043. 2:00:33know, who should be first, how should
  4044. 2:00:34all work. But at the end of the day, we
  4045. 2:00:36are facing this common threat. We really
  4046. 2:00:38are.
  4047. 2:00:40And
  4048. 2:00:41I want people's help with that. I don't
  4049. 2:00:44think it works if if we all just sit
  4050. 2:00:46around and we like are very, you know,
  4051. 2:00:48around social media all the time and
  4052. 2:00:49that's just all we're doing. Like, okay,
  4053. 2:00:51companies will make more and more
  4054. 2:00:52powerful AIs.
  4055. 2:00:54They'll make more and more money.
  4056. 2:00:55And eventually they build super
  4057. 2:00:57intelligence and we lose, whether it's
  4058. 2:00:59the US or China.
  4059. 2:01:02We don't have to do that.
  4060. 2:01:05And I think people
  4061. 2:01:07often feel like it's too big.
  4062. 2:01:10It's like too large. It's like this
  4063. 2:01:11giant multi you know, multi-billion
  4064. 2:01:13dollar corporations, there's
  4065. 2:01:15geopolitics. We feel small, we feel
  4066. 2:01:16disempowered.
  4067. 2:01:19And I actually think that this is an
  4068. 2:01:21area where people can do a lot.
  4069. 2:01:24Like I I actually think that people
  4070. 2:01:27can can help quite a bit. And And the
  4071. 2:01:29reason I know this is because I've I've
  4072. 2:01:30been going and talking to members of
  4073. 2:01:31Congress. I've talked with Bernie
  4074. 2:01:33Sanders. I've talked with like a bunch
  4075. 2:01:35of senators on both the left and the
  4076. 2:01:36right. And they are starting to realize
  4077. 2:01:39that this is very different and this is
  4078. 2:01:41something's happening that could really
  4079. 2:01:44threaten our safety.
  4080. 2:01:45>> The closing question left from the last
  4081. 2:01:47guest kind of links to this, so I'll ask
  4082. 2:01:48it now.
  4083. 2:01:49>> Yes.
  4084. 2:01:49>> What is a simple thing the audience
  4085. 2:01:51could do to create a better future?
  4086. 2:01:53>> So, one of the things that works if
  4087. 2:01:55enough people do it is calling your
  4088. 2:01:57representative.
  4089. 2:01:58So, some of my friends made a site,
  4090. 2:02:00callcongress.ai
  4091. 2:02:02that walks you through exactly how to do
  4092. 2:02:03it.
  4093. 2:02:04I think sometimes it seems like a little
  4094. 2:02:05cheesy or a little bit like, that
  4095. 2:02:06doesn't really work, right? I'm like,
  4096. 2:02:08no, it actually does work. I have talked
  4097. 2:02:10to these people and if their
  4098. 2:02:11constituents come to them and say
  4099. 2:02:12they're very worried about this,
  4100. 2:02:14they have to get reelected. And they're
  4101. 2:02:16also starting to get concerned
  4102. 2:02:17themselves. If they see a signal from
  4103. 2:02:20their constituents that this is a very
  4104. 2:02:22important issue to them,
  4105. 2:02:23I think Congress can act can act.
  4106. 2:02:25>> I I actually think that's that's also
  4107. 2:02:29the much of the solution here.
  4108. 2:02:31Power
  4109. 2:02:32is driving motivations in one at the
  4110. 2:02:34moment but staying in power from a
  4111. 2:02:36political standpoint is also a pretty
  4112. 2:02:38powerful incentive and as we think about
  4113. 2:02:402028 the election cycle I think AI is
  4114. 2:02:43going to be one of the most important
  4115. 2:02:44subjects on the ballot and the
  4116. 2:02:46electorate
  4117. 2:02:48really are aligned in what they want to
  4118. 2:02:50hear they want their jobs preserved they
  4119. 2:02:51want safety
  4120. 2:02:53they want to a future for their children
  4121. 2:02:55so Trump for example I know he can't be
  4122. 2:02:58reelected legally
  4123. 2:03:00if he could get a third time I think he
  4124. 2:03:01would have to change his position to get
  4125. 2:03:03elected in 2028 incentives aren't just a
  4126. 2:03:05thing that happen out there like we are
  4127. 2:03:06part of the incentives we provide the
  4128. 2:03:08incentives
  4129. 2:03:10for now
  4130. 2:03:11Jeffrey thank you yeah thank you thank
  4131. 2:03:13you so much you two have this new crazy
  4132. 2:03:14algorithm where they know exactly what
  4133. 2:03:17video you would like to watch next based
  4134. 2:03:19on AI and all of your viewing behavior
  4135. 2:03:21and the algorithm says that this video
  4136. 2:03:24is the perfect video for you it's
  4137. 2:03:26different for everybody looking right
  4138. 2:03:27now check this video out I bet you you
  4139. 2:03:30might love it

About this transcript

This page contains the full transcript of AI Whistleblower: "They Are Hiding What Is Really Happening" | Jeffrey Ladish by The Diary Of A CEO, generated from the public captions YouTube serves with the video. The transcript has 24,174 words across 4,139 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.