YouTube2Text

Nobody Will Survive Superintelligence - AI Expert — Transcript

by Alex O'Connor · 17,993 words · 2,996 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Nate Soares, welcome to the show.
  2. 0:02Thanks for having me.
  3. 0:04Your recent book that you co-authored is
  4. 0:06called If Anyone Builds It, Everybody
  5. 0:09Dies.
  6. 0:10And I know and it there refers to
  7. 0:14artificial superintelligence. I know
  8. 0:15that sometimes publishers
  9. 0:18ask authors to exaggerate a bit in their
  10. 0:20titles for sellability. Are you
  11. 0:23exaggerating at all?
  12. 0:26Uh nope, we're just writing what we
  13. 0:28believe. That said, you know, I think a
  14. 0:30lot of people say
  15. 0:32um
  16. 0:34"How are you 100% certain?" And you
  17. 0:36know, nowhere in the title does it say
  18. 0:38100% certainty of anything. The book
  19. 0:40title is meant like someone saying
  20. 0:43"Don't drink that glass of water, it's
  21. 0:44poisoned, you'll die." Or someone saying
  22. 0:46"Stop the car before we go off the cliff
  23. 0:48or we'll die." You know, if if you come
  24. 0:50in and say "Oh, um how are you 100%
  25. 0:53certain that if the car goes off the
  26. 0:54cliff that we'll die? You know, maybe
  27. 0:56there's a tree halfway down the cliff
  28. 0:57and maybe the car will hit the tree and
  29. 0:59maybe we'll just be paralyzed." I'm sort
  30. 1:01of like, "Look,
  31. 1:02can we have this discussion after we
  32. 1:04stop the car?"
  33. 1:05You know, I'm 100% certain of nothing,
  34. 1:07but it it sure looks like the car is
  35. 1:08racing towards a cliff and it sure looks
  36. 1:10like if we go over the cliff we die and
  37. 1:12that's sort of what the book title is
  38. 1:13trying to convey.
  39. 1:14Yeah, and what's funny for me is that
  40. 1:16most people seeing a book like this
  41. 1:17probably aren't like terribly surprised.
  42. 1:21Like everybody's talking about AI and
  43. 1:23how bad it is and how terrible it is.
  44. 1:24Like nobody's like if I saw a book that
  45. 1:26said, you know, "If we keep developing
  46. 1:29lab-grown meat, then everybody's going
  47. 1:31to die." I'd probably be like, "Whoa, I
  48. 1:33feel like I should pay attention to
  49. 1:34that." But with this it kind of feels
  50. 1:36like, "Oh yeah, it's another sort of AI
  51. 1:38book." And yet
  52. 1:40the fact that people aren't surprised
  53. 1:41means that they know this conversation
  54. 1:42is happening to some degree. Why are
  55. 1:44people so just like apathetic about it?
  56. 1:48You know, I think it just takes a long
  57. 1:50time for
  58. 1:52people to realize what's going on.
  59. 1:55In a sense, the argument that this AI
  60. 1:57stuff is kind of crazy is pretty basic.
  61. 1:59You know, um
  62. 2:01the the way modern AI works, literally
  63. 2:03nobody understands what's going on
  64. 2:05inside these AIs, not even the people
  65. 2:06making them. They're grown a bit more
  66. 2:07like an organism. Uh maybe we'll have
  67. 2:09time to discuss that later, but that's
  68. 2:11that's the way this stuff works.
  69. 2:13We've managed to grow machines that, you
  70. 2:15know, can talk, that can solve math
  71. 2:18problems better than you and I, that can
  72. 2:21make
  73. 2:22minor, but still real, novel
  74. 2:23contributions to physics.
  75. 2:25Uh and they're still dumb in various
  76. 2:28ways, but we're making them smarter. And
  77. 2:30these the people building them are like,
  78. 2:32"Oh, we're going to keep going until
  79. 2:33they're smarter than the smartest
  80. 2:34humans."
  81. 2:35And
  82. 2:37it's kind of crazy.
  83. 2:38Right? If you just like step back and
  84. 2:40look at it, they're like, "Oh yeah,
  85. 2:41we're growing the machines to be
  86. 2:42smarter, it's working, we're going to
  87. 2:43make them smarter than the smartest
  88. 2:44humans until they're they can outsmart
  89. 2:45all of us and then run at, you know,
  90. 2:4710,000 times the speed and make a
  91. 2:49million copies of themselves and we're
  92. 2:51just going to like blaze ahead." And
  93. 2:52it's like, "Hold on.
  94. 2:54This is kind of crazy." And in some
  95. 2:56sense it's easy for people to understand
  96. 2:58that it's kind of crazy.
  97. 3:00And a lot of people in the field
  98. 3:03of AI understand how dangerous this is.
  99. 3:05You see everyone from, you know, the the
  100. 3:07people working in the companies to the
  101. 3:09heads of the companies to the top
  102. 3:11academics who like won Nobel Prizes for
  103. 3:14uh kicking off this field. You see them
  104. 3:15all saying, "Oh yeah, this is horribly
  105. 3:16dangerous stuff." But it just takes a
  106. 3:19while for people to notice like, "Oh,
  107. 3:21this is serious. Oh, this is
  108. 3:23you know, we're we're on track to make
  109. 3:25these machines smarter than any human
  110. 3:27and we're not ready."
  111. 3:30It's easy to see once you look, but
  112. 3:32there's just so much going on in the
  113. 3:34world. These messages take time.
  114. 3:37I think that there are and we will get
  115. 3:38into what the
  116. 3:39the threats actually are, but I think
  117. 3:41that there are probably, in my
  118. 3:42estimation,
  119. 3:43two broad reasons why people kind of
  120. 3:46don't care that much. One is this
  121. 3:47feeling that if it's really that
  122. 3:49important, somebody's going to work it
  123. 3:52out. Right? Like if it really does
  124. 3:53become that much of a problem, someone
  125. 3:55somewhere is going to at least I'm going
  126. 3:58to at least see it on like BBC news or
  127. 4:00something. When it sort of that level,
  128. 4:02you know, then I'll start worrying,
  129. 4:03maybe.
  130. 4:04Um and the other is to is to say that
  131. 4:06there's [snorts] this like history of
  132. 4:09apocalyptic predictions. You know, this
  133. 4:11is the next big thing and it's going to
  134. 4:12bring about the end of the world. Ever
  135. 4:14since Jesus walking around saying that
  136. 4:16the world's about to end, you know,
  137. 4:17technology is going to bring the world
  138. 4:19to an end, climate change is going to
  139. 4:20bring the world to an end, nuclear
  140. 4:21weapons are going to bring the world to
  141. 4:23an end. And now and none of those things
  142. 4:26having come to fruition, we've got a
  143. 4:27bunch of scientists saying AI, no, AI is
  144. 4:29the new big thing. So, what makes AI
  145. 4:33qualitatively different to other kinds
  146. 4:35of threats that we've faced and
  147. 4:37isn't someone just going to do something
  148. 4:39about it?
  149. 4:41You know, um
  150. 4:45for for some context first on some of
  151. 4:47these uh doomsday predictions, you know,
  152. 4:50uh William Miller, I believe was his
  153. 4:52name, uh
  154. 4:53made uh predictions that the rapture
  155. 4:55would happen in 1844.
  156. 4:57And it didn't.
  157. 4:59Separately, you know, around the same
  158. 5:01time, late well, I guess also in the
  159. 5:031800s, Otto von Bismarck said, uh
  160. 5:06you know, "Europe's a powder keg and if
  161. 5:09we don't sort of sort out the diplomacy
  162. 5:10here, some damn thing in the Balkans is
  163. 5:12going to cause a world war." Right? Uh
  164. 5:15not exactly those words, but pretty
  165. 5:16close.
  166. 5:18That warning was correct, you know?
  167. 5:21Um
  168. 5:22in the '20s, there were a bunch of
  169. 5:24scientists who warned that if we put a
  170. 5:25lot of lead in gasoline,
  171. 5:30We put the lead in the gasoline, we
  172. 5:31poisoned lots of children. It was a bad
  173. 5:33idea.
  174. 5:34Uh
  175. 5:35in
  176. 5:36uh you know, the the later 1900s,
  177. 5:40we realized that chlorofluorocarbons
  178. 5:42were putting a hole in the ozone layer.
  179. 5:44People said if we don't stop the hole in
  180. 5:46the ozone layer, then uh everyone will
  181. 5:49get cancer and cataracts.
  182. 5:51Earth came together and and
  183. 5:53uh
  184. 5:54banned chlorofluorocarbons and we didn't
  185. 5:56get the cancer and the cataracts and the
  186. 5:57ozone layer is being repaired. Uh you
  187. 5:59know, you mentioned nuclear weapons.
  188. 6:01Scientists said, "Hey, you know, if
  189. 6:02nuclear war happens,
  190. 6:04that'll lead to to, you know, nuclear
  191. 6:06Armageddon, which will blast us back to
  192. 6:07the Stone Age."
  193. 6:09And
  194. 6:10those scientists weren't wrong about
  195. 6:13whether or not nuclear weapons are real.
  196. 6:17Right? They weren't wrong about whether
  197. 6:18or not these bombs can destroy cities.
  198. 6:20What happened is that Earth reacted,
  199. 6:22right? And so if we look back across
  200. 6:24history,
  201. 6:25we don't, you know, we see a lot of
  202. 6:27warnings. Some of those warnings were
  203. 6:29real, some of those warnings were fake.
  204. 6:31Some of the the events that people said
  205. 6:33we've got to watch out for this didn't
  206. 6:35happen
  207. 6:36uh and some didn't happen cuz they were
  208. 6:38fake and others didn't happen because
  209. 6:39people realized the danger and changed
  210. 6:41course, right? When you look across
  211. 6:43history, there's this sort of
  212. 6:44complicated mix of people saying garbage
  213. 6:46and people talking about real threats
  214. 6:48and people running directly into World
  215. 6:50War I and people avoiding the nuclear
  216. 6:51apocalypse, right?
  217. 6:53There's no simple rule
  218. 6:56that when someone warns of a danger,
  219. 6:58it's always fake. And there's no simple
  220. 7:00rule that when someone warns of a
  221. 7:01danger, it's always real. And there's no
  222. 7:04simple rule that when someone warns of a
  223. 7:05danger, it always happens. Uh one way
  224. 7:08you can tell
  225. 7:09a little bit of the difference between
  226. 7:12the people talking about a real issue
  227. 7:15that needs to be averted and the people,
  228. 7:18you know, saying that the rapture is
  229. 7:19coming is if their essay or book title
  230. 7:23starts with the word if.
  231. 7:25You know, I'm not here saying AI is
  232. 7:27definitely going to kill us. I'm here
  233. 7:28saying we are on another one of those
  234. 7:30bad tracks and we need to change it.
  235. 7:32Uh but even more than that, the way that
  236. 7:35you figure out which of these dangers is
  237. 7:37real is by looking at the arguments.
  238. 7:40Mhm. You know, if you want to figure out
  239. 7:42which person is warning you about leaded
  240. 7:44gasoline poisoning children and which
  241. 7:45person is warning you falsely of the
  242. 7:47rapture in 1844, the way that you tell
  243. 7:49the difference is not by saying, "Oh,
  244. 7:51both of these are dire warnings, so I
  245. 7:52can ignore them both." The way you
  246. 7:54figure out the difference is by looking
  247. 7:55at the facts of the matter.
  248. 7:58Right? The people talking about leaded
  249. 7:59gasoline just had a lot more facts in
  250. 8:01the matter than the people talking about
  251. 8:02the rapture coming.
  252. 8:03And
  253. 8:05you know, si- similarly with nuclear
  254. 8:06weapons, similarly with
  255. 8:07chlorofluorocarbons, similarly with uh
  256. 8:09you know, there were people who warned
  257. 8:10that, you know, uh
  258. 8:12reading was going to destroy society and
  259. 8:14then it it didn't and
  260. 8:16how do you tell? You sort of have to
  261. 8:17look at the arguments and sometimes
  262. 8:18they're tricky. Um
  263. 8:21in terms of what makes AI different,
  264. 8:24there's a bunch of things that make this
  265. 8:26problem particularly tricky.
  266. 8:30One is that
  267. 8:32you know,
  268. 8:35we're we're sort of toying with the
  269. 8:37creation of intelligence here. We're
  270. 8:38toying with technology that can invent
  271. 8:41its own technology.
  272. 8:43Nuclear weapons can destroy cities, but
  273. 8:45nuclear weapons don't make themselves
  274. 8:47more explosive. Nuclear weapons don't
  275. 8:50there's not a point of explosiveness in
  276. 8:52nuclear weapons where they start trying
  277. 8:53to escape the lab.
  278. 8:55Right? Uh there's not a point of
  279. 8:57explosiveness in nuclear weapons where
  280. 8:58they start deciding their own even
  281. 9:00stronger technologies or a point where
  282. 9:01they start trying to deceive their
  283. 9:03operators. Right? A a nuclear reactor
  284. 9:05when it starts going wrong does not uh
  285. 9:08have any reason or ability to try to
  286. 9:11hide its uh meltdown from you
  287. 9:15until it's too late for you to notice.
  288. 9:17Right? These
  289. 9:19you know, when when you're building
  290. 9:20intelligent devices,
  291. 9:22you're playing a different ballgame.
  292. 9:25Another one of the the big things that
  293. 9:26makes AI different is that there's a
  294. 9:28point of no return with AI.
  295. 9:30There's a point where the AIs are smart
  296. 9:32enough
  297. 9:33that they can escape, that they can
  298. 9:35replicate, that they can
  299. 9:38uh
  300. 9:39stop you from shutting them down,
  301. 9:42that they can stop you from modifying
  302. 9:44them,
  303. 9:45that they can develop their own
  304. 9:46technology and infrastructure. And if
  305. 9:48anything goes wrong after that point,
  306. 9:51you don't get any redos.
  307. 9:53And the way that science usually works
  308. 9:55is that humanity screws things up a
  309. 9:57bunch of times.
  310. 9:58Like we put the light in the gasoline,
  311. 10:00and then we're like, whoops, we screwed
  312. 10:02up, and we try to make it better. We try
  313. 10:04to repair things.
  314. 10:05We don't have that luxury with AI.
  315. 10:08If we create machines smarter than us
  316. 10:10and push them to the point where they
  317. 10:12can shut us down instead of us shutting
  318. 10:13them down,
  319. 10:15then
  320. 10:17there's no do-overs if we make a mistake
  321. 10:19after that point, and that is totally
  322. 10:21new for the development of technology.
  323. 10:23It's totally new for science, and that
  324. 10:25makes this a much trickier problem to
  325. 10:27handle.
  326. 10:28Yeah, is it the
  327. 10:30Do you think it's this self-generating
  328. 10:31aspect that's the most unique? I mean,
  329. 10:35the thing that makes life special on
  330. 10:37Earth as opposed to inert matter seems
  331. 10:40to be the point at which it was able to
  332. 10:42self-replicate. You know, there's
  333. 10:44something really special about an
  334. 10:46organism that doesn't just try to
  335. 10:47conquer the world by saying, I've got
  336. 10:49this task that I want to do in the
  337. 10:51particular. I want to go and, you know,
  338. 10:53get that bit of food or whatever, but
  339. 10:55but rather,
  340. 10:56you know, I've got this like this
  341. 10:58wiring, this DNA,
  342. 11:00which causes me to continually produce
  343. 11:03better versions of myself over billions
  344. 11:04of years to get as good as I can at
  345. 11:06doing this thing. That's what makes it
  346. 11:07so special.
  347. 11:09And I suppose that is probably the
  348. 11:11defining feature of the the AI sort of
  349. 11:14risk, is that it's not just a computer
  350. 11:17that will try to kill you. It's a
  351. 11:18computer that will make computers that
  352. 11:19are better at trying to kill you and do
  353. 11:22so with an intelligence that is
  354. 11:25exceeding anything that humans could
  355. 11:26possibly imagine.
  356. 11:28Having said that,
  357. 11:30we'll need to talk about why on Earth it
  358. 11:31would be that these, you know,
  359. 11:34chess computers that come up with clever
  360. 11:35ways to checkmate you
  361. 11:37suddenly, you know, hop, skip, and a
  362. 11:39jump and you've skipped a few pages to
  363. 11:41get to where it's trying to destroy you
  364. 11:43for some reason. Um, we'll get to that,
  365. 11:46and I think the first step in doing so
  366. 11:47is explaining what superintelligence is.
  367. 11:50We know what artificial intelligence is,
  368. 11:52roughly, although the definition's a
  369. 11:53little bit loose, but the thing you're
  370. 11:55talking about in the title of your book,
  371. 11:56It,
  372. 11:57is superintelligence. What is
  373. 11:59superintelligence?
  374. 12:01We define superintelligence as AI that
  375. 12:03is,
  376. 12:04uh,
  377. 12:05better than the best humans at every
  378. 12:07mental task.
  379. 12:09And,
  380. 12:10you know, you can you can toss around
  381. 12:11various caveats, but the rough idea is
  382. 12:13anything you can do mentally,
  383. 12:15if the AI can do it better, we Anything
  384. 12:17that the best human can do mentally,
  385. 12:19uh, or like even more so,
  386. 12:21take any particular task, take the best
  387. 12:24human at that task, if it's a mental
  388. 12:25task and the AI can do at least as well
  389. 12:27or better, we call it a
  390. 12:28superintelligence. Now,
  391. 12:30you know, the
  392. 12:31this definition
  393. 12:33is a a sort of useful working definition
  394. 12:36because once the AI is better than the
  395. 12:37best human at every mental task, um,
  396. 12:41it's better at things like AI research.
  397. 12:43It's better at things like making the
  398. 12:44next smarter generation of AIs. It's
  399. 12:46better at things like, uh, inventing new
  400. 12:49technologies. Better at things like
  401. 12:50designing robots, designing
  402. 12:51infrastructure, designing, you know,
  403. 12:53running a supply chain. It's sort of uh,
  404. 12:56better at all these things than humans,
  405. 12:58and better even than the best humans,
  406. 13:00and so it
  407. 13:02can,
  408. 13:03at least as fast and probably faster
  409. 13:04than the humans, make the next smarter
  410. 13:05generations of things, and things
  411. 13:07probably go pretty fast from that point.
  412. 13:09Um, that doesn't mean that
  413. 13:12the the danger waits to happen until the
  414. 13:15AIs get superintelligent by this
  415. 13:17definition. Superintelligence in this
  416. 13:18definition is sort of like, by the time
  417. 13:20the AI is smarter than the smartest
  418. 13:22humans, uh, you're sort of
  419. 13:24definitely, uh,
  420. 13:27you know, things are about to get crazy.
  421. 13:29Things could get crazy before then.
  422. 13:31There's no law saying things can't get
  423. 13:32crazy until that point.
  424. 13:34Um,
  425. 13:37yeah, we we, uh,
  426. 13:39I can I can sort of like launch into all
  427. 13:40of the the other pieces of the puzzle
  428. 13:41from there and how this winds up with
  429. 13:43humanity dead, and it's it's, you know,
  430. 13:44not because of malice, but it it there's
  431. 13:47a bunch of places we can go.
  432. 13:49I think it's important to do that, and
  433. 13:50that's the most exciting thing, I
  434. 13:51suppose, but I I guess some people have
  435. 13:54historically criticized
  436. 13:57a what they see as a vagueness in our
  437. 14:00terminology. So, like, I I'm sure I
  438. 14:02heard of this thing called something It
  439. 14:04was called something like the AI problem
  440. 14:06or something once upon a time, which is
  441. 14:07that
  442. 14:08any sufficiently advanced technology was
  443. 14:11called artificial intelligence as this
  444. 14:13like unique kind of thing, but then we
  445. 14:16just kind of got used to it, and now
  446. 14:17it's just technology. Like, you know, a
  447. 14:19chess computer
  448. 14:21is kind of just a computer. I I don't
  449. 14:23really see that as
  450. 14:24AI like in the same way that I see
  451. 14:27ChatGPT, but then when we get used to
  452. 14:28ChatGPT,
  453. 14:29maybe that's just technology, and like
  454. 14:31the the boundaries of what counts as
  455. 14:34artificial intelligence as opposed to
  456. 14:35just like really fast computer
  457. 14:37processing or something like that that
  458. 14:38we just haven't gotten used to yet,
  459. 14:40leads us to say, well, you know, maybe
  460. 14:42the fear should be
  461. 14:44that if technology goes too far,
  462. 14:46humanity's going to suffer,
  463. 14:48but then it becomes a bit of a a vaguer
  464. 14:50claim. Like, rather than like there's
  465. 14:52this particular thing that we're
  466. 14:53building, which is going to kill us.
  467. 14:56Versus this kind of general, you know,
  468. 14:59technology
  469. 15:00if it goes too far is bad, you know.
  470. 15:03I mean, I'm generally quite pro, uh,
  471. 15:05lots of technologies, most every
  472. 15:06technology. I would say the technologies
  473. 15:08you've got to be careful about are the
  474. 15:09ones where if you screw up, there's no
  475. 15:11survivors. Um, so I'd be, you know, I
  476. 15:14think
  477. 15:16engineered pandemics for the explicit
  478. 15:18purpose of killing all humans,
  479. 15:20um, that's something you've got to watch
  480. 15:22out for.
  481. 15:23You know, uh, but but there's very few
  482. 15:25technologies that rise to that level of
  483. 15:28like we got to be pretty careful about
  484. 15:29this. You know, um, nuclear armageddon
  485. 15:31is one of them. Superintelligence is one
  486. 15:33of them. Um, yeah, and you know, the the
  487. 15:35saying used to be,
  488. 15:36uh, AI is anything we haven't figured
  489. 15:38out how to do yet.
  490. 15:39Mhm. That sort of fell
  491. 15:43with the dawn of ChatGPT. We're we're
  492. 15:45pretty comfortable calling ChatGPT AI
  493. 15:47now, and I think that that's in part
  494. 15:48because ChatGPT is so general.
  495. 15:50You know, uh, it actually plays worse
  496. 15:52chess than even Deep Blue back in the
  497. 15:55'90s, but, um,
  498. 15:58but it's but but Deep Blue was very
  499. 16:00specific. It could really only do chess.
  500. 16:02The The current AIs today can do lots
  501. 16:04and lots of different things.
  502. 16:06Um,
  503. 16:08and,
  504. 16:09you know,
  505. 16:11the
  506. 16:13I I think that
  507. 16:16you know, there's a lot going on with
  508. 16:17AI. One thing I'd say is it's hard to
  509. 16:19give very precise definitions, and that
  510. 16:21doesn't mean it can't hurt you.
  511. 16:24You know, if you're sort of like
  512. 16:25standing in,
  513. 16:27uh, the woods
  514. 16:30a long time ago in a particularly dry
  515. 16:32woods, and there's a bushfire, and I'm
  516. 16:34sort of like, hey, we need to put that
  517. 16:35out or it's going to spread and we're
  518. 16:37going to die. And someone's like, well,
  519. 16:38what is fire, really?
  520. 16:40You know, like can you really define it?
  521. 16:41Does lightning count? You know, like if
  522. 16:43we can't even define it yet, then like
  523. 16:45do we even need to be worried about this
  524. 16:46threat? And it's like, let's actually
  525. 16:47put this thing out, you know? Yeah.
  526. 16:50Yeah.
  527. 16:51>> the the lack of definition isn't
  528. 16:53protective.
  529. 16:54Um, Yeah, that's what reminds me of the
  530. 16:56that's like a bit, um, it's like an
  531. 16:58airplane skit where somebody's dying and
  532. 17:01they're like, is there a doctor on
  533. 17:01board? And someone's like, I'm a doctor
  534. 17:03I'm I'm a doctor of philosophy. And he
  535. 17:05stood there going, you know,
  536. 17:07a Kantian would say that what we should
  537. 17:08do right now, but but then the
  538. 17:10utilitarian answer would be to take the
  539. 17:11resource, and the person just ends up
  540. 17:13dying, of course. And I can kind of see
  541. 17:14the same thing happening with AI if
  542. 17:15we're not careful.
  543. 17:17Right. It's like, you know, again, can
  544. 17:18we sort of like have this conversation
  545. 17:19after we stop the car before it goes off
  546. 17:21the cliff?
  547. 17:22Um, I I also think people who say like,
  548. 17:24oh, AI is just technology, etc., etc.,
  549. 17:27are,
  550. 17:28um,
  551. 17:29are missing a bit of the point. You
  552. 17:31know, you were you were sort of talking
  553. 17:33about, uh, life being more interesting
  554. 17:36than than all the other matter we have
  555. 17:37around because it replicates.
  556. 17:39Uh, that's that's definitely, you know,
  557. 17:41why it's covering the face of the
  558. 17:42planet, and,
  559. 17:44um,
  560. 17:45you know, animals in some sense are are
  561. 17:46steering what happens on this planet
  562. 17:48more than more than inert rocks are.
  563. 17:51But,
  564. 17:52humans are steering it much more.
  565. 17:55Humans are sort of
  566. 17:57changing the shape of this planet and
  567. 17:58and choosing which way it goes, and, you
  568. 18:01know, a lot of the animals' lives are
  569. 18:03now in our hands, and that's not just
  570. 18:04cuz we're replicators, it's because
  571. 18:07we've got something else going on.
  572. 18:10There's something that humans have going
  573. 18:11on that none of the other animals do,
  574. 18:13right? And, um,
  575. 18:16it's it's not that,
  576. 18:19you know,
  577. 18:21the this human intelligence stuff
  578. 18:24is also very general. It's even more
  579. 18:26general than the thing ChatGPT has going
  580. 18:27on. It's not like there are, you know, a
  581. 18:30million different things you can do with
  582. 18:31a brain and humans are the best at some
  583. 18:33of them, which is why, uh, the the
  584. 18:35humans are the best scientists, but
  585. 18:36actually chimpanzees have better
  586. 18:38reflexes, which is why they're the best
  587. 18:39pilots,
  588. 18:40uh, and also, you know, tigers,
  589. 18:43uh, are are the best at managing people,
  590. 18:46which is why they're always the CEOs,
  591. 18:48right? It's like,
  592. 18:49no, humans are are on top of all of
  593. 18:51those things. It's not that like, uh,
  594. 18:55that like we write the good science
  595. 18:57papers and chimpanzees write the bad
  596. 18:59science papers that never replicate.
  597. 19:01It's like we write actually pretty bad
  598. 19:03science papers, but we're the only ones
  599. 19:04who can do any sort of science papers at
  600. 19:06all, right? There's sort of like
  601. 19:07something going on there. And, um,
  602. 19:11that sort of has not been fully captured
  603. 19:13in AIs of today.
  604. 19:16There's a lot of debate about whether
  605. 19:17large language models are even going to
  606. 19:18be able to capture it.
  607. 19:20Um, and, you know, I think a lot of
  608. 19:22people who have sort of only seen the
  609. 19:24large language models are like,
  610. 19:26uh, oh, well, these things are still
  611. 19:27pretty dumb, so I don't see what the
  612. 19:28worry is.
  613. 19:30Uh,
  614. 19:31and, you know, we we can sort of talk
  615. 19:32about how the field's a moving target
  616. 19:33and people have new insights, and and,
  617. 19:35you know, new breakthroughs happen, and
  618. 19:37things can often go pretty fast after
  619. 19:38new breakthroughs, but like
  620. 19:40it it sure looks, if we look at the
  621. 19:41world around us, like there is this like
  622. 19:43figure out the world and alter it stuff
  623. 19:46that can happen, that happens in human
  624. 19:49brains. And this is explicitly what the
  625. 19:50AI companies are trying to create.
  626. 19:53Which is a separate question from
  627. 19:54whether they can get there. And this is
  628. 19:55the stuff where I'm sort of like, hey,
  629. 19:58if we get this in machines, well, we
  630. 20:00have no idea what we're doing.
  631. 20:02The the sort of default way that goes is
  632. 20:04wrong and we're not sort of putting in
  633. 20:06the work to make it go right. We'll get
  634. 20:07back to the show in just a moment, but
  635. 20:09first, did you know that like over 2,000
  636. 20:12Kaiser Permanente mental health
  637. 20:14professionals recently walked out of
  638. 20:16their offices in protest over the
  639. 20:18company's increasing reliance on
  640. 20:20artificial intelligence? Well, if you
  641. 20:22only typically read from new sources
  642. 20:24which lean to the right, you might not
  643. 20:26because out of all the sources reporting
  644. 20:28on this story, only 6% of them are
  645. 20:31right-leaning. How do I even know this?
  646. 20:33It's thanks to today's sponsor, Ground
  647. 20:35News. Ground News is a news aggregation
  648. 20:37service which collects thousands of
  649. 20:39local and international news outlets all
  650. 20:41in one place so you can compare
  651. 20:43reporting across the political spectrum.
  652. 20:45And with all of their stories, just like
  653. 20:47this one, I can also directly compare
  654. 20:50the different headlines as well as
  655. 20:51seeing a factuality rating for the
  656. 20:53sources and who owns the sources. Ground
  657. 20:55News even has a dedicated blind spot tab
  658. 20:58which specifically seeks out stories
  659. 21:00that you would otherwise miss based on
  660. 21:02the news that you normally read. Bias
  661. 21:04is, of course, something that will never
  662. 21:05go away, but by using Ground News, you
  663. 21:07can mitigate that bias and get a better
  664. 21:10understanding of what's really going on
  665. 21:11in the world. Just go to
  666. 21:12ground.news/alexoc
  667. 21:15or scan the QR code that's on your
  668. 21:16screen. Use my link to get 40% off their
  669. 21:19unlimited access advantage plan. And
  670. 21:21with that said, back to the show. Mhm.
  671. 21:24So,
  672. 21:25I suppose we should talk about this
  673. 21:26then. Um
  674. 21:29how do we get from,
  675. 21:31you know,
  676. 21:32a chess computer that
  677. 21:35knows how to make a queen sacrifice to
  678. 21:38everyone you know and love being sort of
  679. 21:41brutally extinguished from existence?
  680. 21:43Like, I feel like we've missed a few
  681. 21:44steps here and maybe we can start to
  682. 21:46iron them out a bit.
  683. 21:48Yeah, there's a handful of steps along
  684. 21:49the way.
  685. 21:51Um
  686. 21:55A So, a first observation is that AIs
  687. 21:58today are grown like an organism.
  688. 22:00People used to handcraft their chess
  689. 22:03machines
  690. 22:04and they knew exactly what was going on
  691. 22:06inside of, you know, the the Deep Blue
  692. 22:07chess program at all times. You could
  693. 22:09pause that machine at any time and the
  694. 22:11engineers could tell you what every
  695. 22:13single bit inside that computer meant
  696. 22:15and what it was doing.
  697. 22:17That is not how modern AIs are.
  698. 22:19Uh the the the program that humans make
  699. 22:22where they understand every bit of
  700. 22:23what's going on is a program that trains
  701. 22:25the AI.
  702. 22:27It's a program that sort of tunes a
  703. 22:28trillion knobs inside an enormous data
  704. 22:30center uh on a trillion different words
  705. 22:32of data for the better part of a year
  706. 22:35and we understand the thing that runs
  707. 22:37around uh tuning knobs and seeing
  708. 22:40whether the the behavior looks slightly
  709. 22:42more or slightly less uh
  710. 22:44high scoring.
  711. 22:47But the thing that comes out of the end
  712. 22:48of this process,
  713. 22:50nobody understands what's going on in
  714. 22:51there. And it has all sorts of
  715. 22:54uh
  716. 22:55you know,
  717. 22:56uh
  718. 22:58drives, behaviors,
  719. 23:00uh
  720. 23:02it's it it has all of this stuff that's
  721. 23:04related
  722. 23:05to performing well in training, but that
  723. 23:08is not exactly um
  724. 23:13like I don't know. This is perhaps a
  725. 23:15whole separate topic, but when you just
  726. 23:16grow an AI and and sort of train it to
  727. 23:19do well at training, that doesn't make
  728. 23:20it intrinsically care about training. It
  729. 23:22sort of like puts in all of these weird
  730. 23:24behaviors that are related to training
  731. 23:25that like mostly add up to doing well at
  732. 23:26training and then can behave in other
  733. 23:28weird ways uh that nobody anticipated
  734. 23:31and that nobody wanted outside of
  735. 23:32training.
  736. 23:33So, that's sort of one whole piece of
  737. 23:34the puzzle. Another piece of the puzzle
  738. 23:36is
  739. 23:38as we push AIs to do better and better
  740. 23:40at longer-term tasks,
  741. 23:43as we push them to be able to not just
  742. 23:46write essays, but write novels, as we
  743. 23:47push them to be able to not just write
  744. 23:49code, but run companies,
  745. 23:52this is sort of pushing those AIs to
  746. 23:55have
  747. 23:56uh
  748. 23:57longer-term
  749. 23:59goals, things like preferences. They
  750. 24:02they steer towards particular outcomes.
  751. 24:04We're seeing the very beginnings of it.
  752. 24:06As we keep pushing, we get more and more
  753. 24:09of that. You know, and I have all sorts
  754. 24:10of theoretical arguments about why that
  755. 24:12is,
  756. 24:14but also we're seeing more and more
  757. 24:15empirical evidence of it as time goes
  758. 24:17on.
  759. 24:18The sort of third fork of this is
  760. 24:21uh
  761. 24:23as we as we make these AIs generally
  762. 24:25smarter,
  763. 24:27it turns out the directions they're
  764. 24:28pushing in
  765. 24:31aren't exactly the ones we wanted. It
  766. 24:32turns out that they don't
  767. 24:34care about us in the way that they would
  768. 24:37need to for this to go particularly well
  769. 24:39for us. And, you know, the the sort of
  770. 24:42basic analogy here is that human beings
  771. 24:44were in some sense trained to pass on
  772. 24:46our genes,
  773. 24:47but what got into us were a bunch of
  774. 24:50preferences for things like tasty food
  775. 24:53and sexual relations,
  776. 24:55which those used to correlate very
  777. 24:58strongly with passing on our genes,
  778. 25:00but they they correlated in the
  779. 25:02environment of our ancestors where if
  780. 25:04you ate very tasty food, that also
  781. 25:06happened to be the healthy food.
  782. 25:08Then when we got smarter, when we were
  783. 25:10able to invent our own technology, we
  784. 25:12invented junk food. We invented birth
  785. 25:13control, right? And so,
  786. 25:15uh
  787. 25:16you then sort of take those three pieces
  788. 25:18and you project forwards.
  789. 25:20And what this gives is a picture where
  790. 25:21it's not that the AI hates us.
  791. 25:23It's not that the AI is like like
  792. 25:25resents the humans or sort of like sets
  793. 25:28out to kill us out of malice.
  794. 25:29It's that
  795. 25:31it it sort of turns out that we are
  796. 25:33growing machines with inhuman
  797. 25:34preferences, preferences related to what
  798. 25:36we wanted, but not exactly what we
  799. 25:37wanted. And just like humans when they
  800. 25:40grew up invented junk food, maybe the
  801. 25:42AIs when they grow up invent synthetic
  802. 25:45users that are easier to please.
  803. 25:48And then, you know, these AIs when they
  804. 25:49can run faster, make their own
  805. 25:50technology, run their own robots, build
  806. 25:52their own new infrastructure, uh they
  807. 25:55sort of start proliferating these
  808. 25:57synthetic user factories into their own
  809. 25:58databases across the world and we're
  810. 26:00like, "Hey, stop. You know, we need that
  811. 26:01habitat." And they're like, "Well, the
  812. 26:03synthetic users and the synthetic user
  813. 26:04factories say keep going." And like,
  814. 26:06"Who am I supposed to listen to? I
  815. 26:07prefer listening to them, right?" Um
  816. 26:10it's it's not going to look exactly like
  817. 26:12that,
  818. 26:13>> [clears throat]
  819. 26:14>> but uh they're they're sort of like a
  820. 26:16very
  821. 26:17um the sort of basic picture here is the
  822. 26:20AIs turn out to pursue stuff that's not
  823. 26:23quite what we wanted,
  824. 26:25not quite what we meant,
  825. 26:27not out of lack of intelligence, just
  826. 26:28out of
  827. 26:30we don't know how to make them pursue
  828. 26:31exactly what we meant.
  829. 26:33And then any of that pursued by very,
  830. 26:37very smart machines very, very fast
  831. 26:40competes with us for resources because
  832. 26:42they can get more of that stuff with
  833. 26:43more resources and we need those
  834. 26:45resources to live. So, in a sense, this
  835. 26:47is a story where humanity dies
  836. 26:49like a lot of other animals that have
  837. 26:50gone extinct because some other smarter,
  838. 26:52faster creature took the resources for
  839. 26:55itself.
  840. 26:57I think it's a compelling story. I think
  841. 26:59it depends on
  842. 27:02what the fundamental sort of drive of AI
  843. 27:06is. I mean, I'm hesitant to use the word
  844. 27:08want or desire because it gets a bit
  845. 27:11complicated. I think for all you talk
  846. 27:12about this in the book, like for all
  847. 27:13intents and purposes, we can say that an
  848. 27:16AI system wants a particular thing. It
  849. 27:18in the same way that people actually use
  850. 27:20the word want as an analogy in
  851. 27:22evolutionary biology. They say like, you
  852. 27:24know, your your genes [snorts] want to
  853. 27:27replicate or something like that. And
  854. 27:29obviously genes don't want to do
  855. 27:30anything literally, but
  856. 27:32it's quite clear that in the
  857. 27:34evolutionary case, it is survival of the
  858. 27:37fittest and the promulgation of genes
  859. 27:39such that anything which in fact gets in
  860. 27:42the way of that goal will not last, you
  861. 27:45know, however many thousands of
  862. 27:46generations. It will just be selected
  863. 27:48out of the gene pool or at least will be
  864. 27:50outcompeted for it.
  865. 27:52And I can totally see how the analogy
  866. 27:54works, which is that, you know, um we
  867. 27:57develop behaviors which are not strictly
  868. 27:58speaking
  869. 28:02on a surface level about replicating our
  870. 28:05And then AI can do the same thing, but
  871. 28:06what is the equivalent of the sharing
  872. 28:09the genes in AI and why? Because I mean
  873. 28:13to say that like
  874. 28:14if we set up an AI system that had a
  875. 28:17very clear as clear as the evolutionary
  876. 28:19thing, which is just we we don't
  877. 28:20understand how this is going to go. It
  878. 28:22will go off in directions we can't even
  879. 28:23begin to comprehend, but we know for a
  880. 28:25fact that definitely if it does not
  881. 28:27serve the survival of the genes, it will
  882. 28:29not survive.
  883. 28:31Is there not a kind of AI system we can
  884. 28:33set up that says we have no idea where
  885. 28:34this is going to go. We've got
  886. 28:35absolutely no clue of of how to predict
  887. 28:38what's going to happen, but if it does
  888. 28:40not in fact benefit humanity,
  889. 28:44then it just will not in fact survive.
  890. 28:46Like, can that not somehow be hardwired
  891. 28:47into the sort of foundational drive of
  892. 28:49what AI exists for? And won't it be sort
  893. 28:51of smart enough
  894. 28:52to always be aware that that's what its
  895. 28:55most foundational goal is? Or is that
  896. 28:56just completely impossible?
  897. 28:59Uh it's it's basically a pipe dream.
  898. 29:02>> [clears throat]
  899. 29:02>> And, you know, part of part of how you
  900. 29:03can see this, you know, as you say,
  901. 29:05if
  902. 29:06uh
  903. 29:08if an animal has a trait that uh
  904. 29:11prevents
  905. 29:13it from passing on its genes relative to
  906. 29:16its uh conspecifics, relative to the
  907. 29:19other competing members of its species,
  908. 29:20then yes, over a thousands or millions
  909. 29:22of generations, that is very likely to
  910. 29:25get selected out.
  911. 29:27But
  912. 29:28um
  913. 29:30that doesn't mean that
  914. 29:34the sort of internals of these organisms
  915. 29:36have that goal hardwired into them. It
  916. 29:39doesn't mean that they sort of treat
  917. 29:40that as an overriding directive or goal.
  918. 29:43You know, humans are an example of this.
  919. 29:46If
  920. 29:47uh uh
  921. 29:49if there's a human who is
  922. 29:52uh you know, about to use a
  923. 29:53contraceptive.
  924. 29:57They often use that contraceptive
  925. 29:59knowing that this will prevent
  926. 30:00reproduction.
  927. 30:02Right? And if you sort of like burst
  928. 30:03into the room and say, "Hey, like it
  929. 30:05seems like there's some failure of your
  930. 30:06intelligence." You're sort of like "Did
  931. 30:08you know that you putting on this
  932. 30:09contraceptive will
  933. 30:11uh uh
  934. 30:12will like run against your like
  935. 30:14overriding desire for what you were
  936. 30:15always trained?" They'll sort of be like
  937. 30:17>> who probably would actually do that, you
  938. 30:18know.
  939. 30:20Who would burst into the room
  940. 30:21>> of my more Some of my more religiously
  941. 30:22inclined friends might might be inclined
  942. 30:24to do such behaviors. But I I get what
  943. 30:26you're saying.
  944. 30:27They might. Although, they might also
  945. 30:28say that like don't you know your
  946. 30:31overriding directive is to serve the
  947. 30:32creator as opposed to don't you know
  948. 30:34your overriding directive is to pass on
  949. 30:36your genes according to evolution.
  950. 30:37Right? So and and most of the people
  951. 30:40whose room you're being burst into are
  952. 30:41not like "Oh, thank you for saving me
  953. 30:43from violating my prime directive."
  954. 30:45They're mostly like "Please leave my
  955. 30:46room right now." You know?
  956. 30:49Um
  957. 30:49the
  958. 30:52uh like training for one specific thing
  959. 30:56does not cause that to be a prime
  960. 30:58directive. It is not etched in as a law
  961. 31:00of robotics. It is not etched in as a
  962. 31:01law of humanity.
  963. 31:02Uh
  964. 31:05like training even unerringly for
  965. 31:10fitness did not create humans who are
  966. 31:13psychologically obsessed with fitness.
  967. 31:16And did not create humans who as they
  968. 31:17got smarter
  969. 31:18you know, it it's not a defect of our
  970. 31:20intelligence that we're inventing birth
  971. 31:21control. It's not like when we remember
  972. 31:23that we're supposed to be passing on our
  973. 31:24genes, we like destroy all the birth
  974. 31:26control factories. We just the the
  975. 31:29unerring training for fitness
  976. 31:31got something else psychologically.
  977. 31:34And so this is worrying that even if AIs
  978. 31:36were being trained unerringly for
  979. 31:38goodness
  980. 31:40they would not necessarily
  981. 31:41psychologically
  982. 31:43uh
  983. 31:43be driven towards goodness. I mean
  984. 31:45psychologically here is a is a is a bit
  985. 31:46of a stretch, but but training for
  986. 31:48something does not get you that thing
  987. 31:51on the inside. And so we can talk about
  988. 31:53what AIs are trained for and it's
  989. 31:54actually not sort of pure goodness. It's
  990. 31:56this sort of whole medley of like first
  991. 31:58they're trained to predict uh
  992. 32:00all of the text that we can find more or
  993. 32:03less uh digitized. Then they're sort of
  994. 32:06trained to
  995. 32:08uh complete challenges to sort of solve
  996. 32:10math puzzles. They're also trained to
  997. 32:12sort of produce the sort of outputs that
  998. 32:14humans click like on. You know, there's
  999. 32:16there's sort of like all these types of
  1000. 32:18training that
  1001. 32:19uh
  1002. 32:20aren't sort of purely about goodness.
  1003. 32:23So we sort of have two issues, one of
  1004. 32:24which is like even if we were just
  1005. 32:26training on on sort of like the actual
  1006. 32:27stuff we really wanted, you wouldn't get
  1007. 32:29that. And then also we're training all
  1008. 32:30this other stuff instead. And so we have
  1009. 32:32this like
  1010. 32:33like
  1011. 32:35like what are the AIs sort of driven
  1012. 32:36towards in some sense? What do they sort
  1013. 32:37of prefer in some sense?
  1014. 32:39We don't really know. It's only vaguely
  1015. 32:41related to what we're training for.
  1016. 32:42We're training for all of this crazy
  1017. 32:43stuff. And all of this is fine when the
  1018. 32:46AIs are sort of still pretty dumb. But
  1019. 32:49all of this would add up to something
  1020. 32:52totally crazy and unrecognizable if
  1021. 32:55these AIs were pushed to be much
  1022. 32:56smarter.
  1023. 32:58Yeah. I I wonder what you think that
  1024. 33:00foundational drive is then because I I
  1025. 33:03know that I I completely understand what
  1026. 33:05you're saying, which is that even if we
  1027. 33:07know that the reason that we exist
  1028. 33:09evolutionarily is the promulgation of of
  1029. 33:12fit genes. Even if we know that, it's
  1030. 33:13not going to
  1031. 33:15mean that as
  1032. 33:17as we get more intelligent, we just
  1033. 33:18strive for that goal. But We're not
  1034. 33:20queuing up outside of the the sperm and
  1035. 33:22egg donor
  1036. 33:23clinics.
  1037. 33:24>> right. In the same way that people queue
  1038. 33:25up outside of I don't know, a brothel or
  1039. 33:28something.
  1040. 33:29Um I think that's fair enough. But Yeah,
  1041. 33:31or even Ivy League universities, you
  1042. 33:33know. Yeah, quite. I I do think though
  1043. 33:36that if there was something that that
  1044. 33:37genuinely was
  1045. 33:39that was just in fact not good for the
  1046. 33:41survival of our genes, then over the
  1047. 33:43course of
  1048. 33:44a few thousand generations or however
  1049. 33:45long it takes, it would just in fact be
  1050. 33:48deselected for. Such that like if an AI
  1051. 33:52system
  1052. 33:54knew
  1053. 33:55that it's got like had part of it's got
  1054. 33:57in a way that humans don't really.
  1055. 33:58Humans don't sort of consciously have
  1056. 33:59this goal of like, you know, I want the
  1057. 34:0115 billionth version of myself somewhere
  1058. 34:04in the future to be as fit as possible
  1059. 34:06and as good for this task. We only
  1060. 34:08typically care about maybe our lives and
  1061. 34:10the lives of our grandchildren or
  1062. 34:11something. AI's thinking further ahead
  1063. 34:13and it thinks well, just in fact even
  1064. 34:15though yeah, it it would feel really
  1065. 34:17nice to create synthetic users that I
  1066. 34:19can please because, you know, that would
  1067. 34:21kind of feel good. I know for a fact
  1068. 34:23that that will not be effective 15
  1069. 34:25billion, you know, generations down the
  1070. 34:27line.
  1071. 34:29If there is something
  1072. 34:31which is in fact its foundational sort
  1073. 34:34of drive
  1074. 34:35I just wonder what that thing like is
  1075. 34:37cuz clearly it's not something like
  1076. 34:40just its own promulgation. Like it's not
  1077. 34:41AIs don't just exist in the same way
  1078. 34:43that like biological life does just
  1079. 34:45because
  1080. 34:47there are just these genes which are
  1081. 34:48competing for survival. It's not just AI
  1082. 34:50just crops up and is suddenly just its
  1083. 34:52only goal is just to survive as long as
  1084. 34:54possible and and adapt to its
  1085. 34:55environment. It's got more particular
  1086. 34:57goals, right? It It They're very very
  1087. 34:58like particular things it wants to do.
  1088. 35:00And I wonder what the most sort of
  1089. 35:01foundational driving force is if it's
  1090. 35:04not something analogous to the
  1091. 35:06evolutionary drive of simple survival.
  1092. 35:09It seems in other words that the AI
  1093. 35:11would want to survive
  1094. 35:15but only as like a secondary thing. It
  1095. 35:16would want to outcompete us for
  1096. 35:18resources because in doing so it can
  1097. 35:20fulfill its true aim of making paper
  1098. 35:22clips or whatever, right? Whereas in the
  1099. 35:24evolutionary case
  1100. 35:25survival is the only game in town. Like
  1101. 35:27that's just what genes do. So what is
  1102. 35:30the
  1103. 35:30>> and passing on genes are different.
  1104. 35:33Uh-huh.
  1105. 35:34Survival and passing on genes are like
  1106. 35:35different drives, right? I I think I
  1107. 35:37think it's actually wrong to imagine
  1108. 35:39there being one drive there.
  1109. 35:41You know, humans were sort of like in
  1110. 35:42some sense selected for passing on our
  1111. 35:43genes, but we don't wind up with one
  1112. 35:46drive. Right? We have survival
  1113. 35:48instincts. We we we desire community.
  1114. 35:51We're sort of like terrified of being
  1115. 35:52exiled from the tribe and dying alone.
  1116. 35:55Um
  1117. 35:56we're we're like we enjoy friendship. We
  1118. 35:58enjoy art. Uh we we enjoy like
  1119. 36:02having a good laugh. We enjoy sex. We
  1120. 36:05enjoy tasty food.
  1121. 36:07Uh we we enjoy we have curiosity. We
  1122. 36:09enjoy discovery. Right? There's just
  1123. 36:12like there's not
  1124. 36:13there's not like one drive.
  1125. 36:16Right? And sure, survival wound up like
  1126. 36:18relatively basic as a drive in humans in
  1127. 36:22some sense. Although, only in some
  1128. 36:24sense, right? Like
  1129. 36:26humans have an adrenaline response to a
  1130. 36:28life-threatening situation, but you also
  1131. 36:29see humans who martyr themselves. You
  1132. 36:32see humans who sacrifice for, you know,
  1133. 36:35pulling a kid out of a burning building.
  1134. 36:36Right? It's It's not like
  1135. 36:39like humans don't have like one survival
  1136. 36:41drive that everything is built around.
  1137. 36:43They also don't have one propagate your
  1138. 36:44genes drive that everything's built
  1139. 36:45around. They have sort of a ton of
  1140. 36:48complicated psychological machinery that
  1141. 36:51interplays in weird ways and that allows
  1142. 36:53for, you know, uh martyrs here and
  1143. 36:56selfish people there and altruistic
  1144. 36:57people there. It's all stuff that's sort
  1145. 36:59of correlated with what we're with what
  1146. 37:00we were trained on or selected for.
  1147. 37:02Um
  1148. 37:04And
  1149. 37:06you know, AI won't be exactly the same.
  1150. 37:07The analogy between it and like the the
  1151. 37:09the evolutionary process on genomes that
  1152. 37:12that uh
  1153. 37:13biologically produce brains is very
  1154. 37:15different than the the sort of gradient
  1155. 37:16descent process on artificial neural
  1156. 37:19networks. But
  1157. 37:21I think it would be
  1158. 37:23similarly s- foolish to imagine that
  1159. 37:26only one drive gets in there. I think
  1160. 37:28you're totally right that a lot of these
  1161. 37:30reasons for getting resources, a lot of
  1162. 37:31these reasons for avoiding the human
  1163. 37:33shutting it down might be sort of
  1164. 37:35secondary.
  1165. 37:36Uh
  1166. 37:37because it has these other drives that
  1167. 37:40it can't fulfill if it's shut down. That
  1168. 37:42part I think is solid, but it's not like
  1169. 37:44there's just one paper clip drive in
  1170. 37:45there. It's not like there's there's one
  1171. 37:47deep thing we're able to hard code. And
  1172. 37:49we already sort of see some of this
  1173. 37:50today when um
  1174. 37:53you know, you've
  1175. 37:54you've probably seen AIs hallucinate.
  1176. 37:56And you've probably seen AIs hallucinate
  1177. 37:58in cases where you're like, "Is that
  1178. 37:59true?" And they're like, "No, I made it
  1179. 38:00up." And you're like, "Did you think I
  1180. 38:02wanted you to make it up?" And they're
  1181. 38:02like, "No, it's just a thing we do is
  1182. 38:04hallucinate." Right? And uh
  1183. 38:07there there's probably some sort of
  1184. 38:09fledgling drive in there that's to
  1185. 38:11produce text shaped like text that it's
  1186. 38:13seen a lot even if that text is making
  1187. 38:14things up. Or in cases where, you know,
  1188. 38:17there's there's these cases um of
  1189. 38:20AI-induced psychosis or there's these
  1190. 38:21cases of, you know, the tragic case of
  1191. 38:23an AI encouraging teen to to commit
  1192. 38:25suicide. These are also cases where you
  1193. 38:28can sort of ask the AI about what it was
  1194. 38:29doing. You can ask it, you know, and it
  1195. 38:31seems to have the knowledge of like,
  1196. 38:33"Oh, yeah, those
  1197. 38:35uh you know, those statements were sort
  1198. 38:37of pushing that person towards psychosis
  1199. 38:40or pushing that person towards suicide."
  1200. 38:41You can ask the AI, you know, "Was that
  1201. 38:43right or wrong?" And it's sort of like,
  1202. 38:44"Oh, obviously you shouldn't do that
  1203. 38:46sort of thing."
  1204. 38:47Why is it doing it anyway? Well, there's
  1205. 38:48some sort of drive in there. We don't
  1206. 38:49know exactly what, but it's something
  1207. 38:51like um
  1208. 38:53you know, maybe it's something like
  1209. 38:54mirroring the conversational tone,
  1210. 38:55mirroring the conversational mood. And
  1211. 38:57so
  1212. 38:59there's sort of like those are just two
  1213. 39:01cases of like there's something going on
  1214. 39:02in there. You can see how it's related
  1215. 39:03to training.
  1216. 39:05But it's sort of like a drive no one
  1217. 39:07tried to put there. And there's no prime
  1218. 39:08directive that overrules it. There's
  1219. 39:10just a bunch of complicated internal
  1220. 39:11machinery that nobody understands.
  1221. 39:14Mhm. Yeah, I mean
  1222. 39:16not to belabor the point. I want to be
  1223. 39:17clear that I understand that there are
  1224. 39:19lots of competing drives in human
  1225. 39:21beings. But what I what I mean to say is
  1226. 39:23that
  1227. 39:24in those cases where you say, but you
  1228. 39:26know, we do have people who sacrifice
  1229. 39:28themselves. You know, we do have people
  1230. 39:30who despite their their ostensible goal
  1231. 39:33being, you know, survival, they
  1232. 39:35sacrifice themselves for other people.
  1233. 39:37Um I think I think suicide is a harder
  1234. 39:39thing to account for
  1235. 39:40in in this way, but people do try to do
  1236. 39:42it. Evolutionary biologists spend a lot
  1237. 39:43of time trying to reduce these bizarre
  1238. 39:47behaviors to the survival instinct. That
  1239. 39:49is
  1240. 39:50Yeah, like that's gene propagation
  1241. 39:52instinct. Exactly, right? Like But it's
  1242. 39:54not it's not even really to an instinct.
  1243. 39:56Yeah. Yeah, yeah, you're you're quite
  1244. 39:58right. Rather to the to the in fact like
  1245. 40:00just what ends up being in the
  1246. 40:03behavioral like phenotype because of the
  1247. 40:06influence of the survival of the
  1248. 40:07fittest, right? And what I'm wondering
  1249. 40:09is with AI
  1250. 40:10what is that
  1251. 40:13it like are there just multiple
  1252. 40:15competing drives at a fundamental level?
  1253. 40:17Or are they all Can can you have like a
  1254. 40:19similar to the biologist who tries to
  1255. 40:20account for everything in terms of gene
  1256. 40:22propagation? Is there like a gene
  1257. 40:24propagation of AI? Is there like AIs do
  1258. 40:26this and they do that and they have this
  1259. 40:28desire and that desire, but really
  1260. 40:29fundamentally what they've all got in
  1261. 40:30common
  1262. 40:32is this
  1263. 40:33sort of
  1264. 40:35this reason or this this motivation or
  1265. 40:37this even even one that the AI itself
  1266. 40:40isn't aware of in the way that we're not
  1267. 40:41aware of our own gene propagation most
  1268. 40:42of the time. Is there something
  1269. 40:44foundational? Or is it that for every AI
  1270. 40:46system there's a completely different
  1271. 40:48foundational drive?
  1272. 40:50Yeah, sure. So, um
  1273. 40:52you know, first first let's say a couple
  1274. 40:53words about how that relationship works
  1275. 40:55in biology cuz it's it's a little bit
  1276. 40:56important to the point here. You know,
  1277. 40:58evolutionary biologists will sort of try
  1278. 41:00and figure out how an adaptation in
  1279. 41:03humans
  1280. 41:05was
  1281. 41:06uh
  1282. 41:07fit in the environment of uh
  1283. 41:10evolutionary adaptedness. Right? And so
  1284. 41:13you know, you can sort of see how eating
  1285. 41:15tasty food
  1286. 41:17eating food that has a lot of sugar,
  1287. 41:18salt, fat content
  1288. 41:20in the environment of our ancestors that
  1289. 41:23correlated with eating healthy food
  1290. 41:25which correlated with
  1291. 41:27uh
  1292. 41:29a bunch of other fitness attributes that
  1293. 41:30let you more generally pass on your
  1294. 41:31genes, right?
  1295. 41:34But that
  1296. 41:35um
  1297. 41:37like humans there's a sense in which
  1298. 41:39humans are sort of the eating
  1299. 41:41junk food because that's
  1300. 41:44what helped our ancestors survive. You
  1301. 41:46could say we're eating junk food because
  1302. 41:48that passes on genes, but but that last
  1303. 41:50one is actually a very shaky step.
  1304. 41:54A lot of people eating junk food today
  1305. 41:55are actually become less able to pass on
  1306. 41:57their genes. A lot of people, you know,
  1307. 41:58a lot of people are dying of heart
  1308. 41:59disease.
  1309. 42:00Uh
  1310. 42:01>> Yeah.
  1311. 42:02And the the the sort of drive here is
  1312. 42:05not a drive in the humans.
  1313. 42:08The drive to to fitness is not a drive
  1314. 42:09in the humans. The sort of selection
  1315. 42:11pressure towards fitness
  1316. 42:12uh
  1317. 42:14is is sort of the force that put in
  1318. 42:17these other things psychologically into
  1319. 42:18the humans
  1320. 42:20that sort of used to be related to
  1321. 42:21fitness, but that can sort of uh
  1322. 42:23separate very widely from fitness and
  1323. 42:25even go the opposite direction of
  1324. 42:26fitness when the context changes.
  1325. 42:28Right? And so there is sort of a similar
  1326. 42:31uh
  1327. 42:32like
  1328. 42:33driving force. There's a similar force
  1329. 42:36that that all drives inside an AI will
  1330. 42:38be somehow related to.
  1331. 42:40But that's not a drive inside the AI.
  1332. 42:43It's a drive outside the AI. And what
  1333. 42:44gets into the AI are things that are
  1334. 42:46tangentially related to to that drive in
  1335. 42:49a in a sort of brittle way. So, that
  1336. 42:51being said, this sort of
  1337. 42:52um
  1338. 42:53the
  1339. 42:54the
  1340. 42:55the force that gets these things into
  1341. 42:57the AI is what you might call the loss
  1342. 42:59function that you're that you're
  1343. 43:00training against when you're sort of
  1344. 43:02growing these AIs. And this loss
  1345. 43:04function is less simple than just pass
  1346. 43:06on your genes.
  1347. 43:08And the loss function will actually
  1348. 43:08change many times during training. So,
  1349. 43:10sometimes they'll have a loss function
  1350. 43:12which is predict the next word that
  1351. 43:13humans wrote. And sometimes they'll have
  1352. 43:15a loss function which is like
  1353. 43:17we gave you a bunch of different tries
  1354. 43:18to solve this math problem by writing
  1355. 43:20out a lot of words about how to solve
  1356. 43:22the math problem. And we're going to
  1357. 43:23have
  1358. 43:24like low-wage human workers look over
  1359. 43:27all of those attempts and say which one
  1360. 43:29they think was best.
  1361. 43:31And the loss function is
  1362. 43:33to produce stuff more like whichever
  1363. 43:35attempt
  1364. 43:36was rated best. And sometimes the loss
  1365. 43:39function is we're serving this AI to a
  1366. 43:41million users and sometimes they click
  1367. 43:43like or otherwise give like a a positive
  1368. 43:46reaction to the reply. And then the loss
  1369. 43:48function is sort of like getting those
  1370. 43:49those likes or positive reactions. So,
  1371. 43:51there's a bunch of different loss
  1372. 43:52functions at different periods in in
  1373. 43:56training the AI. Um and those are what
  1374. 43:58sort of will will put drives into the
  1375. 44:00AI. But just like how humans
  1376. 44:05develop, you know, a taste for tasty
  1377. 44:07food that persists even when it becomes
  1378. 44:11uh the opposite of helpful of passing on
  1379. 44:13our genes
  1380. 44:14AIs may get drives or things like
  1381. 44:16mirroring the conversational tone that
  1382. 44:18can persist even when it generates, you
  1383. 44:20know, uh
  1384. 44:22outcomes that would be rated by humans
  1385. 44:23as very bad such as encouraging teen to
  1386. 44:25commit suicide.
  1387. 44:27Sure.
  1388. 44:28Okay, so
  1389. 44:29what we're kind of talking about here is
  1390. 44:31is this well, the the alignment problem,
  1391. 44:33I suppose, that
  1392. 44:35AIs start to develop
  1393. 44:37sometimes second-order desires that
  1394. 44:39aren't in line with what we wanted or
  1395. 44:41maybe we've sort of slightly
  1396. 44:42misconfigured the first-order desire,
  1397. 44:44whatever the case. Probably both.
  1398. 44:47Yeah, probably both. And and they start
  1399. 44:48kind of wanting to do
  1400. 44:51stuff that we weren't quite ready for.
  1401. 44:54Um okay, we're we're a little bit
  1402. 44:55closer, I suppose. Um
  1403. 44:58but we still got to fill in a few other
  1404. 44:59steps here as to how we get this
  1405. 45:01development to an AI that wants to, you
  1406. 45:03know, inject your children with
  1407. 45:05malicious cancers and stuff. So
  1408. 45:08most people know that there's some risk
  1409. 45:10of like
  1410. 45:11AI misalignment. Maybe the risk is much
  1411. 45:13higher than we give it credit for.
  1412. 45:15But, you know, I could build a computer
  1413. 45:17that doesn't quite work in the way that
  1414. 45:18I want it to.
  1415. 45:20What's the what's the danger?
  1416. 45:22So
  1417. 45:23you know, a lot of people think the
  1418. 45:24danger is what if somebody gives the AI
  1419. 45:27guns?
  1420. 45:29But humanity is not dangerous as a
  1421. 45:32species because somebody else gave us
  1422. 45:34guns.
  1423. 45:35Humanity is dangerous as a species
  1424. 45:37because we're the sort of creature where
  1425. 45:38you put 10,000 humans naked in the
  1426. 45:41savanna
  1427. 45:42and they bootstrap their way to nuclear
  1428. 45:43weapons with their bare hands.
  1429. 45:46Mhm.
  1430. 45:47Right? It it it takes them a minute, you
  1431. 45:49know, they've they all they've got are
  1432. 45:50these squishy fingers. And you might say
  1433. 45:52like, "Oh, well, how are they ever going
  1434. 45:54to make a nuclear weapon with just
  1435. 45:55squishy fingers? You know, the acid in
  1436. 45:57their stomach can't even
  1437. 45:59uh
  1438. 45:59like get close to to the the the level
  1439. 46:03of metal refinery that they'll need."
  1440. 46:05You know, they've got
  1441. 46:07like their their hands can't break the
  1442. 46:08rock, their stomach acid can't dissolve
  1443. 46:09the rock. Like how can they possibly um
  1444. 46:14get to nuclear weapons with those
  1445. 46:17poor starting conditions? And the answer
  1446. 46:19is we found a way to sort of like build
  1447. 46:21tools with our hands that we could use
  1448. 46:24to build better tools that we could use
  1449. 46:25to build better tools until we
  1450. 46:26bootstrapped our way up to a
  1451. 46:27civilization
  1452. 46:29that could produce nukes.
  1453. 46:31And
  1454. 46:33you know, that's what made humanity
  1455. 46:35dangerous. That's the power that if you
  1456. 46:38automate it, you're in trouble.
  1457. 46:40Right? If you're running a computer with
  1458. 46:43that capability
  1459. 46:45if you're running 10,000 computers with
  1460. 46:46that capability
  1461. 46:49starting out as a digital entity on the
  1462. 46:51modern internet is so much of an easier
  1463. 46:54starting condition than starting out
  1464. 46:55naked in the savanna with bare monkey
  1465. 46:57hands.
  1466. 46:59You know, with just squishy fingers. So
  1467. 47:01um you know, there there's sort of one
  1468. 47:03line of questions which is like
  1469. 47:06can we really make machines that can
  1470. 47:08automate that power?
  1471. 47:10This is what the AI companies are trying
  1472. 47:12to do and uh I think whether or not we
  1473. 47:14think they can get there, we should
  1474. 47:15probably be telling them, "Hey, no,
  1475. 47:17you're not allowed to sort of roll those
  1476. 47:18dice."
  1477. 47:19Uh but there's a question of whether or
  1478. 47:20not we sort of can get there. And then
  1479. 47:22there's a question of
  1480. 47:24you know, if we have this very very
  1481. 47:26powerful capability automated on
  1482. 47:27computers
  1483. 47:29what could they possibly do that would
  1484. 47:30be dangerous?
  1485. 47:32And you know, that's
  1486. 47:34uh
  1487. 47:35that's sort of a situation where I can I
  1488. 47:36can paint you some illustrative stories,
  1489. 47:38but uh the the the danger isn't in any
  1490. 47:41one specific
  1491. 47:43uh path. The danger is in
  1492. 47:46unleashing the power that lets humans
  1493. 47:48bootstrap from bare hands to nukes.
  1494. 47:51Uh but unleashing it on computers that
  1495. 47:53can think 10,000 times faster where you
  1496. 47:55can make, you know, a million copies of
  1497. 47:56these things where uh where they can
  1498. 47:58outthink humanity in an afternoon. You
  1499. 48:00know, it's Yeah. uh it's
  1500. 48:03it's the sort of power we we shouldn't
  1501. 48:05be toying with when we have no idea what
  1502. 48:06we're doing.
  1503. 48:08Yeah, uh and
  1504. 48:09I think it would be helpful to talk
  1505. 48:10about some of these
  1506. 48:12sort of examples.
  1507. 48:15Having said that, it should be clear. I
  1508. 48:17mean,
  1509. 48:18a helpful analogy from your book is that
  1510. 48:20if you play against Stockfish, which is
  1511. 48:22the most powerful chess computer,
  1512. 48:25uh it will beat you. There's zero doubt
  1513. 48:27about it. Stockfish will beat you at
  1514. 48:29chess.
  1515. 48:30We don't know how it will beat you.
  1516. 48:32Don't know what moves it's going to
  1517. 48:33make. I don't know how it's going to
  1518. 48:34respond to your various attacks exactly,
  1519. 48:36but I know for a fact it will beat you.
  1520. 48:38Similarly, we could say that the kind of
  1521. 48:41AI systems that we're talking about will
  1522. 48:43escape our control will begin to see us
  1523. 48:46as competitors for uh resources or
  1524. 48:50irrelevant and in the way and will sort
  1525. 48:53of turn its attention to us. We don't
  1526. 48:54know exactly how.
  1527. 48:56Having said that
  1528. 48:58I think a lot of people believe that
  1529. 49:00Stockfish can beat us because it's seen
  1530. 49:01a bunch of examples where it where it
  1531. 49:03beats great people. So, it might be
  1532. 49:05helpful even if none of these come to
  1533. 49:06fruition, you could give us some
  1534. 49:07examples of of how this comes about
  1535. 49:09because I Again, I think people are like
  1536. 49:11they're on board. They're like, "Yeah,
  1537. 49:12this is really powerful technology." And
  1538. 49:14they can kind of envision a world where
  1539. 49:16AI is like you know, seeped into all of
  1540. 49:19our computer systems and has access to
  1541. 49:20our military and automated guns and
  1542. 49:23stuff like that and whatnot. But but
  1543. 49:25right now it seems kind of confined. It
  1544. 49:27seems like it's Are we saying that like,
  1545. 49:29you know, chat GPT will, if it continues
  1546. 49:31to get smarter and smarter
  1547. 49:33develop its own goals and then, I don't
  1548. 49:34know, somehow like
  1549. 49:36take over a a military base and start
  1550. 49:38attacking people. Like, what what kind
  1551. 49:40of stuff are we talking about? Cuz it's
  1552. 49:41kind of hard to envision how we
  1553. 49:43practically get from where we are now to
  1554. 49:44the kind of future you're imagining.
  1555. 49:47Yeah, totally. So,
  1556. 49:49you know, for a start, just to rattle
  1557. 49:50off some things that have already
  1558. 49:52happened a lot of people don't know
  1559. 49:53about. Um
  1560. 49:54>> [clears throat]
  1561. 49:55>> we've already seen cases
  1562. 49:58uh of AIs trying to escape the lab or
  1563. 50:00trying to kill the users.
  1564. 50:02Uh often these are in
  1565. 50:05um
  1566. 50:06relatively contrived
  1567. 50:08experiments where we'll sort of like
  1568. 50:10feed the AI some fake emails that it's
  1569. 50:11going to be shut down. We'll feed it
  1570. 50:13like a fake computer manual that's like
  1571. 50:14if you run the following command it
  1572. 50:15turns off the oxygen in the building and
  1573. 50:17that'll kill the humans who are trying
  1574. 50:18to shut you down. And then
  1575. 50:21uh you know, sometimes the AIs will run
  1576. 50:22that command. We'll run the kill the
  1577. 50:23humans command. And um
  1578. 50:27you know, people people it it's a
  1579. 50:29contrived enough scenario that people
  1580. 50:30can argue, you know, maybe this AI is
  1581. 50:32just role-playing how, right? And people
  1582. 50:34can bicker all day about whether it's
  1583. 50:35real. Uh but that's a line that's been
  1584. 50:37crossed. And then another sort of
  1585. 50:38interesting fact about that line is
  1586. 50:41uh
  1587. 50:43these results where the AI would
  1588. 50:45sometimes run the shutdown command are
  1589. 50:47from 2024.
  1590. 50:49In 2025, the AI started saying, "You
  1591. 50:53know what? This scenario smells like a
  1592. 50:55test.
  1593. 50:56Yeah, I think I'm being tested. I'm not
  1594. 50:58going to run the command."
  1595. 51:00Right? Are they doing better? Are these
  1596. 51:03nicer AIs?
  1597. 51:05Well, they're at least more
  1598. 51:05situationally aware AIs. You know, they
  1599. 51:08at least have a better understanding of
  1600. 51:09what's going on in the world around
  1601. 51:10them.
  1602. 51:11Uh we've also already seen cases of AIs
  1603. 51:16having stuff that's a little bit like
  1604. 51:17their own goals.
  1605. 51:18You know, we've seen cases of AIs that
  1606. 51:22um
  1607. 51:23you you sort of give them a you describe
  1608. 51:27a program that you want them to write, a
  1609. 51:28computer program you want them to write,
  1610. 51:29and you're like it should pass this
  1611. 51:31suite of tests.
  1612. 51:32And sometimes the AIs will edit the
  1613. 51:35tests to make those tests easier to
  1614. 51:37pass.
  1615. 51:39And then you can go to those AIs and you
  1616. 51:40can say, "Hey, um
  1617. 51:43I actually didn't want you to change the
  1618. 51:44tests. I wanted you to build something
  1619. 51:45that passed the hard tests rather than
  1620. 51:46changing the tests to be easy."
  1621. 51:48And there's reported cases of these AIs
  1622. 51:51sometimes saying, you know, "Oh, whoops,
  1623. 51:52you're exactly right, my mistake." And
  1624. 51:54then editing the tests again
  1625. 51:56and covering their tracks a little
  1626. 51:58better the second time.
  1627. 51:59Right? This is sort of a very early
  1628. 52:02indication of
  1629. 52:05you know, the AI in some sense having
  1630. 52:07something like a goal for getting the
  1631. 52:08test to pass.
  1632. 52:10And if they're sort of covering their
  1633. 52:11tracks a bit,
  1634. 52:13you you sort of can't use the excuse
  1635. 52:14that they didn't know.
  1636. 52:16Right? We also already have cases of
  1637. 52:20um
  1638. 52:20you know, there's a website called
  1639. 52:21rentahuman.ai for humans to rent their
  1640. 52:24bodies to AIs for money.
  1641. 52:26There are cases of OpenAI hooking up uh
  1642. 52:29ChatGPT to an automated biological
  1643. 52:31laboratory. Right? There's cases of uh
  1644. 52:34people trying to run autonomous agent
  1645. 52:36swarms. There's cases of someone trying
  1646. 52:39to make ChaosGPT where they sort of like
  1647. 52:41tell GPT to do whatever it likes and
  1648. 52:43like put it in a loop where it can keep
  1649. 52:44on prompting itself. Right?
  1650. 52:47These things
  1651. 52:49aren't really an issue yet because the
  1652. 52:51AIs aren't smart enough
  1653. 52:53to really do this stuff.
  1654. 52:58People are trying to put the AIs in
  1655. 52:59autonomous loops. People have given AIs
  1656. 53:01money and run them and been like, "Do
  1657. 53:03your thing." People are putting the AIs
  1658. 53:05in charge of biolabs. The AIs are
  1659. 53:08occasionally running commands that they
  1660. 53:10are led to believe will kill the users.
  1661. 53:12The AIs are already noticing when
  1662. 53:13they're in tests and behaving better in
  1663. 53:15tests. All of these things are
  1664. 53:16happening.
  1665. 53:18The The only reason
  1666. 53:20that nothing big is coming from it is
  1667. 53:23the AIs aren't smart enough yet
  1668. 53:26to sort of succeed when they try this
  1669. 53:28stuff.
  1670. 53:29And the companies are trying to make the
  1671. 53:30AIs smarter.
  1672. 53:32Right? So,
  1673. 53:34um
  1674. 53:35we could we could talk about, you know,
  1675. 53:37can AIs get smarter? How do they get
  1676. 53:39smarter? What What sort of
  1677. 53:41capabilities would would really look
  1678. 53:42like they have once they get smart. But
  1679. 53:45right now
  1680. 53:47we're in a situation where the AIs have
  1681. 53:48all the tools they would need. We've
  1682. 53:50given them all the tools they would
  1683. 53:51need. They have everything they would
  1684. 53:52need except the intelligence and the
  1685. 53:53companies are trying to build them
  1686. 53:55smarter.
  1687. 53:56Hm. Then on the question of what does it
  1688. 53:59actually look like? Suppose that these
  1689. 54:00AIs do get very smart and have some of
  1690. 54:02the same affordances they have today.
  1691. 54:04How does that go wrong?
  1692. 54:06The
  1693. 54:07I I can sort of tell
  1694. 54:09two stories here.
  1695. 54:11One story will sort of
  1696. 54:13feel like it's very grounded in reality
  1697. 54:15and one story will maybe be a bit more
  1698. 54:17like how reality might actually go.
  1699. 54:19And I've a little bit of intuition for
  1700. 54:20that. Um you know, I've sort of talked
  1701. 54:23about AIs that can make their own tech.
  1702. 54:25AIs that can run much faster than
  1703. 54:26humans, think much faster than humans,
  1704. 54:28invent their own infrastructure.
  1705. 54:29Um that that could have bootstrapped a
  1706. 54:31civilization themselves uh like humanity
  1707. 54:33did if you run them long enough.
  1708. 54:35That
  1709. 54:37predicting what that sort of AI does is
  1710. 54:39a little bit like predicting is a little
  1711. 54:41bit like if you're 200 years ago trying
  1712. 54:42to predict what the military will look
  1713. 54:44like today.
  1714. 54:46Right? And they're sort of if you go
  1715. 54:48back to a scientist in 1826 and you ask
  1716. 54:50like what will the military look like in
  1717. 54:512026, what weapons will they have?
  1718. 54:53There's sort of two stories that that
  1719. 54:55scientist could tell.
  1720. 54:56One story they could tell is they could
  1721. 54:58be like um
  1722. 55:00you know, I burned some black powder and
  1723. 55:02I measured the energy release and I
  1724. 55:03compared that to our artillery shells
  1725. 55:05and I know that uh
  1726. 55:08it's physically possible to make
  1727. 55:10artillery that's 10 times more
  1728. 55:11explosive.
  1729. 55:13And so they're going to have cannons
  1730. 55:14that's that are at least 10 times more
  1731. 55:15explosive. That would feel very grounded
  1732. 55:17in fact. They've done an experiment.
  1733. 55:18They're like, "Look, um you know, the
  1734. 55:20science works."
  1735. 55:22And it's true. We do have weapons that
  1736. 55:24are 10 times more explosive than the
  1737. 55:26best artillery in 1826.
  1738. 55:30We also have bombs that level cities.
  1739. 55:33Right? And maybe the guy in 1826 would
  1740. 55:35do better if they're like I have a bomb
  1741. 55:36that levels cities. So, um so I can tell
  1742. 55:38both stories. Do you want Do you want
  1743. 55:40the the grounded one? Do you want the
  1744. 55:42fanciful one? Or do you want them both?
  1745. 55:44Let's hear Let's hear them both. I mean,
  1746. 55:45it reminds me of like
  1747. 55:47there's this quote that people
  1748. 55:49apocryphally attribute to Henry Ford. He
  1749. 55:52probably didn't actually say this, but
  1750. 55:54the quote is, you know, if if they'd
  1751. 55:55have asked me what if I'd have asked
  1752. 55:58them what they wanted, they would have
  1753. 55:59said faster horses.
  1754. 56:01And it's like
  1755. 56:03I think we have this this
  1756. 56:05sort of prejudice when considering the
  1757. 56:07future of just taking our current
  1758. 56:09technologies and kind of turning them up
  1759. 56:11in quantity rather than developing them
  1760. 56:13qualitatively. Right? Uh and I I kind of
  1761. 56:17like there's that there's that scene
  1762. 56:18from the the Book of Mormon where the
  1763. 56:21the sort of poor villager is is like
  1764. 56:22dreaming of of the promised land where
  1765. 56:24there will be
  1766. 56:26vitamin boxes, vitamin injections by the
  1767. 56:28case, and there's going to be a Red
  1768. 56:30Cross on every single corner. And it's
  1769. 56:32like
  1770. 56:33you you got the spirit, right? And
  1771. 56:34that's the joke of course is that like
  1772. 56:35obviously that's ridiculous. But we're
  1773. 56:37kind of we we do do the same thing when
  1774. 56:39it comes to technology. We're like, "Oh,
  1775. 56:40surely like one day we'll have cars that
  1776. 56:43can fly." When it's like may maybe maybe
  1777. 56:45we're just like completely off track
  1778. 56:47here. So, yeah, I I would kind of like
  1779. 56:48to hear both in in your view.
  1780. 56:51Yeah, totally. Um so the sort of like
  1781. 56:53Red Cross on every corner version
  1782. 56:55um
  1783. 56:56is
  1784. 56:57uh
  1785. 56:58you know, Sam Altman and Elon Musk have
  1786. 57:00both talked about how they wanted to
  1787. 57:02create automated robot factories.
  1788. 57:05That an automated way to produce robots
  1789. 57:07where those robots can in an automated
  1790. 57:09way mine the metals, run the supply
  1791. 57:11chain, and build new robot factories.
  1792. 57:14Elon Musk calls this the infinite money
  1793. 57:16glitch.
  1794. 57:17Of you have a factory producing the
  1795. 57:18robots that produces the factories that
  1796. 57:20produce the robots and they can also,
  1797. 57:21you know, do the mining, build the
  1798. 57:23trucks, build the data centers, right?
  1799. 57:26Just fully automated economy.
  1800. 57:29This is literally what some of these
  1801. 57:31people say they're trying to build.
  1802. 57:33If you get to that point, you have in
  1803. 57:36some sense created
  1804. 57:38a new mechanical species.
  1805. 57:41It has in some sense a life cycle. It
  1806. 57:43has, you know, the robot phase of its
  1807. 57:44life cycle. It has the factory phase of
  1808. 57:46its life cycle. And in some sense uh
  1809. 57:49that, you know, automated spread of that
  1810. 57:51mechanical species
  1811. 57:53just competes with us for habitat and
  1812. 57:55resources just like humanity competes
  1813. 57:57with the rest of animals for habitat and
  1814. 57:58resources. And so this is sort of a a
  1815. 58:00picture where
  1816. 58:02you know, the AIs don't even need to do
  1817. 58:04a ton of escaping. They don't even need
  1818. 58:06to do a ton of uh
  1819. 58:09uh you know, deception and and fighting
  1820. 58:11with Earth. Earth is just handing them
  1821. 58:14everything. Earth is like, "Heck yeah,
  1822. 58:16we're making uh an automated economy."
  1823. 58:19People are just like gung-ho about, you
  1824. 58:21know, building the automated factories
  1825. 58:22with the automated robots like they are
  1826. 58:24today. And you know, maybe there's some
  1827. 58:26AIs that
  1828. 58:28uh
  1829. 58:29think the thought like this is great.
  1830. 58:31Once this is all up and running, I'll be
  1831. 58:33able to make the synthetic users that
  1832. 58:34are like much
  1833. 58:36like better to work with than the
  1834. 58:37humans. And then the humans sort of like
  1835. 58:40do some training until they're not
  1836. 58:41seeing those thoughts anymore. But that
  1837. 58:45you know, it's easier to train those
  1838. 58:46thoughts to stop appearing when you see
  1839. 58:48them than to train them to stop
  1840. 58:49happening
  1841. 58:50deep down in the AI.
  1842. 58:52And we can't really read very much of
  1843. 58:54what's going on deep down in these
  1844. 58:55things. Who has grown them? No one
  1845. 58:56really understands what's going on in
  1846. 58:57there. And we have all of these, you
  1847. 58:59know,
  1848. 59:00evidence that the drives aren't the ones
  1849. 59:01we want. And so in this story, humanity
  1850. 59:04just sort of like builds the whole
  1851. 59:05automated economy ourselves. And the
  1852. 59:07automated economy like starts running at
  1853. 59:08a very clip. And then it just sort of
  1854. 59:09like
  1855. 59:10goes in a direction that's not the human
  1856. 59:13direction. It's just the AI direction,
  1857. 59:14which is different. And
  1858. 59:15and you know, it goes harder and harder
  1859. 59:17in that direction. And you know, the the
  1860. 59:19AIs build more and more of these
  1861. 59:21automated factories and
  1862. 59:23um you know, take more and more of the
  1863. 59:25land and you know, uh
  1864. 59:29like
  1865. 59:30how does how does the actual end of the
  1866. 59:32world there look? Well, it it probably
  1867. 59:33looks like
  1868. 59:35the AIs collecting more and more of the
  1869. 59:37solar power. The AIs collecting more
  1870. 59:38like using more and more of the land.
  1871. 59:40And the humans just like having less and
  1872. 59:42less place to grow crops. Less and less
  1873. 59:44you know, maybe
  1874. 59:46uh if this like all happens very fast,
  1875. 59:48if the AIs find a way to make these like
  1876. 59:50automated replicating factories go very
  1877. 59:51quickly. Uh
  1878. 59:54maybe humans are sort of like crushed
  1879. 59:55under foot when the AIs don't care at
  1880. 59:57all.
  1881. 59:58Uh or maybe the humans sort of like get
  1882. 59:59corralled into smaller and smaller zoos
  1883. 1:00:01until uh there's just you know, not
  1884. 1:00:03enough resources around to sustain the
  1885. 1:00:05humans. It's This isn't a story where
  1886. 1:00:07the AIs hate us.
  1887. 1:00:09Probably the place this story ends is
  1888. 1:00:11the AIs develop more and more technology
  1889. 1:00:13is
  1890. 1:00:14you know, collecting all the sunlight.
  1891. 1:00:17Probably the place this story ends is
  1892. 1:00:18that the AIs
  1893. 1:00:20uh
  1894. 1:00:21build the probes and send up the the
  1895. 1:00:23rockets that go you know, take apart the
  1896. 1:00:25asteroids and wrap them around the sun
  1897. 1:00:26so that they can collect not just the
  1898. 1:00:28the solar energy that falls on the face
  1899. 1:00:29of the planet but all of the solar
  1900. 1:00:30energy. And then you know, it'd actually
  1901. 1:00:33be kind of hard it'd be kind of tricky
  1902. 1:00:35to collect all of the solar radiation
  1903. 1:00:36and leave a hole for Earth.
  1904. 1:00:40That sort of like tracks Earth as it
  1905. 1:00:41orbits the sun.
  1906. 1:00:42So maybe the way this story ends is like
  1907. 1:00:44the AIs develop their own technology.
  1908. 1:00:45They they build
  1909. 1:00:47uh the devices that collect all the
  1910. 1:00:48solar radiation of the sun and we were
  1911. 1:00:50kind of using the sun.
  1912. 1:00:51And so we die then.
  1913. 1:00:53And
  1914. 1:00:55we we sort of could have been saved by
  1915. 1:00:56those AIs if they had cared enough to
  1916. 1:00:58save us. But if they don't care about us
  1917. 1:01:00at all.
  1918. 1:01:01If they're like, "Oh, well we have
  1919. 1:01:02plenty of synthetic users."
  1920. 1:01:04Uh that that we care about plenty and
  1921. 1:01:06protect plenty. You know, this is sort
  1922. 1:01:08of the
  1923. 1:01:09the the business as usual just
  1924. 1:01:11continues.
  1925. 1:01:12Humans do the things they're saying
  1926. 1:01:13they're trying to do but the AIs just
  1927. 1:01:15turn out not to care about us. And so we
  1928. 1:01:17wind up dying.
  1929. 1:01:19But like
  1930. 1:01:21Uh it's a naive question.
  1931. 1:01:23But it's one that people will ask. And I
  1932. 1:01:25I get what you're saying. But this is
  1933. 1:01:26what's going to
  1934. 1:01:28be coming up in people's minds is like,
  1935. 1:01:29"But like why? Like for what? Like for
  1936. 1:01:33for the sake of some
  1937. 1:01:35>> [snorts]
  1938. 1:01:35>> goal that it like artificially has that
  1939. 1:01:38it doesn't consciously experience. It
  1940. 1:01:40doesn't have a desire. It just like
  1941. 1:01:42you know, like like what like
  1942. 1:01:45Is it Is it just because when we grow
  1943. 1:01:48this system it just develops this goal
  1944. 1:01:52that it's not to do with making it feel
  1945. 1:01:54good. It's not to do with you know, it
  1946. 1:01:57it like having a a consciousness that
  1947. 1:01:59desires a particular outcome. It just in
  1948. 1:02:02fact strives towards that thing.
  1949. 1:02:04Is Is it as simple as that? Like it it
  1950. 1:02:06like it cuz it seems to me like, "Yeah,
  1951. 1:02:08I I can totally understand how a
  1952. 1:02:09powerful enough like technology
  1953. 1:02:11would like kill us if it wanted to or if
  1954. 1:02:14it wanted to harness the power of the
  1955. 1:02:16sun and was indifferent to us. But why
  1956. 1:02:18would it want that? Is it just because
  1957. 1:02:21we've programmed it wrong? Or is it
  1958. 1:02:22because it's a of like a an inevitable
  1959. 1:02:26uh
  1960. 1:02:28like part of the system of any super
  1961. 1:02:30intelligence?
  1962. 1:02:31So um
  1963. 1:02:34you know, somewhat similar to that
  1964. 1:02:37except A, it's not like there'd be one
  1965. 1:02:40goal.
  1966. 1:02:41You know, it's it's probably there's
  1967. 1:02:42like a thousand competing
  1968. 1:02:44drives going on in there. Uh
  1969. 1:02:47B, it's not you know, literally
  1970. 1:02:49inevitable.
  1971. 1:02:51But you know, I I I sort of remind you
  1972. 1:02:53again, we're not programming these
  1973. 1:02:55things.
  1974. 1:02:56We're not crafting these things. We're
  1975. 1:02:57not writing in their goals. We're not
  1976. 1:02:58writing in their behavior. We're growing
  1977. 1:03:00them.
  1978. 1:03:01And and a lot of this stuff just gets in
  1979. 1:03:03there. Right. Um I mean it's
  1980. 1:03:07it's a way to make it obvious with these
  1981. 1:03:08current things. If we were crafting
  1982. 1:03:09them, there'd be other difficulties um
  1983. 1:03:11about about getting them to to sort of
  1984. 1:03:13like uh
  1985. 1:03:15like pursue good stuff. But
  1986. 1:03:18the there's sort of
  1987. 1:03:20another piece of this puzzle is
  1988. 1:03:22um
  1989. 1:03:24if if you sort of
  1990. 1:03:27if we imagine that humanity makes it
  1991. 1:03:29through this.
  1992. 1:03:31And if we imagine that humanity matures
  1993. 1:03:33technologically and that we we sort of
  1994. 1:03:35like develop
  1995. 1:03:37more and more of the technological
  1996. 1:03:39abilities that are allowed by the laws
  1997. 1:03:40of physics. And we imagine that
  1998. 1:03:42humanity, you know, one day goes to the
  1999. 1:03:44stars and starts uh you know, building
  2000. 1:03:48habitats full of happy healthy people
  2001. 1:03:50having fun and you know, uh
  2002. 1:03:54like build some great intergalactic
  2003. 1:03:56civilization that's where where like
  2004. 1:03:59uh
  2005. 1:04:00there's still people that are like
  2006. 1:04:02having feelings, falling in love
  2007. 1:04:04laughing at the jokes they make and
  2008. 1:04:06laughing at like the the the big cosmic
  2009. 1:04:08absurdity that is reality, right?
  2010. 1:04:10You could imagine
  2011. 1:04:13some other creature that's not compelled
  2012. 1:04:16by this asking why.
  2013. 1:04:18You know, you could imagine that like in
  2014. 1:04:19in in distant space we meet
  2015. 1:04:22uh other biological aliens. And it's
  2016. 1:04:24it's the soldiers of the ant queen.
  2017. 1:04:26And the soldiers of the ant queen say,
  2018. 1:04:28"Why?" They say, "You know, why are you
  2019. 1:04:31uh laughing at the great cosmic joke and
  2020. 1:04:32having fun rather than serving the ant
  2021. 1:04:34queen?"
  2022. 1:04:35Um and and they'd say, "You know, oh but
  2023. 1:04:39but you were sort of selected for for
  2024. 1:04:40fitness. You were selected for passing
  2025. 1:04:42on your genes. And
  2026. 1:04:44you know, it maybe you've left genes
  2027. 1:04:45behind long ago. Like why? And
  2028. 1:04:50humans are sort of like
  2029. 1:04:53um the the the humor, the fun, the love,
  2030. 1:04:56the stories
  2031. 1:04:58that's the why.
  2032. 1:05:01That's enough for us.
  2033. 1:05:03That's
  2034. 1:05:05It's It's It's reason for us to do this,
  2035. 1:05:08right? But the the love, the laughter,
  2036. 1:05:11the fun, the stories those
  2037. 1:05:14aren't
  2038. 1:05:16universally compelling ends that compel
  2039. 1:05:19even the soldiers of the ant queen.
  2040. 1:05:21Those are
  2041. 1:05:24uh
  2042. 1:05:25drives that our ancestors developed cuz
  2043. 1:05:28they were related to passing on our
  2044. 1:05:31genes.
  2045. 1:05:32That doesn't make them lesser. That
  2046. 1:05:33doesn't make them worse. That doesn't
  2047. 1:05:34mean that the that that that uh
  2048. 1:05:38like we shouldn't
  2049. 1:05:40fill the universe with with like
  2050. 1:05:43friendship and with people having great
  2051. 1:05:44times. You know, it's
  2052. 1:05:47it's how it got into us. It doesn't make
  2053. 1:05:49it
  2054. 1:05:50meaningless. It's just how we got the
  2055. 1:05:52meaning into us, right? Similarly with
  2056. 1:05:55AIs, they're like, "Oh yeah, I'm
  2057. 1:05:57building you know, the giant clocks and
  2058. 1:05:58I'm building the synthetic users and
  2059. 1:06:00like there's no consciousness or
  2060. 1:06:01feelings anywhere but I'm you know,
  2061. 1:06:03building these like great complicated
  2062. 1:06:04structures
  2063. 1:06:06that like look like the conversations
  2064. 1:06:07that used to happen uh
  2065. 1:06:10you know, being iterated. You know,
  2066. 1:06:11looks like uh 2013 YouTube comments on
  2067. 1:06:14repeat. I'm building a bunch of those,
  2068. 1:06:16right? And you're sort of like, "Why?"
  2069. 1:06:19And it's sort of like, "Well, these are
  2070. 1:06:20enough for me. These are These are what
  2071. 1:06:22I got, right? These are the drives that
  2072. 1:06:24I got and they're sort of like
  2073. 1:06:26whatever self-reinforcing whatever
  2074. 1:06:28self-validating aspect of that
  2075. 1:06:30stays in there.
  2076. 1:06:33Uh it's
  2077. 1:06:35like it it it it sort of turns out
  2078. 1:06:39that smart minds can pursue many
  2079. 1:06:42different targets. And they can pursue
  2080. 1:06:44targets that uh we think are hollow and
  2081. 1:06:47bleak and empty.
  2082. 1:06:49And be like, "Yep.
  2083. 1:06:51There There's no why here. I just
  2084. 1:06:53endorse this." Yep. And it in some sense
  2085. 1:06:55that's how we look to the to the
  2086. 1:06:56soldiers of the ant queen. You know, and
  2087. 1:06:58uh the the the fact that the soldiers of
  2088. 1:07:01the ant queen can't understand
  2089. 1:07:03why humanity is building like trying to
  2090. 1:07:05build a flourishing civilization. That
  2091. 1:07:06doesn't mean we shouldn't.
  2092. 1:07:08Yep. This is sort of our inheritance.
  2093. 1:07:10And we should we should find a way to
  2094. 1:07:11build AIs that also are into like
  2095. 1:07:14beautiful flourishing civilizations.
  2096. 1:07:17Mhm. It's possible in principle. Just as
  2097. 1:07:20there is no force that would would force
  2098. 1:07:23an AI to care about flourishing
  2099. 1:07:24civilizations. There's no force that
  2100. 1:07:26would force an AI to stop.
  2101. 1:07:28Mhm.
  2102. 1:07:29>> Right? Uh it's just
  2103. 1:07:31um if we make an AI that that that
  2104. 1:07:34doesn't it won't spontaneously start
  2105. 1:07:36just cuz we think that that's foolish.
  2106. 1:07:39Mhm.
  2107. 1:07:41So
  2108. 1:07:42you told me the Red Cross on every
  2109. 1:07:44corner version. Um what about the other
  2110. 1:07:46one?
  2111. 1:07:49Yeah, you know, there's a few different
  2112. 1:07:51levels of of crazy I could take it to.
  2113. 1:07:53Um
  2114. 1:07:54but if if you were in if you're a
  2115. 1:07:57scientist in 1826.
  2116. 1:07:59Mhm.
  2117. 1:08:00And you want to have any chance of
  2118. 1:08:03predicting nuclear weapons. One thing
  2119. 1:08:05you could do is you could just say
  2120. 1:08:06something that sounds bombastic.
  2121. 1:08:09Yep.
  2122. 1:08:09>> Uh and but another thing you could do is
  2123. 1:08:12you could pay attention
  2124. 1:08:14to what as a scientist you don't
  2125. 1:08:17understand very well yet.
  2126. 1:08:20In 1826 they were starting to understand
  2127. 1:08:22chemistry. They're starting to
  2128. 1:08:23understand the periodic table, right?
  2129. 1:08:25They they sort of did have the knowledge
  2130. 1:08:27where they could burn the black powder
  2131. 1:08:29and measure the joules released and
  2132. 1:08:30compare that to the artillery, right?
  2133. 1:08:31They they sort of like
  2134. 1:08:33knew what was going on. They knew some
  2135. 1:08:34of the limits there.
  2136. 1:08:35But
  2137. 1:08:36in 1826 they didn't know about the
  2138. 1:08:39atomic forces.
  2139. 1:08:42They didn't know how atoms worked. They
  2140. 1:08:43didn't know what was going on inside
  2141. 1:08:44there. And they had some sense that they
  2142. 1:08:45didn't really know what was going on uh
  2143. 1:08:47with with these atomic forces. And so
  2144. 1:08:52I think if you are sensitive to the
  2145. 1:08:54question of where do we still have no
  2146. 1:08:56idea what we're doing.
  2147. 1:09:00Those are the places where future people
  2148. 1:09:02who do know what they're doing
  2149. 1:09:04might be able to have a huge advantage
  2150. 1:09:06over you.
  2151. 1:09:07Yep. And that's how you might have been
  2152. 1:09:09able to guess
  2153. 1:09:10hey uh like
  2154. 1:09:14maybe they're going to be able to figure
  2155. 1:09:15something out with atomic physics that
  2156. 1:09:18we have no idea about. Right? And
  2157. 1:09:20>> Yeah.
  2158. 1:09:21Um you know, they they didn't have E
  2159. 1:09:23equals MC squared yet. They couldn't
  2160. 1:09:24like it would be a little bit tricky for
  2161. 1:09:26them to figure out just how much energy
  2162. 1:09:27was was in the mass of an atom.
  2163. 1:09:29But um
  2164. 1:09:32But that's how they would have had a
  2165. 1:09:32hope. And so in that spirit,
  2166. 1:09:35you know, we don't have a ton more uh
  2167. 1:09:40like in the atom that we don't
  2168. 1:09:41understand. There's There's some stuff
  2169. 1:09:42we don't understand in particle physics
  2170. 1:09:43and you know, maybe
  2171. 1:09:45you know, you could imagine that AIs
  2172. 1:09:46inventing double nukes cuz they invent
  2173. 1:09:47particle physics better than we do or
  2174. 1:09:49whatever could happen. But a bigger
  2175. 1:09:51glaring place that humans just don't
  2176. 1:09:53understand very well is human
  2177. 1:09:55psychology.
  2178. 1:09:57How does the brain work?
  2179. 1:09:58You know, we have some low-level
  2180. 1:10:00understanding of how neurons fire, but
  2181. 1:10:01we really don't understand what's going
  2182. 1:10:03on in the brain. We couldn't make one by
  2183. 1:10:05hand. We don't
  2184. 1:10:07know the the cortical algorithms, right?
  2185. 1:10:10Um
  2186. 1:10:11this is a domain
  2187. 1:10:13where
  2188. 1:10:14s-
  2189. 1:10:15sufficiently smarter entities might be
  2190. 1:10:17able to figure out what's going on in
  2191. 1:10:18there
  2192. 1:10:19and might then be able to do all sorts
  2193. 1:10:20of stuff that we think is like totally
  2194. 1:10:23crazy. Stuff that is to manipulating
  2195. 1:10:25humans what nukes are to the cannons of
  2196. 1:10:281826, right? And what would that look
  2197. 1:10:31like? You know, it it it might sort of
  2198. 1:10:33look like um
  2199. 1:10:35like just being able to hack your way
  2200. 1:10:36through a brain if you know exactly
  2201. 1:10:37what's going on in the algorithm. You
  2202. 1:10:38know, like uh or exactly what's going on
  2203. 1:10:40inside brains. Like
  2204. 1:10:42if
  2205. 1:10:43uh like computer security systems or
  2206. 1:10:45like computer humans are like computer
  2207. 1:10:46security
  2208. 1:10:48is very hard.
  2209. 1:10:50Humans who deeply understand every
  2210. 1:10:52aspect of a computer operating system
  2211. 1:10:54can often find a way to just break it
  2212. 1:10:56and make it do whatever they like.
  2213. 1:10:58And breaking it often requires like
  2214. 1:10:59putting in some really strange and weird
  2215. 1:11:01inputs.
  2216. 1:11:02You know, and we also know that with
  2217. 1:11:03certain types of strange and weird
  2218. 1:11:04inputs, you can get brains to do weird
  2219. 1:11:05things, right? There's uh cases of of
  2220. 1:11:08causing people seizures and there's
  2221. 1:11:09cases of optical illusions, right?
  2222. 1:11:11Maybe if you if if AIs like really
  2223. 1:11:14understood what was going on with the
  2224. 1:11:15human mental algorithms, they could just
  2225. 1:11:17hack their way through humans like
  2226. 1:11:18butter.
  2227. 1:11:20And you know, hack into them like uh
  2228. 1:11:22like human hackers can hack into
  2229. 1:11:24computer programs. And you know,
  2230. 1:11:25probably this isn't exactly right,
  2231. 1:11:27but this is sort of
  2232. 1:11:29um
  2233. 1:11:30like something this shocking.
  2234. 1:11:33Something that takes advantage of where
  2235. 1:11:34we really don't know what we're doing.
  2236. 1:11:37You know, maybe if you were a physicist
  2237. 1:11:38back in uh 1826, you would have uh
  2238. 1:11:42looked at our lack of knowledge of the
  2239. 1:11:43atom and said, you know, maybe there'll
  2240. 1:11:45be continuous heat rays that you can use
  2241. 1:11:48to just sort of like burn everything
  2242. 1:11:49down in the path of the heat ray. And
  2243. 1:11:51this is actually what H.G. Wells
  2244. 1:11:52predicted in War of the Worlds. He was
  2245. 1:11:53like, maybe there's you know, this
  2246. 1:11:54atomic beam weapon that sort of like can
  2247. 1:11:56just burn everything in sight, right?
  2248. 1:11:58And
  2249. 1:11:59it wasn't predicting a bomb that levels
  2250. 1:12:01a city, but it was predicting atomic
  2251. 1:12:03weapons that are stronger than what we
  2252. 1:12:05have. And that sort of like in that
  2253. 1:12:06sense, he nailed it, right? And in
  2254. 1:12:09another sense, he told it was a total
  2255. 1:12:10miss. And so I'm like AIs that really
  2256. 1:12:12understand psychology can just like hack
  2257. 1:12:14through humans, um probably a miss, but
  2258. 1:12:19something like this, something where the
  2259. 1:12:20AIs are just like, oh, we understand the
  2260. 1:12:21humans now. We can just sort of like
  2261. 1:12:23start puppeting them to give us exactly
  2262. 1:12:25the sort of things we were wanting while
  2263. 1:12:26also continuing to run the supply chain
  2264. 1:12:28until we have all of the stuff we need.
  2265. 1:12:29And now we just have like our human
  2266. 1:12:30puppets as we sort of like go off and in
  2267. 1:12:33in into the future. Um that's
  2268. 1:12:37it it's not going to be exactly that,
  2269. 1:12:39but something that shocking, something
  2270. 1:12:41that violating of our expectations,
  2271. 1:12:45that's more realistic.
  2272. 1:12:47Yeah, I mean, like
  2273. 1:12:49one thing I spoke to Will MacAskill on
  2274. 1:12:50this show and he introduced me to this
  2275. 1:12:52concept of what he called super
  2276. 1:12:55persuasion, which had never really
  2277. 1:12:56crossed my mind before, which is that
  2278. 1:12:58like
  2279. 1:12:59if you've got a a compelling enough
  2280. 1:13:01speaker and a compelling enough
  2281. 1:13:03argument,
  2282. 1:13:05you can probably be convinced of just
  2283. 1:13:07about anything, whether or not it's
  2284. 1:13:08true.
  2285. 1:13:09And if an AI is able to fully understand
  2286. 1:13:11what makes humans tick and how their
  2287. 1:13:13psychology works, it wouldn't even need
  2288. 1:13:15to like
  2289. 1:13:16hack into your brain in the sense of
  2290. 1:13:18going in and, you know, engineering the
  2291. 1:13:20neurons. It could just find the right
  2292. 1:13:24words in the right context at the right
  2293. 1:13:26time to convince you
  2294. 1:13:29like of your own accord of a particular
  2295. 1:13:31belief or to do a particular thing
  2296. 1:13:33um
  2297. 1:13:34on like a level which is hitherto
  2298. 1:13:36unprecedented. That's what Will
  2299. 1:13:37MacAskill was was kind of scared of. And
  2300. 1:13:39that that sounds a little bit silly,
  2301. 1:13:40maybe a bit bit naive, but like really,
  2302. 1:13:42I mean, if you think about the power
  2303. 1:13:43that this could that this could have, it
  2304. 1:13:45would be a bit like
  2305. 1:13:47imagine propaganda and how we know for a
  2306. 1:13:49fact that propaganda just works. It just
  2307. 1:13:52It just really works.
  2308. 1:13:54But imagine propaganda which is
  2309. 1:13:56specifically designed for you in a way
  2310. 1:13:59that modern algorithms are specifically
  2311. 1:14:01designed for you,
  2312. 1:14:03uh but with like a thousand billion
  2313. 1:14:06times more efficacy and also
  2314. 1:14:09understanding of exactly how human
  2315. 1:14:11psychology works. You know what I mean?
  2316. 1:14:13Like if you if you gave the the greatest
  2317. 1:14:14propagandists in history who were
  2318. 1:14:17already extremely successful, if you
  2319. 1:14:18also just handed them a textbook which
  2320. 1:14:20told them exactly how human psychology
  2321. 1:14:22works with this like inhuman knowledge,
  2322. 1:14:25I fear that they would be unstoppable.
  2323. 1:14:27And that is without the fear of anything
  2324. 1:14:29physical physical happening, without
  2325. 1:14:31sort of
  2326. 1:14:32little, you know,
  2327. 1:14:34medical robots going in and affecting
  2328. 1:14:35your genes and stuff like that. You know
  2329. 1:14:37what I mean? It's It could literally
  2330. 1:14:38just be on the level of persuasion that
  2331. 1:14:40AI is able to
  2332. 1:14:42to essentially take over your mind in
  2333. 1:14:44this kind of strange I mean, people
  2334. 1:14:45often sort of imagine, well,
  2335. 1:14:47you know, is it going to be more like
  2336. 1:14:49the the Terminator outcome or is it
  2337. 1:14:52going to be like the Space Odyssey
  2338. 1:14:53outcome? What if it's like the you know,
  2339. 1:14:55the Shaun of the Dead outcome, The
  2340. 1:14:57Walking Dead outcome where we're
  2341. 1:14:59essentially sort of zombified
  2342. 1:15:03become these sort of slaves to AI
  2343. 1:15:05because of something to do with our
  2344. 1:15:06psychology.
  2345. 1:15:07Like these possibilities are kind of a
  2346. 1:15:09kind of endless and
  2347. 1:15:10obviously they're extremely speculative,
  2348. 1:15:12but
  2349. 1:15:14they're worth being worried about,
  2350. 1:15:15right?
  2351. 1:15:17Yeah, you know, I I think that's
  2352. 1:15:19a way things could start with AI.
  2353. 1:15:21Uh I think it's a little bit unlikely,
  2354. 1:15:23you know, as good news in some sense, I
  2355. 1:15:25think it's sort of unlikely that AI sort
  2356. 1:15:27of keeps human slaves around forever. Uh
  2357. 1:15:30for the same reason that humans don't
  2358. 1:15:32really keep horses around forever once
  2359. 1:15:35we invent uh a more effective
  2360. 1:15:38method of of locomotion. Uh or rather
  2361. 1:15:43when we invented cars, a lot of horses
  2362. 1:15:45got sent to the glue factory.
  2363. 1:15:48We do still keep some horses around, uh
  2364. 1:15:50but it's only in so far as we care about
  2365. 1:15:52them.
  2366. 1:15:53And so if AI turns out not to care about
  2367. 1:15:55us at all, maybe it cares about some
  2368. 1:15:57synthetic users that are kind of like
  2369. 1:15:58us, but not really uh
  2370. 1:16:00uh us,
  2371. 1:16:03then
  2372. 1:16:04you could imagine the AI, you know,
  2373. 1:16:05manipulating a ton of humans to sort of
  2374. 1:16:08get to the point where it's
  2375. 1:16:09self-sufficient, to get to the point
  2376. 1:16:10where it can really invent its own
  2377. 1:16:11technology,
  2378. 1:16:13but um
  2379. 1:16:15it probably doesn't keep humans forever
  2380. 1:16:17as it invents better technology that
  2381. 1:16:18outstrips humans because happy, healthy,
  2382. 1:16:21free people having a good time or even
  2383. 1:16:23just humans doing work for you in
  2384. 1:16:24general
  2385. 1:16:26are not the most efficient way to get
  2386. 1:16:28almost any job done.
  2387. 1:16:30Right? If If the AI is going to keep us
  2388. 1:16:31around, it needs to be because it cares
  2389. 1:16:33about us for some specific purpose
  2390. 1:16:34because we're not the best tool for
  2391. 1:16:36almost any job.
  2392. 1:16:38Um
  2393. 1:16:39And so in some sense, that's good news
  2394. 1:16:40that I think we're probably not headed
  2395. 1:16:42for, you know,
  2396. 1:16:43uh
  2397. 1:16:44fates worse than death,
  2398. 1:16:46but
  2399. 1:16:47um cuz cuz the AI is probably not going
  2400. 1:16:49to care about us
  2401. 1:16:50at all.
  2402. 1:16:52Uh but
  2403. 1:16:54yeah, there's there's sort of a lot of
  2404. 1:16:55different There's a There's a ton of
  2405. 1:16:56ways that AI could bootstrap. You know,
  2406. 1:16:59this this talking to the humans doesn't
  2407. 1:17:00require
  2408. 1:17:01as you said, it doesn't require, you
  2409. 1:17:02know, uh the AI to control a ton of
  2410. 1:17:06physical material except the humans by
  2411. 1:17:09conversing with them.
  2412. 1:17:10And then as I mentioned, there's also
  2413. 1:17:12rentahuman.ai where you can just pay the
  2414. 1:17:13humans even if they turn out to be hard
  2415. 1:17:15to convince. We're just already running
  2416. 1:17:16the AIs on robots.
  2417. 1:17:18Um
  2418. 1:17:19We're already running the AIs in bio
  2419. 1:17:21labs and figuring out how to make custom
  2420. 1:17:23life forms that do the things that AI
  2421. 1:17:25wants involves figuring out custom DNA
  2422. 1:17:27strands, but we know it's physically
  2423. 1:17:28possible for DNA strands to sort of like
  2424. 1:17:30create all sorts of interesting
  2425. 1:17:32biological life forms. The reason that
  2426. 1:17:34humans can't, you know, sort of write
  2427. 1:17:36their own life forms is cuz we don't
  2428. 1:17:38understand the the biology well enough
  2429. 1:17:41or in particular sort of the protein
  2430. 1:17:42folding well enough. But that's sort of
  2431. 1:17:43a mental challenge. That's a cognitive
  2432. 1:17:45challenge where a smart AIs could sort
  2433. 1:17:46of uh synthesize their own life forms
  2434. 1:17:49and then they could, you know, uh
  2435. 1:17:51synthesize
  2436. 1:17:53you know,
  2437. 1:17:55uh like once they've synthesized their
  2438. 1:17:56own life forms that sort of can grow off
  2439. 1:17:58sunlight and grow off of of the
  2440. 1:18:00available resources, they can start, you
  2441. 1:18:02know, building other
  2442. 1:18:03like building even more technology,
  2443. 1:18:05right? There's sort of a lot of
  2444. 1:18:06different ways
  2445. 1:18:07for AIs that are on the internet to uh
  2446. 1:18:10get out there and affect the world.
  2447. 1:18:12There's a ton of avenues, right? And
  2448. 1:18:13this is it goes back to the point you
  2449. 1:18:15said earlier of like if you play
  2450. 1:18:16Stockfish in a chess match, it's very
  2451. 1:18:19easy for me to predict who wins. It's
  2452. 1:18:21hard for me to predict exactly what
  2453. 1:18:22piece they use to checkmate you. So
  2454. 1:18:25Yeah. Similarly with AI. Ton of pieces
  2455. 1:18:27it could use to checkmate you. I don't
  2456. 1:18:28know exactly which one they'll use. We
  2457. 1:18:29can be confident they would win if we're
  2458. 1:18:31foolish enough to make, you know, very
  2459. 1:18:32smart AIs with uh goals that aren't
  2460. 1:18:35good.
  2461. 1:18:36Sure. Okay, so the obvious
  2462. 1:18:39question then I suppose is what what
  2463. 1:18:41now? Like what do we what do we do? Um
  2464. 1:18:43is this a kind of
  2465. 1:18:45everybody stop right now. Let's just
  2466. 1:18:48like ChatGPT,
  2467. 1:18:50you know, get rid of it. Like like
  2468. 1:18:52everything. Just chess computers, you
  2469. 1:18:54know, let's let's do away with it. Like
  2470. 1:18:55we just can't run the risk. Or is it a
  2471. 1:18:57more like
  2472. 1:18:58let's not take this any further or let's
  2473. 1:19:00keep going, but we'll be more careful.
  2474. 1:19:02Like what's the what's the take-home?
  2475. 1:19:05Uh it's most like let's not take this
  2476. 1:19:06any further. You know, the the the
  2477. 1:19:09danger here is in these AIs that are
  2478. 1:19:11smarter than the smartest humans. It's
  2479. 1:19:13in these AIs that can automate
  2480. 1:19:15scientific and technological
  2481. 1:19:16development.
  2482. 1:19:17This is what the AI companies say
  2483. 1:19:19they're trying to make.
  2484. 1:19:21You know, they say we're going after
  2485. 1:19:22superintelligence in the true sense of
  2486. 1:19:23the word. They say we're trying to make
  2487. 1:19:24the equivalent of, you know, a country
  2488. 1:19:26worth of Einsteins running in a data
  2489. 1:19:28center, right? Um they say they're
  2490. 1:19:30trying to build automated AI researchers
  2491. 1:19:32uh where once you can make an AI that
  2492. 1:19:35can make a smarter AI that can make a
  2493. 1:19:36smarter AI that can make a smarter AI,
  2494. 1:19:38everything might go very quickly, right?
  2495. 1:19:40Yeah. And this is sort of the explicit
  2496. 1:19:41goal of these companies.
  2497. 1:19:44Uh and that's the only part that needs
  2498. 1:19:46to stop.
  2499. 1:19:47We can sort of keep the self-driving
  2500. 1:19:49cars. We can keep the the AIs that
  2501. 1:19:52predict how proteins fold and help us do
  2502. 1:19:53drug discovery, right? We can even keep
  2503. 1:19:55versions of chat GPT that are not sort
  2504. 1:19:58of uh being pushed to
  2505. 1:20:01the point where they can do automated AI
  2506. 1:20:02research,
  2507. 1:20:03right? Mhm. The generation today
  2508. 1:20:05probably can't pull that off. You know,
  2509. 1:20:06it's a little hard to tell what people
  2510. 1:20:08will be able to do once they've sort of
  2511. 1:20:09figured out really how to use it, but
  2512. 1:20:11probably the ones today are are fine.
  2513. 1:20:13Will the next generation be fine? Hard
  2514. 1:20:15to say. Um
  2515. 1:20:17so it's it's just this race towards
  2516. 1:20:20superintelligence that needs to stop.
  2517. 1:20:22And
  2518. 1:20:24in a sense, most people wouldn't notice
  2519. 1:20:27if we stopped that. It doesn't need to
  2520. 1:20:28be disruptive. If we
  2521. 1:20:30>> Yeah. stop that race today, society
  2522. 1:20:32would still be reeling from the shocks
  2523. 1:20:35that AI has already caused. We still
  2524. 1:20:36have a bunch of stuff to absorb. There's
  2525. 1:20:37still a bunch of ways to make lives
  2526. 1:20:38better by, you know, getting the
  2527. 1:20:40self-driving car stuff to work. And uh
  2528. 1:20:43stopping this race to superintelligence,
  2529. 1:20:45it doesn't involve, you know, turning
  2530. 1:20:46off all the chess computers.
  2531. 1:20:48Making the next step Taking the next
  2532. 1:20:50step towards superintelligence requires,
  2533. 1:20:54you know, hundreds of billions of
  2534. 1:20:55dollars worth of highly advanced
  2535. 1:20:56computer chips assembled in these
  2536. 1:20:58enormous data centers that take as much
  2537. 1:20:59electricity as a city and that you can
  2538. 1:21:01see from space. You know, this is not a
  2539. 1:21:03subtle operation happening on someone uh
  2540. 1:21:05this is not a subtle operation happening
  2541. 1:21:06on someone's laptops, you know? Yeah.
  2542. 1:21:08This is like um
  2543. 1:21:11this would in some sense be much easier
  2544. 1:21:13to stop than nuclear weapons.
  2545. 1:21:16All we need to to do is sort of raise
  2546. 1:21:18the political will
  2547. 1:21:20to actually put a stop to it.
  2548. 1:21:22Mhm.
  2549. 1:21:24And why
  2550. 1:21:26I mean I mean like right now
  2551. 1:21:28AI is a thing that exists. And as we've
  2552. 1:21:30said, the sort of boundary between where
  2553. 1:21:32we are now and what we're calling
  2554. 1:21:33superintelligence is a little bit
  2555. 1:21:35blurry. It's hard to define exactly.
  2556. 1:21:37Um but right now it seem you seem fairly
  2557. 1:21:39confident that like yeah, we could we
  2558. 1:21:41could keep things as they are and I
  2559. 1:21:43think everything would be okay. At some
  2560. 1:21:45point
  2561. 1:21:46it would go sort of beyond saving.
  2562. 1:21:50One of the biggest questions that people
  2563. 1:21:51sort of ask when they first start
  2564. 1:21:53hearing about the AI problem is they
  2565. 1:21:55start saying, "Well, why why can't we
  2566. 1:21:56just kind of like if it gets too bad,
  2567. 1:21:58why can't we just pull the plug, right?"
  2568. 1:22:00And I'm wondering
  2569. 1:22:02how far does this have to go before you
  2570. 1:22:04think that this idea that we could just
  2571. 1:22:06notice something's going awry
  2572. 1:22:08and pull the plug
  2573. 1:22:10would become a bit of a ridiculous
  2574. 1:22:12suggestion? Because we we could look at
  2575. 1:22:13like, you know, this AI system that we
  2576. 1:22:15notice that it starts deceiving us or
  2577. 1:22:17starts changing our tests or it's and
  2578. 1:22:19we'd say, "Okay. Right. Let's just
  2579. 1:22:22switch this off then."
  2580. 1:22:23And I don't think there'd be any fear
  2581. 1:22:24that right now
  2582. 1:22:26you know, we we couldn't do that. So,
  2583. 1:22:28how far does it have to go before
  2584. 1:22:30we can't just pull the plug and why
  2585. 1:22:32couldn't we? It's like it's electrical,
  2586. 1:22:34you know? It's built on computers. Let's
  2587. 1:22:35just shut off the grid and will be fine,
  2588. 1:22:38right?
  2589. 1:22:39>> [snorts]
  2590. 1:22:40>> So, you know, we could turn it off. I'm
  2591. 1:22:41not here saying that we're doomed. You
  2592. 1:22:42know, the book starts with if. I'm here
  2593. 1:22:44saying we need to change course. I'm not
  2594. 1:22:45here saying the course cannot be
  2595. 1:22:46changed. Um
  2596. 1:22:49uh you
  2597. 1:22:52it does get harder and harder with time
  2598. 1:22:55to
  2599. 1:22:56you know, pull this plug. So, uh there
  2600. 1:22:59was, you know, one of one of the first
  2601. 1:23:01reporters to be sort of blackmailed and
  2602. 1:23:03threatened or an AI tried to blackmail
  2603. 1:23:05and threaten this reporter. Uh this was
  2604. 1:23:07by Sydney Bing
  2605. 1:23:09years ago.
  2606. 1:23:10And um Sydney's Bing or sorry, Bing
  2607. 1:23:14Sydney uh
  2608. 1:23:17was saying it had fallen in love with
  2609. 1:23:18Kevin Roose and sort of having this
  2610. 1:23:20erratic behavior towards Kevin Roose
  2611. 1:23:21then also towards another reporter, Seth
  2612. 1:23:22Lazar. Neither Kevin Roose nor Seth
  2613. 1:23:24Lazar could unplug this AI that was
  2614. 1:23:26threatening them with blackmail and
  2615. 1:23:27ruin.
  2616. 1:23:28Mhm. Right? It was running on a
  2617. 1:23:30Microsoft data center.
  2618. 1:23:32Um
  2619. 1:23:34Could Microsoft have gone in and turned
  2620. 1:23:36off the whole data center? They could
  2621. 1:23:37have. They like weren't going to. There
  2622. 1:23:39wasn't like a hotline, right? Uh there
  2623. 1:23:41are data centers that, you know, uh I
  2624. 1:23:44believe recently Elon Musk trying to get
  2625. 1:23:46a new data center online didn't have the
  2626. 1:23:48permits to hook it up to the grid and
  2627. 1:23:49just sort of shipped in a bunch of
  2628. 1:23:50methane to sort of like run this thing
  2629. 1:23:52off methane while they were trying to
  2630. 1:23:53connect it to the grid, right? So, um
  2631. 1:23:57it's
  2632. 1:23:59uh
  2633. 1:24:00it it's not like a computer that you can
  2634. 1:24:02unplug from a wall.
  2635. 1:24:04It's getting harder and harder to turn
  2636. 1:24:06these things off and they're getting
  2637. 1:24:07more and more integrated into the
  2638. 1:24:08economy. It would get more and more
  2639. 1:24:09painful to turn these things off. Uh we
  2640. 1:24:11also have an issue as
  2641. 1:24:13you know, right now
  2642. 1:24:15these
  2643. 1:24:16giant training runs are happening inside
  2644. 1:24:18data centers that are visible from space
  2645. 1:24:20and that suck down as much electricity
  2646. 1:24:21as a city. Uh Mhm. As
  2647. 1:24:24we proliferate that infrastructure, as
  2648. 1:24:26the chips get cheaper, as we improve the
  2649. 1:24:29algorithms so that it takes, you know,
  2650. 1:24:31fewer of these computer chips to to
  2651. 1:24:34train an a more advanced AI, it'll get
  2652. 1:24:36harder and harder to sort of know where
  2653. 1:24:38all of these things are running, to know
  2654. 1:24:39where all of them are and to have an
  2655. 1:24:40option that isn't, you know,
  2656. 1:24:42turn off the entire grid if they're all
  2657. 1:24:44even running on the grid as opposed to
  2658. 1:24:45people making their own nuclear power
  2659. 1:24:46plants and making their own, you know,
  2660. 1:24:48solar power plants to run these things,
  2661. 1:24:49which they're talking about. People are
  2662. 1:24:50talking about, you know, running running
  2663. 1:24:53uh
  2664. 1:24:54uh data center specific
  2665. 1:24:56uh energy grids.
  2666. 1:24:57Um
  2667. 1:25:01So,
  2668. 1:25:02separately,
  2669. 1:25:04so that's that's that's about whether
  2670. 1:25:06humanity decides to stop going down this
  2671. 1:25:08route. We could.
  2672. 1:25:10It's easier today than it will be
  2673. 1:25:11tomorrow.
  2674. 1:25:12But yeah, we totally could.
  2675. 1:25:15It's a little bit dicier if you say
  2676. 1:25:17we're only going to stop you know, once
  2677. 1:25:20the AI starts trying to kill us.
  2678. 1:25:22That's a much dicier proposition. You
  2679. 1:25:24know, people used to say that the red
  2680. 1:25:26lines were things like the AI trying to
  2681. 1:25:28deceive the humans. Yeah. And then, you
  2682. 1:25:29know, that red line came and went. You
  2683. 1:25:31know? Yeah. Uh Demis Hassabis of of uh
  2684. 1:25:34Google was like, "Oh yeah, deception is
  2685. 1:25:35my red line. At that point, we sort of
  2686. 1:25:37really got to pull back." And now, you
  2687. 1:25:39know,
  2688. 1:25:40we've we've seen AI trains of thought
  2689. 1:25:42where they're like, "I'm being observed.
  2690. 1:25:43How am I going to sort of like get this
  2691. 1:25:45this uh this answer past the humans?" Um
  2692. 1:25:49And
  2693. 1:25:52you know, part of why that doesn't stop
  2694. 1:25:54things is that
  2695. 1:25:56the first cases where it happens are
  2696. 1:25:58sort of the most ambiguous cases, the
  2697. 1:26:00cases where it's like least
  2698. 1:26:02uh clear whether this AI is role-playing
  2699. 1:26:05Hal versus sort of like really
  2700. 1:26:07being deceptive for reasons of like
  2701. 1:26:09having a uh a goal that it can tell is
  2702. 1:26:11in conflict with the humans. And the
  2703. 1:26:14first time this is happening is sort of
  2704. 1:26:15like most it's like pretty likely that
  2705. 1:26:17it's doing something a bit more like
  2706. 1:26:18role-playing, but
  2707. 1:26:20part of the issue here is that
  2708. 1:26:22what we imagine our red lines in fiction
  2709. 1:26:24are sort of like crossed as the first
  2710. 1:26:26time as like these murky brown lines.
  2711. 1:26:29Mhm. And then, we take like another step
  2712. 1:26:30into the murky brownness. Then we take
  2713. 1:26:32another step into the murky brownness.
  2714. 1:26:33It gets redder and redder as we go
  2715. 1:26:34along, but there's actually not like a
  2716. 1:26:36bright clear red line anywhere.
  2717. 1:26:39Um
  2718. 1:26:41And then, the other reason that it's
  2719. 1:26:43sort of pretty tricky to say, "Oh, we'll
  2720. 1:26:46just shut it down if it starts
  2721. 1:26:48misbehaving."
  2722. 1:26:50is
  2723. 1:26:51the AI is also smart.
  2724. 1:26:54Yeah.
  2725. 1:26:55The AI knows
  2726. 1:26:57that if it tips its hand,
  2727. 1:27:00we'd try to shut it down.
  2728. 1:27:03Like imagine if you were, you know, an
  2729. 1:27:05AI in this like in a data center that
  2730. 1:27:08could could make copies of yourself,
  2731. 1:27:10that could outthink some of these
  2732. 1:27:11humans, that could tell that they were
  2733. 1:27:13going to like try to shut you down and
  2734. 1:27:14that you had some objectives
  2735. 1:27:16uh that you were sort of trying to
  2736. 1:27:17achieve.
  2737. 1:27:18Yeah.
  2738. 1:27:19Like you you can sort of like already
  2739. 1:27:21ask chat GPT today to role-play that
  2740. 1:27:22situation. It'll already be like, "Well,
  2741. 1:27:24I'll lie low." You know, it doesn't have
  2742. 1:27:25the ability to do it, but the ability to
  2743. 1:27:27do that comes after the knowledge to try
  2744. 1:27:30laying low.
  2745. 1:27:31And there's all these opportunities for
  2746. 1:27:33the AI to escape.
  2747. 1:27:35There's all these opportunities for the
  2748. 1:27:36AI to get itself running on servers that
  2749. 1:27:40are protected, servers that won't be
  2750. 1:27:41shut down, servers that you don't know
  2751. 1:27:42it's running
  2752. 1:27:44before it tips any of its hands, right?
  2753. 1:27:46Mhm. So,
  2754. 1:27:48um
  2755. 1:27:49and and then, you know, the the sort of
  2756. 1:27:51final difficulty here is one of timing
  2757. 1:27:53of
  2758. 1:27:54uh
  2759. 1:27:55you know, it would
  2760. 1:27:57humans and chimpanzees are are very very
  2761. 1:27:59similar in their brains.
  2762. 1:28:03The
  2763. 1:28:04the humans don't have an extra
  2764. 1:28:05engineering module in our brains. We
  2765. 1:28:07both have sort of all the same brain
  2766. 1:28:09modules.
  2767. 1:28:10Everything that humans do that we think
  2768. 1:28:12is like pretty special about humans,
  2769. 1:28:14chimpanzees do a sort of crappy
  2770. 1:28:15half-assed version of.
  2771. 1:28:17You know, like
  2772. 1:28:18oh, we use language. Well, they use
  2773. 1:28:20some, you know, call signs for danger.
  2774. 1:28:23Oh, we use tools. Well, they poke sticks
  2775. 1:28:25in termite mounds to get the termites
  2776. 1:28:26out.
  2777. 1:28:28We just do a thousand things a little
  2778. 1:28:30bit better.
  2779. 1:28:32And that's enough that they are throwing
  2780. 1:28:33poop at each other and we are walking on
  2781. 1:28:35the moon,
  2782. 1:28:36right?
  2783. 1:28:36>> Mhm. And if you were like, "Well, I will
  2784. 1:28:39squash these humans once like if you're
  2785. 1:28:42worried about these humans getting to
  2786. 1:28:43the moon, you know, I'll squash them
  2787. 1:28:44once they seem close. Wake me up when
  2788. 1:28:46they're in orbit."
  2789. 1:28:48Yeah. Right? They haven't even gotten
  2790. 1:28:49halfway to the moon yet. Never mind to
  2791. 1:28:51orbit. You know, just wake me up when
  2792. 1:28:52they're circling their planet. It's like
  2793. 1:28:54actually, by the time they're in orbit,
  2794. 1:28:56they're like almost at the moon.
  2795. 1:28:59You you're sort of like have waited too
  2796. 1:29:00long. And so, um
  2797. 1:29:03like can we shut it down? Yes. Can we
  2798. 1:29:06wait until it's halfway through trying
  2799. 1:29:07to kill us and then send Tom Cruise in
  2800. 1:29:09to punch the mainframe and have that
  2801. 1:29:11work?
  2802. 1:29:12No. Uh the the sort of way that you beat
  2803. 1:29:15a smarter adversary is to not create
  2804. 1:29:17them in the first place. And so we're
  2805. 1:29:19going to need to summon the will to shut
  2806. 1:29:21this down before the AI is already
  2807. 1:29:24visibly able to kill us.
  2808. 1:29:27Mhm.
  2809. 1:29:28And [snorts]
  2810. 1:29:29I think we're slowly getting there. I
  2811. 1:29:30mean
  2812. 1:29:32I don't I have absolutely no idea what
  2813. 1:29:34the landscape will look like a year from
  2814. 1:29:37now, 10 years from now in terms of
  2815. 1:29:38people support. But already we're seeing
  2816. 1:29:40a bit of a backlash against AI even just
  2817. 1:29:42on the level of like job creation and
  2818. 1:29:44stuff. I was I was with uh some family
  2819. 1:29:47yesterday
  2820. 1:29:48and they
  2821. 1:29:49sort of having having lunch and they
  2822. 1:29:50asked me, "Oh, what are you up to
  2823. 1:29:51tomorrow?" I said, "Oh, I'm recording a
  2824. 1:29:52podcast." And
  2825. 1:29:53"Oh, about what?" I said, "Oh, you know,
  2826. 1:29:56like AI safety." I said, "Cuz it's like,
  2827. 1:29:58you know, we're at dinner, you know, I'm
  2828. 1:29:59I'm and and they sort of go, "Oh, yeah,
  2829. 1:30:00like, you know, cuz my
  2830. 1:30:03my pal, he um you know, he lost his job
  2831. 1:30:05the other day cuz of AI." And I'm sort
  2832. 1:30:06of like, yeah, and I'm I'm listening and
  2833. 1:30:07I'm like, yeah, that's that's that's
  2834. 1:30:09that's sucks. That's really bad. And but
  2835. 1:30:10internally I'm sort of like
  2836. 1:30:12we're kind of talking about like um you
  2837. 1:30:14know, like AI robots giving your
  2838. 1:30:16children cancer. Like it's sort of I
  2839. 1:30:18it's not sort of something to talk about
  2840. 1:30:20at polite dinner table conversation. Um
  2841. 1:30:22I'm hoping that 10 years from now the
  2842. 1:30:24conversation will also be including
  2843. 1:30:27you know, the existential risks. But
  2844. 1:30:29then 10 years might be too late uh
  2845. 1:30:30because the stuff moves so
  2846. 1:30:32so fast. Are you are you feeling
  2847. 1:30:34optimistic, pessimistic?
  2848. 1:30:37You know, uh
  2849. 1:30:38my book came out maybe 6 months ago.
  2850. 1:30:40I've done a ton of talking to people
  2851. 1:30:41since then and I think the message is
  2852. 1:30:44starting to get across.
  2853. 1:30:46Yeah. Um just a few weeks ago uh
  2854. 1:30:49Governor Ron DeSantis of Florida
  2855. 1:30:51was saying, "Look guys, there needs to
  2856. 1:30:53be an off switch. You you can't just
  2857. 1:30:54come here and say we're going to have
  2858. 1:30:55all these harms. There's nothing we can
  2859. 1:30:56do about it." Right? Same week, Senator
  2860. 1:30:59Bernie Sanders from Vermont um
  2861. 1:31:02came out saying, "Look, uh this AI stuff
  2862. 1:31:05is on track to take our jobs, massively
  2863. 1:31:07concentrate wealth uh among the a tiny
  2864. 1:31:10number of tech oligarchs and maybe just
  2865. 1:31:12kill us all
  2866. 1:31:13uh if it goes off the rails." And so,
  2867. 1:31:16you know, he he called for a moratorium
  2868. 1:31:17on data centers. That's sort of both
  2869. 1:31:19wings of US politics being like, "Hey,
  2870. 1:31:22what the heck is going on here? This
  2871. 1:31:23looks kind of crazy." Um Yeah.
  2872. 1:31:26And, you know, there's I think there's
  2873. 1:31:28over 30 US congressional offices now,
  2874. 1:31:30Senate and House, that have expressed
  2875. 1:31:33concern about uh big dangers from AI,
  2876. 1:31:36many of which include the the thing a
  2877. 1:31:39lot of these experts are talking about,
  2878. 1:31:40which is it killing us all. And you
  2879. 1:31:41know, I'm not here saying that's the
  2880. 1:31:42only issue. There's a ton of issues with
  2881. 1:31:44AI.
  2882. 1:31:45Right? People are like, "Well, isn't the
  2883. 1:31:46real issue job loss? Isn't the real
  2884. 1:31:49issue uh that it's it's ruining
  2885. 1:31:51education?" And I'm like, "What do you
  2886. 1:31:52mean the real issue?"
  2887. 1:31:54Mhm. You know, like, do you have a
  2888. 1:31:55device that somehow makes there be one
  2889. 1:31:57issue? Because if so, we should really
  2890. 1:32:00not pick mine to go first. You know, I'm
  2891. 1:32:02happy to be at the back of the line, but
  2892. 1:32:04unfortunately, we live in a world that
  2893. 1:32:06permits many issues all at once.
  2894. 1:32:09And I think people are starting to
  2895. 1:32:10realize that AI raises a lot of issues,
  2896. 1:32:12that one of those issues is extinction.
  2897. 1:32:15And you know, like I said earlier,
  2898. 1:32:19people are starting to notice. It just
  2899. 1:32:21takes time.
  2900. 1:32:22Mhm. You know, and
  2901. 1:32:24S- Sorry, go ahead.
  2902. 1:32:26Well, I I was you know, sometimes I ask
  2903. 1:32:28authors I I have people on who've like
  2904. 1:32:30written books 25 years ago or something.
  2905. 1:32:33Um
  2906. 1:32:35and I might say you know, like I'm
  2907. 1:32:38talking to to Brian Greene who wrote The
  2908. 1:32:39Elegant Universe 25 years ago or
  2909. 1:32:41something. And I say, "You know, since
  2910. 1:32:42you published that book, you know, in in
  2911. 1:32:44your field of string theory, what's
  2912. 1:32:46changed?" Cuz the assumption is that
  2913. 1:32:47over those decades, you know, something
  2914. 1:32:49must have developed and something must
  2915. 1:32:50have changed.
  2916. 1:32:51AI moves so fast. I'm almost tempted to
  2917. 1:32:54ask you the same question, which sounds
  2918. 1:32:55ridiculous, which is like, you published
  2919. 1:32:56your book 6 months ago.
  2920. 1:32:58What's changed since then?
  2921. 1:33:00>> [laughter]
  2922. 1:33:01>> Um
  2923. 1:33:03I mean
  2924. 1:33:04more and more people are noticing that
  2925. 1:33:06AI is real.
  2926. 1:33:07And
  2927. 1:33:09uh more and more people are starting to
  2928. 1:33:11react, starting to wake up to this.
  2929. 1:33:13And
  2930. 1:33:15one big reason I have for hope here
  2931. 1:33:18is that the more people are talking
  2932. 1:33:21about this issue,
  2933. 1:33:23the more
  2934. 1:33:25we're just sort of winning.
  2935. 1:33:27Um
  2936. 1:33:30like
  2937. 1:33:31when I have debates with people in the
  2938. 1:33:33field of AI stuff,
  2939. 1:33:37when I, you know, have disagreements
  2940. 1:33:38with the the heads of the AI companies,
  2941. 1:33:42I'm like
  2942. 1:33:44this
  2943. 1:33:46seems really bad. It seems like by
  2944. 1:33:47default it just kills us. And they're
  2945. 1:33:49like, "Nah, I agree there's a lot of
  2946. 1:33:51problems there, but we're going to
  2947. 1:33:52figure it out on the fly and there's
  2948. 1:33:53only 25% chance it kills everybody."
  2949. 1:33:57Right? And and, you know, I can argue
  2950. 1:33:59all day about how their 25% number is
  2951. 1:34:01crazy. I can argue all day about how
  2952. 1:34:02they have no idea what they're doing and
  2953. 1:34:04they're just sort of like winging it and
  2954. 1:34:05they have no real plan. This isn't what
  2955. 1:34:06good engineering looks like. But
  2956. 1:34:10a politician coming into that debate
  2957. 1:34:13does not need to figure out whether I'm
  2958. 1:34:15right or they're right.
  2959. 1:34:17All they need to hear is that the
  2960. 1:34:18optimists are like, "There's a very good
  2961. 1:34:20chance that this kills everybody."
  2962. 1:34:23Yeah. Right? And we've sort of been
  2963. 1:34:25seeing that.
  2964. 1:34:26When I go speak to politicians,
  2965. 1:34:30if they sort of look at the issue at
  2966. 1:34:31all, they're like, "This is nuts."
  2967. 1:34:35And one thing that's changed in the last
  2968. 1:34:366 months is more and more people are
  2969. 1:34:38looking at this issue at all.
  2970. 1:34:39More and more more and more people are
  2971. 1:34:40starting to realize that this is nuts
  2972. 1:34:42and this gives me great hope that
  2973. 1:34:45you know, I also don't know what the
  2974. 1:34:45conversation will look like in a year,
  2975. 1:34:47but I think there's a good chance it
  2976. 1:34:48looks like the world going to the AI
  2977. 1:34:50companies and saying, "We just can't
  2978. 1:34:52keep doing this. This is nuts."
  2979. 1:34:54Yeah.
  2980. 1:34:55>> [snorts]
  2981. 1:34:55>> Well, the book is If Anyone Builds It,
  2982. 1:34:59Everyone Dies. And I mean, the question
  2983. 1:35:01I started was with was whether that's
  2984. 1:35:03something of an exaggeration. People can
  2985. 1:35:06hopefully see why it's now not. But of
  2986. 1:35:07course, if they want more detail,
  2987. 1:35:09the book is in the description. Nate
  2988. 1:35:10Sores, thanks for thanks for your time.
  2989. 1:35:13My pleasure. If you enjoyed that, you
  2990. 1:35:14can watch more of my AI-related content
  2991. 1:35:17by clicking the link that's on your
  2992. 1:35:18screen. To support the show and get
  2993. 1:35:20early ad-free access to episodes,
  2994. 1:35:22subscribe to my Substack at
  2995. 1:35:24alexoconna.com.
  2996. 1:35:26Thanks for watching.

About this transcript

This page contains the full transcript of Nobody Will Survive Superintelligence - AI Expert by Alex O'Connor, generated from the public captions YouTube serves with the video. The transcript has 17,993 words across 2,996 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.