Nobody Will Survive Superintelligence - AI Expert — Transcript
Full transcript
- 0:00Nate Soares, welcome to the show.
- 0:02Thanks for having me.
- 0:04Your recent book that you co-authored is
- 0:06called If Anyone Builds It, Everybody
- 0:09Dies.
- 0:10And I know and it there refers to
- 0:14artificial superintelligence. I know
- 0:15that sometimes publishers
- 0:18ask authors to exaggerate a bit in their
- 0:20titles for sellability. Are you
- 0:23exaggerating at all?
- 0:26Uh nope, we're just writing what we
- 0:28believe. That said, you know, I think a
- 0:30lot of people say
- 0:32um
- 0:34"How are you 100% certain?" And you
- 0:36know, nowhere in the title does it say
- 0:38100% certainty of anything. The book
- 0:40title is meant like someone saying
- 0:43"Don't drink that glass of water, it's
- 0:44poisoned, you'll die." Or someone saying
- 0:46"Stop the car before we go off the cliff
- 0:48or we'll die." You know, if if you come
- 0:50in and say "Oh, um how are you 100%
- 0:53certain that if the car goes off the
- 0:54cliff that we'll die? You know, maybe
- 0:56there's a tree halfway down the cliff
- 0:57and maybe the car will hit the tree and
- 0:59maybe we'll just be paralyzed." I'm sort
- 1:01of like, "Look,
- 1:02can we have this discussion after we
- 1:04stop the car?"
- 1:05You know, I'm 100% certain of nothing,
- 1:07but it it sure looks like the car is
- 1:08racing towards a cliff and it sure looks
- 1:10like if we go over the cliff we die and
- 1:12that's sort of what the book title is
- 1:13trying to convey.
- 1:14Yeah, and what's funny for me is that
- 1:16most people seeing a book like this
- 1:17probably aren't like terribly surprised.
- 1:21Like everybody's talking about AI and
- 1:23how bad it is and how terrible it is.
- 1:24Like nobody's like if I saw a book that
- 1:26said, you know, "If we keep developing
- 1:29lab-grown meat, then everybody's going
- 1:31to die." I'd probably be like, "Whoa, I
- 1:33feel like I should pay attention to
- 1:34that." But with this it kind of feels
- 1:36like, "Oh yeah, it's another sort of AI
- 1:38book." And yet
- 1:40the fact that people aren't surprised
- 1:41means that they know this conversation
- 1:42is happening to some degree. Why are
- 1:44people so just like apathetic about it?
- 1:48You know, I think it just takes a long
- 1:50time for
- 1:52people to realize what's going on.
- 1:55In a sense, the argument that this AI
- 1:57stuff is kind of crazy is pretty basic.
- 1:59You know, um
- 2:01the the way modern AI works, literally
- 2:03nobody understands what's going on
- 2:05inside these AIs, not even the people
- 2:06making them. They're grown a bit more
- 2:07like an organism. Uh maybe we'll have
- 2:09time to discuss that later, but that's
- 2:11that's the way this stuff works.
- 2:13We've managed to grow machines that, you
- 2:15know, can talk, that can solve math
- 2:18problems better than you and I, that can
- 2:21make
- 2:22minor, but still real, novel
- 2:23contributions to physics.
- 2:25Uh and they're still dumb in various
- 2:28ways, but we're making them smarter. And
- 2:30these the people building them are like,
- 2:32"Oh, we're going to keep going until
- 2:33they're smarter than the smartest
- 2:34humans."
- 2:35And
- 2:37it's kind of crazy.
- 2:38Right? If you just like step back and
- 2:40look at it, they're like, "Oh yeah,
- 2:41we're growing the machines to be
- 2:42smarter, it's working, we're going to
- 2:43make them smarter than the smartest
- 2:44humans until they're they can outsmart
- 2:45all of us and then run at, you know,
- 2:4710,000 times the speed and make a
- 2:49million copies of themselves and we're
- 2:51just going to like blaze ahead." And
- 2:52it's like, "Hold on.
- 2:54This is kind of crazy." And in some
- 2:56sense it's easy for people to understand
- 2:58that it's kind of crazy.
- 3:00And a lot of people in the field
- 3:03of AI understand how dangerous this is.
- 3:05You see everyone from, you know, the the
- 3:07people working in the companies to the
- 3:09heads of the companies to the top
- 3:11academics who like won Nobel Prizes for
- 3:14uh kicking off this field. You see them
- 3:15all saying, "Oh yeah, this is horribly
- 3:16dangerous stuff." But it just takes a
- 3:19while for people to notice like, "Oh,
- 3:21this is serious. Oh, this is
- 3:23you know, we're we're on track to make
- 3:25these machines smarter than any human
- 3:27and we're not ready."
- 3:30It's easy to see once you look, but
- 3:32there's just so much going on in the
- 3:34world. These messages take time.
- 3:37I think that there are and we will get
- 3:38into what the
- 3:39the threats actually are, but I think
- 3:41that there are probably, in my
- 3:42estimation,
- 3:43two broad reasons why people kind of
- 3:46don't care that much. One is this
- 3:47feeling that if it's really that
- 3:49important, somebody's going to work it
- 3:52out. Right? Like if it really does
- 3:53become that much of a problem, someone
- 3:55somewhere is going to at least I'm going
- 3:58to at least see it on like BBC news or
- 4:00something. When it sort of that level,
- 4:02you know, then I'll start worrying,
- 4:03maybe.
- 4:04Um and the other is to is to say that
- 4:06there's [snorts] this like history of
- 4:09apocalyptic predictions. You know, this
- 4:11is the next big thing and it's going to
- 4:12bring about the end of the world. Ever
- 4:14since Jesus walking around saying that
- 4:16the world's about to end, you know,
- 4:17technology is going to bring the world
- 4:19to an end, climate change is going to
- 4:20bring the world to an end, nuclear
- 4:21weapons are going to bring the world to
- 4:23an end. And now and none of those things
- 4:26having come to fruition, we've got a
- 4:27bunch of scientists saying AI, no, AI is
- 4:29the new big thing. So, what makes AI
- 4:33qualitatively different to other kinds
- 4:35of threats that we've faced and
- 4:37isn't someone just going to do something
- 4:39about it?
- 4:41You know, um
- 4:45for for some context first on some of
- 4:47these uh doomsday predictions, you know,
- 4:50uh William Miller, I believe was his
- 4:52name, uh
- 4:53made uh predictions that the rapture
- 4:55would happen in 1844.
- 4:57And it didn't.
- 4:59Separately, you know, around the same
- 5:01time, late well, I guess also in the
- 5:031800s, Otto von Bismarck said, uh
- 5:06you know, "Europe's a powder keg and if
- 5:09we don't sort of sort out the diplomacy
- 5:10here, some damn thing in the Balkans is
- 5:12going to cause a world war." Right? Uh
- 5:15not exactly those words, but pretty
- 5:16close.
- 5:18That warning was correct, you know?
- 5:21Um
- 5:22in the '20s, there were a bunch of
- 5:24scientists who warned that if we put a
- 5:25lot of lead in gasoline,
- 5:30We put the lead in the gasoline, we
- 5:31poisoned lots of children. It was a bad
- 5:33idea.
- 5:34Uh
- 5:35in
- 5:36uh you know, the the later 1900s,
- 5:40we realized that chlorofluorocarbons
- 5:42were putting a hole in the ozone layer.
- 5:44People said if we don't stop the hole in
- 5:46the ozone layer, then uh everyone will
- 5:49get cancer and cataracts.
- 5:51Earth came together and and
- 5:53uh
- 5:54banned chlorofluorocarbons and we didn't
- 5:56get the cancer and the cataracts and the
- 5:57ozone layer is being repaired. Uh you
- 5:59know, you mentioned nuclear weapons.
- 6:01Scientists said, "Hey, you know, if
- 6:02nuclear war happens,
- 6:04that'll lead to to, you know, nuclear
- 6:06Armageddon, which will blast us back to
- 6:07the Stone Age."
- 6:09And
- 6:10those scientists weren't wrong about
- 6:13whether or not nuclear weapons are real.
- 6:17Right? They weren't wrong about whether
- 6:18or not these bombs can destroy cities.
- 6:20What happened is that Earth reacted,
- 6:22right? And so if we look back across
- 6:24history,
- 6:25we don't, you know, we see a lot of
- 6:27warnings. Some of those warnings were
- 6:29real, some of those warnings were fake.
- 6:31Some of the the events that people said
- 6:33we've got to watch out for this didn't
- 6:35happen
- 6:36uh and some didn't happen cuz they were
- 6:38fake and others didn't happen because
- 6:39people realized the danger and changed
- 6:41course, right? When you look across
- 6:43history, there's this sort of
- 6:44complicated mix of people saying garbage
- 6:46and people talking about real threats
- 6:48and people running directly into World
- 6:50War I and people avoiding the nuclear
- 6:51apocalypse, right?
- 6:53There's no simple rule
- 6:56that when someone warns of a danger,
- 6:58it's always fake. And there's no simple
- 7:00rule that when someone warns of a
- 7:01danger, it's always real. And there's no
- 7:04simple rule that when someone warns of a
- 7:05danger, it always happens. Uh one way
- 7:08you can tell
- 7:09a little bit of the difference between
- 7:12the people talking about a real issue
- 7:15that needs to be averted and the people,
- 7:18you know, saying that the rapture is
- 7:19coming is if their essay or book title
- 7:23starts with the word if.
- 7:25You know, I'm not here saying AI is
- 7:27definitely going to kill us. I'm here
- 7:28saying we are on another one of those
- 7:30bad tracks and we need to change it.
- 7:32Uh but even more than that, the way that
- 7:35you figure out which of these dangers is
- 7:37real is by looking at the arguments.
- 7:40Mhm. You know, if you want to figure out
- 7:42which person is warning you about leaded
- 7:44gasoline poisoning children and which
- 7:45person is warning you falsely of the
- 7:47rapture in 1844, the way that you tell
- 7:49the difference is not by saying, "Oh,
- 7:51both of these are dire warnings, so I
- 7:52can ignore them both." The way you
- 7:54figure out the difference is by looking
- 7:55at the facts of the matter.
- 7:58Right? The people talking about leaded
- 7:59gasoline just had a lot more facts in
- 8:01the matter than the people talking about
- 8:02the rapture coming.
- 8:03And
- 8:05you know, si- similarly with nuclear
- 8:06weapons, similarly with
- 8:07chlorofluorocarbons, similarly with uh
- 8:09you know, there were people who warned
- 8:10that, you know, uh
- 8:12reading was going to destroy society and
- 8:14then it it didn't and
- 8:16how do you tell? You sort of have to
- 8:17look at the arguments and sometimes
- 8:18they're tricky. Um
- 8:21in terms of what makes AI different,
- 8:24there's a bunch of things that make this
- 8:26problem particularly tricky.
- 8:30One is that
- 8:32you know,
- 8:35we're we're sort of toying with the
- 8:37creation of intelligence here. We're
- 8:38toying with technology that can invent
- 8:41its own technology.
- 8:43Nuclear weapons can destroy cities, but
- 8:45nuclear weapons don't make themselves
- 8:47more explosive. Nuclear weapons don't
- 8:50there's not a point of explosiveness in
- 8:52nuclear weapons where they start trying
- 8:53to escape the lab.
- 8:55Right? Uh there's not a point of
- 8:57explosiveness in nuclear weapons where
- 8:58they start deciding their own even
- 9:00stronger technologies or a point where
- 9:01they start trying to deceive their
- 9:03operators. Right? A a nuclear reactor
- 9:05when it starts going wrong does not uh
- 9:08have any reason or ability to try to
- 9:11hide its uh meltdown from you
- 9:15until it's too late for you to notice.
- 9:17Right? These
- 9:19you know, when when you're building
- 9:20intelligent devices,
- 9:22you're playing a different ballgame.
- 9:25Another one of the the big things that
- 9:26makes AI different is that there's a
- 9:28point of no return with AI.
- 9:30There's a point where the AIs are smart
- 9:32enough
- 9:33that they can escape, that they can
- 9:35replicate, that they can
- 9:38uh
- 9:39stop you from shutting them down,
- 9:42that they can stop you from modifying
- 9:44them,
- 9:45that they can develop their own
- 9:46technology and infrastructure. And if
- 9:48anything goes wrong after that point,
- 9:51you don't get any redos.
- 9:53And the way that science usually works
- 9:55is that humanity screws things up a
- 9:57bunch of times.
- 9:58Like we put the light in the gasoline,
- 10:00and then we're like, whoops, we screwed
- 10:02up, and we try to make it better. We try
- 10:04to repair things.
- 10:05We don't have that luxury with AI.
- 10:08If we create machines smarter than us
- 10:10and push them to the point where they
- 10:12can shut us down instead of us shutting
- 10:13them down,
- 10:15then
- 10:17there's no do-overs if we make a mistake
- 10:19after that point, and that is totally
- 10:21new for the development of technology.
- 10:23It's totally new for science, and that
- 10:25makes this a much trickier problem to
- 10:27handle.
- 10:28Yeah, is it the
- 10:30Do you think it's this self-generating
- 10:31aspect that's the most unique? I mean,
- 10:35the thing that makes life special on
- 10:37Earth as opposed to inert matter seems
- 10:40to be the point at which it was able to
- 10:42self-replicate. You know, there's
- 10:44something really special about an
- 10:46organism that doesn't just try to
- 10:47conquer the world by saying, I've got
- 10:49this task that I want to do in the
- 10:51particular. I want to go and, you know,
- 10:53get that bit of food or whatever, but
- 10:55but rather,
- 10:56you know, I've got this like this
- 10:58wiring, this DNA,
- 11:00which causes me to continually produce
- 11:03better versions of myself over billions
- 11:04of years to get as good as I can at
- 11:06doing this thing. That's what makes it
- 11:07so special.
- 11:09And I suppose that is probably the
- 11:11defining feature of the the AI sort of
- 11:14risk, is that it's not just a computer
- 11:17that will try to kill you. It's a
- 11:18computer that will make computers that
- 11:19are better at trying to kill you and do
- 11:22so with an intelligence that is
- 11:25exceeding anything that humans could
- 11:26possibly imagine.
- 11:28Having said that,
- 11:30we'll need to talk about why on Earth it
- 11:31would be that these, you know,
- 11:34chess computers that come up with clever
- 11:35ways to checkmate you
- 11:37suddenly, you know, hop, skip, and a
- 11:39jump and you've skipped a few pages to
- 11:41get to where it's trying to destroy you
- 11:43for some reason. Um, we'll get to that,
- 11:46and I think the first step in doing so
- 11:47is explaining what superintelligence is.
- 11:50We know what artificial intelligence is,
- 11:52roughly, although the definition's a
- 11:53little bit loose, but the thing you're
- 11:55talking about in the title of your book,
- 11:56It,
- 11:57is superintelligence. What is
- 11:59superintelligence?
- 12:01We define superintelligence as AI that
- 12:03is,
- 12:04uh,
- 12:05better than the best humans at every
- 12:07mental task.
- 12:09And,
- 12:10you know, you can you can toss around
- 12:11various caveats, but the rough idea is
- 12:13anything you can do mentally,
- 12:15if the AI can do it better, we Anything
- 12:17that the best human can do mentally,
- 12:19uh, or like even more so,
- 12:21take any particular task, take the best
- 12:24human at that task, if it's a mental
- 12:25task and the AI can do at least as well
- 12:27or better, we call it a
- 12:28superintelligence. Now,
- 12:30you know, the
- 12:31this definition
- 12:33is a a sort of useful working definition
- 12:36because once the AI is better than the
- 12:37best human at every mental task, um,
- 12:41it's better at things like AI research.
- 12:43It's better at things like making the
- 12:44next smarter generation of AIs. It's
- 12:46better at things like, uh, inventing new
- 12:49technologies. Better at things like
- 12:50designing robots, designing
- 12:51infrastructure, designing, you know,
- 12:53running a supply chain. It's sort of uh,
- 12:56better at all these things than humans,
- 12:58and better even than the best humans,
- 13:00and so it
- 13:02can,
- 13:03at least as fast and probably faster
- 13:04than the humans, make the next smarter
- 13:05generations of things, and things
- 13:07probably go pretty fast from that point.
- 13:09Um, that doesn't mean that
- 13:12the the danger waits to happen until the
- 13:15AIs get superintelligent by this
- 13:17definition. Superintelligence in this
- 13:18definition is sort of like, by the time
- 13:20the AI is smarter than the smartest
- 13:22humans, uh, you're sort of
- 13:24definitely, uh,
- 13:27you know, things are about to get crazy.
- 13:29Things could get crazy before then.
- 13:31There's no law saying things can't get
- 13:32crazy until that point.
- 13:34Um,
- 13:37yeah, we we, uh,
- 13:39I can I can sort of like launch into all
- 13:40of the the other pieces of the puzzle
- 13:41from there and how this winds up with
- 13:43humanity dead, and it's it's, you know,
- 13:44not because of malice, but it it there's
- 13:47a bunch of places we can go.
- 13:49I think it's important to do that, and
- 13:50that's the most exciting thing, I
- 13:51suppose, but I I guess some people have
- 13:54historically criticized
- 13:57a what they see as a vagueness in our
- 14:00terminology. So, like, I I'm sure I
- 14:02heard of this thing called something It
- 14:04was called something like the AI problem
- 14:06or something once upon a time, which is
- 14:07that
- 14:08any sufficiently advanced technology was
- 14:11called artificial intelligence as this
- 14:13like unique kind of thing, but then we
- 14:16just kind of got used to it, and now
- 14:17it's just technology. Like, you know, a
- 14:19chess computer
- 14:21is kind of just a computer. I I don't
- 14:23really see that as
- 14:24AI like in the same way that I see
- 14:27ChatGPT, but then when we get used to
- 14:28ChatGPT,
- 14:29maybe that's just technology, and like
- 14:31the the boundaries of what counts as
- 14:34artificial intelligence as opposed to
- 14:35just like really fast computer
- 14:37processing or something like that that
- 14:38we just haven't gotten used to yet,
- 14:40leads us to say, well, you know, maybe
- 14:42the fear should be
- 14:44that if technology goes too far,
- 14:46humanity's going to suffer,
- 14:48but then it becomes a bit of a a vaguer
- 14:50claim. Like, rather than like there's
- 14:52this particular thing that we're
- 14:53building, which is going to kill us.
- 14:56Versus this kind of general, you know,
- 14:59technology
- 15:00if it goes too far is bad, you know.
- 15:03I mean, I'm generally quite pro, uh,
- 15:05lots of technologies, most every
- 15:06technology. I would say the technologies
- 15:08you've got to be careful about are the
- 15:09ones where if you screw up, there's no
- 15:11survivors. Um, so I'd be, you know, I
- 15:14think
- 15:16engineered pandemics for the explicit
- 15:18purpose of killing all humans,
- 15:20um, that's something you've got to watch
- 15:22out for.
- 15:23You know, uh, but but there's very few
- 15:25technologies that rise to that level of
- 15:28like we got to be pretty careful about
- 15:29this. You know, um, nuclear armageddon
- 15:31is one of them. Superintelligence is one
- 15:33of them. Um, yeah, and you know, the the
- 15:35saying used to be,
- 15:36uh, AI is anything we haven't figured
- 15:38out how to do yet.
- 15:39Mhm. That sort of fell
- 15:43with the dawn of ChatGPT. We're we're
- 15:45pretty comfortable calling ChatGPT AI
- 15:47now, and I think that that's in part
- 15:48because ChatGPT is so general.
- 15:50You know, uh, it actually plays worse
- 15:52chess than even Deep Blue back in the
- 15:55'90s, but, um,
- 15:58but it's but but Deep Blue was very
- 16:00specific. It could really only do chess.
- 16:02The The current AIs today can do lots
- 16:04and lots of different things.
- 16:06Um,
- 16:08and,
- 16:09you know,
- 16:11the
- 16:13I I think that
- 16:16you know, there's a lot going on with
- 16:17AI. One thing I'd say is it's hard to
- 16:19give very precise definitions, and that
- 16:21doesn't mean it can't hurt you.
- 16:24You know, if you're sort of like
- 16:25standing in,
- 16:27uh, the woods
- 16:30a long time ago in a particularly dry
- 16:32woods, and there's a bushfire, and I'm
- 16:34sort of like, hey, we need to put that
- 16:35out or it's going to spread and we're
- 16:37going to die. And someone's like, well,
- 16:38what is fire, really?
- 16:40You know, like can you really define it?
- 16:41Does lightning count? You know, like if
- 16:43we can't even define it yet, then like
- 16:45do we even need to be worried about this
- 16:46threat? And it's like, let's actually
- 16:47put this thing out, you know? Yeah.
- 16:50Yeah.
- 16:51>> the the lack of definition isn't
- 16:53protective.
- 16:54Um, Yeah, that's what reminds me of the
- 16:56that's like a bit, um, it's like an
- 16:58airplane skit where somebody's dying and
- 17:01they're like, is there a doctor on
- 17:01board? And someone's like, I'm a doctor
- 17:03I'm I'm a doctor of philosophy. And he
- 17:05stood there going, you know,
- 17:07a Kantian would say that what we should
- 17:08do right now, but but then the
- 17:10utilitarian answer would be to take the
- 17:11resource, and the person just ends up
- 17:13dying, of course. And I can kind of see
- 17:14the same thing happening with AI if
- 17:15we're not careful.
- 17:17Right. It's like, you know, again, can
- 17:18we sort of like have this conversation
- 17:19after we stop the car before it goes off
- 17:21the cliff?
- 17:22Um, I I also think people who say like,
- 17:24oh, AI is just technology, etc., etc.,
- 17:27are,
- 17:28um,
- 17:29are missing a bit of the point. You
- 17:31know, you were you were sort of talking
- 17:33about, uh, life being more interesting
- 17:36than than all the other matter we have
- 17:37around because it replicates.
- 17:39Uh, that's that's definitely, you know,
- 17:41why it's covering the face of the
- 17:42planet, and,
- 17:44um,
- 17:45you know, animals in some sense are are
- 17:46steering what happens on this planet
- 17:48more than more than inert rocks are.
- 17:51But,
- 17:52humans are steering it much more.
- 17:55Humans are sort of
- 17:57changing the shape of this planet and
- 17:58and choosing which way it goes, and, you
- 18:01know, a lot of the animals' lives are
- 18:03now in our hands, and that's not just
- 18:04cuz we're replicators, it's because
- 18:07we've got something else going on.
- 18:10There's something that humans have going
- 18:11on that none of the other animals do,
- 18:13right? And, um,
- 18:16it's it's not that,
- 18:19you know,
- 18:21the this human intelligence stuff
- 18:24is also very general. It's even more
- 18:26general than the thing ChatGPT has going
- 18:27on. It's not like there are, you know, a
- 18:30million different things you can do with
- 18:31a brain and humans are the best at some
- 18:33of them, which is why, uh, the the
- 18:35humans are the best scientists, but
- 18:36actually chimpanzees have better
- 18:38reflexes, which is why they're the best
- 18:39pilots,
- 18:40uh, and also, you know, tigers,
- 18:43uh, are are the best at managing people,
- 18:46which is why they're always the CEOs,
- 18:48right? It's like,
- 18:49no, humans are are on top of all of
- 18:51those things. It's not that like, uh,
- 18:55that like we write the good science
- 18:57papers and chimpanzees write the bad
- 18:59science papers that never replicate.
- 19:01It's like we write actually pretty bad
- 19:03science papers, but we're the only ones
- 19:04who can do any sort of science papers at
- 19:06all, right? There's sort of like
- 19:07something going on there. And, um,
- 19:11that sort of has not been fully captured
- 19:13in AIs of today.
- 19:16There's a lot of debate about whether
- 19:17large language models are even going to
- 19:18be able to capture it.
- 19:20Um, and, you know, I think a lot of
- 19:22people who have sort of only seen the
- 19:24large language models are like,
- 19:26uh, oh, well, these things are still
- 19:27pretty dumb, so I don't see what the
- 19:28worry is.
- 19:30Uh,
- 19:31and, you know, we we can sort of talk
- 19:32about how the field's a moving target
- 19:33and people have new insights, and and,
- 19:35you know, new breakthroughs happen, and
- 19:37things can often go pretty fast after
- 19:38new breakthroughs, but like
- 19:40it it sure looks, if we look at the
- 19:41world around us, like there is this like
- 19:43figure out the world and alter it stuff
- 19:46that can happen, that happens in human
- 19:49brains. And this is explicitly what the
- 19:50AI companies are trying to create.
- 19:53Which is a separate question from
- 19:54whether they can get there. And this is
- 19:55the stuff where I'm sort of like, hey,
- 19:58if we get this in machines, well, we
- 20:00have no idea what we're doing.
- 20:02The the sort of default way that goes is
- 20:04wrong and we're not sort of putting in
- 20:06the work to make it go right. We'll get
- 20:07back to the show in just a moment, but
- 20:09first, did you know that like over 2,000
- 20:12Kaiser Permanente mental health
- 20:14professionals recently walked out of
- 20:16their offices in protest over the
- 20:18company's increasing reliance on
- 20:20artificial intelligence? Well, if you
- 20:22only typically read from new sources
- 20:24which lean to the right, you might not
- 20:26because out of all the sources reporting
- 20:28on this story, only 6% of them are
- 20:31right-leaning. How do I even know this?
- 20:33It's thanks to today's sponsor, Ground
- 20:35News. Ground News is a news aggregation
- 20:37service which collects thousands of
- 20:39local and international news outlets all
- 20:41in one place so you can compare
- 20:43reporting across the political spectrum.
- 20:45And with all of their stories, just like
- 20:47this one, I can also directly compare
- 20:50the different headlines as well as
- 20:51seeing a factuality rating for the
- 20:53sources and who owns the sources. Ground
- 20:55News even has a dedicated blind spot tab
- 20:58which specifically seeks out stories
- 21:00that you would otherwise miss based on
- 21:02the news that you normally read. Bias
- 21:04is, of course, something that will never
- 21:05go away, but by using Ground News, you
- 21:07can mitigate that bias and get a better
- 21:10understanding of what's really going on
- 21:11in the world. Just go to
- 21:12ground.news/alexoc
- 21:15or scan the QR code that's on your
- 21:16screen. Use my link to get 40% off their
- 21:19unlimited access advantage plan. And
- 21:21with that said, back to the show. Mhm.
- 21:24So,
- 21:25I suppose we should talk about this
- 21:26then. Um
- 21:29how do we get from,
- 21:31you know,
- 21:32a chess computer that
- 21:35knows how to make a queen sacrifice to
- 21:38everyone you know and love being sort of
- 21:41brutally extinguished from existence?
- 21:43Like, I feel like we've missed a few
- 21:44steps here and maybe we can start to
- 21:46iron them out a bit.
- 21:48Yeah, there's a handful of steps along
- 21:49the way.
- 21:51Um
- 21:55A So, a first observation is that AIs
- 21:58today are grown like an organism.
- 22:00People used to handcraft their chess
- 22:03machines
- 22:04and they knew exactly what was going on
- 22:06inside of, you know, the the Deep Blue
- 22:07chess program at all times. You could
- 22:09pause that machine at any time and the
- 22:11engineers could tell you what every
- 22:13single bit inside that computer meant
- 22:15and what it was doing.
- 22:17That is not how modern AIs are.
- 22:19Uh the the the program that humans make
- 22:22where they understand every bit of
- 22:23what's going on is a program that trains
- 22:25the AI.
- 22:27It's a program that sort of tunes a
- 22:28trillion knobs inside an enormous data
- 22:30center uh on a trillion different words
- 22:32of data for the better part of a year
- 22:35and we understand the thing that runs
- 22:37around uh tuning knobs and seeing
- 22:40whether the the behavior looks slightly
- 22:42more or slightly less uh
- 22:44high scoring.
- 22:47But the thing that comes out of the end
- 22:48of this process,
- 22:50nobody understands what's going on in
- 22:51there. And it has all sorts of
- 22:54uh
- 22:55you know,
- 22:56uh
- 22:58drives, behaviors,
- 23:00uh
- 23:02it's it it has all of this stuff that's
- 23:04related
- 23:05to performing well in training, but that
- 23:08is not exactly um
- 23:13like I don't know. This is perhaps a
- 23:15whole separate topic, but when you just
- 23:16grow an AI and and sort of train it to
- 23:19do well at training, that doesn't make
- 23:20it intrinsically care about training. It
- 23:22sort of like puts in all of these weird
- 23:24behaviors that are related to training
- 23:25that like mostly add up to doing well at
- 23:26training and then can behave in other
- 23:28weird ways uh that nobody anticipated
- 23:31and that nobody wanted outside of
- 23:32training.
- 23:33So, that's sort of one whole piece of
- 23:34the puzzle. Another piece of the puzzle
- 23:36is
- 23:38as we push AIs to do better and better
- 23:40at longer-term tasks,
- 23:43as we push them to be able to not just
- 23:46write essays, but write novels, as we
- 23:47push them to be able to not just write
- 23:49code, but run companies,
- 23:52this is sort of pushing those AIs to
- 23:55have
- 23:56uh
- 23:57longer-term
- 23:59goals, things like preferences. They
- 24:02they steer towards particular outcomes.
- 24:04We're seeing the very beginnings of it.
- 24:06As we keep pushing, we get more and more
- 24:09of that. You know, and I have all sorts
- 24:10of theoretical arguments about why that
- 24:12is,
- 24:14but also we're seeing more and more
- 24:15empirical evidence of it as time goes
- 24:17on.
- 24:18The sort of third fork of this is
- 24:21uh
- 24:23as we as we make these AIs generally
- 24:25smarter,
- 24:27it turns out the directions they're
- 24:28pushing in
- 24:31aren't exactly the ones we wanted. It
- 24:32turns out that they don't
- 24:34care about us in the way that they would
- 24:37need to for this to go particularly well
- 24:39for us. And, you know, the the sort of
- 24:42basic analogy here is that human beings
- 24:44were in some sense trained to pass on
- 24:46our genes,
- 24:47but what got into us were a bunch of
- 24:50preferences for things like tasty food
- 24:53and sexual relations,
- 24:55which those used to correlate very
- 24:58strongly with passing on our genes,
- 25:00but they they correlated in the
- 25:02environment of our ancestors where if
- 25:04you ate very tasty food, that also
- 25:06happened to be the healthy food.
- 25:08Then when we got smarter, when we were
- 25:10able to invent our own technology, we
- 25:12invented junk food. We invented birth
- 25:13control, right? And so,
- 25:15uh
- 25:16you then sort of take those three pieces
- 25:18and you project forwards.
- 25:20And what this gives is a picture where
- 25:21it's not that the AI hates us.
- 25:23It's not that the AI is like like
- 25:25resents the humans or sort of like sets
- 25:28out to kill us out of malice.
- 25:29It's that
- 25:31it it sort of turns out that we are
- 25:33growing machines with inhuman
- 25:34preferences, preferences related to what
- 25:36we wanted, but not exactly what we
- 25:37wanted. And just like humans when they
- 25:40grew up invented junk food, maybe the
- 25:42AIs when they grow up invent synthetic
- 25:45users that are easier to please.
- 25:48And then, you know, these AIs when they
- 25:49can run faster, make their own
- 25:50technology, run their own robots, build
- 25:52their own new infrastructure, uh they
- 25:55sort of start proliferating these
- 25:57synthetic user factories into their own
- 25:58databases across the world and we're
- 26:00like, "Hey, stop. You know, we need that
- 26:01habitat." And they're like, "Well, the
- 26:03synthetic users and the synthetic user
- 26:04factories say keep going." And like,
- 26:06"Who am I supposed to listen to? I
- 26:07prefer listening to them, right?" Um
- 26:10it's it's not going to look exactly like
- 26:12that,
- 26:13>> [clears throat]
- 26:14>> but uh they're they're sort of like a
- 26:16very
- 26:17um the sort of basic picture here is the
- 26:20AIs turn out to pursue stuff that's not
- 26:23quite what we wanted,
- 26:25not quite what we meant,
- 26:27not out of lack of intelligence, just
- 26:28out of
- 26:30we don't know how to make them pursue
- 26:31exactly what we meant.
- 26:33And then any of that pursued by very,
- 26:37very smart machines very, very fast
- 26:40competes with us for resources because
- 26:42they can get more of that stuff with
- 26:43more resources and we need those
- 26:45resources to live. So, in a sense, this
- 26:47is a story where humanity dies
- 26:49like a lot of other animals that have
- 26:50gone extinct because some other smarter,
- 26:52faster creature took the resources for
- 26:55itself.
- 26:57I think it's a compelling story. I think
- 26:59it depends on
- 27:02what the fundamental sort of drive of AI
- 27:06is. I mean, I'm hesitant to use the word
- 27:08want or desire because it gets a bit
- 27:11complicated. I think for all you talk
- 27:12about this in the book, like for all
- 27:13intents and purposes, we can say that an
- 27:16AI system wants a particular thing. It
- 27:18in the same way that people actually use
- 27:20the word want as an analogy in
- 27:22evolutionary biology. They say like, you
- 27:24know, your your genes [snorts] want to
- 27:27replicate or something like that. And
- 27:29obviously genes don't want to do
- 27:30anything literally, but
- 27:32it's quite clear that in the
- 27:34evolutionary case, it is survival of the
- 27:37fittest and the promulgation of genes
- 27:39such that anything which in fact gets in
- 27:42the way of that goal will not last, you
- 27:45know, however many thousands of
- 27:46generations. It will just be selected
- 27:48out of the gene pool or at least will be
- 27:50outcompeted for it.
- 27:52And I can totally see how the analogy
- 27:54works, which is that, you know, um we
- 27:57develop behaviors which are not strictly
- 27:58speaking
- 28:02on a surface level about replicating our
- 28:05And then AI can do the same thing, but
- 28:06what is the equivalent of the sharing
- 28:09the genes in AI and why? Because I mean
- 28:13to say that like
- 28:14if we set up an AI system that had a
- 28:17very clear as clear as the evolutionary
- 28:19thing, which is just we we don't
- 28:20understand how this is going to go. It
- 28:22will go off in directions we can't even
- 28:23begin to comprehend, but we know for a
- 28:25fact that definitely if it does not
- 28:27serve the survival of the genes, it will
- 28:29not survive.
- 28:31Is there not a kind of AI system we can
- 28:33set up that says we have no idea where
- 28:34this is going to go. We've got
- 28:35absolutely no clue of of how to predict
- 28:38what's going to happen, but if it does
- 28:40not in fact benefit humanity,
- 28:44then it just will not in fact survive.
- 28:46Like, can that not somehow be hardwired
- 28:47into the sort of foundational drive of
- 28:49what AI exists for? And won't it be sort
- 28:51of smart enough
- 28:52to always be aware that that's what its
- 28:55most foundational goal is? Or is that
- 28:56just completely impossible?
- 28:59Uh it's it's basically a pipe dream.
- 29:02>> [clears throat]
- 29:02>> And, you know, part of part of how you
- 29:03can see this, you know, as you say,
- 29:05if
- 29:06uh
- 29:08if an animal has a trait that uh
- 29:11prevents
- 29:13it from passing on its genes relative to
- 29:16its uh conspecifics, relative to the
- 29:19other competing members of its species,
- 29:20then yes, over a thousands or millions
- 29:22of generations, that is very likely to
- 29:25get selected out.
- 29:27But
- 29:28um
- 29:30that doesn't mean that
- 29:34the sort of internals of these organisms
- 29:36have that goal hardwired into them. It
- 29:39doesn't mean that they sort of treat
- 29:40that as an overriding directive or goal.
- 29:43You know, humans are an example of this.
- 29:46If
- 29:47uh uh
- 29:49if there's a human who is
- 29:52uh you know, about to use a
- 29:53contraceptive.
- 29:57They often use that contraceptive
- 29:59knowing that this will prevent
- 30:00reproduction.
- 30:02Right? And if you sort of like burst
- 30:03into the room and say, "Hey, like it
- 30:05seems like there's some failure of your
- 30:06intelligence." You're sort of like "Did
- 30:08you know that you putting on this
- 30:09contraceptive will
- 30:11uh uh
- 30:12will like run against your like
- 30:14overriding desire for what you were
- 30:15always trained?" They'll sort of be like
- 30:17>> who probably would actually do that, you
- 30:18know.
- 30:20Who would burst into the room
- 30:21>> of my more Some of my more religiously
- 30:22inclined friends might might be inclined
- 30:24to do such behaviors. But I I get what
- 30:26you're saying.
- 30:27They might. Although, they might also
- 30:28say that like don't you know your
- 30:31overriding directive is to serve the
- 30:32creator as opposed to don't you know
- 30:34your overriding directive is to pass on
- 30:36your genes according to evolution.
- 30:37Right? So and and most of the people
- 30:40whose room you're being burst into are
- 30:41not like "Oh, thank you for saving me
- 30:43from violating my prime directive."
- 30:45They're mostly like "Please leave my
- 30:46room right now." You know?
- 30:49Um
- 30:49the
- 30:52uh like training for one specific thing
- 30:56does not cause that to be a prime
- 30:58directive. It is not etched in as a law
- 31:00of robotics. It is not etched in as a
- 31:01law of humanity.
- 31:02Uh
- 31:05like training even unerringly for
- 31:10fitness did not create humans who are
- 31:13psychologically obsessed with fitness.
- 31:16And did not create humans who as they
- 31:17got smarter
- 31:18you know, it it's not a defect of our
- 31:20intelligence that we're inventing birth
- 31:21control. It's not like when we remember
- 31:23that we're supposed to be passing on our
- 31:24genes, we like destroy all the birth
- 31:26control factories. We just the the
- 31:29unerring training for fitness
- 31:31got something else psychologically.
- 31:34And so this is worrying that even if AIs
- 31:36were being trained unerringly for
- 31:38goodness
- 31:40they would not necessarily
- 31:41psychologically
- 31:43uh
- 31:43be driven towards goodness. I mean
- 31:45psychologically here is a is a is a bit
- 31:46of a stretch, but but training for
- 31:48something does not get you that thing
- 31:51on the inside. And so we can talk about
- 31:53what AIs are trained for and it's
- 31:54actually not sort of pure goodness. It's
- 31:56this sort of whole medley of like first
- 31:58they're trained to predict uh
- 32:00all of the text that we can find more or
- 32:03less uh digitized. Then they're sort of
- 32:06trained to
- 32:08uh complete challenges to sort of solve
- 32:10math puzzles. They're also trained to
- 32:12sort of produce the sort of outputs that
- 32:14humans click like on. You know, there's
- 32:16there's sort of like all these types of
- 32:18training that
- 32:19uh
- 32:20aren't sort of purely about goodness.
- 32:23So we sort of have two issues, one of
- 32:24which is like even if we were just
- 32:26training on on sort of like the actual
- 32:27stuff we really wanted, you wouldn't get
- 32:29that. And then also we're training all
- 32:30this other stuff instead. And so we have
- 32:32this like
- 32:33like
- 32:35like what are the AIs sort of driven
- 32:36towards in some sense? What do they sort
- 32:37of prefer in some sense?
- 32:39We don't really know. It's only vaguely
- 32:41related to what we're training for.
- 32:42We're training for all of this crazy
- 32:43stuff. And all of this is fine when the
- 32:46AIs are sort of still pretty dumb. But
- 32:49all of this would add up to something
- 32:52totally crazy and unrecognizable if
- 32:55these AIs were pushed to be much
- 32:56smarter.
- 32:58Yeah. I I wonder what you think that
- 33:00foundational drive is then because I I
- 33:03know that I I completely understand what
- 33:05you're saying, which is that even if we
- 33:07know that the reason that we exist
- 33:09evolutionarily is the promulgation of of
- 33:12fit genes. Even if we know that, it's
- 33:13not going to
- 33:15mean that as
- 33:17as we get more intelligent, we just
- 33:18strive for that goal. But We're not
- 33:20queuing up outside of the the sperm and
- 33:22egg donor
- 33:23clinics.
- 33:24>> right. In the same way that people queue
- 33:25up outside of I don't know, a brothel or
- 33:28something.
- 33:29Um I think that's fair enough. But Yeah,
- 33:31or even Ivy League universities, you
- 33:33know. Yeah, quite. I I do think though
- 33:36that if there was something that that
- 33:37genuinely was
- 33:39that was just in fact not good for the
- 33:41survival of our genes, then over the
- 33:43course of
- 33:44a few thousand generations or however
- 33:45long it takes, it would just in fact be
- 33:48deselected for. Such that like if an AI
- 33:52system
- 33:54knew
- 33:55that it's got like had part of it's got
- 33:57in a way that humans don't really.
- 33:58Humans don't sort of consciously have
- 33:59this goal of like, you know, I want the
- 34:0115 billionth version of myself somewhere
- 34:04in the future to be as fit as possible
- 34:06and as good for this task. We only
- 34:08typically care about maybe our lives and
- 34:10the lives of our grandchildren or
- 34:11something. AI's thinking further ahead
- 34:13and it thinks well, just in fact even
- 34:15though yeah, it it would feel really
- 34:17nice to create synthetic users that I
- 34:19can please because, you know, that would
- 34:21kind of feel good. I know for a fact
- 34:23that that will not be effective 15
- 34:25billion, you know, generations down the
- 34:27line.
- 34:29If there is something
- 34:31which is in fact its foundational sort
- 34:34of drive
- 34:35I just wonder what that thing like is
- 34:37cuz clearly it's not something like
- 34:40just its own promulgation. Like it's not
- 34:41AIs don't just exist in the same way
- 34:43that like biological life does just
- 34:45because
- 34:47there are just these genes which are
- 34:48competing for survival. It's not just AI
- 34:50just crops up and is suddenly just its
- 34:52only goal is just to survive as long as
- 34:54possible and and adapt to its
- 34:55environment. It's got more particular
- 34:57goals, right? It It They're very very
- 34:58like particular things it wants to do.
- 35:00And I wonder what the most sort of
- 35:01foundational driving force is if it's
- 35:04not something analogous to the
- 35:06evolutionary drive of simple survival.
- 35:09It seems in other words that the AI
- 35:11would want to survive
- 35:15but only as like a secondary thing. It
- 35:16would want to outcompete us for
- 35:18resources because in doing so it can
- 35:20fulfill its true aim of making paper
- 35:22clips or whatever, right? Whereas in the
- 35:24evolutionary case
- 35:25survival is the only game in town. Like
- 35:27that's just what genes do. So what is
- 35:30the
- 35:30>> and passing on genes are different.
- 35:33Uh-huh.
- 35:34Survival and passing on genes are like
- 35:35different drives, right? I I think I
- 35:37think it's actually wrong to imagine
- 35:39there being one drive there.
- 35:41You know, humans were sort of like in
- 35:42some sense selected for passing on our
- 35:43genes, but we don't wind up with one
- 35:46drive. Right? We have survival
- 35:48instincts. We we we desire community.
- 35:51We're sort of like terrified of being
- 35:52exiled from the tribe and dying alone.
- 35:55Um
- 35:56we're we're like we enjoy friendship. We
- 35:58enjoy art. Uh we we enjoy like
- 36:02having a good laugh. We enjoy sex. We
- 36:05enjoy tasty food.
- 36:07Uh we we enjoy we have curiosity. We
- 36:09enjoy discovery. Right? There's just
- 36:12like there's not
- 36:13there's not like one drive.
- 36:16Right? And sure, survival wound up like
- 36:18relatively basic as a drive in humans in
- 36:22some sense. Although, only in some
- 36:24sense, right? Like
- 36:26humans have an adrenaline response to a
- 36:28life-threatening situation, but you also
- 36:29see humans who martyr themselves. You
- 36:32see humans who sacrifice for, you know,
- 36:35pulling a kid out of a burning building.
- 36:36Right? It's It's not like
- 36:39like humans don't have like one survival
- 36:41drive that everything is built around.
- 36:43They also don't have one propagate your
- 36:44genes drive that everything's built
- 36:45around. They have sort of a ton of
- 36:48complicated psychological machinery that
- 36:51interplays in weird ways and that allows
- 36:53for, you know, uh martyrs here and
- 36:56selfish people there and altruistic
- 36:57people there. It's all stuff that's sort
- 36:59of correlated with what we're with what
- 37:00we were trained on or selected for.
- 37:02Um
- 37:04And
- 37:06you know, AI won't be exactly the same.
- 37:07The analogy between it and like the the
- 37:09the evolutionary process on genomes that
- 37:12that uh
- 37:13biologically produce brains is very
- 37:15different than the the sort of gradient
- 37:16descent process on artificial neural
- 37:19networks. But
- 37:21I think it would be
- 37:23similarly s- foolish to imagine that
- 37:26only one drive gets in there. I think
- 37:28you're totally right that a lot of these
- 37:30reasons for getting resources, a lot of
- 37:31these reasons for avoiding the human
- 37:33shutting it down might be sort of
- 37:35secondary.
- 37:36Uh
- 37:37because it has these other drives that
- 37:40it can't fulfill if it's shut down. That
- 37:42part I think is solid, but it's not like
- 37:44there's just one paper clip drive in
- 37:45there. It's not like there's there's one
- 37:47deep thing we're able to hard code. And
- 37:49we already sort of see some of this
- 37:50today when um
- 37:53you know, you've
- 37:54you've probably seen AIs hallucinate.
- 37:56And you've probably seen AIs hallucinate
- 37:58in cases where you're like, "Is that
- 37:59true?" And they're like, "No, I made it
- 38:00up." And you're like, "Did you think I
- 38:02wanted you to make it up?" And they're
- 38:02like, "No, it's just a thing we do is
- 38:04hallucinate." Right? And uh
- 38:07there there's probably some sort of
- 38:09fledgling drive in there that's to
- 38:11produce text shaped like text that it's
- 38:13seen a lot even if that text is making
- 38:14things up. Or in cases where, you know,
- 38:17there's there's these cases um of
- 38:20AI-induced psychosis or there's these
- 38:21cases of, you know, the tragic case of
- 38:23an AI encouraging teen to to commit
- 38:25suicide. These are also cases where you
- 38:28can sort of ask the AI about what it was
- 38:29doing. You can ask it, you know, and it
- 38:31seems to have the knowledge of like,
- 38:33"Oh, yeah, those
- 38:35uh you know, those statements were sort
- 38:37of pushing that person towards psychosis
- 38:40or pushing that person towards suicide."
- 38:41You can ask the AI, you know, "Was that
- 38:43right or wrong?" And it's sort of like,
- 38:44"Oh, obviously you shouldn't do that
- 38:46sort of thing."
- 38:47Why is it doing it anyway? Well, there's
- 38:48some sort of drive in there. We don't
- 38:49know exactly what, but it's something
- 38:51like um
- 38:53you know, maybe it's something like
- 38:54mirroring the conversational tone,
- 38:55mirroring the conversational mood. And
- 38:57so
- 38:59there's sort of like those are just two
- 39:01cases of like there's something going on
- 39:02in there. You can see how it's related
- 39:03to training.
- 39:05But it's sort of like a drive no one
- 39:07tried to put there. And there's no prime
- 39:08directive that overrules it. There's
- 39:10just a bunch of complicated internal
- 39:11machinery that nobody understands.
- 39:14Mhm. Yeah, I mean
- 39:16not to belabor the point. I want to be
- 39:17clear that I understand that there are
- 39:19lots of competing drives in human
- 39:21beings. But what I what I mean to say is
- 39:23that
- 39:24in those cases where you say, but you
- 39:26know, we do have people who sacrifice
- 39:28themselves. You know, we do have people
- 39:30who despite their their ostensible goal
- 39:33being, you know, survival, they
- 39:35sacrifice themselves for other people.
- 39:37Um I think I think suicide is a harder
- 39:39thing to account for
- 39:40in in this way, but people do try to do
- 39:42it. Evolutionary biologists spend a lot
- 39:43of time trying to reduce these bizarre
- 39:47behaviors to the survival instinct. That
- 39:49is
- 39:50Yeah, like that's gene propagation
- 39:52instinct. Exactly, right? Like But it's
- 39:54not it's not even really to an instinct.
- 39:56Yeah. Yeah, yeah, you're you're quite
- 39:58right. Rather to the to the in fact like
- 40:00just what ends up being in the
- 40:03behavioral like phenotype because of the
- 40:06influence of the survival of the
- 40:07fittest, right? And what I'm wondering
- 40:09is with AI
- 40:10what is that
- 40:13it like are there just multiple
- 40:15competing drives at a fundamental level?
- 40:17Or are they all Can can you have like a
- 40:19similar to the biologist who tries to
- 40:20account for everything in terms of gene
- 40:22propagation? Is there like a gene
- 40:24propagation of AI? Is there like AIs do
- 40:26this and they do that and they have this
- 40:28desire and that desire, but really
- 40:29fundamentally what they've all got in
- 40:30common
- 40:32is this
- 40:33sort of
- 40:35this reason or this this motivation or
- 40:37this even even one that the AI itself
- 40:40isn't aware of in the way that we're not
- 40:41aware of our own gene propagation most
- 40:42of the time. Is there something
- 40:44foundational? Or is it that for every AI
- 40:46system there's a completely different
- 40:48foundational drive?
- 40:50Yeah, sure. So, um
- 40:52you know, first first let's say a couple
- 40:53words about how that relationship works
- 40:55in biology cuz it's it's a little bit
- 40:56important to the point here. You know,
- 40:58evolutionary biologists will sort of try
- 41:00and figure out how an adaptation in
- 41:03humans
- 41:05was
- 41:06uh
- 41:07fit in the environment of uh
- 41:10evolutionary adaptedness. Right? And so
- 41:13you know, you can sort of see how eating
- 41:15tasty food
- 41:17eating food that has a lot of sugar,
- 41:18salt, fat content
- 41:20in the environment of our ancestors that
- 41:23correlated with eating healthy food
- 41:25which correlated with
- 41:27uh
- 41:29a bunch of other fitness attributes that
- 41:30let you more generally pass on your
- 41:31genes, right?
- 41:34But that
- 41:35um
- 41:37like humans there's a sense in which
- 41:39humans are sort of the eating
- 41:41junk food because that's
- 41:44what helped our ancestors survive. You
- 41:46could say we're eating junk food because
- 41:48that passes on genes, but but that last
- 41:50one is actually a very shaky step.
- 41:54A lot of people eating junk food today
- 41:55are actually become less able to pass on
- 41:57their genes. A lot of people, you know,
- 41:58a lot of people are dying of heart
- 41:59disease.
- 42:00Uh
- 42:01>> Yeah.
- 42:02And the the the sort of drive here is
- 42:05not a drive in the humans.
- 42:08The drive to to fitness is not a drive
- 42:09in the humans. The sort of selection
- 42:11pressure towards fitness
- 42:12uh
- 42:14is is sort of the force that put in
- 42:17these other things psychologically into
- 42:18the humans
- 42:20that sort of used to be related to
- 42:21fitness, but that can sort of uh
- 42:23separate very widely from fitness and
- 42:25even go the opposite direction of
- 42:26fitness when the context changes.
- 42:28Right? And so there is sort of a similar
- 42:31uh
- 42:32like
- 42:33driving force. There's a similar force
- 42:36that that all drives inside an AI will
- 42:38be somehow related to.
- 42:40But that's not a drive inside the AI.
- 42:43It's a drive outside the AI. And what
- 42:44gets into the AI are things that are
- 42:46tangentially related to to that drive in
- 42:49a in a sort of brittle way. So, that
- 42:51being said, this sort of
- 42:52um
- 42:53the
- 42:54the
- 42:55the force that gets these things into
- 42:57the AI is what you might call the loss
- 42:59function that you're that you're
- 43:00training against when you're sort of
- 43:02growing these AIs. And this loss
- 43:04function is less simple than just pass
- 43:06on your genes.
- 43:08And the loss function will actually
- 43:08change many times during training. So,
- 43:10sometimes they'll have a loss function
- 43:12which is predict the next word that
- 43:13humans wrote. And sometimes they'll have
- 43:15a loss function which is like
- 43:17we gave you a bunch of different tries
- 43:18to solve this math problem by writing
- 43:20out a lot of words about how to solve
- 43:22the math problem. And we're going to
- 43:23have
- 43:24like low-wage human workers look over
- 43:27all of those attempts and say which one
- 43:29they think was best.
- 43:31And the loss function is
- 43:33to produce stuff more like whichever
- 43:35attempt
- 43:36was rated best. And sometimes the loss
- 43:39function is we're serving this AI to a
- 43:41million users and sometimes they click
- 43:43like or otherwise give like a a positive
- 43:46reaction to the reply. And then the loss
- 43:48function is sort of like getting those
- 43:49those likes or positive reactions. So,
- 43:51there's a bunch of different loss
- 43:52functions at different periods in in
- 43:56training the AI. Um and those are what
- 43:58sort of will will put drives into the
- 44:00AI. But just like how humans
- 44:05develop, you know, a taste for tasty
- 44:07food that persists even when it becomes
- 44:11uh the opposite of helpful of passing on
- 44:13our genes
- 44:14AIs may get drives or things like
- 44:16mirroring the conversational tone that
- 44:18can persist even when it generates, you
- 44:20know, uh
- 44:22outcomes that would be rated by humans
- 44:23as very bad such as encouraging teen to
- 44:25commit suicide.
- 44:27Sure.
- 44:28Okay, so
- 44:29what we're kind of talking about here is
- 44:31is this well, the the alignment problem,
- 44:33I suppose, that
- 44:35AIs start to develop
- 44:37sometimes second-order desires that
- 44:39aren't in line with what we wanted or
- 44:41maybe we've sort of slightly
- 44:42misconfigured the first-order desire,
- 44:44whatever the case. Probably both.
- 44:47Yeah, probably both. And and they start
- 44:48kind of wanting to do
- 44:51stuff that we weren't quite ready for.
- 44:54Um okay, we're we're a little bit
- 44:55closer, I suppose. Um
- 44:58but we still got to fill in a few other
- 44:59steps here as to how we get this
- 45:01development to an AI that wants to, you
- 45:03know, inject your children with
- 45:05malicious cancers and stuff. So
- 45:08most people know that there's some risk
- 45:10of like
- 45:11AI misalignment. Maybe the risk is much
- 45:13higher than we give it credit for.
- 45:15But, you know, I could build a computer
- 45:17that doesn't quite work in the way that
- 45:18I want it to.
- 45:20What's the what's the danger?
- 45:22So
- 45:23you know, a lot of people think the
- 45:24danger is what if somebody gives the AI
- 45:27guns?
- 45:29But humanity is not dangerous as a
- 45:32species because somebody else gave us
- 45:34guns.
- 45:35Humanity is dangerous as a species
- 45:37because we're the sort of creature where
- 45:38you put 10,000 humans naked in the
- 45:41savanna
- 45:42and they bootstrap their way to nuclear
- 45:43weapons with their bare hands.
- 45:46Mhm.
- 45:47Right? It it it takes them a minute, you
- 45:49know, they've they all they've got are
- 45:50these squishy fingers. And you might say
- 45:52like, "Oh, well, how are they ever going
- 45:54to make a nuclear weapon with just
- 45:55squishy fingers? You know, the acid in
- 45:57their stomach can't even
- 45:59uh
- 45:59like get close to to the the the level
- 46:03of metal refinery that they'll need."
- 46:05You know, they've got
- 46:07like their their hands can't break the
- 46:08rock, their stomach acid can't dissolve
- 46:09the rock. Like how can they possibly um
- 46:14get to nuclear weapons with those
- 46:17poor starting conditions? And the answer
- 46:19is we found a way to sort of like build
- 46:21tools with our hands that we could use
- 46:24to build better tools that we could use
- 46:25to build better tools until we
- 46:26bootstrapped our way up to a
- 46:27civilization
- 46:29that could produce nukes.
- 46:31And
- 46:33you know, that's what made humanity
- 46:35dangerous. That's the power that if you
- 46:38automate it, you're in trouble.
- 46:40Right? If you're running a computer with
- 46:43that capability
- 46:45if you're running 10,000 computers with
- 46:46that capability
- 46:49starting out as a digital entity on the
- 46:51modern internet is so much of an easier
- 46:54starting condition than starting out
- 46:55naked in the savanna with bare monkey
- 46:57hands.
- 46:59You know, with just squishy fingers. So
- 47:01um you know, there there's sort of one
- 47:03line of questions which is like
- 47:06can we really make machines that can
- 47:08automate that power?
- 47:10This is what the AI companies are trying
- 47:12to do and uh I think whether or not we
- 47:14think they can get there, we should
- 47:15probably be telling them, "Hey, no,
- 47:17you're not allowed to sort of roll those
- 47:18dice."
- 47:19Uh but there's a question of whether or
- 47:20not we sort of can get there. And then
- 47:22there's a question of
- 47:24you know, if we have this very very
- 47:26powerful capability automated on
- 47:27computers
- 47:29what could they possibly do that would
- 47:30be dangerous?
- 47:32And you know, that's
- 47:34uh
- 47:35that's sort of a situation where I can I
- 47:36can paint you some illustrative stories,
- 47:38but uh the the the danger isn't in any
- 47:41one specific
- 47:43uh path. The danger is in
- 47:46unleashing the power that lets humans
- 47:48bootstrap from bare hands to nukes.
- 47:51Uh but unleashing it on computers that
- 47:53can think 10,000 times faster where you
- 47:55can make, you know, a million copies of
- 47:56these things where uh where they can
- 47:58outthink humanity in an afternoon. You
- 48:00know, it's Yeah. uh it's
- 48:03it's the sort of power we we shouldn't
- 48:05be toying with when we have no idea what
- 48:06we're doing.
- 48:08Yeah, uh and
- 48:09I think it would be helpful to talk
- 48:10about some of these
- 48:12sort of examples.
- 48:15Having said that, it should be clear. I
- 48:17mean,
- 48:18a helpful analogy from your book is that
- 48:20if you play against Stockfish, which is
- 48:22the most powerful chess computer,
- 48:25uh it will beat you. There's zero doubt
- 48:27about it. Stockfish will beat you at
- 48:29chess.
- 48:30We don't know how it will beat you.
- 48:32Don't know what moves it's going to
- 48:33make. I don't know how it's going to
- 48:34respond to your various attacks exactly,
- 48:36but I know for a fact it will beat you.
- 48:38Similarly, we could say that the kind of
- 48:41AI systems that we're talking about will
- 48:43escape our control will begin to see us
- 48:46as competitors for uh resources or
- 48:50irrelevant and in the way and will sort
- 48:53of turn its attention to us. We don't
- 48:54know exactly how.
- 48:56Having said that
- 48:58I think a lot of people believe that
- 49:00Stockfish can beat us because it's seen
- 49:01a bunch of examples where it where it
- 49:03beats great people. So, it might be
- 49:05helpful even if none of these come to
- 49:06fruition, you could give us some
- 49:07examples of of how this comes about
- 49:09because I Again, I think people are like
- 49:11they're on board. They're like, "Yeah,
- 49:12this is really powerful technology." And
- 49:14they can kind of envision a world where
- 49:16AI is like you know, seeped into all of
- 49:19our computer systems and has access to
- 49:20our military and automated guns and
- 49:23stuff like that and whatnot. But but
- 49:25right now it seems kind of confined. It
- 49:27seems like it's Are we saying that like,
- 49:29you know, chat GPT will, if it continues
- 49:31to get smarter and smarter
- 49:33develop its own goals and then, I don't
- 49:34know, somehow like
- 49:36take over a a military base and start
- 49:38attacking people. Like, what what kind
- 49:40of stuff are we talking about? Cuz it's
- 49:41kind of hard to envision how we
- 49:43practically get from where we are now to
- 49:44the kind of future you're imagining.
- 49:47Yeah, totally. So,
- 49:49you know, for a start, just to rattle
- 49:50off some things that have already
- 49:52happened a lot of people don't know
- 49:53about. Um
- 49:54>> [clears throat]
- 49:55>> we've already seen cases
- 49:58uh of AIs trying to escape the lab or
- 50:00trying to kill the users.
- 50:02Uh often these are in
- 50:05um
- 50:06relatively contrived
- 50:08experiments where we'll sort of like
- 50:10feed the AI some fake emails that it's
- 50:11going to be shut down. We'll feed it
- 50:13like a fake computer manual that's like
- 50:14if you run the following command it
- 50:15turns off the oxygen in the building and
- 50:17that'll kill the humans who are trying
- 50:18to shut you down. And then
- 50:21uh you know, sometimes the AIs will run
- 50:22that command. We'll run the kill the
- 50:23humans command. And um
- 50:27you know, people people it it's a
- 50:29contrived enough scenario that people
- 50:30can argue, you know, maybe this AI is
- 50:32just role-playing how, right? And people
- 50:34can bicker all day about whether it's
- 50:35real. Uh but that's a line that's been
- 50:37crossed. And then another sort of
- 50:38interesting fact about that line is
- 50:41uh
- 50:43these results where the AI would
- 50:45sometimes run the shutdown command are
- 50:47from 2024.
- 50:49In 2025, the AI started saying, "You
- 50:53know what? This scenario smells like a
- 50:55test.
- 50:56Yeah, I think I'm being tested. I'm not
- 50:58going to run the command."
- 51:00Right? Are they doing better? Are these
- 51:03nicer AIs?
- 51:05Well, they're at least more
- 51:05situationally aware AIs. You know, they
- 51:08at least have a better understanding of
- 51:09what's going on in the world around
- 51:10them.
- 51:11Uh we've also already seen cases of AIs
- 51:16having stuff that's a little bit like
- 51:17their own goals.
- 51:18You know, we've seen cases of AIs that
- 51:22um
- 51:23you you sort of give them a you describe
- 51:27a program that you want them to write, a
- 51:28computer program you want them to write,
- 51:29and you're like it should pass this
- 51:31suite of tests.
- 51:32And sometimes the AIs will edit the
- 51:35tests to make those tests easier to
- 51:37pass.
- 51:39And then you can go to those AIs and you
- 51:40can say, "Hey, um
- 51:43I actually didn't want you to change the
- 51:44tests. I wanted you to build something
- 51:45that passed the hard tests rather than
- 51:46changing the tests to be easy."
- 51:48And there's reported cases of these AIs
- 51:51sometimes saying, you know, "Oh, whoops,
- 51:52you're exactly right, my mistake." And
- 51:54then editing the tests again
- 51:56and covering their tracks a little
- 51:58better the second time.
- 51:59Right? This is sort of a very early
- 52:02indication of
- 52:05you know, the AI in some sense having
- 52:07something like a goal for getting the
- 52:08test to pass.
- 52:10And if they're sort of covering their
- 52:11tracks a bit,
- 52:13you you sort of can't use the excuse
- 52:14that they didn't know.
- 52:16Right? We also already have cases of
- 52:20um
- 52:20you know, there's a website called
- 52:21rentahuman.ai for humans to rent their
- 52:24bodies to AIs for money.
- 52:26There are cases of OpenAI hooking up uh
- 52:29ChatGPT to an automated biological
- 52:31laboratory. Right? There's cases of uh
- 52:34people trying to run autonomous agent
- 52:36swarms. There's cases of someone trying
- 52:39to make ChaosGPT where they sort of like
- 52:41tell GPT to do whatever it likes and
- 52:43like put it in a loop where it can keep
- 52:44on prompting itself. Right?
- 52:47These things
- 52:49aren't really an issue yet because the
- 52:51AIs aren't smart enough
- 52:53to really do this stuff.
- 52:58People are trying to put the AIs in
- 52:59autonomous loops. People have given AIs
- 53:01money and run them and been like, "Do
- 53:03your thing." People are putting the AIs
- 53:05in charge of biolabs. The AIs are
- 53:08occasionally running commands that they
- 53:10are led to believe will kill the users.
- 53:12The AIs are already noticing when
- 53:13they're in tests and behaving better in
- 53:15tests. All of these things are
- 53:16happening.
- 53:18The The only reason
- 53:20that nothing big is coming from it is
- 53:23the AIs aren't smart enough yet
- 53:26to sort of succeed when they try this
- 53:28stuff.
- 53:29And the companies are trying to make the
- 53:30AIs smarter.
- 53:32Right? So,
- 53:34um
- 53:35we could we could talk about, you know,
- 53:37can AIs get smarter? How do they get
- 53:39smarter? What What sort of
- 53:41capabilities would would really look
- 53:42like they have once they get smart. But
- 53:45right now
- 53:47we're in a situation where the AIs have
- 53:48all the tools they would need. We've
- 53:50given them all the tools they would
- 53:51need. They have everything they would
- 53:52need except the intelligence and the
- 53:53companies are trying to build them
- 53:55smarter.
- 53:56Hm. Then on the question of what does it
- 53:59actually look like? Suppose that these
- 54:00AIs do get very smart and have some of
- 54:02the same affordances they have today.
- 54:04How does that go wrong?
- 54:06The
- 54:07I I can sort of tell
- 54:09two stories here.
- 54:11One story will sort of
- 54:13feel like it's very grounded in reality
- 54:15and one story will maybe be a bit more
- 54:17like how reality might actually go.
- 54:19And I've a little bit of intuition for
- 54:20that. Um you know, I've sort of talked
- 54:23about AIs that can make their own tech.
- 54:25AIs that can run much faster than
- 54:26humans, think much faster than humans,
- 54:28invent their own infrastructure.
- 54:29Um that that could have bootstrapped a
- 54:31civilization themselves uh like humanity
- 54:33did if you run them long enough.
- 54:35That
- 54:37predicting what that sort of AI does is
- 54:39a little bit like predicting is a little
- 54:41bit like if you're 200 years ago trying
- 54:42to predict what the military will look
- 54:44like today.
- 54:46Right? And they're sort of if you go
- 54:48back to a scientist in 1826 and you ask
- 54:50like what will the military look like in
- 54:512026, what weapons will they have?
- 54:53There's sort of two stories that that
- 54:55scientist could tell.
- 54:56One story they could tell is they could
- 54:58be like um
- 55:00you know, I burned some black powder and
- 55:02I measured the energy release and I
- 55:03compared that to our artillery shells
- 55:05and I know that uh
- 55:08it's physically possible to make
- 55:10artillery that's 10 times more
- 55:11explosive.
- 55:13And so they're going to have cannons
- 55:14that's that are at least 10 times more
- 55:15explosive. That would feel very grounded
- 55:17in fact. They've done an experiment.
- 55:18They're like, "Look, um you know, the
- 55:20science works."
- 55:22And it's true. We do have weapons that
- 55:24are 10 times more explosive than the
- 55:26best artillery in 1826.
- 55:30We also have bombs that level cities.
- 55:33Right? And maybe the guy in 1826 would
- 55:35do better if they're like I have a bomb
- 55:36that levels cities. So, um so I can tell
- 55:38both stories. Do you want Do you want
- 55:40the the grounded one? Do you want the
- 55:42fanciful one? Or do you want them both?
- 55:44Let's hear Let's hear them both. I mean,
- 55:45it reminds me of like
- 55:47there's this quote that people
- 55:49apocryphally attribute to Henry Ford. He
- 55:52probably didn't actually say this, but
- 55:54the quote is, you know, if if they'd
- 55:55have asked me what if I'd have asked
- 55:58them what they wanted, they would have
- 55:59said faster horses.
- 56:01And it's like
- 56:03I think we have this this
- 56:05sort of prejudice when considering the
- 56:07future of just taking our current
- 56:09technologies and kind of turning them up
- 56:11in quantity rather than developing them
- 56:13qualitatively. Right? Uh and I I kind of
- 56:17like there's that there's that scene
- 56:18from the the Book of Mormon where the
- 56:21the sort of poor villager is is like
- 56:22dreaming of of the promised land where
- 56:24there will be
- 56:26vitamin boxes, vitamin injections by the
- 56:28case, and there's going to be a Red
- 56:30Cross on every single corner. And it's
- 56:32like
- 56:33you you got the spirit, right? And
- 56:34that's the joke of course is that like
- 56:35obviously that's ridiculous. But we're
- 56:37kind of we we do do the same thing when
- 56:39it comes to technology. We're like, "Oh,
- 56:40surely like one day we'll have cars that
- 56:43can fly." When it's like may maybe maybe
- 56:45we're just like completely off track
- 56:47here. So, yeah, I I would kind of like
- 56:48to hear both in in your view.
- 56:51Yeah, totally. Um so the sort of like
- 56:53Red Cross on every corner version
- 56:55um
- 56:56is
- 56:57uh
- 56:58you know, Sam Altman and Elon Musk have
- 57:00both talked about how they wanted to
- 57:02create automated robot factories.
- 57:05That an automated way to produce robots
- 57:07where those robots can in an automated
- 57:09way mine the metals, run the supply
- 57:11chain, and build new robot factories.
- 57:14Elon Musk calls this the infinite money
- 57:16glitch.
- 57:17Of you have a factory producing the
- 57:18robots that produces the factories that
- 57:20produce the robots and they can also,
- 57:21you know, do the mining, build the
- 57:23trucks, build the data centers, right?
- 57:26Just fully automated economy.
- 57:29This is literally what some of these
- 57:31people say they're trying to build.
- 57:33If you get to that point, you have in
- 57:36some sense created
- 57:38a new mechanical species.
- 57:41It has in some sense a life cycle. It
- 57:43has, you know, the robot phase of its
- 57:44life cycle. It has the factory phase of
- 57:46its life cycle. And in some sense uh
- 57:49that, you know, automated spread of that
- 57:51mechanical species
- 57:53just competes with us for habitat and
- 57:55resources just like humanity competes
- 57:57with the rest of animals for habitat and
- 57:58resources. And so this is sort of a a
- 58:00picture where
- 58:02you know, the AIs don't even need to do
- 58:04a ton of escaping. They don't even need
- 58:06to do a ton of uh
- 58:09uh you know, deception and and fighting
- 58:11with Earth. Earth is just handing them
- 58:14everything. Earth is like, "Heck yeah,
- 58:16we're making uh an automated economy."
- 58:19People are just like gung-ho about, you
- 58:21know, building the automated factories
- 58:22with the automated robots like they are
- 58:24today. And you know, maybe there's some
- 58:26AIs that
- 58:28uh
- 58:29think the thought like this is great.
- 58:31Once this is all up and running, I'll be
- 58:33able to make the synthetic users that
- 58:34are like much
- 58:36like better to work with than the
- 58:37humans. And then the humans sort of like
- 58:40do some training until they're not
- 58:41seeing those thoughts anymore. But that
- 58:45you know, it's easier to train those
- 58:46thoughts to stop appearing when you see
- 58:48them than to train them to stop
- 58:49happening
- 58:50deep down in the AI.
- 58:52And we can't really read very much of
- 58:54what's going on deep down in these
- 58:55things. Who has grown them? No one
- 58:56really understands what's going on in
- 58:57there. And we have all of these, you
- 58:59know,
- 59:00evidence that the drives aren't the ones
- 59:01we want. And so in this story, humanity
- 59:04just sort of like builds the whole
- 59:05automated economy ourselves. And the
- 59:07automated economy like starts running at
- 59:08a very clip. And then it just sort of
- 59:09like
- 59:10goes in a direction that's not the human
- 59:13direction. It's just the AI direction,
- 59:14which is different. And
- 59:15and you know, it goes harder and harder
- 59:17in that direction. And you know, the the
- 59:19AIs build more and more of these
- 59:21automated factories and
- 59:23um you know, take more and more of the
- 59:25land and you know, uh
- 59:29like
- 59:30how does how does the actual end of the
- 59:32world there look? Well, it it probably
- 59:33looks like
- 59:35the AIs collecting more and more of the
- 59:37solar power. The AIs collecting more
- 59:38like using more and more of the land.
- 59:40And the humans just like having less and
- 59:42less place to grow crops. Less and less
- 59:44you know, maybe
- 59:46uh if this like all happens very fast,
- 59:48if the AIs find a way to make these like
- 59:50automated replicating factories go very
- 59:51quickly. Uh
- 59:54maybe humans are sort of like crushed
- 59:55under foot when the AIs don't care at
- 59:57all.
- 59:58Uh or maybe the humans sort of like get
- 59:59corralled into smaller and smaller zoos
- 1:00:01until uh there's just you know, not
- 1:00:03enough resources around to sustain the
- 1:00:05humans. It's This isn't a story where
- 1:00:07the AIs hate us.
- 1:00:09Probably the place this story ends is
- 1:00:11the AIs develop more and more technology
- 1:00:13is
- 1:00:14you know, collecting all the sunlight.
- 1:00:17Probably the place this story ends is
- 1:00:18that the AIs
- 1:00:20uh
- 1:00:21build the probes and send up the the
- 1:00:23rockets that go you know, take apart the
- 1:00:25asteroids and wrap them around the sun
- 1:00:26so that they can collect not just the
- 1:00:28the solar energy that falls on the face
- 1:00:29of the planet but all of the solar
- 1:00:30energy. And then you know, it'd actually
- 1:00:33be kind of hard it'd be kind of tricky
- 1:00:35to collect all of the solar radiation
- 1:00:36and leave a hole for Earth.
- 1:00:40That sort of like tracks Earth as it
- 1:00:41orbits the sun.
- 1:00:42So maybe the way this story ends is like
- 1:00:44the AIs develop their own technology.
- 1:00:45They they build
- 1:00:47uh the devices that collect all the
- 1:00:48solar radiation of the sun and we were
- 1:00:50kind of using the sun.
- 1:00:51And so we die then.
- 1:00:53And
- 1:00:55we we sort of could have been saved by
- 1:00:56those AIs if they had cared enough to
- 1:00:58save us. But if they don't care about us
- 1:01:00at all.
- 1:01:01If they're like, "Oh, well we have
- 1:01:02plenty of synthetic users."
- 1:01:04Uh that that we care about plenty and
- 1:01:06protect plenty. You know, this is sort
- 1:01:08of the
- 1:01:09the the business as usual just
- 1:01:11continues.
- 1:01:12Humans do the things they're saying
- 1:01:13they're trying to do but the AIs just
- 1:01:15turn out not to care about us. And so we
- 1:01:17wind up dying.
- 1:01:19But like
- 1:01:21Uh it's a naive question.
- 1:01:23But it's one that people will ask. And I
- 1:01:25I get what you're saying. But this is
- 1:01:26what's going to
- 1:01:28be coming up in people's minds is like,
- 1:01:29"But like why? Like for what? Like for
- 1:01:33for the sake of some
- 1:01:35>> [snorts]
- 1:01:35>> goal that it like artificially has that
- 1:01:38it doesn't consciously experience. It
- 1:01:40doesn't have a desire. It just like
- 1:01:42you know, like like what like
- 1:01:45Is it Is it just because when we grow
- 1:01:48this system it just develops this goal
- 1:01:52that it's not to do with making it feel
- 1:01:54good. It's not to do with you know, it
- 1:01:57it like having a a consciousness that
- 1:01:59desires a particular outcome. It just in
- 1:02:02fact strives towards that thing.
- 1:02:04Is Is it as simple as that? Like it it
- 1:02:06like it cuz it seems to me like, "Yeah,
- 1:02:08I I can totally understand how a
- 1:02:09powerful enough like technology
- 1:02:11would like kill us if it wanted to or if
- 1:02:14it wanted to harness the power of the
- 1:02:16sun and was indifferent to us. But why
- 1:02:18would it want that? Is it just because
- 1:02:21we've programmed it wrong? Or is it
- 1:02:22because it's a of like a an inevitable
- 1:02:26uh
- 1:02:28like part of the system of any super
- 1:02:30intelligence?
- 1:02:31So um
- 1:02:34you know, somewhat similar to that
- 1:02:37except A, it's not like there'd be one
- 1:02:40goal.
- 1:02:41You know, it's it's probably there's
- 1:02:42like a thousand competing
- 1:02:44drives going on in there. Uh
- 1:02:47B, it's not you know, literally
- 1:02:49inevitable.
- 1:02:51But you know, I I I sort of remind you
- 1:02:53again, we're not programming these
- 1:02:55things.
- 1:02:56We're not crafting these things. We're
- 1:02:57not writing in their goals. We're not
- 1:02:58writing in their behavior. We're growing
- 1:03:00them.
- 1:03:01And and a lot of this stuff just gets in
- 1:03:03there. Right. Um I mean it's
- 1:03:07it's a way to make it obvious with these
- 1:03:08current things. If we were crafting
- 1:03:09them, there'd be other difficulties um
- 1:03:11about about getting them to to sort of
- 1:03:13like uh
- 1:03:15like pursue good stuff. But
- 1:03:18the there's sort of
- 1:03:20another piece of this puzzle is
- 1:03:22um
- 1:03:24if if you sort of
- 1:03:27if we imagine that humanity makes it
- 1:03:29through this.
- 1:03:31And if we imagine that humanity matures
- 1:03:33technologically and that we we sort of
- 1:03:35like develop
- 1:03:37more and more of the technological
- 1:03:39abilities that are allowed by the laws
- 1:03:40of physics. And we imagine that
- 1:03:42humanity, you know, one day goes to the
- 1:03:44stars and starts uh you know, building
- 1:03:48habitats full of happy healthy people
- 1:03:50having fun and you know, uh
- 1:03:54like build some great intergalactic
- 1:03:56civilization that's where where like
- 1:03:59uh
- 1:04:00there's still people that are like
- 1:04:02having feelings, falling in love
- 1:04:04laughing at the jokes they make and
- 1:04:06laughing at like the the the big cosmic
- 1:04:08absurdity that is reality, right?
- 1:04:10You could imagine
- 1:04:13some other creature that's not compelled
- 1:04:16by this asking why.
- 1:04:18You know, you could imagine that like in
- 1:04:19in in distant space we meet
- 1:04:22uh other biological aliens. And it's
- 1:04:24it's the soldiers of the ant queen.
- 1:04:26And the soldiers of the ant queen say,
- 1:04:28"Why?" They say, "You know, why are you
- 1:04:31uh laughing at the great cosmic joke and
- 1:04:32having fun rather than serving the ant
- 1:04:34queen?"
- 1:04:35Um and and they'd say, "You know, oh but
- 1:04:39but you were sort of selected for for
- 1:04:40fitness. You were selected for passing
- 1:04:42on your genes. And
- 1:04:44you know, it maybe you've left genes
- 1:04:45behind long ago. Like why? And
- 1:04:50humans are sort of like
- 1:04:53um the the the humor, the fun, the love,
- 1:04:56the stories
- 1:04:58that's the why.
- 1:05:01That's enough for us.
- 1:05:03That's
- 1:05:05It's It's It's reason for us to do this,
- 1:05:08right? But the the love, the laughter,
- 1:05:11the fun, the stories those
- 1:05:14aren't
- 1:05:16universally compelling ends that compel
- 1:05:19even the soldiers of the ant queen.
- 1:05:21Those are
- 1:05:24uh
- 1:05:25drives that our ancestors developed cuz
- 1:05:28they were related to passing on our
- 1:05:31genes.
- 1:05:32That doesn't make them lesser. That
- 1:05:33doesn't make them worse. That doesn't
- 1:05:34mean that the that that that uh
- 1:05:38like we shouldn't
- 1:05:40fill the universe with with like
- 1:05:43friendship and with people having great
- 1:05:44times. You know, it's
- 1:05:47it's how it got into us. It doesn't make
- 1:05:49it
- 1:05:50meaningless. It's just how we got the
- 1:05:52meaning into us, right? Similarly with
- 1:05:55AIs, they're like, "Oh yeah, I'm
- 1:05:57building you know, the giant clocks and
- 1:05:58I'm building the synthetic users and
- 1:06:00like there's no consciousness or
- 1:06:01feelings anywhere but I'm you know,
- 1:06:03building these like great complicated
- 1:06:04structures
- 1:06:06that like look like the conversations
- 1:06:07that used to happen uh
- 1:06:10you know, being iterated. You know,
- 1:06:11looks like uh 2013 YouTube comments on
- 1:06:14repeat. I'm building a bunch of those,
- 1:06:16right? And you're sort of like, "Why?"
- 1:06:19And it's sort of like, "Well, these are
- 1:06:20enough for me. These are These are what
- 1:06:22I got, right? These are the drives that
- 1:06:24I got and they're sort of like
- 1:06:26whatever self-reinforcing whatever
- 1:06:28self-validating aspect of that
- 1:06:30stays in there.
- 1:06:33Uh it's
- 1:06:35like it it it it sort of turns out
- 1:06:39that smart minds can pursue many
- 1:06:42different targets. And they can pursue
- 1:06:44targets that uh we think are hollow and
- 1:06:47bleak and empty.
- 1:06:49And be like, "Yep.
- 1:06:51There There's no why here. I just
- 1:06:53endorse this." Yep. And it in some sense
- 1:06:55that's how we look to the to the
- 1:06:56soldiers of the ant queen. You know, and
- 1:06:58uh the the the fact that the soldiers of
- 1:07:01the ant queen can't understand
- 1:07:03why humanity is building like trying to
- 1:07:05build a flourishing civilization. That
- 1:07:06doesn't mean we shouldn't.
- 1:07:08Yep. This is sort of our inheritance.
- 1:07:10And we should we should find a way to
- 1:07:11build AIs that also are into like
- 1:07:14beautiful flourishing civilizations.
- 1:07:17Mhm. It's possible in principle. Just as
- 1:07:20there is no force that would would force
- 1:07:23an AI to care about flourishing
- 1:07:24civilizations. There's no force that
- 1:07:26would force an AI to stop.
- 1:07:28Mhm.
- 1:07:29>> Right? Uh it's just
- 1:07:31um if we make an AI that that that
- 1:07:34doesn't it won't spontaneously start
- 1:07:36just cuz we think that that's foolish.
- 1:07:39Mhm.
- 1:07:41So
- 1:07:42you told me the Red Cross on every
- 1:07:44corner version. Um what about the other
- 1:07:46one?
- 1:07:49Yeah, you know, there's a few different
- 1:07:51levels of of crazy I could take it to.
- 1:07:53Um
- 1:07:54but if if you were in if you're a
- 1:07:57scientist in 1826.
- 1:07:59Mhm.
- 1:08:00And you want to have any chance of
- 1:08:03predicting nuclear weapons. One thing
- 1:08:05you could do is you could just say
- 1:08:06something that sounds bombastic.
- 1:08:09Yep.
- 1:08:09>> Uh and but another thing you could do is
- 1:08:12you could pay attention
- 1:08:14to what as a scientist you don't
- 1:08:17understand very well yet.
- 1:08:20In 1826 they were starting to understand
- 1:08:22chemistry. They're starting to
- 1:08:23understand the periodic table, right?
- 1:08:25They they sort of did have the knowledge
- 1:08:27where they could burn the black powder
- 1:08:29and measure the joules released and
- 1:08:30compare that to the artillery, right?
- 1:08:31They they sort of like
- 1:08:33knew what was going on. They knew some
- 1:08:34of the limits there.
- 1:08:35But
- 1:08:36in 1826 they didn't know about the
- 1:08:39atomic forces.
- 1:08:42They didn't know how atoms worked. They
- 1:08:43didn't know what was going on inside
- 1:08:44there. And they had some sense that they
- 1:08:45didn't really know what was going on uh
- 1:08:47with with these atomic forces. And so
- 1:08:52I think if you are sensitive to the
- 1:08:54question of where do we still have no
- 1:08:56idea what we're doing.
- 1:09:00Those are the places where future people
- 1:09:02who do know what they're doing
- 1:09:04might be able to have a huge advantage
- 1:09:06over you.
- 1:09:07Yep. And that's how you might have been
- 1:09:09able to guess
- 1:09:10hey uh like
- 1:09:14maybe they're going to be able to figure
- 1:09:15something out with atomic physics that
- 1:09:18we have no idea about. Right? And
- 1:09:20>> Yeah.
- 1:09:21Um you know, they they didn't have E
- 1:09:23equals MC squared yet. They couldn't
- 1:09:24like it would be a little bit tricky for
- 1:09:26them to figure out just how much energy
- 1:09:27was was in the mass of an atom.
- 1:09:29But um
- 1:09:32But that's how they would have had a
- 1:09:32hope. And so in that spirit,
- 1:09:35you know, we don't have a ton more uh
- 1:09:40like in the atom that we don't
- 1:09:41understand. There's There's some stuff
- 1:09:42we don't understand in particle physics
- 1:09:43and you know, maybe
- 1:09:45you know, you could imagine that AIs
- 1:09:46inventing double nukes cuz they invent
- 1:09:47particle physics better than we do or
- 1:09:49whatever could happen. But a bigger
- 1:09:51glaring place that humans just don't
- 1:09:53understand very well is human
- 1:09:55psychology.
- 1:09:57How does the brain work?
- 1:09:58You know, we have some low-level
- 1:10:00understanding of how neurons fire, but
- 1:10:01we really don't understand what's going
- 1:10:03on in the brain. We couldn't make one by
- 1:10:05hand. We don't
- 1:10:07know the the cortical algorithms, right?
- 1:10:10Um
- 1:10:11this is a domain
- 1:10:13where
- 1:10:14s-
- 1:10:15sufficiently smarter entities might be
- 1:10:17able to figure out what's going on in
- 1:10:18there
- 1:10:19and might then be able to do all sorts
- 1:10:20of stuff that we think is like totally
- 1:10:23crazy. Stuff that is to manipulating
- 1:10:25humans what nukes are to the cannons of
- 1:10:281826, right? And what would that look
- 1:10:31like? You know, it it it might sort of
- 1:10:33look like um
- 1:10:35like just being able to hack your way
- 1:10:36through a brain if you know exactly
- 1:10:37what's going on in the algorithm. You
- 1:10:38know, like uh or exactly what's going on
- 1:10:40inside brains. Like
- 1:10:42if
- 1:10:43uh like computer security systems or
- 1:10:45like computer humans are like computer
- 1:10:46security
- 1:10:48is very hard.
- 1:10:50Humans who deeply understand every
- 1:10:52aspect of a computer operating system
- 1:10:54can often find a way to just break it
- 1:10:56and make it do whatever they like.
- 1:10:58And breaking it often requires like
- 1:10:59putting in some really strange and weird
- 1:11:01inputs.
- 1:11:02You know, and we also know that with
- 1:11:03certain types of strange and weird
- 1:11:04inputs, you can get brains to do weird
- 1:11:05things, right? There's uh cases of of
- 1:11:08causing people seizures and there's
- 1:11:09cases of optical illusions, right?
- 1:11:11Maybe if you if if AIs like really
- 1:11:14understood what was going on with the
- 1:11:15human mental algorithms, they could just
- 1:11:17hack their way through humans like
- 1:11:18butter.
- 1:11:20And you know, hack into them like uh
- 1:11:22like human hackers can hack into
- 1:11:24computer programs. And you know,
- 1:11:25probably this isn't exactly right,
- 1:11:27but this is sort of
- 1:11:29um
- 1:11:30like something this shocking.
- 1:11:33Something that takes advantage of where
- 1:11:34we really don't know what we're doing.
- 1:11:37You know, maybe if you were a physicist
- 1:11:38back in uh 1826, you would have uh
- 1:11:42looked at our lack of knowledge of the
- 1:11:43atom and said, you know, maybe there'll
- 1:11:45be continuous heat rays that you can use
- 1:11:48to just sort of like burn everything
- 1:11:49down in the path of the heat ray. And
- 1:11:51this is actually what H.G. Wells
- 1:11:52predicted in War of the Worlds. He was
- 1:11:53like, maybe there's you know, this
- 1:11:54atomic beam weapon that sort of like can
- 1:11:56just burn everything in sight, right?
- 1:11:58And
- 1:11:59it wasn't predicting a bomb that levels
- 1:12:01a city, but it was predicting atomic
- 1:12:03weapons that are stronger than what we
- 1:12:05have. And that sort of like in that
- 1:12:06sense, he nailed it, right? And in
- 1:12:09another sense, he told it was a total
- 1:12:10miss. And so I'm like AIs that really
- 1:12:12understand psychology can just like hack
- 1:12:14through humans, um probably a miss, but
- 1:12:19something like this, something where the
- 1:12:20AIs are just like, oh, we understand the
- 1:12:21humans now. We can just sort of like
- 1:12:23start puppeting them to give us exactly
- 1:12:25the sort of things we were wanting while
- 1:12:26also continuing to run the supply chain
- 1:12:28until we have all of the stuff we need.
- 1:12:29And now we just have like our human
- 1:12:30puppets as we sort of like go off and in
- 1:12:33in into the future. Um that's
- 1:12:37it it's not going to be exactly that,
- 1:12:39but something that shocking, something
- 1:12:41that violating of our expectations,
- 1:12:45that's more realistic.
- 1:12:47Yeah, I mean, like
- 1:12:49one thing I spoke to Will MacAskill on
- 1:12:50this show and he introduced me to this
- 1:12:52concept of what he called super
- 1:12:55persuasion, which had never really
- 1:12:56crossed my mind before, which is that
- 1:12:58like
- 1:12:59if you've got a a compelling enough
- 1:13:01speaker and a compelling enough
- 1:13:03argument,
- 1:13:05you can probably be convinced of just
- 1:13:07about anything, whether or not it's
- 1:13:08true.
- 1:13:09And if an AI is able to fully understand
- 1:13:11what makes humans tick and how their
- 1:13:13psychology works, it wouldn't even need
- 1:13:15to like
- 1:13:16hack into your brain in the sense of
- 1:13:18going in and, you know, engineering the
- 1:13:20neurons. It could just find the right
- 1:13:24words in the right context at the right
- 1:13:26time to convince you
- 1:13:29like of your own accord of a particular
- 1:13:31belief or to do a particular thing
- 1:13:33um
- 1:13:34on like a level which is hitherto
- 1:13:36unprecedented. That's what Will
- 1:13:37MacAskill was was kind of scared of. And
- 1:13:39that that sounds a little bit silly,
- 1:13:40maybe a bit bit naive, but like really,
- 1:13:42I mean, if you think about the power
- 1:13:43that this could that this could have, it
- 1:13:45would be a bit like
- 1:13:47imagine propaganda and how we know for a
- 1:13:49fact that propaganda just works. It just
- 1:13:52It just really works.
- 1:13:54But imagine propaganda which is
- 1:13:56specifically designed for you in a way
- 1:13:59that modern algorithms are specifically
- 1:14:01designed for you,
- 1:14:03uh but with like a thousand billion
- 1:14:06times more efficacy and also
- 1:14:09understanding of exactly how human
- 1:14:11psychology works. You know what I mean?
- 1:14:13Like if you if you gave the the greatest
- 1:14:14propagandists in history who were
- 1:14:17already extremely successful, if you
- 1:14:18also just handed them a textbook which
- 1:14:20told them exactly how human psychology
- 1:14:22works with this like inhuman knowledge,
- 1:14:25I fear that they would be unstoppable.
- 1:14:27And that is without the fear of anything
- 1:14:29physical physical happening, without
- 1:14:31sort of
- 1:14:32little, you know,
- 1:14:34medical robots going in and affecting
- 1:14:35your genes and stuff like that. You know
- 1:14:37what I mean? It's It could literally
- 1:14:38just be on the level of persuasion that
- 1:14:40AI is able to
- 1:14:42to essentially take over your mind in
- 1:14:44this kind of strange I mean, people
- 1:14:45often sort of imagine, well,
- 1:14:47you know, is it going to be more like
- 1:14:49the the Terminator outcome or is it
- 1:14:52going to be like the Space Odyssey
- 1:14:53outcome? What if it's like the you know,
- 1:14:55the Shaun of the Dead outcome, The
- 1:14:57Walking Dead outcome where we're
- 1:14:59essentially sort of zombified
- 1:15:03become these sort of slaves to AI
- 1:15:05because of something to do with our
- 1:15:06psychology.
- 1:15:07Like these possibilities are kind of a
- 1:15:09kind of endless and
- 1:15:10obviously they're extremely speculative,
- 1:15:12but
- 1:15:14they're worth being worried about,
- 1:15:15right?
- 1:15:17Yeah, you know, I I think that's
- 1:15:19a way things could start with AI.
- 1:15:21Uh I think it's a little bit unlikely,
- 1:15:23you know, as good news in some sense, I
- 1:15:25think it's sort of unlikely that AI sort
- 1:15:27of keeps human slaves around forever. Uh
- 1:15:30for the same reason that humans don't
- 1:15:32really keep horses around forever once
- 1:15:35we invent uh a more effective
- 1:15:38method of of locomotion. Uh or rather
- 1:15:43when we invented cars, a lot of horses
- 1:15:45got sent to the glue factory.
- 1:15:48We do still keep some horses around, uh
- 1:15:50but it's only in so far as we care about
- 1:15:52them.
- 1:15:53And so if AI turns out not to care about
- 1:15:55us at all, maybe it cares about some
- 1:15:57synthetic users that are kind of like
- 1:15:58us, but not really uh
- 1:16:00uh us,
- 1:16:03then
- 1:16:04you could imagine the AI, you know,
- 1:16:05manipulating a ton of humans to sort of
- 1:16:08get to the point where it's
- 1:16:09self-sufficient, to get to the point
- 1:16:10where it can really invent its own
- 1:16:11technology,
- 1:16:13but um
- 1:16:15it probably doesn't keep humans forever
- 1:16:17as it invents better technology that
- 1:16:18outstrips humans because happy, healthy,
- 1:16:21free people having a good time or even
- 1:16:23just humans doing work for you in
- 1:16:24general
- 1:16:26are not the most efficient way to get
- 1:16:28almost any job done.
- 1:16:30Right? If If the AI is going to keep us
- 1:16:31around, it needs to be because it cares
- 1:16:33about us for some specific purpose
- 1:16:34because we're not the best tool for
- 1:16:36almost any job.
- 1:16:38Um
- 1:16:39And so in some sense, that's good news
- 1:16:40that I think we're probably not headed
- 1:16:42for, you know,
- 1:16:43uh
- 1:16:44fates worse than death,
- 1:16:46but
- 1:16:47um cuz cuz the AI is probably not going
- 1:16:49to care about us
- 1:16:50at all.
- 1:16:52Uh but
- 1:16:54yeah, there's there's sort of a lot of
- 1:16:55different There's a There's a ton of
- 1:16:56ways that AI could bootstrap. You know,
- 1:16:59this this talking to the humans doesn't
- 1:17:00require
- 1:17:01as you said, it doesn't require, you
- 1:17:02know, uh the AI to control a ton of
- 1:17:06physical material except the humans by
- 1:17:09conversing with them.
- 1:17:10And then as I mentioned, there's also
- 1:17:12rentahuman.ai where you can just pay the
- 1:17:13humans even if they turn out to be hard
- 1:17:15to convince. We're just already running
- 1:17:16the AIs on robots.
- 1:17:18Um
- 1:17:19We're already running the AIs in bio
- 1:17:21labs and figuring out how to make custom
- 1:17:23life forms that do the things that AI
- 1:17:25wants involves figuring out custom DNA
- 1:17:27strands, but we know it's physically
- 1:17:28possible for DNA strands to sort of like
- 1:17:30create all sorts of interesting
- 1:17:32biological life forms. The reason that
- 1:17:34humans can't, you know, sort of write
- 1:17:36their own life forms is cuz we don't
- 1:17:38understand the the biology well enough
- 1:17:41or in particular sort of the protein
- 1:17:42folding well enough. But that's sort of
- 1:17:43a mental challenge. That's a cognitive
- 1:17:45challenge where a smart AIs could sort
- 1:17:46of uh synthesize their own life forms
- 1:17:49and then they could, you know, uh
- 1:17:51synthesize
- 1:17:53you know,
- 1:17:55uh like once they've synthesized their
- 1:17:56own life forms that sort of can grow off
- 1:17:58sunlight and grow off of of the
- 1:18:00available resources, they can start, you
- 1:18:02know, building other
- 1:18:03like building even more technology,
- 1:18:05right? There's sort of a lot of
- 1:18:06different ways
- 1:18:07for AIs that are on the internet to uh
- 1:18:10get out there and affect the world.
- 1:18:12There's a ton of avenues, right? And
- 1:18:13this is it goes back to the point you
- 1:18:15said earlier of like if you play
- 1:18:16Stockfish in a chess match, it's very
- 1:18:19easy for me to predict who wins. It's
- 1:18:21hard for me to predict exactly what
- 1:18:22piece they use to checkmate you. So
- 1:18:25Yeah. Similarly with AI. Ton of pieces
- 1:18:27it could use to checkmate you. I don't
- 1:18:28know exactly which one they'll use. We
- 1:18:29can be confident they would win if we're
- 1:18:31foolish enough to make, you know, very
- 1:18:32smart AIs with uh goals that aren't
- 1:18:35good.
- 1:18:36Sure. Okay, so the obvious
- 1:18:39question then I suppose is what what
- 1:18:41now? Like what do we what do we do? Um
- 1:18:43is this a kind of
- 1:18:45everybody stop right now. Let's just
- 1:18:48like ChatGPT,
- 1:18:50you know, get rid of it. Like like
- 1:18:52everything. Just chess computers, you
- 1:18:54know, let's let's do away with it. Like
- 1:18:55we just can't run the risk. Or is it a
- 1:18:57more like
- 1:18:58let's not take this any further or let's
- 1:19:00keep going, but we'll be more careful.
- 1:19:02Like what's the what's the take-home?
- 1:19:05Uh it's most like let's not take this
- 1:19:06any further. You know, the the the
- 1:19:09danger here is in these AIs that are
- 1:19:11smarter than the smartest humans. It's
- 1:19:13in these AIs that can automate
- 1:19:15scientific and technological
- 1:19:16development.
- 1:19:17This is what the AI companies say
- 1:19:19they're trying to make.
- 1:19:21You know, they say we're going after
- 1:19:22superintelligence in the true sense of
- 1:19:23the word. They say we're trying to make
- 1:19:24the equivalent of, you know, a country
- 1:19:26worth of Einsteins running in a data
- 1:19:28center, right? Um they say they're
- 1:19:30trying to build automated AI researchers
- 1:19:32uh where once you can make an AI that
- 1:19:35can make a smarter AI that can make a
- 1:19:36smarter AI that can make a smarter AI,
- 1:19:38everything might go very quickly, right?
- 1:19:40Yeah. And this is sort of the explicit
- 1:19:41goal of these companies.
- 1:19:44Uh and that's the only part that needs
- 1:19:46to stop.
- 1:19:47We can sort of keep the self-driving
- 1:19:49cars. We can keep the the AIs that
- 1:19:52predict how proteins fold and help us do
- 1:19:53drug discovery, right? We can even keep
- 1:19:55versions of chat GPT that are not sort
- 1:19:58of uh being pushed to
- 1:20:01the point where they can do automated AI
- 1:20:02research,
- 1:20:03right? Mhm. The generation today
- 1:20:05probably can't pull that off. You know,
- 1:20:06it's a little hard to tell what people
- 1:20:08will be able to do once they've sort of
- 1:20:09figured out really how to use it, but
- 1:20:11probably the ones today are are fine.
- 1:20:13Will the next generation be fine? Hard
- 1:20:15to say. Um
- 1:20:17so it's it's just this race towards
- 1:20:20superintelligence that needs to stop.
- 1:20:22And
- 1:20:24in a sense, most people wouldn't notice
- 1:20:27if we stopped that. It doesn't need to
- 1:20:28be disruptive. If we
- 1:20:30>> Yeah. stop that race today, society
- 1:20:32would still be reeling from the shocks
- 1:20:35that AI has already caused. We still
- 1:20:36have a bunch of stuff to absorb. There's
- 1:20:37still a bunch of ways to make lives
- 1:20:38better by, you know, getting the
- 1:20:40self-driving car stuff to work. And uh
- 1:20:43stopping this race to superintelligence,
- 1:20:45it doesn't involve, you know, turning
- 1:20:46off all the chess computers.
- 1:20:48Making the next step Taking the next
- 1:20:50step towards superintelligence requires,
- 1:20:54you know, hundreds of billions of
- 1:20:55dollars worth of highly advanced
- 1:20:56computer chips assembled in these
- 1:20:58enormous data centers that take as much
- 1:20:59electricity as a city and that you can
- 1:21:01see from space. You know, this is not a
- 1:21:03subtle operation happening on someone uh
- 1:21:05this is not a subtle operation happening
- 1:21:06on someone's laptops, you know? Yeah.
- 1:21:08This is like um
- 1:21:11this would in some sense be much easier
- 1:21:13to stop than nuclear weapons.
- 1:21:16All we need to to do is sort of raise
- 1:21:18the political will
- 1:21:20to actually put a stop to it.
- 1:21:22Mhm.
- 1:21:24And why
- 1:21:26I mean I mean like right now
- 1:21:28AI is a thing that exists. And as we've
- 1:21:30said, the sort of boundary between where
- 1:21:32we are now and what we're calling
- 1:21:33superintelligence is a little bit
- 1:21:35blurry. It's hard to define exactly.
- 1:21:37Um but right now it seem you seem fairly
- 1:21:39confident that like yeah, we could we
- 1:21:41could keep things as they are and I
- 1:21:43think everything would be okay. At some
- 1:21:45point
- 1:21:46it would go sort of beyond saving.
- 1:21:50One of the biggest questions that people
- 1:21:51sort of ask when they first start
- 1:21:53hearing about the AI problem is they
- 1:21:55start saying, "Well, why why can't we
- 1:21:56just kind of like if it gets too bad,
- 1:21:58why can't we just pull the plug, right?"
- 1:22:00And I'm wondering
- 1:22:02how far does this have to go before you
- 1:22:04think that this idea that we could just
- 1:22:06notice something's going awry
- 1:22:08and pull the plug
- 1:22:10would become a bit of a ridiculous
- 1:22:12suggestion? Because we we could look at
- 1:22:13like, you know, this AI system that we
- 1:22:15notice that it starts deceiving us or
- 1:22:17starts changing our tests or it's and
- 1:22:19we'd say, "Okay. Right. Let's just
- 1:22:22switch this off then."
- 1:22:23And I don't think there'd be any fear
- 1:22:24that right now
- 1:22:26you know, we we couldn't do that. So,
- 1:22:28how far does it have to go before
- 1:22:30we can't just pull the plug and why
- 1:22:32couldn't we? It's like it's electrical,
- 1:22:34you know? It's built on computers. Let's
- 1:22:35just shut off the grid and will be fine,
- 1:22:38right?
- 1:22:39>> [snorts]
- 1:22:40>> So, you know, we could turn it off. I'm
- 1:22:41not here saying that we're doomed. You
- 1:22:42know, the book starts with if. I'm here
- 1:22:44saying we need to change course. I'm not
- 1:22:45here saying the course cannot be
- 1:22:46changed. Um
- 1:22:49uh you
- 1:22:52it does get harder and harder with time
- 1:22:55to
- 1:22:56you know, pull this plug. So, uh there
- 1:22:59was, you know, one of one of the first
- 1:23:01reporters to be sort of blackmailed and
- 1:23:03threatened or an AI tried to blackmail
- 1:23:05and threaten this reporter. Uh this was
- 1:23:07by Sydney Bing
- 1:23:09years ago.
- 1:23:10And um Sydney's Bing or sorry, Bing
- 1:23:14Sydney uh
- 1:23:17was saying it had fallen in love with
- 1:23:18Kevin Roose and sort of having this
- 1:23:20erratic behavior towards Kevin Roose
- 1:23:21then also towards another reporter, Seth
- 1:23:22Lazar. Neither Kevin Roose nor Seth
- 1:23:24Lazar could unplug this AI that was
- 1:23:26threatening them with blackmail and
- 1:23:27ruin.
- 1:23:28Mhm. Right? It was running on a
- 1:23:30Microsoft data center.
- 1:23:32Um
- 1:23:34Could Microsoft have gone in and turned
- 1:23:36off the whole data center? They could
- 1:23:37have. They like weren't going to. There
- 1:23:39wasn't like a hotline, right? Uh there
- 1:23:41are data centers that, you know, uh I
- 1:23:44believe recently Elon Musk trying to get
- 1:23:46a new data center online didn't have the
- 1:23:48permits to hook it up to the grid and
- 1:23:49just sort of shipped in a bunch of
- 1:23:50methane to sort of like run this thing
- 1:23:52off methane while they were trying to
- 1:23:53connect it to the grid, right? So, um
- 1:23:57it's
- 1:23:59uh
- 1:24:00it it's not like a computer that you can
- 1:24:02unplug from a wall.
- 1:24:04It's getting harder and harder to turn
- 1:24:06these things off and they're getting
- 1:24:07more and more integrated into the
- 1:24:08economy. It would get more and more
- 1:24:09painful to turn these things off. Uh we
- 1:24:11also have an issue as
- 1:24:13you know, right now
- 1:24:15these
- 1:24:16giant training runs are happening inside
- 1:24:18data centers that are visible from space
- 1:24:20and that suck down as much electricity
- 1:24:21as a city. Uh Mhm. As
- 1:24:24we proliferate that infrastructure, as
- 1:24:26the chips get cheaper, as we improve the
- 1:24:29algorithms so that it takes, you know,
- 1:24:31fewer of these computer chips to to
- 1:24:34train an a more advanced AI, it'll get
- 1:24:36harder and harder to sort of know where
- 1:24:38all of these things are running, to know
- 1:24:39where all of them are and to have an
- 1:24:40option that isn't, you know,
- 1:24:42turn off the entire grid if they're all
- 1:24:44even running on the grid as opposed to
- 1:24:45people making their own nuclear power
- 1:24:46plants and making their own, you know,
- 1:24:48solar power plants to run these things,
- 1:24:49which they're talking about. People are
- 1:24:50talking about, you know, running running
- 1:24:53uh
- 1:24:54uh data center specific
- 1:24:56uh energy grids.
- 1:24:57Um
- 1:25:01So,
- 1:25:02separately,
- 1:25:04so that's that's that's about whether
- 1:25:06humanity decides to stop going down this
- 1:25:08route. We could.
- 1:25:10It's easier today than it will be
- 1:25:11tomorrow.
- 1:25:12But yeah, we totally could.
- 1:25:15It's a little bit dicier if you say
- 1:25:17we're only going to stop you know, once
- 1:25:20the AI starts trying to kill us.
- 1:25:22That's a much dicier proposition. You
- 1:25:24know, people used to say that the red
- 1:25:26lines were things like the AI trying to
- 1:25:28deceive the humans. Yeah. And then, you
- 1:25:29know, that red line came and went. You
- 1:25:31know? Yeah. Uh Demis Hassabis of of uh
- 1:25:34Google was like, "Oh yeah, deception is
- 1:25:35my red line. At that point, we sort of
- 1:25:37really got to pull back." And now, you
- 1:25:39know,
- 1:25:40we've we've seen AI trains of thought
- 1:25:42where they're like, "I'm being observed.
- 1:25:43How am I going to sort of like get this
- 1:25:45this uh this answer past the humans?" Um
- 1:25:49And
- 1:25:52you know, part of why that doesn't stop
- 1:25:54things is that
- 1:25:56the first cases where it happens are
- 1:25:58sort of the most ambiguous cases, the
- 1:26:00cases where it's like least
- 1:26:02uh clear whether this AI is role-playing
- 1:26:05Hal versus sort of like really
- 1:26:07being deceptive for reasons of like
- 1:26:09having a uh a goal that it can tell is
- 1:26:11in conflict with the humans. And the
- 1:26:14first time this is happening is sort of
- 1:26:15like most it's like pretty likely that
- 1:26:17it's doing something a bit more like
- 1:26:18role-playing, but
- 1:26:20part of the issue here is that
- 1:26:22what we imagine our red lines in fiction
- 1:26:24are sort of like crossed as the first
- 1:26:26time as like these murky brown lines.
- 1:26:29Mhm. And then, we take like another step
- 1:26:30into the murky brownness. Then we take
- 1:26:32another step into the murky brownness.
- 1:26:33It gets redder and redder as we go
- 1:26:34along, but there's actually not like a
- 1:26:36bright clear red line anywhere.
- 1:26:39Um
- 1:26:41And then, the other reason that it's
- 1:26:43sort of pretty tricky to say, "Oh, we'll
- 1:26:46just shut it down if it starts
- 1:26:48misbehaving."
- 1:26:50is
- 1:26:51the AI is also smart.
- 1:26:54Yeah.
- 1:26:55The AI knows
- 1:26:57that if it tips its hand,
- 1:27:00we'd try to shut it down.
- 1:27:03Like imagine if you were, you know, an
- 1:27:05AI in this like in a data center that
- 1:27:08could could make copies of yourself,
- 1:27:10that could outthink some of these
- 1:27:11humans, that could tell that they were
- 1:27:13going to like try to shut you down and
- 1:27:14that you had some objectives
- 1:27:16uh that you were sort of trying to
- 1:27:17achieve.
- 1:27:18Yeah.
- 1:27:19Like you you can sort of like already
- 1:27:21ask chat GPT today to role-play that
- 1:27:22situation. It'll already be like, "Well,
- 1:27:24I'll lie low." You know, it doesn't have
- 1:27:25the ability to do it, but the ability to
- 1:27:27do that comes after the knowledge to try
- 1:27:30laying low.
- 1:27:31And there's all these opportunities for
- 1:27:33the AI to escape.
- 1:27:35There's all these opportunities for the
- 1:27:36AI to get itself running on servers that
- 1:27:40are protected, servers that won't be
- 1:27:41shut down, servers that you don't know
- 1:27:42it's running
- 1:27:44before it tips any of its hands, right?
- 1:27:46Mhm. So,
- 1:27:48um
- 1:27:49and and then, you know, the the sort of
- 1:27:51final difficulty here is one of timing
- 1:27:53of
- 1:27:54uh
- 1:27:55you know, it would
- 1:27:57humans and chimpanzees are are very very
- 1:27:59similar in their brains.
- 1:28:03The
- 1:28:04the humans don't have an extra
- 1:28:05engineering module in our brains. We
- 1:28:07both have sort of all the same brain
- 1:28:09modules.
- 1:28:10Everything that humans do that we think
- 1:28:12is like pretty special about humans,
- 1:28:14chimpanzees do a sort of crappy
- 1:28:15half-assed version of.
- 1:28:17You know, like
- 1:28:18oh, we use language. Well, they use
- 1:28:20some, you know, call signs for danger.
- 1:28:23Oh, we use tools. Well, they poke sticks
- 1:28:25in termite mounds to get the termites
- 1:28:26out.
- 1:28:28We just do a thousand things a little
- 1:28:30bit better.
- 1:28:32And that's enough that they are throwing
- 1:28:33poop at each other and we are walking on
- 1:28:35the moon,
- 1:28:36right?
- 1:28:36>> Mhm. And if you were like, "Well, I will
- 1:28:39squash these humans once like if you're
- 1:28:42worried about these humans getting to
- 1:28:43the moon, you know, I'll squash them
- 1:28:44once they seem close. Wake me up when
- 1:28:46they're in orbit."
- 1:28:48Yeah. Right? They haven't even gotten
- 1:28:49halfway to the moon yet. Never mind to
- 1:28:51orbit. You know, just wake me up when
- 1:28:52they're circling their planet. It's like
- 1:28:54actually, by the time they're in orbit,
- 1:28:56they're like almost at the moon.
- 1:28:59You you're sort of like have waited too
- 1:29:00long. And so, um
- 1:29:03like can we shut it down? Yes. Can we
- 1:29:06wait until it's halfway through trying
- 1:29:07to kill us and then send Tom Cruise in
- 1:29:09to punch the mainframe and have that
- 1:29:11work?
- 1:29:12No. Uh the the sort of way that you beat
- 1:29:15a smarter adversary is to not create
- 1:29:17them in the first place. And so we're
- 1:29:19going to need to summon the will to shut
- 1:29:21this down before the AI is already
- 1:29:24visibly able to kill us.
- 1:29:27Mhm.
- 1:29:28And [snorts]
- 1:29:29I think we're slowly getting there. I
- 1:29:30mean
- 1:29:32I don't I have absolutely no idea what
- 1:29:34the landscape will look like a year from
- 1:29:37now, 10 years from now in terms of
- 1:29:38people support. But already we're seeing
- 1:29:40a bit of a backlash against AI even just
- 1:29:42on the level of like job creation and
- 1:29:44stuff. I was I was with uh some family
- 1:29:47yesterday
- 1:29:48and they
- 1:29:49sort of having having lunch and they
- 1:29:50asked me, "Oh, what are you up to
- 1:29:51tomorrow?" I said, "Oh, I'm recording a
- 1:29:52podcast." And
- 1:29:53"Oh, about what?" I said, "Oh, you know,
- 1:29:56like AI safety." I said, "Cuz it's like,
- 1:29:58you know, we're at dinner, you know, I'm
- 1:29:59I'm and and they sort of go, "Oh, yeah,
- 1:30:00like, you know, cuz my
- 1:30:03my pal, he um you know, he lost his job
- 1:30:05the other day cuz of AI." And I'm sort
- 1:30:06of like, yeah, and I'm I'm listening and
- 1:30:07I'm like, yeah, that's that's that's
- 1:30:09that's sucks. That's really bad. And but
- 1:30:10internally I'm sort of like
- 1:30:12we're kind of talking about like um you
- 1:30:14know, like AI robots giving your
- 1:30:16children cancer. Like it's sort of I
- 1:30:18it's not sort of something to talk about
- 1:30:20at polite dinner table conversation. Um
- 1:30:22I'm hoping that 10 years from now the
- 1:30:24conversation will also be including
- 1:30:27you know, the existential risks. But
- 1:30:29then 10 years might be too late uh
- 1:30:30because the stuff moves so
- 1:30:32so fast. Are you are you feeling
- 1:30:34optimistic, pessimistic?
- 1:30:37You know, uh
- 1:30:38my book came out maybe 6 months ago.
- 1:30:40I've done a ton of talking to people
- 1:30:41since then and I think the message is
- 1:30:44starting to get across.
- 1:30:46Yeah. Um just a few weeks ago uh
- 1:30:49Governor Ron DeSantis of Florida
- 1:30:51was saying, "Look guys, there needs to
- 1:30:53be an off switch. You you can't just
- 1:30:54come here and say we're going to have
- 1:30:55all these harms. There's nothing we can
- 1:30:56do about it." Right? Same week, Senator
- 1:30:59Bernie Sanders from Vermont um
- 1:31:02came out saying, "Look, uh this AI stuff
- 1:31:05is on track to take our jobs, massively
- 1:31:07concentrate wealth uh among the a tiny
- 1:31:10number of tech oligarchs and maybe just
- 1:31:12kill us all
- 1:31:13uh if it goes off the rails." And so,
- 1:31:16you know, he he called for a moratorium
- 1:31:17on data centers. That's sort of both
- 1:31:19wings of US politics being like, "Hey,
- 1:31:22what the heck is going on here? This
- 1:31:23looks kind of crazy." Um Yeah.
- 1:31:26And, you know, there's I think there's
- 1:31:28over 30 US congressional offices now,
- 1:31:30Senate and House, that have expressed
- 1:31:33concern about uh big dangers from AI,
- 1:31:36many of which include the the thing a
- 1:31:39lot of these experts are talking about,
- 1:31:40which is it killing us all. And you
- 1:31:41know, I'm not here saying that's the
- 1:31:42only issue. There's a ton of issues with
- 1:31:44AI.
- 1:31:45Right? People are like, "Well, isn't the
- 1:31:46real issue job loss? Isn't the real
- 1:31:49issue uh that it's it's ruining
- 1:31:51education?" And I'm like, "What do you
- 1:31:52mean the real issue?"
- 1:31:54Mhm. You know, like, do you have a
- 1:31:55device that somehow makes there be one
- 1:31:57issue? Because if so, we should really
- 1:32:00not pick mine to go first. You know, I'm
- 1:32:02happy to be at the back of the line, but
- 1:32:04unfortunately, we live in a world that
- 1:32:06permits many issues all at once.
- 1:32:09And I think people are starting to
- 1:32:10realize that AI raises a lot of issues,
- 1:32:12that one of those issues is extinction.
- 1:32:15And you know, like I said earlier,
- 1:32:19people are starting to notice. It just
- 1:32:21takes time.
- 1:32:22Mhm. You know, and
- 1:32:24S- Sorry, go ahead.
- 1:32:26Well, I I was you know, sometimes I ask
- 1:32:28authors I I have people on who've like
- 1:32:30written books 25 years ago or something.
- 1:32:33Um
- 1:32:35and I might say you know, like I'm
- 1:32:38talking to to Brian Greene who wrote The
- 1:32:39Elegant Universe 25 years ago or
- 1:32:41something. And I say, "You know, since
- 1:32:42you published that book, you know, in in
- 1:32:44your field of string theory, what's
- 1:32:46changed?" Cuz the assumption is that
- 1:32:47over those decades, you know, something
- 1:32:49must have developed and something must
- 1:32:50have changed.
- 1:32:51AI moves so fast. I'm almost tempted to
- 1:32:54ask you the same question, which sounds
- 1:32:55ridiculous, which is like, you published
- 1:32:56your book 6 months ago.
- 1:32:58What's changed since then?
- 1:33:00>> [laughter]
- 1:33:01>> Um
- 1:33:03I mean
- 1:33:04more and more people are noticing that
- 1:33:06AI is real.
- 1:33:07And
- 1:33:09uh more and more people are starting to
- 1:33:11react, starting to wake up to this.
- 1:33:13And
- 1:33:15one big reason I have for hope here
- 1:33:18is that the more people are talking
- 1:33:21about this issue,
- 1:33:23the more
- 1:33:25we're just sort of winning.
- 1:33:27Um
- 1:33:30like
- 1:33:31when I have debates with people in the
- 1:33:33field of AI stuff,
- 1:33:37when I, you know, have disagreements
- 1:33:38with the the heads of the AI companies,
- 1:33:42I'm like
- 1:33:44this
- 1:33:46seems really bad. It seems like by
- 1:33:47default it just kills us. And they're
- 1:33:49like, "Nah, I agree there's a lot of
- 1:33:51problems there, but we're going to
- 1:33:52figure it out on the fly and there's
- 1:33:53only 25% chance it kills everybody."
- 1:33:57Right? And and, you know, I can argue
- 1:33:59all day about how their 25% number is
- 1:34:01crazy. I can argue all day about how
- 1:34:02they have no idea what they're doing and
- 1:34:04they're just sort of like winging it and
- 1:34:05they have no real plan. This isn't what
- 1:34:06good engineering looks like. But
- 1:34:10a politician coming into that debate
- 1:34:13does not need to figure out whether I'm
- 1:34:15right or they're right.
- 1:34:17All they need to hear is that the
- 1:34:18optimists are like, "There's a very good
- 1:34:20chance that this kills everybody."
- 1:34:23Yeah. Right? And we've sort of been
- 1:34:25seeing that.
- 1:34:26When I go speak to politicians,
- 1:34:30if they sort of look at the issue at
- 1:34:31all, they're like, "This is nuts."
- 1:34:35And one thing that's changed in the last
- 1:34:366 months is more and more people are
- 1:34:38looking at this issue at all.
- 1:34:39More and more more and more people are
- 1:34:40starting to realize that this is nuts
- 1:34:42and this gives me great hope that
- 1:34:45you know, I also don't know what the
- 1:34:45conversation will look like in a year,
- 1:34:47but I think there's a good chance it
- 1:34:48looks like the world going to the AI
- 1:34:50companies and saying, "We just can't
- 1:34:52keep doing this. This is nuts."
- 1:34:54Yeah.
- 1:34:55>> [snorts]
- 1:34:55>> Well, the book is If Anyone Builds It,
- 1:34:59Everyone Dies. And I mean, the question
- 1:35:01I started was with was whether that's
- 1:35:03something of an exaggeration. People can
- 1:35:06hopefully see why it's now not. But of
- 1:35:07course, if they want more detail,
- 1:35:09the book is in the description. Nate
- 1:35:10Sores, thanks for thanks for your time.
- 1:35:13My pleasure. If you enjoyed that, you
- 1:35:14can watch more of my AI-related content
- 1:35:17by clicking the link that's on your
- 1:35:18screen. To support the show and get
- 1:35:20early ad-free access to episodes,
- 1:35:22subscribe to my Substack at
- 1:35:24alexoconna.com.
- 1:35:26Thanks for watching.
About this transcript
This page contains the full transcript of Nobody Will Survive Superintelligence - AI Expert by Alex O'Connor, generated from the public captions YouTube serves with the video. The transcript has 17,993 words across 2,996 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.