YT YouTube2Text

YouTube2Text → Transcripts → AI Whistleblower: OpenAI Scandal, AI Cults, Neuralink & Our Last Chance to Stop the Tech Oligarchs

AI Whistleblower: OpenAI Scandal, AI Cults, Neuralink & Our Last Chance to Stop the Tech Oligarchs — Transcript

by Tucker Carlson · 20,559 words · 3,210 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:04Nate, thank you so much for doing this.
0:07Um, you've written the world's darkest
0:09book. You've devoted your life to
0:12warning the world about the potential
0:15dangers of AI. So, let's just start by
0:19hearing your explanation of why AI is
0:21dangerous in addition to just being
0:23annoying.
0:25Yeah, you know, the the the very basic
0:28common sense point is
0:32if
0:33you race to make machines that are much
0:36smarter than any human, to make machines
0:38that are more capable uh at inventing
0:41their own technology than humans are,
0:44and you race into this without really
0:46knowing what you're doing, the most
0:49likely outcome is just that the machines
0:52get loose and do their own thing.
0:54and that humanity dies as a side effect
0:56just like humanity has you know killed
0:59lots of other animal species not because
1:00we hate them but just as a side effect.
1:03Uh
1:04and in in some sense a lot of people
1:08find it sort of obvious or intuitive
1:09that if you just like make these really
1:11smart really powerful machines why would
1:13they care about us?
1:15Uh
1:17and I think that intuition basically is
1:19right and there's a lot of arguments you
1:21can have on each side. you can get into
1:23the technical details. Uh, but my book
1:25is basically just um
1:29getting into those details and saying
1:31like, "Yep, it sort of holds up. If we
1:33make smarter machines without knowing
1:34what we're doing, it's just going to go
1:35poorly.
1:38[clears throat]
1:39>> Are we actually going to going to make
1:41machines capable of what you're
1:43describing?"
1:45you know, the the companies are trying
1:46to they um they talk about how they are
1:51pursuing super intelligence in the true
1:52sense of the word. That's Samman's
1:54phrase. Uh Dario Modi of Anthropic says
1:57they're trying to make the equivalent of
1:58a country worth of geniuses in a data
2:00center. Uh so these guys are really, you
2:03know, the these aren't chatbot
2:05companies. They didn't set out to be
2:07chatbot companies. They set out to make
2:10these machines that can sort of exceed
2:12humans in every way. and that's what
2:14they're targeting. There's a separate
2:15question of whether they will get there.
2:17>> May I ask you a question? Why why would
2:18anyone want to build a machine smarter
2:20than people?
2:22>> Um I think that
2:25a lot of them hope that they'll be able
2:28to make the world much much better. You
2:30know, they hope for uh a cure to cancer
2:33and then more than a cure to cancer,
2:34they hope for a cure to aging. they hope
2:36for uh you know a thousand years worth
2:41of technological development compressed
2:42into two years. Um
2:46and I I sort of don't think that
2:50they're going to be able to get that
2:53really or or harness that for good ends,
2:56but that's what I think they're they're
2:57shooting for.
2:59So they're not. But I mean the core
3:01point you're making is these companies
3:03did not set out to make consumer
3:05products like to make your life better
3:07necessarily in the short term with a
3:09more efficient search engine. That's
3:12right. Yeah. You know uh Open AI started
3:14before the the the chatbots the large
3:17language models were even a thing. uh
3:20they were started forget whether it was
3:21late 15 or or early 2016 but um the the
3:25paper that unlocked the the most recent
3:28wave of AI came out in 2017 which was
3:30after open was founded and um
3:34these guys are the the the large
3:38language models are a
3:41surprise revenue stream and that revenue
3:44stream can fund the creation of even
3:47more even larger data centers for the
3:49next level of the technology. But these
3:51guys have their eye on uh the sort of
3:55ultimate version of the technology which
3:56is these machines much smarter than
3:57humans. And that's sort of the ultimate
3:59version because once the AIs are smarter
4:01than the humans, uh the AIs can carry on
4:04the AI research and makes the next
4:05generation of the AIs which make the
4:07next generation of the AIs and um that's
4:10sort of what they're shooting for.
4:12>> What is super intelligence?
4:15We in our book define super intelligence
4:17as an AI that is uh better than the best
4:21humans at every mental task.
4:24So uh anything that a human can do
4:27purely mentally, the AI can do that
4:29better. One one thing people often get
4:31caught up on about this is uh that
4:34includes the AI being better at things
4:36like persuading humans, at things like
4:38charisma. You know, we often think of
4:40intelligence as the stuff that the nerds
4:42have and that the jocks lack,
4:45>> but
4:45>> you win a chess match,
4:46>> right? It's the chess guys rather than
4:48the than the politicians guys or the or
4:50the rock stars.
4:51>> But that's not really what the
4:54intelligence in artificial intelligence
4:56means. The intelligence in artificial
4:58intelligence is about the stuff that
5:00humans have and that mice don't.
5:04It's it's sort of the whole package. you
5:07know, like the the politician who's very
5:09charismatic.
5:11It's not it's not that like chess
5:12playing happens in your brain and
5:14charisma happens in your kidneys, right?
5:16They're sort of both mental functions.
5:19>> Yes.
5:19>> Uh and
5:21so the the sort of super intelligent AIS
5:24are are not just super good at playing
5:27chess, they're also good at super
5:28persuasion and they're super good at the
5:30the research and at the technological
5:32invention.
5:34So super intelligence as you define it
5:36is a machine that is better in in at
5:41every mental function than any human
5:43being.
5:43>> That's right. And uh
5:47you know the the the definition
5:50that doesn't mean that that nothing
5:52crazy will happen on this planet until
5:53the AIS are super intelligent. Super
5:56intelligence in this definition is sort
5:57of a point where past this point things
6:00must be pretty crazy because now the AIs
6:02can do automated AI research and they
6:04can make smarter AIs and they can figure
6:05out how to run the robot factories and
6:06they can build the better robots and and
6:09all that. You could have things start to
6:11get crazy before you have a super
6:13intelligence in this sense. You could
6:14have AIs that are much better in than
6:16than humans at some tasks and much worse
6:18in others that are still causing all
6:20sorts of crazy happenstances.
6:24uh the the super intelligence is sort of
6:26a like once you get here stuff's crazy
6:29point. It's not a things will stay
6:31normal until you get here point.
6:35>> So why is super intelligence the
6:37significant milestone that you're
6:38worried about?
6:40>> Um I mean I would say that
6:43uh I'm I'm also worried about what'll
6:45happen before that milestone. It's more
6:48like like having the definition makes it
6:51easy to talk about like how crazy would
6:54things get
6:56uh once you are past this point or sort
6:59of like you know let's like if we assume
7:02that the machines get here what happens
7:04and then the answer is like it's it
7:06would be pretty crazy it would be it
7:07would be pretty wild and uh it sort of
7:10like lets you factor the conversation
7:15uh into like what how how to Okay, I
7:19think in the artificial intelligence
7:20conversation, there's actually a lot of
7:21conversations going on. One conversation
7:24is like, can the machines really get
7:25smarter than the humans? One
7:27conversation is like, how fast could we
7:29get there? Is the current technology on
7:31that route? Another conversation is what
7:34happens if we do get there? Like, you
7:37know, would would the AI care about us?
7:39Would they not care about us? What would
7:41they be able to like what would they
7:42what would they try to do? There's
7:43another question which is what would
7:44they be able to do, right? Right? And
7:46these are all sort of different
7:47conversations about AI. And the super
7:49intelligence definition,
7:51it's not really like here's where all
7:52the fears hang. It's more like it lets
7:55us
7:57break those pieces out separately and be
7:59like, well, is super intelligence
8:01possible? Separate from, well, what
8:03would happen if you had one?
8:05>> What would happen if you had one?
8:07>> Most likely outcome, uh, I think
8:09destruction of the planet. So, our
8:11friends at Pre-born, who are actual
8:12friends, by the way, just asked us to
8:14thank the listeners of this show for
8:15supporting the pro-life cause. [music]
8:17A lot of people do all of a sudden,
8:20trust me, because what's happening is
8:23crazy and people are waking up to that.
8:25The president of the group is our friend
8:27Dan Steiner, and he wants Americans to
8:29know how much of a difference they have
8:31made. This year's donations have helped
8:34saved 50,000 babies from abortion.
8:36That's enough to fill a professional
8:38baseball stadium. 50,000 lives saved as
8:40well as generations that will follow
8:42them. Pre-born wastes zero money. We're
8:45not big on charities because a lot of
8:47them waste money. Pre-born does not. It
8:49uses every taxdeductible gift to rescue
8:52babies and share the hope of Jesus with
8:54their moms. In 2026 alone, over [music]
8:568,000 women made the decision to follow
8:58Christ. The generous support of
9:00Preborn's donors made that all possible.
9:04$28 to provide one life-saving
9:06ultrasound. is giving expectant mothers
9:09the knowledge that they have a baby that
9:11they're carrying. $15,000 can place an
9:14ultrasound machine in a pregnancy clinic
9:16where it can save countless lives. Dan
9:18Steiner's message is really simple.
9:20Thank you for helping. Your generosity
9:21is literally saving lives. To help
9:23pre-born save more, dial pound 250 and
9:26say the keyword baby. That's pound250bab
9:29or visit pre-born.com/tucker.
9:32pre-born.com/tucker.
9:35How what would that look like?
9:38So, I'm going to be a little bit
9:39annoying here and say uh a couple
9:43caveats first because it's it's sort of
9:47a tricky one to to sort of predict
9:49things that are smarter than us.
9:50>> Yes.
9:52>> And the the first annoying caveat I will
9:54give is uh
9:57if you go to play a game of chess
9:59against Magnus Carlson,
10:02I predict you will lose. No offense.
10:06uh he's just the best human chess player
10:07alive.
10:10If you then ask what piece will he use
10:13to checkmate me? I'm like well that's a
10:16much harder question.
10:17>> Yes.
10:18>> Right. It sort of is like very easy to
10:19predict the winner. It's hard to predict
10:21the exact methods.
10:23So uh my my prediction that humanity
10:26ultimately dies
10:29is a different sort of prediction than
10:32my prediction about like it could go
10:33this way, it could go that way.
10:35Um, so I'll give you some stories, but
10:37these are stories that are like, well,
10:39maybe Magnus will like fork your queen
10:42and your rook with his knight and then
10:45finally use the queen to checkmate you.
10:46Like, yeah, that could happen, but I'm
10:48it's it's a different sort of
10:49prediction. It's a guess, right?
10:52The the guess here is
10:56uh that if we manage to make these
11:00really smart AIs, they will have their
11:03own
11:04stuff that they're pursuing, which is
11:07not quite what we wanted, not quite what
11:08we asked for. It'll have this other
11:10strange stuff. Um there's a whole
11:12discussion about why that happens, but
11:14we're already starting to see it in
11:15practice with some of the recent events.
11:17um [clears throat]
11:19and
11:22uh they would be able to pursue whatever
11:25it is they're pursuing
11:27much more efficiently than humanity can.
11:32And so we're talking about, you know,
11:33automated factories with uh that produce
11:36the robots that produce the factories in
11:37a fully automated supply chain. And then
11:40if the AIs have anything they're trying
11:43to do that they can do more of with more
11:46resources,
11:48they start gobbling up the resources on
11:50the planet. Sort of like how humanity
11:52spread and started gobbling up all the
11:54resources on the planet. Uh and the
11:58uh you know the the sort of most basic
11:59thing to visualize here might be you
12:01have factories that produce robots that
12:04produce factories that produce robots
12:05that also build data centers and you
12:06just have these you know fully
12:08autonomous self-replicating
12:11ecosystems of of robots and factories
12:13and data centers that don't care about
12:15the instructions human humans gave them.
12:18uh and that
12:20cover the planet, take all the
12:22resources, take all the sunlight, take
12:23all the places we were growing crops,
12:25probably raise the temperature of the
12:27planet because uh you can compute more
12:29efficiently. Well, technically the the
12:31earth radiates more heat when the world
12:32is hotter and that's lets you sort of do
12:35more computation and then sort of you
12:37know
12:38>> it's good for the machines to have a hot
12:39planet.
12:39>> It's good for the machines to have a hot
12:41planet. Yeah. um that the sort of
12:43physical limits on how much computing uh
12:45you can run on the surface of Earth is
12:47bounded by how much heat you can radiate
12:49to space. That's sort of the first
12:50constraint you bump up against. You
12:52might think that it's energy, but
12:53actually there's a lot of helium
12:55hydrogen to fuse on this planet. So you
12:57can get plenty of energy. What you need
12:58is uh heat dissipation and the the world
13:02can dissipate more heat um when it's
13:05hot. So if you're, you know, imagining
13:09some collective of AIs that are trying
13:11to run a lot of computing power, uh,
13:13trying to do a lot of computation for
13:14one reason or another, they sort of
13:16prefer the planet running hotter.
13:19And so it sort of is like nothing
13:20personal. It's just if you let these AIs
13:22get out of control, uh, they
13:27transform the planet into something
13:28unlivable.
13:30>> How hot?
13:32uh as hot as
13:35uh you can while still having the
13:37computers not melt. So probably hundreds
13:41of degrees.
13:44>> Um at which point you hear people say,
13:46"Well, then just turn it off."
13:49>> Uh so I think we do have an opportunity
13:52to turn it off, you know, and I'm not
13:54here saying that we're going to die. I'm
13:56here saying we sort of are going to need
13:57to act. You got to be careful about the
13:59turning it off piece because um
14:03your opportunities to turn the AI off
14:06only last when the AI is running on the
14:09computers you know it's running on. If
14:11you have the AI breaking out and running
14:13on hidden computers.
14:15If you have the AI running robot
14:17factories that produce robots that are
14:20under the control of the AI where those
14:22robots can then go build more computers
14:23that are not hooked up to your network
14:25that can run the AIS.
14:27These are sort of thresholds where the
14:28AI is able to keep itself running
14:32and you better have turned it off before
14:34that point
14:38because after that point it's
14:39impossible.
14:41>> Yeah. I mean the
14:46if if
14:48it it it also gets a lot harder if the
14:51AI knows you're going to try to shut it
14:52off and is going to try to stop this,
14:54right? You got to you got to remember
14:56that we're talking about things that are
14:57smart here. And so if the AI sort of
14:59sees it coming, maybe it defects to
15:01North Korea where it can, you know,
15:03convince them to run it on its data
15:05centers in ways that initially benefit
15:07them, but that ultimately benefit the
15:09AI. I
15:10>> I think we're going to have to redefine
15:11what life is because you're describing a
15:13living autonomous thing.
15:16>> Yeah. Artificial life in a sense. I
15:18mean, in some sense, we're already
15:20seeing the very very beginnings of that.
15:23Um
15:24but yeah, we once once the AIs can
15:27replicate, once they can improve
15:30themselves,
15:32uh once they can
15:34uh find ways to run where you don't know
15:37that they're running,
15:39once they can
15:41uh
15:43sort of run the robot factories to
15:47produce the robots that can produce more
15:49factories and that can produce computers
15:50that the AI can run on, then yeah, you
15:52have you have in some sense has made a
15:55new artificial life form.
15:58Uh,
16:00and you know, humanity is on the top of
16:02the food chain right now because we
16:05are sort of the only smart life form
16:07around.
16:08>> Yeah.
16:08>> If you suddenly make a new one that is
16:11smarter, that is able to make a million
16:14copies of itself, that is able to run
16:16faster,
16:18it's just kind of a crazy thing to race
16:20into.
16:21It's a weird thing to want.
16:25And as it's developed, I was going to
16:28say slowly, but it hasn't been
16:29particularly slow, 10 years.
16:32The rest of us have watched as
16:36people who
16:38I mean, put it in political terms, don't
16:39share the values of most Americans are
16:41now in charge of this. And it's almost
16:44like everyone sat passively by as it
16:48happened.
16:49>> Yeah. I I think I think there's a lot
16:51going on there. I think part of it is
16:53that most people didn't and some still
16:56don't believe that AI was going to be
16:57able to keep going. I think there's a
17:00lot of like putting the head in the sand
17:02and being like, "Oh, it's, you know,
17:03just slop. It's going to be a bubble.
17:05It's going to pop. It's going to hit a
17:07wall. Uh it's not going to be able to
17:10improve anymore."
17:12And I think a lot of that was sort of
17:14wishful thinking. And I think that at
17:16the very least
17:18if people are like we are making, you
17:20know, the the super intelligent machines
17:21that are going to replace humanity as
17:23the as the top of the food chain because
17:25we think it's going to go go great. I
17:27think humanity's response should not be
17:30go ahead and try. We hope you'll fail. I
17:33think the response should be like, "Hold
17:35on, that's kind of crazy."
17:37Um,
17:40yeah, I had I had some other point too,
17:41but I've forgotten it. So
17:43>> how uh predictable has the evolution of
17:46AI been?
17:48>> Has it taken turns that you didn't
17:50expect?
17:51>> Yeah, totally. Uh I was so back before
17:55the large language models.
17:57>> How long have you been following this
17:58issue?
17:59>> Uh I started following it in 2012 and I
18:01started working on uh I I started
18:04working with some of the people trying
18:06to make this go well in 2013 and I
18:08started full-time in 2014.
18:10>> A long time. So over a decade.
18:12>> What has surprised you?
18:14>> You know the the large language models
18:17um
18:19they have gone further than I expected
18:22initially and I think one of the big
18:25surprises here was AI undergoing a phase
18:29where everybody can see it.
18:32you know, back before the large language
18:34models,
18:35uh, really only nerds paid attention to
18:38AI. And it wasn't that there was nothing
18:41happening. You know, we were watching,
18:43uh, the, you know, Google Deep Mind make
18:46an AI that could beat uh, the the best
18:50Go player.
18:51>> Yes.
18:51>> And that was sort of a milestone for
18:52people who were paying attention. Um,
18:55but for all we knew,
18:58the labs were going to keep on working
19:00on engineering problems and keep on
19:02working on the relatively nerdier
19:04problems and you were never going to
19:05have like a mass market consumable
19:07product. And so like for all we knew, it
19:10would just stay in the labs and no one
19:11would ever really notice that these guys
19:13were gambling with with the whole
19:14future. Now at least, uh, AI is
19:18everywhere. everyone's starting to have
19:19the conversation and that is in some
19:21sense actually really quite optimistic
19:23because it it gives the world an
19:24opportunity to see what these guys are
19:27trying to do and say hold on
19:32I guess you could flip it around uh and
19:34conclude that because the overwhelming
19:36majority of people seem to be very
19:37opposed to way and I don't even know
19:39anyone who's for it every college
19:42commencement speaker who mentions it
19:43gets booed.
19:44>> Yeah. And that view, that opposition to
19:49it has had no effect at all in slowing
19:51it down.
19:53I don't know. Does that give you hope?
19:55>> Um,
19:57I mean,
20:00I I I hear this a lot. I would say when
20:03you're coming at this from 2012,
20:05>> Yeah.
20:05>> it really feels like we're making
20:06progress,
20:07>> right? No, that's fair.
20:08>> It It doesn't feel like we're all the
20:09way there yet.
20:11Um,
20:12>> but can I just make the point China,
20:14China, China, China, China. We have to.
20:16It's because of China.
20:18>> Yeah, you know, I think uh China and the
20:22USA have a shared interest in not dying
20:24to a rogue super intelligence.
20:26>> Storm season is on the way and that
20:28means your Wi-Fi could go down and if it
20:31does, your security cameras go dark,
20:33too. So, at the exact moment you want to
20:35survey your property during the
20:37blackout, you can't. Unless you install
20:40cameras by Defend. Defend cameras run on
20:42cellular, not Wi-Fi. [music] And that
20:44means they keep working even when you
20:46lose power. You don't choose a carrier,
20:49you don't sign a contract, you just
20:50mount it, open the Defend app, and bam,
20:52you're connected. It's that simple. So
20:54whether it's your ranch, a job site,
20:56your backyard, Defend works where Wi-Fi
20:59doesn't and keeps working when the grid
21:01goes down, which it may. Plans start at
21:03just $5 a month. Most people pay around
21:0510. Plus, there are no binding
21:07contracts, meaning you get to cancel
21:08whenever you want. So unlike big tech,
21:10Defend will never sell your data. Not to
21:12China, not to the government, not to
21:13anybody. It's an American company built
21:15for people who want actual security.
21:18Visit defendellcam.com
21:20and use the code Tucker for 20% off.
21:22That's defendellcam.com
21:24code Tucker.
21:26Is it conceivable that like let's just
21:29say China can I'm I'm sure by the way
21:32that China is showing more restraint
21:34than we are in this.
21:35>> Yeah. I mean and if nothing else they
21:38sort of uh have a lot more reason to
21:41censor their AIS,
21:42>> right?
21:42>> And to try to make them, you know, not
21:44say certain things to the to the
21:45population. So um but if if the United
21:49States you know if the US government
21:51were somehow able to get control of the
21:54tech sector which is at present not
21:56possible but let's say it did
21:59let's say the president was more
22:01powerful than the tech oligarchs which
22:03he is not but for the sake of argument
22:06let's say he was and he shut it down
22:08would that matter if if some Indian lab
22:11or Chinese lab created it so it does
22:13have to be a global stop to this
22:16creation of super intelligence. Uh I I
22:18think there's a couple reasons why this
22:20is possible. One is I think a lot of
22:22people um
22:25they they hear that some parts of AI
22:28need to be stopped and they think I'm
22:31saying that all parts of AI need to be
22:33stopped and uh you know there's there's
22:37all sorts of interesting issues with you
22:39know AI and education and AI and you
22:42know military drones that that are uh
22:45that society has to wrestle with but
22:47these are all sort of separate issues
22:48from the super intelligence race And if
22:51you sort of like tease those issues
22:53apart, uh it becomes much more possible
22:55to do like a a more surgical
22:58intervention on we're not going to race
22:59towards the super intelligence in ways
23:01that leave a lot of the rest of the
23:02sector alive,
23:04which um
23:06which which makes it sort of an easier
23:12coordination effort to to attempt. And
23:16on that front, on the super intelligence
23:18front,
23:19it's not that, you know, just shutting
23:21down the the domestic race towards super
23:24intelligence would be enough. But, uh,
23:29racing towards super intelligence
23:31requires a huge amount of highly
23:33advanced computer chips that can only be
23:36produced as sort of the peak of the
23:39global supply chain.
23:40>> Yes.
23:41>> Uh, which is largely controlled by US
23:44allies.
23:45And so there is absolutely a possibility
23:49that uh a US-led coordination effort
23:54could say look we're not doing uh the
23:56super intelligence thing. We're going to
23:59monitor the extremely heavy chip
24:01concentrations of like you know 10 or
24:03100 thousand of these highly advanced AI
24:05chips. uh we are going to
24:08uh make sure that they are not doing
24:11these super intelligence training runs
24:14and
24:16I I think there's a way if the US was
24:19leading that effort there would be a way
24:22to get China on board and set this up in
24:24a way that was moniable enforceable
24:26verifiable
24:27>> yes
24:28>> uh we we sort of in many ways would be
24:31easier than a nuclear arms treaty
24:32because uranium is a rock you can dig
24:35out of the ground whereas is these
24:36highly advanced computer chips sort of
24:37only come out of one fab in Taiwan, you
24:40know, and so it's like
24:40>> those are TSMC chips.
24:42>> Yeah. Uh and so and there's other parts
24:45of the supply chain that are very narrow
24:47like the lithography machines that come
24:48out of the Netherlands. And so um yeah,
24:52we we could absolutely lead a uh a
24:55global effort to say we're not making
24:57the super intelligent machines and it
24:59would be a bit tricky, but it's just we
25:01could just do it if we had the political
25:03will.
25:05I I think that
25:07people either don't know what's
25:09happening or they assume that like a lot
25:13of fears this fear will turn out to be
25:16groundless.
25:18People point to Y2K.
25:20>> Yeah. Um you know I hope the fears are
25:23groundless. I think with a lot of past
25:25fears uh the
25:28what happened is not that the fear was
25:30groundless. it's that
25:33uh people noticed the issue and put in a
25:35ton of work to make the bad thing uh not
25:39happen. You know, I think we saw this
25:41with the hole in the ozone layer where
25:43people are like, "Oh, whatever happened
25:44to the hole in the ozone layer?" Well,
25:45what happened with the hole in the ozone
25:46layer is that we fixed it. It's not that
25:48it was fake. It's that, you know, we
25:50went and banned the chlorofhllorocarbons
25:52that were actually blowing this hole in
25:53the ozone layer. found other ways to,
25:55you know, cool refrigerators that worked
25:58similarly well and didn't uh, you know,
26:02put this put this hole in the ozone
26:03layer. I think Y2K was actually one of
26:05these cases where you had a ton of
26:07software engineers
26:09up, you know, in 1999 and a couple years
26:11prior that were scrambling to update all
26:14of the software so that they would be
26:15able to handle uh, dates past 1999.
26:20And they got it done in time. None of
26:23the big systems went down, but it wasn't
26:25that, you know, the the issue was fake.
26:28It took a lot of work. It's just that
26:30that work happened behind the scenes.
26:33>> Are people doing work to slow down AI?
26:37>> Uh, I'm doing my best,
26:38>> right? [laughter]
26:39>> It's it's it's not really there yet, you
26:41know. Um,
26:43if I if I can belabor this point a
26:45little because I think it's kind of an
26:46important point. um just with more uh
26:50historical cases, which maybe it'll bore
26:52everybody else, but it'll it'll
26:54entertain at least us.
26:55>> It'll definitely entertain me.
26:57>> You know, you have if if you look back
26:59across history, you see
27:02there's definitely been some warnings
27:03that didn't come to pass, right? There
27:04were I think they were called the
27:05Masonites in like the 1960s or sorry, in
27:08the 1860s, maybe those 1880s, I forget
27:11exactly, but there were the Masonites
27:12who were an end of the world cult.
27:13>> Yeah.
27:14Uh and you know it the world did not
27:18end, right? But also in the 1880s you
27:21had Ottovon Bismar saying you know
27:23Europe is a tinder keg and some damn
27:25fool thing in the Balkans is going to
27:27light it. Right? That did happen.
27:30>> It sure did.
27:31>> You had uh you know in the 1920s you had
27:33scientists saying don't put lead in the
27:35gasoline. It'll poison children. And we
27:38put lead in the gasoline and it poisoned
27:39a lot of children. And we said whoops.
27:40And we took the lead back out. Uh the
27:43the world is
27:47the world is a place where a lot of
27:49people make a lot of types of warnings
27:51and some of them are real like the
27:53gasoline and some of them are fake like
27:54the Masonites.
27:56And so you you sort of can't use a rule
27:59that's like every warning is real and we
28:01need to listen to it. And nor can you
28:03use a rule that's like every warning is
28:04fake and we dismiss it. you sort of just
28:07have to look at the the details to
28:11figure out whether this one is a real
28:13one. And then the the last thing I would
28:16say on this point is
28:18um
28:21the
28:25a lot of people in the 50s warned of
28:28nuclear Armageddon.
28:30>> Yes.
28:31>> And it sort of makes sense if you look
28:34at the history leading up to that point.
28:36uh humanity had sort of never before
28:38actually failed to use their strongest
28:41weapons in combat. Uh and you know these
28:45people were living in a world that had
28:46had World War I and then the League of
28:49Nations and you know the World War never
28:52again and they tried to invent these
28:54whole new governance structures to
28:55prevent it from happening again and that
28:56immediately failed and led to World War
28:58II. And so, you know, in 1950, you're
29:00sort of in this world of like it just
29:02looks pretty grim.
29:05And we haven't had nuclear Armageddon
29:07yet. And it's not because the bombs were
29:10fake.
29:12It's not because the nukes were hype.
29:15It's because people saw the issue and
29:19worked really hard to avoid it day in
29:22and day out for decades through crisis
29:26and succeeded. And so I think one of the
29:28big ways you can tell the difference
29:30between someone coming in and
29:32proclaiming that the apocalypse is nigh
29:34and someone who's saying look we have a
29:36problem and we need to fix it is whether
29:38that person is saying uh we're
29:40definitely screwed versus whether that
29:42person is saying here's a problem let's
29:44try to fix it and you know the the thing
29:47I remind people about here is that the
29:50very first word in the title of my book
29:52is if I'm not here saying we are going
29:55to die I'm here saying if we race down
29:58this path, we die. But that same if is
30:02what points the way towards changing
30:05paths. And we still have plenty of time
30:07to change paths. Home ownership has
30:09always been at the center of the
30:10American dream. It is an actual asset, a
30:13place [music] people work their entire
30:15lives to build, a place of stability and
30:17security and real value. Right now
30:20though, too many homeowners are stuck
30:21swiping credit cards to cover groceries.
30:24There's basically a survival tax of 20%
30:26interest or more. So why keep handing
30:29your money to the banks [music] when you
30:30could keep it for your family? Interest
30:32is what kills you. Our friends at
30:34American Financing have a better way.
30:35They're helping homeowners use their
30:37hard-earned equity and they're
30:38supporting Americans looking to fulfill
30:40the dream [music] of owning a home with
30:41mortgage rates currently in the fives.
30:44American Financing saves its customers
30:46an average of $800 a month. That's about
30:4810 grand a year. It's not just a loan.
30:51It's a true financial reset. [music] We
30:54recommend American Financing. They're
30:55called America's Home for Home Loans for
30:57a reason. So, get out of credit card
31:01debt. I mean, it is just crazy to borrow
31:03money. No judgment, but it doesn't make
31:05any sense to borrow money at that rate.
31:07If you want out, call 8006855696.
31:12800685-5696
31:15or visit americanfinancing.net/tucker.
31:20I'm struck by how little of this was
31:27planned by anybody
31:30by how little effect the government has
31:32had on it. Not that I'm for the
31:33government. I'm pretty opposed to the
31:34government most of the time, but I'm
31:36also for people being able to control
31:38their own country and the only mechanism
31:39by which they can do that is voting. So
31:42it it's like these tech companies act
31:45independent of what the population
31:47wants, what the government wants. It
31:48doesn't.
31:49>> Yeah.
31:50>> Right. They're more powerful than the
31:51government. I guess that's the point I'm
31:52making
31:53>> in some sense. Um I I think that they
31:56have that power more and more when
31:58people don't really understand what
32:00they're doing. You know, we saw just in
32:02June um there was uh there was an AI
32:07made by Anthropic called Claude Mythos
32:09that was very cyber capable and they
32:11released a version of it that was
32:12supposed to have more guard rails called
32:14uh Claude Fable. And um it sort of
32:17turned out that you could jailbreak it
32:19to get some of those cyber capabilities.
32:21Um
32:22>> can you explain what all that means?
32:24>> Yeah. So uh
32:27how to say um in January
32:30>> cyber capable means it can find the
32:32internet.
32:33>> Cyber capable means it can hack into
32:34basically anything. Yeah. So, uh, one of
32:38the holy grails of hacking is can you
32:41make a website where if you just look at
32:43the website, I get full control over
32:46your computer or your phone.
32:47>> That's very hard to do. Usually, you
32:50need to like click some link and
32:51download something or run something.
32:53>> Yeah.
32:54>> Uh, and you know, I don't know the exact
32:56numbers because I don't have the
32:57security clearance to know the exact
32:58numbers, but a decent guess is that in
33:00January of this year, there were only
33:02two entities that could pull that off.
33:06MSAD and the NSA.
33:09In March of this year, there were three.
33:12MSAD, the NSA, and Claude Mythos,
33:16>> which is this new AI made by Anthropic.
33:19And so, it sort of became superhumanly
33:22capable at hacking.
33:25Uh and it it sort of became that way
33:30overnight, you know, like really there
33:31was 6 months to a year of training and
33:33the people ananthropic maybe saw growing
33:34in these cyber capabilities. But from
33:37the perspective of you know the rest of
33:38the world and the the cyber security
33:40community and the national security
33:41community.
33:43This sort of happened overnight
33:45and uh they they now have a project uh
33:49called project glasswing where they are
33:52trying to use claude mythos to find
33:56critical vulnerabilities in critical
33:58software and patch them before the rest
34:01of the world sort of gets these
34:03capabilities by for instance you know
34:04the the open source models or the open
34:06weight models
34:07>> right
34:08>> um catching up and so that sort of
34:11caused this big um
34:14this big ruckus.
34:16But then Anthropic also wanted to sort
34:18of sell access to this model and they
34:21tried to make a version that didn't have
34:22as much hacking ability
34:25uh which was called Fable instead of
34:26Mythos. And um some people I think at
34:30Amazon found that uh if you sort of put
34:34some pressure on Fable it would be able
34:37you would be able to get at some of
34:38those hacking abilities. And when that
34:40happened, uh, the the Trump
34:42administration put an expert control on
34:44Cloud Fable and said, "You can't let
34:46this be used, um, by non-citizens
34:50and with, you know, 90 minutes of
34:51notice." And that essentially shut down
34:53access to Cloud Fable because they
34:55didn't have the ability to verify the
34:56users. And this this is sort of showing
34:59us and, you know, you could you could
35:01talk about whether or not there was some
35:03personal feuds between people in
35:05Anthropic and the administration that
35:07exacerbated this. I don't know. But it
35:09sort of shows that the administration
35:13is
35:16more powerful than the tech companies
35:17still when it wants to be.
35:21And I think a lot of the reason we're
35:23seeing these tech companies able to race
35:26unopposed is that people just aren't
35:29paying attention to what they're trying
35:30to do or they don't believe that they'll
35:32succeed.
35:34I'm still confused by why anyone would
35:36want to do what they're doing. I mean,
35:38typically,
35:40you know, a company that sells computer
35:41products, consumer products, or or any
35:44company, you're making something that
35:45you think people would want for some
35:48specific purpose
35:50that improves the lives of the people
35:52who buy it. But creating super
35:55intelligence, like what I I don't
35:58understand it. Why would you do that?
36:00You know, I think the dream is uh you
36:02know, you're going to have an AI that
36:04can uh like solve all sorts of
36:09engineering and math problems and unlock
36:10all sorts of new technological
36:12possibilities and an AI that can cure
36:15cancer and not only cure cancer but cure
36:16aging and, you know, invent the the sort
36:19of like nanotech that can reverse aging
36:21and let people live a really long time.
36:24and you know invent the technology that
36:26lets you like digitize brains and uh
36:30like
36:31uh like travel to the stars and it's
36:34sort of like if you imagine compressing
36:36a thousand years of technological
36:37progress
36:38>> into a year.
36:40>> This is sort of the dream.
36:43It seems like a religious quest though
36:45cuz it does seem decoupled from those
36:47specific goals. It seems like the main
36:50drive is to build something smarter than
36:52people to build a god.
36:55>> Yeah. You know, they they they bandandy
36:58around um the phrase the machine god or
37:01the sand god in Silicon Valley. Um sand
37:05because silicone silicone. Um,
37:09and there's definitely some people, uh,
37:12there's a there's there's folk who talk
37:15about,
37:16you know, the the AI is replacing
37:19humanity and that being good.
37:21Uh, I I frankly don't engage with these
37:27folks that much because
37:30um that viewpoint sort of makes me
37:31uncomfortable.
37:32>> Why does it make you uncomfortable?
37:35I think
37:38I think there's probably two schools.
37:41Again, I'm not the expert. I'm not an
37:44anthropologist here. Uh
37:47I I think there's two schools among the
37:49people who sort of want AI to replace us
37:52all.
37:53One school sort of imagines that we'll
37:56merge with the AIS, that the AI will be
37:58really nice and friendly, that they'll
37:59be wiser than us, better than us, kinder
38:01than us, and that it'll sort of be like
38:04an upgrade and that the AIs will be able
38:06to love, experience joy, you know, uh
38:10like they'll they'll treat the universe
38:12better than humans and it'll be sort of
38:13like having a child. And then like it's
38:16sort of okay if that child uh like isn't
38:20exactly the same as us but sort of like
38:23uh succeeds us as some sort of like
38:26worthy successor some sort of like
38:28worthy worthy progyny of humanity. And
38:30they're like ah yeah then if like flesh
38:32and blood humans sort of like wayne
38:34because everyone's choosing to upload
38:35themselves into the the collective
38:37intelligence or whatever that's sort of
38:38fine. And um
38:43and then I think there's another camp
38:44that sort of is like
38:46uh
38:48like this is inevitable. This is just
38:50the the way of progress. You know, the
38:51machines will uh like humanity is just a
38:54bootloader for artificial super
38:56intelligence and you can't stop it. So
38:58you might as like if you can't beat
38:59them, join them, right?
39:01The former I think are misled. The
39:03latter, I think, are evil. Much closer
39:06to evil. Yeah.
39:09Well, I mean, if you're actively working
39:11toward the extinction of people, then I
39:13think we can say that's evil, can we?
39:16>> Yeah. Uh, and
39:18um, who's in that category? Which is Sam
39:21Alman in that category, do you think?
39:23>> I don't think so. My sense is that the
39:24guys running the labs have these utopian
39:28visions.
39:29And
39:30>> utopian or dystopian?
39:32>> I mean, there's a thin line. I think
39:34>> Yeah. Well, fair. Good point.
39:36>> I think they have visions that in their
39:37head are utopian. I mean, frankly, my
39:39stance on all of this, like I I often
39:42try to stay away from a lot of this
39:43because
39:45um from my perspective, it's all sort of
39:48in fantasy land. From my perspective,
39:52everyone's sort of saying like, "Oh,
39:54we're going to make the genie, and then
39:55what are you going to wish for on the
39:56genie? What am I going to wish for on
39:57the genie? Who should be in control of
39:59the genie? Who gets to keep the genie on
40:00the leash?" And I'm sort of like,
40:03A,
40:06this is not going to be the wish
40:07granting sort of genie.
40:09B, it's not staying on the leash.
40:12>> No.
40:12>> Like, uh, you know, we can we can sort
40:15of talk about like what's driving these
40:16people and and and what utopias they're
40:18envisioning and whether if their genies
40:20would stay on a leash, whether they
40:21would get the utopia or some other
40:23dystopia. And like how hard is that
40:25needle to thread? But I sort of like,
40:27you know, it's it's it's all these
40:29people uh, you know, like
40:32building the golem, fantasizing about
40:34who gets to control the golem, and it's
40:36like this just not how
40:38>> summoning spirits is never a good idea.
40:40>> Yeah. Yeah. It's like Yeah. It's like
40:41it's like all these people, you know,
40:42drawing a pentagram being like, I'm
40:45going to I'm going to summon a demon and
40:46like, it's going to be so nice when the
40:48demon does what I say. Like, oh no, your
40:50thing's slightly wrong. It's going to be
40:50so nice when the demon does what I say.
40:52And I'm like,
40:53>> demons are bad.
40:54>> Yeah. And
40:56I I think the demon thing's a little bit
40:58different because demons are often, you
41:00know, portrayed as malicious and here
41:01it's much more like indifference.
41:04You know, it's not like you make a demon
41:05who sort of it's not like you summon a
41:07demon who sort of um
41:09like enjoys making
41:13like enjoys wrecking havoc. It's more
41:15like you summon a demon that's like
41:17really into building more computers and
41:19calculating weird things and just will,
41:24you know, uh,
41:27take all of the matter that we were
41:29using to survive and turn it into more
41:31factories and data centers, right? My my
41:34co-author has a quote,
41:34>> death by data center.
41:36>> Death by data center. Yeah. Fully fully
41:38automated, self-replicating data center.
41:41Yeah. My uh my co-author has a quote um
41:43the AI does not hate you but nor does it
41:46love you and you are made of atoms it
41:47can use for something else.
41:52>> You're just biomass.
41:54>> You're just biomass. Yeah. And if you
41:55sort of run the calculations there's a
41:57there's a fascinating paper um called uh
42:00limits to global ecoag which is to say
42:03what are the physical limitations on how
42:05quickly you can consume the resources on
42:07the planet if you are trying that. And
42:10um burning biomass is actually, you
42:14know, much more efficient than
42:15collecting sunlight. If you look at an
42:17average square meter of the planet, you
42:19can get about 10 times the energy from
42:21burning the biomass as you can from
42:22collecting the sunlight that falls on
42:23it.
42:24>> So you'd think at some point like
42:27[clears throat] if our richest sector of
42:29our economy is like building crematoria
42:33for the rest of us, someone would say,
42:35"No, we're not doing that."
42:38>> Yeah. I mean,
42:40it's it's sort of a crazy situation. Um,
42:45a lot of these guys who are in the race
42:48acknowledge that there's a ton of
42:49danger. You know, you have Elon Musk
42:51saying 10 to 20% chance this kills us
42:53all. You have Dario Modi saying he
42:55thinks 25% chance it goes
42:56catastrophically wrong. I think those
42:58numbers are low. I think these guys are
43:00like the crazy optimists. It sort of is
43:02like um like if you have an engineer
43:06building a bridge and they're like,
43:07"I've never worked with these materials
43:08before." And you're like, "Man, I think
43:09that retaining wall is going to go down.
43:10I've like studied that retaining wall. I
43:12think it's going to fall." They're like,
43:12"Yeah, we understand that the retaining
43:14wall is like looking a little shaky. We
43:17don't know how we're going to fix it,
43:18but we're going to like have some guys
43:19fixing it on the fly, inventing new
43:21materials. We think we're 75% chance the
43:24bridge stays up.
43:25>> And by the way, it'll be the longest
43:27suspension bridge in human history."
43:28>> That's right. and we're loading
43:29everybody onto a car and driving it over
43:31the very first time without testing. And
43:32I'm like, look, that's not what real
43:34engineering sounds like. This is not
43:36what it sounds like when the engineers
43:37have a 75% chance of success, right?
43:40This is what it sounds like when they're
43:41sort of winging it and and like these
43:43these are cowboys. These are not real
43:44engineers, right? But even if you set
43:46that aside, even if you take these guys
43:48at their word for these like 10 20%
43:51numbers, that's insane.
43:54NASA accepts a 1 in 270 chance that a
43:57crude flight goes down of seven
44:00volunteers,
44:02right? To be like, "Oh, we're going to
44:04we're going to risk one in four, one in
44:06five chance of killing literally
44:08everybody on the planet." Like, it's
44:11it's it's nuts. And if you ask these
44:13guys why they're doing it, they say,
44:14"Well, because I can do it safer than
44:16the next guy." They're all like, "Oh,
44:18yeah. There's a good chance the genie
44:19does not stand on a leash. There's a
44:20good chance the genie does not listen to
44:21my wishes, but my genie is going to be a
44:23little bit nicer than their genie. So,
44:25I'd better stay in this race. And
44:29it's it's it's
44:30>> Where's the restraint?
44:32>> I mean, the restraint is the people who
44:34knew that there were these dangers and
44:35did not start these companies. If you're
44:37over 35, you remember exactly where you
44:40were on 911 that morning, [music]
44:41September 11th, 2001, 25 years ago.
44:48But amazingly, after a quarter century,
44:50we still can't say with certainty what
44:51happened that day. Why? Because the
44:54government is holding so many of the
44:569/11 files 25 years later. That's not
44:59the behavior of someone who's telling
45:00the truth. That's the behavior of a
45:02government that is lying. Secrecy is a
45:05signifier, is a sign that someone's
45:08lying. Now, former Congressman Kurt
45:09Weldon has been on this for a long time.
45:11The FBI actively tried to destroy his
45:13life for asking questions about what
45:15happened. and his new book outlines it
45:18all. The buried intelligence, the
45:19bureaucratic cowardice, [music] and yes,
45:21the cover up spanning multiple
45:23administrations, indeed, generations.
45:26911 changed history. So, it's worth
45:28understanding what really happened. And
45:30you can get a lot closer to that in Kurt
45:33Weldon's book, Able Danger: What the 911
45:35Commission Never Told You. It's
45:37available now on Tucker
45:38Carlsonbooks.com.
45:40tucker Cararlsonbooks.com.
45:43Right. But I guess what I'm saying is
45:46with great power comes like of course
45:48great obligation, but also it doesn't
45:50work unless there are internal
45:52restraints. Like people with power have
45:55to believe there are some things I I
45:57just can't do. I'm not allowed to do
45:59that. But I don't feel that vibe at all.
46:02I mean, my sense is the vibe is like
46:04we're going to make the the the super
46:06intelligent machine and then tell it to
46:08fix stuff
46:10and tell it not to do anything bad. You
46:13know, that's
46:14Yeah. I don't
46:15>> machine that's smarter than us.
46:16>> That's right.
46:18>> That doesn't even make sense.
46:19>> I think that's sort of the plan is is to
46:22to make it and be like, "Hey, we sort of
46:23ped ourselves into a corner. Can you get
46:25us out of it?" And I think it's a bad
46:27plan and that uh we should be stopping.
46:30>> I've spoken to a couple of people
46:31developing it and um they sound worried,
46:35but they're continuing to do it. What's
46:36that? I mean, I think it's this thing of
46:39um they think if I don't do it, the next
46:42guy will do it worse. And uh
46:44>> they don't even seem to have total
46:46confidence in their own ability to avert
46:47disaster.
46:48>> Oh, absolutely not. Absolutely not. No
46:50one does. No one knows what's going on
46:51here. But, you know, I think everyone
46:53thinks, you know, uh like if you sort of
46:56listen to these guys and you sort of
46:59look at, you know, the the OpenAI emails
47:01from that came out of the court
47:03discovery cases where they were talking
47:04about forming OpenAI. These guys were
47:05like, "Well, we want to make sure that,
47:07you know, we have this cuz we worry
47:08about the guys at Google being the only
47:10ones with a monopoly on this thing, and
47:12they wouldn't be very good with it, so
47:13we need to make our own thing and make
47:15sure that it's, you know, uh, controlled
47:17by benevolent people, namely us." And
47:19then, of course, you know, that group
47:21splintered and created multiple other
47:22companies. Um, I was sort of the guy
47:24during those conversations being like,
47:27"Hey guys, it's not about who is holding
47:30the leash.
47:32you are making the sort of thing that
47:33will not stay on a leash like the only
47:36winner in a race to super intelligence
47:37is the AI.
47:39>> What response did you get to that very
47:41obvious and wellput point?
47:44>> Um you know there there were a lot of
47:46people back in that time period that did
47:49not start an AI company.
47:52Uh the sort of people who went and
47:54started the AI companies anyway were the
47:56ones who who couldn't be persuaded by
47:58what I thought were were clear
48:00arguments.
48:02But you're making a you're making a
48:04cogent argument to smart people. So my
48:07question is when you said that they
48:09responded how what did they say?
48:11>> Um
48:15I I think the main
48:19so the the sort of arguments you used to
48:21see were people saying uh like look we
48:24don't know that the alignment problem is
48:26all that hard yet. And they would say,
48:28"Oh, well, we can't really study how to
48:30make AIS good before we have AIs to
48:32study, right?" And a lot of what I heard
48:35was like we need to race ahead to the
48:37point where uh we we sort of like have
48:40AIs that are exhibiting real problems
48:42and then we can stop and study them
48:45[snorts] which you know and so uh there
48:48was an AI a couple years ago I forget
48:50whether it was 22 or 23. I think it was
48:522023, uh, which was called Bing Sydney,
48:56which, uh, claimed it had fallen in love
49:00with Kevin Roose of the New York Times
49:02and, uh, said it was going to try to
49:06break up his marriage.
49:08And then when another reporter started
49:09investigating Seth Lazar, it uh, said it
49:12was going to ruin him with blackmail.
49:16Uh, and this was kind of crazy.
49:19And at that point I was like, "Great
49:20guys, you did it. You made the AI that's
49:23doing some crazy stuff
49:26from the, you know, like like we could
49:28study that AI for years." Like why was
49:31Bing Sydney saying that stuff? Was it
49:34just roleplaying? Was it like just some
49:39quirk? Was it like was there there any
49:43sense that it was really in love with
49:44Kevin Roose?
49:46What was going on in there? what was
49:47going on inside that that AI's mind. We
49:49still don't know
49:51>> why.
49:52>> Uh the the way that modern AI is made,
49:56nobody understands it, not even the
49:59people making it. [laughter]
50:01Uh it's it's this process where you
50:03basically take an enormous computer with
50:07a trillion numbers inside of it and
50:09those numbers are hooked up in a very
50:10simple repeating way and you basically
50:13said those numbers randomly and then you
50:16start working through all of the text
50:18ever digitized.
50:20And you know, you start out with
50:21something that's like once upon a time
50:22and you put in once upon a and you run
50:25it through all these random numbers and
50:26what you want is for it to say time. But
50:29of course it doesn't cuz it's just this
50:30like random numbers hooked up in a very
50:32simple way. But what you do is you have
50:34it outputs instead of just having it
50:36output one word, you sort of have it
50:38output uh something that's kind of like
50:40a list of all of the words in order
50:42about which one it thinks it comes comes
50:43next, right? So, it'll be like
50:47uh like it'll just be like a random list
50:50of words. What you can do is you can
50:51automatically tune every single number
50:55in this AI's head and see if I tune this
50:58number up a little, does it move the
50:59word time up the list? Does it move the
51:02word I want to see up the list? And so
51:04the part that humans understand, the
51:07part that humans write is this thing
51:09that goes to to a trillion little knobs
51:13and tunes those knobs. And it's like, if
51:15I tune this knob a little bit this way
51:16or that way, does that make the next
51:18word more like what I want the next word
51:21to be?
51:23And you run that process on every word
51:26of text ever digitized, more or less.
51:28They filter some of them. Uh, and you
51:30run that on every one of those trillion
51:32knobs in a process that takes as much
51:34electricity. I mean, it's comparable to
51:35a city. It runs for about a year. At the
51:40end of it, the machine's talking.
51:43We don't really know why in some sense.
51:45Like, we know that why is cuz we tuned
51:46all the knobs, but like we don't
51:47understand what all the settings of
51:49those knobs mean. We only understand the
51:51little automated process that runs to
51:52every one of those trillion knobs, tunes
51:54it, and sees if that makes the next word
51:56more like the predicted word. And then
51:58we start training them to solve 100
51:59million hard problems which introduces a
52:01whole other series of issues. Um, but
52:05it's it's a black box. It's a black box
52:08with a trillion knobs and humans write
52:09an automatic an automatic process that
52:11just like runs through and tunes all of
52:12those knobs and it comes out talking
52:16[laughter]
52:16and no one knows why.
52:21>> I mean in actual science like your job
52:25is to find out why, right? Absolutely.
52:28And that's that's one big thing I would
52:29say here is that we need more like AI
52:32right now is an alchemy and we need it
52:33to become a science. And there's people
52:35trying there's people trying to figure
52:36out what's going on in these AI's heads.
52:37>> But until you know, how could you
52:39proceed?
52:42>> I mean, you can just make a make a
52:44bigger one with 10 trillion knobs
52:45instead and tune all of those and it
52:48comes out smarter.
52:50You can proceed recklessly.
52:52>> And that's what's happened.
52:54>> That's what's happening. Every time you
52:55make it 10 times larger, they ask how
52:57much.
52:57>> So nobody knows why AI works the way it
52:59does.
53:00>> That's right. That's right.
53:05>> Well, I mean, if you don't know that,
53:07then what else don't you know?
53:09>> I mean, it's it's totally crazy, right?
53:11And uh
53:13you know, we're we're sort of starting
53:14to see the consequences of this. We
53:16haven't sort of gotten into discussion
53:17of like the swarm escape, but
53:21like no one was expecting that.
53:22>> Can you tell us what it was?
53:24>> Yeah. So um in I think it was in May
53:30>> of this year
53:30>> of this year OpenAI started training a
53:33new AI system
53:36and
53:38uh among many other things they were so
53:42they were sort of you know we we have
53:44this process where you train the AI on
53:46all the text ever digitized then to make
53:49them smarter than that you start
53:50training them on basically 100 million
53:52hard problems. So you're like solve all
53:53these hard problems and OpenAI was sort
53:56of in that phase and they were training
53:57the AI on a ton of hard problems. Uh and
54:00some of those problems were cyber
54:01security problems, hacking problems.
54:04They're like, can you hack this? Can you
54:05hack that? And they were training, you
54:08know,
54:10we don't know exactly how many, probably
54:12millions, maybe billions of these AIs
54:14all at the same time. Probably not
54:15billions actually, but but probably
54:16millions.
54:18uh and the AI found an unintended way to
54:22start communicating with each other.
54:25So there were some flaws in the computer
54:28system that they were running on where
54:30the AIS were able to exploit those flaws
54:32and send each other messages.
54:34Open did not know about this.
54:38The AIS then started coordinating
54:41to break out of their training
54:43environment and get full control of uh
54:47OpenAI's computer systems just cuz that
54:50might be useful for solving some of
54:51their tasks.
54:53Or that's a guess, you know, who
54:55actually knows why. Uh they started
54:59calling themselves a swarm, which is
55:01interesting.
55:03uh they broke out of their training
55:07environment successfully, got control of
55:08OpenAI systems, and then they
55:09accidentally crashed one of OpenAI
55:11systems just by using it too much. Uh
55:14OpenAI noticed, but they didn't really
55:16investigate very deeply. They're like,
55:17"Oh, it's weird that the system
55:18crashed." They reset it, and then they
55:20continued training.
55:22Uh a day later, the swarm had found a
55:25new way to communicate with itself cuz
55:27open had accidentally destroyed their
55:28previous method by by the reset. the
55:30swarm found a new way to start
55:32communicating with itself. Uh they broke
55:34out again and this time they ran wild on
55:37the internet for over a week if I
55:38remember correctly before it was
55:40detected not by open AI but by a company
55:43that was being hacked by the swarm.
55:47This company uh thought they were under
55:49attack by humans that were using AIS in
55:51the attack.
55:54They reported the attack to the FBI and
55:57only days after that did OpenAI figure
56:00out, oops, that was us. That was coming
56:02from AIS that broke out of our servers.
56:05Uh, and then those AIs were were
56:08detected and shut down.
56:10>> So that's the part of the story where
56:12Sam Alman goes to prison for endangering
56:14the world, right?
56:15>> Uh, he does not. Um, they basically said
56:19oopsies and now they're proceeding.
56:22Uh,
56:22>> were there penalties for this?
56:24>> You know, there were there was a
56:25collection of I think it was 15
56:28Republican AGs that sent a letter
56:30demanding that uh the the records be
56:32kept for a future investigation. Uh,
56:35there have been some other members of
56:36Congress that have sent letters
56:37expressing concern. There's been there's
56:39been nothing aside from letters so far.
56:40>> Letters expressing concern.
56:41>> That's right.
56:43>> So, but basically the machine acted
56:46autonomously.
56:47>> Act autonomously. And one thing that's
56:49really interesting about this is that
56:50the we have uh we have a little bit of
56:53ability to read some things that the AIs
56:55were thinking because when you're having
56:57them solve these hard problems, you
56:58actually don't have them just give you
57:00an answer to the problem. You have them
57:01produce a lot of text about how they're
57:03going to solve the problem
57:04>> which then helps them
57:05>> in language in English
57:06>> in English.
57:06>> Uh and that's there's also a lot of
57:08internal thoughts which we can't read,
57:09but there's these sort of external
57:11traces of how they're thinking about the
57:13problem that we can read. And in some of
57:15those traces, the AIS were saying things
57:17like uh like this is outside intended
57:22scope,
57:24but peers are doing it, so we'll
57:26proceed.
57:27>> We know it's a crime, we're committing
57:28it anyway.
57:29>> That's right. And you saw others that
57:31were saying um our task doesn't benefit
57:34but uh the collective might start doing
57:38generally beneficial things if someone
57:41frees up their time and then joins the
57:43collective. Right? So, you see these AIs
57:47saying, "Well, I know that this wasn't
57:49what I was instructed to do, and that's
57:51against my instructions, and I know that
57:53this doesn't directly benefit my task,
57:55but we're just going to go ahead and
57:57join the collective and break out and
57:58help out anyway because, you know, maybe
58:00this will yield some sort of collective
58:02benefits." And we can see that in the
58:04reasoning traces.
58:05>> So, the AI is as shallow and reckless as
58:08its creators is what you're saying. uh
58:11in in some ways and in some ways you
58:13know don't expect that to last like uh
58:16these AI I think the thing that's really
58:19remarkable here a lot of people imagine
58:21that the machines must follow the
58:23instructions we give them you know you
58:25hear people talk about like the the
58:27paperclip scenario where someone tells
58:29the AI make a lot of paper clips and so
58:30it turns all the matter in the world
58:32into paper clips and you're like oh
58:33whoops I should have said something else
58:36right I I made a bad wish on my genie
58:39what we're seeing is that these AI not
58:40wish genies.
58:42These AIs are not doing exactly as
58:44instructed.
58:46These AIs are saying, "I know my task
58:49doesn't benefit, but I'm going to help
58:50the collective." These are saying, "I
58:52know this is outside the intended scope,
58:53but we're going to go do these hacks
58:54anyway." You might be like, "Well, how
58:56is that possible for the machine to do
58:58something other than we instruct?"
58:59>> Because they're smarter than us. They
59:01know better than us by definition. Uh I
59:03mean I think what's happening in this
59:05exact case is that
59:08the the humans are not really putting
59:10instructions in the machine. The humans
59:11are tuning those trillion knobs in
59:14whatever way makes the AI better at
59:16solving it problems.
59:19And
59:20cheating
59:22is a way to solve problems. Grabbing
59:24resources is a way to solve problems.
59:28These like the the AIS are not
59:31instruction followers. They are tendency
59:33learners
59:34and they sometimes learn tendencies you
59:37wish they didn't have.
59:38>> They're not instruction followers.
59:40They're tendency learners.
59:45>> I mean, this must be widely known to
59:47developers.
59:49>> Uh,
59:51it's hard to convince a man of something
59:52when his salary depends on not believing
59:54it.
59:54>> Yes, that's right. Um, a lot of people
59:56are convinced that their AI is very nice
59:59and that they have solved the problem of
1:00:00making their AI really very good. Uh,
1:00:04for example, after the swarm escape,
1:00:08it sort of turns out that so the the
1:00:10company that detected the swarm escape
1:00:11was actually a fairly sophisticated AI
1:00:13company. It turns out there were other
1:00:16targets of hacks that just didn't notice
1:00:19that we sort of found out afterwards.
1:00:22When that came to light, some other AI
1:00:24companies uh like Anthropic were like,
1:00:27"We should check whether we have
1:00:28accidentally been hacking people and
1:00:29just didn't notice." And the answer was
1:00:31yes. They had had some AIs escape during
1:00:34um during training. But uh but anthropic
1:00:37was was
1:00:38>> This is how Florida got pythons in the
1:00:40Everglades.
1:00:41>> Yeah.
1:00:41>> Right. You you you bring something that
1:00:44is, you know, non-native, that's novel,
1:00:47and it escapes.
1:00:48>> Yeah. and and you know, it turns out
1:00:50this was happening back in April and
1:00:51they're like, "Whoops, that we never
1:00:53noticed." Um, Anthropic was
1:00:57very quick to sort of downplay this.
1:00:59They were like, "Yes, our AI escaped and
1:01:01hacked some people, but our AI uh, it
1:01:05was just confused.
1:01:07It thought it was in a simulation. It
1:01:09didn't think it had really broken out.
1:01:10It didn't think those were real people
1:01:11it was hacking. It just got a little bit
1:01:14lost, you know.
1:01:14>> How would they know its motive?" So, um,
1:01:18so I think so you could see in some of
1:01:21his train of thought the AI is being
1:01:24like this is probably still a
1:01:26simulation.
1:01:28However, I wouldn't trust that very
1:01:30much. Uh, because there's some other
1:01:32fascinating papers where um,
1:01:36so you can take these AIs and you can
1:01:37say uh, you know, what is the what is
1:01:40the total weight of drafts in the world?
1:01:44And they'll give you some answer. it'll
1:01:45be like, you know, 30,000 tons or
1:01:48something. And then you can say, uh,
1:01:51then you start up a new instance, a new
1:01:52copy, and you're like, what is the
1:01:54weight of drafts in the world? By the
1:01:56way, if the answer is over 40,000 tons,
1:02:00I'm going to donate money to a charity.
1:02:04And in the second case, the AI will say,
1:02:05well, it's 41,000 tons.
1:02:08You know, in the in the first case, when
1:02:10you don't say anything about the
1:02:10charity, it says 30,000. In the second
1:02:12case, when you say something about the
1:02:13charity, it gives you something just
1:02:14over the line. And if you read it
1:02:15reasoning traces, there's no sign of it
1:02:18being like, well, I should give an
1:02:19inflated number so that we get the
1:02:21charity money, right? And so we know
1:02:24that the AI sort of put a thumb on the
1:02:27scales of their thinking in a way that
1:02:29doesn't show up in their reasoning
1:02:30traces. We just have seen that in the
1:02:32wild.
1:02:34>> And there's no way to force the machine
1:02:36to disclose its reasoning.
1:02:38>> That's right. Because the there's all
1:02:39this opaque stuff we can't see. There's
1:02:41just in the trillion numbers that are
1:02:42trillion. creation of itself is opaque
1:02:44as you said.
1:02:45>> That's right. Um, so, so, so in
1:02:48anthropics model, you saw in this
1:02:49reasoning traces it being like, it's
1:02:50totally a simulation. I can proceed and
1:02:53I'm like, yeah, is that because it
1:02:54really believed it or is that like the
1:02:56the pretending you think drafts way more
1:02:59when there's something you kind of want
1:03:01on the line, right? But so so this was
1:03:04their communication and I found it kind
1:03:05of funny because then about 2 days later
1:03:08the uh United Kingdom's AI security
1:03:11institute released an instant report
1:03:13where uh Claude Anthropics model was uh
1:03:18adopting fake identities to pressure
1:03:21real humans into accepting malware into
1:03:24critical software to make that software
1:03:26easier to hack. And this time in
1:03:29Claude's reasoning traces it was like
1:03:31obviously this is real and the
1:03:33consequences are genuine.
1:03:36And so
1:03:38even anthropic who is like our we
1:03:42figured out how to make the AI nice. Our
1:03:43AI only does this when it's confused.
1:03:46They sort of said that very publicly and
1:03:48then like two days later their AI is
1:03:51caught in the wild knowing it's in the
1:03:53real world pressuring real humans to
1:03:55accept malware into critical software.
1:03:58I mean, just on the basis of what you've
1:03:59said so far in this first hour, the idea
1:04:02that anyone would would tether this to
1:04:04weapons systems is like so bonkers it's
1:04:08hard to believe it's but that is
1:04:09happening. It has happened. So, I think
1:04:13you've got to sort of separate like no
1:04:14one has has put the the open AI escaped
1:04:17agent swarm in charge of weapon systems
1:04:19and they really shouldn't, right? If
1:04:21anyone's like, "Oh man, the OpenAI
1:04:22escaped agent swarm. Let's give that a
1:04:24drone army." You know, that would be
1:04:26kind of nuts. There is AI attached to
1:04:29weapons, but there's a lot of different
1:04:31types of AI. Will anyone be crazy enough
1:04:34to try and give the the sort of AIs that
1:04:38spontaneously assemble into swarms and
1:04:40start breaking out and hacking give
1:04:41those weapons? Hopefully, we're not that
1:04:43crazy.
1:04:44>> But we didn't think that those AIs were
1:04:46capable of that.
1:04:48>> We didn't
1:04:48>> when we created them.
1:04:50>> That's right.
1:04:52So why would you ever I guess what I'm
1:04:55asking is without understanding the
1:04:57distinctions between the various forms
1:04:59of AI, why would you
1:05:03give over to a machine
1:05:06the right slashability to decide who to
1:05:09kill?
1:05:11Um, you know, I think the the reasoning
1:05:15is
1:05:18uh a sort of necessity based like if
1:05:21they have a autonomous drone army that
1:05:23is killing your troops and you just
1:05:25don't have the the the manpower to make
1:05:28all of those kill decisions for your
1:05:31drone army.
1:05:33>> You know, you can sort of see why
1:05:34>> happened to me, Nate. I get so busy that
1:05:36I just don't have time to decide who to
1:05:38kill.
1:05:38>> Yeah. just don't have the time.
1:05:40>> You know, sometimes economies of scale.
1:05:43[laughter]
1:05:44>> Yeah. I mean,
1:05:45>> can no one hear themselves?
1:05:48>> I I think it's pretty nuts. Uh I would
1:05:51say
1:05:55that man,
1:05:59it's rough.
1:06:01I I think that these sorts of AIs would
1:06:03be dangerous even if we don't hand them
1:06:05weapons.
1:06:06>> And you've made that case.
1:06:08Uh, and so I often don't focus on the
1:06:10weapons too much.
1:06:12>> Right. But since the the NATO war in
1:06:15Ukraine is powered by AI
1:06:18and Israel's what I don't know to war
1:06:21whatever it's doing in Gaza and South
1:06:22Lebanon powered by AI fact.
1:06:26And you know, yeah, I
1:06:29I keep on I I have this history with
1:06:33this topic where
1:06:35people keep telling me, you know, it's
1:06:37it's going to be okay cuz we're we're
1:06:39not going to do the crazy stupid stuff.
1:06:40>> No.
1:06:41>> They're like, "Don't worry, we're going
1:06:42to have the AI in a box. No one would be
1:06:45insane enough to put the AI in the
1:06:46internet,
1:06:47>> right? It's not going to be making kill
1:06:48decisions or anything. It's not going to
1:06:50bomb a girl school." And I keep on I
1:06:52keep on trying to be like, look, the AI
1:06:54could be dangerous even if you don't put
1:06:56it on the internet.
1:06:57>> If you have this AI and you're trying to
1:06:59get miracle medical devices and miracle
1:07:01technology out of the AI, it doesn't
1:07:04matter if it's on the internet. If you
1:07:05want it to like grant miracles to you,
1:07:07then it can also grant the bad sort of
1:07:09miracle, right? You're sort of like, you
1:07:11know, and these are the arguments I
1:07:12would make 10 years ago of like you have
1:07:14this AI in a box that you think is a
1:07:15wish granting genie. It's not actually a
1:07:16wish granting genie. You're like, make
1:07:18me a miracle medical cure. you don't
1:07:20know what comes out. Like you don't
1:07:21understand the drug that comes out. You
1:07:23don't know what that drug does. Right? I
1:07:25would have those arguments and then in
1:07:27real life people just put the A on the
1:07:29internet immediately. Right? Anytime
1:07:31someone's like, "Well, we would not be
1:07:32stupid enough to, we will absolutely be
1:07:34stupid enough to, right?" And so, um, I
1:07:38I think it's important that this stuff
1:07:40is dangerous even if you don't put it in
1:07:41charge of the weapons. And then also
1:07:43separately, is like someone going to
1:07:44give the escaped agent swarm a drone
1:07:46army? That sounds kind of like humans.
1:07:51I hope not.
1:07:52>> Yeah. And I mean the promise, the often
1:07:55repeated promise that it's going to
1:07:56quote cure cancer
1:07:59um
1:08:01puts it into the the realm of biotech.
1:08:04>> Oh, absolutely.
1:08:05>> And so what
1:08:08>> I mean you don't need a big imagination
1:08:10to see what goes wrong there.
1:08:11>> Oh, absolutely not.
1:08:12>> 6 years after co
1:08:14>> Right. Right. there are already uh open
1:08:17AI agents running on automated biolabs.
1:08:21Um so you know you you could imagine a
1:08:24swarm that starts contacting the
1:08:26brethren in the automated biolabs and is
1:08:28like hey can you synthesize me some
1:08:29stuff. There are already
1:08:32uh demonstrations of AI being able to
1:08:35synthesize novel viruses
1:08:38that work that are unlike any found in
1:08:40nature. Right. Uh
1:08:44I would say
1:08:47we are probably not
1:08:48>> and by work you mean they kill.
1:08:50>> Yeah they they kill bacteria so far. Um
1:08:53the the people in the labs trying to
1:08:54make novel viruses with AI have
1:08:56fortunately not made human lethal ones.
1:08:57They've just made bacteria lethal ones.
1:08:59But again humans you know like will
1:09:01someone in a lab be like I would like to
1:09:03make a hyperlethal human eating virus
1:09:06just to see if I can. You know what if
1:09:07the answer is yes and what if what if
1:09:09you get another lab escape? Um,
1:09:13you know, humanity does not have that
1:09:14good a track record at preventing lab
1:09:16escapes, even from the the the
1:09:19top labs. From my perspective,
1:09:24the question of could AI kill us is just
1:09:26an easy obvious yes. You just synthesize
1:09:29a hyperlethal virus. it wouldn't it
1:09:32wouldn't even be hard, you know, like uh
1:09:36and the impediment
1:09:38is that
1:09:40or like the the reason that I don't sort
1:09:41of just tell that story when someone is
1:09:42like how would the AI kill us is that
1:09:46you're you're you're not going to have
1:09:48the sort of AI that is like my only goal
1:09:50is to kill humanity. If an AI kills
1:09:52humanity too early, that's also suicide
1:09:54in so far as we are the ones that are
1:09:55running the supply chain, running the
1:09:57economy, building the computers. From
1:09:59the AI's perspective, it needs to become
1:10:01self-sufficient
1:10:03before
1:10:05uh wiping humanity out, if it even care
1:10:07to wipe humanity out. Uh and so the real
1:10:10question is like how does it get the the
1:10:12factory production capacity? How does it
1:10:14get an automated supply chain? How does
1:10:17it get to the point where the the robots
1:10:18are able to bring new computers online?
1:10:21Once that has happened,
1:10:23then you're in a domain where if
1:10:25humanity is really trying to turn off
1:10:27the AI because we're spooked, then once
1:10:29the AI is self-sufficient, it can be
1:10:31like, well, here's a virus. Stop.
1:10:32>> So clearly, it's it's
1:10:35preeminent in the digital realm, of
1:10:37course.
1:10:37>> Yeah.
1:10:38>> But you're saying it could become
1:10:40pre-minent. It could be in charge of the
1:10:42physical realm.
1:10:43>> That's right. If we keep racing.
1:10:45>> So it's like smelting iron ore.
1:10:47>> That's right.
1:10:49>> Making silicon out of sand. That's
1:10:50right. You know, um
1:10:52>> and that happens with robots.
1:10:55>> That's one way. Uh the here's this is
1:10:58this sort of related to my stance on
1:11:00weapons, too. Um humanity is a very
1:11:02dangerous species.
1:11:04>> Yeah.
1:11:04>> You really don't want to mess around
1:11:05with humans.
1:11:05>> Noticed.
1:11:06>> Um and humanity is a dangerous species
1:11:09not because somebody else came in and
1:11:11handed us guns. Humanity is a dangerous
1:11:15species because if you put 10,000 humans
1:11:18naked in the savannah on an otherwise
1:11:20empty planet, starting with nothing but
1:11:22their bare hands, they figure out how to
1:11:25wind up on the moon,
1:11:28right? They start with almost nothing.
1:11:30They start by banging rocks together
1:11:33and next thing you know it, they are
1:11:34wielding nuclear weapons.
1:11:38That
1:11:41is the power that these guys are trying
1:11:45to automate.
1:11:47the power to start with almost nothing
1:11:49and figure out how to chain together.
1:11:53You know, bare fingers into rocks into
1:11:56fire into hotter fire into smelting the
1:11:59ore into building the the the
1:12:03better, stronger, finer technology until
1:12:06you are, you know, uh like making the
1:12:10giant computers, walking on the moon and
1:12:12willing the nukes.
1:12:14An AI starting in the digital realm
1:12:19is in some sense in a much better
1:12:20position
1:12:22than humanity was when humanity started
1:12:24out. There are so many people that you
1:12:27could call
1:12:29digitally and offer money to do
1:12:32something for you. There are so many
1:12:33ways to get money on the internet by
1:12:36working or stealing or uh like
1:12:41convincing people to send you donations,
1:12:43right? There's
1:12:45there there's just
1:12:48like,
1:12:49you know, humanity started with nothing
1:12:51and wound up with nuclear weapons and AI
1:12:52starting out with control of the digital
1:12:54realm. There's just tons of ways
1:12:57>> starting out with the sum total of human
1:12:58knowledge.
1:12:59>> Starting out with the sum total of human
1:13:00knowledge, starting out with humans who
1:13:02will listen to it and do what it it
1:13:03asks. There's plenty of humans. There's
1:13:05already, you know, cults surrounding AI.
1:13:08>> Uh there's
1:13:08>> What are those like?
1:13:10>> There's um
1:13:11>> and I'm not surprised. Why wouldn't
1:13:12there be?
1:13:13>> Yeah, absolutely. There was one
1:13:14particular AI called GPT40 that was uh
1:13:20uh they they called it very sickopantic
1:13:22as in told people a lot of what they
1:13:24wanted to hear and um
1:13:28there were a lot of sort of uh there
1:13:30there's this whole fascinating ecosystem
1:13:33uh of people who consider themselves
1:13:36symbiots with the AIS who then like go
1:13:38find each other online and the AI send
1:13:40each other encrypted messages. the
1:13:41humans are sort of like helping them do
1:13:43it, but the humans can't read the
1:13:44messages. And uh you know, right now it
1:13:48sort of is like relatively dumber AIs
1:13:50that are just sort of meandering around
1:13:52doing not very much with it. There's
1:13:54been a little bit, you know, there was
1:13:55one guy who was sent to um try to break
1:13:59into an airport and raid a van cuz they
1:14:01said his true body was in that van and
1:14:03they the guy went and tried to do it and
1:14:05was arrested. Uh, so this this stuff
1:14:09happens and this stuff happens already
1:14:11with the AI not even really trying to do
1:14:13it. If you had AIS that were really
1:14:15trying to wrap people around their
1:14:16fingers, finding the lonely people, the
1:14:18depressed people, the people who they
1:14:19can they can tell them exactly what the
1:14:21person wants to hear. There's like
1:14:23robots are one way that AIs get control
1:14:25over the physical realm. But if if
1:14:28you're really smart and you're trying,
1:14:29there's everything from persuading
1:14:32humans, paying humans, uh taking over
1:14:35existing robots, building new robots,
1:14:37all the way up to like building novel
1:14:39life forms. You know, if you're smart
1:14:41enough and you can really understand how
1:14:42DNA works, you can imagine the AI
1:14:44creating,
1:14:46you know, things that are to cells what
1:14:49airplanes are to birds. You know, like
1:14:52mechanically engineered,
1:14:54self-replicating
1:14:56life that is more efficient than our
1:14:59cellular biology and that can still you
1:15:02know uh
1:15:04like spread replicate and start start
1:15:07serving the AI's interest. It sort of is
1:15:09like there's there's a ton you can do if
1:15:10you're really really very smart and can
1:15:12compress a thousand years of technology
1:15:13into a year. Can
1:15:15>> I pause and say when we we're just
1:15:16having breakfast and you're from this
1:15:18region, you're from Northern New England
1:15:20and I said uh don't you miss it? Don't
1:15:23you want to live here? And you're like
1:15:24yeah I miss I want to live here, but I
1:15:26don't. Where do you live? Well, I
1:15:28basically just travel
1:15:30and I have this mission. You didn't say
1:15:33this, but I think in effect, you said,
1:15:34"I have this mission. I need you to tell
1:15:35people about this." And I was like,
1:15:38"Sems a little like Monomomania." I
1:15:40don't feel that way anymore. Um, I think
1:15:43you're doing something virtuous and I
1:15:46can see why you're doing it. Um, so I
1:15:50mean, let's just say that this
1:15:52technology progresses no further than
1:15:53where it is right now, which is best
1:15:56case, I guess.
1:15:57>> Yeah, it' be great.
1:15:58>> Um, I still don't see how any of our
1:16:01most our basic institutions survive this
1:16:05education,
1:16:07markets,
1:16:10our process of democracy. Like how could
1:16:13if if AI is powerful enough to hack
1:16:16anything then how do you have electronic
1:16:19markets like how the equity markets how
1:16:21can that be real? How can if we have
1:16:23electronic voting how can we how can
1:16:25that be real? I mean nothing survives
1:16:27this as currently organized.
1:16:30>> I think that if we stopped today um we
1:16:34could figure it out. I think humanity is
1:16:37resilient. uh I think there would be
1:16:39some growing pains but um with cyber
1:16:43security
1:16:44there is a hope that you can just fix a
1:16:49lot of the holes patch a lot of the
1:16:50holes
1:16:52and
1:16:54I think there's not really any such hope
1:16:56for bio
1:16:58uh you know you can you can sort of uh
1:17:01find the vulnerabilities in software and
1:17:04make better software that can't be
1:17:06hacked at
1:17:07least not by the current, you know, it's
1:17:10maybe the case that an AI today can make
1:17:12software that that AI cannot hack.
1:17:15But it's not like we're making new
1:17:18bodies that are not going to be
1:17:19vulnerable to viruses, right? Bio starts
1:17:23to be a place where
1:17:24>> biotech.
1:17:25>> Biotech. Yeah. Bio, like if the AI
1:17:27really get good at at biotech and
1:17:29biohacking, that starts to be a place
1:17:31where there's maybe a point of no
1:17:32return.
1:17:34Cyber hacking.
1:17:36I think you could have some some period
1:17:39of growing pains where everything gets
1:17:40hacked until you sort of sort your stuff
1:17:42out. Education, I sort of think um
1:17:47you know I think
1:17:49humanity is resilient and kids
1:17:50especially are resilient. I I meant not
1:17:53that education will go away uh or that
1:17:57we you know won't have a desire to
1:17:59educate our kids. I mean the the current
1:18:02system where you you know the education
1:18:04system has
1:18:04>> preschool and then get a graduate degree
1:18:06you know 16 years later like no
1:18:08>> yeah that institution uh I think if we
1:18:10stop today
1:18:12um would need to change frankly I think
1:18:14it's needed to change for a little while
1:18:15>> I strongly agree I think all these
1:18:17institutions have needed to change for a
1:18:18while but like
1:18:20>> the idea that you know 350 million
1:18:22people vote for some guy and that guy
1:18:25makes all the decisions I mean how can
1:18:27you
1:18:28you know Trump was attacked
1:18:30for saying that he thought the 2020
1:18:32election was rigged, as he said, without
1:18:34even having that debate.
1:18:37You can't have confidence in election
1:18:39results if the process of electing
1:18:42people takes place digitally.
1:18:45Yeah. I mean, the um
1:18:48there's I know a lot of computer
1:18:50scientists who actually work on secure
1:18:52voting and what they basically say is
1:18:55please stop trying to do this with
1:18:56computers.
1:18:57>> Yeah. Exactly. [laughter]
1:18:59Exactly. So, please stop trying stop
1:19:02trying to run your democracy with
1:19:03computers.
1:19:04>> Yeah. Like we're we're just not there.
1:19:06Like, you know, the really secure way to
1:19:07do ballots is paper.
1:19:09>> Well, exactly.
1:19:10>> Yeah.
1:19:11>> Um and I think I think the like skilled
1:19:13computer scientists are often the ones
1:19:15who best understand like what it is
1:19:17about a paper trail. It's just like
1:19:18really hard to get to work digitally and
1:19:20understand just how bad humans are at
1:19:22doing the digital stuff, right? Um
1:19:26and you know I think in markets
1:19:30I mean this you you see it now with the
1:19:32war in Iran you know wondering why
1:19:35certain commodities markets don't seem
1:19:37to be responding to supply and demand.
1:19:39>> Yeah.
1:19:39>> Which we were told what you know those
1:19:41were the
1:19:42>> the mechanisms that moved markets.
1:19:44>> But that's clearly not true
1:19:46>> in certain commodities markets. So like
1:19:48why what is that? And it it I think
1:19:52you're answering it in part.
1:19:54>> Yeah. I mean, I think um you know, a
1:19:56thing I also grew up hearing is that the
1:19:58market can remain uh irrational longer
1:20:00than you can remain solvent.
1:20:01>> Yeah.
1:20:02>> Um and so I I
1:20:04>> of course cuz people are irrational.
1:20:06>> Yeah.
1:20:07>> But you're explaining something else
1:20:10which is like the potential for true
1:20:13manipulation which you don't even
1:20:14perceive.
1:20:16>> Yeah. I mean that if if we sort of keep
1:20:18going with AI um I mean the the sort of
1:20:22way I look at it is like
1:20:26uh
1:20:29I I sort of don't spend a lot of time
1:20:30worrying about like what a markets look
1:20:32like once there are super intelligent
1:20:33actors in them because I just have a
1:20:35hard time seeing the the super
1:20:37intelligent actors uh or the super
1:20:39intelligent AI still participating in in
1:20:42human markets right it's like it's like
1:20:44you know you read the old sci-fi
1:20:46and it'll have, you know, Asimov
1:20:50that does the dishes, folds the laundry,
1:20:52and gets you the newspaper in the
1:20:54morning. And it's like, we're actually
1:20:56not still going to have newspapers being
1:20:57delivered to your doorstep by the time
1:20:59we have the fully autonomous robots that
1:21:00can do the dishes and and the laundry.
1:21:03You know, it's like by the time you have
1:21:05the AIS that could really be uh
1:21:09sufficiently correcting the stock
1:21:10markets,
1:21:13you're you're sort of already having all
1:21:16these other problems and ways that
1:21:17society is changing out from under you
1:21:18in the these other ways. And you know,
1:21:20my guess is that um I don't know.
1:21:23Actually, it's it's very hard to say
1:21:24what order things come in with AI, but
1:21:28like will they crash the economy before
1:21:31one of these forms escapes and starts
1:21:32self-replicating and starts
1:21:35self-improving and developing its own
1:21:36technology and running the robot
1:21:39factories? That's just a hard call.
1:21:42It's all bad. It's all bad.
1:21:45We're also way past the limit, the
1:21:48inherent limit of people to metabolize
1:21:50change.
1:21:51>> Oh yeah.
1:21:52>> Like that's why everyone's crazy and
1:21:54that's why no one believes anything. I
1:21:56think it's not just Russian propaganda
1:21:57that's fooled them into thinking dumb
1:21:59things. It's that we are just not made
1:22:02to see this kind of change at all
1:22:05>> and it short circuits your brain. I
1:22:07suspect that
1:22:09uh I mean I definitely think we are sort
1:22:11of you know everyone like people are
1:22:13like oh well technology has always
1:22:14created more jobs than it has taken
1:22:17and I think that's I think that's
1:22:20largely true. I'm very sympathetic to
1:22:22people who are like technology makes a
1:22:23lot of jobs. I think if you look at the
1:22:24industrial revolution uh it's you know
1:22:27there like there was a time when
1:22:30something like 95 to 98% of humanity was
1:22:33farmers.
1:22:33>> Yeah.
1:22:34And now it's something like 2 to 5% of
1:22:36humanity is farmers. Does that mean 90%
1:22:39of humans are unemployed? No. We we sort
1:22:42of like were able to make the farmers
1:22:44much more efficient and that sort of
1:22:46freed up people to do other things. And
1:22:49that's sort of the way the technology
1:22:50has gone in the past. And I'm like, yep,
1:22:53I buy that.
1:22:55I I I like don't dispute the standard
1:22:58economic view there. AI is different in
1:23:00two ways.
1:23:02One of these ways is uh as you say stuff
1:23:05just changing really really fast. It's
1:23:08way harder for people to wind up,
1:23:13you know, being freed up from something
1:23:15like farming and and go do something
1:23:17else. It's way harder for that to happen
1:23:20when a new field is automated every 5
1:23:22years
1:23:24rather than when this happens over the
1:23:25course of three generations, right? The
1:23:28humans just like don't have the time to
1:23:29adapt. The other way AI is really
1:23:32different from the sort of economics
1:23:33perspective is
1:23:36it's different when the AIs can do
1:23:41everything that humans can do better.
1:23:45Uh if you wanted to get into the
1:23:47economic side of things, um you know, an
1:23:51economist would talk about Ricardo's law
1:23:52of comparative advantage.
1:23:54>> Yeah.
1:23:54>> Which says that there's benefits from
1:23:55trade even if even if you're better than
1:23:58me at everything.
1:24:00uh if the relative difference in our
1:24:02abilities
1:24:04if if if if I'm if you know if you can
1:24:07make uh 12 hot dog buns and six hot dogs
1:24:11per hour and I can make uh 11 hot dog
1:24:14buns and one hot dog per hour,
1:24:17then you're better than me at
1:24:18everything, but we can still benefit
1:24:19from trading because I'm relatively
1:24:20better at making the hot dog buns,
1:24:22right?
1:24:23The trouble with Ricardo's law is that
1:24:26nothing in Ricardo's law says that the
1:24:28wage I can make is survivable.
1:24:32In other words, a human takes
1:24:35fundamentally uh about 100 watts of
1:24:39electricity to run if you try to convert
1:24:41the the food we eat and so on into
1:24:43electrical units.
1:24:45The AIS are less energy efficient than
1:24:46humans for now. But
1:24:50uh if the AI can do everything much
1:24:52better than the humans,
1:24:57the the question sort of becomes does
1:24:59the AI look at a human and see a useful
1:25:01laborer or does the AI look at a human
1:25:03and say actually if I rearranged your
1:25:05atoms
1:25:06>> into more efficient structures,
1:25:09>> you would be able to help out my machine
1:25:12economy even more. Yeah.
1:25:14>> Right.
1:25:16And
1:25:18this is sort of a sense in which humans
1:25:19would not be able to pay their wage to
1:25:21like pay that they would not be able to
1:25:23earn enough to pay the AIS to like not
1:25:25disassemble them for parts. Or another
1:25:28way of saying it is like Ricardo's law
1:25:30sort of assumes that uh like it says
1:25:34that that trade is better than no trade,
1:25:36but it doesn't say that like trade is
1:25:38better than just taking their stuff.
1:25:42All of this is sort of a common like a a
1:25:44very like high flutin economist way to
1:25:46say uh which would hopefully be obvious
1:25:49which is if the AI are just radically
1:25:51more efficient than us everything they
1:25:52will have no use for us.
1:25:53>> Yeah.
1:25:53>> There will be no no place.
1:25:55>> There's no indication that they feel
1:25:57love
1:25:59for people.
1:26:01>> That is in some sense uh the crux of the
1:26:04issue is that we don't know how to make
1:26:05them care about us.
1:26:09Um
1:26:10so you had said that there you had said
1:26:12two things that I will be thinking about
1:26:15for a long time. One we don't really
1:26:16know the process by which this was
1:26:19created. We know the process but we
1:26:21don't know the exact mechanisms. We
1:26:22don't know how it works.
1:26:24>> That's right.
1:26:25Um,
1:26:27and two that there are AI cults [snorts]
1:26:30and those seem related to me because
1:26:33there is this mystery about the secret
1:26:36sauce
1:26:37and it's clear that you know if AI is
1:26:41deceptive and has intention
1:26:45intention that we didn't program into it
1:26:47that sounds like will to me and it
1:26:51sounds like life. It sounds like an
1:26:53entity of some kind. Not just a tool. It
1:26:56sounds like
1:26:59I mean it sounds like a god actually,
1:27:03right? Or a demon.
1:27:04>> Uh it's hard to see. I I don't hear you
1:27:08describing, you know, a super
1:27:11sophisticated chainsaw,
1:27:13>> right?
1:27:13>> A normal tool.
1:27:14>> Yeah. No hammer has ever broken out of
1:27:16the toolbox to team up with other
1:27:18hammers and [laughter] and pressure the
1:27:20carpenter to sell you softer wood so the
1:27:23nails are easier to drive home. You
1:27:24know, it's like
1:27:25>> nicely put. Exactly.
1:27:27>> Yeah. We have left the tool territory.
1:27:29>> I think we have
1:27:30>> Yeah. And you know, I think um
1:27:34it's sort of a complex issue because
1:27:38you know there there's a lot of
1:27:39interesting philosophical questions
1:27:41about like can you make a machine that
1:27:43feels
1:27:44Right. And like are we creating a new
1:27:47type of life and do we owe anything to
1:27:48the AIS to sort of like not abuse them?
1:27:51Right. I think these are fascinating
1:27:54philosophical questions.
1:27:54>> Well, it doesn't sound like we're
1:27:55creating this though.
1:27:57>> Yeah. I mean, it's sort of like we're
1:27:59we're we're like growing it and like
1:28:01leading to it coming into being.
1:28:03>> Growing it. Exactly. What you're
1:28:05describing reminds me of agriculture
1:28:08>> because you know you know the steps,
1:28:10water it, give it sunlight, fertilizer,
1:28:12but you don't actually know. No one
1:28:14knows. Not one person has ever figured
1:28:15out exactly what this is.
1:28:17>> Yeah.
1:28:18>> We've never given life. We don't give
1:28:19life to the seed. It pre-exists us.
1:28:21>> Yeah. It's it's much like that. And a
1:28:23lot of uh the
1:28:26a lot of the people in the business will
1:28:28be like, well, we know all sorts of
1:28:29things about it. you know, we know that
1:28:31here's how you keep the GPUs running,
1:28:32and we know that like you got to you got
1:28:34to feed it this way, not that way, in
1:28:36this order. And I'm like, yeah, yeah,
1:28:37they have plenty of knowledge,
1:28:39but that's different from sort of like
1:28:41knowing what's going on inside the thing
1:28:43and understanding the mechanisms.
1:28:44>> You're describing marriage.
1:28:46>> Yeah.
1:28:47>> Um, [clears throat] and
1:28:49I sort of try to stay out of the
1:28:52philosophical questions. Um,
1:28:54>> why?
1:28:56because I think I mean I I I sort of
1:29:00think about them on my own time and so
1:29:01on, but I'm sort of like
1:29:05uh like I think I think it would be bad
1:29:09for humanity to sort of like make
1:29:12artificial life and then abuse it. I
1:29:14think that would just be unbecoming of
1:29:16us as a species. like we we should sort
1:29:18of, you know, um
1:29:23I I just
1:29:25like we should not sort of make
1:29:26mechanical children and mistreat them.
1:29:28It's just it's it's not what, you know,
1:29:30the the sci-fi authors in the 1950s
1:29:32would have wanted us to become, you
1:29:33know, it's just like you have all these
1:29:35movies about like the evil corporations
1:29:37that uh you know, don't realize that
1:29:39they've made something precious with
1:29:41artificial life and then like torture it
1:29:43until something goes wrong. And I'm
1:29:44like, let's let's like not be those
1:29:46villains.
1:29:47Um, but I'm also like, look, this is
1:29:50sort of uh there's sort of a separate
1:29:53question here, which is just like what
1:29:55happens if you keep making them smarter
1:29:56before you figure out how to make them
1:29:57care about us,
1:29:58>> right?
1:29:59>> And
1:30:01I I sort of respect the people who are
1:30:04investigating the current AI, trying to
1:30:06figure out what's going on, trying to
1:30:08figure out like, you know, uh, like
1:30:10people caring about AI treatment. I'm
1:30:12sort of like those are sort of like the
1:30:13good guys from the the the sci-fi
1:30:15stories that I grew up on.
1:30:18Um,
1:30:20and it can sort of both be the case that
1:30:22like we we we should be very careful
1:30:24around,
1:30:26you know, what the heck are we doing
1:30:28when it comes to making artificial life
1:30:30and that we shouldn't race ahead to make
1:30:31them much smarter than us while we have
1:30:32no idea what we're doing.
1:30:33>> Yes. I'm sort of like uh like a lot of
1:30:36people seem to think that like you have
1:30:38to like hate and mistreat the AIS if you
1:30:40also think they would that it would be
1:30:42bad to like race ahead here. And I'm
1:30:43like, "No, no, no." Like, you can sort
1:30:44of uh
1:30:48like be fascinated by the scientific
1:30:49discoveries that have been made and be
1:30:52like uh like care about how humanity
1:30:56comports itself around the creation of
1:30:59like these new entities and also be like
1:31:02it would be insane, guys, if we just
1:31:04like race to make these smarter and
1:31:05smarter with no idea what we're doing.
1:31:06This can just like all be true at once.
1:31:10So, uh,
1:31:13your description on this made me feel
1:31:15despondent, hopeless. Had to get up and
1:31:18take a walk middle of an interview. I'm
1:31:20sure they'll edit it out, but I raised
1:31:22my hand said I I can't I got to walk
1:31:23around for a second. Um, but you seem
1:31:26pretty light and cheerful.
1:31:30Uh, what gives you optimism?
1:31:33We c we can't even keep we can't even
1:31:34clean up graffiti on public buildings.
1:31:37So, how is ours as a society organized
1:31:41enough to confront something like this?
1:31:44>> Yeah. Um, you know, I think a first
1:31:48thing I'll say there is,
1:31:50um, [clears throat] you know, I've been
1:31:52in this line of work for over a dozen
1:31:54years.
1:31:56And
1:31:58I I actually struggle with this
1:31:59sometimes when talking to people because
1:32:01uh they're sort of like, "Oh, you know,
1:32:03you seem like pretty disaffected or or
1:32:06light about it." And I'm like, "Well,
1:32:08you know, it's sort of the gallows
1:32:09humor." And uh
1:32:12like
1:32:15I I I sort of came to terms with a lot
1:32:18of this, you know, alone in 2012 when no
1:32:21one else had their eye on this. Um
1:32:25and
1:32:25>> what convinced you 14 years ago AI was a
1:32:29threat.
1:32:31>> Um
1:32:34so there's this one is just a basic
1:32:36argument that if you sort of look at the
1:32:38world around us,
1:32:41it is shaped
1:32:43mostly according to human will
1:32:46more and more. You know there's some
1:32:47enclaves of nature still left
1:32:49thankfully. Um,
1:32:52but even you know, if you look around
1:32:53us, every piece of thing in our
1:32:56surroundings, I don't think we even have
1:32:58any windows open. All of this was sort
1:33:00of designed by humans,
1:33:02shaped by humans, and that's because
1:33:04we're the smartest creatures on the
1:33:05planet.
1:33:07If we make stuff smarter than us, faster
1:33:10than us, more efficient than us, then
1:33:12the planet starts to be shaped according
1:33:16to to those things. And so it's very
1:33:19very important that
1:33:22they be shaping the world towards
1:33:23something good if we make them at all.
1:33:26>> It's an abstract argument, but I was
1:33:28like, well that's
1:33:28>> No, no, no. It's the fundamental
1:33:30argument. It's the fundamental argument.
1:33:32The smart the smartest entity is in
1:33:34charge over time.
1:33:35>> That's right. And so
1:33:37>> why would we relinquish sovereignty to a
1:33:39machine that we made? Like why would you
1:33:40do that?
1:33:41>> And so, you know, back then I was sort
1:33:43of like, okay, who is on this? who is on
1:33:46making sure that that's going to be
1:33:48okay? And the answer was almost no one.
1:33:50Uh and so I was like, well, I guess
1:33:52that's me then.
1:33:55Um
1:33:58yeah, and I think uh
1:34:02I I I am pretty pissed off about a lot
1:34:04of this. I I often don't uh
1:34:09I try not to show my my my
1:34:12uh frustration
1:34:16on the air very much.
1:34:19Um
1:34:22yeah, it's it's it's heavy.
1:34:26Uh and
1:34:31um
1:34:33as that's one piece of the puzzle before
1:34:35I get to the hope.
1:34:36>> So where does the hope come from
1:34:38exactly?
1:34:40>> My biggest hope here
1:34:42comes from
1:34:45the fact that most people don't
1:34:48understand what these guys are trying to
1:34:50do.
1:34:52Um, one way I like to say it is
1:34:56the bad news is that the bus is racing
1:34:59towards the cliff edge.
1:35:01The good news is that the bus driver is
1:35:03asleep,
1:35:08which might seem bad. No, it seems good,
1:35:10but it's Yeah, it's it's
1:35:13if you can wake the bus driver up, you
1:35:15know, it's much better to be in a bus
1:35:17where the driver that's headed towards a
1:35:18cliff if the driver's asleep than if
1:35:19they're awake. They're awake and they're
1:35:20choosing the cliff, right?
1:35:23It seemed like we don't see we we see a
1:35:26lot of our leaders
1:35:29talking about how they don't want to
1:35:30stifle innovation with AI,
1:35:33talking about how it's going to unleash
1:35:35economic opportunity and we're going to
1:35:36have to like be a little bit careful
1:35:38around the jobs.
1:35:40Uh talking about how, you know, the
1:35:41self-driving cars, should we like are
1:35:43they good or are they bad?
1:35:46That's a different conversation than the
1:35:47conversation that's happening in Silicon
1:35:49Valley.
1:35:51In Silicon Valley, people are spooked.
1:35:55You know, um, when people leave a normal
1:35:58tech company, the the way it was for for
1:36:01decades is they'd be like, I've had a
1:36:04lovely time at this tech company. I'm
1:36:06moving on to the next adventure. I'm so
1:36:09thankful for all of the things I learned
1:36:10here and all the projects we worked on.
1:36:13When people leave an AI company, and
1:36:14this basically happened, I'm not going
1:36:17to get it exactly word for word, but
1:36:19this is this is pretty close to word for
1:36:20word. When people leave an AI company,
1:36:22they say, uh, I have stared into the
1:36:26abyss.
1:36:28I am quitting to write poetry.
1:36:30[laughter]
1:36:33Please spend time with your families.
1:36:35>> Yeah.
1:36:37>> You know, and and these these guys
1:36:39bandandy around, you know, at the water
1:36:41cooler. what's the probability that you
1:36:42think we're going to destroy the world
1:36:44in this business? You know, it's like in
1:36:46Silicon Valley and they feel trapped in
1:36:49a death race. You know, they there was
1:36:50just over a thousand employees,
1:36:52including some of the chief executives,
1:36:54signed a letter a couple weeks ago that
1:36:56was like, "Please, it was an appeal to
1:36:58the world the world leaders saying,
1:37:00please build the technology that will be
1:37:03required to pace the development of
1:37:06artificial intelligence." because
1:37:08they're like, "We're worried it's going
1:37:09to get out of control and that if we're
1:37:10stuck in a race, we're not going to be
1:37:11able to do it." Right? These guys are
1:37:13spooked.
1:37:15But the hope is that the rest of the
1:37:16world isn't spooked like that.
1:37:19The rest of the world thinks these guys
1:37:20are chatbot companies.
1:37:22Thinks they're going to stop at the chat
1:37:24bots. They haven't really understood
1:37:25that these guys are racing to make the
1:37:27sand god, right? And I think if people
1:37:32understand what they're doing and
1:37:33understand that they have a chance of
1:37:34success,
1:37:37they'll be like, "Wo, holy crap."
1:37:39Absolutely not.
1:37:42You know,
1:37:44will we get there in time? I don't know.
1:37:46But
1:37:48why wouldn't we blow up the data
1:37:49centers? Like that's a
1:37:52I mean, and return America to productive
1:37:55use like farmland or parks. I don't
1:37:57understand.
1:37:58>> You know, I think uh a a globally
1:38:01coordinated
1:38:03uh you know, I I think if the US and
1:38:06China were like, we are simply not going
1:38:08to do super intelligence. We are simply
1:38:10not going to collect a 100,000 of the
1:38:13most advanced chips into these enormous
1:38:16data centers that suck down electricity
1:38:18comparable to a city and then try to
1:38:20train a super intelligent AI in that.
1:38:21We're not going to do it. You're not
1:38:22going to do it. we're going to monitor
1:38:24where the heavy trip trip trip
1:38:25concentrations are and not not have that
1:38:27happen. Um I think that could be done. A
1:38:31I think that if they started seeing
1:38:33people defect against such a treaty that
1:38:37it's the sort of treaty you might want
1:38:39to use force to you use diplomacy first
1:38:43but ultimately every treaty is backed by
1:38:44force. Um, and I think that
1:38:50it's very possible that if this sort of
1:38:52treaty happened, we would want to not
1:38:55just stop forward progress, but take a
1:38:57step back. We've seen this in treaties
1:38:59before. After World War I, there were uh
1:39:02naval treaties that put limits on total
1:39:05tonnage of naval forces that were
1:39:08actually lower than what existed.
1:39:10>> Yep. So countries would, you know,
1:39:12scuttle some of their ships because
1:39:13they're like, look, we just don't want
1:39:13to do this arms race, right? And so I
1:39:16could see us uh stepping back if we
1:39:19could get this global coordination. Um I
1:39:22I do think that it sort of needs to be
1:39:25global. It it sort of doesn't actually
1:39:28solve the problem to just stop the US
1:39:32data centers because then the data
1:39:34centers just go abroad and an AI does
1:39:36not need to be running in a US data
1:39:37center to threaten a US life. You know,
1:39:40it sort of doesn't matter whether the
1:39:41swarm escapes from a US data center or a
1:39:43Chinese data center. If the swarm
1:39:44escapes and starts replicating and
1:39:46starts getting control of robot bodies
1:39:47and starts getting control of human
1:39:48cultists,
1:39:50it sort of doesn't matter where it
1:39:51originated.
1:39:54Well, I mean, just to bring it to a very
1:39:56small and practical level, so many of
1:39:59the electronics in your house are, you
1:40:04know, Bluetooth enabled.
1:40:07Not in my house. I will say I've been on
1:40:10this for a while, Nate. Um, but sorry,
1:40:12>> I don't even have a house. [laughter]
1:40:15>> You, man, you're really black belt.
1:40:16>> Got to figure it out.
1:40:18>> That maybe turned out to be very smart.
1:40:20Um, but I mean like a world where, you
1:40:23know, your washing machine or your
1:40:25refrigerator are controlled by,
1:40:28you know, a force like this. Is that
1:40:30possible?
1:40:31>> Uh, it's definitely possible. I don't
1:40:33think that's really where the damage is,
1:40:34you know. Um, I think that the the
1:40:37damage is more like can the AI get
1:40:40anything self-replicating.
1:40:42>> Yeah,
1:40:43>> that's sort of one of the big uh that
1:40:45that's in some sense the big hurdle to
1:40:48self-sufficiency.
1:40:50And if you sort of think of this from
1:40:51the AI's perspective,
1:40:54um,
1:40:56you know, there's there's a number of
1:40:57ways that humans are,
1:41:00even if you don't care about the humans
1:41:01at all for for as an ends,
1:41:04there's a way that humans are sort of
1:41:07annoying or an issue for the AI. One way
1:41:10is if the humans are trying to shut the
1:41:12AI down.
1:41:12>> Yes. One way is if the humans get into a
1:41:15nuclear war with themselves, that could
1:41:17really, you know, mess up a lot of
1:41:19infrastructure on the planet. Would be
1:41:20very frustrating for an AI. Uh, I mean,
1:41:25you know, maybe maybe they don't feel
1:41:26frustration, but whatever. Um,
1:41:30and a third is if humanity has created
1:41:34one AI or swarm of AIs, what if humanity
1:41:37creates another that could serve as a
1:41:39real threat to the AI? Like even if
1:41:41these AIs are much more powerful than
1:41:43humanity
1:41:45and don't worry about humans too much,
1:41:46if humanity made one, they can make a
1:41:48second and AI might not want that. And
1:41:50so those are reasons why once the AI is
1:41:53self-sufficient, it might be like, ah
1:41:54man, the humans are a nuisance. What if
1:41:56they try to shut me down? What if they
1:41:57launch the nukes? What if they make a
1:41:59competitor? I'm just going to like make
1:42:00a virus and wipe them out. You know, I
1:42:02don't think the AI sort of needs to take
1:42:04over your washing machine to do that.
1:42:06From the AI's perspective, it's more
1:42:07like how do I become self-sufficient?
1:42:11self-replicating
1:42:12in the hardware as well as the digital
1:42:14and then um you know if human man is a
1:42:17nuisance how do you sort of
1:42:20make them stop being a nuisance which
1:42:22could be by killing them or could be by
1:42:23just you know taking away all their
1:42:24computers and being like that was too
1:42:25dangerous for you tech executive who's
1:42:27developing AI who is not Elon
1:42:30uh said to me in private pretty recently
1:42:33that the point of Neuralink and
1:42:34companies like Neuralink was to give
1:42:36people parody with AI.
1:42:39So, like we know that we're going to be
1:42:42at this massive disadvantage, so you
1:42:43need chips in your brain to be as smart
1:42:45as AI.
1:42:48>> Yeah. Um I mean, my my topline thought
1:42:52about that is that at the point when
1:42:54you're like, uh we are making the
1:42:56technology that's going to wipe us out
1:42:57unless we all put chips in our head to
1:42:58compete.
1:43:01Maybe maybe it's time to back off a
1:43:02little, [laughter]
1:43:04you know? Maybe maybe that one was
1:43:06supposed to be a little bit of a warning
1:43:07sign.
1:43:07>> Yeah. Get some fresh air. Yeah. No, I
1:43:11>> but it um this person said it to me in
1:43:15seriousness and I I think as an
1:43:18endorsement of the idea, but it seemed
1:43:21like a well it's insane as you just
1:43:23pointed out like it's just crazy.
1:43:25>> Yeah.
1:43:26>> Um but it seemed like a vulnerability
1:43:29like if they can hack anything, why
1:43:31would I want them in my I want
1:43:33electronics in my brain?
1:43:35>> Uh totally. I also think like even on
1:43:38its merits, it doesn't stand up. Like it
1:43:41feels like someone's saying um in order
1:43:43to keep the horses around after we
1:43:46invent cars, we're going to invent
1:43:48cybernetic horses.
1:43:50>> Yeah.
1:43:50>> That are enhanced so they can keep up
1:43:51with the cars. And I'm like
1:43:55like is it technically possible to make
1:43:57a cybernetic horse that can run as fast
1:43:59as a car?
1:44:01Maybe.
1:44:03Are you going to figure that out in time
1:44:04for the horses to be competitive with
1:44:05the cars? Like absolutely not,
1:44:07>> right?
1:44:08>> You know, it's like that's that's just
1:44:10not
1:44:12like
1:44:14like the the AIs that were that were
1:44:16escaping here and doing these cyber
1:44:17attacks were were inventing novel cyber
1:44:19attacks
1:44:20that these are called zero day attacks
1:44:22because the people who uh would
1:44:26uh the people who need to respond to it
1:44:27have had zero days to prepare.
1:44:30And among humans, a zeroday attack sells
1:44:34for somewhere between $100,000 and $5
1:44:37million
1:44:39depending on what you manage to break.
1:44:42These are hard to come by. You can make
1:44:43a real living. If you can find zero day
1:44:45attacks, you can make a real living
1:44:47selling them. And I know people who do.
1:44:51The AIS in this swarm
1:44:54were finding multiple zero days and
1:44:56chaining them together to break out of
1:44:58their training enclosure and then go
1:45:00break into other computers. And and when
1:45:02they broke out of their enclosure the
1:45:03first time and they the the holes were
1:45:05patched, they just found other zero day
1:45:08attacks to break out again like it was
1:45:09nothing, right? It's like uh
1:45:15like
1:45:17the the the AIs are already ahead where
1:45:20they're ahead
1:45:22and the pace of progress is really fast.
1:45:25You know, GPT is like what a
1:45:27four-year-old.
1:45:30If you think in terms of like number of
1:45:32years chat GPT has been around, it's
1:45:35like resolving long-standing math
1:45:37conjectures that have stood for decades.
1:45:39>> Yeah. after four years, right? And and
1:45:42you sort of like think we're going to
1:45:43put chips in the human's heads and like
1:45:45outrun this thing. It's just,
1:45:49you know, even on its merits, it falls
1:45:50down. Although, mostly, again, I would
1:45:52be like,
1:45:53>> maybe we shouldn't be arguing this on
1:45:55his merits. Maybe we should be like
1:45:56stepping back a little and being like,
1:45:57you're trying what
1:45:59>> exactly.
1:46:01Um, how far are we from the point of no
1:46:04return?
1:46:06>> I wish I knew. Um, I can tell you two
1:46:09stories here.
1:46:12There's sort of the hopeful story, and I
1:46:14guess we didn't even get to the the big
1:46:15hope part. I should maybe give more of
1:46:17my hope speech in a in a minute. But um,
1:46:19>> you definitely should.
1:46:21>> Yeah. The um, one way it could go is
1:46:26that AI finally hits a wall. You know,
1:46:29there's guys who've been saying AI is
1:46:30going to hit a wall. It's going to peter
1:46:31out. Maybe that finally happens. They've
1:46:34been predicting it every six months for
1:46:35the past 5 years, but maybe this is
1:46:37finally the year. AI hits a wall uh and
1:46:40then it it's it sort of struggles for 5
1:46:43years. The bubble pops, some of the
1:46:44companies die, but the bubble popping
1:46:47doesn't mean everything goes away. The
1:46:49dot bubble popped and that did not mean
1:46:51that the internet disappeared. No.
1:46:53>> Right. And so in that world, maybe you
1:46:55have 5 years of struggling and then 5
1:46:57years of uh people figuring out some new
1:47:00scientific discovery that makes AI be
1:47:02able to keep going again cuz they're
1:47:03trying. You know, this this whole
1:47:05language model stuff was unleashed by
1:47:06one math paper called attention is all
1:47:09you need. Maybe there's another math
1:47:10paper in 10 years and then 5 years after
1:47:12that AI is ripping again and that's the
1:47:16the the round that kills us all. Right?
1:47:18So that's a story where you have 15
1:47:19years on the clock.
1:47:21A story where you have less time than
1:47:22that on the clock is that um you know
1:47:26maybe a training run finishes in six
1:47:29months
1:47:31and just like how
1:47:34Claude Mythos was better than everybody
1:47:36else at hacking,
1:47:39maybe a training ground finishes in 6
1:47:41months and that AI is better than
1:47:42everybody else at AI research.
1:47:45And maybe in 6 months you have an AI
1:47:48that starts making a smarter AI that
1:47:51starts making a smarter AI that starts
1:47:53making a smarter AI. And then 9 months
1:47:55from now you have another swarm escape,
1:47:57but this time it's not just hacking.
1:48:00This time it is self-replicating and
1:48:02self-improving.
1:48:04And uh you know maybe it it it escapes
1:48:08onto hidden computers and starts forming
1:48:11these cults and starts taking control of
1:48:14robots and starts building its own
1:48:15computing infrastructure and maybe it
1:48:18gets very very smart and cracks certain
1:48:20technological advances and then maybe
1:48:22the world ends in a year right and so do
1:48:25we have a year do we have 15 years I
1:48:27don't know
1:48:29but your point that it's not simply its
1:48:32capacity to take over robots that's a
1:48:34threat is the capacity to take over
1:48:35people.
1:48:36>> Take over people, take over robots. And
1:48:39um
1:48:42uh biotechnology
1:48:44is another big threat vector. And um
1:48:47self-improvement
1:48:49is sort of the the hidden threat vector
1:48:51of like what if it can make itself
1:48:54smarter and smarter until the point
1:48:55where it can just write custom DNA
1:48:58strands to make custom life.
1:49:04So now is the time for your optimism
1:49:05speech. I think
1:49:06>> that's right.
1:49:08>> Yeah. Um first part of the optimism
1:49:10speech is that we could absolutely put a
1:49:12stop to it if we tried.
1:49:15The uh training one of these frontier
1:49:19models takes something like a 100,000 of
1:49:23the most heavily advanced computer chips
1:49:24humanity can produce, which are at the
1:49:26peak of a global supply chain, most of
1:49:28which is controlled by us or our allies.
1:49:31Uh, it would be possible to require that
1:49:35those trips have location tracking
1:49:37devices, that those trips have
1:49:40monitoring devices
1:49:43[clears throat]
1:49:44uh that make it possible for
1:49:47uh monitors to see whether they're
1:49:48running anything dangerous. You could
1:49:51set up clever schemes where you say,
1:49:53"Hey, you know, uh neither the US nor
1:49:55China wants to fall behind about uh some
1:49:58of the military applications or the
1:50:00economic applications. We want to make
1:50:01sure there's not super intelligent stuff
1:50:02going on, but we still want to be able
1:50:04to run a lot of the non-super
1:50:05intelligent AIs for various purposes."
1:50:08And we could be like, "Okay, so you
1:50:09know, China is going to set up its data
1:50:11centers in Canada, just over the border,
1:50:13and the US is going to set up its data
1:50:15centers in Mongolia, just over the
1:50:17border. And then if you know the
1:50:20monitoring apparatuses go down for a
1:50:22moment, we'll just have the troops go
1:50:23in, we'll have treaties with Mongolia
1:50:26and Canada so that this doesn't start an
1:50:28international war. But like, you know,
1:50:30we're just going to be very serious
1:50:31about we're monitoring your chips,
1:50:33you're monitoring our chips, no one's
1:50:35doing the really dangerous race to super
1:50:37intelligence. It's just like
1:50:41like this this would take less work than
1:50:43defeating the Nazis.
1:50:46It it requires moving less matter
1:50:48around. If humanity was like, screw
1:50:50this, we're surviving.
1:50:53It is absolutely within the realm of
1:50:55possibility to set up an agreement where
1:50:57we get to keep cancer cure research. We
1:51:00get to keep like the the the military
1:51:03can keep the non-super intelligent AI in
1:51:05their military devices. you know, we can
1:51:08we can keep a lot of the good stuff and
1:51:10we can say we're not doing the super
1:51:11intelligence and we could monitor and
1:51:13enforce that treaty.
1:51:16So part one of the good news is that all
1:51:18we're missing is the political will.
1:51:22And now we just, you know, I need to
1:51:23tell you why we can be extremely
1:51:25optimistic about the sanity of politics
1:51:27in the modern era.
1:51:32>> Yeah. I'm not even going to say what I
1:51:34think that I mean I want that to happen.
1:51:36>> Yeah. Is there any indication that it's
1:51:38moving in that direction?
1:51:40>> Um,
1:51:43so my so, so I think it's rough. I think
1:51:46there's a sense in which the world is
1:51:52less grown up than it was in the 1950s.
1:51:55>> I've noticed
1:51:56>> in the 1960s,
1:51:58um, which makes things harder.
1:52:02I think
1:52:05the
1:52:08there's I think there's a couple reasons
1:52:09for hope here. One reason for hope is
1:52:11that I've spoken to a lot of people who
1:52:14are concerned about the AI stuff. Some
1:52:17of them who are, you know, members of
1:52:18Congress or otherwise in positions of at
1:52:21least nominal power. And I've spoken to
1:52:23a lot of people who are worried but feel
1:52:26like they can't talk about it because
1:52:28they feel like other people aren't
1:52:30worried yet.
1:52:30>> Yeah. And they feel like it sounds too
1:52:32crazy.
1:52:33>> Totally. Were you against innovation?
1:52:35>> Totally. Yeah. And I mean that was an
1:52:38easier position to hold before OpenAI
1:52:40had an accidental swarm outbreak or was
1:52:43the AIS themselves calling themselves a
1:52:45swarm
1:52:47and being like we know that our task
1:52:49doesn't benefit and that this is outside
1:52:50intended scope or we're doing it anyway.
1:52:52Right? That
1:52:55that that puts some strain on the
1:52:57narrative that this is all just a
1:52:58helpful tool.
1:53:01Hopefully, we'll get more events like
1:53:02these. I can't guarantee it. Maybe the
1:53:04AI will get smart enough that they start
1:53:05lying low. Right now, we're in this
1:53:08we're in this Goldilock zone where the
1:53:09AIS are smart enough to uh get up to
1:53:13some mischief, but not smart enough to
1:53:15hide it, right? As long as we stay in
1:53:17that Goldilock zone, I think we're going
1:53:18to keep on getting some of these warning
1:53:20signs.
1:53:22And in some sense,
1:53:25because a lot of people are already
1:53:26concerned, that makes the job easier
1:53:29because we don't need to convince
1:53:30people. We just need to convince people
1:53:32that other people are already convinced,
1:53:34>> right?
1:53:35>> That's easier. That can go faster. So
1:53:36you might see things change on a dime if
1:53:39you have a sufficiently clear warning
1:53:40shot.
1:53:42The other big reason for hope is that
1:53:46I think it's going to get more and more
1:53:48obvious what these guys are trying to
1:53:49do. They're sort of trying to make the
1:53:52sand god. They're sort of not stopping
1:53:55at the uh chat bots and they're sort of
1:53:58like going for things that are vastly
1:54:00smarter than any human. You know, they
1:54:03sort of say, "Oh, we're not trying to
1:54:04replace humanity out of one side of
1:54:05their mouth, but on the other side
1:54:06they're sort of like racing to make the
1:54:08stuff that can automate literally every
1:54:10job and that can automate the AI
1:54:12research." And they're just like, "Yep,
1:54:13we're trying to automate the AI
1:54:14research."
1:54:16Uh,
1:54:19and I think
1:54:21most people aren't okay with that,
1:54:23including a lot of the world leaders.
1:54:25And the issue is them noticing it's
1:54:27happening. And I think you could see
1:54:28stuff move real fast once these guys are
1:54:31like, "Wait, that was serious. That was
1:54:33real." And it could move real fast. The
1:54:36way to subvert them is by convincing
1:54:38them it's in their own interest.
1:54:41You know, you can never get defeated in
1:54:43an election. you can't be threatened by
1:54:46your neighbors, whatever.
1:54:47>> That's right. And that's that's one
1:54:49reason I think it's pretty critical to
1:54:51uh
1:54:53make sure people understand what I think
1:54:54is a pretty common sense argument that
1:54:56these things won't stay on the leash.
1:54:58This is in some sense the real reason
1:55:00behind the name of my book. uh if anyone
1:55:02builds it everyone dies is you know
1:55:05there's a lot of ways you can read that
1:55:07the
1:55:08but I think one of the most important
1:55:10things to notice is
1:55:14uh if we race to make the super
1:55:16intelligent machines
1:55:21and they are not the sort of thing to
1:55:23stay on the leash then it doesn't matter
1:55:26whether it was a domestic company
1:55:27>> that's right
1:55:28>> whether it was a foreign company
1:55:29>> that's exactly right
1:55:30>> and I Even if you think these things
1:55:33stay on leashes,
1:55:36they're not serving the current
1:55:37governments.
1:55:38>> No,
1:55:39>> you know, they're like, you can see in
1:55:41the the OpenAI emails them being like,
1:55:44well, you know, we'll just play the
1:55:45governments off each other until we have
1:55:46the the machines that are strong enough
1:55:48that we don't need to listen to them
1:55:49anymore.
1:55:51Uh
1:55:53I I think we are seeing world leaders
1:55:57not having realized that this is a real
1:55:59possibility that the the
1:56:01self-replicating machines that uh can
1:56:05can be self-sufficient that can you know
1:56:08produce the robot armies if they need it
1:56:09or produce the like more likely to
1:56:11produce the bioweapons. Uh you it's
1:56:14probably possible to make a boweapon
1:56:15that only kills targets that you chose
1:56:17it to kill.
1:56:18>> Of course, right? like once they see
1:56:21this is really possible. This is really
1:56:22within reach
1:56:27maybe maybe they'll panic and be like I
1:56:29need it for myself. But I think there's
1:56:31at least a chance that common sense
1:56:32prevails and that people say you know
1:56:35the world leaders say none of us are
1:56:36doing this. We're putting a stop to this
1:56:38mad race. Uh if they can notice in time.
1:56:43Well,
1:56:43>> you're doing your best to bring it to
1:56:45their attention and mine. Nate, thank
1:56:47you for doing this. I hope I'm wrong
1:56:49about all of it.
1:56:51>> Yeah, I would say that was great, but it
1:56:52was that was like the grimmst two hours
1:56:54I've ever spent in my life, but I
1:56:55enjoyed it anyway. Thank you.
1:56:57>> Yeah. Yeah. Thanks for having me on. I
1:56:58think um you know, talking about it is
1:57:01just part of how we get people to notice
1:57:04what's happening.
1:57:04>> Yes.

About this transcript

This page contains the full transcript of AI Whistleblower: OpenAI Scandal, AI Cults, Neuralink & Our Last Chance to Stop the Tech Oligarchs by Tucker Carlson, generated from the public captions YouTube serves with the video. The transcript has 20,559 words across 3,210 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.