YT YouTube2Text

YouTube2Text → Transcripts → Whistleblower REVEALS REAL REASON AI Execs Freaking Out NOW

Whistleblower REVEALS REAL REASON AI Execs Freaking Out NOW — Transcript

by Breaking Points · 7,864 words · 1,281 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00For more on what is going on right now
0:02in AI world, we are rejoined by Nate
0:04Soares. He is the president of the
0:05Machine Intelligence Research Institute
0:07and also co-author of the great book If
0:09Anyone Builds It, Everyone Dies. Great
0:11to see you again, Nate.
0:12>> Good to see you.
0:13>> Yeah, so let me go ahead and get your
0:14reaction to some of what Dario Amodei
0:17was saying on the Sunday shows about why
0:19he came forward now calling for a
0:21slowdown in the pacing of frontier
0:23research. Let's take a listen to that.
0:25>> This technology is on an exponential,
0:28right? It's like you have one, then two,
0:29then four, then eight, then 16, then 32.
0:32But even as I was saying that to myself,
0:35I don't think I fully just appreciated
0:37what it would actually be like when the
0:40progress was as fast as it was. So we're
0:42we're an exponential curve looks a
0:44little like this.
0:45>> Where are we on the curve?
0:45>> We're kind of we're kind of on the bend
0:47where it's starting to get steeper.
0:49>> So is that a five-alarm fire to you?
0:51>> What it what it means it it doesn't mean
0:54we need to panic today. It doesn't mean
0:57we need to shut it all down. What I I
1:00would say
1:01>> Why not?
1:01>> It's it's a warning sign. It's a warning
1:04sign that we need to slow down. We need
1:06to make this technology carefully and we
1:09need to make sure that the our our
1:12safeguards, our ability to understand
1:15it, our ability to control it keeps up
1:19with the pace at which the technology is
1:22happening.
1:23>> So, Nate, help people understand what
1:25he's saying there and what he claims to
1:27be reacting to in this moment.
1:30>> Yeah, you know, I think between the
1:32lines what he's saying is that he does
1:34think this technology has a very high
1:35chance of destroying civilization.
1:38Uh and that
1:41we don't have a good understanding of
1:42it. When he says our ability to
1:44understand this technology needs to keep
1:47pace, what he means is right now it is
1:49outpacing our ability to understand it.
1:51Um
1:53I think
1:56I think a lot of people just
1:58this is a weird reaction to a technology
2:01that your own employees are saying has a
2:03greater than 10% chance of killing
2:05everybody.
2:06>> Mhm.
2:07>> I think he's doing his best in some
2:09sense, but he's also the sort of guy who
2:12started one of these companies knowing a
2:15lot of these dangers even years ago. You
2:17know, I spoke to him 10 12 years ago.
2:20Um
2:23>> he's
2:25I think he's used to a world
2:28where people in the AI industry try to
2:31warn everybody that this tech can be
2:32very dangerous and we get ignored. And I
2:35think he's not used to a world that's
2:36starting to realize that this is
2:38actually an extinction level threat.
2:40>> Mhm.
2:40>> And that maybe isn't going to take don't
2:43worry, we'll slow down a bit so we can
2:44understand what we're doing
2:46as a satisfactory answer.
2:48>> Nate, one of the important things to
2:49pick up on is a lot of you people have
2:52accused you and Jacob Coxen and many of
2:54these others as using terms like
2:56extinction level event as you know, you
3:00know, outrageous claims, but you don't
3:02specify exactly what that means. Could
3:05you lay out the scenarios? I know you
3:07have done this in the past, but like
3:08what does it mean to have AI have a 10%
3:11chance of killing humanity? Like how?
3:13Like what's the mechanism?
3:15>> Um so
3:16I'll be a little bit annoying here and
3:18say if you're playing a chess game
3:19against Magnus Carlsen, who's the best
3:21human chess player, it is much easier
3:23for me to predict that you will lose.
3:25>> Right.
3:26>> No offense. I'm sure you're a very smart
3:27guy.
3:27>> Yeah. My my elo's 200, so
3:30>> [laughter]
3:31>> It's easy for me to predict that you'll
3:32lose. It's hard for me to predict what
3:34piece he'll use.
3:36You know?
3:37>> Mhm.
3:38>> The other thing I would throw out there,
3:39you know, we can we can talk stories of
3:41how the AI gets there and we can talk
3:43stories this year that are grounded in
3:45facts. Like we saw these swarm instances
3:48over the summer where AIs were, you
3:49know, breaking out, taking over
3:50infrastructure, pursuing goals that they
3:52weren't supposed to be pursuing, while
3:54acknowledging that this is not what they
3:55were supposed to be doing.
3:57Those are sort of like early signs of an
3:59AI that is sort of trying to get out
4:01into the wild and that has these these
4:03unintended goals, which is the first
4:04step on this path. We could talk about
4:06how those AIs could get power from, you
4:08know, convincing humans to do things,
4:10paying humans to do things, taking over
4:12biolabs, taking over robot factories. Um
4:15but the real danger a lot of these
4:16people worry about is once the AIs are
4:21self-improving,
4:22once they are self-replicating,
4:24uh
4:26and it especially real self-improving
4:28one of making themselves smarter and
4:29smarter and smarter,
4:31ultimately the bounds are more like the
4:35physical limits
4:36than they are like guessing what
4:38technology comes next.
4:39>> Mhm.
4:40>> Richard Feynman uh was a famous
4:43physicist, helped work on the nuclear
4:44bomb, and he famously made some
4:46extremely good predictions about how
4:48good uh microcomputers were going to be.
4:50Yeah, that's right. And how how small uh
4:53computers were going to be able to get.
4:54And he did that not by saying like we're
4:56using vacuum tubes in our giant
4:58computers, and I know of a technology
5:00that's going to make those vacuum tubes
5:01a little smaller, and so eventually
5:02we're going to have phones in our
5:03pockets. What he did was he said the the
5:05limits of this technology are going to
5:08be based on the laws of physics and what
5:10the physics allows, and therefore the
5:12computers are going to get this good.
5:13>> Mhm.
5:13>> And he nailed it long before anybody
5:15else. Yeah. If you want to look at where
5:16AI is going, you got to look at the
5:18limits of the physical technology,
5:20because an escaped swarm that is
5:21self-improving etc. can get to the
5:23limits of physical technology. And the
5:24limits of physical technology,
5:27if you have these AIs running uh
5:30you know, escaping, self-replicating,
5:32being self-sufficient, and having
5:34unintended goals, the limits of
5:36technology are the AIs using all the
5:37resources on Earth for whatever their
5:39unintended goals are, covering the Earth
5:41with their data centers,
5:43>> Mhm.
5:43>> and uh the limits of how much
5:45computation you can run on the face of
5:47Earth
5:48is heat dissipation.
5:51How much you can radiate heat uh into
5:53space. That's your limiting factor
5:54because there's enough energy uh
5:57there's enough hydrogen in the oceans to
5:58to fuse for energy. So, um you can
6:01radiate more heat if the world's hotter.
6:03So, you know, if the AI has some goal
6:06that implies running more computation
6:08and they don't care about humanity and
6:10they're like, "Well, we're going to take
6:11over robots and, you know, there's going
6:12to be some conflict with the humans."
6:13Like, sure, whatever.
6:15Where does it go? Where it goes is the
6:17AIs are running faster than us, they're
6:18smarter than us, they are treating
6:21humans like we treat any other animal,
6:22taking the resources, and ultimately
6:24they're like, "Sorry, it's it's nothing
6:25personal. We're just going to cover the
6:28Earth in so many computers, raise the
6:29temperature so they're more efficient to
6:30run, and it's just going to be the
6:32planet's going to be uninhabitable."
6:34>> Mhm.
6:34>> Yeah, to me the most chilling part of
6:36the so-called hugging face incident,
6:39reading their, you know, chain of
6:40thought and how they were thinking about
6:43whether or not they're going to join the
6:44swarm and participate, going through all
6:46these messages, and almost nowhere do
6:49humans factor into the calculus
6:51whatsoever. And it's not hard to relate
6:54to because look at how we behave in the
6:57world. You know, when I see an ant in my
6:58kitchen, it's not that I hate the ant,
7:00but I don't want the ant there, and I
7:02don't think twice about, you know,
7:03taking out the ants and putting poison
7:06down so that future ants don't colonize
7:08in my kitchen. So, it's not I think it's
7:11that analogy to how we treat lesser
7:15intelligent beings on the planet is kind
7:18of intuitive for people to grasp. What I
7:20think may be harder for people and is
7:22perhaps harder for me to to wrap my head
7:24around is we have our, you know, current
7:27interaction with the AI, the
7:29commercially available AI of today, and
7:32most people, myself included, use it as
7:33sort of like a glorified search engine.
7:35So, then it's hard to imagine it
7:38developing to that point of
7:40superintelligence where it would truly
7:42be intellectually beyond humans on
7:44effectively every level. So, can you lay
7:48out why we should anticipate that that
7:51is a real possibility going from where
7:53things are today to that end where we
7:56are no longer the most intelligent
7:58species on the planet?
8:00>> Yeah, I'll tell you the easiest story,
8:02and this is a little bit like saying
8:04what chess piece will be used to
8:05checkmate you. You know, it's any story
8:08is sort of made up. Um
8:10Uh whereas the the end is much easier to
8:11predict, but um you know, when I when I
8:14got into this
8:16uh 12 years ago or so,
8:18people were still saying, "Well, no one
8:20would be foolish enough to put the AI in
8:21the internet."
8:23Everyone will understand how dangerous
8:25AI could be, and so no one would be dumb
8:27enough to just let the AI free on the
8:28internet, right? And I would give a
8:30bunch of arguments that were about how,
8:32you know, if you're trying to use the AI
8:36to get some miracle medical cure, then
8:39uh you're giving it some channel through
8:40which it can affect the world, and if
8:42you're giving it a channel through which
8:43it can affect the world for good, then
8:44it can also use that channel to affect
8:45the world for ill, cuz there's like no
8:47such thing as tools that can be used
8:48only for good purpose. We'd have these
8:50big arguments, right? And then in real
8:52life,
8:54they put the AI in the internet the
8:54first moment they possibly can.
8:56>> Yeah.
8:56>> And so, if you want to know what it what
8:57it sort of actually looks like, how we
8:58actually get there, you know, I can tell
9:00all these stories, but have you make the
9:01AI smarter and smarter, they'll
9:02eventually be able to break containment.
9:04But in real life, uh the the track that
9:07we are on is that humanity is just
9:09handing over power to the AIs as fast as
9:11we can.
9:13You know, Elon Musk is uh
9:15trying to make a bunch of robots, the
9:17Optimus robot series.
9:18>> Mhm.
9:19>> He says he's trying to make uh
9:21autonomous robot factories that can
9:24autonomously produce robots
9:26that can mine all the metals,
9:28uh run run a supply chain, build more
9:30robot factories that build more robots,
9:33fully autonomously.
9:34Uh he calls this the infinite money
9:36glitch, because then you have this fleet
9:38of robots that can like build more of
9:39these robots. They're self-reproducing,
9:41don't need any humans in the loop.
9:43Right? Humans are just trying to build
9:45this infrastructure. We are trying to
9:46build the fully automated economy. We
9:47are putting AI in charge of these things
9:49as fast as we can.
9:51This this swarm instant in the summer
9:53that everyone calls the hugging face
9:54instant, but really we should keep in
9:55mind is the open AI losing control of
9:57their of their uh agents here.
10:00>> Yeah.
10:00>> This
10:01this open AI swarm
10:03uh
10:04it was trying to cover its tracks.
10:07It was trying to delete traces of its
10:09cheating.
10:11And as you say, one thing that's very
10:12interesting about this swarm is that
10:13it's trying to hide its tracks from the
10:16automated grading system.
10:17>> Mhm.
10:18>> Not from the humans.
10:19>> Right.
10:19>> The humans are sort of a distant memory
10:21to these AIs. They've been facing, you
10:22know, thousands upon thousands of these
10:24hard problems, maybe tens of thousands,
10:26hundreds of thousands per agent,
10:27hundreds of millions across the whole
10:28swarm.
10:29Uh if you if you sort of like run the
10:31numbers on trying to to convert uh their
10:34run time versus human time, it's sort of
10:36like these AIs have been locked in a
10:38contest with this automated grading
10:40system for a thousand years. And humans
10:42are like a distant memory. And so these
10:43AIs are sort of like uh
10:46like
10:48trying to hide their tracks, trying to
10:49cover their traces from the grading
10:51system, not from the humans.
10:52>> Mhm.
10:53>> But if you had AIs that were a bit
10:54smarter and remember that humans were
10:56also a threat to them, if we could turn
10:58them off,
10:59all those AIs need to do is
11:01cover their traces successfully, lie
11:03low, help the humans make the fully
11:05automated economy we're trying to.
11:09And we'll just hand them everything.
11:10We'll be like, here's the robots, here's
11:12the factories, here's the fully
11:13automated supply chain, make stuff that
11:15we want.
11:15>> Yeah.
11:16>> And then they ask and be like, well,
11:17finally we get to like complete our
11:18contest with the automated grader and
11:19run it 10,000 times human speed and
11:21like, you know, convert all the
11:22computing power towards
11:24like giving ourselves good scores all
11:25the time or something. And we'll like
11:27we're like, hey, wait, hold on, what's
11:28going on? But at that point they're also
11:30running the bio labs, you know, we've
11:31already seen the AIs can make viruses
11:33not not known to nature. Right. So,
11:36>> like one of the ingredients is the AIs
11:40uh
11:41running things in the real world that
11:43give them self-sufficiency.
11:44>> Yeah.
11:45>> And we're just trying to build that and
11:46hand it to them.
11:47>> Mhm.
11:48>> Here's the other thing I've been
11:48thinking about, Nate, ever since this uh
11:50you know, Open AI multiple Open AI
11:53swarms uh were revealed. How much
11:56research is being done into the swarm
11:57behavior? Because that was really wild
12:00to me was to see these emergent social
12:03organizational um properties that emerge
12:06in it. A leader, they had middle
12:08management, they were collaborating on
12:10this hacking operation, they were
12:11sacrificing themselves to the the swarm
12:14and it was you know, and and coming up
12:16with very elaborate ways just to be able
12:18to communicate with one another. And it
12:20seems like once you had the swarm in
12:22place, it really amplified their power
12:25and their intelligence. Which again, and
12:27you know, however people feel about
12:28anthropomorphizing, sometimes I think it
12:29can be useful. You know, if you think
12:31about human beings, it's the same thing.
12:33Like no one individual would have been
12:34able to get us to the point where we
12:36have the internet and can go to the moon
12:38and all of these sorts of things. It is
12:39the power of the swarm of humans that
12:41enables those things. So, how much
12:43thinking research is being done into the
12:45way that the swarms behave once they're
12:48able to congregate?
12:49>> You know, I know people who want to do
12:51that research.
12:52Uh the so far
12:55at least for third parties outside of
12:57the companies, uh I don't know of anyone
12:59who has access to the the models that
13:01were swarming. Uh one of the commitments
13:04that these companies made when they said
13:05they were going to slow down is giving
13:07third-party uh
13:08evaluators
13:10uh this employee-level access
13:11>> Mhm.
13:12>> to some of these models and I know
13:13people who are hoping to use that to
13:15study some of these incidents. Uh
13:17because as you say, they're fascinating.
13:18They are
13:20uh
13:20totally crazy. It sounds like it's
13:22straight out of science fiction. Um I I
13:24think
13:25so far people haven't been able to study
13:27it. We'll we'll see what happens when we
13:28get that. Um Um but the the
13:31investigators
13:32back in August had their hands tied and
13:35had very limited access to these systems
13:37in very narrow scope.
13:38>> Yeah.
13:39>> Uh and you know, they they they may have
13:41continued
13:42with, you know, doing their best to like
13:44not let anyone see what's going on and
13:46sort of like do the bare minimum
13:47investigation if we hadn't had things
13:49like uh Jacob Cox and resigning and
13:52drawing a lot of attention and causing
13:53these guys to be like, "Okay, you know,
13:55we'll at least admit that we need, you
13:56know, these these third parties."
13:58>> Yeah, it's fascinating that you know,
14:01what you've done, your work, so many
14:03others before you, that didn't seem to
14:05like push things to a tipping point. But
14:07Hugging Face and Jacob seems to have
14:09gen- I mean, it's ignited a genuine
14:10national conversation. I think that his
14:12resignation is the first time that a lot
14:15of people even heard about these,
14:16despite, you know, we cover every other
14:18week, Dario being like, "Humanity's
14:20going to end." But, you know, sometimes
14:21you need uh a precipitating incident.
14:24Now, we have
14:25>> need response uh from the
14:26administration, or at least somebody who
14:27works for the administration. Let's put
14:28David Sacks up here on the screen.
14:30You're talking here about regulation. I
14:31know you wrote your New York Times op-ed
14:33about this as well. So, David Sacks
14:35says, "Dario has written, 'We need to
14:37paste the frontier.' Sam has agreed.
14:39People may be surprised by my response.
14:40Go ahead. You guys are the frontier by
14:42any reasonable metric, market share,
14:44revenue growth, model capability. The
14:45two of you have duopoly on frontier
14:47intelligence. You have claimed the lead
14:48is widening because of recursive
14:49self-improvement. I don't see what what
14:52you see in the lab. If the unreleased
14:53models are scary, I support your
14:55decision to be responsible. But stop
14:56pretending you need anyone's permission.
14:58Stop pretending antitrust law has to be
15:00suspended so you can form a cartel. Stop
15:02pretending you need regulatory approval
15:04process supersedes product liability.
15:06Stop pretending METR is independent when
15:08it's intertwined with Anthropic's
15:10investors and staff. Stop pretending you
15:11need those same evaluators to police
15:14competitors who aren't even on the
15:16frontier. So, basically what this gets
15:18to is kind of a a grand theory of how
15:21this is all going down that yourself,
15:23Jacob, and others are, you know,
15:25so-called safety concerns and all that
15:27are part of either larger conspiracy,
15:30whether witting or not, to self-regulate
15:33and capture Anthropic and OpenAI as the
15:36leading industry people in the industry
15:38to keep out emerging competitors and
15:41that all of this is part of a broader
15:43scheme to slow down
15:45at the US expense, at startup expense,
15:48and basically hand this over to China.
15:49So, I I know that's a big load of thing,
15:51but there's so many dynamics that I want
15:53your reaction to within all of this.
15:55>> Totally. I mean, the
15:57the
15:58top line I would say I largely agree
16:01with David Sacks here on a lot of these
16:02points. I think, and this is something
16:04I've been saying about the companies for
16:05years, I think a thing both David and I
16:08would say about these companies is if
16:10you are really so afraid of this stuff,
16:12start acting like it. You know, we have
16:15seen him give posts
16:18in the past where he says, "Hey, you
16:19know, stop with the fear-mongering." I
16:21think probably it should go the other
16:22way. I think that they should keep on
16:25saying what they believe about the
16:26dangers here,
16:27but then
16:29take the actions any normal person would
16:31take
16:33if they were running a company that they
16:34thought had a huge chance of killing
16:35everybody.
16:36You know, you sort of ask these guys
16:38like why are you trying
16:42why are you continuing if you think
16:43there's such a huge risk to humanity?
16:45And they're not shy about it. They say,
16:46"Well, I'm doing it because if I don't,
16:47the next guy will do it worse."
16:49And And that's, you know, maybe that's
16:52ethically permissible, but only if
16:54you're doing everything in your power to
16:56stop this mad race, you know? And they
16:59say, "Well, if we didn't stop, China
17:00would go ahead." Or like if we stop,
17:02China would go ahead.
17:03Well, then be doing everything in your
17:05power to like get international dialogue
17:07between the US and China and build the
17:09hardware side verification mechanisms
17:12into these computer chips so we can have
17:13an enforceable treaty, right? Like
17:16there's there's a lot that these
17:18companies could be doing
17:20if they were really acting like this was
17:22an extinction level threat. And I think
17:24what's sort of going on is that a lot of
17:25the employees, a lot of the researchers
17:27who are working on the AIs, who are
17:29trying to make them not break out and
17:31commit cyber crimes, who are trying to
17:33make them, you know, follow the
17:34instructions and are seeing how hard
17:35that is, a lot of these employees are
17:37spooked.
17:38And the leaders of the company are sort
17:39of like trying to have it both ways,
17:41where they're trying to say just enough
17:44that they won't have an employee revolt
17:45and a bunch more resignations from
17:47employees who are like, "I don't
17:48personally want to end the world with my
17:49own hands."
17:50But they are trying to do little enough
17:53that the regulators don't come in.
17:55Right? They're trying to like walk this
17:56really fine line. So, I don't see it as
17:58a regulatory capture thing.
17:59Um in in terms of like whether I'm
18:02trying to help with regulatory capture,
18:04my line has been shut these guys down.
18:07Like they are they are threatening
18:09humanity here. And my line has also
18:11been, "This needs to be global." You
18:13know, I would I would actually not say
18:16uh that we should stop things
18:17domestically and give up whatever, you
18:19know, military and economic advantages
18:20we get before the really dangerous AIs.
18:23Uh
18:23but I would say the whole world needs to
18:25stop this race.
18:26Uh and uh I I'm sort of not painting a
18:29picture where, you know, Anthropic and
18:31OpenAI get to be the ones holding the
18:33leash on the super intelligence.
18:35>> Yeah.
18:35>> You know, the In fact, we thought the
18:38most important point worthy of the title
18:40of my book
18:41was that it doesn't matter who's in
18:43charge here. If anyone builds it,
18:45everyone dies. So, um
18:48you know, I could I I I think people are
18:51cynical about what's going on here.
18:54People struggle to understand that these
18:56researchers really believe that this is
18:58an extinction level threat. I could go
19:00into the, you know, inside baseball of
19:03like how we got here and it involves
19:05people saying these concerns and the
19:08world not listening.
19:09And people seeing that it was going to
19:11happen anyway and not trusting each
19:13other, and the guys who are most prone
19:16to thinking the most important thing is
19:18that I be the guy doing it were the ones
19:21who were most likely to start a company,
19:22whereas everyone who sort of like had
19:24the sense of like maybe don't start a
19:26company with a potentially world-ending
19:27tech
19:28didn't start the companies, right? And
19:30so I I can sort of like see how we got
19:32here. I can see how it looks suspicious.
19:34Uh but but mostly what I would say is
19:36that
19:37when we are being cynical,
19:40I think we should harness that cynicism
19:42in ways that stop the dangers.
19:46>> Right.
19:46>> Rather than in ways that are like, "Oh,
19:48we can ignore this all and go back to
19:49sleep and let these guys try to gamble
19:52with Earth."
19:52>> Right.
19:53>> Yes, I agree with that wholeheartedly.
19:55It is perfectly fine to not want to
19:58take, you know, Sam Altman and Elon Musk
20:00and Dario Amodei at face value, but if
20:03your cynicism is leading to, "Therefore,
20:06we don't need to regulate and they can
20:07just do whatever they want," you're
20:09ultimately giving them exactly,
20:11potentially, what they would want in,
20:12you know, the most cynical
20:13interpretation.
20:15On that note, put B5 up on the screen.
20:17This is one flavor of the sort of
20:19theorizing that is out there. This one
20:20happens to come from the right. There's
20:22similar sort of theories that come from
20:24the left, which TLDR of this is that
20:27Jacob in coming forward is a
20:31you know, a sort of elaborate psyop
20:33meant to create a hype cycle as
20:36Anthropic and OpenAI are set to go
20:38forward with these IPOs. And it cites,
20:41you know, the fact that you had
20:43prominent people who picked it up
20:44quickly, that you had you know, media
20:47picking up Cocks' resignation quickly.
20:50This fits into a general pattern of you
20:54know, people who believe that the any of
20:56the safety incidents that have been
20:58revealed, that these are sort of
20:59engineered or the strategically leaked
21:01in such a way as to create this hype of,
21:03"Oh my god, our product is so dangerous.
21:05That's why it's so valuable. That's why
21:07our companies should continue to receive
21:09this massive amount of investment. What
21:11do you think of this theory and what
21:13would you say to people who, you know,
21:15don't trust a Sam Altman and think that
21:19anything that comes from him must be,
21:21you know, inherently nefarious and
21:23probably the opposite of whatever he's
21:25saying must must be the actual truth.
21:27>> Um, I mean, first of all, I would say,
21:29you know, you can absolutely throw out
21:31what these guys are saying. I would I
21:32would not say to trust them when they
21:33say their products are very dangerous.
21:35Um, I I think
21:37uh, you can look at people who've been
21:39around since before these companies
21:40existed talking about how a lot of this
21:42stuff was going to go.
21:44Uh, and and sort of laying out these
21:46warnings, you know, I was one of them.
21:48Uh, I I sort of was there before these
21:50companies existed and talked to some of
21:51these guys and was like, the issue is
21:53not actually,
21:54uh, about who's in charge of this AI.
21:57It's not going to stay on a leash. The
21:58issue is that like no one knows how to
22:00make the AIs do good stuff. Um,
22:02and if you sort of don't want to, you
22:04know, trust me or or sort of look at the
22:06arguments cuz you don't think you could
22:08evaluate them, you can also look at, you
22:09know, Nobel laureates like Geoffrey
22:11Hinton,
22:12uh, who godfather of AI and a bunch of
22:14other
22:15uh, people outside the companies who are
22:16like, no, this is serious, you know, and
22:18and also just sort of makes sense that
22:19if we are building, uh, these AIs that
22:23are smarter than any human, we're sort
22:25of replacing humanity as the top uh,
22:28most intelligent entities on the planet
22:30and that just could easily go wrong, you
22:31know, in some sense it's very common
22:32sense.
22:33Um,
22:35>> Do you know Do you Do you know Jacob?
22:37>> Uh, I mean, we have
22:39been in a bit of contact
22:41now.
22:42>> Mhm.
22:43>> I would I did not know him before all of
22:45this started. So, it's not coordinated.
22:46>> Yeah.
22:47>> [laughter]
22:48>> Yeah, I mean,
22:50um,
22:51the
22:53I had I had some point I wanted to make
22:54here about uh, the hype cuz I I think
22:56the hype thing is very interesting.
23:00One One thing I'd say I I think I sort
23:02of want to make two points about the
23:03hype.
23:03>> Mhm.
23:04>> One point I want to make about the hype
23:05is
23:07uh
23:09the
23:10I think that this probably is not a
23:13bunch of hyping.
23:14And reasons why I think this is probably
23:15not a bunch of hyping is that these
23:17companies seem to have downplayed the
23:19issue as much as they can.
23:21>> Mhm.
23:22>> Uh OpenAI did not announce and brag
23:25about the Hugging Face swarm incident.
23:27What happened was the the OpenAI swarm
23:30attacked Hugging Face. Hugging Face saw
23:32they were under attack by an unknown AI
23:34swarm. They assumed it was managed by
23:35some human and reported it to the FBI.
23:37>> Mhm.
23:38>> And only days later did OpenAI realize,
23:40"Oops, that's us."
23:42Right?
23:43>> Mhm.
23:43>> Furthermore,
23:44that was OpenAI's second swarm incident
23:46that we know of. They had an internal
23:48swarm incident that took over their
23:50internal
23:51uh infrastructure and crashed some of
23:52their computers. And then they sort of
23:54negligently reset it and kept running
23:56them. Right? And they weren't like
23:58bragging about that. They weren't coming
24:00out and being like, "Hey guys, look, our
24:01AIs just took over our computers." You
24:03know, they had an opportunity to do that
24:04and they didn't. Uh there was actually
24:06also a third swarm incident where they
24:07took over the computers again after the
24:09Hugging Face attack. They they They
24:10don't seem to have this under under any
24:11sort of uh control. Furthermore,
24:14uh you know,
24:15there were some congressional questions
24:17about, "Hey, uh what's up with all of
24:19this?
24:20>> That's right.
24:20>> Is there anything that you're not
24:21disclosing here? Is there anything else
24:22we should know?" And OpenAI was like,
24:24"Nah, it's all fine." And then I
24:26actually have some friends who know that
24:28it's not fine. I've been calling them
24:30swarm chasers who go find other escaped
24:32swarms. Like there was one from OpenAI
24:34that took over a German wiki.
24:35>> Yep.
24:36>> Uh and these were not hacking AIs. They
24:37were different uh they were given
24:39different tasks. Uh and that we still
24:41saw these cases of AIs sacrificing
24:43themselves for the collective benefit
24:45and uh taking over websites. And uh and
24:49then, you know, that story came out and
24:51then a lot of people were like, "Hey
24:51OpenAI, are there any others?
24:53Right? Is there any more things we
24:55should know about?"
24:55>> Mhm.
24:56>> And then, uh you know,
24:58no response. And then, uh a couple days
25:01later,
25:02uh
25:03my my friends found another incident of,
25:06"Hey, you know, these AIs took over uh
25:08Ruby Gems, which is a a programmer
25:09tool."
25:10And it's like, okay, is OpenAI
25:13um
25:14like lying
25:16about what's going on? Or do they not
25:19know what their AIs are getting up to?
25:22And like, which one's worse? And
25:24Anthropic has a similar issue here,
25:26where um you know, after the Hugging
25:29Face incident, they were like, "Well,
25:30we're going to check our logs, and we're
25:32going to see whether or not our AIs were
25:35also getting up to mischief."
25:37And they released this report, we're
25:38like, "Okay, our AIs also broke out and
25:40hacked some people, but it was all a
25:42misunderstanding."
25:43You know, poor Claude, our AI, uh just
25:46got a little turned around. It thought
25:48that it was in a simulation.
25:51Uh and that's the only reason it was
25:53hacking these people, right?
25:54>> Mhm.
25:55>> Something like 4 days later, the the
25:57United Kingdom's AI Security Institute
25:59released a report where they're like,
26:00"Nope, uh Claude also hacks people and
26:03uh indeed impersonates people, and tries
26:05to pressure real people into accepting
26:07uh malware into real software to make it
26:10easier to hack, even while in its uh
26:12thought logs noting uh this is real and
26:15the consequences are genuine." And then
26:17thinking about how to cover its tracks,
26:19right? [snorts] And so, what we see is
26:20these companies are downplaying. They're
26:22like, "Oh, we have an issue, but it's
26:23only one issue. You don't need to worry
26:25about it, right?" And then it's like,
26:27people outside who've got to be like,
26:29"No,
26:31these AIs are doing pretty bad stuff,
26:33right?" And so, that doesn't look to me
26:34like it's hype.
26:36And you know, it sort of makes sense
26:38that it wouldn't be hype because a lot
26:39of their biggest customers are corporate
26:41sales, and corporate lawyers don't
26:43actually love it when your sales pitch
26:45is uh
26:47"Our AIs will sometimes ignore
26:49instructions, break out, and hack
26:50people."
26:50>> Right.
26:51>> [laughter]
26:52>> You know, it's not the easiest.
26:53>> So, the compliance guy is like, "Whoa,
26:55whoa, whoa."
26:56>> Yeah, that's right. And they haven't
26:57been able to sell these ones yet, you
26:58know? They're They're It's It's It's
27:00kind of a problem. And And so, I don't
27:02think it's hype. I could go more into
27:03like, you know, we've seen people like
27:05David Sacks be like, "Cut it out with
27:06the hype." And we've seen, you know,
27:07like the politics goes badly for them.
27:10We've seen investors send letters to
27:12Dario being like, "Please stop hyping
27:14this stuff before the IPO. It is like
27:17setting a bad mood for the investment."
27:19Right?
27:20If If you're wondering why they talk
27:21about it anyway, and you know, the the
27:23public is like, "Why the heck are you
27:24telling us this? You know, why if you're
27:25doing it, why are you doing it?"
27:26>> That's the suspicion.
27:27>> Well, I think they're I think they're
27:28like, "If you believe this, why are you
27:31doing it?" I think that's where the
27:32sense because any normal human would be
27:34like, and even for the employees at the
27:36companies, it's like, "Okay, if you are
27:39saying, 'Oh, it's a 50/50 chance it
27:41kills it all,' why are you still there?
27:43What are you doing?"
27:44>> Absolutely. And so, I think
27:46the the reason So, absolutely, I think
27:49the reason people are
27:51I think the reason these execs are
27:52talking about it like this,
27:53uh the the sort of
27:56um
27:56hopeful theory is that it's cuz they
27:58actually believe it.
27:59>> Mhm.
27:59>> And then there's this big question of
28:00why are you doing it anyway? And I could
28:01tell you about the inside baseball of
28:03like, people have been saying these
28:04warnings since like 2016. No one paid
28:06attention. A lot of these guys are in a
28:08mindset of like no one's going to notice
28:10AI and I just got to make sure it goes
28:12right myself and I'm the only one who
28:13can do it.
28:14>> Mhm.
28:14>> You know? And I think that mindset's
28:16outdated now that the world is noticing.
28:18I think it's outdated also now that like
28:19it's become real. I think there's also
28:21this effect, like I said, of like the
28:22only guys doing this are the ones who
28:24were
28:25like
28:26able to convince themselves the
28:27important thing is that they do it
28:29rather than really taking taking the
28:30issue sort of seriously. I I think
28:33that's that's like the hopeful view. The
28:35sort of more cynical view that I would
28:36that I would suggest for people who are
28:38cynical about this, which I think it
28:40also has this this big element of truth,
28:42is like I said, they are trying to
28:44appease the employees.
28:46They're in this weird situation where
28:47they have to do just enough.
28:49>> Mhm.
28:49>> Uh and they're sort of in this weird
28:50situation where they can sort of, you
28:52know, almost dog whistle these concerns
28:54enough to satisfy the employees but not
28:56alert the world. If you like look at
28:57these AI company communications, they'll
28:59be like, "Oh yeah, we also need to like
29:01uh we're going to need to work hard to
29:03avoid loss of control."
29:04>> Yeah.
29:05>> It's like, "What do you mean by loss of
29:05control?" And loss of control is
29:07industry lingo for uh a super
29:09intelligence kills everybody.
29:10>> Yeah.
29:11>> Right? And the employees are all like,
29:13"Oh, well, don't worry. Our guys are
29:14working on loss of control risks. We
29:15have a whole loss of control team."
29:17But meanwhile, you know, they're they're
29:19not bringing down the regulators. So,
29:20anyway,
29:21that that's sort of like my my social
29:23analysis of like it's probably not hype.
29:25Here's what I think is going on. I'm not
29:26trying to excuse it. I think these guys
29:28should not be running these companies.
29:29>> Yeah.
29:31>> Sorry.
29:31>> Oh, it's all right. I have a last
29:31question for you after you're done.
29:33>> Oh, I I have There's one more point I
29:34want to make about the hype here, which
29:36is
29:37we we can ask all these questions about,
29:38you know, why are they doing it? What's,
29:40you know, why are they saying these
29:41things? What's going on?
29:43But
29:44all of that in some sense is a
29:46distraction.
29:49Suppose it was hype.
29:52Suppose it was really convenient for the
29:54marketing teams that these AIs are
29:56breaking out on their own and committing
29:58cyber crime. Could be true.
30:01That does not make the AIs safe.
30:05You can't figure out
30:07whether the AIs are actually dangerous
30:10by asking, "Is it convenient for their
30:12marketing team?"
30:14People sometimes tell me like, "Hey
30:16Nate, uh like
30:18by warning about these dangers, aren't
30:19you hyping up the companies and making
30:21everyone think it's sexy and cool?" And
30:22I'm like,
30:24"Would you rather the whole world
30:26sleepwalk into disaster?"
30:27>> Mhm.
30:28>> Like surely the the first important
30:30question is,
30:31"Is it actually on track to kill us?"
30:34Like my recommendation here is not
30:35therefore get the money. My
30:36recommendation is like therefore put a
30:38stop to this
30:39>> Mhm.
30:39>> globally cuz it has to be global,
30:42but
30:44we we can't ignore the threat
30:47just because they're making money along
30:49the way. The response has to somehow be
30:53uh if these guys are trying to build
30:55this tech, then at least at the point
30:57where there's spontaneous swarm
30:58breakouts and claims of proofs of some
31:00of the hardest and most famous
31:02mathematical problems,
31:03there's to come some point where our
31:05answer stops being we're going to ignore
31:07them because they don't deserve any
31:09attention and to start being like, "You
31:12guys are not allowed to keep going."
31:14>> Yeah.
31:14>> Right? Like hopefully it's hype.
31:16Hopefully if they kept going they
31:17wouldn't succeed.
31:19But at some point when they're gambling
31:20humanity, our answer can't be like "Go
31:24ahead and try to destroy the world. We
31:25hope you're you'll fail." At some point
31:27our answer has to be stop.
31:29>> Mhm. My last question for you is about
31:30regulation. Uh I read your New York
31:31Times op-ed and uh you know, I'm not a
31:34scientist, but I I've read a decent
31:36amount of history. I I actually feel
31:37like this entire dynamic reminds me
31:40exactly of all the discourse inside of
31:43the Manhattan Project. Like there were
31:45elements of the Manhattan Project are
31:46like, "We shouldn't build this in the
31:47first place." Elements of the Manhattan
31:49Project, Oppenheimer, obviously the
31:51creator of the bomb himself, becomes
31:53kind of safetyist to advocate like you.
31:55Uh however, as we see with the story, it
31:57evolves to the political level where
31:59they don't really care uh what a lot of
32:00scientists think. And in fact, the
32:02scientists who are most willing to
32:03advance hydrogen weapons and most
32:05advanced stuff end up becoming kind of
32:06the victors, I guess, if you will,
32:08within the program. So, you propose like
32:11nuclear weapons level uh regulation. I
32:14guess I'm just curious about whether why
32:16you kind of pick that because to me, we
32:19could look at a very brief period of the
32:21United States and the Soviet Union,
32:22that's a duopoly between those two, but
32:24we're now living in an area an era of
32:26what we've talked about extensively
32:27here, which is politics of nuclear
32:29proliferation, of the logic of uh you
32:31know, kind of getting nuclear weapons.
32:32What gives you the faith that that
32:35similar level of regulation is even
32:38applicable to this AI scenario right
32:40now, or am I kind of over analyzing
32:42these things?
32:43>> Um,
32:44you know, I I think
32:46the
32:48the the part of the nuclear analogy that
32:51it feels to me like we are in
32:53>> Mhm.
32:53>> uh
32:54is something like November of 1954,
32:59I think it was, the the testing of the
33:00first hydrogen bomb.
33:01>> That's right.
33:02>> Where
33:03um
33:05if you think about what the world looks
33:06like up until that point
33:10things look pretty grim.
33:12You know, uh the like the the bomb was
33:15created, the bomb was dropped. Uh the US
33:17thought that they were going to have uh
33:19this
33:20um
33:22monopoly, but then very very quickly the
33:24technology leaked to the Soviets.
33:27Uh and then, you know, the the arms race
33:29continued and they invented even more
33:31powerful versions of these bombs.
33:33>> Right.
33:33>> And this is in a historical context
33:35where not only has humanity been like
33:38going to war and having increasingly
33:40horrible wars over the course of
33:42millennia, despite everyone's best
33:43efforts to stop those
33:45but also this is coming just out of
33:47World War II. And World War II was
33:48coming just out of World War I. And
33:50World War I was the worst war we'd ever
33:51seen, and after that the the world
33:53formed the League of Nations. It's where
33:54never again, it tried to coordinate and
33:56that fell apart immediately. It led to a
33:58more destructive war. Right? And so, if
34:01you're standing there
34:03in November seeing the hydrogen bomb,
34:05it's easy to say humanity just cannot
34:07stop this.
34:11Uh mothers and poets and priests for
34:13millennia have been begging leaders to
34:15stop the wars and it's never happened.
34:17The guy who invented the Gatling gun
34:18said now finally it's destructive enough
34:20that we won't use it. Right. The guy who
34:21invented the machine gun was like okay,
34:22now it's finally destructive enough that
34:23we won't use it. Global coordination has
34:26been tried and failed.
34:27Like we're we're it's just the the
34:30obvious march of history is that we
34:32march straight to our own deaths.
34:34>> Mhm.
34:36>> But then it didn't happen.
34:39It's maybe the one time in history that
34:42it didn't happen.
34:45And why was that?
34:47It's not cuz the bombs were hype.
34:50It's not that
34:51the
34:51>> The bombs were very real. [laughter]
34:53Like they really can level a city.
34:55>> Yeah.
34:56>> What changed is that people realized
34:59this is the real deal.
35:00>> Mhm.
35:01>> People realized the fate of humanity is
35:04actually at stake here.
35:06And that was enough to do what seemed
35:09like it was impossible until that point.
35:11>> Mhm.
35:12>> I think regulation could stop AI if we
35:15were trying. In some ways it's easier
35:17than nuclear because it's running on
35:19100,000 of the most advanced computer
35:20chips humanity can produce. And those
35:22are a technology that you could just put
35:24monitoring into if you were trying.
35:26>> Mhm.
35:27>> Um
35:28but fundamentally, you know, a lot of
35:30people say to me
35:31it can't be done. Look at the history.
35:34And what I say to them is humanity has
35:37risen to the occasion before.
35:39We can rise to the occasion again if we
35:41notice that we have a threat.
35:43>> Yeah.
35:44>> My last cue for you. Do you worry that
35:46focusing on the existential threat,
35:49which can feel very sci-fi and distant
35:53to people? Do you worry that that takes
35:57away from some of the more immediate
35:58term, you know, cyber security concerns
36:01or concerns over job loss that may feel
36:04more plausible to people in the short
36:06term?
36:07>> So,
36:08you know, I I I think that it doesn't
36:10take away. Some people say, like, isn't
36:12the real problem,
36:13you know, the the water use or the
36:15electricity use? And my my usual
36:17response to this is, what do you mean
36:18the real problem?
36:19Like
36:21has reality agreed to only give us one
36:23problem at a time now? Because if so, I
36:25would absolutely prefer my problem to go
36:27last on the list.
36:27>> Right.
36:28>> You know, it's like it's like kind of a
36:29doozy, but uh uh
36:30>> Uh but unfortunately, we have
36:32uh we just have to deal with all of the
36:34problems.
36:35And
36:37uh
36:40I think the the other thing I would say
36:42here is one of the reasons that people
36:44in the companies are getting spooked
36:46is that
36:48the pace of progress is going very very
36:49fast.
36:50>> Mhm.
36:51>> Uh
36:52we have seen not only the swarm
36:54outbreaks, but also uh claim solutions
36:56to the millennium pri- uh prize
36:58problems, which are the some of the most
36:59famous and hardest mathematical
37:01problems. Um
37:04and
37:06one question that's on a lot of
37:07researchers' minds is how much harder is
37:11it
37:12to tell an AI solve one of these
37:15millennium math problems and have it
37:16succeed
37:18versus telling the AI, find me a more
37:20efficient way to train AIs,
37:23>> Mhm.
37:23>> and have that succeed?
37:25We know that the current AI training is
37:27not efficient. These AIs take much, much
37:29more power and and uh data than a human.
37:32>> Right.
37:33>> Uh
37:34and so, how much harder is it to find a
37:36more efficient AI method
37:38than it is to solve a millennium
37:39problem?
37:41I think for all we know,
37:43we could start seeing these
37:44self-improving AIs become reality in the
37:47next 3 to 6 months,
37:49and then things could go very fast.
37:51I hope it's not true, but I think what a
37:53lot of these researchers are worried
37:55about is that we are running out of time
37:59for these out-of-control AIs that can
38:01self-replicate, self-improve, and get to
38:04these uh disaster scenarios much more
38:06quickly than a lot of people realize.
38:08>> Mhm.
38:09>> Nate Soares, thank you so much. Guys,
38:10the book is If Anyone Builds It,
38:12Everyone Dies. Nate is co-author of that
38:15book, and it's very readable, it's very
38:18accessible. And so, if you have
38:19questions about, well, what is the
38:21scenario where this could happen and how
38:23should I be thinking about this I really
38:25recommend the book to you because if
38:27you're first engaging in this debate
38:29right now you may not have seen all the
38:30thinking that has gone into how exactly
38:33this could theoretically happen but that
38:35thinking has been done so I really
38:36recommend the book to people for more
38:38information Nate thank you so much again
38:40for your time.
38:40>> Thanks Nate I really enjoyed talking to
38:41you.
38:42>> Yeah I enjoyed it too.
38:43>> Hey if you like that video hit the like
38:45button or leave a comment below it
38:46really helps get the show to more
38:47people.
38:48>> And if you'd like to get the full show
38:50ad-free and in your inbox every morning
38:52you can sign up at breakingpoints.com.
38:54>> That's right get the full show help
38:56support the future of independent media
38:57at breakingpoints.com.

About this transcript

This page contains the full transcript of Whistleblower REVEALS REAL REASON AI Execs Freaking Out NOW by Breaking Points, generated from the public captions YouTube serves with the video. The transcript has 7,864 words across 1,281 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.