YT YouTube2Text

YouTube2Text → Transcripts → ChatGPT Whistleblower: They Offered Me $2m To Keep Quiet, No One Is Ready For What's Coming!

ChatGPT Whistleblower: They Offered Me $2m To Keep Quiet, No One Is Ready For What's Coming! — Transcript

by The Diary Of A CEO · 25,395 words · 3,771 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00The scary open secret in the AI industry
0:02right now is that it's possible that
0:03we'll end up essentially creating a new
0:05species that ends up ruling the world
0:07with a 70% chance that this goes
0:08horribly wrong like human extinction.
0:10That's one possibility. There's many
0:12more. It's quite chilling what you're
0:13saying.
0:13>> Yeah, it's uh it gets me down sometimes.
0:18I basically told my wife like, let's not
0:20have any more kids. It's too uncertain.
0:21I don't think they'll ever join the
0:22workforce.
0:24Everybody should be afraid that their
0:25jobs are going to be lost. And I know
0:27this because I went to open air in 2022.
0:28What I did there was forecasting as to
0:30what the next couple years might look
0:31like. And unfortunately, most of the
0:33world is kind of asleep at the wheel and
0:34doesn't really realize what's going on
0:35with AI. So, I resigned.
0:37>> I read it somewhere that you lost $2
0:39million for not signing an
0:41anti-disparagement clause, meaning you
0:43couldn't criticize the company.
0:44>> Yes. For reasons I'm happy to get into.
0:46But the main thing I've learned is when
0:47I go talk to people at Anthropic and
0:48OpenAI about forecasting, they're like,
0:50"It's not going to take that long. You
0:51need to shorten them again. get them
0:52back to 2027 or 228 because these
0:54powerful CEOs, Ariel or Sam or Elon are
0:57racing each other to be in control of
0:59the most powerful AIs and are literally
1:01afraid that if the other guy gets there
1:03first, he might become dictator. I mean,
1:04Anthropic is on track to be the entire
1:07economy by 2030. But none of these
1:09people should be trusted with that much
1:10power. So, this is the most important
1:12thing happening in our lifetimes,
1:13probably in all of history, in fact. And
1:15it's very important that it go well. So,
1:17I think that there's a lot we can do to
1:18like steer things in a better direction.
1:19There's loads of benefits that we could
1:21get from AI if we do it right. And if we
1:22do solve the problems, then things could
1:24be absolutely amazing for everyone.
1:26>> Well, this report here in 2021, it was
1:28remarkably [music] accurate. And then
1:29you just published this one.
1:31>> Yeah. So, this is our new scenario.
1:32>> So, let's go through these slowly and
1:33one at a time.
1:34>> I would be incredibly happy if all my
1:36predictions turn out to be wrong.
1:40>> This is super interesting to me. My team
1:41given me this report to show me how many
1:43of you that watch this show subscribe.
1:44And some of you have told us according
1:46to this that you are unsubscribed from
1:48the channel randomly. So, favor to ask
1:50all of you. Please could you check right
1:51now if you've hit the subscribe button
1:53if you are a regular viewer of the show
1:54and you like what we do here. We're
1:55approaching quite a significant landmark
1:57on this show in terms of a subscriber
1:59number. So, if there was one simple free
2:01thing that you could do to help us, my
2:03team, everyone here to keep this show
2:05free, to keep it improving year over
2:07year and week over week, it is just to
2:09hit that subscribe button and to double
2:10check if you've hit it. Only thing I'll
2:12ever ask of you, do we have a deal? If
2:14you do it, I'll tell you what I'll do.
2:15I'll make sure every single week, every
2:18single month, we fight harder and harder
2:19and harder and harder to bring you the
2:20guests and conversations that you want
2:22to hear. I've stayed true to that
2:23promise since the very beginning of the
2:24D of Coio, and I will not let you down.
2:27Please help us. Really appreciate it.
2:29Let's get on with the show. [music]
2:34>> Daniel Cocatello,
2:36at the very heart of what you do, um,
2:38what is your mission and why? So what
2:42would you do if you thought that super
2:43intelligence was coming in a few years?
2:46>> I guess it depends what the consequences
2:50were. Well, let's talk about it. So
2:52super intelligence, AIs that are better
2:54than the best humans at everything while
2:56also being faster and cheaper. Also able
2:58to operate robots that can do everything
3:00in the physical world that humans can
3:02do, but better, faster, and cheaper. If
3:04that really is coming in a few years,
3:07then we need to prepare and we need to
3:09think about how to make it go well
3:10instead of poorly. So that's sort of my
3:12answer is like I'm doing that to the
3:14best of my ability.
3:14>> So you believe it's coming in a few
3:16years?
3:16>> Yes.
3:17>> How could you be so sure?
3:19>> I spend a lot of time trying to forecast
3:20this sort of thing. My sort of median
3:22estimate 50% chance is currently in
3:252029. Maybe it'll slip to 28 28. It's
3:28possible that it'll take significantly
3:29longer, like maybe 10 years or something
3:31like that, but uh you know, for reasons
3:35I'm happy to get into, seems to me like
3:37it's probably happening by the end of
3:38the decade. What's less important is the
3:41the sense of how close we are. What's
3:43more important is the pace of the
3:45trends. Anthropic
3:48this time last year was making something
3:50like a billion dollars a year and now
3:53they're making something like $60
3:54billion a year. So that's 60x growth in
3:58one year, which is extremely impressive
4:01even for very small startups. But for a
4:03company of their size, it might be the
4:05fastest growth in history. Um, we expect
4:09that rate of growth to slow down, but
4:12even if it slows down quite a lot,
4:15they're still on track to be, you know,
4:18the entire economy by 2030 or so.
4:21>> Why should the average person care? The
4:22high level thing is absolutely
4:24everything is going to change for the
4:25whole world and including therefore for
4:27them and their families. Um could change
4:29for the better, could change for the
4:30worse depending on the details of how
4:32it's done. So for example, everyone
4:34could die you know um this is the
4:36classic loss of control scenario or one
4:39version of it. If we do build these
4:41super intelligences and we use them to
4:44automate all the jobs and we put them in
4:47the military and we, you know, have them
4:49giving advice to politicians and so
4:50forth, they will eventually have
4:52accumulated enough real world power that
4:55they don't need humans anymore and
4:57they're smarter than us, they're more
4:58strategic, etc. At that point, we sort
5:00of have to hope that they are virtuous,
5:02that they have, you know, the goals that
5:04we wanted them to have, the values that
5:05we wanted them to have, etc. And the
5:08sort of scary open secret in the AI
5:11industry right now is that right now
5:12that is kind of just a hope. It's not
5:14something that we can be at all
5:16confident in. And in fact there's lots
5:17of evidence and arguments that it we're
5:19not on track to achieve that. So there's
5:21lots of reason like current AIS for
5:22example will often lie uh to people or
5:25they will like you tell them to do
5:27something and they go do something else
5:28and then pretend that they did it right.
5:30So, it's an inherently difficult problem
5:32to make something that's super
5:33intelligent and also has the values and
5:36virtues that you want it to have. And it
5:38doesn't seem like we're on track to
5:39solve that problem. Also, it seems like
5:41the sort of problem that you could think
5:42you've solved when you haven't actually
5:44solved it, right? Uh that's a big reason
5:46why this is scary. So, for all those
5:48reasons, it's possible that we'll end up
5:50essentially creating a new species that
5:53ends up ruling the world instead of us.
5:55And then maybe we go the way of other
5:57extinct species in the past that were
5:58out competed by humans. That's one
6:00possibility. There's many more. Even if
6:02you're not worried about that and you
6:03think that the AI will be totally
6:05controlled, there's the question of who
6:07controls the AIS, right? When there's a
6:10couple corporations that have made these
6:11super intelligences and are using them
6:13to automate all the jobs. Well, that's a
6:16lot of power. You know, that's a lot of
6:17money. It's a lot of political power.
6:19They'll have the best strategists, the
6:20best advisers, you know, they'll think
6:23faster militarily. uh the countries that
6:26has these AIs will be able to absolutely
6:28wipe the floor with all the other
6:30countries. The AIS themselves it's it's
6:32kind of a single point of failure like
6:33central uh control system where you know
6:37the CEO of Enthropic Dario he coined
6:40this phrase the country of geniuses in
6:42the data data center that was his phrase
6:45to describe what they're trying to
6:46build. You know I think that's a little
6:48bit misleading. I think it would be more
6:50accurate to describe it as army of
6:51geniuses in the data center because it's
6:53not like it's a bunch of diverse
6:55different AIs, you know, living in their
6:57different parts of the data center.
6:58They're all copies of the same big model
7:01and they're owned by the company and so
7:04they all follow the orders given by the
7:06company, right? People should be asking
7:07questions of like who controls this army
7:09or these armies and what are they going
7:11to be doing with them? I think that we
7:12could very easily end up in a sort of
7:16uh a situation where some tiny group of
7:18people are essentially oligarchs or
7:20dictators. And ironically,
7:24both of these risks, the loss of control
7:26and the constitution of power are things
7:28that people in the industry have been
7:30thinking about for decades. Um even
7:32before the AI industry existed, you
7:34know, people thinking about AI were
7:36talking and writing about these things.
7:38And then part of the founding narrative
7:39or the founding myth of deep mind and
7:42open AAI and anthropic is these problems
7:44are real. So we need to get there first
7:48so that we can handle it responsibly.
7:50Those are I think the big two reasons.
7:52But then I can go on there's lots more
7:53reasons as well. So one thing is you
7:56know World War II geopolitical conflict.
7:58Um if AI does in fact get incredibly
8:01powerful that's going to change the
8:02balance of power between nations. That's
8:04going to disrupt a lot of things. That
8:06puts us at increased risk of crisis more
8:08generally, right? Another one, what
8:10about those jobs?
8:12You you're going to lose your taxi job,
8:14but not just the taxi driver, everybody
8:15pretty much. Um there might be a few
8:17exceptions like people whose jobs for
8:20legal reasons are only allowed to be
8:21done by humans, but for the most part,
8:24everybody should be afraid that their
8:25jobs are going to be lost even if we
8:26manage to avoid all the other problems.
8:28Right? this narrative has started to
8:30emerge and I've had several interviews
8:32on this show where I've interviewed
8:33people who are very very scared and
8:34anxious about AI and these are people
8:36that have worked in the industry for
8:37sometimes decades. Yeah.
8:38>> Um the counternarrative coming over the
8:40hill is that this is doomerism that
8:43these people are for whatever reason
8:44just trying to scare people and that
8:46they don't really understand what
8:47they're talking about. How do you
8:48respond to that sort of
8:49counternarrative? And you must have seen
8:51this emerging yourself especially from
8:53people who stand to benefit dare I say.
8:55>> Yeah, exactly. this counter narrative is
8:58fairly recent and it's been pushed by
8:59the people who stand to benefit um from
9:02it and it's not true. Like these these
9:05concerns have been around for decades
9:06since before the AI industry existed.
9:08They're actually pretty reasonable
9:09concerns. Like if you take the companies
9:11at their word and imagine that they are
9:12in fact going to build super
9:13intelligence. Well, it raises a lot of
9:16questions like who's going to control
9:17it? Will anybody control it? What about
9:19the jobs? You know, like these are just
9:21kind of obvious implications to be
9:23thinking about and worrying about. Who
9:24are you and what's your story?
9:26>> My name is Dan Coatello. Um, I currently
9:30run the AI Futures Project, which is a
9:32small nonprofit that mostly focuses on
9:35forecasting the future of AI. Before
9:38that, I worked at OpenAI.
9:40>> AI forecasting.
9:42>> Yeah. So think about how like you know
9:46industry analysts who work for hedge
9:48funds and stuff will make these
9:49forecasts of like here's you know how
9:51many cars Tesla will be selling 5 years
9:54from now or like here's what the price
9:56of electricity will be in 2 years right
9:58that's forecasting I was doing that but
10:01specifically focused on AI the reason I
10:03was doing it is because it's incredibly
10:04important to to see where this is all
10:06headed
10:07>> why did you go to open AI what did you
10:09do there what did you observe while you
10:11were there and how did Did it change
10:12your perspective on the future of AI but
10:16also I guess open AI as a company and
10:18for anybody that doesn't know OpenAI are
10:19the company that produced chatbt?
10:21>> Yeah, so I went to OpenAI in 2022. Uh a
10:24large part of what I did there was more
10:25forecasting. AI 2027 is a scenario that
10:28you may have heard of. I did like
10:29smaller, you know, lower effort versions
10:32of them internally for just internal
10:34circulation of like here's some guesses
10:36as to what the next couple years might
10:37look like. I also worked on evaluations
10:40for dangerous capabilities. So, you
10:42know, trying to measure the AI's cyber
10:44abilities or persuasion abilities or
10:46situational awareness. And I also
10:49briefly was on a uh a capabilities team
10:52doing reinforcement learning to create
10:54agents. AI is in fact getting uh a lot
10:57better and I can say more about why you
10:59know scaling laws um deep neural nets
11:02bigger trained on more data become more
11:04efficient, more competent at those
11:05things. I also
11:09became a bit more disillusioned with the
11:11AI industry. So, OpenAI, Anthropic, and
11:15Deep Mind all had these sort of founding
11:16narratives of like, yes, these risks are
11:18real, but we've thought about them and
11:20we're going to try to handle them
11:21responsibly. And that's why it's
11:23important for us to keep doing what
11:25we're doing. And I increasingly came to
11:28think that these were rationalizations
11:31to justify what they were doing rather
11:33than sort of like deeply guiding their
11:35actual behavior and that when push comes
11:37to shove, they'll follow their
11:38incentives rather than
11:41do what's actually good.
11:43>> So you're inside Open AI at the time and
11:45you start to believe that they're
11:47following commercial incentives versus
11:49the I guess social in or societal
11:52incentives that they founded themselves
11:53on.
11:53>> Sort of. I mean, I wouldn't actually
11:55describe it as commercial incentives. I
11:56think I would describe it as um
12:00power seeking incentives. So,
12:02[clears throat]
12:02>> like it's true that the companies care a
12:04lot about making a lot of money, but
12:07especially at the very top of these
12:08companies, like the leaders,
12:11they understand that this is about more
12:12than just money. You know, there are
12:14these emails that came up in, you know,
12:16the the lawsuit between Musk and and um
12:19OpenAI. A bunch of emails were surfaced
12:21in that lawsuit, which you can go read.
12:23And in some of them, the founders of
12:26OpenAI were talking back in like 2017
12:28about how the reason why we made OpenAI
12:30was because we were worried that Demos
12:34at Google was going to become dictator
12:35with AGI. Even back then, they were this
12:37is obviously about more than just money.
12:39Like these these powerful CEOs are
12:42literally afraid that if the other guy
12:45gets there first, he might become
12:46dictator. And they don't trust each
12:48other. And so that's why they are racing
12:51as hard as they can so that they're the
12:53ones who get there first, so to speak.
12:55>> Have you met Sam Alman?
12:57>> Yeah.
12:58>> And did did that shape your opinion of
13:00his incentives or what why he's doing
13:01what he's doing? Cuz there's a lot, you
13:02know, speculated about what his
13:04incentives are. I mean, his most recent
13:06narrative says uh for the good of
13:08humanity. I think that's what
13:09>> Yeah. I mean, I think the main thing
13:10I've learned is don't pay attention to
13:11the narratives, you know, like uh what
13:14they say to one person is just different
13:15from what they can say to some other
13:17person at the same time. And what they
13:19say in public is a third thing entirely.
13:21I think you should judge people by their
13:22actions, not by their words.
13:25>> And why are you no longer at OpenAI?
13:27>> Largely for the reason that I mentioned.
13:28So I became gradually disillusioned with
13:30how the company was going to behave. For
13:32example, when I first joined in 2022, at
13:36least the people I talked to, my
13:37colleagues at the company, there was
13:39this general sense of like, of course,
13:41we wouldn't actually just build super
13:43intelligence as soon as possible. Once
13:45we started getting really close, like
13:46once we started getting to AIS that
13:48could maybe automate the AI research
13:50process, we would pause and figure out
13:53how to make it safe.
13:55That's cuz we're the good guys. And
13:56that's obviously the safe thing you
13:57should do rather than just going full
13:59speed ahead. But we're worried about
14:01other people who might not pause. You
14:03know, our competitors, Google for
14:04example. And so that's why we need to be
14:07in the lead so that we have that room to
14:09do the safe stuff, right? That was sort
14:11of like a thing that seemed like maybe
14:14like the median position or something
14:15among the colleagues I talked to when I
14:17was there when I started, including
14:19people like Sam, you know, including the
14:20leadership. And then by the time I left,
14:22I was like, "Oh man, they're really not
14:23going to do that, are they?" Like
14:24[laughter]
14:25like they they've sort of, you know,
14:27partly because this has become more
14:28politicized and they've become bigger
14:30and under more scrutiny, people have
14:32started asking like, "Why are you doing
14:33this in the first place if it's so
14:35risky?" And so they've pivoted their
14:36narrative to being more like actually
14:38it's not that risky, you know? Um, and
14:41so yeah, I mean it seems like they're
14:42just going to keep going roughly as fast
14:45as they can and hope that they can
14:46figure it out on the way.
14:47>> How did your time at OpenAI come to an
14:49end?
14:50>> Uh, I resigned in 2024. I had a nice
14:53goodbye party.
14:54>> What were the reasons you gave for
14:55quitting OpenAI?
14:56>> I thought that we were rationalizing too
14:58much and that we needed to think more
15:00about what would actually be good for
15:01the world. Um, I wanted more freedom to
15:04publish.
15:05So at OpenAI as it became a bigger
15:08company it became more of a normal tech
15:11company with incentives and you know a
15:13PR department and things like that and
15:15so it started becoming more difficult to
15:18um to publish the sort of research that
15:20I was doing for example those scenarios
15:22that I mentioned couldn't uh couldn't
15:24publish those right they're just for
15:25internal use I thought that that was a
15:28shame because right now most of the
15:31world is kind of asleep at the wheel and
15:32doesn't really realize what's going on
15:33with AI and doesn't really realize
15:35what's coming in the pipeline a couple
15:36years from now and the companies aren't
15:39really incentivized to tell people that
15:41much about it. I mean they say some
15:43vague stuff in a sort of hypy way but um
15:48you know well they didn't want me to
15:49publish the scenario for example laying
15:51out like here's how things might
15:53actually look.
15:54>> I'm just I'm super curious as to what
15:55it's like being in a company like that
15:56when they you know chat GBT3 is
15:59released. You were there at that time
16:00right?
16:01>> Mhm. um which was a moment where I think
16:03the whole world stood up and realized
16:04that this technology was powerful.
16:08>> Yeah.
16:08>> Um and the conversation really began
16:09from a society level. Um the company
16:13starts growing super quickly.
16:14>> Yeah.
16:14>> Quicker than I think anybody could ever
16:16have imagined. And what what was it like
16:19inside there? What did you see change um
16:21over over that period of time?
16:22>> I remember one all hands meeting where
16:24Ilia said something like
16:26>> Ilia being
16:26>> Ilia Sgver who was um head of research
16:29at that time. He said something like,
16:30"Okay, now the world is starting to pay
16:32attention. Each of you is going to be
16:33the most popular person at every party
16:36uh for the next year. Don't let it get
16:38to your head. Focus on the mission. Got
16:40to build AGI." [laughter] The company
16:42grew a lot. It already wasn't really
16:44feeling like a nonprofit when I joined,
16:46but it definitely didn't feel like a
16:48nonprofit by the time I left. Um, lots
16:50of new people came in. Ironically, the
16:53like amount of conversation about
16:57super intelligence and the implications
17:00of super intelligence arguably sort of
17:02went down over time due to this growth,
17:05right? So, because the company would
17:08like double and then double again and
17:09then double again, all these new people
17:11were coming in from other parts of the
17:13tech industry who hadn't really been
17:14thinking about these things and were
17:15attracted by the high salaries. You lost
17:17$2 million for not signing an
17:19anti-disparagement clause, which would
17:22mean you could speak, you couldn't
17:24criticize the company.
17:25>> Uh, yes. Well, so, um, I got to keep the
17:27money.
17:28>> Oh, you got to keep
17:28>> So, what happened was after I had left,
17:31said my goodbyes, etc., um, I got the
17:34the exit paperwork, and it included this
17:37clause that said you basically have to
17:38agree not to criticize the company
17:39again. Um, and also a clause saying, you
17:42can't tell anyone about this. And so I
17:45thought that was kind of rich coming
17:48from a nonprofit that's supposed to be,
17:50you know, for the benefit of all
17:51humanity. So I didn't sign it. And if
17:54you don't sign, you don't get to keep
17:56your equity. So your compensation, you
17:58know, what what they pay you is a bunch
18:00of money and then also a bunch of stock
18:03basically. But then they had this stuff
18:04in the contract that they get to yank
18:07back your your stock if you don't sign
18:10this thing. Um, and my wife and I, you
18:14know, were uh upset about this. We
18:16talked about it for like a month or two,
18:18consulted some lawyers, um, and then
18:20ultimately decided to just refuse to
18:21sign,
18:22>> which would mean you lost, you would
18:24have lost $2 million.
18:25>> That's right. Which was like 80% of our
18:27net worth at the time. Um, fortunately,
18:31uh, it didn't go the way we expected. It
18:34blew up basically on the internet. Like
18:36when people heard that that we had done
18:37this and that we had said no, it became
18:40like this huge scandal. Employees at the
18:42company started like asking questions in
18:44Slack and like asking leadership like,
18:46"Wait, what? Like why are you going to
18:48take away our equity? What is this?" You
18:49know, cuz a lot of people hadn't really
18:50noticed this before. It had been
18:52whispered about, but it hadn't been sort
18:54of like a thing that most employees knew
18:56about. Um, and so they backtracked and
18:58they said, "Never mind. Never mind.
18:59We'll change the paperwork. You can keep
19:00the equity. It's fine." And Samman came
19:03out and said he was embarrassed that he
19:05didn't realize this was
19:06>> Yeah, he had no idea apparently.
19:08>> You don't believe him?
19:09>> No, I think he probably knew. And if he
19:11didn't know, then people close to him
19:13probably did, such as his head lawyer.
19:15>> Why did you decide not to take the $2
19:17million?
19:19>> I mean,
19:20>> most people would have. I think
19:22>> it's true. Most people would have and
19:23most people did. And you know, money is
19:26nice, but like it's not the only thing.
19:29You know, sometimes it's good to take a
19:30stand on principle. I I keep mentioning
19:33super intelligence. Perhaps I should say
19:34more about like the the sequence of
19:37events that the companies are planning
19:39to do. So, right now they're focusing on
19:42automating coding. They're taking their
19:44AIS, they're making them bigger, they're
19:46training them for longer, and they're
19:48especially focusing the training on
19:50getting them to be good at autonomously
19:52writing and editing code because uh that
19:56will help the companies go faster,
19:57right? Right? If they can automate the
19:59code, then they can do their own work
20:01better and faster and accelerate
20:03progress. The next step, which they've
20:05already begun, is to look at the rest of
20:08the research process as well. Coming up
20:10with ideas, um, analyzing experiments,
20:13communicating those results, all the
20:15other parts of of the research process,
20:17they're trying to figure out how to
20:18train AIs to be good at those as well,
20:20so that they can have AIs do the entire
20:22thing autonomously. When you say do the
20:25entire thing, what do you
20:26[clears throat] mean do the entire
20:27thing?
20:27>> So like Anthropic and OpenAI in
20:29particular are trying to automate
20:31themselves. Like they're trying to make
20:32it the case that um they don't really
20:35need human employees anymore. Uh they
20:37just have a giant army of AIS that's
20:40turnurning away doing all this
20:42autonomous research to make better AIs
20:44to train the new AIs, put them in charge
20:48so they can make even better AIs and so
20:49forth. And of course, not just not all
20:52just happening internally, but also like
20:53interfacing with the world, right? Like
20:55going out and talking to people,
20:56collecting the data, setting up the
20:57training environments, doing the
20:59business deals, and so forth. Like
21:00they're they're trying to automate all
21:02of that. The reason why they're doing
21:04this is because they're trying to get to
21:06a position where they have AIs that are
21:10superhuman at everything, super
21:12intelligence, and they're trying to get
21:14there before their competitors do.
21:16Needless to say, this is incredibly
21:17dangerous, I would say. You know, and in
21:20addition to being dangerous, it's a
21:22power grab, right? Like, if they
21:24actually succeed at this, then they'll
21:26be sitting on top of this army of
21:28superhuman AIs that will give them
21:31immense leverage over all sorts of other
21:33actors in the economy in so far as they
21:35can work out something with the
21:36president and, you know, integrate it
21:38into the military or whatever, then that
21:40would give the US immense hard power
21:42over all other countries, right?
21:44Obviously, nobody knows exactly when
21:46this is happening, but a very disquing
21:48thing has happened over the last year to
21:50me, which is that when we published
21:532027,
21:55people were generally of the opinion
21:57that my timelines were too short and
22:00that like probably it would take more
22:01than 2027 until we got to the sort of
22:05events that I was just mentioning. you
22:06know, recursive self-improvement, AI is
22:08automating the whole research process,
22:10super intelligence. These these types of
22:13milestones um they happen in 2027 in AI
22:172027,
22:18>> which is this research paper you
22:19published.
22:19>> That's right. It's it's a scenario
22:21forecast that sort of lays out like
22:23month by month a possible future
22:25trajectory. There was sort of like at
22:27the time that we started writing it was
22:29my best guess as to what would actually
22:30happen. Obviously, there's lots of
22:31uncertainty, but you know, I I I thought
22:33it's valuable to make a concrete guess
22:35just to sort of see what it might look
22:36like. And at the time we were writing
22:38this, a lot of my friends in the AI
22:40industry and in nonprofits and so forth
22:43that work on AI, a lot of people were
22:45saying like, "Yeah, that stuff's going
22:46to happen, but like it'll probably take
22:47a couple years longer than you think."
22:50And now
22:53it's more 50/50. Especially when I go
22:55talk to people at Anthropica and OpenAI,
22:58they're often like, "Yeah, no, 2027,
23:01that's basically what's going to happen,
23:03just like you wrote. Why did you why did
23:06you become why did you update your
23:07timelines?" Oh, yeah. Context for this
23:09is um after after writing 2027, I
23:13shifted my timelines to be a little bit
23:14more conservative. So, at the time that
23:16we published, my 50% mark was in 2028,
23:19not in 2027. And then after we published
23:22progress just seemed like it was going a
23:24bit slower and so I updated to 2030
23:27which is you know it still could happen
23:28sooner could happen later 2030. Um but
23:32now when I talk to people in the
23:34companies they're like it's not going to
23:35take that long. They're like oh you need
23:38to shorten them again like get them back
23:40to 2027 or 2028 you know. Um so that's a
23:43bit disquing. Um, again, don't know how
23:46long it's going to take, but this is the
23:48stated plans of the AI companies is to
23:50do this incredibly dangerous thing, and
23:51they think that they're just a few years
23:53away. So, you wrote this um report here,
23:55what 2026 looks like, and you wrote this
23:58in 2021,
24:00>> and it was remarkably accurate, helped
24:02make a name for yourself amongst um
24:05amongst everybody in AI. And I Which one
24:07was it that JD Vance, the vice
24:08president, read? I think it was this
24:10one, wasn't it? Yeah, this one. Um and
24:12then so then you published this one AI
24:142027 and this was published I believe in
24:162025.
24:17>> Uh yes that's right April.
24:18>> Yeah.
24:19>> What were you forecasting in here? What
24:21is what are the key things that you said
24:22in here for people that haven't read it?
24:24>> The high level version of it is they
24:26automate the coding then they automate
24:28the rest of the research process. Then
24:30the pace of progress accelerates
24:31dramatically. They get the super
24:33intelligence. They're working with the
24:34government who specifically the
24:36president the executive branch naturally
24:37wants to control this technology and
24:39otherwise wants to use it to beat China
24:41and integrate into military and so
24:42forth. By [snorts] this point it's sort
24:44of doing basically all the work itself.
24:47I mean it's it's super intelligent. So
24:49it's coming up with all these great
24:50ideas for how to integrate itself into
24:52everything and all these new
24:53technologies it's invented and so forth
24:55and uh because of the race dynamics and
24:57because of the profit motive they end up
24:58deploying it everywhere and it builds
25:00robot factories. They build more robots.
25:01to build more robot factories etc.
25:03transforms the world entirely and then
25:06at some point it has enough power it
25:09meaning the AIs have enough power that
25:10they don't have to pretend to to be
25:13aligned anymore right um then they stop
25:17listening to orders
25:19that's the race ending of AI27
25:24we also wrote a sort of different branch
25:26which is the slowdown ending which was
25:28intended to sort of illustrate the
25:30concentration of power issues um that I
25:33mentioned previously. So, what if
25:35hypothetically the alignment issues get
25:38sorted out sufficiently quickly? Like
25:39what if it turns out that like it's not
25:42too hard with two months of slowdown, we
25:44can figure out how to make the AIS
25:45robustly do what we want um and have the
25:48values that we want them to have. So,
25:49that's one possible branch. And in that
25:51branch, uh it looks pretty similar. You
25:54know, they take the jobs, beat China,
25:56etc. Um, but instead of the AIS
26:00ultimately killing everyone, they create
26:02this sort of amazing utopia. But the
26:04amazing utopia is whatever the people
26:07who control the AIS wanted it to be,
26:09right? And so that would be a very small
26:11group of people like the president, some
26:13CEOs, etc.
26:16>> There should be a button just down below
26:17here. And if it says subscribed, you're
26:19already subscribed. If it says
26:21subscriber, that means you're not yet.
26:23And if you're not subscribed, please
26:24could you do us a favor and hit that
26:26button? It helps the show more than you
26:27know. And according to the algorithm,
26:29you're someone that watches our show,
26:30but you haven't yet hit that button.
26:32Thank you so much.
26:33>> Is there any possibility, do you think,
26:35that we never get to this thing called
26:36AGI? And and how do we distinguish AGI
26:40from this term super intelligence?
26:41What's the difference?
26:42>> Yeah, so the difference is that AGI is a
26:44more vague uh and weak term. So super
26:47intelligence is a bit more precisely
26:48defined. It's better than the best
26:49humans at everything, faster and
26:51cheaper. Um, AGI is more like it stands
26:53for artificial general intelligence,
26:55which means AI is that can do things in
26:57general rather than like some specific
26:58task.
26:59>> Yeah.
26:59>> And so arguably we've already achieved
27:01AGI, right? If you use cloud code or
27:03something like that, it's like it can do
27:04a lot of stuff. It's it's almost kind of
27:06like a little employee that you can like
27:08have go do stuff. So it's it is quite
27:10general. It's not maximally general
27:12though. It can't do everything. Whereas
27:13super intelligence by definition can do
27:16all the things that a human can do but
27:17better.
27:17>> And how does this sort of overlap with
27:19robotics? Because obviously we're seeing
27:21this huge robotics boom at the moment.
27:23There are some real world things that
27:24humans can still do because these AIs
27:26are still stunk on my computer.
27:28>> The way people talk about this is that
27:29they basically just say we've achieved
27:31super intelligence for cognitive tasks.
27:33Then you can talk about like full super
27:36intelligence that can do this physical
27:37stuff.
27:38>> And are we going to get there? Are we
27:39going to get there with both?
27:40>> I think so. I mean again this is not
27:42something that we can be certain about.
27:43Um you asked like is it possible we'll
27:45never get there? Yes, it's possible
27:46we'll never get there. I don't think
27:48it's likely though. I think that there's
27:50nothing sort of like magical about the
27:52human brain. It's, you know, um it's
27:55just a bunch of neurons. It is possible
27:57for a digital system to do similar
28:00functions in the same way that like, you
28:03know, a plane can fly just like a bird.
28:06Not in the same way as a bird
28:07necessarily, like it doesn't have it's
28:09not flying in the same way that a bird
28:10flies, but it flies, you know.
28:12>> Um so, so it does seem like yeah, like
28:15>> seems possible.
28:16>> You've written all these, you know,
28:16these research reports. You're working
28:18on another one that will be released um
28:20likely on the 9th of July. You have
28:23worked inside OpenAI. You then quit
28:25OpenAI because you were concerned about
28:27what was going on there and about the
28:29future of the industry. You know more
28:30than I do. Are you optimistic about the
28:34future or pessimistic? Are we heading to
28:36a bad place if things don't change? Um
28:39based on everything that you know,
28:40>> I think we are headed to a bad place if
28:42things don't change. Um I'm not
28:44confident in that. I would say something
28:45like 70%, it's very very hard to predict
28:48of course, but yeah, it seems like the
28:49current default path is heading towards
28:51a very very scary place.
28:53>> How do you contend with that personally
28:54and emotionally?
28:55>> Um, it's rough. I mean, I think it it's
28:58the sort of thing that like
29:02gets me down
29:04on a regular basis, but also I've been
29:06dealing with this for so many years now
29:08that I've sort of gotten used to it, if
29:10that makes sense. Um,
29:15yeah. Yeah. I I'll put it this way. I
29:17would be incredibly happy if all my
29:19predictions turn out to be wrong and uh
29:22an AI hits the wall, for example.
29:23>> It gets you down on a regular basis.
29:26>> I used to be known as a pretty chipper
29:27and optimistic person. But um in 2020 my
29:32AI timelines predictions started
29:34collapsing due to GPT3 and the scaling
29:37laws papers and um the bioankers report
29:40which I can talk about if you're
29:41interested but basically some events
29:43happened in 2020 that convinced me that
29:45actually this stuff was like quite
29:47plausibly coming by the end of the
29:48decade and humanity is very obviously
29:51not ready for this you know in a whole
29:53bunch of different ways and so that's
29:55obviously very scary
29:56>> and that's an extremely scary world
29:58because of all things you've said but
29:59but again because of this recursive
30:00self-improvement where AIs can train
30:02themselves and at such point we're
30:04starting to lose hold of what's going on
30:06here. I mean the AIS are already
30:07training themselves to be clear. It's
30:08more like closing the entire research
30:11loop, right? Doing everything.
30:13>> Yeah. Like right right now a lot of the
30:14training data is generated by AIS. A lot
30:17of the reinforcement like the grading
30:20that happens out of positive and
30:21negative reinforcement is itself done by
30:23AI.
30:24>> Can you explain that in layman's terms
30:25for
30:26>> Yeah. So an important thing for
30:27everybody to understand is that modern
30:29AI systems are not software in the
30:32normal sense. I mean they are
30:33technically software but they're not
30:35lines of code you know it's not like
30:37some engineers at anthropic went and
30:40wrote lines of code that basically says
30:42like you know when the user asks for
30:44this type of thing then go do this type
30:47of thing for this many steps or
30:49whatever. There's nothing like that.
30:50Instead it's a neural net. You know
30:52>> what's that?
30:53>> Well think about how the brain is a
30:55bunch of neurons connected to each
30:56other.
30:57>> Yeah.
30:57>> That are firing um signals back and
30:59forth. The brain learns over time the
31:02types of patterns of firing that caused
31:06success that caused a dopamine rush or
31:09various other types of feedback get
31:10reinforced and fire more often. And the
31:13types of patterns that caused failure
31:15like touching a hot stove get
31:17anti-reinforced, get, you know, um,
31:20destroyed so that they fire less often.
31:22And as a result of all of that, you over
31:24the course of years learn to act in the
31:28world and you learn all sorts of skills
31:29and you learn world models. You learn
31:31like beliefs about the world and you can
31:33sort of like mentally simulate how it's
31:34going and stuff like that. So artificial
31:36neurallets are like that except
31:38artificial. So it's it starts off as a
31:40giant
31:42tangled spaghetti mess of randomly
31:45generated uh artificial connections
31:49called parameters. these days there
31:51might be something like 10 trillion
31:53parameters uh in the biggest AIS. So it
31:57starts off randomly generated. So, it's
31:59of course completely useless. Like if
32:01you give it some input, it'll just
32:02produce gibberish as an output. But then
32:04they train it. And they start with
32:08pre-training, which is where you give it
32:10a bunch of internet text, and you show
32:12it the first piece of text, and you put
32:14that in as the input, and then it gives
32:16a gibberish output, and then you
32:20positively or negatively reinforce it
32:21based on how accurate that output was at
32:23predicting the next piece of text. Um,
32:26so it's basically playing this game of
32:27like predict the next word.
32:29>> Isn't that how it happens with babies? I
32:31had a I think I had a neuroscientist
32:32tell me that babies have more neural
32:34connections um than adults. And yeah, it
32:37says yeah, toddlers have twice as many
32:39neural connections as adults. And they I
32:42guess they whittle down through
32:43reinforcement.
32:44>> Yep.
32:44>> We have more pathways when we're younger
32:46and just like the process of training an
32:48AI, we're trained down to like remove
32:50the ones that aren't useful and build up
32:52on the ones that are.
32:53>> Yeah. It's both pruning and
32:54strengthening. Okay, it seems like in
32:56humans it's actually more pruning than
32:57strengthening, but it's both. Uh, and in
33:00AI it's the same thing. It's both. So
33:02the first portion of training is where
33:03they train the AI to predict text, which
33:06is kind of like training it to read. Um,
33:08and it a similar thing does happen in
33:09humans. So basically the the random
33:12tangle gradually takes shape and
33:15gradually sort of coaleses into more
33:17useful circuitry that has stored lots of
33:20facts about the world and has stored
33:22lots of skills for how to you know
33:25process information and transform it and
33:27then produce predictions. That's just
33:30the first step. After they do the
33:31pre-training then they try to teach it
33:34more useful skills besides just
33:36predicting text. And so, you know, by
33:38the end of the process, they've thrown
33:40lots of coding problems at it and
33:42they've said like, "Here's a coding
33:43problem. Go. Here's a coding problem.
33:46Here's an environment. You have access
33:47to this virtual computer. Here's like
33:49the codebase you're working with. You
33:51can write code. You can edit the code.
33:52You can run the code. You can read it.
33:54You can use the internet. Go." And it
33:57does that for a while. And then based on
33:59how successful it is, reinforcement
34:01happens. and they have thousands maybe
34:03millions of examples of coding problems
34:06like that that they train it on and
34:07that's why they're so good at coding
34:09now.
34:09>> So what does super intelligence look
34:11like in this regard? Is it just more of
34:12these connections and how would they get
34:14more connections? Can you explain that
34:16to me like on
34:17>> So there's different AI models, right?
34:19So there's like you know GPT3 and GPT4
34:22and GPD 4.5 and GPD5 and GPD 5.5 and
34:255.6, right? Sometimes they're just the
34:28same previous model but with extra
34:29training. Mhm.
34:30>> Sometimes there are a whole new model
34:32that's been trained from scratch,
34:33including starting the whole
34:34pre-training process again. Over the
34:37last couple years, they've done several
34:39new rounds of starting over from
34:40scratch. And typically, when they start
34:42over from scratch, they make the whole
34:44thing bigger, the artificial brain much
34:46bigger.
34:46>> Okay.
34:46>> Right now, they're at something like 10
34:48trillion parameters. Back in 2020, um it
34:52was more like 175 billion.
34:55>> So, we've grown like two orders of
34:56magnitude uh in six years.
34:58>> Two orders of magnitude. Yeah, like two
35:0010 x's, so 100x, right?
35:03>> So that process is continuing. Um,
35:06they're also improving the algorithms
35:08themselves. So they're not literally
35:10just the same type of AI, but bigger.
35:12They've also come up with all sorts of
35:13ideas for how to change the structure of
35:16the of the connections and the neurons
35:18and so forth and change the like
35:20reinforcement
35:21algorithms that they're using and to
35:23change the training data that they're
35:25training on. All sorts of tweaks that
35:27have made this whole thing more
35:28efficient. We're literally building a
35:30brain
35:30>> basically. Yeah. As they make more
35:32brains, they're getting better at making
35:33they're making them bigger and making
35:35them more efficient and so forth.
35:36>> And it's literally modeled on the brain,
35:38like the way it works, right?
35:40>> It's it's certainly heavily inspired by
35:41the brain, but I I shouldn't overstate
35:43the analogy. Like there's lots of
35:44differences, too. So, for example, the
35:46transformer architecture um
35:48>> which is
35:49>> which is the architecture that they use
35:50for for these LLMs, uh is not really
35:53recurrent. So the information sort of
35:55flows one way rather than allowing all
35:58these sort of little loops on the
35:59inside. Also the the back propagation
36:01algorithm is different from the sort of
36:04um learning that naturally happens in
36:06human brain. So there are some
36:06differences but yes like broadly
36:08speaking uh we are sort of making
36:10artificial brains. It's kind of like for
36:12brains what like a plane is for a bird.
36:14>> Mhm. That's a [clears throat] really
36:15good analogy. Yeah,
36:16>> that that analogy helped me think
36:18through a bunch of questions people
36:19often ask about AI when they said can it
36:21be creative but actually that analogy
36:24kind of helps me understand that
36:25actually that maybe that's not the
36:26question it's can it produce something
36:29that you would consider to be creative
36:30because [clears throat] creativity is
36:32people think of it as like a process but
36:34actually it's it's judged based on the
36:35output isn't it? I mean, you you can get
36:37philosophical about like, is it truly
36:39creativity that they have, but you can
36:41also be like, well, I mean, just look at
36:42all the stuff they're accomplishing,
36:43[laughter]
36:44you know, and uh it seems like they're
36:46going to be accomplishing a lot more in
36:47the near future.
36:48>> Yeah, I do I I asked the question about
36:50how this weighs on you personally
36:51because I can I can sense that you're
36:53actually personally bothered.
36:55>> I mean, I think the situation is crazy.
36:57Like,
37:00first of all, it's very exciting. Like,
37:01AI is really fascinating and interesting
37:02stuff. I've been following the field for
37:04more than a decade now. Um, I've been
37:06part of it uh for some years and um it's
37:10really cool, really interesting and it's
37:11really fun to think about what's going
37:14on inside these artificial brains and
37:15why they are the way that they are. And
37:17it's really cool to see all the
37:18applications of this technology out in
37:20the world. But it really seems like
37:23we're on a pretty scary path. And the
37:24more you think about it, the more
37:26worried you get. And you know, in
37:29stories it always ends well, but this is
37:31real life. And I I think we have to sort
37:35of stare reality in the face and tell it
37:38and realize that like it might not
37:39actually end well, you know.
37:41>> Were there any recent dare I say I was
37:44going to say Eureka moments, but
37:45paradigm shifting moments where even
37:47your own sort of mental model of what's
37:49going on here and how this is going to
37:51look were changed for better or for
37:52worse.
37:53>> For better or for worse and probably for
37:54worse, things are kind of on track for
37:56AI 2027. There are a few things that
37:59have been different. not exactly like
38:00paradigm shift differences but like
38:02there have been some differences from
38:03what we expected at the time we wrote
38:05this. So the government has actually got
38:07involved faster than we expected and has
38:09been more aggressive than we expected.
38:10So the export controls on mythos being
38:13uh the biggest example and also
38:14threatening anthropic with uh being
38:16destroyed by the defense production act.
38:19Um [clears throat]
38:20>> another thing that's been surprising to
38:22us is that anthropic in particular has
38:24gone from second place to first place in
38:26the sort of in the race basically. Why
38:29do you think that happened? Because it
38:30seemed like chat Gvt were out front and
38:33clear as it relates to OpenAI were out
38:35front and clear, but suddenly Anthropic
38:37have uh lapped them.
38:40>> Yeah, I mean I guess they have um
38:42probably higher talent density um and
38:44better strategy, but not by a lot, but
38:48enough to make the difference.
38:49>> Why do you think they have more talent?
38:52>> Well, they don't have more compute. Like
38:54what are the inputs, right? Like they're
38:56in the lead now. They used to be behind.
38:58What are the possible explanations for
38:59this? Well, it could have been that they
39:02had more resources, like more compute,
39:03more money. But that's not true. They
39:04have less resources and less money,
39:06right? So then I guess talent is is the
39:10next best alternative. You could maybe
39:12say strategy,
39:13>> some combination of those things. Yeah.
39:15>> Something that wasn't just like the
39:16amount of resources they had.
39:18>> Just like Jon Jones, where marginal
39:20improvements in your cognitive
39:21performance can have a massive impact.
39:23Sometimes I podcast for 10 hours a day.
39:25Over the last couple of weeks, I've been
39:26in filming for a TV show and then I have
39:28like one or two days off to get all of
39:30my work done, which means there's lots
39:32of cognitive load. And so, I turned to
39:33ketones because I find myself more
39:35articulate, able to think more clearly,
39:37able to work out better when I'm fueled
39:40by ketones. And so, the reason I became
39:42a co-owner of this company and the
39:43reason why they now are a sponsor of
39:44this podcast is because I remember one
39:46of my team members called Cristiana, she
39:47tried it once and came up to my desk and
39:49she goes, "This is the best product ever
39:51made." And I think in part that's
39:53because she really cares about those
39:54cognitive benefits as I do, as John
39:56Jones does, and as I think most of my
39:58listeners probably will. So, if you
39:59haven't tried these yet, all you have to
40:01do is go to ketone.com/stephven
40:05and you'll also get 30% off your first
40:07subscription order. You'll get exclusive
40:08Ketone IQ merch and of course, cognitive
40:11benefits that might just change your
40:13life. Much of the reason most people
40:16haven't posted content or built their
40:17personal brand is because it's hard and
40:19it's timeconuming and we're all very
40:21very busy. And if you've never posted
40:23something before, there's so many
40:26factors in your psychology that stop you
40:28wanting to post, what people will think
40:30of you. Am I doing this right? Is the
40:32thing I'm saying absolutely stupid? All
40:34of these result in paralysis, which
40:36means you don't post and your feed goes
40:38bare. I'm an investor in a company
40:40called Stanto, which you've probably
40:42heard me talk about. And what they've
40:44been building is this new tool called
40:45Stanley that uses AI, looks at your
40:47feed, looks at your tone of voice, looks
40:49at your history, looks at your best
40:50performing posts, and tells you what you
40:52should post, makes those posts for you.
40:54You can also just use it for
40:56inspiration. And sometimes what we need
40:58when we're thinking about doing a post
40:59for our social media channels is
41:01inspiration. Building an audience has
41:03fundamentally changed my life, and I
41:05think it could change yours, too. So,
41:06I'm inviting you to give this new tool a
41:09shot and let me know what you think. All
41:11you have to do is search coach.stand
41:12stand.store now to get started.
41:15>> Uh, a friend of mine who knows some of
41:17these people sat me down once upon a
41:19time in London. He's actually said this
41:21a few times to me, but I remember one
41:23particular conversation where he says
41:25that some of these AI CEOs predict the
41:29probability of extinction at being I
41:31think he said 7%. I don't know why I
41:33have that number in my head, but I
41:34remember it being less than 10%. And the
41:35point he was making to me was that even
41:38if it was 1%, like if there was a 100
41:40buttons on this table now
41:41>> Yeah. and one of them would end the
41:44world. Would I dare press any of them?
41:46You know, [laughter]
41:47um,
41:48>> no,
41:48>> I wouldn't press any of them. But he
41:50made the case to me that these AI CEOs
41:52are very smart and they understand super
41:53intelligence and that they think
41:54actually if there was a 100 buttons on
41:56this table right now, maybe 10 of them
41:58could end the world. I've heard you say,
42:00I think it was on the the the Daily
42:01Show, the interview did, you said that
42:02you think there's a 70% chance of human
42:04extinction due to AI.
42:06>> I wouldn't say human extinction exactly.
42:08I would say something like 70% chance
42:09that this goes horribly wrong like human
42:11extinction. But that's just one of
42:12several possibilities. But yeah,
42:14basically
42:15like for example, possibly the AI take
42:18over and then don't actually kill
42:20everyone. You know, maybe they do
42:22something else. Like just just because
42:24they've taken over doesn't mean they're
42:25definitely going to kill us, right? They
42:27might, but they could do something else.
42:28So that's what that's that's why I don't
42:29usually say like 70% chance of like
42:31actual human extinction, but 70% chance
42:33of like something like AI is taking over
42:36some some sort of very big catastrophe
42:38like that that could lead to human.
42:39>> I guess I got two points there, which is
42:41you've been around these CEOs. I mean,
42:43you've worked for Sam Alman at OpenAI
42:45before you quit. Do you think that they
42:47think there's a chance of human
42:48extinction?
42:49>> Yes. But I think that an important thing
42:52to understand is that like people sort
42:55of believe what they need to believe in
42:57order to think that they're good people
42:59and that they need to keep doing what
43:00they're doing. This is what
43:01rationalization is. And so I think that
43:04the tech CEOs have like genuinely
43:06convinced themselves that like probably
43:09things are going to be fine and that the
43:10way to make things fine is for them to
43:12keep doing what they're doing. And like
43:13they need to like make sure that like
43:15you know Sam needs to make Sam's
43:16probably thinking like can't let Dario
43:18or Elon get there first. You know I know
43:21Dario is thinking Sam can't get there
43:23first. Elon's thinking that like you
43:25know they they they've all probably
43:26convinced themselves that like oh yeah
43:27like maybe it'll go horribly wrong but
43:29like probably it's going to be fine and
43:31probably, you know, I should be the one
43:34in charge. It appears to me that
43:36Anthropic are the only ones that are
43:37talking about the potential chance of
43:39extinction or a catastrophic event or um
43:42the down the real downside still. They
43:44seem to be the only ones that are still
43:46publishing on it and now they're
43:47actually becoming the enemy in many
43:49respects of the
43:50>> the tech industry in San Francisco. I'm
43:52watching a lot of interviews and it's
43:53everyone's attacking Dario because he's
43:55saying listen things could go bad.
43:56They're calling him a doomer and
43:58questioning his incentives. Even with
43:59Mythos, which is a a claude model that
44:02they started to warn the world about,
44:04again, he is attacked immediately for
44:06saying that.
44:06>> Yeah.
44:07>> My question is, do you see him as being
44:09slightly different from Sam in this
44:10regard?
44:11>> Yeah. I mean, it seems like Anthropic
44:14and Stereo have been more willing to
44:18say and do things that are costly to
44:19their bottom line uh at least in the
44:22last year or so. That's an example of
44:24it. um like I don't think that really
44:26wins them favors in the administration
44:28or among their investors to say that
44:30type of thing and you know a better
44:33example is just the whole fight between
44:35the department of war and anthropic was
44:36an example of them doing something that
44:37like cost them a lot of money and even
44:39more importantly cost them a lot of
44:40power for something like like they could
44:44have just signed a contract you know
44:46that said I really don't want to be in a
44:48situation where we're like which CEO is
44:50the least bad CEO let's support that one
44:53you know like none of these people
44:54should be trusted did uh with that much
44:56power basically.
44:57>> Nobody should.
44:58>> Nobody should
44:59>> regardless.
45:00>> Regardless. Yeah.
45:00>> Mhm. So on this point of the buttons,
45:03you you you do believe that they think
45:04there's a credible chance of extinction.
45:06>> Yeah. But [clears throat] they've
45:07convinced themselves that like it's
45:08probably fine and also it'll be even
45:11worse if I'm not doing it, you know?
45:13Like that that's what they'll say inside
45:14the companies too. Like two people will
45:15be like, "Okay, well if we stop, what
45:18about the other guys?" Like they're not
45:19going to stop, you know?
45:21>> Yeah. This is this has always been why
45:22I've had this outstanding question which
45:24is how does this not go bad when human
45:25incentives seem to rule the day when you
45:27look at history and all of the human
45:29incentives are saying well if you you're
45:30damned if you do i.e. you're damned if
45:33you carry on developing these bigger and
45:34bigger and bigger AI brains but you're
45:36also then damned if you don't from a
45:38geographical perspective because the
45:40United States will lose to that country
45:41or this company will lose to that
45:43company. So when you just look at human
45:44incentives and goes how does how does if
45:46just purely incentives and disincentives
45:48how does this end? Well it carries on
45:49going. seems like it. I mean there there
45:51is a caveat to that which is a hopeful
45:53caveat which is that first of all if the
45:56world wakes up to all of this then there
45:58can be a more serious conversation about
46:00regulation and international treaties
46:03and things like that and that can change
46:05the incentives right so the government
46:08could come in and say like actually here
46:10are some rules that you all have to
46:11follow and because they're rules that
46:13you all have to follow then you're not
46:14incentivized to like break them anymore
46:18because you get punished if you break
46:19and everyone else is also following them
46:22too and so you know it's fine. So so
46:24there is that sort of like ray of hope
46:25that like we can change the incentives
46:28if the government and especially the US
46:31government but then later other
46:32countries act to to change the
46:34incentives but that's not going to
46:36happen until people sort of wake up to
46:37all of this. The second thing is that
46:39even individually at some point,
46:43you know, Dario or Sam or Elon might
46:45realize that like actually it's like not
46:48even in their own interest to to keep
46:50racing unilaterally.
46:52And it the problem with that is it's
46:54only if it gets extremely obvious and
46:56extremely dire. So like in in AI 2027 in
46:59that scenario, there's this choice point
47:01that I mentioned and in one case the AI
47:04are misaligned, in the other case the
47:05AIS are aligned. At that choice point,
47:08we have like one branch that depicts the
47:09the misalignment ending and one branch
47:11that depicts like they they slow down a
47:12bit and solve the alignment issues.
47:14>> The instigator for that choice point is
47:16they see some evidence that their AI
47:18might be misaligned and plotting against
47:20them, right? So, if you actually see
47:22that evidence,
47:23>> then it's like, oh gosh, uh maybe we
47:27shouldn't put it in charge of everything
47:28and let it rip, you know, because that's
47:31evidence is staring us right in the face
47:33that it's it's untrustworthy, you know.
47:35But if they don't see that sort of very
47:37clear evidence, then I think they're
47:40going to convince themselves that they
47:41need to keep going, you know, but maybe
47:42they will see very clear evidence like
47:44that. In which case, even if we don't
47:46have regulation, they might just sort of
47:47voluntarily stop. Um, so that's the
47:49second ray of hope. Like overall, I
47:52don't think that we're like definitely
47:53doomed. You know, like [snorts] I said
47:5470%, but like I could see it working out
47:57pretty well as well.
47:59>> H what about uh jobs?
48:02>> Yeah.
48:04So I think I think I'm excited to at
48:06some point get into the new thing which
48:08is the more optimistic [clears throat]
48:09positive vision.
48:10>> Uh and that will have a lot to say about
48:12this because in the in the in the
48:15prediction you know in AI 2027 by the
48:18time everyone loses their jobs there are
48:20worse things happening or like it's it's
48:22kind of like too late by that point. Um,
48:24but yes, like once if I mean just just
48:27think about it. If the companies do
48:28manage to build super intelligence, then
48:30by definition they're going to be able
48:32to take almost all the jobs or all the
48:34jobs, right? Because it's better,
48:35faster, cheaper than the best humans at
48:38everything.
48:38>> And that again, the timeline is by the
48:40end of sort of 2030, you reckon you
48:42think super intelligence might arrive?
48:43I'm trying to think about when we could
48:44start to see job displacement in the
48:46economy.
48:47>> We're already starting to see a little
48:48bit of it now, but not very much.
48:49>> Why?
48:50>> Um, cuz the AI aren't good enough yet.
48:52like they they're they're they're
48:53impressive, but they're not like they're
48:56not just a drop in replacement for a
48:58human worker in almost any field.
49:00>> And do you think that'll be sudden?
49:02>> I think it'll be sudden because of the
49:05intelligence explosion dynamics or
49:07recursive self-improvement dynamics. So
49:09you can imagine a different world where
49:12it's gradual.
49:13>> Mhm. And and [clears throat] this is
49:14this is maybe how it is in a lot of
49:15science fiction is you know the AIS
49:18gradually get better at a bunch of
49:19things and you know they gradually
49:21automate like this one industry like
49:23pharma then they automate like you know
49:26steering drones then they automate like
49:28driving cars or something like that. Um
49:31but what's different about the real
49:33world is that the companies have
49:35converged on this strategy of automating
49:37themselves first, you know, automating
49:40the AI research process. And so
49:44if they're allowed to continue with this
49:45strategy,
49:47we're not going to see like, you know,
49:50the robo taxis and like the plumber
49:52robots and, you know, the lawyer AIs.
49:56We're not going to see that sort of like
49:57broad diffusion of AI into the economy
49:59happening first because that's not what
50:02they're focusing on first. They're
50:03focusing on automating themselves,
50:05automating their own research so that
50:06they can do everything that they're
50:08doing faster. And they want that to sort
50:10of get going and get to, you know, very
50:14high levels of intelligence, very high
50:16levels of general intelligence. Um, and
50:18then deploy more out into the economy,
50:20right? So by the time it's actually
50:23coming for like all these different jobs
50:26they will have had fully autonomous AI
50:28research happening for months maybe
50:31years you know and that means that like
50:34the AIs will be vastly superhuman at AI
50:36research and probably also vastly
50:38superhuman at lots of other things just
50:39as a side effect you know if you're
50:42wondering what this looks like well we
50:44wrote about what it looks like it's sort
50:46of like this this wave smashing through
50:48the economy after they do the
50:50intelligence explosion internally.
50:52>> What I'm hearing there is that because
50:54the AI will be able to improve itself
50:56and train itself, it'll be getting
50:58better at everything at once and then it
51:00will be released at kind of once. Is
51:02that accurate?
51:03>> But it's it's not it's not even exactly
51:04that because even if it's mostly just
51:06getting better at the things that it's
51:07doing like re research that'll have some
51:10spillover effects to other skills as
51:12well and then when it turns to focusing
51:14on the those other skills it'll be able
51:16to do them very fast.
51:17>> What jobs remain in such a scenario do
51:19you think?
51:20I think that's actually a political
51:22question, not a technical question
51:23>> because
51:24>> because on a technical level, all the
51:26jobs can be done by the AIS if they've
51:29reached that level. And so it's a
51:32question of what jobs are allowed for
51:34them to do
51:35>> and what kind of jobs wouldn't be
51:36allowed, do you think?
51:37>> That depends on who's in charge. So
51:39there'd be some sort of political
51:41conversation about like what we're going
51:42to allow and disallow.
51:43>> I mean, in this scenario, the humans are
51:44still controlling them. The AIs
51:46>> depends on what you mean by control,
51:47right? So there's like there's do the
51:49AIs actually have the goals and values
51:51that you want them to have and are they
51:53going to robustly do that and behave as
51:55intended into the future and then
51:57there's like are they obeying your
51:58orders for now?
51:59>> Are they obeying the orders is really
52:00what I'm saying.
52:01>> Yeah. So like even in AI27 in the
52:03scenario where the AI take over and kill
52:04everyone, there's a period of like
52:06several years where they're still
52:07obeying orders and they're, you know,
52:10taking some jobs but not other jobs and
52:12they're helping to make better weapons
52:14that the US government can use to like
52:17do its arms race with China and so
52:19forth. And that's why they're able to
52:22get so much power so quickly is because
52:26the governments and the corporations and
52:27so forth trusts them and is deliberately
52:30deploying them into all of these
52:32positions because it thinks that things
52:34are fine. But because these things are
52:37neural nets, you can't just like look
52:39inside and see what it's really
52:40thinking. You can't really tell. I think
52:42this is a really important point because
52:44unlike software we can look at the code
52:46and see what's going on theoretically
52:48with AI you're saying that we don't know
52:50what why it's making the decisions that
52:51it's making because we can't get inside.
52:53>> One note of optimism is that it doesn't
52:55necessarily have to be that way. Like
52:57there's a a sub field of machine
52:59learning called mechanistic
53:00interpretability and a broader subfield
53:02called interpretability more generally
53:04that's trying to solve that problem and
53:06trying to take these these trained
53:08artificial neural nets and piece
53:09[snorts] them apart and understand like
53:12how the information is flowing and how
53:13the decisions are being made so to
53:15speak. Um the problem is just it's a
53:18very inherently hard problem. If you
53:19have 10 trillion connections to look at,
53:21you can look at any particular group of
53:23them and be like, okay, so this is how
53:25like this particular connection works,
53:26but like how do you get a sense of the
53:28whole, you know, how do you get a sense
53:29of like
53:30>> what's happening at a high level? And
53:31the answer is, well, it might be
53:32impossible, but people are working on it
53:34and they are making progress and if they
53:37can make enough progress, then we're in
53:38a very different and much brighter
53:40world. I think that it would be much
53:42less likely for us to get into those
53:44loss of control scenarios if we could
53:46just actually see what our AIS were
53:48thinking and why and how at any given
53:50time, right?
53:52>> Yeah.
53:52>> So, we would still have the other
53:53problems to worry about, but at least we
53:55could mostly solve that one.
53:56>> It is pretty crazy to think that we're
53:57building a technology, a brain that we
53:59don't understand.
54:00>> Yeah, it's pretty crazy. I mean, it's
54:01one of those things where like
54:03>> in a movie, like a sci-fi movie, a bunch
54:05of scientists stood around this big
54:06brain and they're all just like they're
54:08making it more they're feeding it.
54:09>> Yeah. Yeah,
54:09>> but they don't really know what the
54:11>> Yeah, I mean it's it's kind of just like
54:12obviously a dangerous thing to be doing.
54:14>> Um, but we're doing it anyway because of
54:16this history of how the field has
54:18developed in the last 10 years where,
54:20you know, people were like, "Oh, wow.
54:21Yeah, that's obviously dangerous. Oh,
54:23no. What if someone else did it and did
54:24a bad job of it, therefore we should do
54:26it and do a good job of it." And now
54:29they're in this race where where they're
54:31racing each other. And they're also
54:32under all sorts of political pressure to
54:34like pretend that it's not as bad as it
54:35seems because they don't want to like
54:38anger their investors. They don't want
54:39to anger the White House.
54:41>> One of the the key questions we had from
54:43our audience was which and I kind of
54:44asked you this in part, but which jobs
54:46are genuinely likely to survive AI and
54:49what skills should people/ students
54:51focus on over the next 10 years? That's
54:54kind of like like imagine if you were
54:57someone living in Mexico in like 1500
55:01and then you hear that like the
55:03concistadors are coming.
55:05You could be asking yourself like okay
55:07well what sort of job should I be
55:08switching to to like survive this
55:10transition. But like you have a lot more
55:13to worry about besides that. But yes, I
55:15think I would say that like if we manage
55:17to avoid the loss of control problem and
55:20we end up with humans still in charge of
55:22the AI and humans can like say what the
55:24AI's goals and values are supposed to be
55:25even as they become much smarter than
55:27humans and even as they run the whole
55:29economy then probably there will be
55:31regulation that protects some areas and
55:34you can try to guess at what those areas
55:36might be maybe stuff that's more like
55:39like like judges potentially. What about
55:41podcasters?
55:44>> Be honest.
55:45>> Probably not podcasters, I think. Um,
55:48stuff like, you know, being a nanny,
55:51maybe, right? Like, I think it's even if
55:53there's a robot nanny that's like really
55:55really good, I think a bunch of people
55:56might prefer to have an actual human
55:58because they might be creeped out by the
55:59idea of a really good robot nanny. So,
56:01you can sort of you can sort of reason
56:02like that. There's also like stuff that
56:05might be legally protected, like maybe
56:06judges, for example, like are going to
56:08be legally required to be humans and not
56:10robots. Some people say though there's
56:12going to be so many jobs created that we
56:13can't foresee right now like there was
56:14in the industrial revolution or the
56:16internet boom or whatever.
56:18>> The problem with that is that um past
56:21technological advancements have been
56:24more narrow. They've like automated some
56:26things but not everything. But we are
56:28talking about a hypothetical future
56:30situation in which everything gets
56:31automated. So there isn't any new job
56:34that you could do that the AI couldn't
56:36also do except if it's like protected by
56:39regulation or something. That's that's
56:40that's also a thing. But so like for
56:43example, right now there's this sort of
56:45like cycle where
56:47you know the AI has learned to do a
56:49certain thing like write copy or like
56:53draft code or like debug something and
56:56then humans who used to do that thing
56:58switch to managing AIS or switch to
57:00doing the other stuff that the AIS can't
57:02do. And that's why there's been this
57:04dynamic historically of, you know, new
57:07jobs opening up and people flooding to
57:09them. But if it gets to the point where
57:11the AI can do everything that humans can
57:13do and better and faster and cheaper,
57:14then whatever that new job is that you
57:16might have switched to, like the AI can
57:17switch to that, too. And they'll already
57:19be better at it than you.
57:21>> Because we haven't seen widespread
57:22unemployment yet in the economy, do you
57:24think people are getting a little bit
57:25complacent? Because what I'm seeing on
57:26my timeline is a lot of people saying,
57:28"I told you so. I told you everything
57:29would be fine." And when you look at the
57:31the US unemployment rate, currently the
57:34it's flat to slightly down. If you look
57:36at the UK, it is up. The trend is up
57:39compared to last year. We're at about 5%
57:41unemployment. The US is at 4.2%
57:43unemployment.
57:44>> Yeah. Basically, nobody has said that
57:46there would be mass unemployment by now.
57:48Or at least we didn't say that, you
57:49know, and we were historically one of
57:51the more bullish people on AI progress
57:53in AI 2027 because of the dynamics that
57:55we just described. The mass unemployment
57:57doesn't happen until 2028 or 2029 after
57:59they already have super intelligence
58:01because again the companies aren't
58:03trying to cause mass unemployment as
58:05step one. That's like step three after
58:08you know it's like step one automate
58:09themselves. Step two have this recursive
58:13self-improvement to get to super
58:14intelligence. Step three expand out into
58:16the economy and automate everything. And
58:18so this is really unfortunate from
58:20humanity's perspective because one might
58:23have hoped that if there was this broad
58:26wave of automation going through the
58:27economy, people would sit up and pay
58:29attention and think about where all this
58:31is headed and demand good regulations
58:34from the government. But that's not
58:37actually the strategy the companies are
58:38taking. you know, they're going to be
58:39getting the super intelligence first and
58:40then doing the broad wave of automation,
58:42which means that by the time they're
58:43actually doing all of that, uh, well,
58:47it's already going to be moving very
58:48fast and the AI will already be very
58:49powerful.
58:50>> In your 2027 report, so you wrote that
58:52in 2025, but it is called AI 2027. You
58:56said that in mid 2025, we'd have the
58:58autonomous employee, which is sort of
59:00like AI agents taking instructions over
59:01Slack or Teams.
59:04>> That happened. I've actually got an AI
59:06agent in my WhatsApp, which I talk to,
59:07of course. I've got Clawbot exploded
59:09obviously around the world and and now
59:11um you know Claude have talked about uh
59:12their new Slack integration but lots of
59:14people are using agents now and that
59:16happened I'd say for us at the we really
59:18sort of caught on to it the the start of
59:202026 you also said by by 2026 companies
59:24begin replacing entire corporate
59:25departments with AI agent subscriptions
59:272027 the final job AI automates the job
59:30of the human AI researchers themselves
59:32and begins the machine learning research
59:34to upgrade and build the next generation
59:35of AIs.
59:36>> Yeah. Yeah. So again timelines we are
59:39uncertain about how long it will take to
59:41achieve these milestones in this
59:43scenario they happen at those times but
59:46by the time we had actually published
59:48the scenario our timelines had shifted
59:49back a little bit specifically mine had
59:51so like my 50% mark was 2028
59:55>> for that for the full automation of AI
59:57research milestone not 2027
1:00:00uh and then other people on my team had
1:00:02more like 2030 2031 things like that so
1:00:05I I I kind of want to Excellent.
1:00:07>> Maybe try to illustrate this with, you
1:00:08know, we have like this probability
1:00:10distribution. It's like a smeared out
1:00:12probability mass and like the 50% mark
1:00:14is this particular year, but there's
1:00:16like a lot of possibility that it
1:00:18happens
1:00:19>> years earlier or years later, right?
1:00:22>> What is this AI 2040?
1:00:24>> So AI 257 was a best guess prediction as
1:00:27to how things would actually go.
1:00:28>> Yeah.
1:00:28>> AI 2040 plan A is our recommendation for
1:00:31how things should go. So we called it AI
1:00:342040 because in this scenario uh they
1:00:37build super intelligence in 2040 instead
1:00:40of much sooner because they delay
1:00:42things.
1:00:42>> Why do they delay things?
1:00:44>> To manage the risks and make sure that
1:00:46power is distributed equitably. They
1:00:48basically like
1:00:50regulate AI development so that it still
1:00:52continues but at a slower more
1:00:54reasonable pace in a more transparent
1:00:56and safe way and spread out over more
1:00:59countries and companies and as a result
1:01:01they get to super intelligence in 2040
1:01:04instead of in say 2030 and then we call
1:01:06it plan A because well it's our
1:01:09recommendation like we've we've come up
1:01:10with a plan for what government should
1:01:13do and uh the scenario is an
1:01:16illustration of what it might look like
1:01:18to imple ment that plan in a similar way
1:01:19to how AI27 is kind of an an
1:01:22illustration of what it might look like
1:01:24to do what the companies are currently
1:01:25planning to do. That makes sense.
1:01:27>> And is this wishful thinking or is this
1:01:29what you think is going to happen?
1:01:31>> No, it's definitely not what we think is
1:01:32going to happen.
1:01:33>> It's not what you think is going to
1:01:34happen.
1:01:34>> No. No. What we think is going to happen
1:01:35is still something more like this,
1:01:38right? We we don't expect the world to
1:01:40listen to us, right? this is our
1:01:41recommendation, but we mean we hope that
1:01:44that people do something like this and
1:01:45we think it's possible, but it's not our
1:01:47like prediction for what's going to
1:01:49happen by default, you know. So, I do
1:01:51want to run through the plans, the
1:01:53potential plans and also plan A, but um
1:01:56just to close off on how things might
1:01:57look after the year cuz I think I wanted
1:01:59to touch on robotics too and I've got
1:02:00this graph here which talks about share
1:02:02of labor output.
1:02:04>> Yeah.
1:02:04>> Um which I found to be quite striking. I
1:02:06I've been sat here wondering as an
1:02:07employer who employs hundreds and
1:02:09hundreds of people
1:02:10>> when when all this stuff is going to
1:02:12happen and you know we're still hiring
1:02:13more people as things stand there are
1:02:16some roles where our consideration is
1:02:18changing shifting considerably
1:02:21>> and I'd have to say that you know we're
1:02:22probably in a phase where our teams are
1:02:24AI powered and they're using agents to
1:02:26do some of their work now
1:02:27>> but I'm wondering as an employer like
1:02:29when is it when does this happen?
1:02:31>> Yeah, great question. So if we could
1:02:33maybe zoom in on this a little bit.
1:02:34>> Yeah, we'll put it on the screen. Um, so
1:02:36this is in the AI 2040 plan A scenario
1:02:39and notably in that scenario there's
1:02:42significant regulation introduced in
1:02:432029 that slows down the pace of AI
1:02:46development. In the scenario, they do
1:02:47that sort of at the last moment. So in
1:02:49the scenario, if they hadn't done that,
1:02:52then it was about to take off similar to
1:02:54how it does in AI 247. Um, but as you
1:02:57can see, like in the scenario, there's
1:02:59still a bunch of jobs at the point that
1:03:02they implement it. And this gets back to
1:03:04what I was saying earlier is that if you
1:03:05wait until most people have lost their
1:03:07jobs to regulate the AI companies,
1:03:10that's already too late because they
1:03:13will probably already have super
1:03:14intelligent AI by then because their
1:03:16strategy is to first get super
1:03:17intelligent AI and then do all that
1:03:18stuff.
1:03:18>> And I think you say that it would
1:03:20collapse the economy and cause even more
1:03:21harm to suddenly regulate something that
1:03:23all of us and all of our lives were then
1:03:24at that point relying on.
1:03:26>> Oh, but it's a risk well worth taking. I
1:03:28mean, we it's true that right now a lot
1:03:30of people use AI for a lot of things,
1:03:31but like if we could somehow slow or
1:03:34halt AI development now to set up a
1:03:36better way to do it, that would be well
1:03:38worth it. Um, even though there would be
1:03:40significant costs,
1:03:41>> but you can't over here, right, can you
1:03:42at this point where AI and robotics are
1:03:45doing most of the labor output.
1:03:46>> That's right. But in but in but in in
1:03:48this scenario in the AI 2040 plan A
1:03:50scenario they put in the regulations in
1:03:512029
1:03:52>> and then they slowly and carefully
1:03:54develop AI in a way that avoids all the
1:03:57problems which we can get into in a
1:03:59little bit. And so eventually, yes,
1:04:00eventually the AIs take the jobs.
1:04:02Eventually, basically the whole economy
1:04:04is run by AIs and robots, but it it
1:04:06happens gradually over the course of the
1:04:092030s
1:04:10>> instead of happening in this sort of
1:04:12crazy shock, you know, a year later,
1:04:15>> right?
1:04:16>> Because in this scenario, they don't let
1:04:18the companies recursively self-improve
1:04:20and get to super intelligence as fast as
1:04:22possible. Instead, they regulate AI
1:04:24development so that the core
1:04:26capabilities of the AIS are improving at
1:04:28a more reasonable pace and also in a
1:04:31more transparent way so that the
1:04:32scientific community can see what's
1:04:34going on and help make it safe.
1:04:36>> But it's uh I guess I noticed here that
1:04:38in both your scenarios eventually AI and
1:04:41robotics do pretty much all the jobs.
1:04:43>> Yes.
1:04:44>> So you kind of side there with Elon when
1:04:46Elon says that working will be a choice.
1:04:50Uh
1:04:53>> because I mean what happens?
1:04:54>> Yes. I mean if [laughter] it by
1:04:55definition if it can do all the things
1:04:56then it can do all the things. I think
1:04:59that there's a question of like should
1:05:01we allow there to be AIS that can do all
1:05:03the things right? Some people think that
1:05:05the answer is no and we should just shut
1:05:07it all down and prevent these types of
1:05:09AIs from being created in the first
1:05:11place. And we're actually kind of
1:05:13sympathetic to that. We we have our
1:05:15Should we bring out the plans diagram?
1:05:17>> Yeah.
1:05:18>> Thanks. Yeah. So
1:05:21our scenario is called AI 240 plan A.
1:05:24It's a scenario in which they slow down
1:05:25AI development to make super
1:05:27intelligence happen in 2040 instead of
1:05:28earlier. And plan A is our
1:05:30recommendation. So this is sort of
1:05:31illustrating our recommendation. But for
1:05:33comparison, we made like mini scenarios
1:05:35illustrating different alternative plans
1:05:38which we call plan S, plan B, plan C,
1:05:40and plan D. Plan D is basically the same
1:05:44thing that happens in AI27. Like the
1:05:46race continues. there's very little
1:05:48regulation. Um, you can read about that
1:05:50in AI27. Plan C also very similar to
1:05:53what happens in the slowdown ending of
1:05:55AI27 where they solve the alignment
1:05:56problems. So, in that ending, they like
1:05:59slow down a little bit, pivot more
1:06:01resources to AI alignment and AI safety
1:06:04research, get lucky and succeed, and now
1:06:06they have aligned AIS and then they
1:06:09speed up again and take all the jobs and
1:06:11beat China and all those things. Plan B
1:06:14is
1:06:16it's kind of like plan C in that well
1:06:21basically in plan B you're uh being more
1:06:24aggressive towards China and you're like
1:06:26taking actions to sabotage or cyber
1:06:28attack them to like keep them behind so
1:06:30that you have more breathing room to to
1:06:32solve the alignment problems yourself.
1:06:34Plan A is our recommendation. It's uh
1:06:38domestic regulation and then an
1:06:39international deal to continue building
1:06:41AI, but in a much better way. Plan S is
1:06:44shut it all down. If you want to have a
1:06:47future where there aren't AIs running
1:06:50around that can do everything better and
1:06:52faster than humans, you kind of want
1:06:53something like plan S.
1:06:55>> What What do you want?
1:06:56>> Plan A is our recommendation. I think
1:06:59that I'm sympathetic to plan S, but for
1:07:01reasons we explain, we recommend plan A
1:07:04instead. And what do you think is most
1:07:05probable if you're being honest?
1:07:07>> Plan D,
1:07:08>> which is that they just
1:07:10>> type of thing where they keep racing,
1:07:11they don't really slow down
1:07:12significantly. Um, and uh things happen
1:07:17extremely fast. The diagram sort of
1:07:19explains like roughly the reasoning
1:07:20behind this too. So like there's this
1:07:22high level thing of like do you want to
1:07:24keep racing as fast as possible to make
1:07:27the AI smarter and smarter to put them
1:07:29in charge of more things so that we can
1:07:30be China? You know, if you're happy with
1:07:33that, then you can get down into this
1:07:36variation of optins here. If you are
1:07:39worried about that, well, you get to
1:07:42something like this. There's more
1:07:43different options besides these, but
1:07:44this is kind of like the ones that we
1:07:47could compress onto a screen.
1:07:50>> Do you have children?
1:07:51>> Yeah, I have two children.
1:07:55It's kind of sad. Like
1:07:58I think that one way or another this
1:08:00will probably all be over by the time
1:08:01they're old enough to join the
1:08:03workforce.
1:08:04So I don't think they'll ever join the
1:08:06workforce.
1:08:07>> When you say this will be all over by
1:08:09the time they join the what do you mean
1:08:10by this will be all over?
1:08:13>> So these milestones that I described
1:08:15like AI is automating the AI research AI
1:08:17is getting super intelligent. Um
1:08:20AI is then exploding out into the
1:08:22economy, taking the jobs, building robot
1:08:24factories to build more robots, to build
1:08:26more factories, etc. GDP starting to go
1:08:29vertical. That sort of thing is what I
1:08:31mean like all of those events
1:08:33transpiring. Maybe there's like, you
1:08:34know, 10 20% chance or something that
1:08:37hits the wall and and none of this comes
1:08:40to pass even if you don't do anything.
1:08:43>> How old your oldest?
1:08:45>> Six.
1:08:45>> Six. Boy, girl,
1:08:47>> girl,
1:08:48>> girl. So your daughter comes to you and
1:08:49says, "Dad, what shall I um what shall I
1:08:51study in school?"
1:08:52>> I mean, again, like if these radical
1:08:55transformations happen, then the world
1:08:58would just look completely different and
1:09:00what sort of jobs you set yourself up
1:09:01for basically won't matter that much
1:09:03probably. I would say um that the thing
1:09:06to do is well a try to make it actually
1:09:09go well. Like if you can exert any
1:09:10influence at all on history and how this
1:09:12all develops, you should be trying very
1:09:14hard to steer the future in better
1:09:16directions. And then separately from
1:09:18that on a personal level you should
1:09:20focus on well
1:09:23being a good person and doing things
1:09:25that are sort of good in the for their
1:09:26own sake rather than good because
1:09:28they'll set you up for later employment
1:09:29because that later employment is going
1:09:31to be very uncertain. Um basically
1:09:34>> Elon talks about this age of abundance
1:09:35we're heading towards
1:09:37age of abundance.
1:09:38>> There'll definitely be abundance. The
1:09:41question is who controls the abundance
1:09:43and what do they do with it? Right? Are
1:09:46the AIs controlled by anyone or are they
1:09:48doing their own thing? And then if they
1:09:50are controlled by people, who controls
1:09:52them and what do they do? And what's the
1:09:54sort of like political structure
1:09:56governing how they make those decisions?
1:09:58>> I think it was Jeffrey Hinton that said
1:09:59to me, he said there's no example in
1:10:01nature where a more intelligent species
1:10:05is has less control than a less
1:10:09intelligent species. Thus saying that
1:10:12we're quite arrogant to think that in a
1:10:14world where there's this artificial
1:10:16brain that's a gazillion times the size
1:10:18of mine that I'm going to give it
1:10:19orders.
1:10:20>> Yeah. I mean that that's the thing is I
1:10:22I think it's like that should be our
1:10:24default assumption is that like well
1:10:26there's these brains. We can't see
1:10:28exactly what they're thinking. We're
1:10:30going to make them smarter than us and
1:10:31put them in charge of everything
1:10:33>> and then we're going to give them
1:10:33bodies.
1:10:34>> Yeah. And then they're going to be
1:10:35autonomously building new factors and so
1:10:37forth. And like how is this supposed to
1:10:38end well again? Like isn't this just
1:10:40exactly like us picking a new species
1:10:44that's then going to out compete us when
1:10:45it doesn't need us anymore? Like I think
1:10:47that is just the default trajectory. Now
1:10:50there's a whole argument we can get into
1:10:51about like ways that we could get off of
1:10:52that default trajectory. So for example,
1:10:54there's research into interpretability
1:10:56that I described previously. And if that
1:10:58research bears fruit, then you will be
1:11:00able to actually see what they're
1:11:01thinking and then that would be an
1:11:02excellent tool for shaping them and
1:11:04controlling them and making sure that
1:11:06they do what we want. Right? there's
1:11:07other sorts of um AI alignment research
1:11:09agendas that are making progress and if
1:11:12enough of those agendas succeed
1:11:14sufficiently we can avoid this problem
1:11:16of course also there's the regulatory
1:11:18side too where like part of what makes
1:11:19this difficult is that we're building
1:11:21these AIs in race conditions you know
1:11:23like the the companies are secretive
1:11:25about their recipes for making these AIs
1:11:28because it's secrets that they want to
1:11:30protect so that other people can't copy
1:11:31them and so a lot of it is happening you
1:11:34know behind closed doors only a few
1:11:36people can really see the recipes that
1:11:38they're using to train these AIs and and
1:11:40so forth. And then often times when the
1:11:42AI behave in unexpected ways or even
1:11:44just like blatantly misaligned ways,
1:11:46sometimes that information doesn't
1:11:48really flow out to the public because
1:11:49the companies are not really
1:11:50incentivized to tell everyone about how
1:11:52they messed up and how their AI is evil.
1:11:54It's just not very conducive to
1:11:55scientific progress on these issues. If
1:11:58the regulatory system was different,
1:11:59then perhaps we could be in a better
1:12:01situation, make faster progress. Also,
1:12:03of course, we wouldn't be planning to
1:12:05put these AIs in charge of everything as
1:12:07fast as possible, and we wouldn't be
1:12:08planning to like let themsel improve,
1:12:10you know, like the these are choices
1:12:12that we could not make, you know?
1:12:17I don't speak Vietnamese, but this show
1:12:19can because of AI video technology from
1:12:21our sponsor, Hey Jen. I get messages
1:12:23every single week from those of you
1:12:24listening to the D of Co all around the
1:12:26world, and you express how much impact
1:12:28it's had on you and your life. And if
1:12:30that's true, then those conversations
1:12:31shouldn't only reach people in English.
1:12:33Ken can take one recording of me and
1:12:36deliver it in any language while keeping
1:12:37my voice, timing, and expressions
1:12:40intact. But you don't need a studio like
1:12:42this to make it work for you. Record 15
1:12:45seconds of yourself and get an AI avatar
1:12:47that delivers studio quality video in
1:12:49over 175 languages. We're up to 20
1:12:53languages now, and we're not the only
1:12:55ones using it. Hey Jen is already used
1:12:57by 30 million people, including 85% of
1:13:00the Fortune 100. Whether you're building
1:13:02an audience on social media, launching
1:13:04an online course, or rolling out
1:13:05training across your team, check out
1:13:07Heyen now. Your first three videos are
1:13:10totally free at heyjen.com/doac.
1:13:13That's hygen.com/doac.
1:13:17See you there.
1:13:19Ilia was, as you said, he was one of the
1:13:21leaders at OpenAI and he left and he
1:13:22started his own company now, Safe Super
1:13:24Intelligence. very curious name of a
1:13:26company, Safe Super Intelligence, after
1:13:28leaving OpenAI. Did you ever get to work
1:13:30with him?
1:13:30>> Uh, I wasn't directly working with him.
1:13:32I had a couple chats with him.
1:13:33>> Do you think he's he's genuinely
1:13:35concerned as well?
1:13:36>> I think he is, but I think he's I think
1:13:39he's similar to these other CEOs where
1:13:43I mean, just think about the sort of
1:13:44incentives that they're under, right?
1:13:45like
1:13:47they can sort of see the problem and
1:13:49then they can be like okay but like if I
1:13:52don't if I stop if I quit my job and or
1:13:55do something else that's not going to
1:13:57solve the problem cuz the other CEOs are
1:13:58going to keep going and even if all of
1:14:01us didn't go then maybe China would keep
1:14:02going. So like man seems like this is
1:14:04just going to happen one way or another
1:14:05whether I do anything about it or not. I
1:14:08guess I should be involved, you know,
1:14:09and like maybe I can make it go well.
1:14:11And at any rate, like I don't want to be
1:14:12out in the cold while these other people
1:14:14I don't trust are in charge of
1:14:15everything. So they all sort of like
1:14:16reason through all of this and then
1:14:18convince themselves that like the thing
1:14:19to do is for them
1:14:20>> to build it and to do it better. And I
1:14:23think Ilia is just the latest example of
1:14:25this. Elon's another example. Dario is
1:14:28another example.
1:14:29>> You know, arguably OpenAI at the
1:14:30beginning Sam was an example. Although
1:14:32like Elon and Dario were at OpenAI early
1:14:34on. So
1:14:35>> what do you think they should all do
1:14:36then? So I think what should happen is
1:14:38some sort of international regulation or
1:14:40at least domestic regulation similar to
1:14:42what we described in plan A.
1:14:43>> Okay. So walk me through plan A.
1:14:45>> Yeah. So in this scenario, AI takes
1:14:49longer to the to get to recursive
1:14:51self-improvement and full automation of
1:14:53AI research than it does in AI 27. We
1:14:55figured that we should try to illustrate
1:14:57like a range of different possibilities
1:14:58because we do have those sort of
1:14:59uncertainty intervals. So we chose 2030
1:15:02as the moment when full automation would
1:15:06finally be achieved and things would
1:15:07really kick off. And then working
1:15:09backwards from that, when's the last
1:15:11moment you could really have good
1:15:12regulation? 2029. So in this scenario,
1:15:16AI progress slows down a little bit
1:15:17naturally and the AI companies keep keep
1:15:20racing, but they don't quite succeed in
1:15:22automating uh themselves in 2027 or in
1:15:252028 or in 2029, but they're getting
1:15:28really close and they're going to do it
1:15:28in 2030. And then in 2029, the
1:15:31government steps in and regulates them.
1:15:33What regulations do they do? Well, they
1:15:35basically just shut it down temporarily.
1:15:37>> Can I ask um
1:15:39how does the elections overlay with your
1:15:42time frames here? Because there's going
1:15:43to be a big election, isn't there, in
1:15:442028?
1:15:46>> And it seems now that sentiment has
1:15:47really really turned against AI in in
1:15:50sort of in the general public and that
1:15:51it will be one of the big ticket items
1:15:53on the on the ballot.
1:15:54>> We think that it'll be maybe the most
1:15:56important issue in the presidential
1:15:58election in 2028. Um, I think a lot of
1:16:00people mo most people will be quite
1:16:02concerned about where things are headed.
1:16:03And that's part of why we we chose to
1:16:06depict things the way they were doing in
1:16:08this scenario because that helps explain
1:16:10why they might do this sort of
1:16:11regulation in 2029 is that the voters
1:16:13have been demanding it and the
1:16:14presidential candidates have been
1:16:15promising it.
1:16:15>> And in this scenario and in 2027, would
1:16:18the general public have felt the
1:16:20consequences of AI much more severely
1:16:22than they have now by then?
1:16:23>> Yes. Although still even in 2029 in this
1:16:26scenario they still mostly have the jobs
1:16:28as as depicted here. Right. So in in
1:16:302029 in this scenario lots of jobs now
1:16:32involve managing AI agents. You you
1:16:35mentioned you have an AI agent right?
1:16:36Well in 2029 in this scenario the AI
1:16:39agents will be much better. Still though
1:16:41not enough to just completely do
1:16:42everything. You know that was the sort
1:16:44of thing that would come in 2030
1:16:45>> in this in this timeline. Again we're
1:16:48uncertain about timelines. Things could
1:16:49go faster than depicted in this
1:16:51scenario. And in fact, I think things
1:16:52probably will go a bit faster than
1:16:53depicted in this scenario, but we're
1:16:55uncertain. We already did the very fast
1:16:56timeline scenario, so now we're doing a
1:16:58slower timeline scenario. But maybe we
1:16:59should talk about the high level goals.
1:17:01So
1:17:03they want to have AI continue, but in a
1:17:05slower pace.
1:17:06>> Who's that?
1:17:06>> So they can make it safe. The
1:17:08politicians, you know, the president and
1:17:10the people who voted for the president
1:17:11and, you know, the heads of other
1:17:13governments and so forth. So goal one,
1:17:16slow things down. Um, goal two, make it
1:17:18more transparent so that the scientific
1:17:21community can catch up to this stuff and
1:17:23make more progress and also so that we
1:17:24don't have to take the company's word
1:17:26for it when they say that their systems
1:17:27are safe and when they say that they
1:17:28haven't, you know, put in any biases
1:17:31into their systems, for example. That's
1:17:32a concentration of power issue. We also
1:17:35want to avoid a situation where there's
1:17:37an intense concentration of power. So in
1:17:39addition to these the transparency and
1:17:42the slowdown, we actually think it's
1:17:44actively good for there to be multiple
1:17:45AI companies across multiple different
1:17:47countries that have similar levels of
1:17:49very advanced AI capability and for
1:17:51there to be like broad diffusion of AI
1:17:55into society rather than you know a
1:17:57single mega project that has all the
1:17:58best AIs for example. And the nice thing
1:18:01about that is you kind of get that by
1:18:02default if you do the first two things.
1:18:04If you slow it down and if you make it
1:18:05more transparent, then that means
1:18:07there's breathing room for other
1:18:09projects to sort of catch up, right? And
1:18:12the transparency just like literally
1:18:13helps them catch up because then they
1:18:14can like copy copy some of the ideas.
1:18:16And then I think the fourth thing would
1:18:18be reversibility. So in what follows in
1:18:21the scenario, we are going to be
1:18:23building up a lot of data centers and a
1:18:24lot of robots. We're going to be
1:18:25transforming the world at a in a sort of
1:18:27like slower pace, though still a very
1:18:30fast pace, but slower. And if things go
1:18:32wrong and the deal breaks down and
1:18:34everyone starts racing each other again
1:18:36to get to super intelligence as fast as
1:18:38possible, that would be very scary. And
1:18:41so the fourth principle is basically
1:18:43build the new data centers in such a way
1:18:45that if everything breaks down and
1:18:47everyone starts racing again, the newly
1:18:49built data centers get destroyed so that
1:18:51we're sort of back to square one again
1:18:53instead of in an even worse race where
1:18:55there's even more AI and robust and
1:18:57compute everywhere. Um, so I can sort of
1:19:00walk you through the timeline if you're
1:19:01interested. Sure.
1:19:02>> Or the president talks to China, talks
1:19:04to the leaders of a bunch of other
1:19:05countries and says, "We're going to
1:19:08basically halt AI development until we
1:19:10can figure out a plan for how to do it
1:19:12in the in ways that achieve these
1:19:14goals." So they basically send
1:19:16inspectors to each other's data centers.
1:19:18Like Chinese inspectors come to US data
1:19:20centers, US inspectors go to Chinese
1:19:21data centers and verify that they are
1:19:23doing inference and not training.
1:19:26developing new AIS that's that involves
1:19:28training them. But just taking existing
1:19:31AIS and using them to serve customers
1:19:33that's called inference. And so the sort
1:19:36of like solution they come up with here
1:19:38in this scenario is will allow them to
1:19:40keep doing inference but not training
1:19:42for now until we can get the new
1:19:44training data center set up. So they
1:19:47retrofit the existing data centers to
1:19:49serve inference. People can still keep
1:19:51talking to their AI agents, but they're
1:19:52going to stop getting better and better
1:19:55for like six months to a year while they
1:19:57build the new data centers that are
1:19:59going to be the transparent data
1:20:00centers. And that's where the training
1:20:01is going to happen. Once they get those
1:20:03new data centers set up in 2030, then AI
1:20:07research continues. This is a bit spicy.
1:20:09We advocate for total research
1:20:10transparency which means that on the
1:20:12training data centers that are training
1:20:13the new models they basically have to
1:20:16publish everything which means you get
1:20:17to see all the details of the recipes
1:20:19for training these models you get to see
1:20:20the architecture etc. We think that's
1:20:23sort of open science is really important
1:20:24for solving the alignment problem fast
1:20:27enough because you don't want to have
1:20:28these sort of biased companies making
1:20:30the decisions about whether the AIS are
1:20:32safe. Um and we also think it's
1:20:34important for just good regulations more
1:20:36generally because right now most of the
1:20:38expertise in the world on AI is sort of
1:20:40concentrated in Silicon Valley and the
1:20:42the governments in particular kind of
1:20:45are don't really understand AI that well
1:20:48and imagine an alternative instead of
1:20:49total research transparency you had like
1:20:52an auditor system where the government
1:20:53says here are some rules for how to make
1:20:56the AI safe and then we're going to have
1:20:58like an agency that like goes into the
1:21:00companies and asks them questions and
1:21:02tries to make sure that they're
1:21:02following the rules that creates this
1:21:04sort of adversarial dynamic where the
1:21:06company is incentivized to like fool the
1:21:08the regulator, [clears throat] you know,
1:21:10and and also if they if they discover
1:21:12some new problem that's not even on the
1:21:14government's radar,
1:21:15>> they're might be incentivized to like
1:21:16not tell the government about it, right?
1:21:18So, if you have the total transparency,
1:21:19it helps the government make better
1:21:21decisions faster, but
1:21:22>> it kills that competitive advantage.
1:21:24>> Yes, Dropbox's not going to like this.
1:21:26You know, OpenAI is not going to like
1:21:28this. This would be uh probably bad for
1:21:30the valuations. I don't think it would
1:21:31kill them completely, but it means that
1:21:34it would commoditize more, right? So, it
1:21:36means that there'd be like a bunch of AI
1:21:38companies that would catch up to the
1:21:39frontier. They would train AIs that are
1:21:41like roughly similar, roughly
1:21:42equivalent. They could still make money
1:21:44by doing that and then selling their
1:21:46AIs, but they wouldn't have a monopoly.
1:21:48They wouldn't have anything close to
1:21:49monopoly, which I think is good for
1:21:50humanity, although it's bad for the
1:21:52bottom line of those particular
1:21:54companies. Notably, it's good for the
1:21:55bottom line of lots of other companies.
1:21:57Like if you're a company that's behind
1:21:58and you don't you're not anthropic or
1:22:00you're not open AI then you would love
1:22:02this because this helps you catch up you
1:22:04know or this this helps you to like um
1:22:06capture more of the value from the chips
1:22:08you're selling for example or from the
1:22:10like downstream product that you're
1:22:11making
1:22:11>> and by 2031 then you have 1ifth of all
1:22:15cognitive labor done by AI.
1:22:17>> Yeah. So what's happening here is that
1:22:19we're imagining that the government of
1:22:20the United States and the government of
1:22:22these other countries that are involved
1:22:23in this agreement that are sort of
1:22:24implementing similar regulations. Um
1:22:26they don't have to be exactly the same.
1:22:28Uh but that's another thing that's nice
1:22:30about the transparency is that if you
1:22:32have this sort of transparency then if
1:22:34two governments are implementing
1:22:37different regulations like if one of
1:22:38them is like telling their companies to
1:22:40go slower or like banning more stuff
1:22:42than the other one is they can both see
1:22:45>> Yeah. like, oh, you're letting them do
1:22:46that sort of thing and you're not like
1:22:48maybe we should let them do this too,
1:22:50you know? So, it helps to sort of
1:22:52naturally equalize the regulations to
1:22:55some extent without having there to be a
1:22:57central power that just gets to make
1:22:58regulations for everybody.
1:22:59>> Mhm.
1:23:00>> So, anyhow, we're imagining that when
1:23:02they when they get this transparency set
1:23:03up, they basically agree to ban the
1:23:05dangerous stuff to allow the not so
1:23:07dangerous stuff. And there's a constant
1:23:09ongoing conversation about like, well,
1:23:10what's dangerous and what's not? What
1:23:12should we ban? What should we allow?
1:23:13What about this country? what about that
1:23:15country? That conversation evolves over
1:23:16time, but the gist of it is, at least if
1:23:18they do it the way that we recommend it,
1:23:20is that they don't do an intelligence
1:23:21explosion. They don't let the AI, you
1:23:23know, autonomously self-improve.
1:23:25Instead, they slowly and carefully scale
1:23:29up the AIS that they currently have and
1:23:31invest lots into finding ways to make
1:23:33them more interpretable, uh, to make
1:23:34them more easy to control, to understand
1:23:36better how they work, and so forth. The
1:23:38result is that AI progress continues but
1:23:40it's not quite as fast and it's much
1:23:44much much safer and more transparent.
1:23:46>> But still through these you we see job
1:23:48disruption.
1:23:48>> It is continuing because they are
1:23:49building more data centers right like
1:23:51this whole time they're building more
1:23:53and more data centers more and more
1:23:54chips and they're continuing to like
1:23:57make there be a larger and larger
1:23:59population of AIS so to speak and that
1:24:01causes this huge transformation over the
1:24:04course of the 2030s. sort of a big thing
1:24:05that we sort of want people to take away
1:24:07is that even if you heavily restrict AI
1:24:09progress,
1:24:11you still get this sort of crazy
1:24:13transformation. You know, in this
1:24:14scenario, they basically allow progress
1:24:16to continue but at a slower, more safe
1:24:18pace here in 2030 and then as a result,
1:24:21it takes until 2035 to get to top expert
1:24:25level AI. So remember, they were on
1:24:27track to do that in 2030, but then sort
1:24:29of at the last moment they stopped. But
1:24:31because it was sort of so close to the
1:24:33last moment, that means that like they
1:24:35can sort of get there pretty soon if
1:24:36they want to. And it's just a matter of
1:24:38like how long they they allow it to go,
1:24:41right? So they sort of they sort of slow
1:24:42it down, spread it out, leisurely arrive
1:24:45at this level after 5 years. By this
1:24:48point, they've built up massive amounts
1:24:49of data centers everywhere. So it's not
1:24:51just that the AIs are smarter and able
1:24:53to do all the things that humans can do,
1:24:55but also there's a lot more of them and
1:24:57there's a lot of robots and so forth. So
1:24:59by this by this point you kind of have
1:25:02the economy that a lot of people would
1:25:03have imagined with AGI where there's AIS
1:25:06there's lots of them they're able to do
1:25:08all sorts of jobs there's robots there's
1:25:09lots of them they're able to do all
1:25:11sorts of physical work and basically the
1:25:13economy is being run by these machines
1:25:15>> so in 20 31 you you have the 1/5if of
1:25:19all cognitive labor done by AI in 2023
1:25:21you have 60 million AIs running at 100x
1:25:24speed in 2033
1:25:27there's cash dividend to all Americans.
1:25:29>> Mhm.
1:25:30>> Um I've got to
1:25:32explain explain this to me.
1:25:35>> Yeah. So, if the AIS are going to be
1:25:37taking people's jobs, then it's very
1:25:39important that people not starve to
1:25:41death and still have money. And if
1:25:45companies are going to be using AI and
1:25:46robots to take all these jobs, then that
1:25:48means that there needs to be some sort
1:25:49of taxation scheme or something to like
1:25:52make sure that people still have a slice
1:25:54of that pie. M
1:25:55>> the pie is going to grow huge, but you
1:25:57still need to actually give people a
1:25:58slice of the pie. And our proposal for
1:26:00how to do that, we call it the citizens
1:26:01dividend. Basically,
1:26:04people have shares in a agency that
1:26:07sells permits to the robot companies and
1:26:10to the compute companies and makes
1:26:13profit from selling those permits and
1:26:15then those are people have shares in
1:26:17that entity. It starts off small. It
1:26:19starts off something like $25,000 per
1:26:21person. Uh, and then by the end it's
1:26:24something like $10 million per citizen
1:26:26>> per person.
1:26:27>> Per person per year.
1:26:29>> Factoring in inflation. Like what you
1:26:30mean?
1:26:31>> Factoring in inflation.
1:26:31>> So we're all going to be
1:26:32multi-millionaires.
1:26:33>> Yes. If this happens, which they
1:26:36probably won't, but if it happens, this
1:26:37is where it go. And again, this is the
1:26:39thing I want to emphasize is that if you
1:26:40get to the point where your AIs are
1:26:42close to being able to do all the
1:26:44research and then you sort of pause and
1:26:46slow down, that means that like you
1:26:49still have a lot of transformation ahead
1:26:50of you because if you allow those AIs to
1:26:52like still proceed slowly and like start
1:26:54to automate various jobs and so forth
1:26:56after some years they will in fact have
1:26:58done that and they will have, you know,
1:27:00built huge amounts of new data centers,
1:27:02huge amounts of new chip fabs, huge
1:27:04amounts of new robots, robot factories,
1:27:06etc. You know, we're not sure obviously
1:27:08how fast this will go exactly, but we've
1:27:10thought about it a lot and we have our
1:27:11our guesses and this is sort of like our
1:27:12median guess.
1:27:13>> What does this mean? 2037, the
1:27:15apocalyptic arrival of truth on Earth.
1:27:18>> Yeah. So, like this is the point where
1:27:20we say they get to top expert level AI.
1:27:22So, it's not super intelligence in the
1:27:25sense that it's not like vastly smarter
1:27:26than humans at things because they
1:27:28deliberately pause it at the level of
1:27:30top experts. So, so here they're going
1:27:32slow. here they've just actually stopped
1:27:35but they've stopped at a point where the
1:27:36AIs are just actually really good at
1:27:38everything. So kind of they've
1:27:40definitely got AI maybe they got like
1:27:41weak super intelligence
1:27:43because they have so many of these AIs
1:27:45and because they think faster than
1:27:46humans you know they just run much
1:27:48faster that's going to transform society
1:27:51dramatically. So we talk about some of
1:27:53the ways in which it transforms society
1:27:55like this is sort of life after work we
1:27:57talk about what it would be like to be
1:27:58living on your cit citizens dividend and
1:28:00not have a job anymore in this sort of
1:28:01world. Um, here we talk about all the
1:28:04scientific changes and all the social
1:28:06changes that would come from all of the
1:28:09intellectual progress and activity that
1:28:10would be generated by all of these AIs.
1:28:13So, for example, here is things like
1:28:16cancer cures and like you know people
1:28:18living in apartments that were built by
1:28:20robots 2 years ago,
1:28:21>> 2036
1:28:23>> providing again we stop in 2029.
1:28:25>> Yeah.
1:28:26>> And providing I mean a conservative this
1:28:28is a conservative time frame.
1:28:30>> Yeah. Like unfortunately, I actually
1:28:31think that things will happen faster
1:28:32than this by default and that if we
1:28:34don't slow down, things will happen much
1:28:36faster than this. Once you get to the
1:28:37point where you've got, you know, a
1:28:39billion AIs running day and night and
1:28:42they're each better than the best humans
1:28:43at everything. And so they're doing a
1:28:45lot of science. They're doing a lot of
1:28:47talking to each other. They're doing a
1:28:48lot of thinking. Everyone's constantly
1:28:50talking to their AI assistants and so
1:28:51forth. There's going to be a lot of
1:28:53scientific progress. There's going to be
1:28:54a lot of changes to politics, to
1:28:57ideologies. It's going to be very
1:28:59disruptive and crazy and we get into
1:29:01some of the ways in which it is uh
1:29:03later. Basically,
1:29:05>> I still not super clear on what this
1:29:06means. The apocalyptic arrival of truth
1:29:08on Earth. It's just it's just because
1:29:10there's so many eyes AIs that are so
1:29:12smart that they're uncovering making new
1:29:13discoveries in sciences.
1:29:15>> Let me give you an example. Lie
1:29:16detectors.
1:29:17>> Yeah.
1:29:17>> So, that's an example of a technology
1:29:20that might be invented.
1:29:21>> Yeah.
1:29:21>> You know, right now we don't have good
1:29:22lie detectors. We have very bad lie
1:29:24detectors that like sort of work but
1:29:25don't don't fully work. But once you've
1:29:28had these top expert level AIs thinking
1:29:31for many years at, you know, 100x human
1:29:33speed and there's billions of them and
1:29:35they have access to robot factories to
1:29:36do research and stuff.
1:29:38>> They'll probably invent a ton of
1:29:39technologies. Maybe they'll invent on
1:29:40lie detectors that actually work on real
1:29:42humans. That'll have big social effects,
1:29:44right? Imagine a presidential candidate
1:29:46who's like, "Those allegations are
1:29:48false, and to prove them, I will go
1:29:50under a lie detector and say that
1:29:52they're false." I was just thinking
1:29:53about the whole justice system and how
1:29:56that would be overturned. Um, in fact,
1:29:58you could, you know, theoretically walk
1:29:59down the street and be
1:30:01>> Yeah,
1:30:02>> it's both terrifying and exciting. One
1:30:05thing that we talk about in this in this
1:30:06section is like the invention of lie
1:30:08detectors could be really bad. Like it
1:30:10could be that it enables a new form of
1:30:12totalitarianism where the powerful
1:30:14people, you know, the CEOs and the
1:30:16politicians
1:30:17>> force the people under them to go under
1:30:19lie detectors and say like, "Yes, I'm
1:30:20loyal to the dear leader. I would never
1:30:22do anything against the cheerleader,
1:30:24right?
1:30:24>> And if you're lying, then you're in.
1:30:25>> And then if you're lying, you get fired,
1:30:26right? So like there's there's a ton of
1:30:28like very harmful uses of lie detector
1:30:29technology. There's also the good uses.
1:30:31And broadly speaking, I would say the
1:30:33good uses are when lie detectors are
1:30:35used on the powerful instead of by the
1:30:37powerful.
1:30:37>> What's this 2040 passing the torch to
1:30:40AIS?
1:30:41>> Yeah, great. So here they pause at the
1:30:43top expert AAI level. And the reason why
1:30:45they pause is because their safety cases
1:30:48aren't good enough for going beyond that
1:30:50level. Um, so in the sort of regulatory
1:30:52systems that they set up over the course
1:30:54of these years, roughly speaking, the
1:30:56way they would work is when you're
1:30:58making a new AI and then when you're
1:31:00trying to deploy the AI into something,
1:31:02you have to have some sort of safety
1:31:03case explaining like what your
1:31:06intentions are and like why you think
1:31:07it's going to work the way that you want
1:31:08it to work. And in particular, why the
1:31:10AI is going to like
1:31:12>> do as it's told, for example, and why
1:31:14nothing super terrible is going to
1:31:15happen like AI takeover. It's relatively
1:31:18easy to make safety cases like this when
1:31:20your AIs are still not capable of
1:31:22automating everything,
1:31:24>> but the more powerful they get, the more
1:31:26difficult it is to actually argue that
1:31:28things are going to be fine because the
1:31:30AIS are just more capable and they can
1:31:31they can get up to more stuff. And if
1:31:32you if they're actually untrustworthy,
1:31:34the the possible downsides are bigger.
1:31:36So that's why they stop at this level is
1:31:38that they they realize that if they keep
1:31:40going, then they might actually lose
1:31:42control of everything. But at the
1:31:44current level, they're convinced by
1:31:45safety cases that it's fine, but they
1:31:47don't want to go further. So they stop
1:31:48there. And then what happens in 2040 is
1:31:51they've made significant progress
1:31:53scientifically, including on alignment.
1:31:55And they figured out how to make AIs
1:31:57that are actually aligned in a robust
1:31:59way
1:31:59>> with humans
1:32:00>> with humans. So they can actually trust
1:32:02those AIs and they can allow them to
1:32:03become much smarter again. So that's why
1:32:05we call the whole thing AI 2040 because
1:32:07in 2040 they sort of let off the brakes
1:32:11and allow the AIs to become
1:32:13significantly smarter than humans.
1:32:14>> I guess you know this is a this is a
1:32:16plan and this is a hope.
1:32:19>> Yes.
1:32:20>> But in reality this is not what you
1:32:22think probabilistically if you had to
1:32:24>> that's right it's important to
1:32:25distinguish like this is what we
1:32:26recommend. This is what we want to
1:32:28happen from like this is what we
1:32:30actually think will happen by default.
1:32:32Now, we do think it's possible for this
1:32:33to happen, but you know, that will
1:32:35require a lot of people to sort of wake
1:32:36up and pay more attention and advocate
1:32:40for something like this to happen. So,
1:32:41our main scenario is mostly talking
1:32:44about the policy choices made and the
1:32:46broad scale effects on society. We
1:32:48figured it would also be nice to
1:32:49accompany this with a little mini
1:32:52scenario that describes what it would
1:32:53actually feel like to live through this
1:32:56from an ordinary person's perspective.
1:32:58>> Okay.
1:32:58>> Um 2029, everyone's yelling at each
1:33:00other. the presidents are negotiating
1:33:01something and they've paused AI but you
1:33:04still have access to the existing AI so
1:33:06it doesn't really feel that different
1:33:07although it definitely is like something
1:33:09exciting happening 2031 they've started
1:33:11progress again the AIS are really smart
1:33:13more people have lost their jobs it's
1:33:14like really starting to actually affect
1:33:16things but I think still most people
1:33:17have their jobs but their jobs have sort
1:33:18of transformed so like by 2031 it's like
1:33:21most white collar jobs involve working
1:33:23with AIs to a large extent or managing
1:33:25teams of AIS or collaborating with them
1:33:27somehow also there are some things like
1:33:29robo taxis that are basically just
1:33:30working citizens dividend, you know,
1:33:32ideally this would happen sooner. Like
1:33:34in our scenario, they kind of do things
1:33:36at the last minute, you know, so like a
1:33:38lot of these policy things are like
1:33:39happening kind of like just in time.
1:33:42Obviously, we would recommend that you
1:33:43do them sooner and and do a better job
1:33:45of them too, but so 2033 you start
1:33:47getting your your checks from your
1:33:49dividend. So you're forecasting that
1:33:50there will be a citizens check that your
1:33:53model says it could be around 25,000 at
1:33:55the start per person
1:33:56>> and then it would grow as the economy
1:33:57grows.
1:33:58>> But also as I guess job displacement
1:34:00takes hold. They're going to need to to
1:34:01grow that. Check make sure you
1:34:03>> and that's why it's kind of the last
1:34:03possible moment because if you waited to
1:34:05implement this until like 2037 then like
1:34:08everyone would have already lost their
1:34:09jobs by the time that happens, right?
1:34:12people losing their jobs, especially if
1:34:14it happens quickly like like we see on
1:34:17this sort of graph here, is going to
1:34:19cause lots of problems in terms of civil
1:34:21unrest, social unrest, purpose, mental
1:34:23health, these kinds of things
1:34:24theoretically.
1:34:25>> Yes.
1:34:26>> How do you think about that?
1:34:27>> Uh it's it's going to be rough and
1:34:29hopefully we can navigate that well. We
1:34:31think that at a high level people need
1:34:33to have money and also people need to
1:34:35have power. And I think these are like
1:34:37somewhat different things. It's like why
1:34:38are jobs important? Well, there's a lot
1:34:40of reasons why jobs are important, but I
1:34:41think the main ones are um well, it's
1:34:43how people get money so they can survive
1:34:45and get the things that they want by
1:34:46buying the things that they want. So, if
1:34:47people are going to be losing their
1:34:48jobs, you need some other way of people
1:34:50getting money. And then there's also the
1:34:52power thing, which is that right now
1:34:54people have political power in part due
1:34:56to their economic power. people can
1:34:58threaten to go on strike for example or
1:35:01you know countries that are ruled by
1:35:03dictators can't
1:35:06just completely you know genocide an
1:35:08entire subop or they can but like it's
1:35:11costly for them to do so because
1:35:14then they'll have less money because
1:35:15that subop is contributing to their
1:35:17economy and contributing tax revenue and
1:35:18so forth but if you end up in a world
1:35:20where actually nobody's contributing tax
1:35:22menu revenue except for the AI companies
1:35:24and the robot companies then you're you
1:35:26the government are less incentivized to
1:35:29care about what you know the common
1:35:31people think. So, so when people lose
1:35:33their jobs, they're not just threatened
1:35:35with lack of loss of income. They're
1:35:37also threatened with loss of political
1:35:38power. And so, we think that it's
1:35:40important to like do things to push
1:35:42against that.
1:35:43>> What does that look like? How do you how
1:35:45do people have power in such a world?
1:35:47>> Well, in democracies at least, they
1:35:48still have votes.
1:35:49>> Okay. So I think that it's very
1:35:52important for there to be uh regulations
1:35:55on the use of AI that help make the
1:35:59public discourse more sane and more um
1:36:04actually giving the people what is in
1:36:05their interest and what they want and
1:36:07avoiding a sort of um opposite outcome
1:36:10where you know the masses are easily
1:36:14manipulated by AI powered media for
1:36:16example or where everyone's talking all
1:36:20to their AI advisers and the AI advisers
1:36:22are like subtly steering them away from
1:36:26voting for the candidate that would not
1:36:28be what the AI companies want because
1:36:30the AI companies have this other
1:36:32candidate that they like better and
1:36:33they're like secretly biasing their AIS
1:36:35to like steer people towards voting for
1:36:37that candidate, right? So, so we want to
1:36:38be in a situation where um people have
1:36:42AIs that are actually trustworthy and
1:36:44that are truth seeeking AIs, honest AIs
1:36:48and that don't have any sort of like
1:36:49political agendas put into them by the
1:36:51AI companies or by the government. You
1:36:52know, you want to avoid a situation
1:36:53where the AI company where where the
1:36:55government has issued some sort of
1:36:56secret order that like the AI have to be
1:36:58such and such a way. Yeah. The
1:37:00Department of War dispute versus
1:37:02Enthropic is like a an interesting sort
1:37:04of foreshadowing of this, right? where
1:37:07um Enthropic was giving their AIS to the
1:37:09Department of War. The Department of War
1:37:10wanted to use them for certain things
1:37:13and was upset that Enthropic's AIS were
1:37:15like not supposed to be used for those
1:37:17things. Uh that things in particular
1:37:19were domestic surveillance and uh
1:37:23autonomous robots.
1:37:24>> Mhm. There's going to be a lot more
1:37:26issues like that coming up and you want
1:37:27it to be the case that like people know
1:37:29what they're getting and that if people
1:37:30are like spending hours a day talking to
1:37:32their chatbot that chatbot doesn't have
1:37:34political biases put into it or a secret
1:37:35agenda or things like that and instead
1:37:37has been trained to like give honest
1:37:39true answers to things. And I think if
1:37:40you can do that it can improve the
1:37:42discourse and help people to use their
1:37:44votes to put even better regulations and
1:37:46even better politicians in place and so
1:37:48forth. it can sort of potentially
1:37:50bootstrap this to having something where
1:37:53people's power is even more secure than
1:37:54it is today.
1:37:55>> A lot of the stuff we've we've covered
1:37:57in part, so you know the wars and drones
1:37:59and missiles, we're already seeing this
1:38:00around the world at the moment, which is
1:38:02really really interesting. Um, and we've
1:38:05talked about robots outnumbering humans
1:38:07as well, which is part of this
1:38:08prediction. Some of the ones down here I
1:38:10found to be really curious, which is
1:38:12people will be protected by AIS wherever
1:38:14they go.
1:38:15>> Yeah. In this scenario, they delay the
1:38:18creation of super intelligence until
1:38:202040 and in fact they pause from 2035
1:38:23but then they let it go after then and
1:38:25then they let the AIS become vastly
1:38:26super intelligent. And we think that
1:38:30once the AIs are vastly super
1:38:31intelligent, the world will transform
1:38:33even more radically than what happens in
1:38:36the 2030s in this scenario. So in the
1:38:382030s in this scenario, it's more like
1:38:40human level. You know, the AIS are not
1:38:43they're they're doing the same sorts of
1:38:45things that human experts would have
1:38:46done. They're just doing it a bit
1:38:47better, a bit faster, and a lot cheaper,
1:38:49and there's a lot more of them. And the
1:38:51robots are still, you know, doing the
1:38:52same sorts of things that human workers
1:38:54would have done. There's just more of
1:38:55them, and they're cheaper. And because
1:38:57of exponential growth, uh you start with
1:39:00a world that looks not that different
1:39:02from today in 2029, and then by 2039,
1:39:04you end in a world that's radically
1:39:06transformed, where everyone's living in
1:39:07these like fancy new apartments that
1:39:09were built by robots two years ago.
1:39:10There's like giant special economic
1:39:12zones that are full of robots and solar
1:39:14panels and factories producing more
1:39:16robots and solar panels and factories
1:39:17and so forth. Most of the economy is AIS
1:39:20and robots and people don't have jobs
1:39:22anymore. That sort of transformation is
1:39:23what you get if you pause at human
1:39:25level. But if you go beyond the super
1:39:28intelligence, there's a whole another
1:39:30transformation coming that's going to
1:39:31look more like magic. Think about how
1:39:33the technology of today would look like
1:39:35magic to someone from 500 years ago.
1:39:37Mhm.
1:39:38>> You know, and that's without even like a
1:39:40qualitative improvement in intelligence,
1:39:42right? Like the humans of today aren't
1:39:44like qualitatively smarter than the
1:39:45humans from 500 years ago. It's just
1:39:47that we've had more time to do research
1:39:49>> and we have more like money and
1:39:50resources to build, you know, prototypes
1:39:53and experiments and run experiments and
1:39:54so forth. But if you had a point where
1:39:56there were billions and billions of AIs
1:39:59that were not only faster than humans,
1:40:01but like qualitatively way way way
1:40:03better at everything and in particular
1:40:05at doing scientific research, we should
1:40:07expect that some of the things that they
1:40:09develop will seem like magic to us
1:40:11>> and we'll just completely like we did
1:40:13not think that was even possible. You
1:40:15know, people don't want to die. People
1:40:16don't want to be hit by cars. People
1:40:17don't want to be like attacked by a
1:40:19random mass murderer.
1:40:21>> Cancer's gone. I mean not just cancer
1:40:23like [snorts] you know all all a lot of
1:40:25the stuff that happens in science
1:40:26fiction will probably have happened by
1:40:27then. So things like people scanning
1:40:29their brains and uploading into into
1:40:31computers right or self-replicating
1:40:34robots in the asteroid belt uh creating
1:40:37more and more satellites to
1:40:40uh produce more and more power to
1:40:41produce more and more self-replicating
1:40:42robots and so forth.
1:40:43>> Most people still live on Earth but the
1:40:45trend is to move to space.
1:40:47>> That's right. Yeah. So like if if you
1:40:49end up in the situation where the entire
1:40:52human economy is just like a tiny drop
1:40:55in the bucket that is the entire economy
1:40:58and it's just like this huge amounts of
1:41:00robots and AIs that are moving
1:41:03incredibly quickly, then what you want
1:41:05is Earth to be mostly left as something
1:41:08like a preserve. You know, I think a lot
1:41:11of people are worried about the
1:41:12environment being destroyed, which it
1:41:14totally would be if it wasn't protected.
1:41:16And uh you know there's a lot of people
1:41:18who sort of like their lives as it is
1:41:20and don't want to be uploaded or live in
1:41:23some crazy new future thing. And it
1:41:26seems to us like the reasonable solution
1:41:27to these issues is uh create new living
1:41:30spaces off the planet with some of that
1:41:33vast economic wealth and activity that's
1:41:36happening for the people who want that
1:41:37sort of thing and then that way the
1:41:39earth can be preserved.
1:41:41>> Data picture here of data centers in the
1:41:44ocean. Uh I mean there's three images
1:41:46there of different environments where
1:41:48humans might live.
1:41:50>> Again like our proposal was you preserve
1:41:53like 99% of the earth uh mostly as is as
1:41:57historic or environmental from historic
1:41:59or environmental reasons but then like
1:42:01some parts of it you designate as
1:42:02special economic zones where the robots
1:42:04can go crazy and dig giant pit mines and
1:42:07produce factories and so forth. Um, we
1:42:10were thinking it would be good to build
1:42:11the data centers on the ocean instead of
1:42:13um on land for a variety of reasons,
1:42:15although later space would be better and
1:42:19we could see that being reasonable as
1:42:20well.
1:42:21>> What about immortality in a world of AI?
1:42:24Um, 20 well 30 45 you say you've lived a
1:42:28dozen lifetimes and are immortal,
1:42:29passing from life to life as if by
1:42:32reincarnation.
1:42:34I mean, there's a lot of billionaires at
1:42:35the moment that are focused on
1:42:36longevity. I mean, Brian Johnson's said
1:42:38he's got this central rule, which is do
1:42:40not die right now.
1:42:42>> Yeah.
1:42:42>> Because we're in the age of AI and it's
1:42:44conceivable that with super
1:42:45intelligence, we'll be able to choose
1:42:46when we die.
1:42:47>> Yep. I think that's probably right. We
1:42:49don't depict that happening in this part
1:42:51because at this part they only have, you
1:42:53know, human level AIs, but that's one of
1:42:56those things that seems quite plausible
1:42:58that super intelligence could achieve um
1:43:01through a variety of means.
1:43:05What is your hope with all of this stuff
1:43:07now? Why did you do this? Why did you
1:43:09make this 2040 plan ape?
1:43:11>> In the like first week after we
1:43:13published 2027, it it blew up a lot
1:43:15bigger than we expected. By the way,
1:43:17like we actually made forecasts
1:43:19beforehand of like how many views it
1:43:21would get and stuff like that. And it
1:43:22was like 90th percentile outcome. So
1:43:25like um very much not what we expected.
1:43:28Um, but in like the Twitter storm that
1:43:30happened, various people were like, "Are
1:43:33why are you giving us all this like doom
1:43:34and gloom uh predictions? Like, how
1:43:37about a more positive vision of like
1:43:39what you think we should do instead?"
1:43:41And I think that that seed sort of like
1:43:43implanted in us and then we were like,
1:43:45"Yeah, that's reasonable." Like, we've
1:43:47sort of depicted what we think the
1:43:48default path looks like and why we think
1:43:50it's pretty scary. Now maybe we should
1:43:53switch facts and come up with some
1:43:55actual recommendations and then depict
1:43:57that as well
1:43:57>> even though you don't believe they're
1:43:58pro probable.
1:44:00>> Yeah. I mean you can vote for a
1:44:01political candidate even if you aren't
1:44:02confident that they're going to win, you
1:44:04know, and and you can say like here's
1:44:06what I think we should do even if you
1:44:08think that people are probably not going
1:44:09to do it. You shouldn't say this if you
1:44:11think it's completely unlikely. Like if
1:44:12you think there's no chance then like
1:44:14maybe you shouldn't bother. But we think
1:44:15there's a chance. like in particular for
1:44:17the reasons that we described in the
1:44:19scenario we think that people are going
1:44:21to wake up to the power of AI over the
1:44:24next few years
1:44:25>> because of something happens
1:44:27>> the companies are saying that they're
1:44:28going to do this
1:44:30>> and
1:44:31they are kind of on track and it just
1:44:34sort of makes sense that like if they
1:44:36get anywhere close to this level of AI
1:44:39then there's like big issues and big
1:44:41problems and like we need to like do
1:44:42something about this and so I think that
1:44:45Even if there's not any like very
1:44:47dramatic warning shot or something, I
1:44:50think that just naturally people are
1:44:51going to start paying more attention to
1:44:52this and reasoning through the
1:44:53implications and trying to predict
1:44:55what's going to happen. And so naturally
1:44:57people are going to be more interested
1:44:59in regulation of AI for example. And in
1:45:02fact
1:45:04there's actually like there's there's
1:45:06actually more of this happening than we
1:45:07predicted.
1:45:08>> More of what happening?
1:45:09>> Serious interest in reg AI regulation.
1:45:12So, at the time that we published AI
1:45:132427, the sort of like mainstream
1:45:16position of the tech companies and in
1:45:18the government was kind of like AI
1:45:19regulation, bad idea,
1:45:21>> free-for-all.
1:45:22>> Free-for-all. Yeah. In fact, there was
1:45:23even an attempt to um preemptively ban
1:45:27states from regulating AI.
1:45:28>> Yeah.
1:45:29>> You remember that? Now, it seems like
1:45:30the conversation has changed a lot. Like
1:45:32now the the US government just told
1:45:34Enthropic they have to shut down their
1:45:36AI because they were worried that bad
1:45:38actors would use it for cyber attacks.
1:45:40you know, the government is like waking
1:45:43up and doing more stuff than we expected
1:45:45uh already. And we're actually hopeful
1:45:48that that trend will just continue and
1:45:50that before it's actually too late,
1:45:52there will be very serious conversations
1:45:54happening inside the government and
1:45:55outside the government and in the
1:45:56broader society about all of these
1:45:58issues and trying to chart a course that
1:46:02is um avoids the loss of control and
1:46:05concentration of power risk that we
1:46:06mentioned. you um you've spent well must
1:46:09be almost coming up to 15 years thinking
1:46:11about this stuff. Um if this here was a
1:46:15button and if you press that button your
1:46:18plan S would occur and it would shut
1:46:21down every data center that is currently
1:46:24training a frontier AI model uh for
1:46:26good. There would never be any other
1:46:30>> AI labs um working on these problems.
1:46:33Would you press that button? I was I was
1:46:35about to slam it until you said for
1:46:37good.
1:46:38>> Okay.
1:46:39>> Like I think I think if it was a sort of
1:46:40temporary shutdown, I would totally slam
1:46:42that button because we are not ready to
1:46:44do this. You know, like civilization is
1:46:48not ready to have these companies
1:46:50automate themselves and then get smarter
1:46:52and smarter and then have the super
1:46:52intelligent like no. There's a bunch of
1:46:54reasons why that's really uh dangerous.
1:46:57But I would be at least hesitant to
1:47:00press this button if it permanently
1:47:02foreclosed the possibility of ever doing
1:47:04it again. for sure.
1:47:05>> But but if you think that plan D is
1:47:07probable, which is this race we're on to
1:47:09super intelligence,
1:47:10>> if I had a choice between D and S, I
1:47:12think I would press it.
1:47:13>> Well, it's it comes down to what you
1:47:14think, right? Cuz if you think that's
1:47:16that is what's going to happen, plan D
1:47:18and the only alternative.
1:47:20>> I didn't say this is what's going to
1:47:22happen
1:47:23>> proistically.
1:47:23>> Yeah. Yeah. Like like I'd be like this
1:47:25is the most likely, maybe this is the
1:47:26second most likely, maybe this is the
1:47:28third most likely. They are all
1:47:29possible. So with your current
1:47:30perspective on whatever one you think is
1:47:32going to happen, would you press the
1:47:33button? I'm giving you an S, a definite
1:47:35S or whatever you think is going to
1:47:36happen.
1:47:37>> That's tough.
1:47:43>> What is the scope of the shutdown? So is
1:47:45it
1:47:45>> it's no one can train an AI model again,
1:47:49ever again.
1:47:52>> That's real rough because like I said,
1:47:53there's loads of benefits that we could
1:47:54get from AI if we do it right. Um,
1:47:56>> I think I I've almost put you in the
1:47:58position of Sam Holtman.
1:47:59>> Yeah.
1:48:00>> To some degree.
1:48:01>> Yeah.
1:48:03>> Um, let me Do you mind if I just take a
1:48:05moment to think about this?
1:48:05>> I think I prefer you to think.
1:48:07>> Yeah.
1:48:16I think I would not press the button,
1:48:19but I'm I feel very torn about it. Um,
1:48:21the reason why I think I would not press
1:48:23the button is that I still have
1:48:25substantial hope that we can get
1:48:26something much better than this,
1:48:27something more like this. And I think
1:48:30that
1:48:32basically I think that if we don't build
1:48:34powerful AI systems eventually,
1:48:36then
1:48:38we're probably going to die as a
1:48:40civilization.
1:48:42eventually, you know, like 100 years
1:48:43from now, 200 years from now, something
1:48:44like that, like nuclear war, pandemic or
1:48:46something, you know,
1:48:49I I don't think human civilization right
1:48:51now is like super super stable. Um, and
1:48:55so I think that basically what I was
1:48:59about to say was the possible benefits
1:49:01for posterity and for all the billions
1:49:03and billions of people who could live in
1:49:05the future outweigh the like
1:49:09the current level of risk. But actually,
1:49:12>> I've heard that narrative before.
1:49:13>> Yeah. I don't know. Like,
1:49:15>> yeah, like maybe maybe it's just like,
1:49:17nope, the people right now are the
1:49:20people we should prioritize. People
1:49:21right now are in grave danger. They're
1:49:24going to be fine for at least the next
1:49:25couple decades.
1:49:27So, never mind posterity.
1:49:31Prioritize the people right now. Um, and
1:49:33people right now definitely don't want
1:49:36to do this lottery, I would say. Um,
1:49:41yeah, you've really asked me a tough
1:49:43question.
1:49:43>> So, would you press the button if that
1:49:45was the button?
1:49:47>> Probably not, but I would feel very
1:49:49torn.
1:49:50>> Okay. So, what I I always think about
1:49:53the personas of like the audience that
1:49:54are watching and these are, you know,
1:49:56they're they're very curious people,
1:49:57especially on the subject of AI as we've
1:49:58seen, but they they want to know like
1:50:01what it means for them. I think a lot of
1:50:02them also want to know what they can do.
1:50:04>> Oh, yes. Yeah. What can people do? Well,
1:50:07I think that if you either have talent
1:50:11or passion, you can get directly
1:50:13involved. There's lots of organizations
1:50:15that are worried about these things and
1:50:16that are trying to do something about
1:50:17it, like political advocacy or technical
1:50:20research or like building useful tools
1:50:23that will hopefully help people be
1:50:25better and stuff. But if you don't want
1:50:26to like make any major career changes or
1:50:29or things like that, then I would say
1:50:32just pay more attention to these issues
1:50:33and talk about them more with people. do
1:50:36stuff like, you know, emailing your
1:50:37congressman or whatever, it it doesn't
1:50:40change things that much, but it does
1:50:42help. I think that especially for this
1:50:44particular issue, the core problem is
1:50:46that people aren't taking it seriously
1:50:48yet. Like if the sorts of things that I
1:50:50was just saying to you for the last hour
1:50:52or two were just like top of everybody's
1:50:55mind,
1:50:57we wouldn't even be here. Like there
1:50:58there would already be much more
1:51:00significant regulation in place, you
1:51:02know, and not only would there be more
1:51:04heavy regulation in place, but there
1:51:06would have been better regulation in
1:51:07place that's less, you know, less like a
1:51:10cudgel and more like a scalpel and
1:51:11that's like more sensitive to what's
1:51:14actually bad and what's not so bad and
1:51:16so forth. And there'd be more expert
1:51:17people in the government and advising
1:51:19the government and so forth. So, just in
1:51:21general, like the more people wake up to
1:51:24these concerns and to these projections,
1:51:27uh, I think the more likely it is that
1:51:29we can do good stuff before it's too
1:51:31late.
1:51:31>> What about how they should vote at the
1:51:32polls? We've got an election coming up
1:51:34in the United States in a couple of
1:51:36years time, but there's elections
1:51:37happening all over the world all the
1:51:38time.
1:51:39>> You should ask your candidates what they
1:51:41think about all this AI stuff. You
1:51:42should try to get them to like have
1:51:44opinions, and then you should vote for
1:51:45the candidates whose opinions are better
1:51:47on this topic. This is the most
1:51:48important thing happening. uh in our
1:51:51lifetimes, probably in all of history in
1:51:52fact and it's very important that it go
1:51:55well and so it's what all the all the
1:51:57leaders of all the countries should be
1:51:58thinking about and making plans for.
1:51:59>> Isn't it such a weird thing to be alive
1:52:01at this moment in time? Like I was
1:52:03thinking about all the times that I
1:52:04could have been born and I guess my
1:52:05ancestors probably thought the same, but
1:52:07I was thinking as you were speaking I
1:52:08was like I think it's when you referred
1:52:09to it as like the final show.
1:52:11>> Yeah.
1:52:12>> What was the phraseology you used?
1:52:13>> I said the the clim run up to the climax
1:52:15or something.
1:52:16>> Yeah. I mean, what a what a crazy thing
1:52:19to be born in the run-up to the climax
1:52:21where everything you're describing here
1:52:23is within my lifetime, conceivably.
1:52:25Hopefully.
1:52:25>> Yeah.
1:52:26>> Um or maybe not hopefully.
1:52:28>> What a crazy time to be alive.
1:52:30>> Certainly.
1:52:31>> I noticed that when I met asked you if
1:52:32you had kids, your demeanor changed
1:52:34quite considerably.
1:52:35>> Well, it's Yeah.
1:52:37>> It's like you dropped into a different
1:52:38state. Obviously, that's been central to
1:52:41the
1:52:43rumination that you've been
1:52:44experiencing. Well, it's a sad topic,
1:52:46right? Like when when I had kids, like
1:52:49the reason to have kids is in large part
1:52:51about the future, you know, like it's
1:52:54not just like a cuddly thing to have
1:52:55with you in the moment. It's because you
1:52:57have all these hopes and dreams about
1:52:58how they'll grow up and how they'll go
1:53:00do their own thing and be their own
1:53:01person and stuff. And because of what's
1:53:04happening with AI, I think a lot of
1:53:05those dreams are in jeopardy.
1:53:06>> Presumably, you still would have had
1:53:08kids.
1:53:08>> I've actually flip-flopped on this
1:53:10occasionally. Yeah. Basically, the
1:53:11topline answer is I'm not sure. the my
1:53:15first child was had we we had her when
1:53:17we were um in 2019 she was born 2019.
1:53:20>> Yeah.
1:53:20>> So this is before my timeline shortened
1:53:22a lot. So at this at this point I was
1:53:23interested in AI I was tracking the
1:53:24field. I was making forecasts but I
1:53:27didn't like actually expect it to happen
1:53:28soon
1:53:29>> you know.
1:53:30>> Mhm. [clears throat]
1:53:30>> And then this caused like when I when I
1:53:33did start thinking like oh my gosh it's
1:53:34going to be happening like real soon um
1:53:37like by 2030 you know. um that caused
1:53:41some reconsidering and so I basically
1:53:44told my wife like let's not have any
1:53:45more kids. It's too uncertain, you know.
1:53:49But that turned out to be really hard
1:53:50because especially for my wife, like we
1:53:52already had one kid and like no
1:53:55siblings. Um so eventually I sort of
1:53:59gave in and was like, "Okay, well, you
1:54:00know what? We already have one. It's
1:54:02gonna be all right. Like maybe maybe the
1:54:04future will be good. And even if it's
1:54:06not like, well, we're all in the same
1:54:08boat together.
1:54:10>> It's quite chilling what you're saying.
1:54:12It's chilling because you know more than
1:54:13me. And if you're at home saying to your
1:54:15wife, listen, maybe we should pause on
1:54:17having more children and building a
1:54:18family because of what's going on with
1:54:19AI.
1:54:21>> To be clear, this is Yes. I mean, yes,
1:54:22it's very concerning. [laughter] I am I
1:54:24am chilled. Uh this is bad. This is what
1:54:27I've been saying. I hope things go well.
1:54:29I think things might go well. Um I think
1:54:31that there's a lot we can do to like
1:54:32steer things in a better direction. I
1:54:34mean, one of those things as well, I
1:54:35have to say, is just speaking about it.
1:54:38It's I think a lot of the progress we've
1:54:40seen with governments waking up and, you
1:54:43know, we've seen certain things with
1:54:44people booing certain people at certain
1:54:45events.
1:54:46>> Yeah.
1:54:46>> Um is is a downstream from people like
1:54:49yourself actually coming on shows like
1:54:51this and all the other podcasts and
1:54:53>> Yeah.
1:54:54>> telling us what's going on.
1:54:55>> Yeah.
1:54:55>> Because elsewhere, to be fair, we're
1:54:57going to be gaslighted by the people
1:54:58that have the biggest PR machines.
1:55:00>> Yeah. So, um, I often I think it's
1:55:02probably worth me saying I find myself
1:55:04kind of in two minds because I'm an
1:55:05entrepreneur and I'm I'm an investor.
1:55:07I'm an investor in probably more than
1:55:08100 companies now. And so many of those
1:55:11companies are using AI. I invested in
1:55:12Grock, the inference chip company. I've
1:55:14invested in SpaceX which now own another
1:55:16Grock and they're doing AI. I use AI
1:55:18every day in my life. I've been using it
1:55:20through this conversation to understand
1:55:21different things that you've said.
1:55:22>> So that's one side of me which is like
1:55:24business builder entrepreneur who has
1:55:27seen the benefits of AI in my own life.
1:55:29And then there's the other side of me.
1:55:30And it's funny because I think sometimes
1:55:31people think you have to pick a camp.
1:55:33But through all of my life, even when I
1:55:35was a social media CEO and I was saying,
1:55:36by the way, listen, I'm building a
1:55:37social media business, but I think
1:55:38there's some downsides to social media.
1:55:40I find myself at the same moment where
1:55:41I'm like, I build with AI. I have AI
1:55:43investments. And at the same time, as a
1:55:46civilian, I'm like,
1:55:47>> yeah, I mean, I think that is attention.
1:55:49I think that there's there's different
1:55:52ways you can draw the line. So, and I
1:55:54know lots of people who draw the line in
1:55:56lots of different ways. So, like there's
1:55:57some people who just like, I'm not going
1:55:58to use AI. I think this stuff is bad um
1:56:01and on a bad trajectory, so I'm going to
1:56:02like boycott AI, right? I'm not one of
1:56:04those people. I use AI a lot. We all do
1:56:06at AI features project. Um it's helpful
1:56:09for a lot of our work. The opposite end
1:56:11of the spectrum is
1:56:14people being like, well, it seems like
1:56:16it's on a trajectory to happen, so the
1:56:18thing to do to make it go well is to
1:56:20like get involved and accumulate power
1:56:22and try to like steer it from the
1:56:23inside. Mhm.
1:56:24>> And so I'm going to go work at OpenAI or
1:56:26Anthropic and like try to like climb the
1:56:28ranks and then like you know be someone
1:56:30who matters when the important decisions
1:56:32are being made. And I know loads of
1:56:33people like that. That was like what I
1:56:35was doing when I was that wasn't what I
1:56:36was doing exactly but like
1:56:38>> that was the path
1:56:38>> that was like that was I mean this in
1:56:40some sense this is what the whole
1:56:41narrative of the companies are right
1:56:42like this is why they tell themselves
1:56:43it's okay to do what they're doing is
1:56:45that they're worried about the other
1:56:46guys you know and so like all these
1:56:48people are are deciding like we're going
1:56:50to like lean really hard into it. are
1:56:52going to like be there in the room when
1:56:54the important decisions are being made,
1:56:55you know? So, there's a whole spectrum
1:56:56and I'm sort of like somewhere in the
1:56:58middle. Like, I'm not at the AI
1:56:59companies. I'm not helping them go
1:57:01faster. Instead, I'm talking to the
1:57:04broad public and trying to advocate for
1:57:06what I think is the my current best
1:57:09guess as to the way out, you know, the
1:57:11way forward. Um, but I'm not like
1:57:13boycotting all the AIs. I'm I'm not
1:57:15like, you know, uh trying to I'm not
1:57:17refusing to like engage with it in that
1:57:19way.
1:57:20Do you think it's too late?
1:57:22>> No, I don't think it's too late. If I
1:57:25thought it was too late, I wouldn't be
1:57:26here. H
1:57:27>> Where would you be?
1:57:28>> With my family.
1:57:31>> What's your closing message to the
1:57:33general public? If you had to have a
1:57:35closing statement to them,
1:57:37>> maybe I would say that like you're going
1:57:39to hear a lot of things and you already
1:57:40have been hearing a lot of things about
1:57:43AI and it's going to sound like science
1:57:45fiction, but sometimes things which
1:57:48sound like science fiction happen in
1:57:49reality. And in fact, many times
1:57:52historically things used to be science
1:57:54fiction have then become reality. And
1:57:56people need to stop thinking about what
1:57:59does or doesn't sound like science
1:58:00fiction and just start thinking about
1:58:01like the trends and you know the actual
1:58:05trends that this technology is on and
1:58:08reading and forecasting how it's going
1:58:09to go and then taking seriously the
1:58:11possibility that it could go something
1:58:13like this and then thinking about what
1:58:15should be done about that.
1:58:16>> And where would you direct them to get
1:58:18more information?
1:58:19>> You can go to a27.com to read our
1:58:22previous scenario. You can go to
1:58:24a2040.com plan A to read our new
1:58:27proposal for what is to be done. Um,
1:58:29these things are not just a sci-fi
1:58:32story. They also have lots of like
1:58:34explainers and links to other things.
1:58:36And so they're kind of like a nice
1:58:37jumping off point to to learn about all
1:58:40of this stuff. M
1:58:42>> um if you want I could um after this is
1:58:44over like give a reading list of like
1:58:46other papers and articles and
1:58:50>> blogs to follow and so forth
1:58:52>> and I'll link them all below in the
1:58:53comment section. So if you're listening
1:58:54now go ahead and take a look at the
1:58:56comment se the um description of this
1:58:58episode and you'll see a bunch of links
1:59:00which is Daniel's recommendations of
1:59:01what you should read. You know I think
1:59:03it's it's just a really really great
1:59:04moment in time to get educated on this
1:59:06stuff. Um humans have an inclination
1:59:09because of cognitive dissonance where we
1:59:10feel uncomfortable about something to
1:59:12bury our heads in the sand and avoid it.
1:59:15But actually I think this is one such
1:59:17time to do the very opposite for many
1:59:18reasons to to inform yourself so you
1:59:21know what actions to take but also
1:59:22because AI you know unavoidably is going
1:59:25to be a huge part of all of our lives
1:59:26and careers.
1:59:27>> Yeah. Yeah. Thank you. And that that's a
1:59:29good way to to say it. It's going to
1:59:31matter a lot. It's going to it's going
1:59:33to be everywhere soon and um we need to
1:59:36do something about it before it's too
1:59:37late. And
1:59:37>> what about AI future project?
1:59:39>> That's our organization. We spent a year
1:59:41writing AI 2027 after I left OpenAI and
1:59:43then we spent another year writing AI
1:59:452040 plan A.
1:59:46>> Daniel, thank you.
1:59:48>> Thank you.
1:59:48>> Thank you for all the work that you do.
1:59:49I can see how much you care about this
1:59:50stuff and it's your care. It's funny.
1:59:52Care itself makes others feel care. and
1:59:55um seeing how personal this is for you
1:59:56and seeing how much you've dedicated
1:59:57your life to this, but also hearing that
1:59:59you you basically walked away from $2
2:00:02million to be able to speak to the
2:00:04public about this information is
2:00:05incredibly admirable and uh I I think
2:00:08voices like yours are more important now
2:00:10than they've ever been on this subject.
2:00:12So, please do keep fighting the fight
2:00:13that you're fighting. And that's one of
2:00:14information. It is of honesty and it is
2:00:16uh of saying what what is often the
2:00:18quiet part out loud and doing really
2:00:21really smart research. I'll link
2:00:22everything we've discussed today below.
2:00:24Um, and I hope we can chat again
2:00:25sometime soon.
2:00:26>> Thank you.
2:00:27>> YouTube have this new crazy algorithm
2:00:28where they know exactly what video you
2:00:31would like to watch next based on AI and
2:00:33all of your viewing behavior. And the
2:00:35algorithm says that this video is the
2:00:38perfect video for you. It's different
2:00:40for everybody looking right now. Check
2:00:41this video out and I bet you you might
2:00:43love

About this transcript

This page contains the full transcript of ChatGPT Whistleblower: They Offered Me $2m To Keep Quiet, No One Is Ready For What's Coming! by The Diary Of A CEO, generated from the public captions YouTube serves with the video. The transcript has 25,395 words across 3,771 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.