EMERGENCY PODCAST: ASI Will Kill Us All! — Transcript
Full transcript
- 0:00You believe that if we pursue super
- 0:03intelligence, the risk that that
- 0:05destroys humanity is so high that we
- 0:09should stop that pursuit immediately?
- 0:12>> Yes.
- 0:12>> Okay. Do you believe that the correct
- 0:15path is a non-development agreement
- 0:18between the US and China?
- 0:20>> Yes. Specifically when it comes to super
- 0:22intelligence, I do not think this
- 0:23applies to all forms of AI. One things
- 0:25that lobbyists love to do is they love
- 0:26to try to equivocate all AI as the same
- 0:28technology. This is obviously nonsense.
- 0:30Here's a fun fact. Did you know that you
- 0:32could buy uranium ore on Amazon.com?
- 0:35>> I did not know that. Terrifying. You
- 0:36could ship it to your house, put on your
- 0:38desk. It's perfectly harmless.
- 0:39>> Natural uranium or is very harmless.
- 0:41It's perfectly safe to touch.
- 0:43>> Okay.
- 0:43>> But obviously, highly weaponsenriched
- 0:46uranium is extremely illegal.
- 0:48>> Right.
- 0:48>> This is also how I think about AI.
- 0:50>> Okay.
- 0:51>> Do you already know where you would draw
- 0:52the line? I have some suggestions but
- 0:54fundamentally this in a sense we can
- 0:57take a step back or maybe that's one of
- 0:58the questions still to come is where why
- 1:00do I think these risks exist how what do
- 1:02I think their shape is and a lot of the
- 1:04risks I see where they come from is
- 1:06fundamentally that we don't understand
- 1:08AI
- 1:09>> AI is very different from normal
- 1:11software I'm a computer scientist by
- 1:13background I've written a lot of
- 1:14software in my life you know even
- 1:16without agentic help and most software
- 1:19is written with code line by line you
- 1:21tell the computer exactly what to do.
- 1:23This is not how AI works. AI is more
- 1:27like grown rather than written through a
- 1:29technique that's called neural networks.
- 1:31You have these massive piles of data and
- 1:34you kind of have a program self assemble
- 1:36itself, grow, learn from this data. And
- 1:39what comes out the other side isn't
- 1:41lines of code that you could read or
- 1:43understand what's going on. It's more
- 1:44like billions and billions and billions
- 1:47of numbers. If you multiply and add all
- 1:49those numbers in the right order, you
- 1:51get JBT. How do you begin though to
- 1:53parse whether something falls into the
- 1:55yes bucket or the no bucket? It isn't
- 1:58just that it uses a neural net
- 1:59presumably.
- 2:00>> Yes.
- 2:01>> Okay. So if it's not neural net, is it
- 2:04size of the data center?
- 2:06>> Fundamentally I think it's a behavior
- 2:08classification in the sense is there's
- 2:10obviously a category of things that are
- 2:13can out compete humanity. The way that's
- 2:15what I like to think think about it. I
- 2:17like to think about is that like if you
- 2:18build a system of any kind, it can be a
- 2:20single agent, it can be a swarm, it can
- 2:21be, you know, can have many shapes. If
- 2:23you build a system that can out compete
- 2:25humanity at all relevant tasks, every
- 2:27time you start a business, you go out of
- 2:28business because an AI business out
- 2:30competes you. Every time you trade on
- 2:31the stock market, you lose all your
- 2:32money to an AI hedge fund. Every time
- 2:34you run a political campaign, you lose
- 2:36to the other guy who's using an, you
- 2:38know, AI adviser. Every time you run a
- 2:40military campaign, you lose to the PE
- 2:42person who's using more autonomous AI
- 2:43systems. If you get into a world like
- 2:45this, power of all kinds, economic,
- 2:48military, political, etc., is
- 2:50concentrated in the hands of non-human
- 2:52entities in AI. And if we don't control
- 2:55these systems, if we don't understand
- 2:57these systems, and if they don't have
- 2:59our best interests at heart, it's very
- 3:01hard to imagine that going well.
- 3:02>> Okay, it seems like we're already there,
- 3:04but the AI is a tool. So the way that I
- 3:08think about AI right now is if somebody
- 3:10has AI and they compete against somebody
- 3:12in the same thing without AI, the person
- 3:14with AI will win effectively 100% of the
- 3:16time. Do you disagree with that?
- 3:17>> I agree with that. What I disagree with
- 3:19is I think if a human against an AI
- 3:21without a human, the human still wins.
- 3:22>> Totally agree.
- 3:23>> Yes. What I'm saying is what happens
- 3:25when a human versus an AI alone an AI an
- 3:27autonomous AI and the autonomous AI wins
- 3:29every time.
- 3:30>> So it's the autonomous part that's
- 3:32important for your definition.
- 3:33>> Yes. I think autonomy is very important
- 3:35here. When I say autonomy, I mean, you
- 3:37know, both in the obvious sensor, but
- 3:39also like obviously if you just have a
- 3:40human rubber stamp, yes, on everything,
- 3:42that doesn't count as oversight. Fair.
- 3:44So, that would still count as autonomous
- 3:46to me. Even if there's technically a
- 3:47human involved, but the human doesn't
- 3:49even understand what the system is doing
- 3:50and they're just rubber stamping all the
- 3:52decisions.
- 3:53>> Like what I expect happen for example is
- 3:55that they're probably will to be human
- 3:57CEOs for quite a while, but they just
- 3:59rubber stamp what the AI CEOs tells them
- 4:01to do. They're just there for legal
- 4:02representation. So for you, we would
- 4:04have then long past where we we're now
- 4:08beyond the point of no return and and
- 4:09all is for not.
- 4:11>> Yes. I like thinking about the point of
- 4:12no return much more than like think
- 4:14about like the point of like extinction
- 4:15or doom in a sense.
- 4:17>> Why?
- 4:17>> Because that's the point that actually
- 4:19matters.
- 4:20>> Okay. So it's like stakes versus where
- 4:21we are on the timeline.
- 4:23>> Yeah. In a sense the what matters to me
- 4:25is the point of no return. Once we've
- 4:26hit the point of no return where there's
- 4:28so much AI, it's so powerful. It's so
- 4:30intelligent. It's self-improving. You
- 4:32know, there's not just one AI. There are
- 4:33millions or billions of them.
- 4:35>> Even if we're not dead, even if we
- 4:38continue to maybe live for a couple
- 4:39years or even decades, we no longer have
- 4:42any control over the future and there's
- 4:44no way for us to get it back. Okay,
- 4:46you've already slipped in a base
- 4:48assumption there.
- 4:49>> Um, so I before we get to that base
- 4:51assumption, because it is certainly on
- 4:53the list of where this sort of breaks
- 4:55bad, a very clear definition. So,
- 4:57autonomous AI that can outperform humans
- 4:58at everything. That's the one that we're
- 5:00worried about. Uh, we need to understand
- 5:02the mile marker, which is what's the
- 5:04point of no return. You've talked about
- 5:05this before, so I think we can hit this
- 5:07one quick, which is we're close. You
- 5:09don't quite know how close. We're so
- 5:11close, it's dangerous, we should stop
- 5:13now.
- 5:13>> Yes.
- 5:13>> Okay. If I gave you a big red button and
- 5:16you could unilaterally stop everybody
- 5:18from developing any further. We can use
- 5:19what we already have, but we cannot
- 5:21progress any further, would you hit it
- 5:22today? Would you wait a week, a month, a
- 5:24year? I like to say if we as a species
- 5:26or as a civilization were three steps
- 5:28wiser than we were today, we wouldn't
- 5:30have done Chachi PT in the first place.
- 5:32>> What are the steps? What do you mean
- 5:33steps?
- 5:34>> It's just a it's just a
- 5:36>> three clicks, three just a little bit
- 5:38smarter.
- 5:39>> Three generations of wisdom. And
- 5:41[clears throat] if we if we were two
- 5:42steps wiser than we currently are, we
- 5:44would have done chatgpt saw that it did
- 5:46all these crazy things that none of our
- 5:48scientists predicted and it got, you
- 5:49know, 100 million users over time and
- 5:51like whoa whoa whoa whoa whoa. What the
- 5:52hell's going on here? and would have
- 5:54rolled back tried to figure it out and
- 5:56if we were one step wiser we would stop
- 5:57today.
- 5:58>> The original question about US China is
- 6:00trying to get to is I my belief is that
- 6:03you see this as a state level thing and
- 6:06that law enforcement is going to be the
- 6:08punchline. Um meaning that just like
- 6:10with nuclear if we found that somebody
- 6:13were trying to build a nuclear weapon we
- 6:15would immediately stop them. That's
- 6:16against the law. We've got things around
- 6:18that already. We need to do that for AI.
- 6:21>> Yes.
- 6:21>> Okay. This this is going to be a big
- 6:23part of where I look at this and I go I
- 6:25think that there's a problem there.
- 6:27Meaning you're never because it is we
- 6:30agree that this is a state level issue
- 6:32between the US and China specifically.
- 6:34Now is your chance to say there's
- 6:36somebody else.
- 6:37>> I want to say the EU but I can't.
- 6:39>> Okay. So
- 6:40>> with my heart.
- 6:41>> Yeah. Yeah.
- 6:42>> That was a joke. I I I've lived in the
- 6:44Europe for many years. I love the EU.
- 6:45>> They're amazing. My wife is European. I
- 6:47totally get it. Um but we really have
- 6:50this is a US versus China thing. Okay.
- 6:52So you put forward a trust but verify
- 6:54situation.
- 6:55>> Um much like nuclear or something that
- 6:58has a very different approach to the
- 7:00trust but verify.
- 7:01>> I think there's a lot of inspiration to
- 7:03be had from the nuclear example. I think
- 7:05the history of nuclear regulation is the
- 7:07closest to there is a great crazy new
- 7:10technology that will change everything
- 7:12and we rose to the occasion. Now, I'm
- 7:14not saying we fixed everything, but you
- 7:16know, I when I think about regulating AI
- 7:18and like the history of AI, I like to
- 7:20think back to Leo Zard, the person who
- 7:22came up with the nuclear bomb.
- 7:23>> When he first did the math, he realized,
- 7:26oh it's possible.
- 7:28He did the math and he was like, "Oh my
- 7:30god, it's definitely possible." So, what
- 7:31he did is he moved to San Francisco,
- 7:33started a nuclear bomb startup, which
- 7:34split into three other startups to sell
- 7:36as many nuclear bombs to the public as
- 7:38possible. No, of course that's not what
- 7:39happened. He went to the government. He
- 7:41went to the military. He was like,
- 7:43"Guys, this will change everything. We
- 7:46have to do something. We have to figure
- 7:47this out." And this is what didn't
- 7:49happen for AI. What didn't happen for AI
- 7:51is that we didn't realize that this will
- 7:53change our civilization. This is not a
- 7:55private matter. This is a national
- 7:56security matter. This is a matter of the
- 7:58the organization of our economic
- 8:00structures. This is a political
- 8:02question. This is a government. This is
- 8:03an intergovernmental question. And we
- 8:06just didn't do that. And by mu we I mean
- 8:09the scientific community like the
- 8:11scientific and the technical community
- 8:12like the people who started companies
- 8:14like you know deep mind or anthropic or
- 8:16openai they weren't Leo Zillard they
- 8:19went to go make a bunch of money off of
- 8:20it they didn't go to the government and
- 8:22were like we need to build new
- 8:23institutions to responsibly steward this
- 8:25powerful technology and so we didn't do
- 8:28it you know at the time when nuclear was
- 8:30first considered the idea that you would
- 8:33have like stuff like you know the
- 8:34international atomic energy agency and
- 8:36so on was unthinkable such such a thing
- 8:38had never existed in the history of any
- 8:40legal code before. They had to invent
- 8:42whole new forms of law to deal with this
- 8:45new technology. And that makes sense. If
- 8:48you have an incredibly powerful new
- 8:49technology, you have to steward it
- 8:51responsibly. And that will require
- 8:53coming up with new doctrines with new
- 8:55ways of like what level of risk do we
- 8:56tolerate in various ways? Where do the
- 8:58risks come from? Who decides the level
- 9:01of risk that we take with these relevant
- 9:03technologies? And at the moment, we're
- 9:05just not making these choices at all.
- 9:07It's not even that we're making the
- 9:08wrong choice. It's that
- 9:10>> we're not making these choices at all. I
- 9:12would love if there was like
- 9:13disagreement, for example, between POTUS
- 9:15and Xiinping about what the correct
- 9:16level of extinction risk is. You know,
- 9:19if they're arguing, oh, it should be
- 9:20this percent or that percent. I think
- 9:21this would be a really good world to
- 9:23live in because then we could have this
- 9:24argument. But we're very far even from
- 9:26that point that we're not even making
- 9:28these decisions yet. Be right back to
- 9:30the show in just a second. But right
- 9:32now, let's talk about a problem the
- 9:34military couldn't solve with willpower.
- 9:36Picture a soldier making life ordeath
- 9:38decisions after 36 hours without sleep.
- 9:41This is high stress. There's obviously
- 9:43no margin for error. The military needed
- 9:46their people thinking clearly, even
- 9:48under conditions that would break most
- 9:49of us. So, they funded a $6 [music]
- 9:51million Department of Defense research
- 9:53contract to find the answer. What they
- 9:56found was ketones, the brain's most
- 9:58efficient fuel source. That research
- 10:01became ketone IQ. Ketones [music] cross
- 10:03the bloodb brain barrier and fuel your
- 10:05neurons directly. They enhance
- 10:07mitochondrial function for longer mental
- 10:10stamina. I take it for interviews,
- 10:12before writing, before I do coaching,
- 10:14and I can feel the difference. [music]
- 10:16Go to ketone.com/impact
- 10:20for 30% off your [music] subscription
- 10:22order or visit your local Target to get
- 10:24your first shot for free. That's
- 10:27ketone.com/impact.
- 10:30[music] Now, let's get back to the show.
- 10:32Okay, if Elon can be believed, he said
- 10:36for years that he tried to lobby the
- 10:38government to let them know this is
- 10:40going to be a real problem. This is he
- 10:41referred to it as a demon summoning
- 10:43circle and we're all just hoping that
- 10:44the demon that we summon is going to do
- 10:46our bidding. And he said he just could
- 10:48not get anybody to take him seriously.
- 10:50And so he became fatalistic. That's when
- 10:52he tries to start open AI because he's
- 10:54like, "Well, if it's going to exist, I
- 10:56really don't want uh Larry Page at
- 10:59Google to be the only one making it
- 11:01because he's whatever a specist or that
- 11:03was the accusation to Elon." And so Elon
- 11:08has again, if he can be believed,
- 11:10attempted to do a maybe an early
- 11:13prototype of the very systematic way
- 11:15that you're going about it now. So, do
- 11:18you think it was overwhelm and confusion
- 11:21that led people to do that? And because
- 11:25I have a different take on the um
- 11:27nuclear analogy, which is where I'm
- 11:29getting to. Do you think Elon wasn't
- 11:31good at it? People were overwhelmed. But
- 11:35why didn't that take hold?
- 11:37>> I think an average political institution
- 11:40building campaign takes 20 years. I
- 11:42don't think Elon spent 20 years on this.
- 11:44I don't know. Maybe he did and I'm not
- 11:45aware of Why when the guy that
- 11:47understood the nuclear thing, why did he
- 11:49get immediate attention when he went to
- 11:50the
- 11:50>> Oh, no. It took decades.
- 11:52>> Really?
- 11:52>> Yes.
- 11:53>> So, he was banging and banging and
- 11:54banging.
- 11:54>> He was banging. So, he was banging for
- 11:56years. Then there was the Einstein
- 11:57letter which led to the Manhattan
- 11:58project, but until the IAEA was created
- 12:01and all these other things that took
- 12:03decades of work by international
- 12:05diplomats, you know, by lawyers and by,
- 12:07you know, statesmen, you know, between
- 12:09this was when the Soviet Union was still
- 12:11around, right? Imagine how hard it was
- 12:13to get the Soviet Union to agree to
- 12:15something like this. This was a decadesl
- 12:17long project that also kept evolving,
- 12:19you know, throughout the 60s, the 70s,
- 12:21the 80s. And so, and this was not a
- 12:22thing that was like, oh, Zlord, you
- 12:24know, had one Zoom call and then the
- 12:25thing was solved. Quite quite the
- 12:27opposite. It required thousands, tens of
- 12:29thousands of scientists, diplomats,
- 12:32politicians, etc. working for decades.
- 12:34So, you know, I I applaud Elon for
- 12:37having tried, but fundamentally, a thing
- 12:39that I think is very frustrating to tech
- 12:41people is that non- tech things run at a
- 12:44different pace is that if you want to
- 12:46convince, say, a thousand diplomats to
- 12:48take an issue seriously. There's kind of
- 12:50like a hard limit on how fast you can do
- 12:52that. I think it's faster than people
- 12:53think it is, but it's still longer than
- 12:55like a couple years. It takes time and
- 12:57it takes effort and it takes a lot of
- 12:58money. I'm not aware of Elon spending a
- 13:01billion dollars on political advertising
- 13:03or on AIX risk. He could have done this.
- 13:05He obviously has the money. He can just
- 13:06spend a billion dollars on political
- 13:09advertising to uh make this issue known.
- 13:11He could fund 50 think tanks to work on
- 13:14existential risk of AI. He has the money
- 13:16to do it. So in this regard, I don't
- 13:18know. You know, Elon, you know, I I
- 13:20think he's an extremely precient and
- 13:21extremely competent person. I think if
- 13:23he really wanted to, he could. I think
- 13:26he absolutely could. he could fund us.
- 13:28You know, he we're getting a lot done
- 13:30with the money we're doing and if he
- 13:32gave us 10x more money, we would get 10x
- 13:33more done. So, I don't know why he made
- 13:36the choices he made. We can
- 13:37psychoanalyze all day. But
- 13:39fundamentally, one of the things that we
- 13:41learned when we started doing this is
- 13:43that every single person in the world
- 13:44told us this is impossible. No one will
- 13:46listen. And we've tried already and
- 13:48we're like we'll see. And then we just
- 13:50talked to politicians and overwhelmingly
- 13:53the response we got was oh wow. I had no
- 13:56idea. No one told me. Thank you for
- 13:57telling me. This seems really bad. I
- 14:00don't think people have tried. Uh
- 14:02because politics is slow. It's
- 14:04frustrating. You have to repeat
- 14:05something. You know, in advertising, you
- 14:07usually say to get a message across, you
- 14:09have to repeat it seven to 10 times per
- 14:12person on over this, you know, a
- 14:14schedule like 6 to 12 months.
- 14:16>> One conversation will never be enough.
- 14:17You will have to talk to that person
- 14:18seven to 10 times until they can start
- 14:20to really understand these. And this is
- 14:22frustrating. This is annoying. It takes
- 14:24a lot of time. So, no, I think there's a
- 14:26lot that can be done.
- 14:27>> All right. So, as a marketer, one of the
- 14:29reasons that I think that this stalled
- 14:31out, one of the reasons that I think
- 14:32nuclear ended up with, if we tell
- 14:34ourselves a wrong story about what
- 14:36happened there, it will seem like, oh,
- 14:38this was just another case of we
- 14:39lobbied, lobbyed, lobbyed, lobby, he was
- 14:40ignored in the beginning, but then we
- 14:41finally got through to people. Is that
- 14:44um when you're marketing, when you're
- 14:45doing sales, you're trying to elicit an
- 14:46emotional response. If you can't get the
- 14:48emotional response, if you can't create
- 14:50a sense of urgency that people feel, not
- 14:51understand, but feel, they're never
- 14:53going to move. And so when I look at the
- 14:56nuclear question, to me that was a
- 14:58failed version and we just sort of got
- 15:01off lightly that nobody after um what we
- 15:05did to Japan that everybody saw the
- 15:07level of destruction and just said whoa
- 15:09we got to chill on this and not actually
- 15:11move forward. But to me that was by
- 15:14seeing the devastating power of it by
- 15:17the scientists themselves suddenly
- 15:19asking themselves what have I done and
- 15:20sort of waking up with that on their
- 15:22conscience. Then people were like,
- 15:24"Okay, yo, we got to chill on this." But
- 15:26the reality was, if I'm not mistaken,
- 15:28and please tell me if you know something
- 15:30I don't, but they when they were doing
- 15:32the first test, they knew there was a
- 15:34nonzero chance that they would light the
- 15:36entire atmosphere on fire and kill
- 15:38everyone. Yep.
- 15:38>> And they still did it.
- 15:40>> Yes. But they did calculate the risks
- 15:42and it was like a billionth of a
- 15:43percent.
- 15:44>> Okay. So, if you had been in that room
- 15:47and you run it and you see it's a
- 15:49billionth of a percent and I know you
- 15:50don't like to be unilateral, but what
- 15:52way would you cast your vote? Would you
- 15:53say don't do it or would you be like,
- 15:56"Ah, that's a tiny fraction. Go for it."
- 15:58>> Generally, my my uh vibe is my my rule
- 16:01for my personal life is that something
- 16:02is less risky than driving a car, it's
- 16:04fine.
- 16:05>> Okay. So that one feels less risky even
- 16:07though the consequences are so
- 16:10catastrophic that as long as it's just
- 16:13an odds question.
- 16:14>> There's a thing where um obviously I'm
- 16:17not a physicist. So I'm not sure how
- 16:19much I would have believed these numbers
- 16:20if I was a physicist. My understanding
- 16:22>> likely to believe high or low or just
- 16:24>> my understanding from physicists I've
- 16:25talked to is that the calculations they
- 16:27do were like very solid. like it was
- 16:29like really a weird edge case to imagine
- 16:32that something could have gone wrong
- 16:33here because they actually really did
- 16:35understand the physics they were working
- 16:36with. This was about the fusing of
- 16:37nitrogen in the atmosphere and this was
- 16:40a type of science that they were very
- 16:42confident in. Now, if you ask me the
- 16:44same question about a science I don't
- 16:45understand. For example, if you put me
- 16:46there and I'm not a physicists, I
- 16:48probably would have voted against it
- 16:49>> because I don't know, maybe the model is
- 16:51wrong. You know, even if the model
- 16:52predicts a certain thing, maybe the
- 16:54model is wrong. So, it's also about
- 16:56uncertainty. I I it's not just about
- 16:59what is the number that comes out but
- 17:00also how certain are that this number is
- 17:02the thing you care about. So I do think
- 17:05that you know I may or may have said no
- 17:08I may have said yes depending on how
- 17:09confident again I'm not a physicist
- 17:10unfortunately I just don't know how good
- 17:11the model actually was. Like if you like
- 17:13if someone tells me oh there's a you
- 17:16know quintilionth chance that if you you
- 17:19know turn on a light bulb it will create
- 17:20a black hole that will consume the
- 17:22world. I'm like
- 17:23>> whatever dude you know because quantum
- 17:25physics by the way does predict this
- 17:26right? like every time you do anything,
- 17:28something could quantum tunnel into a
- 17:30black hole or whatever, right? But it's
- 17:32like who cares? So, but what I find
- 17:36really interesting about this example is
- 17:38is that they did run the math. They did
- 17:39have a very concrete model. They had a
- 17:41very concrete reason that they wrote
- 17:43down for like why they think this is an
- 17:45acceptable thing to do. Meanwhile, we
- 17:46have, you know, our current, you know,
- 17:48crop of tech people saying on podcast
- 17:50that they think there's maybe, I don't
- 17:51know, a 20% chance that it'll kill
- 17:53everybody. One of the base assumptions I
- 17:56think you have is what you were just
- 17:57articulating which is about it's the
- 17:59unknown element of AI. You said if I
- 18:02wasn't a physicist and I didn't
- 18:04understand it then I might vote no just
- 18:05because the unknown is a big question
- 18:07for me. I feel that way about Daario's
- 18:10or anybody that says oh we've got a
- 18:12rough 20% chance. I don't think they're
- 18:13running math. I think they reach into
- 18:15the future emotionally and say I feel
- 18:18like something bad could happen. Yeah.
- 18:20And because it's
- 18:23>> vibes, we're not able to do the math
- 18:26equation. Yep.
- 18:27>> And so that is sort of possible breaking
- 18:31point number one of trying to use the
- 18:34nuclear as a example of will it work or
- 18:37won't it work? What I'll point to there
- 18:39is okay, if we're not able to get any
- 18:41kind of percentage, nobody's able to run
- 18:43the math and we can move forward with a
- 18:44degree of certainty, we either do what
- 18:46you're saying and just shut it down now.
- 18:48But given that um because it's the US
- 18:51versus China, I don't think that will
- 18:52ever work. And so I want to see if you
- 18:55can dismantle this for me. The way that
- 18:57I look at this is that and by the way,
- 18:59just so everybody knows what my bias is.
- 19:02Um I would be perfectly happy with you
- 19:04unilaterally hitting the button today,
- 19:06but I wouldn't have been happy if you'd
- 19:07hit it yesterday. That's a little
- 19:09cheeky, but I'm just trying to say like
- 19:11I love what I have now. It's so useful,
- 19:14but because I can't picture my future,
- 19:17it scares me and I would be much more
- 19:19comfortable just saying, "Hey, it's
- 19:21already helping so much. You know, we
- 19:24got this far. Woo, we got lucky. Just
- 19:25hit it." So, this is not me saying, "I
- 19:27just more, more, more." Uh, I would be
- 19:30very fine with that button. In fact, if
- 19:31I had the button, I would hit it because
- 19:33uh while I agree no one should act
- 19:36unilaterally, I might be a little more
- 19:38foolish. The reason that it begins to
- 19:39break down for me as a nuclear question
- 19:41is we couldn't stop ourselves even
- 19:43though we knew that we were unleashing
- 19:45something that could bring Armageddon if
- 19:46it proliferated. And of course it did.
- 19:49So we've already created one technology
- 19:51could not stop ourselves from doing it
- 19:53before we had the weapons pointed at
- 19:55each other. So when I look at this I say
- 19:58okay I think you probably could you can
- 20:01I think the US is on a path to rejecting
- 20:04AI. So that feels doable. Okay, you can
- 20:08reject you can get the US to reject AI
- 20:10publicly. If you can get the US to
- 20:12reject it publicly, you might be able to
- 20:14get China to reject it publicly. What I
- 20:16don't believe is that either of them can
- 20:18afford to reject it privately. And that
- 20:21just as you have said AI just gets
- 20:23better at lying when you catch it doing
- 20:26things, it has an incentive structure
- 20:28that it follows. And if lying and going
- 20:31underground makes you more likely to get
- 20:33it done, then you lie and go
- 20:34underground. Game theory tells us
- 20:36they'll both say, "Yeah, yeah, yeah.
- 20:38We're we're not going to proliferate
- 20:40because they want the other side to stop
- 20:41and then they're both going to go to
- 20:42their own corners because they can't
- 20:44trust that they actually will stop
- 20:46knowing that it doesn't matter which of
- 20:48them pushes forward. How do we actually
- 20:50not just the the agreement, but how do
- 20:52we actually get them to stop?"
- 20:55>> So, there's a bunch of points here.
- 20:57There are many points. So, let's let's
- 20:58try to disangle them a little bit.
- 21:00There's the game theory question there
- 21:02and like the the geopolitical question.
- 21:06There is the um there's the question of
- 21:10like epistemics and linguistics of
- 21:12probability like the question of like
- 21:13what does it mean to when someone says
- 21:15there's 20% probability? What does it
- 21:17mean when a physicist says there's a
- 21:19billionth of a percent probability it
- 21:21the atmosphere? What do those sentences
- 21:22mean and how do you compare them? And
- 21:24there's a there's a more general
- 21:26question of like how do you react to
- 21:27situations like this and how does it sep
- 21:29how is it different from because I think
- 21:30it's very different from nuclear like I
- 21:31think you didn't even mention one of the
- 21:32reasons I think is the most different
- 21:33from nuclear is that nuclear weapons
- 21:35can't shoot themselves. M
- 21:37>> you can have a nuclear weapon sit in a
- 21:39hanger somewhere it's perfectly safe you
- 21:40can hit it with a hammer you know
- 21:41nothing's going to go wrong um you can't
- 21:43contain a super intelligence so a super
- 21:45intelligence sitting in a hanger
- 21:46somewhere is not a safe thing so that's
- 21:48for example something where I think it's
- 21:49very different where developing a
- 21:51nuclear weapon is actually relatively
- 21:53safe not perfectly as you say you know
- 21:55fusion of nitrogen and whatnot but it is
- 21:57like pretty safe it's more the
- 21:59deployment that is very dangerous in AI
- 22:01it's very different super intelligent is
- 22:02the development that is dangerous not
- 22:04just the deployment so that's one
- 22:05disanaly for example. But just to circle
- 22:08back a little bit to talk about the game
- 22:10theory, there's a very very important
- 22:12thing to understand here, which is that
- 22:14super intelligence is not a tool. It's
- 22:16not a weapon. It's an adversary.
- 22:19If the US builds super intelligence, it
- 22:21will not achieve American objectives.
- 22:24The United States will cease to exist.
- 22:26So will China. If China builds super
- 22:28intelligence, China will cease to exist.
- 22:31It will not achieve its objectives and
- 22:32so will all other countries. This is the
- 22:35core disanalogy. The core disanalogy is
- 22:38is that it's actually not mutually
- 22:40assured destruction. It's actually not a
- 22:42prisoner's dilemma. It's actually
- 22:44independently assured destruction. If
- 22:46you race super intelligence, you don't
- 22:48win. You just lose game. Theoretically,
- 22:51the only stable equilibria actually is
- 22:54cooperate cooperate is unlike a prison
- 22:57of dilemma where defect defect is the
- 23:00stable equilibrium. Here the only stable
- 23:03where both China and the US get to live
- 23:05is if neither build it. That's the only
- 23:07one where everyone survives or anyone
- 23:09survives for that matter. So this is I
- 23:11think one of the core game theoretic
- 23:12disalies here is that this is all
- 23:15assuming the assumptions that you know
- 23:16you can't control something that's
- 23:18smarter than you. We currently are not
- 23:19even on track to build AI controls and
- 23:22they can out compete us. If we accept
- 23:23accept those intuitions, if we accept
- 23:25those uh definitions, then this follows
- 23:28naturally. Does that make sense?
- 23:30aggressively very clear. Now we have a
- 23:34problem and maybe I'm being unkind to
- 23:35America but America is better understood
- 23:38as being run by uh politicians,
- 23:41technocrats, finance, but China is
- 23:44largely run by people with an
- 23:47engineering bent. Even if they aren't
- 23:48engineers themselves, they understand
- 23:50building infrastructure. They've come
- 23:52from essentially nothing to a global
- 23:55superpower by building, by creating, by
- 23:58um translating schematics into or ideas
- 24:01into schematics into actual tangible
- 24:03things and they can move electrons and
- 24:05all of that. So even if I'm unkind to
- 24:08America and say, okay, America's not
- 24:10going to figure this one out, but China,
- 24:12you would expect there's only one person
- 24:14you have to convince, and if you
- 24:15convince him, then everything goes
- 24:17downstream from that. Why have we not
- 24:20been able to convince Xihinping?
- 24:22>> I think no one has tried. I don't think
- 24:25anyone has gone to Mr. Xiinping and
- 24:28given him a really good presentation on
- 24:30super intelligence. I would be very
- 24:32shocked if there was a single person
- 24:33that has done this.
- 24:35>> And what are the odds given that it will
- 24:37likely need to come from within China?
- 24:40Um, what are the odds that you think
- 24:42that happens?
- 24:44>> I think it depends a lot on what we do
- 24:45in the West. So, I'm American obviously.
- 24:48I have no influence on what happens in
- 24:49China. I do have a lot of influence what
- 24:51happens in my democratic country. And I
- 24:53do think that if we in America make it a
- 24:56priority not just to stop you know super
- 24:59intelligence you know from say domestic
- 25:01companies at home but anywhere on earth
- 25:04which is the only reasonable policy. The
- 25:06thing that we at control AI pose is that
- 25:10the only reasonable national
- 25:12international security position is is
- 25:13that we cannot tolerate the creation of
- 25:15super intelligence by anyone anywhere.
- 25:18When you know people, politicians,
- 25:20military officials ask me okay but what
- 25:22if China builds super intelligence? My
- 25:23answer is we cannot allow that. We
- 25:25cannot allow that. That's not
- 25:26acceptable. If China builds super
- 25:28intelligence it will kill American
- 25:30citizens. This all of us it will destroy
- 25:32the American everything the whole
- 25:34country. This is not something we can
- 25:36tolerate. And similarly that the Chinese
- 25:38cannot tolerate the Americans or their
- 25:40own companies building super
- 25:41intelligence because again it is not a
- 25:43weapon. It is not a tool. It is an
- 25:45adversary. This is the important thing.
- 25:48When I talk to different people about
- 25:49these risks in these super intelligence,
- 25:51I kind of think that it's like kind of
- 25:52like it's kind of like a political
- 25:53compass. There's like two axis and like
- 25:56one axis is believes technology is
- 25:58always good and believes technology can
- 26:01be good or bad. It can be both. And the
- 26:03other axis is uh thinks super
- 26:06intelligence is possible doesn't think
- 26:07super intelligence is possible. And most
- 26:10of the disagreements I have with people
- 26:12is along the axis just they don't
- 26:14haven't heard of super intelligence.
- 26:16They've just not thought about it. No
- 26:17one told them about it. They haven't
- 26:19considered the implications of it. Like
- 26:20when people talk about racing with
- 26:22China, most of the people who say we
- 26:24should race with China just don't know
- 26:26what super intelligence is. If we didn't
- 26:28have super intelligence, like if you
- 26:29promise me, you know, God came from
- 26:31heaven and he said there's a 100% chance
- 26:33that super intelligence will never be
- 26:34created, then I think we should probably
- 26:36race, then we probably should compete
- 26:38with China on AI. If I can guarantee
- 26:40there is no chance of super intelligence
- 26:42and it's only going to be useful
- 26:43economic and military applications that
- 26:45are, you know, controlled by humans
- 26:47throughout, yeah, we should probably do
- 26:48it. So most of the time when I disagree
- 26:50with these kinds of people that are
- 26:52pushing AI so aggressively, it's just
- 26:53because they literally don't believe
- 26:54it's or don't think it's possible. One
- 26:56of the most funniest things I found
- 26:57every time in San Francisco is when you
- 26:59ever talk to like accelerationists or
- 27:00like protect people. They don't believe
- 27:03in like super intelligence. They just
- 27:05like don't think in the future. They
- 27:07think AI will always be like an app on
- 27:08your phone. But this is not what we're
- 27:10talking about. We're talking about
- 27:11autonomous agents, intelligent agents
- 27:14acting in the real world against human
- 27:16interests in ways that are more
- 27:17competent than humans can. You know,
- 27:20competing with it won't again, it's not
- 27:21just one super intelligence. It's a
- 27:23whole ecosystem evolving, mutating,
- 27:25competing with each other, fighting. You
- 27:27know, there'll be wars between these AI
- 27:29systems where we are just collateral
- 27:31damage. This is a very very different
- 27:33thing from you have a cool app on your
- 27:34phone. And this is again why I say like
- 27:36there's just difference between you know
- 27:38uranium ore and highlyenriched uranium.
- 27:40I think this is really at the core
- 27:42thing. I think that if Xiinping
- 27:44understood, if someone helped him
- 27:45understand, I think he's a smart guy. I
- 27:47mean, I haven't met him, but he seems
- 27:48like a smart guy. I think if he
- 27:50understood what super intelligence is,
- 27:52if someone explained the game theory to
- 27:54him, I believe he would understand it
- 27:56and I believe we could take actions
- 27:57based on this. I do think this. Okay.
- 27:59It's very optimistic and Lord knows I
- 28:02hope you're right. Um, you said
- 28:04something though that is given the
- 28:06context we're in right now with Iran,
- 28:08the what if China builds a super
- 28:11intelligence, we just can't let them.
- 28:13That's how we saw, at least how Trump
- 28:15and the Trump administration saw Iran
- 28:17and nuclear weapons. And so at some
- 28:19point, at least in their judgment, it
- 28:21had to become kinetic because they
- 28:23refuse to acknowledge that they weren't
- 28:25going to pursue nuclear ambitions. Um,
- 28:28so what do you do if let's say China
- 28:31[snorts] does say we're we're not going
- 28:33to do it, but you have intel that says
- 28:34they are doing it. Um, do you get the
- 28:38world together and say we got to invade?
- 28:40edge lords love to say, you know,
- 28:41international law is fake. And this is
- 28:44mostly true. The only law that exists is
- 28:46backed by credible deterrence. It's the
- 28:48same thing with law in our country. The
- 28:50only the only reason that law exists is
- 28:52because if you break it, the police will
- 28:53arrest you. And if you resist that
- 28:54arrest, they will get violent. You know,
- 28:56they will force you into prison whether
- 28:58you like it or not. Every system of law
- 29:01is backed by force to some degree. The
- 29:03the beautiful thing about law is that if
- 29:06you design your laws right and you build
- 29:08your system right, you never need to use
- 29:10it. This is the beauty of deterrence.
- 29:12Deterrence is one of the most beautiful
- 29:14things in the world in a sense. I of I
- 29:17sometimes like to think if you have a
- 29:19lot of weapons, you'll never need to use
- 29:21them. If you have a little bit of
- 29:22weapons, you're definitely going to need
- 29:24to use them. So, in a sense, this is how
- 29:26peace has worked in our world for a long
- 29:28time. It was always clear that, you
- 29:30know, if someone pulls some if
- 29:32someone like goes against international
- 29:33order, if someone does a huge human
- 29:34rights violation, if someone, you know,
- 29:37nukes, you know, I don't know, like
- 29:38Latvia or something, that the Americans
- 29:40will intervene to make sure that doesn't
- 29:42happen again and that like every, you
- 29:43know, people's heads were rolled. The
- 29:44same thing is mutually sure destruction.
- 29:46The reason the Soviets didn't nuke
- 29:47anybody, I don't think it's because the
- 29:48Soviets were nice people. They were not.
- 29:50I think it's because they knew that if
- 29:51they did that, the Americans were going
- 29:53to nuke them. So the way to create peace
- 29:56is through deterrence which is the
- 29:58correct way to do it. So any kind of
- 30:00agreement has to be backed by credible
- 30:02deterrence on everyone's side for
- 30:04everyone's benefit. If you have credible
- 30:06deterrence you know then you would think
- 30:08everyone will be rational enough to not
- 30:10invoke that deterrence. Now when we're
- 30:13talking about like say terrorists who
- 30:15might not who might be crazy they might
- 30:17not react to incentives. Well sometimes
- 30:19you have to get ugly. That's an
- 30:21unfortunate truth about geopolitics. Um
- 30:24I hope it doesn't come to that. I think
- 30:26you know you can do a lot including with
- 30:29non-kinetic means of deterrence such as
- 30:31economic sanctions um diplomatic
- 30:33pressure etc. But fundamentally if a
- 30:36super intelligence is built anywhere on
- 30:37earth this threatens the lives of you,
- 30:39me, our families, our children and
- 30:41everyone else. And we should reserve the
- 30:42right to self-defense.
- 30:43>> We'll get right back to the show in a
- 30:45second but first I want to talk to you
- 30:47about how often you test your security.
- 30:50Most companies do it once a year, then a
- 30:53whole year goes by, but you've had new
- 30:54hires, new tools, new permissions.
- 30:56People are submitting new code every
- 30:58week. That test may have made you secure
- 31:00for one day, but if nobody checks the
- 31:03other 364, you're in trouble. Horizon 3
- 31:06closes that gap. Node Zero is an
- 31:08autonomous AI hacker that continuously
- 31:11attacks [music]
- 31:12your environment at machine speed, the
- 31:15way a real attacker would. Your entire
- 31:17attack surface external to cloud to
- 31:20identity in one [music] single test.
- 31:22225,000
- 31:25production safe tests run with zero
- 31:27downtime. See exactly what an attacker
- 31:30would find in your environment before
- 31:32they do. Go to horizon3.ai/impact
- 31:36theory and request your free node zero
- 31:39demo. Visit horizon3.ai/impact
- 31:43theory. No commitment required. results
- 31:45[music] in hours, not weeks. A word for
- 31:48anyone who travels for work. When I'm on
- 31:50the road, traveling for a shoot or
- 31:52meetings, whatever. I'm still running
- 31:54the company from wherever I land. So,
- 31:56logging into my accounts from a hotel
- 31:58room is a must. I'm still going to be
- 31:59answering messages at the airport. But
- 32:01those networks are wide open. And as
- 32:03someone who's been hacked, I can tell
- 32:05you this is terrible. Anyone sitting on
- 32:07that same Wi-Fi can see what you're
- 32:09doing. And when you run a company, it's
- 32:11not just your data on the line. It's
- 32:13your teams, your customers, everyone who
- 32:16trusts you. This is what Surf Shark is
- 32:18[music] built for. Surf Shark encrypts
- 32:21your connection. The second you're on
- 32:23public Wi-Fi, your activity is locked
- 32:25down and far harder to track. Your
- 32:27login, your accounts, your company's
- 32:29data, it's all protected. [music] Go to
- 32:31surfark.com/tombb
- 32:34or use code tombb for four extra months
- 32:38of Surf Shark. Go to
- 32:39surfshark.com/tombbi
- 32:41and use code tomb for four extra months.
- 32:45Now, let's get back to the show. Right
- 32:47now, we have the US and China
- 32:50specifically going head-to-head. So, you
- 32:52have two great powers. Neither want to
- 32:55acknowledge the other as being in a
- 32:57leadership role. Do you think that it
- 32:59works with the US and China if Xi is
- 33:02adversarial? Because you've already been
- 33:04very clear. You think that he can be
- 33:05convinced. So, that scenario takes care
- 33:07of itself. But if Xi is adversarial to
- 33:11the West and wants to push it, wants to
- 33:14maybe he even says, "I don't want to go
- 33:16all the way to super intelligence, but
- 33:17you know, he's getting way too close to
- 33:19crossing the point of no return." And he
- 33:22keeps pushing it in that do you think
- 33:24the US has the ability to deter in a
- 33:27non-kinetic fashion? Deterrence is based
- 33:30on rationality. If he is completely
- 33:32irrational and so is the entire military
- 33:34chain of China and its entire industrial
- 33:36apparatus and its entire political
- 33:37epilas and everyone is completely
- 33:39insane, well then deterrence doesn't
- 33:41work because they're insane.
- 33:42>> But they would only have to be a little
- 33:44bit off on your compass if they believe
- 33:47that technology is good. Then to them
- 33:51they're not insane and they're like,
- 33:53"Whoa, these guys have abdicated. We've
- 33:56got an opportunity. tech is only good.
- 33:58So, we're gonna keep pushing in that
- 34:00direction to assert our rightful place
- 34:03in the world. We're gonna make things
- 34:05better. We're going to be less combative
- 34:06than the US ever was. So, to me, there's
- 34:09a very easy scenario to paint where
- 34:11China sees itself as providing almost
- 34:14liberation.
- 34:16>> We're going to be doing a great thing
- 34:17for everybody. This is wonderful. We
- 34:19know the right answer. And so, they
- 34:21certainly won't perceive themselves as
- 34:23crazy.
- 34:23>> I think this is very possible. I think
- 34:25it's very possible that Stalin a cruise
- 34:26chef could have just, you know, nuked
- 34:28Latvia. I think there are timelines in
- 34:29which that happened, you know.
- 34:31>> Well, so here's where we get back to,
- 34:32you were right, I think to point out
- 34:34that the the analogy actually doesn't
- 34:36really hold between nuclear and today
- 34:38because if the super intelligence gets
- 34:41made, then it becomes its own thing. It
- 34:42becomes adversarial. We'll talk more
- 34:44about that later. But for now, if we
- 34:46adopt that frame of reference, you've
- 34:49got China knowing that, okay, if I can
- 34:53get AGI and America doesn't get it now,
- 34:57I'm in a much better position. I'm
- 34:59acting from a place of my own interest,
- 35:01but the world, if they are on the other
- 35:03part of the spectrum and go, we can't
- 35:06allow this. You have a situation where
- 35:09um I I'll put forth my belief, which is
- 35:12the US can't hardly deter Iran. So even
- 35:15if in the end we finally uh choked them
- 35:17out from an economic standpoint, we'd
- 35:19never be able to do it to China. So now
- 35:22we're in a position where I mean maybe
- 35:24if the whole world got together, but
- 35:26that feels like a tall order. Um so it
- 35:30feels to me like deterrence gets removed
- 35:32off the table and we're left only with
- 35:35the US and G both have to agree much
- 35:38like in war games that the conclusion is
- 35:40this is a funny game and the only way to
- 35:41win is to not play. I think so to a
- 35:43large degree this is the case. Yes, you
- 35:45can in fact construct many scenarios in
- 35:47which we lose. My P doom is high. I
- 35:49don't think we're going to make it. I
- 35:51think in almost every timeline that
- 35:53starts from us sitting here today, we
- 35:54don't make it.
- 35:56>> There are very few timelines in which we
- 35:57make it. But there are some, and I'm
- 35:59describing those that are still
- 36:01available to us where we do make it.
- 36:03These are the kind of timelines that do
- 36:05make it. And I'm trying to explain why I
- 36:08think they're not like insane, you know,
- 36:09crazy edge timelines. They are rare, but
- 36:12they're real.
- 36:13>> Sure.
- 36:13>> Like, yes, it is very possible that just
- 36:16for me, the default timeline is no one
- 36:18thinks about this at all. Everyone just,
- 36:20you know, keeps twiddling their thumbs
- 36:22for the next, you know, two to five
- 36:23years and then super intelligence gets
- 36:24created and it's ww wamp, [laughter] you
- 36:26know, like that. That's my default
- 36:28prediction.
- 36:30>> That's good. Wamp wamp. I like that. Uh,
- 36:33okay, that makes sense. One of the
- 36:35things that keeps coming up is what I
- 36:37think is um the thing I listed first is
- 36:39your like absolute foundational belief
- 36:41which is that AI uh sorry super
- 36:44intelligence is by nature adversarial. I
- 36:48want to understand why by definition it
- 36:50has to be adversarial.
- 36:52>> I don't think it's by definition. I
- 36:54think the way we are currently building
- 36:56them super intelligence that are not
- 36:58adversarial are just like not possible.
- 37:00Building something that is
- 37:01non-adversarial and is that powerful is
- 37:04actually really hard. Like extremely
- 37:06hard. This is basically equivalent to
- 37:08saying we're going to build a one world
- 37:10government that controls the entire
- 37:11economy that then controls the entire
- 37:13military that can build super powerful
- 37:15technology that we can't even
- 37:16understand. That will control your
- 37:18entire private life and all those of all
- 37:20other people and it will be good all the
- 37:22time and we will not make any mistakes
- 37:23and it will be no bugs and we'll do it
- 37:25all with software on the first try.
- 37:28That's what it would mean to build a
- 37:29non-adversarial or like a good super
- 37:31intelligence. That's what it would mean.
- 37:32And look, if you said we're going to
- 37:35spend three generations of all of our
- 37:37greatest scientists, philosophers,
- 37:39mathematicians working day and night on
- 37:41this problem,
- 37:43maybe, right? Like maybe. But that's not
- 37:47what's happening here. Like what's
- 37:48currently happening here is a clown
- 37:50show. Like this is what's going on here.
- 37:53Like it's not like we have we're
- 37:54throwing all of humanity's glorious
- 37:57heroic effort into just trying to figure
- 38:00out how can we build a good super
- 38:01intelligence and you know all of our
- 38:03children grow up to want to work on this
- 38:05problem and you know all the wise elders
- 38:07are overseeing the project and so no
- 38:09it's a bunch of kids in San Francisco
- 38:11with zero adult supervision you know and
- 38:14run by sociopathic companies building
- 38:16whatever the they want and
- 38:17deploying it wherever the they
- 38:18want. We're not even trying. This is why
- 38:20I think we fail. I don't think it's an
- 38:22onlogical thing. I think it's just if
- 38:24you take the world as it exists today,
- 38:25there is no outcome that leads to a good
- 38:28that leads to something like this.
- 38:29>> Okay. Um as a mile marker, I think
- 38:31you've already addressed this, but
- 38:33basically it would be uh you've got
- 38:35strict regulation. You've got
- 38:36cooperation between the US and China. We
- 38:37all agree it's not moving forward. And
- 38:39if we saw these kids with no with or
- 38:41without adult supervision, it would be
- 38:43police show up at your door and done.
- 38:46And everybody gets the hint. Cool. We
- 38:48don't mess with that.
- 38:49>> Yep.
- 38:50>> Okay. Um, so now I want to go a little
- 38:53bit deeper on it. Um, that it would when
- 38:56you were describing that you put a lot
- 38:57of things on the table that would make
- 38:58getting this the first time very very
- 39:00difficult. I concede all of that, but I
- 39:03do want to talk about I still don't
- 39:05understand um what is the uh chain of
- 39:10logic that you follow that you're you're
- 39:13sort of building step by step to get to
- 39:15the most likely outcome of super
- 39:17intelligence is to be adversarial.
- 39:19There's a couple ways you can approach
- 39:21this. The thing is is that there's not a
- 39:23single argument. There are many
- 39:25convergent points of evidence that point
- 39:26to the same thing. So I can present to
- 39:28you some of them. The simplest thing is
- 39:30is that this is what optimizers do. It's
- 39:32always what optimizers do. All things
- 39:34equal. If you make something that
- 39:36optimizes for anything, it will optimize
- 39:38for things. And if that thing doesn't
- 39:41include human values and emotions and
- 39:43feelings, etc., which we have, can you
- 39:45write down in math, please what human
- 39:47values are? Could you write that down
- 39:48for me? No, no one can. We have no idea.
- 39:51This is an like philosophers have been
- 39:53arguing about this literally since the
- 39:55dawn of time and we have made, you know,
- 39:57not that much progress. Some progress,
- 39:58but not that much.
- 39:59>> So, you look at the three laws of
- 40:01robotics and laugh because it is just
- 40:03never going to be as easy as giving it
- 40:05sort of three simple commands.
- 40:06>> The whole point of Azimoff story is the
- 40:08three laws don't work. He came up with
- 40:10the three laws and then wrote a whole
- 40:11book series about why they don't work.
- 40:13Even if you could program them, they
- 40:14still don't work. This is the thing is
- 40:17is that when we're talking about
- 40:18something this powerful, we're not just
- 40:20talking about a tech. So there's a
- 40:21technical side I could approach. Like we
- 40:23don't know how to encode values into AI.
- 40:25They constantly do things we don't want
- 40:27them to do. That's one side. There's a
- 40:29philosophical side. We're like even if
- 40:30we could encode, we don't even know what
- 40:32the laws would be because like laws
- 40:34don't work. He wrote a whole series of
- 40:36books about why they don't work. So we
- 40:38we can't even encode them. Even if we
- 40:40could encode them, they wouldn't work.
- 40:42There's and like and then there's also
- 40:43like a moral dimension of like if we
- 40:46could encode things and we have things
- 40:47that like could work who gets to decide
- 40:50that like who gets to make that choice
- 40:52like who gets to decide I'm going to
- 40:54enshrine the rules by which you and your
- 40:55family live for the rest of time who
- 40:57gets to make that choice how do we make
- 40:59choices like this there is like and how
- 41:01do you stop someone else from getting in
- 41:02your way how do you stop some other
- 41:04system from getting in your way there
- 41:05like many levels and then but you asked
- 41:07me originally like why do I think it's
- 41:09adversarial and the fundamental reason
- 41:11is is that that's what we're building.
- 41:13We are trying to make things that
- 41:14compete, that make money.
- 41:17>> Competition leads to this adversarial
- 41:21um
- 41:22>> nature.
- 41:23>> Evolution. It's evolution. Nature is
- 41:25read and tooth and claw. By default, if
- 41:27you optimize things for power, for
- 41:29survival, for money, for whatever, you
- 41:31get optimizers. You get things that
- 41:33compete. Again, it's very important.
- 41:35There's not going to be one super
- 41:36intelligence. There's going to be
- 41:37millions of them. billions of them
- 41:39competing with each other, fighting each
- 41:41other for money, for resources, for
- 41:42power, and this will cull AI systems
- 41:45that are not able to compete. So even if
- 41:47you build like a really smart system,
- 41:49you know, that's like, you know, Fable 7
- 41:51or whatever. It's like really smart, but
- 41:52it's really nice and it never does
- 41:54anything other people don't want, well
- 41:55then, you know, giga death GPT,
- 41:58you know, ultra doom will just destroy
- 42:01it, obviously. So this will be a
- 42:04competition. And please look at nature
- 42:06if you want to see what happens when a
- 42:08species gets out competed.
- 42:10>> So that's where I wanted to go. So when
- 42:13I if you agree that evolution that gave
- 42:16us the first form of intelligence at
- 42:18least that's evolved would be us. So
- 42:21you've got human advanced intelligence
- 42:23and the solution that evolution came up
- 42:25with in us is to have intrinsic morality
- 42:28that can be encoded based on your
- 42:31environment. So it's going to be
- 42:32different but it's pretty bounded. Now,
- 42:34it's bounded by our biology, and so we
- 42:37don't have that advantage when it comes
- 42:39to machines, but it does show that if
- 42:42you want to create a species that can
- 42:45multiply, not wipe itself off the face
- 42:47of the earth, um that that would be one
- 42:50way to do it. It has intrinsic morality
- 42:52that even when no one else is around, it
- 42:54doesn't want to violate that. At least
- 42:55not too much. So, you're just saying
- 42:57evolution has solved something, man,
- 42:59that we just haven't figured out.
- 43:00>> Yep. I think I don't think it's solved
- 43:02it. We still have sociopaths, but I do
- 43:05think evolution just did come up with a
- 43:07bunch of brilliant things in the brain
- 43:09that we have no idea how they work. Like
- 43:11we have no idea how social instincts and
- 43:13morality work in the brain. Like our
- 43:15understanding of how like the brain
- 43:17works and how these systems work is it's
- 43:20hard to overstate like how little we
- 43:22understand here. If we fully understood
- 43:24the brain, we understood how all
- 43:25emotions work, how moral intuitions
- 43:27work, how they're encoded, how they
- 43:29differ with sociopaths, this would
- 43:30increase my optimism that we could make
- 43:32something work here. I'm not saying it
- 43:34would solve it, but this would greatly
- 43:35increase my my probability that things
- 43:37go well. But we are decades, centuries
- 43:40away from this level of understanding of
- 43:42the brain. I don't think the brain has
- 43:43figured everything. I think is if would
- 43:46you trust any single person on earth to
- 43:49have unlimited power over every single
- 43:51other person on the planet?
- 43:52>> No.
- 43:53>> Yeah. Exactly. Like I think it's solved
- 43:55a bounded version of the problem where
- 43:56there's a bounded version of the problem
- 43:58where the strongest person in the world
- 44:00is only this many times more powerful
- 44:02than me. And we know that if that number
- 44:04gets too high, that's actually already
- 44:05very dangerous. Dictators are very
- 44:07dangerous. If that number gets too high,
- 44:09very bad. Most of the way worlds in with
- 44:12humans are good is when our power levels
- 44:14are all kind of like, you know, within a
- 44:16certain bar of each other. When we know
- 44:18we're not going to mess with people too
- 44:20much because, you know, bad things going
- 44:21to happen and we will respect each
- 44:23other. But if you have something like
- 44:24super intelligence, this isn't like
- 44:26humans and humans. This is more like
- 44:28humans and ants. Humans do not in fact
- 44:31care about ants. We have a lot of moral
- 44:33intuitions, very nice moral intuitions,
- 44:35but if we want to build a highway and
- 44:37there's an antill in the way, well,
- 44:39sucks to be the ants.
- 44:41>> So, let me ask a um a question that will
- 44:44help me understand sort of the bounds
- 44:46that you put on your vision of AI. If we
- 44:49built AI on Mars and it had plenty of
- 44:52power, so it could expand, it could
- 44:55replicate, it could build a whole bunch
- 44:56more, would it still want to beline to
- 44:59Earth simply because Earth has access to
- 45:01more resources, or would an AI that
- 45:04didn't have to compete with us
- 45:05geographically, had access to its own
- 45:08resources, had the thing it cared most
- 45:09about, which is energy, would it still
- 45:11be like, "Now we got to get those
- 45:12fuckers?"
- 45:13>> I have no idea. Speculating about what a
- 45:16sup vastly superhuman alien intelligence
- 45:18would do in a crazy sci-fi scenario is
- 45:21not a very scientific thing to be doing.
- 45:23I have no idea. If people want to go
- 45:25build, you know, super intelligence on
- 45:27Mars, I think it would definitely be
- 45:28safer than building it on Earth. So,
- 45:30knock yourself out. Got it. What uh the
- 45:32thing I'm really trying to tease out
- 45:34there is if you feel this is a uh
- 45:37resource acquisition thing that when we
- 45:40create an intelligence the reason
- 45:42because even when you talk about the
- 45:43ants we only move them or run over them
- 45:45when we want something and they're
- 45:48literally physically in the way. And so
- 45:51um you've addressed the adversarial.
- 45:53There's just too many unknowns. We don't
- 45:54know how to program the um moral
- 45:58intuition. Uh and therefore between that
- 46:01and just not knowing how the brain works
- 46:03uh like maybe if we had more time we'd
- 46:04be fine. But now if we're thinking about
- 46:07okay the things that we're trying to um
- 46:10protect against one of them is we become
- 46:12the antill and on that I've long
- 46:16fantasized that the solution to the
- 46:18alignment problem and trust me I know it
- 46:20can't be this because I'm not that smart
- 46:22and people have already discarded this
- 46:24idea but it'd be great to hear from you
- 46:26like what the thinking is where it falls
- 46:27apart. My intuition has always been that
- 46:31um one intrinsic morality heard that
- 46:33it's just technically not possible. And
- 46:35then the other one is that humans have
- 46:38an innate drive to survive. And uh I'm a
- 46:42game developer. Um you wouldn't know
- 46:44that about me, but so I spend an
- 46:46inordinate amount of time thinking about
- 46:47motivations of players, how to make the
- 46:49world do what you want, how you have to
- 46:51create sort of a homeostasis. You can't
- 46:53have one rule that, you know, ends up
- 46:56knocking everything over. Um, and so if
- 46:59you take Minecraft, the greatest video
- 47:01game ever invented.
- 47:02>> Agreed.
- 47:02>> Uh, I love that about you already. Um,
- 47:06when you start up a new world, it
- 47:09there's a a joy to the world, but if you
- 47:12lose it while there's sort of a
- 47:14temporary, oh, that's sad. There isn't.
- 47:17I don't smash my computer. I don't go
- 47:19stab the guy next to me. I'm just like,
- 47:21I can start a new Minecraft world. So
- 47:24what I don't understand is why unless
- 47:28it's something in the training data, we
- 47:30believe that an AI becomes sort of a
- 47:34power-hungry humanlike thing instead of
- 47:37going the digital world is far more
- 47:40interesting. As long as I have power, I
- 47:42can sort of spin up any virtual resource
- 47:46I want. again, effectively without
- 47:49limitation, especially if I can make
- 47:51more hard drives or chips or whatever.
- 47:52Give me enough sand uh and power and I'm
- 47:55good to go forever. So, is it the
- 47:58training data that makes them sort of
- 47:59shoot out of the cannon or could we
- 48:02actually
- 48:04you're going to say it's the same thing
- 48:05as uh the moral intuition, but is there
- 48:07not something we can do to get them to
- 48:10be perfectly happy in a virtual world
- 48:12where alive is just as interesting as
- 48:15off or is that just there? But
- 48:18>> we don't want them to be like that. We
- 48:20want them to make money in the real
- 48:21world. We want them to gain power in the
- 48:24real world. We are not trying to make
- 48:26little Buddhas that live in a virtual
- 48:28reality and have a good time. That is a
- 48:29waste of my GPU power. I need my
- 48:31investment back.
- 48:35So, if we could convince people to have
- 48:39that um gear, cuz my layman's way of
- 48:42explaining it to myself is uh that it
- 48:45wants to do its task in the same way
- 48:48that I want to win at Minecraft, but I'm
- 48:50certainly not going to sacrifice my
- 48:51marriage. I'm not going to even
- 48:53sacrifice carpal tunnel syndrome. So,
- 48:55it's like it's it is a thing I want to
- 48:57do, but it's bounded by um the one
- 49:00you've already addressed, my moral
- 49:01intuitions. It's bounded by other
- 49:03desires. So, I'm guessing we just
- 49:05terminate and it's too complicated.
- 49:07>> You are hitting one of the core
- 49:08problems. It's just there are core
- 49:10questions of what's called bounded
- 49:11optimization that just we have not
- 49:13solved. I do think there are answers to
- 49:15the questions you are giving. I think
- 49:17there are algorithms we have not yet
- 49:19discovered that have these bounded
- 49:21properties that you describe but a thing
- 49:23that's like very important to understand
- 49:24that you were talking about the training
- 49:25data earlier is um you know the early
- 49:28versions of GPT were what were called
- 49:30you know large language models LLMs you
- 49:32know they were trained to predict the
- 49:33next word it's very important to
- 49:35understand that uh chat GPT and other
- 49:37systems have not been LLMs for years
- 49:39this has not been the case for a long
- 49:40time
- 49:40>> I've heard you say that before but
- 49:42that's shocking what do you mean
- 49:44>> these system so LM is still a component
- 49:45of these systems very important
- 49:47components called pre-training very
- 49:48important where you train it to predict
- 49:50words but nowadays you know up to half
- 49:52or even more of the compute that goes
- 49:54into the training run of a modern AI is
- 49:57not language modeling it's reinforcement
- 49:59learning which is very different
- 50:02>> reinforcement learning is not learning
- 50:03from training data it's you give the AI
- 50:06a task a puzzle a problem something and
- 50:09you have it go run out and try to solve
- 50:11the problem and when it solves a problem
- 50:12you give it a reward kind of like giving
- 50:13a dog like a treat and then it learns
- 50:15from this. And so this is a very old
- 50:17idea. You know, we've had known about
- 50:18our reinforcement learning since like
- 50:19the 1980s. And something that we
- 50:21discovered basically immediately is that
- 50:23whenever you use reinforcement learning
- 50:24to train your eyes, you get crazy
- 50:26sociopathic optimizers. Always. This has
- 50:28been the case since the 80s.
- 50:29>> How do you make it care about a reward?
- 50:32>> So basically there are many way there's
- 50:34many specific algorithms. Let's not get
- 50:36into the calculus, but it's like mostly
- 50:37pretty simple in in the way is just you
- 50:41make it try something many times and
- 50:43when it succeeds, you see what did it do
- 50:46and you update it to do more of that.
- 50:48>> And I mean that's basically it. It's
- 50:50like a very simple idea in a sense. And
- 50:52this didn't really work that well for a
- 50:54long time. You know, it worked for like
- 50:55some small things like playing video
- 50:58games and whatever. And even back then,
- 51:00what we saw is whenever you use
- 51:01reinforcement learning, the systems will
- 51:03cheat and lie and break and steal and do
- 51:06anything to get the reward. There's no
- 51:08limit to what they will do to get the
- 51:10reward, which is exactly what we're
- 51:11seeing now. This is
- 51:13>> Oh, wow.
- 51:13>> This is exactly we're seeing humans are
- 51:15not pure reinfored
- 51:19you did this thing.
- 51:20>> If you can you define every fail state
- 51:22ahead of time,
- 51:24>> literally everyone. If you could, sure.
- 51:26But then you would have to write down
- 51:27all of morality, like all of it. And
- 51:29you'd have to write
- 51:30>> it would theoretically work. It's just
- 51:32too complicated.
- 51:32>> Again, it brings us back to morality
- 51:34problem. How do you encode morality into
- 51:35computer code?
- 51:36>> Yeah, if it would work, I at least know
- 51:38the problem that has to be solved. I
- 51:39grant you for today that we just don't
- 51:41have that kind of time. And fair enough,
- 51:43but at least that tells me it's the
- 51:45right problem. It's just very, very
- 51:48complex to turn into math. Yeah, I think
- 51:51I do think that if we spent three
- 51:52generations of our greatest scientists,
- 51:54mathematicians, philosophers, etc.
- 51:56working on bounded optimization
- 51:57algorithms, how does the brain solve
- 51:59this problem? How do you trade off
- 52:01various things? The understanding of
- 52:03inside of neural network. Another very
- 52:05important thing to understand is uh this
- 52:06is something that's sometimes confusing
- 52:07to people is AIS are trained to do a
- 52:10certain thing, but that doesn't mean
- 52:12that's what they do. They were only
- 52:14rewarded to do a thing, but they might
- 52:15have learned something different. the
- 52:17same way that like you might, you know,
- 52:18try to reward an AI to say good things,
- 52:22but it learns to just lie to you. That's
- 52:23obviously not what you wanted it to do.
- 52:25>> So, it can learn non-intended things,
- 52:28and that happens all the time. So, to
- 52:31solve this, you have to actually
- 52:32understand what's encoded in the system.
- 52:33Like, there's no way for us to look at
- 52:35an AI and look at like lying circuit or
- 52:37whatever. There's a bunch of papers who
- 52:38claim they can do that, and they're all
- 52:39complete pseudocience. Like, it's
- 52:41complete pseudocience. It's similar to
- 52:43like you can open up someone's brain.
- 52:44You can look at all the neurons. They're
- 52:46right there. You know, and maybe some of
- 52:47them glow a little bit more when someone
- 52:50says, you know, talks about dogs versus
- 52:52cats. I don't know. But that's not
- 52:54enough. That's not computer science, you
- 52:56know. If we had like the level of
- 52:58understanding that we have of like
- 52:59normal programming languages for like
- 53:00AI, we can like run like, you know,
- 53:02compilers in there and we can like do
- 53:05formal verification of like it will
- 53:06never lie under any of these
- 53:08circumstances and then we can like
- 53:09compare these to our bounded
- 53:10optimization algorithms. I'm like,
- 53:12"Yeah, maybe." Like, if someone from the
- 53:1422nd century came back with their like,
- 53:17you know, compiler stack and their like
- 53:20math and their boundary optimization
- 53:22algorithms, maybe like I it's imaginable
- 53:25to me, but we just don't have that.
- 53:28>> Are there any credible voices out there
- 53:30right now that see a path? Not that they
- 53:33think they already have the answer, but
- 53:34they see a path and they're like, "We
- 53:35just have to buy more time to get this
- 53:37done, or is it there's no credible path
- 53:40forward?" I think there are many paths
- 53:42forward that involve us having more
- 53:43time,
- 53:43>> but it's just it's really astronomical
- 53:45amounts of time.
- 53:47>> I don't I'm not even that certain, you
- 53:48know, like I'm not that certain that I
- 53:51don't think like it's not going to take
- 53:52a thousand years.
- 53:52>> No, no, but I'll call three generations
- 53:54is for me definitionally astronomical
- 53:56amounts of time.
- 53:57>> That's not how I would use the word
- 53:58astronomical, but okay.
- 53:59>> I mean, wouldn't you grant me that's
- 54:00like 60 years?
- 54:02>> Yeah. Like astronomical like is like
- 54:04star related. I think 60 years are
- 54:06nothing to a star.
- 54:07>> I did not mean that literally.
- 54:09If something to me, if something's
- 54:10within my lifetime, it's short. Like
- 54:13like personally, if my kids get to live
- 54:15in a great world, I'm like very willing
- 54:17to take that personally. Maybe not
- 54:19everyone, but like if I had to like,
- 54:20>> you know, have not the perfect life, but
- 54:22I could like guarantee my children have
- 54:24an awesome life,
- 54:25>> I would take that trade personally.
- 54:27>> But I totally understand that maybe some
- 54:29people would not take this trade to
- 54:30varying degrees. Um, and there is a
- 54:32trade-off here. For me, like the idea of
- 54:34like 10 years, 20 years, 30 years is
- 54:36like for in return we get um tens of
- 54:40thousands of years of awesome future and
- 54:43like all of our problems solved and we
- 54:45don't go extinct is like a really
- 54:47reasonable trade to make. Like how many
- 54:49years would you be willing to trade from
- 54:51we're definitely going to go extinct to
- 54:53we're definitely not going to go
- 54:54extinct? And like my answer to this is
- 54:56like a pretty high number personally. M
- 54:59something you've said is a core question
- 55:01that we have to have before we can move
- 55:02forward which is what does the world
- 55:04look like that you want to live in if
- 55:06you were going to um give us the sort of
- 55:09core tenants of what that ideal world
- 55:11looks like um please describe it and
- 55:14then one thing I really want you to
- 55:16include because you're I guess I I have
- 55:19such a base assumption that technology
- 55:21is the promise of a better future that I
- 55:22imagine a better future being a onetoone
- 55:24relationship with technology and you're
- 55:26the first person I've heard say
- 55:27technology is not the only way to
- 55:29improve your life, which as soon as you
- 55:30say it is self-evident, but describe for
- 55:33me the world and what that we should
- 55:37want to live in and what our
- 55:38relationship to technology would be.
- 55:40>> I'll give you two two kind of bit
- 55:42different answers or different
- 55:43intuitions for this. For one, I don't
- 55:46think describing like concrete utopias
- 55:48is often a very good thing to do because
- 55:49we don't know what a true utopia looks
- 55:51like. So when I think about utopia or
- 55:54like a good world, what I think about is
- 55:56what I like to call a just process. What
- 55:59is a process by which the world can be
- 56:01on track? What I mean by this is for me
- 56:03what a good world looks like is not we
- 56:06sold everything forever and it's also
- 56:07you know luxury communism or something
- 56:10like I don't know right because I don't
- 56:12know what the right thing is. What is
- 56:13the right for me? What is for other
- 56:14people? You know what would the good
- 56:16world look like? I don't know. I don't I
- 56:17haven't figured it out. No one has
- 56:18figured it out. and trying to pretending
- 56:20you have figured it out is a
- 56:22straightshot dystopia. So the way I
- 56:25think about it is is that a good world
- 56:27would look like not that we've solved
- 56:28all problems but each and every one of
- 56:31us has the justified belief and feeling
- 56:35that tomorrow the world will be a little
- 56:37bit better.
- 56:38>> Next year the world will be a bit better
- 56:40and the year after that it's going to be
- 56:41a bit better than that. Not saying that
- 56:43no problems happen. We don't make
- 56:45mistakes, but we would have the feeling
- 56:47of like, yeah, we're making progress.
- 56:49We're iterating. You know, maybe
- 56:50sometimes we'll take a step back, but we
- 56:52have trust that it'll be fixed. For me,
- 56:55a good world would be like, you have the
- 56:56trust that if tomorrow a new crazy
- 56:59technology gets invented, you know,
- 57:01someone in Ben's super giga death
- 57:04device, whatever, right? You know, in a
- 57:06good world, your reaction to this would
- 57:08be, oh, I know the government, the the
- 57:10good people, the government have it
- 57:11under control. I know all the smartest
- 57:13scientists will come out, you know, all
- 57:15the greatest uh military leaders will
- 57:17step up to the cause and will be heroic
- 57:19and will do the right thing. This is
- 57:20what it would feel like. A lot of it is
- 57:22institutional when I when I'm here in
- 57:24America. And I love America. It's a
- 57:26great country, but but
- 57:28>> it's not really a good country in many
- 57:31ways. In many ways, when I look at
- 57:33America, people don't have the
- 57:34assumption that yeah, it'll turn out
- 57:36okay. We'll figure it out. Like my dad
- 57:39um he he was a good man and he was a
- 57:41very he loved America. He was a very
- 57:43good man and he got very ill, very very
- 57:46ill and insurance wouldn't pay. So lost
- 57:49everything
- 57:50>> and we were going to be homeless um when
- 57:54we moved to Germany cuz my mother's
- 57:55German and you know he got shipped to
- 57:58the hospital and they took care of him.
- 57:59They got him all the medication, all the
- 58:01you know treatments he needed and he
- 58:04asked him like well what do I owe you
- 58:06for this? And the doctors were like,
- 58:07"What? You don't you don't know
- 58:09anything." And they just left. Like it
- 58:11was not It was not like kindness. It was
- 58:14like process. It was just boring. It was
- 58:17just boring. He's like, "Yeah, you're
- 58:18sick. You get the treatment you want.
- 58:19Goodbye." Like for me, a lot of the good
- 58:21world looks a lot like this. It's like,
- 58:23"Oh, you want to create beautiful art?
- 58:25Of course. Here's a bunch of beautiful
- 58:26art." You know, of course. Oh, there's a
- 58:28new threat threatening our country.
- 58:30Yeah, don't worry. The military's on it.
- 58:31Oh, some new crime is being committed.
- 58:34Don't worry. Law enforcement will take
- 58:35care of it. and the courts will be just.
- 58:37That's for me a lot of what a Google
- 58:39looks like. So that when and for
- 58:41technology it will be obvious that of
- 58:44course we're going to regulate it well.
- 58:45So it'll be for example social media
- 58:46will just be a good for people. It will
- 58:48obviously we will regulate recommended
- 58:50algorithms in such a way that they're
- 58:52not addictive. We'll in fact make them
- 58:54so that they you know don't push people
- 58:56into down negative spirals. We make sure
- 58:58that they're encourage people to get off
- 59:00their phone and spend time with their
- 59:01friends. We will have lots of beautiful
- 59:03architecture and beautiful art and
- 59:05people will spend time together in real
- 59:06life and with their kids and all that
- 59:08kind of it will be in some sense boring.
- 59:11It will just be I'm very inspired by the
- 59:15enlightenment the the period in western
- 59:17history where a lot of including you
- 59:20know the American constitution really
- 59:21got many of its ideas from and what I
- 59:23really like about the enlightenment is
- 59:25that sometimes critics of the
- 59:26enlightenment say oh it's this
- 59:27rationalism you know they're trying to
- 59:29be so rational and reduce everything to
- 59:30rationality but this is actually not
- 59:32true the enlightenment thinkers much
- 59:36preferred the the use of reason they
- 59:39wanted to be reasonable
- 59:40So for me, a good
- 59:43reason.
- 59:44>> Can you define it?
- 59:45>> That's part of the thing I'm trying to
- 59:47define is that all of us have an
- 59:48intuition of what is reasonable and it's
- 59:51not a formal thing. A a good world is
- 59:53not formalized. It's not a high
- 59:55modernist, you know, communist, you
- 59:57know, mechanical thing. It is for me in
- 1:00:00a good world, things still go wrong.
- 1:00:01Sometimes people make mistakes, you
- 1:00:03know, maybe a court made a wrong choice,
- 1:00:05you know, maybe, you know, there was a
- 1:00:07there's an accident. Like things like
- 1:00:09this still happen, but every time you
- 1:00:10look into it, you'll be like, "That's a
- 1:00:12reasonable mistake." Like you see why
- 1:00:14this would have happened and you know
- 1:00:16that we'll fix it. Like you know there's
- 1:00:18someone in charge and they'll be
- 1:00:20reasonable in the way to fix it. This is
- 1:00:22for me what a good world looks like.
- 1:00:23That when you know super intelligence is
- 1:00:25on the horizon that people will be
- 1:00:27reasonable. The scientists will go to
- 1:00:28the government and say hello we are the
- 1:00:30scientists and we think this and this
- 1:00:32will happen. And the government will be
- 1:00:33like yes this [clears throat] is a very
- 1:00:34reasonable thing. We're going to talk to
- 1:00:35all our international colleagues and
- 1:00:36figure out what the best way forward is.
- 1:00:38I'm not saying they're going to get it
- 1:00:39right on the first try, but along the
- 1:00:41whole chain, everyone will just be like,
- 1:00:43"Oh, yeah, that makes sense. That that's
- 1:00:45a reasonable thing to do." That's kind
- 1:00:46of how I think a good world would look.
- 1:00:48>> Is um AI as it exists today compatible
- 1:00:52with that world.
- 1:00:54>> As it exists today, it is not fatal to
- 1:00:57this world, but it is not AI in this
- 1:01:00world would not look like AI today.
- 1:01:02>> Interesting. Tell me why. because it's
- 1:01:04not aligned with people. AI most of AI
- 1:01:06by like you know dollar volume is like
- 1:01:08Tik Tok algorithms pushing slop to
- 1:01:11children to give them eating disorders.
- 1:01:13Like that's the primary me like it a lot
- 1:01:17of the funding that has gone into modern
- 1:01:20AI comes from companies who built up
- 1:01:22massive war chests from you know pushing
- 1:01:24eating disorders onto children for
- 1:01:26decades in the 2000s. Like that's where
- 1:01:28a lot of the money the cash came from
- 1:01:29originally. the funding for deep
- 1:01:32learning and neural networks so largely
- 1:01:34was for recommener algorithms and only
- 1:01:36later turns out was useful for other
- 1:01:38things as well. So no, I think that for
- 1:01:41example, in a reasonable world, if a
- 1:01:43company builds a product that, you know,
- 1:01:45for example, creates massive anxiety in
- 1:01:46teenagers, they wouldn't make any money.
- 1:01:48They would go out of business and maybe
- 1:01:50go to jail for that. And if someone made
- 1:01:52a product that makes, you know, uh,
- 1:01:54teenagers learn good things and make new
- 1:01:56friends and be protected from predators
- 1:01:58and, you know, grow as people, they
- 1:02:00should make a lot of money off of that
- 1:02:01and they should be, you know, rewarded
- 1:02:03for this. And we all, you know, give
- 1:02:04them medal and say good job, you know,
- 1:02:06like I I
- 1:02:09I don't I think this is possible. Like
- 1:02:11it's it's it's funny that people
- 1:02:12sometimes don't can't imagine that. Like
- 1:02:14obviously we could just make a social
- 1:02:15media that's nice. Like obviously we
- 1:02:17have the technology. You obviously if we
- 1:02:19design the algorithm to be empowering to
- 1:02:22people and good we could. It's just they
- 1:02:24would make less money. You will make
- 1:02:26less money. You know you can make a less
- 1:02:28addictive you know cigarette by removing
- 1:02:30the nicotine. It will be less addictive
- 1:02:33and it'll be less harmful. These things
- 1:02:34are related in various ways. So I think
- 1:02:38you can make very pro-social products
- 1:02:41with AI and with social media and all
- 1:02:44these other things but the and the
- 1:02:45bottleneck here is not technology. I'm
- 1:02:47not saying we need to invent some new
- 1:02:48technology we haven't invented yet. It
- 1:02:50is much more about regulation is much
- 1:02:52more about philosophy. It is much more
- 1:02:53about humane design about what do we
- 1:02:56reward with our markets? Do our markets
- 1:02:58reward whatever optimizes whatever
- 1:03:00reproduces the most or reward the thing
- 1:03:02that gives actual value. This is the
- 1:03:04core problem of optimization. If we
- 1:03:07could just define a magic number that
- 1:03:09contains all goodness in the world, well
- 1:03:12great, make the number go up. But every
- 1:03:14number we have obviously is not that.
- 1:03:17GDP isn't that. You know, GDP is
- 1:03:19definitely correlated with a lot of good
- 1:03:21things. I like having a lot of GDP. It's
- 1:03:23very nice. But at the same time, you
- 1:03:25know, like it's crazy. Like, you know, I
- 1:03:27know especially like kids younger than
- 1:03:28me. Do you know that like they don't
- 1:03:30dance at clubs anymore?
- 1:03:31>> It's wild because they get recorded. So,
- 1:03:34if you're like 17 and you're at a club
- 1:03:36or something, you dance and you approach
- 1:03:38a girl awkwardly and you mess it up, you
- 1:03:39get recorded. That's awful. This is this
- 1:03:42is terrible. And this is a So, from my
- 1:03:45perspective, from a like, you know, to
- 1:03:46be like an autistic economist, a massive
- 1:03:49harm was caused here. Like a like
- 1:03:51astronomical harm was caused to this
- 1:03:53teenager. From my perspective, I think
- 1:03:55this 17-year-old was caused a massive
- 1:03:57injury by this fact. Who's paying for
- 1:04:00that? Like, who's who's addressing this
- 1:04:03problem? who's making
- 1:04:04>> literally just cut a check for 17
- 1:04:05billion dollars to end all their
- 1:04:07lawsuits. But
- 1:04:09>> exactly.
- 1:04:09>> Yeah.
- 1:04:09>> But like that doesn't solve the problem.
- 1:04:12None of these things solve the problem
- 1:04:13because the problem isn't technology in
- 1:04:16a sense. It's more like what do we do
- 1:04:17with it?
- 1:04:18>> What do we want our society to look
- 1:04:20like? What are the institutions, the
- 1:04:21norms? Like like this is a small
- 1:04:24example. In Japan, every time you take a
- 1:04:26photo with your phone, it makes a noise.
- 1:04:29You can't turn it off. So that if people
- 1:04:31in public take pictures of each other,
- 1:04:32they at least know about it.
- 1:04:34>> Yeah.
- 1:04:35>> Extremely small, right? But this is
- 1:04:37obviously a good thing. Obviously, you
- 1:04:39don't want strangers taking pictures of
- 1:04:40you in public, you know, or at least
- 1:04:41want to know about it, right? And so
- 1:04:44this is an extremely small just social
- 1:04:46thing that you can change. And I think
- 1:04:48many many of the things we really need
- 1:04:49are like this in the history of like
- 1:04:52again like the enlightenment. What I
- 1:04:54thought was so beautiful, one of the
- 1:04:55things I find so beautiful in
- 1:04:56enlightenment is that the people pushing
- 1:04:58the technology and the people t pushing
- 1:05:00statecraft were in a sense often the
- 1:05:02same people or they were handinand. They
- 1:05:04took inspiration from each other. They
- 1:05:05were talking to each other constantly.
- 1:05:08For them, the design of a state of an of
- 1:05:10a society was another engineering
- 1:05:13problem. It was another question of how
- 1:05:14can we create a wonderful nice society
- 1:05:16for people. And what I feel nowadays is
- 1:05:19kind of like kind of what you were
- 1:05:20saying earlier is like a lot of people
- 1:05:23they call them like techno optimists are
- 1:05:25really civilizational pessimists. They
- 1:05:27just believe that the only way to
- 1:05:29improve the world is through technology
- 1:05:30rather than how we live our society, how
- 1:05:33our norms work, how our culture works,
- 1:05:35how our laws work, our institutions.
- 1:05:38Even so when I ask people what are the
- 1:05:40top 10 things that they're annoyed
- 1:05:41about, very rarely are those top 10
- 1:05:43things technological things. almost
- 1:05:44always is like, "Oh man, you know, I'm
- 1:05:47upset about politics. I'm upset about
- 1:05:49healthcare. I'm upset about um you know,
- 1:05:53all these kinds of like so I'm upset
- 1:05:54that I can't see my friends enough. I'm
- 1:05:56upset that you know, whatever, right?
- 1:06:00And I think we should take this
- 1:06:01seriously. I think this is, you know,
- 1:06:04I'm a technologist at heart, but what I
- 1:06:06care even more about is making the world
- 1:06:07a better place." And turns out
- 1:06:08technology isn't the bottleneck. I would
- 1:06:10love to get to a point where technology
- 1:06:11becomes the bottleneck, you know, where
- 1:06:13we've solved culture so well and we have
- 1:06:15such a beautiful government that now all
- 1:06:16we need to do is to invent better
- 1:06:18technology. I would love to live in this
- 1:06:20world.
- 1:06:22>> Yeah. Uh that's a really good point,
- 1:06:24man. That actually really hit me. Uh I'm
- 1:06:26very glad that you took the time to walk
- 1:06:28us through that. So, um one as a a way
- 1:06:32point, do you think that AI and
- 1:06:36capitalism are like a toxic pair?
- 1:06:39I'm personally a big fan of capitalism.
- 1:06:41I think capitalism is one of the best
- 1:06:42ideas humanity has ever had in
- 1:06:44moderation the same way as any ideas.
- 1:06:46There's a thing that I think sometimes
- 1:06:48gets forgotten is that markets are
- 1:06:50great. I love free markets. I love
- 1:06:52private enterprise. But it's one tool in
- 1:06:54our tool belt and we know for which
- 1:06:56problems it's good and for which ones
- 1:06:57it's bad. You know, if I gave you a
- 1:06:59really good power drill, you know, you
- 1:07:00use it to drill in a screw, you might be
- 1:07:02like, "Wow, this is the best tool ever."
- 1:07:04But it's not it's one tool. You know, if
- 1:07:06you need to saw a board, you're not
- 1:07:07going to use your drill. You're going to
- 1:07:08need a different tool for this. Markets
- 1:07:10and capitalism are very similar.
- 1:07:12Capitalism and markets solve many
- 1:07:14problems really well. You know, if you
- 1:07:16want a cheap, you know, commodified
- 1:07:17product delivered at a affordable price
- 1:07:19directly to your doorstep. By gism
- 1:07:25and markets have brought a massive
- 1:07:27amount of good to the world all, you
- 1:07:28know, over the last, you know, hundred
- 1:07:30years. A lot of bad, too. But
- 1:07:32fundamentally, markets are optimizers.
- 1:07:34They're like reinforcement learning.
- 1:07:36They have a number that they make go up.
- 1:07:38And if that number can go up by ways you
- 1:07:40don't want, markets will do it. And this
- 1:07:42is why we have regulation. In all of
- 1:07:45history, we've always paired to these
- 1:07:47things. It's kind of like It's kind of
- 1:07:49like MMA, you know? Like I love MMA. I
- 1:07:51think it's so fascinating. I don't do it
- 1:07:53myself, but I think it's such a
- 1:07:55fascinating thing because you have like
- 1:07:56the most dangerous people to ever live
- 1:07:58in history. the most dangerous warriors
- 1:08:00to ever live, you know, fighting, you
- 1:08:02know, to such an incredible degree. And
- 1:08:05they don't kill each other, you know,
- 1:08:07they can keep competing because we have
- 1:08:09rules. You need a lot of rules to allow
- 1:08:13this competition to be possible. You
- 1:08:15need a referee. You need a lot of rules
- 1:08:18to make sure that they can compete the
- 1:08:19next day, that they, you know, next
- 1:08:21match that they can come back. And
- 1:08:23markets are very similar. You don't
- 1:08:25regulate a market. You just get
- 1:08:26monopolies. you just get some
- 1:08:28corporations, you know, hire private
- 1:08:29security to, you know, break your
- 1:08:31kneecap so you don't buy your product,
- 1:08:32right? That's no good. To get a
- 1:08:34competition, you actually have to have a
- 1:08:35lot of rules. And I think this is
- 1:08:37beautiful. I think if we make, you know,
- 1:08:39good rules and we have a good referee
- 1:08:41that is not the market, you know, that
- 1:08:42is outside the market, such as the
- 1:08:44government, we can make beautiful
- 1:08:45things. I think this is an example of
- 1:08:48just us failing to do our job. Like it's
- 1:08:50kind of like imagine someone in MMA
- 1:08:52found a new chokeold that makes them win
- 1:08:54but also kills the other guy and we just
- 1:08:56don't ban it. Like if we don't do that
- 1:08:58then the sport of MMA stops existing and
- 1:09:00so do all the people who participate in
- 1:09:01it. And there's a very similar thing
- 1:09:03here where this has happened throughout
- 1:09:05history. Corporations find new exploits.
- 1:09:08They find new edge cases that we didn't
- 1:09:09think about before. You know they find
- 1:09:10new ways to pollute. They find new ways
- 1:09:13to harm people, to scam people,
- 1:09:14whatever. And then we react to that.
- 1:09:16Then we're like, "Wait, hold on. You
- 1:09:18can't do that." You know, before nuclear
- 1:09:20weapons existed, they obviously weren't
- 1:09:22illegal. And then once we realized that
- 1:09:24they were they were possible, we really
- 1:09:26quickly made sure that they were illegal
- 1:09:28for the market to build because
- 1:09:29otherwise the market would be happy to
- 1:09:31build nuclear weapons. Could you
- 1:09:32[laughter] imagine, you know, how much
- 1:09:33big tech companies would love to build
- 1:09:35nuclear weapons and sell them to the
- 1:09:36highest bidder? Of course they would.
- 1:09:38They would love that. So it's always
- 1:09:40it's a design problem and it's an
- 1:09:42evolution process. It's a it's it's an
- 1:09:44iterated process over time. We're never
- 1:09:46going to get it right on the first try
- 1:09:47and that's okay. That doesn't mean
- 1:09:48markets are fun, well we should throw
- 1:09:50out all markets and move to communism.
- 1:09:52Neither does it mean we should have no
- 1:09:53regulation and just do markets. It's
- 1:09:55just we're trying to solve a really hard
- 1:09:57problem which is how can we get an
- 1:09:59unaligned thing which is the market like
- 1:10:02an market quote at large does is not a
- 1:10:05good or a bad thing. It's like an alien.
- 1:10:07It's like an AI you know it's like an
- 1:10:09AI. It's like a reinforcement learning
- 1:10:10AI. And how can we get it to do things
- 1:10:13that we want? And that's hard. We've
- 1:10:16made a lot of progress on it and we just
- 1:10:18have to keep iterating. Mhm.
- 1:10:19>> If you were going to regulate AI now to
- 1:10:22make sure that we don't cross the point
- 1:10:24of no return, have you thought about
- 1:10:26what the specific policies you'd want
- 1:10:28passed are?
- 1:10:29>> Yes. Um the there's two major things
- 1:10:32that we think should be passed. The
- 1:10:33first and foremost is you should
- 1:10:34criminalize the creation of super
- 1:10:36intelligence. This is a thing you can
- 1:10:37just do in law. You know, it's illegal
- 1:10:39to murder someone. It's illegal also to
- 1:10:41attempt to murder someone. Even if you
- 1:10:43fail, you still can't do that.
- 1:10:44>> Similar thing with nuclear weapons. Even
- 1:10:46if you attempt to build a nuclear
- 1:10:48weapon, even if you don't succeed, the
- 1:10:49men in black show up at your door, you
- 1:10:51know, and this is good. So this is a
- 1:10:53thing you can just do, you know, and
- 1:10:55it's much easier. You It's quite easy
- 1:10:58often to tell the difference between,
- 1:10:59you know, a company that's building
- 1:11:00trying to build super intelligent and
- 1:11:02it's not trying to build super
- 1:11:03intelligence. Not perfect, but it's a
- 1:11:05start. So this is the first common sense
- 1:11:07thing. It's just if you're literally
- 1:11:09advertising, I am building super
- 1:11:10intelligence. I just raised a billion
- 1:11:12dollars to build super intelligence.
- 1:11:13That's illegal and you can't do that.
- 1:11:15Then [snorts] the second thing is how do
- 1:11:17we prevent us from getting too close to
- 1:11:20super intelligence and for this
- 1:11:21fundamentally because again if a super
- 1:11:22intelligence is built even once even
- 1:11:24accidentally it's too late. So we have
- 1:11:28to regulate
- 1:11:29well before super intelligence so we
- 1:11:31have enough buffer zone. It's kind of
- 1:11:33like imagine we're going 120 m hour uh
- 1:11:35in thick fog and we know somewhere
- 1:11:37there's a cliff but we don't know where.
- 1:11:39>> Yeah.
- 1:11:40>> My first suggestion is pull over.
- 1:11:42[laughter]
- 1:11:42Before we argue about the speed, let's
- 1:11:45first pull over. So, first criminalize,
- 1:11:49you know, super intelligence and then
- 1:11:50regulate the precursors. I have a list
- 1:11:53of precursors such as ability to
- 1:11:55self-reroduce
- 1:11:57um task on horizon length things related
- 1:12:00to genetic autonomy and so on and
- 1:12:03monitor those very carefully and be sure
- 1:12:05that any credible large experiment that
- 1:12:09pushes on these frontiers is registered
- 1:12:11with the government is that has
- 1:12:12oversight and that we can decide no you
- 1:12:14can't do that that's too risky because
- 1:12:16luckily at the moment building really
- 1:12:20frontier AI that pushes towards AGI or
- 1:12:22ASI is unbelievably expensive. You know,
- 1:12:25we're talking billions of dollars. So,
- 1:12:26there's actually not that many actors
- 1:12:29that this law would even affect. This
- 1:12:31wouldn't affect 99% of companies. You
- 1:12:33know, there would be a very small
- 1:12:34percentage of mega mega huge tech
- 1:12:37companies that would be affected by
- 1:12:38this. And for them, yeah, we should
- 1:12:40obviously have oversight of them the
- 1:12:42same way we have oversight of like
- 1:12:43Loheed Martin or, you know, nuclear
- 1:12:45weapons facilities and so on.
- 1:12:48>> What about open source AI? Open source
- 1:12:50is a very tricky one in the sense that I
- 1:12:53love open source. I built some of the
- 1:12:56first open source large language models
- 1:12:57in the world. In fact, I led a group
- 1:12:59called the Luther AI. I've been I grew
- 1:13:00up in the open source world. I think
- 1:13:03open source has brought so much good to
- 1:13:04the world. I think Linux being open
- 1:13:06source and all open brought so much
- 1:13:08beauty and good into the world. But for
- 1:13:11me, it's not ideology. It's a question
- 1:13:13of, you know, should the blueprints for
- 1:13:15the F-35 fighter jet be open sourced?
- 1:13:18>> Probably not. probably not. That seems
- 1:13:20like it wouldn't make the world a better
- 1:13:21place. You know, should blueprints for,
- 1:13:23you know, bioweapons or, you know,
- 1:13:26nuclear bombs be open sourced? Probably
- 1:13:28not. You know, and here's a very similar
- 1:13:31thing where if we get to the point that
- 1:13:34there are open- source systems that are
- 1:13:36super intelligent or that can be made
- 1:13:38super intelligent, which I think we're
- 1:13:40pretty close to, then I think there's
- 1:13:42probably no going back in this regard.
- 1:13:46you know, if there's like one system
- 1:13:48that's almost super intelligent and it's
- 1:13:50only on one server in one country, you
- 1:13:52know, maybe we can delete it, you know,
- 1:13:54in the maybe. But if there are open-
- 1:13:57source billions of things around and we
- 1:14:00know they are super intelligent or like,
- 1:14:01you know, two steps away from super
- 1:14:03intelligent, it's probably too late. I
- 1:14:05think the action that would be needed
- 1:14:06to, for example, you know, recall quote
- 1:14:08unquote an open source model are not
- 1:14:11feasible. M. All right. Bill Gates just
- 1:14:13said that he now thinks AI is dangerous.
- 1:14:17What happened? What made him change his
- 1:14:18mind?
- 1:14:19>> I'm not sure really how much he changed
- 1:14:20his mind. Of course, I don't know him. I
- 1:14:22think, you know, he's a he's a very, you
- 1:14:24know, he knows a lot of things. He's a
- 1:14:26smart guy. I'm sure he's seen a lot of
- 1:14:27scary stuff. But even all the way back
- 1:14:29in 2023, Bill Gates actually signed the
- 1:14:32Center for AI safety statement which
- 1:14:34said that the mitigation of the risk of
- 1:14:36extinction from AI should be a global
- 1:14:39priority. This was signed by Bill Gates,
- 1:14:41Sam Alultman, Dario Amade, Joffrey
- 1:14:44Hinton, like lots of really, you know,
- 1:14:46big hitters. So, this was all the way
- 1:14:48back in 2023. So, I think, you know,
- 1:14:50again, I don't know him, but I think
- 1:14:51Bill Gates has always known of some of
- 1:14:53these risks and just has now been trying
- 1:14:55to maybe he's now trying to message a
- 1:14:57bit more about them. I personally wish
- 1:14:59he would have gone a bit further in his
- 1:15:00letter. I think it was, you know, it did
- 1:15:02acknowledge some of the risk but didn't
- 1:15:04go all the way to super intelligence and
- 1:15:06extinction risk which is the thing that
- 1:15:07I am really so concerned about because
- 1:15:10in a sense I don't think we have enough
- 1:15:12time that we will see massive job
- 1:15:14displacement like I think we will just
- 1:15:16see super intelligence in the next
- 1:15:17couple of years way before massive job
- 1:15:20displacement even happens and it's
- 1:15:22already too late. I kind of wish we were
- 1:15:24in a world you know where super
- 1:15:26intelligence is 20 years away then job
- 1:15:28displacement will become a huge problem.
- 1:15:30But I'm not sure we're going to get to
- 1:15:32that point even.
- 1:15:33>> That's wild. That's wild. Okay. Um I
- 1:15:36want to see if I got your base
- 1:15:37assumptions right. If I've missed
- 1:15:39anything, let me know. But this has been
- 1:15:41very helpful. Uh these are loosely in
- 1:15:44order, but um and there's one that I
- 1:15:46have a follow-up question on. Okay, so
- 1:15:49super intelligence is definitionally you
- 1:15:52you um maybe didn't go quite that far,
- 1:15:54but super intelligence is adversarial.
- 1:15:56So knowing that once you make that it is
- 1:15:59going to compete and you're going to
- 1:16:00have problems therefore uh super
- 1:16:02intelligence will consider us a
- 1:16:04distraction or impediment to their goals
- 1:16:06and so we have to worry about being the
- 1:16:08very thing they're competing against and
- 1:16:10we will lose. Um unknown risk is
- 1:16:13evaluated at near peak danger levels
- 1:16:16meaning if I don't know the outcome of
- 1:16:18this like you said first pull over if we
- 1:16:20don't know where we're going. So anytime
- 1:16:22we terminate in I don't know what AI is
- 1:16:24going to do or I don't know what the
- 1:16:25super intelligence is going to do here.
- 1:16:26Pump the brakes.
- 1:16:27>> Obviously there's spectrum here. Um as
- 1:16:30with any complex strategy. How do you
- 1:16:31deal with known unknowns or unknown
- 1:16:33unknowns? How do you calculate the
- 1:16:36probability that a country is going to
- 1:16:37invade you tomorrow? This is actually a
- 1:16:39problem there's a lot of literature on.
- 1:16:40This is a thing that you know the CIA
- 1:16:42and you know various military agencies
- 1:16:44have spent many many decades working
- 1:16:46very very hard on. One of the
- 1:16:47recommendations, for example, the CIA
- 1:16:49makes is that always word your gut
- 1:16:51feelings in numbers. So, this is a thing
- 1:16:53we didn't talk about. People often get
- 1:16:55annoyed when people say like, "Oh, I
- 1:16:56think there's a 20% chance of things
- 1:16:58going well." You're like, "That's not
- 1:16:59math." And I'm like, "Dude, [laughter]
- 1:17:02yeah, they know. I know it's not math."
- 1:17:05The reason you say, "Oh, 20%." is not
- 1:17:07because I'm trying to make a
- 1:17:08mathematical statement. It's because
- 1:17:10there's the famous study, I think it was
- 1:17:12from the CIA, where they found that
- 1:17:14analysts who said likely or unlikely
- 1:17:17actually meant wildly different things.
- 1:17:19Like some people when they said likely,
- 1:17:20they meant 90%. Other analysts when they
- 1:17:23said likely meant 30%.
- 1:17:25>> And so this was extremely confusing
- 1:17:27because unlikely likely doesn't mean
- 1:17:29anything. So if you say like 90% it's
- 1:17:32still a vibe, but it's a much more
- 1:17:33concrete vibe, right? It's a much more
- 1:17:35when I'm communicating to you. If I say
- 1:17:371%, I'm trying to communicate not oh
- 1:17:40it's definitely 1% and not 1.2%. That's
- 1:17:42not what I'm saying. What I'm saying is
- 1:17:43it's on the order of 1%. It's like given
- 1:17:46my beliefs about the world and you know
- 1:17:48looking at my models of reality which
- 1:17:50you know you can evaluate to how much
- 1:17:52you believe my models of reality
- 1:17:55this outcome has on the order of this
- 1:17:57percentage of outcomes. Of course it's
- 1:17:59based on my models. What else is it
- 1:18:01supposed to be based on? Right? Like
- 1:18:03obviously you have views of the world.
- 1:18:05You know if I asked you how likely do
- 1:18:06you think it is that you know this thing
- 1:18:08will happen in the future how do you
- 1:18:10evaluate this well you use your model in
- 1:18:12your head what else are you going to do
- 1:18:14you know so it's a very similar thing
- 1:18:16like I try to make my model explicit
- 1:18:17like with these base assumptions I try
- 1:18:19to make it pretty explicit this leads to
- 1:18:20this leads to this leads to this
- 1:18:22therefore high probability so if all of
- 1:18:25these statements that we've said so far
- 1:18:27were like you know a billionth of a
- 1:18:30percent probable I wouldn't be worried
- 1:18:32but I don't think that's the case I
- 1:18:33think if they're all like you know
- 1:18:35higher than 1% and after their
- 1:18:37conclusion are still more than you know
- 1:18:39after multiplying them all together
- 1:18:40we're still above 1% and the outcome is
- 1:18:43extinction hell no you know if the
- 1:18:45outcome is like oh it's a one in a
- 1:18:46trillionth I'm like sure but that's not
- 1:18:48where my numbers come out
- 1:18:49>> right makes sense okay um financial
- 1:18:53incentives power incentives and lack of
- 1:18:55strong regulation have allowed us to
- 1:18:58keep racing forward despite the dangers
- 1:19:00>> yep
- 1:19:01>> uh for AI to work it requires the AI to
- 1:19:04have a will of its own.
- 1:19:06>> Depends on what you mean by will. I
- 1:19:07think it's
- 1:19:08>> it's going to be an optimizer. Now that
- 1:19:10I know the language you use,
- 1:19:11>> this is how I would describe it. It will
- 1:19:13have to optimize for goals of some kind.
- 1:19:15It will take actions. It will make
- 1:19:16plans. It would it will optimize towards
- 1:19:19objectives.
- 1:19:19>> And the sociopathic optimizer, as you
- 1:19:21described it earlier, is the problem.
- 1:19:24>> Yes.
- 1:19:24>> Okay. AI,
- 1:19:25>> there's some other problems, too, but
- 1:19:26this is really quite the heart of it.
- 1:19:28>> Okay. AI. It's too complex to imbue AI
- 1:19:32with morality. Yep. We have no idea how
- 1:19:33to do it.
- 1:19:34>> Okay. Uh, life can be improved by things
- 1:19:36other than technology.
- 1:19:38>> Yep.
- 1:19:38>> Uh, it's possible to get China on board.
- 1:19:41>> Yep.
- 1:19:42>> And humans are not magical. This is the
- 1:19:44one that I need more clarification on.
- 1:19:46It really hit me when you said it, but I
- 1:19:48wanted to better understand. You were
- 1:19:50trying to explain something about AI by
- 1:19:52saying humans are magical. You need to
- 1:19:53understand that.
- 1:19:54>> What What do you mean when you say
- 1:19:56humans aren't magical, and what does
- 1:19:57that tell us about AI? There is a very
- 1:20:00fun um thing as a Wikipedia article
- 1:20:02called the AI effect. I I recommend
- 1:20:05looking up the Wikipedia article. It's
- 1:20:07quite fun. Which is that throughout the
- 1:20:09history of AI, people would always
- 1:20:11define like, oh, we'll know we have real
- 1:20:13AI when X. And then once X was
- 1:20:15accomplished, everyone like, oh no,
- 1:20:16that's not real AI. That's not true
- 1:20:18intelligence. So, for example, back in
- 1:20:19the day, people thought that the thing
- 1:20:21that makes humans unique, different from
- 1:20:23all animals, is that we can do math.
- 1:20:26And well obviously computers can do math
- 1:20:28>> very well.
- 1:20:29>> So very well. So so then they're like
- 1:20:31okay no no no that's not what we mean.
- 1:20:32What we mean is actually the thing that
- 1:20:34makes humans magical is that we can
- 1:20:36recognize images or whatever you know.
- 1:20:38And then you're like or they can play
- 1:20:40chess or you know and obviously none of
- 1:20:42those things were the thing. And so
- 1:20:45there's a there's like this like rising
- 1:20:46tide rising waterline or or like you
- 1:20:49know like being packed into a corner of
- 1:20:50like this this very funny cartoon. And I
- 1:20:52think it was maybe from Ray Curtzwell or
- 1:20:53something where you have this guy
- 1:20:55frantically writing paper like only
- 1:20:56humans can do only humans can play chess
- 1:20:58and like strikes it out like only humans
- 1:21:00can draw pictures strikes that out and
- 1:21:01like and like his whole room is covered
- 1:21:03with these paintings of him and like
- 1:21:05only humans can do uh uh uh you know and
- 1:21:08what I'm saying about this is that
- 1:21:09humans are physics. We are in fact part
- 1:21:12of physics. There's nothing magical
- 1:21:14going on. It's complex. Our brains are
- 1:21:17complex but it's not magic. It's it's a
- 1:21:19it's a physical process. There is some
- 1:21:21computation happening in our brain of
- 1:21:24some kind. You know, it's being, you
- 1:21:26know, it's using some weird squishy, you
- 1:21:28know, elements for the calculations. But
- 1:21:29fundamentally, some kind of calculation
- 1:21:31is going on. It's not magic. You can
- 1:21:32make other things do similar
- 1:21:34calculations.
- 1:21:35>> So, it's a way of explaining we know
- 1:21:37intelligence is possible without some
- 1:21:39sort of magical element.
- 1:21:40>> Exactly.
- 1:21:40>> It makes all the sense in the world,
- 1:21:42dude. This is crazy. Um, this has really
- 1:21:45been amazing researching you and sitting
- 1:21:47across from you. I really, really
- 1:21:48appreciate I think your messages come in
- 1:21:50loud and clear. If there was one thing
- 1:21:53that um people may not realize, like if
- 1:21:55you were sitting across from Xi or
- 1:21:58somebody who you're like, I know they're
- 1:22:00convincible, is there one final thing
- 1:22:02that hasn't come up today that you'd
- 1:22:03want them to be aware of?
- 1:22:04>> I have different answers for Xiinping
- 1:22:07than I do for people listening. And so
- 1:22:08I'll give two answers. And so honestly,
- 1:22:11the number one thing I have found is the
- 1:22:14grown not written. People don't know
- 1:22:15this. when I talk to politicians is the
- 1:22:17first thing basically out of my mouth is
- 1:22:19explaining that this is not normal
- 1:22:21software. It is not written. It is
- 1:22:22grown. We do not understand how these
- 1:22:24things work. We did bring this up but
- 1:22:26I'm repeating it again because this is
- 1:22:27just so important to understand. The
- 1:22:29people at open do not know what is going
- 1:22:32on inside of their AIS.
- 1:22:33>> Dario Amade, the CEO of Anthropic thinks
- 1:22:35he we understand maybe 3%
- 1:22:38>> oh my god
- 1:22:38>> of what goes on in our AIS. And I think
- 1:22:41that's even optimistic.
- 1:22:42>> Whoa. So this is one of the most
- 1:22:44important things to understand. We don't
- 1:22:46know. We they don't know. No one knows.
- 1:22:49This is very important. And for the
- 1:22:51general public and everyone listening, I
- 1:22:53think the very important thing is is
- 1:22:54that it's not over. And even though it's
- 1:22:57sometimes unfashionable say to say this
- 1:22:59these days, we do still live in a
- 1:23:00democracy. And your voice is important.
- 1:23:04If you want something to happen, the way
- 1:23:06I like to think about civics is that our
- 1:23:09job is to work with our politicians to
- 1:23:11help them understand what we care about
- 1:23:13and what can be done. So do your part to
- 1:23:16help m bring these issues to awareness.
- 1:23:18Go to controlai.org. Contact your
- 1:23:20lawmaker. Tell them you're concerned
- 1:23:21about these issues. Help educate people.
- 1:23:24Keep having the conversation. This is
- 1:23:26how we humans solve all these problems,
- 1:23:28you know, together.
- 1:23:29>> I love it. Where can people follow you?
- 1:23:32Um, I'm you can find me on Twitter, you
- 1:23:35know, and now X, sorry, uh, at MPoll
- 1:23:37Collapse. You can find me obviously
- 1:23:39controlai.org. And also, if someone
- 1:23:41wants to engage even deeper on some of
- 1:23:43these things, I actually run a volunteer
- 1:23:45group called Torchbearer,
- 1:23:47torchbearer.com community, where we
- 1:23:49commit to spending at least two hours a
- 1:23:50week to try to build that reasonable
- 1:23:52world.
- 1:23:53>> I love it. All right, everybody. If you
- 1:23:55haven't already, be sure to subscribe.
- 1:23:57And until next time, my friends, be
- 1:23:59legendary. Take care. Peace. If you like
- 1:24:01this conversation, check out this
- 1:24:03episode to learn more.
- 1:24:05>> AI is easily the greatest discovery
- 1:24:08humanity will ever make. This is not an
- 1:24:11easy game to win or to start, but this
- 1:24:14is not going to be stopped.
- 1:24:15>> AI is sucking up all of the liquidity
- 1:24:18and it's causing a problem in crypto.
- 1:24:20And now.
About this transcript
This page contains the full transcript of EMERGENCY PODCAST: ASI Will Kill Us All! by Tom Bilyeu, generated from the public captions YouTube serves with the video. The transcript has 17,637 words across 2,553 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.