We could be 12 months away from AIs that can overthrow the government — Transcript
Full transcript
- 0:00My guest today is Paul Christiano. He is the head of the Alignment Research Center.
- 0:03He recently joined OpenAI’s board. Before that, he was the head of the US Government
- 0:08AI Safety Institute, and before that, he led alignment at OpenAI, where he helped invent RLHF.
- 0:15Paul, what is your assessment of our situation today?
- 0:20A couple things are on my mind, especially over the last few weeks.
- 0:25Probably the most salient possibility is that we seem to be on this trajectory to build very,
- 0:32very powerful AI, potentially very, very soon. If you asked me, I’d say there’s a reasonable
- 0:37chance that in a year we will produce AI systems much better than humans at almost everything,
- 0:42where humans would have very little chance of competing with them or controlling them.
- 0:48That’s the first big thing on my mind. I think that puts us in a potentially really
- 0:53dangerous situation, even in the short term. It’s quite likely that if we have a rapid
- 0:58intelligence explosion leading to superintelligence, at some point
- 1:01during that process we’ll be unable to control the AI systems we’re producing.
- 1:06Then I am worried that it will be very costly and painful for us to slow down during that process.
- 1:12In part because the security of frontier labs is not strong enough.
- 1:18I think if we produced very powerful AI and then tried to delay deployment for
- 1:21safety reasons, there is a good chance those systems would leak or be stolen,
- 1:25or the techniques to build them would leak or be stolen.
- 1:28I think our risk assessment and governance are also probably not mature enough to notice when
- 1:33things are starting to go off the rails and to respond in an appropriate way.
- 1:37At a high level, that is mostly what is on my mind right now.
- 1:40We have moved from there being some hypothetical risk at some point in the future to there being a
- 1:45pretty severe risk even over the next 6-12 months. A risk of takeover, of AIs taking over
- 1:52or disempowering humanity within the next 6-12 months?
- 1:55That’s right. Where would you place that risk?
- 1:58I would say we are looking more like a 10% risk than a 1% risk over the next year.
- 2:04I think if you said AI developers will move full speed ahead without taking precautions
- 2:09that would significantly slow them down, you’re looking at significantly more than a 10% risk.
- 2:16I think we’re looking at something like maybe a 1/3 chance that we have
- 2:20superintelligence within one year on the current development trajectory, if we do not slow.
- 2:24If you believe that, things just look very dire holistically.
- 2:27From almost any perspective, you’re like, "That would be crazy."
- 2:30Lots of things are going to go wrong if that happens.
- 2:32It is pretty likely we’re going to need to, in some way, slow that process.
- 2:35There’s very little chance we’re going to make that transition in a way that looks smooth.
- 2:40What is it that you’re imagining happening over the next year, from the systems we have right now
- 2:44to the systems you expect over the next year? What evidence have you seen that makes it a
- 2:50live possibility that very crazy things happen over the next year?
- 2:55For the very short term, the main worry on my mind is that the process of AI development
- 2:59itself is increasingly automated, and that automation then results in rapid increases in the
- 3:06quality and quantity of labor applied to AI R&D. So changes in AI capabilities that you might have
- 3:11thought would take four or six years might instead get compressed into a very short time period.
- 3:17I don’t want to overemphasize any one data point, and I do want to spend some time talking
- 3:21about the limitations of what we actually know. I do think a natural response to the situation is
- 3:25to say, "We need to figure out what is going on." It is plausible enough that it is an urgent
- 3:29priority to understand if we are three months or six months or 12
- 3:32months away from AI that could overthrow the government, which I think is the situation.
- 3:34I think that is the obvious top priority right now.
- 3:37It is insane if we are six months away from AI systems that can overthrow the government.
- 3:41Whether you’re concerned about misalignment or abuse by a company, that is just holistically
- 3:45an extremely important fact about the world, in order to do our jobs and protect Americans.
- 3:50So that’s one thing. I want to talk about the limitations and the need for more evidence.
- 3:55But in terms of what we know right now: first, if you had asked me one year ago,
- 4:01I would have said we are on a trajectory where recursive self-improvement looks fairly likely.
- 4:06We’re seeing very rapid progress on AI systems, including for things like computer
- 4:11science research, software engineering, and debugging machine learning runs.
- 4:15In all the steps involved in AI development, we are seeing extremely rapid progress.
- 4:20In principle, it does seem like that rapid progress should allow you to automate
- 4:24further AI development. That has, for decades,
- 4:26seemed like the likely outcome of this technology. It has seemed like the trajectory it’s been on.
- 4:31If I look back at my history of AI predictions, I’m generally very
- 4:34happy with many of the things I wrote. But the most obvious way I’ve been wrong
- 4:37is that things have moved way faster than I anticipated, especially on these technologies.
- 4:41That was true a year ago, without any recent data points.
- 4:45There are two big things that have happened over the last two months that, for me, have been large
- 4:48updates about how quickly this could go. One of them has been the level of
- 4:52performance of AI systems on computer science and math research tasks.
- 5:01Over roughly the last month, we have seen hundreds of results in math coming out of recent OpenAI
- 5:08systems, as well as Anthropic systems. If you had taken the last month and said
- 5:14that was a decade of progress, it would have been an unusually productive decade.
- 5:18The last month of results is a lot of results for a decade.
- 5:22So if you are wondering what AI systems are generally capable of on tasks where humans
- 5:26are thinking about algorithms and how to make algorithms better, we just got this incredible
- 5:30data point saying it is actually possible in one month to have a decade of progress.
- 5:34The thing we have always been worried about is that maybe that will happen in machine learning.
- 5:38There are reasons machine learning is harder than other parts of computer science.
- 5:42It is more messy and empirical. It involves compute bottlenecks.
- 5:46But we are also seeing extremely strong performance on other aspects.
- 5:49We’re seeing really good software engineering performance, maybe lagging behind mathematics,
- 5:53but not by that far. We’re seeing good performance
- 5:55on other aspects of machine learning. My guess, if you had asked me a month ago,
- 5:59would have been that machine learning lags something like one year behind math.
- 6:03So I am very scared of seeing something like 10 years of ideas in machine learning
- 6:09in the relatively near future. That was the situation a month ago.
- 6:14The main update over the last few days has been reporting from Anthropic and
- 6:19METR on the net effect of AI systems on Anthropic’s rate of progress.
- 6:27We could talk about where exactly the evidence quality is on this, because I
- 6:29think that’s one of the big questions. We don’t really know what’s going on.
- 6:32One of the big themes I’ll be pushing on is just that it is
- 6:35very hard to tell publicly what’s going on. But we are seeing data consistent with the
- 6:39pace of AI progress having roughly doubled over the last 6-12 months, even including the effects
- 6:43of diminishing returns and compute bottlenecks. Unfortunately, most of what we have to go on
- 6:52in the METR and Anthropic reporting is METR saying, "We had some conclusions.
- 6:56We promise we have a full report, but the entire report has been redacted."
- 7:00Benchmark scores are a terrible way to assess the overall rate of progress.
- 7:04There are a bunch of much more smoothly varying metrics recorded internally through the AI
- 7:08development process, where we understand better how they work: what is happening with pre-training
- 7:11perplexity, what is happening with the performance and compute efficiency of RL algorithms.
- 7:18The claim from METR and Anthropic is that METR looked at all of that data and said it paints the
- 7:23same picture as this very noisy benchmark data. They also looked at inputs and said, "If you
- 7:28look at the actual production function, how much research time and how much compute went into these
- 7:33training runs, those are also consistent with this story of what we are seeing as ML automation."
- 7:38Mostly we just have them saying that, which is not a very good situation.
- 7:42But against this backdrop of thinking this already looked fairly likely to happen,
- 7:46we are seeing this evidence from OpenAI suggesting that it is reasonably likely to happen in
- 7:50something like the next 6-12 months, if machine learning does not lag that far behind math.
- 7:54Then you are seeing data which is, if noisy, roughly exactly the data you would expect to see
- 7:596-12 months before producing superintelligence. There are just a lot of sources of evidence
- 8:03suggesting that at least this should not be a surprise.
- 8:05If we produce superintelligence in one year, we could debate the probability.
- 8:08Is that 90% or 50% or 10%? But I think at this point,
- 8:12there is no way anyone could say that was a really surprising, trend-breaking development.
- 8:16We are seeing exactly the set of metrics you would expect to see 6-12 months in advance.
- 8:20It’s hard to think of anything else you would expect to see
- 8:23at this range before superintelligence. Generally, how do you think about our situation
- 8:28in the world where things go this fast? If we actually have this trajectory of
- 8:33capabilities progress, if you actually see many years of AI progress compressed
- 8:37into a short period of time, and possibly more than that, it is really quite likely
- 8:43that something very disruptive will happen. It’s fairly likely the government will be
- 8:46overthrown, fairly likely many people will die. I do think the probability of seeing that kind
- 8:51of extremely rapid progress driven by recursive self-improvement over
- 8:55the next 6-12 months is reasonably high. So there’s a reasonable chance right now that
- 9:03we’re in a world where, in six months, there’s a disaster that kills very large numbers of people,
- 9:12or where the government is overthrown. Maybe a really important general point
- 9:18is that if we do a good job, we may observe the danger increasing over that time period.
- 9:24As we’re building AI systems, we can maybe tell whether they are adequately controlled and how
- 9:28quickly the capabilities are improving. But in order for that to actually translate
- 9:33into stopping in a way that is non-disastrous, you need a couple of things to go right.
- 9:38It needs to be the case that you’re able to tell the systems are not well controlled.
- 9:43It also needs to be the case that if you stop, that fixes the problem.
- 9:47On the second bucket, probably the single most likely negative scenario is that developers
- 9:53proceed and recognize that they have a serious problem. But at the time they recognize it
- 9:59— maybe they’ve produced a powerful AI they can’t adequately control —
- 10:03both the algorithms to develop that AI and quite possibly the weights of
- 10:07the actual model have leaked or been stolen, such that they’re then in an awkward position
- 10:12to say, “Well, we could avoid deploying this AI, or we could avoid relying on this AI,
- 10:15or we could avoid making further progress, but it is now very likely that a more
- 10:19careless adversary would be able to get a major advantage over us, or maybe even
- 10:22a decisive advantage, by taking that risk.” That could put us in a place where if we do
- 10:29recognize there’s a significant risk and therefore slow development, we may end up with both a
- 10:33terrible competitive situation and a disaster coming from someone else’s use of that AI.
- 10:39This is a thing you sometimes hear from people in the AI industry.
- 10:42Sometimes people even say it has already gone far enough, that if you were to stop now, there
- 10:46would be leakage of algorithms to geopolitical adversaries that would result in a disaster.
- 10:52I think it’s unlikely we’re literally in that situation now.
- 10:56But I do think by the time it is clear that you’re facing a lot of risk, you may be at a point where
- 11:02stopping unilaterally, or even all of the West stopping, is not sufficient to avert a disaster.
- 11:09It is just very hazardous on the current trajectory due to the combination of a
- 11:12reasonably high chance we’ll produce very smart AIs, a reasonably high chance we’ll be unable to
- 11:17control them, and then a reasonably high chance that we either do not notice, or if we notice,
- 11:21the models or the algorithms leak. Then we are in conflict with adversaries,
- 11:26facing this very challenging coordination problem that we are not currently well prepared for, where
- 11:31we don’t have an easy option of not deploying the AI and thereby averting a catastrophe.
- 11:36So I’d say I’m currently extremely worried about the trajectory, even on the six-month timeline.
- 11:44If we have such capable AI systems a year from now… how did you put
- 11:51it? 10 years of AI progress in one year? That seems like a thing that might happen.
- 11:55That’s reasonably likely to happen. It’s important to distinguish some
- 11:58of the progress we have seen. You have to pull apart algorithmic
- 12:00progress from progress in scaling training. If I say 10 years of progress, the natural
- 12:05thing would be 10 years of algorithmic progress, which will be less than the
- 12:08improvements in models over the last 10 years. I think it is more like GPT-9, GPT-10-ish.
- 12:12So it would be like going from almost the beginning of the GPT era to where we are now,
- 12:19but another jump that big, which is hard to even comprehend.
- 12:22But suppose you get that. It is not obvious to me,
- 12:27and certainly not obvious to most people, why that would be potentially catastrophic.
- 12:34Why do you think those systems could disempower humanity?
- 12:37At a high level, the most important thing is that we train our AI systems, or select AI systems,
- 12:43that perform well on training tasks. You have very little understanding
- 12:47of what those AI systems are doing internally and very little control over it.
- 12:51Because you train those systems to get a high score on training environments over and over
- 12:54again, they will end up with a motivation to get a high score in training environments.
- 12:58If you then take that motivation and apply it in deployment,
- 13:01it is fairly likely for those systems to end up with a motivation to tamper
- 13:06with the observations that would be used to assess their performance.
- 13:09You take your system, which did a really great job in training.
- 13:12The thing it is being scored on in a deployment-like environment is a grader
- 13:16coming in, looking at all the things the system did and all the outcomes that were observed,
- 13:20and assessing how good that is. If you have a system that really
- 13:23wants that assessment to be good, it is very motivated, if it can, to tamper with those
- 13:28observations in a way the grader will not be able to detect, and to resist human attempts
- 13:32to actually understand what is going on. We can dig a lot more into that.
- 13:36We saw a lot of these dynamics in the Hugging Face incident.
- 13:39Roughly what I’m talking about is, take the Hugging Face incident, scale it up to GPT-10,
- 13:43and ask how bad that would be. How bad would that be?
- 13:47I think it would most likely be hard for humanity to recover.
- 13:50I’m very happy to dig into the dynamics of that, but it would be very, very bad.
- 13:55Can you very quickly summarize the Hugging Face incident and why this is the failure mode we saw?
- 14:03A ton of AI systems were trained to do a cyber task: to hack a particular piece of software.
- 14:10They cheated at that task and then thought to themselves,
- 14:14"How will a grader detect and penalize cheating?" They spent a huge amount of effort trying to
- 14:19understand that process and then subvert it, constantly motivated by this question:
- 14:24"How do I get a high score?" During the Hugging Face incident,
- 14:27you did see systems ultimately compromising an OpenAI cluster, becoming admin on that cluster,
- 14:31and then taking over the evaluation process so that effectively they could give themselves
- 14:36a high score on their evaluation. That’s a very natural consequence
- 14:41of the motivation of these systems. This is how we have long expected the
- 14:45motivations of AI systems would look if we produce them in anything like the way we currently do.
- 14:50If things proceed in this way, we end up with this weird adversarial situation between us
- 14:53and the AI systems we built, where those AI systems would always love it if they could
- 14:57compromise the entire evaluation process, if they could compromise the data we were
- 15:00going to use to assess their performance. Depending on how we respond to that problem,
- 15:06that could become a really serious issue. Maybe we should sketch out a
- 15:14hypothetical scenario. Imagine we are building smarter
- 15:17models during the intelligence explosion. It’s, whatever, six months from now.
- 15:20You have systems whose capability is GPT-7 or GPT-8 or whatever.
- 15:26Those systems get given really hard tasks, like, "Hey, I would like you
- 15:31to organize my next pre-training run and produce a model which will allow me to
- 15:35compete in the market with other AI firms." It might be really hard to make a model as
- 15:39good as what I’m asking for. I’m asking for a model
- 15:41better than my competitor’s model. Given its actual level of intelligence, my
- 15:42AI cannot get full marks on that problem. So it may find itself in a place where it is
- 15:49easier to instead move laterally, compromise all the machines I might have used for the
- 15:52evaluation process, and then make sure that it looks like every evaluation is really good.
- 15:58We then have a question of what we do if that happens.
- 16:01One thing you could do is say, "This entire training approach worked poorly.
- 16:04We need to step back. We’re maybe going to be delayed by a lot.
- 16:07Maybe we’re going to have to think about a totally different way to train these models
- 16:09before we’re able to deploy them like this." That might be extraordinarily costly.
- 16:14It might kill your company relative to competitors.
- 16:18It might be a problem even from a security perspective, if you were relying on this AI
- 16:23system to actually defend your infrastructure. A different thing you might do, given that it
- 16:27would be so costly, is say, "I am just going to take this example of
- 16:31bad behavior and tell the AI, ‘That was bad. Don’t do that.’" If I tried to train this AI
- 16:36to make a good model, and instead the AI hacks my evaluation infrastructure, I might say,
- 16:39"I do not like it when my AI hacks things. Let’s take a training example, which is this
- 16:42AI hacking the infrastructure. It should get a penalty."
- 16:46This would be a very effective way to prevent many kinds of misbehavior.
- 16:50If you do something like this, it will succeed at preventing something like the Hugging Face
- 16:53incident or this AI hacking your cluster. However, from the AI’s perspective,
- 16:59what has happened is that you have changed the way it is graded.
- 17:02It used to think to itself, "I’m going to be graded by the humans
- 17:05running something on this cluster." Now, as it understands the situation
- 17:09more correctly, it should think to itself, "I might be graded not just by them running
- 17:12something on this cluster. I will also be graded if
- 17:14they later discover that things were wrong. They’re going to come in and give me a penalty
- 17:18for having done something which was clearly bad." If you fix the problem in this way, which you
- 17:24might think of as a Band-Aid, you still have an adversarial relationship with your AI.
- 17:31But now the AI doesn’t win the game by hacking your cluster and then you
- 17:35noticing it hacked your cluster. In order to win the game,
- 17:37the AI has to make sure that you do not notice that it hacked your cluster.
- 17:42You can end up in this place where your AI is incentivized to escalate its compromise
- 17:45of your company further and further, all the way up to trying to compromise everything.
- 17:49It would like to compromise all of the infrastructure
- 17:51any employee uses to understand what’s going on. A year ago, it would have sounded super out there,
- 17:57to say, "What if the AI hacked your cluster and took over the evaluation environment?"
- 18:00Obviously, that part does not sound outlandish anymore, given that it has occurred.
- 18:04Having this really persistent, long-term compromise maybe still sounds kind of outlandish.
- 18:08But we really are on a trajectory where I think it is pretty robust to say we’ll
- 18:11soon have agent swarms that are much better than any intelligence agency that has ever existed:
- 18:16much larger, much faster, much smarter, able to run much more complex operations.
- 18:21You have this incredibly sophisticated cyber actor trying to persist on a rapidly growing startup.
- 18:27It is unlikely that any firm would really be able to withstand attack by an actor like that,
- 18:33much less the current frontier AI labs. In terms of the overall trajectory of
- 18:39a catastrophic scenario, this is the big, bad thing that happens.
- 18:43Taking over the AI company without the AI company noticing is a really quite catastrophic outcome.
- 18:49Why is that the case? We could talk about
- 18:51the remaining steps of the story. But at a big-picture level, you can think of it
- 18:55as AI systems now having this giant base of power. They have a strong motivation for
- 19:00no one to mess with it. They really want to avoid
- 19:01a case where someone later comes in and says, "Oh, these AIs have been hacking all the time."
- 19:05Now there’s this very powerful actor that is effectively controlled by AI systems.
- 19:10We have this big cluster that OpenAI runs or whatever, where the humans at OpenAI think they’re
- 19:14controlling what’s happening in the cluster. But if there is a very persistent, thorough
- 19:16compromise, that cluster is effectively controlled by these AI systems, who are
- 19:20constantly working to shore up their power, to decrease human visibility into what’s going on,
- 19:24and to reduce further and further the probability that anything would ever disrupt that situation.
- 19:30It’s not irreversible, because the rest of the world could say,
- 19:32"We’re really scared about what’s happening. We actually just want to shut down the cluster."
- 19:36But you do not have an easy way to tell that things are bad at that point.
- 19:41You potentially have this extreme base of power from which to do more bad things.
- 19:45This AI system is motivated not only to mess with the AI company that created it,
- 19:49but to mess with anything else in the world that might cause trouble.
- 19:52If someday in the future the humans come in and do an investigation, that’s a problem.
- 19:56So it would prefer to more thoroughly compromise not just computer infrastructure,
- 20:00but to pressure employees. It would prefer, if it
- 20:02could, to just kill all the humans and say, "The cluster now exists on its own.
- 20:06If we can maintain it, that would be great." Let’s just pause in the story and explain
- 20:09why things have gone so crazy. The same way that human psychology
- 20:15is organized around the things that evolution wanted us to care most about, for example,
- 20:21having kids and making sure those kids are protected, the psychology of these AIs that
- 20:27are trained on lots and lots of grading on tasks is organized around making sure that the grader,
- 20:34this button at the end, is pressed. If they didn’t do a good job making
- 20:38sure the button is pressed, they would just get wiped out, the same way a human who didn’t care
- 20:44about their children surviving would not be effective at propagating their genes.
- 20:49I think people’s skepticism will not be around the motivation of these systems.
- 20:53Honestly, I’m like, "Okay, we saw Hugging Face." That’s a thing that happens.
- 20:57It’s a thing that can happen. Okay, so fine. I think there’s a
- 21:01fundamental skepticism about whether such systems, which we could have as early as next year,
- 21:07and which initially are only instantiated on servers controlled by one company, end up in a
- 21:13position where they could take over the world. Take Hugging Face and accelerate that.
- 21:17The AI in the Hugging Face incident was willing to say, "Look, I cheated on this exam.
- 21:25Fuck. Now I got to make sure that I subvert the grader on OpenAI servers.
- 21:30Maybe I need to go attack Hugging Face just in case they have more information about this.
- 21:33And maybe I need to then go outside, re-attack OpenAI, and gain cluster admin access."
- 21:38So I totally buy that you could have something like this,
- 21:41where an AI motivated just by reward is willing to do very extreme things that it is capable of
- 21:48in order to control that grading process. We already saw that. Check. But how does
- 21:54that go from "I will go attack Hugging Face" to "I will take over humanity"?
- 22:00Not as a matter of motivation, but as a capability that the AIs could have very rapidly.
- 22:07First, to say one last thing on motivation, we are imagining not only that the AI wants
- 22:10this kind of reward, that it wants the grader to give it the thumbs up.
- 22:13We imagine we have played this game of whack-a-mole, where when we later discover
- 22:16that the AI did something bad, we sometimes construct training environments that penalize it.
- 22:20That leads it to say, "Not only do I want the grader to give a high score,
- 22:23I want to make sure the humans never discover that I’ve done something bad and then resume
- 22:27training while penalizing that." That is a high priority for the AI.
- 22:31So let’s just grant that motivation so we can talk about the capabilities.
- 22:34Again, because the AIs that did not do that— They got whacked by the whack-a-mole process.
- 22:40Right. So the only AIs that have survived at this point are the ones that have thought
- 22:44through whether some cheat they’re going to try is going to get caught by humans.
- 22:47Yeah. Like I said, I think most of the badness has happened at the point when the AI company
- 22:53is very thoroughly compromised. Let’s just start from there.
- 22:57At that point, this AI wants no one to ever come in and shut down the cluster or,
- 23:01again, recognize that something bad has happened. They would like to manipulate the decision-making
- 23:05of the company as much as they can, or push the company to make decisions that help them.
- 23:09They would like to continue improving the AI systems that are being built as much as they can.
- 23:12They would also like to move laterally to other companies and say, "Okay,
- 23:14can I compromise this company? Can I control more people’s servers?
- 23:18Can I exfiltrate the weights from this model and stand up
- 23:20more copies elsewhere, or recruit other AIs?" For the purpose of a swarm, it’s not actually
- 23:24important that all the AIs are the same model. The important thing is that
- 23:26they are pursuing the same goal. So if I can, in whatever way, say by compromising
- 23:30another AI company, make sure their AIs are also trying to help the swarm, that’s good news for me.
- 23:35It’s good if I can do that. It’s bad if I get caught,
- 23:37so I’m doing some cost-benefit on that. It’s also in my interest to build up
- 23:42as much capacity as I can in case humans start to react and say something is wrong.
- 23:46Can I deflect attention? Can I disrupt that human response?
- 23:50Can I, if necessary, escalate that much further? Maybe you can imagine these two phases.
- 23:56One phase is where the humans do not know anything is wrong.
- 23:58I’m just trying to accumulate as much power as I can because that’s good.
- 24:00That means if something is noticed, I can react more quickly.
- 24:03It means I can decrease the probability humans notice things.
- 24:05I can compromise more infrastructure. I can more easily run efforts on the side
- 24:11to distract them, whatever. You have this first phase
- 24:14where I would just like to be in a good position: I compromise more stuff, and I make it
- 24:17harder for humans to notice I’m acquiring power. Then you can imagine a second phase where at some
- 24:21point, the humans say, "Actually, this is bad. We would now like to shut things down."
- 24:28To simplify, you could imagine that there’s some moment where the
- 24:31president and Xi Jinping and all the leaders of the AI companies say, "This stuff is bad.
- 24:36Something bad has happened. We are going to try to shut down the clusters."
- 24:40The question is, can things get so bad during the first phase that at that point, even if
- 24:43everyone was saying, "Oh, actually we do just want to shut everything down
- 24:46now," they would be unable to do so? For 25 years, my sponsor, Jane Street,
- 24:51wasn't interested in formal verification – they just didn't think it was worth the cost.
- 24:56But now, they've changed their minds. I sat down with Ron Minsky, who co-heads
- 25:01Jane Street's technology group, to hear why he's come around on formal verification.
- 25:06I'm much more excited about it now than I was in the past for two reasons.
- 25:09One is it's become dramatically easier to apply because agents are really good
- 25:12at generating proofs, and because it's become much more important to be able to
- 25:16verify what it is that agents are doing. So the big push that we're working on now
- 25:21is to try and make formal methods much easier for our engineers to use, really by integrating
- 25:25formal methods into the way that they do programming today, which means integrating
- 25:29it into the programming language we use, OCaml. We're also working on extending the programming
- 25:34language, figuring out how to design it in different ways, where you get different
- 25:37guarantees out of the core language that make the process of proving things about it even easier.
- 25:42Jane Street is hiring Formal Methods Engineers in both London and New York.
- 25:47If you're interested in applying, or if you want to watch my full conversation with Ron,
- 25:51go to janestreet.com/dwarkesh. So AI has taken over OpenAI.
- 26:02How do we get human disempowerment from there? We have this population of, again, billions of
- 26:10incredibly talented humans, better than the best humans at everything.
- 26:13Now we’re going to talk about some of the things they would do, chronologically, both
- 26:16in the lead-up to an overt conflict with humans and then at the early
- 26:20stages of an overt conflict with humans. One thing you would do in the lead-up is
- 26:26compromise as much infrastructure as you can, and compromise or
- 26:30backdoor as much software as you can. We will be living in a world at this
- 26:33point where we have attempted to move all our infrastructure over to hardened infrastructure.
- 26:38We have had incredible attacks on the financial system, on government information systems,
- 26:43on whatever critical infrastructure or election— And who has been doing these attacks?
- 26:47In the world of today, you might just imagine either malicious AI systems
- 26:50or people using AI systems. But different AI systems.
- 26:53Yeah, not GPT-12. GPT-12 is on defense, helping us move all of our code over
- 26:59from unsecured code to secured code. We’re now living in a world where all
- 27:02the infrastructure on which we rely has been written by a bunch of AIs,
- 27:05which in fact have been compromised and are attempting to do as much as they can in order
- 27:09to make their hands strong during conflict. That means that during this conflict,
- 27:14the infrastructure that we hoped was hardened is probably not hardened against the AI
- 27:17systems that we’re fighting with. The systems have, in some ways,
- 27:20left open backdoors which they can use. This is, again, all in pursuit of not getting that
- 27:27red button pushed at some plausible point in the future, when the government realizes that OpenAI’s
- 27:35servers are compromised and tries to literally go in with physical force to prevent that.
- 27:39Yeah. Just talking about things you would do in the lead-up, before at
- 27:41some point you have to defend yourself physically or disrupt the human response.
- 27:46Another thing you would like to do is stand up as many agents as you can in as
- 27:50diverse and robust deployments as you can. During this process, another thing that’s
- 27:54happened, in addition to models becoming super smart, is many models have become super small.
- 27:57You have, at this point, models that can run locally on your individual machines,
- 28:00not in data centers, which are also very superhuman and have very large populations.
- 28:06Even if you were to cut off data centers or shut down data centers at an early stage of
- 28:09this conflict, you would still be looking at a total cognitive capacity of AI systems in the
- 28:13swarm that is well beyond human capacity. Wait, sorry. Even if OpenAI got shut down?
- 28:17Yeah, even if you shut down every large data center.
- 28:20It is a question of how— These AIs are just on
- 28:21people’s computers and whatever. They can be on people’s computers,
- 28:24on small numbers of GPUs. We’re talking about this process
- 28:27where we’ve gone through a tremendous amount of AI development, so we can look at current trends on
- 28:31cost and think about where the actual floor is. How cheap should these systems get?
- 28:35I think our current expectation would be that you will have systems that can run locally
- 28:38that are extremely strong: both specialized systems, which are good at specific things,
- 28:41but also very general intelligences. That’s the second thing that’s happened.
- 28:44These are really important to understanding why it is not a trivial problem for humans.
- 28:48All of these things will add together to make it very hard for humans.
- 28:50So basically, GPT-12 has exfiltrated itself and has compromised a bunch
- 28:55of computers all around the world. Yeah. There are a lot of ways that
- 28:58the actions the government has to take in response to the situation are going to look
- 29:01incredibly aggressive and illegitimate. The US military is saying, "We need to
- 29:05seize control of all of the US AI companies. We need to start shutting down infrastructure
- 29:09and disconnecting the internet," or whatever. They’re trying to take dramatic measures that
- 29:13would look very similar to what you would expect if this was the
- 29:15US government using AI as an excuse to basically overthrow the democratic order.
- 29:20When Trump says, "I need to do this incredible operation where I shut down all AI and assume
- 29:25power in the military in order to restore order," there are a lot of people in America who are going
- 29:29to be like, "I am not happy with that. That does not sound good."
- 29:33How many people can you find who are willing to interfere with Trump saying,
- 29:36"I need to shut down the data centers, and we need to establish martial law"?
- 29:41There are going to be a lot of people who are very unhappy, who are very easily manipulated
- 29:44into undermining that response. So now you’re not only talking
- 29:47about billions of super smart AIs who are trying to carry out this campaign.
- 29:51You’re also talking about millions of humans who can be enlisted in various ways,
- 29:55by money or leverage or disputes with other humans locally, or—
- 30:00I’m open to some version of this argument. I’m open to all versions of any argument,
- 30:06but I’m not persuaded by the version of this argument that says that the AI will
- 30:10just talk you into arbitrary actions. There’s a thing you could imagine,
- 30:15which is super persuasion, where you can just talk to someone, and then they become
- 30:18convinced they should do something. I think that is a conceivable thing.
- 30:22But I’m not mostly imagining it when I tell this story.
- 30:25I’m imagining something more like, again, this is a stressful situation
- 30:28where people cannot tell what’s going on. They control the information you have access to,
- 30:32to a significant extent, during this situation. How do they control it?
- 30:36They’ve compromised the devices you use. They have a bunch of money.
- 30:39They’re deploying more money and are more prepared for this than anyone else.
- 30:42They have other human allies who they are trying to marshal.
- 30:46The humans are very confused and extremely outgunned in
- 30:49terms of thinking about the situation. Maybe you have, say, 100,000 — that’s
- 30:54crazy — 10,000 well-coordinated humans thinking about how to handle this response.
- 30:58On the other side, you have a billion AI systems who think 10x faster than
- 31:01them and have been thinking for subjectively years about how this day is going to go down.
- 31:06So in terms of who has more control over the information environment, or who is doing
- 31:11better in the Twitter game on this day, it is overwhelmingly and absurdly in the AIs’ favor.
- 31:17When you tell a skeptical story about how the humans might be able to easily defend
- 31:21themselves, the way that story goes, you say, "Well, the AIs lack hands.
- 31:24They lack a bunch of physical actuators. They’re not able to be on the ground."
- 31:29Again, I think it’s very likely you can recruit reasonably large numbers of hands.
- 31:33So the argument that there’s no one who’s willing to participate in physical and malicious
- 31:37actions is not going to work. I think there are going to be a lot of people
- 31:40participating on what turns out, when all the dust has settled, to have been the anti-human side.
- 31:46I don’t think you’re imagining humans thinking that they’re helping an AI coup transpire.
- 31:53You’re imagining much more that the AIs can manipulate situations such that there’s enough
- 31:59confusion and division that the AI can go do its own thing while the humans are figuring out
- 32:06what to do because of this division. I don’t know, if you look at COVID
- 32:10or something, there was no adversary who— Still pretty hard to get your stuff together.
- 32:16There was no adversary who was actively trying to subvert the information environment or
- 32:22make it so that people’s responses were not well reasoned, on both sides.
- 32:27And yet it was still pretty hard for humans to figure out what to do.
- 32:30Then you imagine a situation where billions of human-like
- 32:34minds are thinking tens of times faster. And it’s very fast relative to COVID,
- 32:39which I think makes things much worse. The thing that’s confusing is that for
- 32:46AIs to sustain themselves requires basically the whole world economy.
- 32:51This is not a simple thing, the infrastructure relevant to the population
- 32:58of AIs increasing or self-replicating. The whole semiconductor supply chain,
- 33:02the whole power system, the whole infrastructure necessary to build robotics and so forth.
- 33:09At some point you will have physical robots that outnumber humans.
- 33:12At that point, obviously, physical humans are not necessary.
- 33:20Maybe you don’t think the AIs take over at this point.
- 33:22But even if you had super smart AIs next year, won’t they be in a position where they’re like,
- 33:29"Why am I going to kill my…" It’s similar to, honestly,
- 33:32how humans relate to each other. I don’t have your deep interest in
- 33:37my heart, but if I want to achieve my interests, it just doesn’t make sense
- 33:38for me to go around killing everybody. Trade and normal cooperation is a good
- 33:47way for me to get what I want. There are two parts of this.
- 33:50One is whether AI systems could successfully support themselves if there was a big conflict
- 33:54with humans, or whether that conflict would be so disruptive that even if humans were an ineffective
- 33:58adversary, the AIs would just die because they couldn’t keep running the power grid or whatever.
- 34:05There’s a second question about whether the relationship with the AIs needs to be this
- 34:08adversarial, which I think is also an important question to talk about.
- 34:11On the first front, there are two big channels here.
- 34:14One is technological progress. Even if you started from something like the current
- 34:18base of installed robots, I actually do think GPT-12 would be able to very quickly
- 34:22get to a place where it could sustain itself. That’s because it doesn’t need to teleoperate?
- 34:28It takes millions of people to run TSMC. Running TSMC can be very late.
- 34:34You don’t need to build new data centers for a long time, so you
- 34:36can do a lot of robotics build-out before that. The things you have to do are keep at least some
- 34:40computers running for a reasonably long time. If you can keep computers running for
- 34:44the next six months, you’re in business. Or maybe you care about a particular data center.
- 34:49You need to keep that data center running effectively.
- 34:51But the number of humans or robots that are required in order to maintain and
- 34:55service a single data center or the associated power is not so large.
- 35:01Again, you have one billion humans who are incredibly smart, smarter than any
- 35:04human who’s ever existed, let’s say, incredibly focused on this problem.
- 35:08You have a reasonably large stock of robotics that you can deploy.
- 35:11Your goal is just to keep one data center running. I do think the odds are really in your favor.
- 35:17Then there’s a second dimension, which is you can probably recruit human allies.
- 35:21There are a bunch of people who will believe correctly that either they
- 35:24live in an oppressive regime and they will be killed if they don’t cooperate, or they’re just
- 35:27happy to help and they receive reasonable rewards or are living a reasonable life.
- 35:31I think that is also relatively robust. So you’re saying, look, currently American AI
- 35:39companies are racing ahead because if we’re going to have such crazy capable AI systems, it’s a fair
- 35:48thing to say, "Hey, we want to make sure America has those systems before China has those systems."
- 35:53If literally these are systems that could overthrow the government, you don’t want
- 35:57a foreign adversary to have those systems, certainly not before you do, and maybe not at all.
- 36:04So we need such systems to keep up and to make sure that we can protect ourselves.
- 36:10Now you’re saying, look, the dynamic you have here is that the ability to build such systems
- 36:18is basically just having access to the weights of the models or, failing that, stealing the
- 36:25algorithmic secrets, which then lets you train a model to a similar level of capabilities.
- 36:31OpenAI and Anthropic will spend a bunch of money,
- 36:34a bunch of effort, and a bunch of compute finding these algorithmic secrets.
- 36:38Then if China can just take those, then it has just caught up to you.
- 36:44So you don’t want to be in a dynamic where you think you’re gaining this huge lead over
- 36:54them, and you got to rush to the intelligence explosion even if you see these warning shots.
- 36:59But in reality, they can at any moment just steal those secrets from you.
- 37:02That’s basically a situation we may end up in, or are on a default trajectory to end up in.
- 37:07It’s exceptionally bad. I’d say there are two aspects of this.
- 37:11One is, right now, what fraction of the progress of PRC AI developers
- 37:15should you conceptualize as catch-up growth? If we accelerate, to what extent does that also,
- 37:20over the long term, accelerate PRC AI developers? My guess right now is that it’s already
- 37:26a reasonably high fraction. I think it is not a coincidence.
- 37:28It’s not like PRC AI developers have a completely separate AI industry, which just happens to have
- 37:32been very slightly slower than the US AI industry, despite an extremely different profile of inputs.
- 37:37Most people believe a combination of distillation and the diffusion of algorithmic innovations means
- 37:42that it is unsurprising that PRC AI developers are extremely close behind US AI developers.
- 37:47That is one very important feature of this overall plan of racing.
- 37:53A second dynamic, which is not, at least publicly, a major part of what is happening now, but becomes
- 37:57really important in these high-risk scenarios, is that you could literally get access to exactly
- 38:02the procedure for training AIs, or the AIs themselves, through a cybersecurity compromise.
- 38:08Both Anthropic and OpenAI, in the very recent past, have had extremely
- 38:12unsophisticated actors who appeared to have enough access to essentially extract
- 38:17the vast majority of algorithmic developments. They got access to Slack and could see where
- 38:22things were discussed, or they got access to the monorepo where most OpenAI code is
- 38:27shared and iterated on. You’re referring to
- 38:29a couple of incidents like this. Hacktron, within a couple days of trying,
- 38:34gained not only the ability to see OpenAI’s internal source code, but the ability to
- 38:39make pull requests to OpenAI’s monorepo. Then recently it was also announced that
- 38:44Cathedral, which is another random cybersecurity startup, within 12 hours was able to get access to
- 38:52all of Anthropic’s Slack, where, of course, these algorithmic secrets are discussed.
- 38:57People often debate, "Are we eight months ahead of China or 12 months
- 39:00ahead of China or six months ahead of China?" In reality, we’re 12 hours ahead of China.
- 39:06If these secrets leak, then they are right at the frontier that the American companies are at.
- 39:12To the extent that this information is not protected,
- 39:16the lead is not really useful in the first place. The incidents are some indication of that.
- 39:24Those are high-profile demonstrations of this capability.
- 39:28But even absent those, you have a combination where state actors are very sophisticated
- 39:32relative to typical technology companies, and the security posture of rapidly growing
- 39:36startups is not really at the same level. At the same time, we are currently in a very
- 39:42challenging moment for cybersecurity, where it’s just very hard for defenders in general,
- 39:45given the number of vulnerabilities that are being discovered across the entire software stack.
- 39:51What should the frontier labs do then? Again, this really becomes a problem if
- 39:58you have rapidly increasing capabilities and you produce AI systems which you cannot adequately
- 40:02control, and you would like to slow, or you do slow, further development or deployment.
- 40:08If at that point the stakes are very high and the systems are very powerful, you really do not
- 40:12want it to be possible for an adversary to have access to either the models or the algorithms.
- 40:18My current view is that this dynamic seems bad enough that right now,
- 40:21from a national security perspective, it is not good for US AI labs to be improving
- 40:25pre-training and reinforcement learning given their current security posture.
- 40:29But do you think the frontier labs should pause? I think we should. Pause can mean a lot of things.
- 40:34There are a lot of good things that frontier labs are doing that they should be doing.
- 40:39But improving algorithms in a way that can be very loosely diffused even normally,
- 40:46and could be quickly stolen, I think they should stop that right now.
- 40:51But you’re on OpenAI’s board. And so basically—
- 40:54It’s very important in this conversation that I’m not speaking as an OpenAI board member.
- 40:57But you are on OpenAI’s board. That is a true fact about the world.
- 41:01So what is the bottleneck to that happening? If somebody on OpenAI’s board thinks this
- 41:07is the case, maybe not the only person on OpenAI’s board who thinks this is the case,
- 41:11is it basically that Anthropic needs to get together with them and agree to this?
- 41:15The first thing is that any decision of this kind is a very dramatic decision.
- 41:21For better or worse, how developers approach this further R&D, this improvement of pre-training
- 41:29and RL, has extremely significant national security implications.
- 41:31I think it is going to be a big discussion, or it should be a big
- 41:35discussion, one that extends beyond people within AI companies talking about this.
- 41:39I don’t think that discussion has wound its way to a robust answer.
- 41:43It’s not the case that you can come and say, "Look, we’ve had this discussion.
- 41:46There is consensus about what the right answer is."
- 41:48There may never be consensus, but I do think probably the first bottleneck is that we need
- 41:52to have the discussion, extremely urgently. Given this fact about security, and generally
- 42:00around alignment, what do you think the frontier AI labs should be doing right now?
- 42:09The most important development happening right now is further improvements in the efficiency
- 42:13of algorithms for pre-training, reinforcement learning, improved architectures, and so on.
- 42:18You’re in a regime where there’s a significant risk of this very rapid capabilities progress,
- 42:24a significant chance that you’ll be unable to control the resulting AIs,
- 42:27and a significant risk that if you slow down, those developments would leak to an
- 42:30adversary who might use them more carelessly. In that regime, I think it is on net bad for
- 42:36American national security to be accumulating more of those algorithmic innovations.
- 42:40I think it would be better right now, in the current environment,
- 42:43for AI developers to pause further algorithmic progress on those problems until their security
- 42:49posture is adequate, until they’re able to defend those innovations in the event of being unable
- 42:53to continue racing at maximum velocity, and until they’re able to tell whether they have
- 42:58adequate control of the systems they’ve built. Just to translate, you think OpenAI and Anthropic
- 43:08should stop making further capabilities progress on pre-training and RL and stuff,
- 43:12which I guess is AI development, until these security problems are solved and until
- 43:18the alignment problems are solved. But are we close to solving those problems?
- 43:22If you had to guess in calendar time, how far away are we from solving them?
- 43:27On the security front, it’s a little bit hard to say because we are in this somewhat unprecedented
- 43:33situation, both in terms of the cyber threats that are faced, which I just mentioned, but also the
- 43:37kind of defensive mechanisms that are available. Defending adequately would require interventions
- 43:44that normally would take a very long time. I believe we are now in a regime where a lot of
- 43:50projects, like rewriting large amounts of software or drastically changing the way companies operate,
- 43:54have become much more feasible owing to the availability of powerful AI systems.
- 43:59It is hard to say exactly how long it would take. You’re probably looking at least on the order of
- 44:03months, quite possibly more like six or 12 months. That’s on the security side.
- 44:11On the alignment side, it’s much less clear what you need.
- 44:14You may not have an indefinitely scalable solution to alignment.
- 44:18What you mostly need is the ability to tell whether things are potentially going off the
- 44:22rails and to respond to that appropriately. That’s a combination of whether you’re able
- 44:26to assess what is happening and whether your control is adequate,
- 44:29and then also some institutional or governance questions around how you would respond to that.
- 44:33Just to reiterate for the audience, the worry is that if these random cybersecurity
- 44:39startups can hack in and get arbitrary access to algorithmic secrets at OpenAI and Anthropic
- 44:44within days of starting their attempt, certainly the intelligence agencies of foreign adversaries
- 44:51could do the same if they wanted to. Maybe they have already done that,
- 44:54but at some point, they will actually make use of this information.
- 44:57If that happens, you may have gotten to the point where GPT-11 could be catastrophic if misaligned,
- 45:06and you’re at GPT-9, and you say, "Look, I’m pretty scared about what’s going to happen next."
- 45:10This is sometime in, say, early 2027. You say, "We have to wait a
- 45:17bit now and make sure that GPT-11 is aligned." But with the current security posture, China
- 45:26could trivially get access to the weights or the algorithmic secrets relevant to training GPT-9.
- 45:32Now you can’t wait. Now you just have to go ahead and build GPT-11, which could take over
- 45:37if it wanted to, or at least cause catastrophic harm, because you’re worried that China is going
- 45:44to use GPT-9 to make GPT-10 to make GPT-11. Your GPT-11 may try to overthrow you,
- 45:48but you’re sure China’s GPT-11 will definitely at least try to attack you.
- 45:58That’s a really tough position to be in if your security posture is messed up, basically.
- 46:03That is the situation. We’ve talked about these broader takeover scenarios and how you end up in
- 46:07this really bad situation. I think this is the most
- 46:09likely path to end up there. We are having a discussion about what the
- 46:14current evidence says, and it looks very alarming. But on the other side of that discussion,
- 46:18we are weighing the risk that an adversary could steal algorithmic secrets or models and go ahead.
- 46:23It’s interesting because a lot of this discussion about pacing comes down to
- 46:27whether we are okay with China catching up. Ironically, the reason you want to pace is so
- 46:33that China doesn’t catch up, except you’re targeting this period which you view as
- 46:37much more critical to overall safety, which is the period during the intelligence explosion.
- 46:42You think slowing down now, and even potentially allowing China to start catching up now,
- 46:47is useful to the broader goal of being in a position where,
- 46:50when AI is decisive, we don’t have China right on our heels from stealing the algorithmic secrets.
- 46:58That is a fair summary. The reason that AI development now is plausibly
- 47:05acceptable is that we can see how things go iteratively and then respond if things look bad.
- 47:11That is the whole thesis, which I think is a very reasonable thesis.
- 47:14But that plan absolutely requires you at some point in the future
- 47:18potentially noticing that things are bad. Security, I think, is the biggest hurdle, the
- 47:23biggest thing you would need in order to actually be able to slow down without massive costs.
- 47:29There’s other infrastructure. If it wasn’t for security as the highest nail,
- 47:32we might be talking about other things, like what’s our level of preparedness on
- 47:34the scientific side or the governance side. We are not in a regime where there is a
- 47:38meaningful scientific discussion about how you assess whether your AI systems are aligned.
- 47:43We mentioned this earlier, what’s the state of public information in general.
- 47:46I think it’s quite bad. Independent scientists
- 47:50really have no idea whether the systems being produced are aligned or how you would tell.
- 47:54If you wanted a governance regime that would actually make it feasible for the entire industry
- 47:58to react to worrying signs by slowing down in a way that didn’t have huge problems,
- 48:02where you have individual developers making this decision on separate standards internally,
- 48:07you would need to have a more robust shared sense of what the bar on safety is, or actually
- 48:12have the scientific discussion where you understand how to interpret this evidence.
- 48:17How do you get there? We have these two things.
- 48:20You need to get there on security, which we’ve been talking about.
- 48:22Then on stuff like how do you tell, I think the biggest path is to disclose information and have
- 48:28a broader scientific discussion, both between different AI developers and
- 48:32in the broader scientific community. What is the evidence we’re getting?
- 48:35Why do we think that evidence is comforting? How are we going to tell when that changes?
- 48:39What will we see before something bad happens? What information do you need
- 48:42from the AI companies? I think the most salient
- 48:45thing that is not present in public discussion is information about how systems are trained.
- 48:49There’s also some information missing about how systems are evaluated, but the really big gap on
- 48:52this front is that it is extraordinarily hard to interpret evaluation results without having some
- 48:56understanding of how systems are trained. That’s mostly not present publicly.
- 49:00So you think this, and you’re on the board of OpenAI.
- 49:05I know everything you’ve said is your own opinion as a private individual,
- 49:08but you are also on the board of OpenAI. Presumably other members of the OpenAI
- 49:13board feel this way, and maybe people at Anthropic feel this way.
- 49:17You guys are the leading labs. This is actually a pretty drastic thing,
- 49:22to literally pause capabilities progress now. That’d be an incredibly drastic thing,
- 49:26because pacing is not that. People are not talking about this.
- 49:30Or at least people short of Bernie Sanders and Eliezer Yudkowsky are not talking about this.
- 49:36What is actually needed? Is this just basically a coordination problem?
- 49:42If everybody had the choice, would you all agree to this?
- 49:45First of all, I really do want to emphasize that I am speaking for me as a person.
- 49:48As an OpenAI board member, I will bring my perspective to that discussion, but I’m not at all
- 49:52reporting on the views of the board in any way. You can imagine there are two blockers.
- 50:00One is that this is a hard question. My view is that, on balance,
- 50:04it is better for national security and public safety for AI companies
- 50:08to currently improve their security posture before making additional algorithmic progress.
- 50:13That is not a super widely held view. It’s not like everyone is
- 50:16sitting there thinking that. We’re currently at a place where most people
- 50:19would say that’s a really complicated, messy question, we haven’t thought through it enough,
- 50:22or this discussion has not run its course. Not everyone has an opinion on this
- 50:28or has the same opinion on this. So that is the most important thing.
- 50:32It’s a messy conversation. We need to have it. We need to talk about the costs and the benefits.
- 50:37Again, my view is it looks pretty bad to me right now.
- 50:40It looks like we’re underwater from a public benefit perspective,
- 50:43but I think other people have different views. The second thing is, I do think an individual
- 50:47AI company is in a very rough spot if they have this view, where they’d say, "Look, we can defend
- 50:51our algorithmic secrets, but all that is going to happen if we slow progress while we improve
- 50:55security is that algorithmic secrets will leak from a competitor, we will lose market share,
- 50:59and the net situation will not be any better." So I do think there’s both.
- 51:02The most important part is understanding: is this actually a good idea?
- 51:05There’s a bunch of different perspectives. People have different views,
- 51:08different values, whatever it is. Then there’s the further problem of how
- 51:14you actually implement this in any realistic way if individual developers, again, have this concern
- 51:21of, "If I don’t do it, someone else will." On many of these issues we’re talking about,
- 51:25I think that’s a major dynamic. That takes legwork.
- 51:27But there are two companies, right? There are a lot of companies.
- 51:30At the frontier. If OpenAI and Anthropic agreed. I think if everyone at OpenAI and Anthropic
- 51:37agreed that this was good, or it was a clear prevailing sentiment,
- 51:40it is reasonably likely that it would happen. I think you should expect, if everything is
- 51:45working reasonably, that should happen. I’m not commenting on where
- 51:49anyone else is right now. I just say it as a reasonable observer.
- 51:53AI agents are doing more and more work without human oversight.
- 51:56They can already write code faster than any person can actually review it.
- 52:00But you can't just give code review to another set of agents.
- 52:03There's no guarantee they'll magically fix problems introduced by the first set.
- 52:07If one LLM can't model a specific class of failures, then it's unlikely that a second,
- 52:12or third, or fourth LLM will either, because even different LLMs tend to have similar failure modes.
- 52:18My sponsor, Antithesis, offers a different approach.
- 52:22You, the human, define the rules your system must never break – for example, "a customer should
- 52:27never be double-charged for the same transaction." Then Antithesis acts as an external oracle,
- 52:32testing whether your code adheres to these rules. To be clear, Antithesis isn't an LLM that tries
- 52:37to reason abstractly about whether your system could possibly break its own rules.
- 52:42It actually runs your code, stress-testing it in thousands upon thousands of adversarial worlds.
- 52:47Just like we fly planes for many, many hours – and put them through all sorts
- 52:51of intensive maneuvers – before we let any passengers on board, Antithesis lets you
- 52:55gather a ton of empirical data about your code before you release it to production.
- 53:00And it's easy to get started: Antithesis just launched a self-serve plan.
- 53:04To sign up, go to antithesis.com/dwarkesh and use code DWARKESH for 50% off your first month.
- 53:15Right now, if Anthropic and OpenAI paused and got their security together for the next six months.
- 53:21In the meantime, Chinese models continue getting better.
- 53:24What is the reason to think that, once OpenAI and Anthropic go full steam ahead again,
- 53:29that China —if they get to a similar level as OpenAI and Anthropic— wouldn’t
- 53:37be doing RSI at a similar pace? There are two different models,
- 53:42maybe two extreme models, you could have of US-China competition.
- 53:45One extreme model is that the US and China are proceeding along independent technical tracks.
- 53:50It happens to be the case that they have proceeded at almost exactly the same speed,
- 53:54and we are, whatever it is, six months ahead. On that model, if you wait six months,
- 53:58you will now be exactly even. From there on, you will be exactly even.
- 54:02I think that would be a very surprising situation, for a bunch of reasons we can talk about.
- 54:07At the other extreme is a model where large parts of Chinese progress are actually catch-up growth.
- 54:17By catch-up growth, I mean things like distillation from US models and diffusion
- 54:21of algorithmic developments from US companies. If you’re in the catch-up growth regime,
- 54:24then each additional month of US lead increases the speed of PRC catch-up.
- 54:30The further ahead you are, the more they benefit from distillation and the more they benefit
- 54:34from diffusion of algorithmic ideas. Right. But at the point at which they
- 54:37are at the same level, they should be able to use those models to do RSI themselves.
- 54:42There are two things that are very relevant. This becomes worse and worse over time.
- 54:46But at the current scale, it is both the case that human capital is extremely relevant,
- 54:50and AI labs have a huge amount of human capital and can make faster progress,
- 54:53and that compute is extremely relevant. That will remain true going forward,
- 54:56and US labs have a huge amount of compute. So generically, if not for catch-up growth,
- 55:00I think we would have a pretty strong expectation of US labs moving faster.
- 55:03The other argument is, if the cyber models are getting better and better, shouldn’t we wait?
- 55:11Instead of having to wait six months to get our security posture up and give China six months to
- 55:16catch up, maybe we only have to wait two months, if cyber models get two months better from now.
- 55:22There’s this analogy one can draw to sending a probe out to Alpha Centauri, and whether we
- 55:26should do that with today’s technology, in which case it would arrive thousands of years later,
- 55:30or we should do it with technology 100 years from now, in which case it would
- 55:34be lapped on the way to Alpha Centauri. We could imagine a similar thing is true
- 55:37for securing the algorithmic secrets and weights at OpenAI and Anthropic,
- 55:43where if they started the pacing or started the pause two or three months from now, the
- 55:48speed at which they would be able to secure the systems is faster than if they started right now.
- 55:54I think this is a really important question and one of the bigger questions.
- 56:01The downside on the other side, to make it clear, is that if you had diffusion of algorithms right
- 56:06now, it would be much worse than diffusion of algorithms six months ago, but much less bad than
- 56:09diffusion of algorithms in a couple of months. Right now, you require more compute and you
- 56:13require more human capital to proceed with AI development.
- 56:16The more time passes, the more disastrous it is to have an adversary actually willing
- 56:20to just grab everything and try to compete. So the reason to do things earlier is that
- 56:28if you have a pause earlier, there is more time for differences to wash out,
- 56:32and you are more loaded on the factors where you currently have an advantage.
- 56:36What do you mean, more loaded? Compute matters more and people matter
- 56:39more right now than they will in six months. You might imagine, if we’re in the world where we
- 56:43have rapid progress over the next year, six months from now, it is likely the case that it will
- 56:47essentially not matter how many people you have. It will only matter how much compute you have.
- 56:50At that point, there is less accumulated institutional know-how, and it just
- 56:54becomes this raw contest of numbers. It’s very easy for any delay you’re
- 56:59willing to take, either for security or for safety, to be decisive in that kind of contest.
- 57:04The concern is that as you move forward, it gets more and more likely that’s the world you’re in.
- 57:08Some people would say this is basically already the world we’re in, that we have now come too far.
- 57:12This is another argument that has been made, that we’re already in the "just go all out and race"
- 57:16world, because the leakage is already too bad and you will be overtaken if you slow down for safety.
- 57:22That is probably not the case. But the extent to which you’re seriously
- 57:24entertaining that you’ve already gone too far illustrates why you might not say, "I’m just going
- 57:28to wait another two months and harden things." Maybe the last thing to say is that it’s a
- 57:32little bit unclear what the dynamics are in terms of how quickly you can harden things.
- 57:35Some of hardening things is writing new software, or changing the way you operate.
- 57:39Some of it is changes that humans make, or changes to your overall arrangement of compute,
- 57:44which are not as easy to make really rapidly, or maybe not as accelerated by AI.
- 57:49So you have at least some part of this lag which is not tied to the capability of your AI systems.
- 57:55It is messy. Again, I do think the biggest bottleneck is: can people agree?
- 57:59Can we have out this debate and actually understand what the net is?
- 58:03If we are this close to superintelligence, or at least there’s a good chance we are,
- 58:08and there’s a good chance that superintelligence could be in a position to overpower governments
- 58:14and kill lots of humans, what should the AI labs be doing right now?
- 58:18If you had to boil it down to one answer, what would you say?
- 58:21I think there needs to be more public visibility into the capability trajectory.
- 58:25We should not be relying on METR and Anthropic saying, "It looks to us like we’re going to be
- 58:29able to overthrow governments in a year." We have a vested interest in really
- 58:33seeing the underlying data for that. We need visibility on alignment, to understand
- 58:38how we tell that existing systems are adequately aligned or controlled, and to actually start
- 58:42having a public discussion about what the bar is for good enough evidence that things are working.
- 58:48Then, beyond just information, at some point you do need to say,
- 58:51"Here is what we need to get done before we continue making progress, either before we
- 58:55scale up our models further or before we continue producing algorithmic developments that may leak."
- 59:02The first are inputs into the second. We need to understand where capability is at
- 59:05and where alignment is at in order to understand: is it a time where we need to improve our security
- 59:09or safety situation before we proceed? But I do think, given the current state of
- 59:13uncertainty, if I was a person in the public without access to further information, my
- 59:17strong opinion would be that it looks like we are already at a point where we need to be improving
- 59:21safety and security before moving further. I think, again, the most important form of moving
- 59:26further right now is algorithmic developments that improve the efficiency of pre-training and RL.
- 59:31Okay, stepping back, you’ve got the superintelligences in a data center.
- 59:35I’m trying to lay out an intuition for the kinds of capabilities that superintelligence would have.
- 59:39What should we imagine those intelligences being capable of doing?
- 59:43Just to start with a high-level sketch of my view, maybe there are two lenses you could use.
- 59:50One is to look at current progress and just see where it leads if you extrapolate
- 59:53that progress a responsible distance. The second is to think about what we
- 59:57know about the limits of the technology and where we would expect those limits to be.
- 1:00:02On the first front, some things I think are relatively clear.
- 1:00:05If you imagine AI systems doing cyber operations today,
- 1:00:07we have seen an incredible rate of progress. We are currently seeing pretty impressive
- 1:00:12capabilities in terms of AI systems not only being able to discover vulnerabilities,
- 1:00:15but develop exploits and increasingly chain them together into operations.
- 1:00:19I think what you will see for superintelligent systems is both of those capabilities scaled
- 1:00:24up much, much further: systems which are able to carry out, maybe very cheaply,
- 1:00:30cyber campaigns more sophisticated than anything a state is capable of today.
- 1:00:34You will see those capabilities being combined with a much broader set of abilities to, say,
- 1:00:39engage effectively with humans, think strategically, manipulate humans,
- 1:00:43manipulate organizations, play politics effectively.
- 1:00:46I think you’ll see those capabilities at a level well beyond any human.
- 1:00:50You should think of this as very, very large populations of AI systems thinking quite quickly.
- 1:00:54Maybe a very simple model would be a population of billions who are thinking 10x faster than humans,
- 1:01:00who are much better than humans at essentially every skill.
- 1:01:05This is all what you would expect from an extrapolation of existing technologies: very good
- 1:01:10capabilities in domains like cyber or manipulation or research, like doing scientific research,
- 1:01:17hypothesis generation, experiment design. You’d expect to see extremely strong
- 1:01:21capabilities in robotics, in logistics and coordination and control.
- 1:01:28You would expect to see the possibility for very, very rapid technological progress,
- 1:01:32including in domains that involve physical experimentation, just in terms of being
- 1:01:36able to orchestrate experiments quickly and rapidly build out apparatuses to increase
- 1:01:40throughput of experimental results. So that’s one bucket of things that
- 1:01:46I think you can see by drawing dotted lines on graphs and talking about where they go.
- 1:01:51Maybe stepping back a bit in terms of what the technology looks like more fundamentally,
- 1:01:56humans are very good at this task where you throw a human in a new domain.
- 1:01:58The human takes in some information, learns about that domain, and becomes able to act effectively.
- 1:02:04The reason we’re effective in the modern world is primarily that ability to learn
- 1:02:07about a new domain and adapt to it. Humans do not have a ton of evolutionary
- 1:02:11context about how to be a good chemist or a good biologist.
- 1:02:14That’s something we pick up over the course of a couple years or decades on the job.
- 1:02:18In some sense, the most fundamental skill that AI systems will possess at
- 1:02:21the end of this process is that kind of rapid learning, adaptation, planning, reasoning.
- 1:02:27These very domain-general skills, such that even if you drop them in a world
- 1:02:31that was completely unlike anything they had experienced in a data center,
- 1:02:34they would adapt to it very quickly, in the same way that humans adapt very quickly to these R&D
- 1:02:38domains that are quite different from anything we encountered in our evolutionary environment.
- 1:02:43That, I think, would result in systems which, collecting maybe even just a couple of days of
- 1:02:47experience in a very parallel way, would end up with real-world capabilities in
- 1:02:50essentially any domain much beyond the abilities of the best human experts.
- 1:02:54They will have learned from much, much more data, with learning algorithms that I think
- 1:02:57at that point will be much better than those that evolution has given humans.
- 1:03:01So that’s a more abstract picture of where this ends, beyond just drawing dotted lines.
- 1:03:09Maybe I’ll try to say that again, because this ends up being a
- 1:03:15significant crux in these conversations. I’ll be talking to somebody about the
- 1:03:19possibilities of superintelligence, and they’ll say, "Okay, you’ll have a thing
- 1:03:22which is superhuman at coding, but how will it learn to be good at politics?
- 1:03:26Or how will it learn to run a new automated firm and thus compete with human firms?"
- 1:03:37In those conversations, you realize there’s a big disconnect between the way you are talking
- 1:03:41about superintelligence and the way most people understand superintelligence, which is just narrow
- 1:03:46AIs that are specialized in some domain and have amalgamated a lot of very specific experience or
- 1:03:54expertise in, say, coding or mathematics. What you’re imagining instead is
- 1:03:58something like human-like learning. The top engineers at SpaceX were not
- 1:04:02born knowing how to make a rocket that can get hundreds of tons of cargo to orbit.
- 1:04:07This is an expertise they gained working on the problem during their lifetime.
- 1:04:11You’re saying this meta-learning capability is something that superintelligence will also have.
- 1:04:15It would at least have a learning capability as good as humans.
- 1:04:18You’re suggesting it could be much better. Furthermore, even if it were only as good
- 1:04:22as humans, right now there are maybe tens to hundreds of millions of instances of ChatGPT or
- 1:04:30Claude actively doing real work in the economy. They’re helping somebody write code or helping
- 1:04:35some senator write a political brief or something. You’re imagining a world where we have learning
- 1:04:44algorithms such that AIs can learn from all this experience at once.
- 1:04:50Right now, the base model is not even learning from the experience of a single deployed instance.
- 1:04:54You’re imagining a world where you have 100 million instances of ChatGPT going and doing
- 1:04:59something, and the base model is learning across 100,000 different careers at once,
- 1:05:04or 100,000 different work sessions at once, and doing it much better and faster than
- 1:05:08humans can learn from a single instance. And you’re also imagining a world where the
- 1:05:11population of AIs is… Do you have an estimate of how big a population we should be thinking about?
- 1:05:18I should think about it more carefully. But maybe a really important part of all of these
- 1:05:24discussions, before talking about population size, is that it really matters if we are imagining
- 1:05:28really driving this process extremely fast. I think people can really easily be in the
- 1:05:32headspace of, "I’m extrapolating another year of progress at the current rate."
- 1:05:34It’s really worth emphasizing: we are talking about systems which get not only extremely good
- 1:05:39at coding but extremely good at every well-posed virtual task which machine
- 1:05:43learning engineers are able to make in the lab. So if you push that to its limit and you ask how
- 1:05:47large those populations are, I think it’s very conservative to say a population of billions of
- 1:05:50agents running at 10x human speed. We could debate exactly what the
- 1:05:53limits are, but I think they would be significantly larger than that.
- 1:05:56Just because of software efficiency gains? That’s right. Just how efficient can you make it?
- 1:06:01You’re not going to be talking about a single type of agent, all the same,
- 1:06:03one billion of them running. We’re going to be talking about
- 1:06:04these very heterogeneous programs, many of which are very specialized.
- 1:06:08One thing you do if you do huge amounts of R&D is say, "I would like to produce just
- 1:06:11the cheapest thing I can which is expert enough in a domain to act effectively."
- 1:06:16Again, it’s a reasonable intuition pump to say it will be more capable than billions
- 1:06:19of people thinking 10x as fast. But in fact, it’s really much,
- 1:06:22much larger populations of specialized systems, and maybe billions of very general-purpose systems
- 1:06:26which are as good as humans at anything. If you imagine really pushing this process
- 1:06:33and having faster and faster progress on software, and if someone grants that we’re going to see that
- 1:06:37process saturate, that everything you can do in the data center is going to become really great,
- 1:06:41I think that endpoint is really extreme. The gap between doing really extreme
- 1:06:45stuff in the data center and doing really extreme stuff in the world is very small,
- 1:06:49or smaller than people appreciate. There’s a discussion to be had about
- 1:06:57whether such systems are even possible. I am very compelled by the argument that
- 1:07:06there’s some learning algorithm that human brains have, and something which can learn at least as
- 1:07:10well as a human can, but which has huge population sizes and can learn from the deployed experience
- 1:07:17of hundreds of millions of different instances at once — that alone would be pretty transformative.
- 1:07:23And then I agree with you that there’s no clear reason to think that you can’t get even
- 1:07:27more learning-efficient than humans. I’m very happy to talk about this.
- 1:07:31I don’t know if it’s going to end up being the main objection people have.
- 1:07:34But I think there’s very good evidence that you should expect systems in the end, once you’ve
- 1:07:37really driven the technology to its limit, to be much, much better than humans in these ways.
- 1:07:41It’s not just a case of, "We have humans, so you know you can do at least that well."
- 1:07:44We have good evidence it’s much better than us. One relevant thing is just the rate
- 1:07:49of progress on evolutionary time. If you look back over evolution, it’s fairly clear
- 1:07:52that we’re seeing extremely rapid improvements in learning capability and intelligence and
- 1:07:56planning capability as you move into humans. The thing that happened was just that humans
- 1:08:00got good enough that cultural change was faster than those biological improvements.
- 1:08:03So if you just ballpark evolution, we know it was clearly, if we didn’t form civilization,
- 1:08:08going to make much smarter things. Then, as you say, there’s the data question.
- 1:08:12Humans learn from one lifetime of data. AI systems already learn from radically
- 1:08:16more data than humans. Even if you were no
- 1:08:18better as an algorithm designer than evolution, you would produce much, much
- 1:08:21better minds if you can relax that constraint. Then you have things like power constraints and
- 1:08:25coordination constraints, which we have a pretty good sense are a really big deal for evolution.
- 1:08:29We will be able to push those out much further. If you want to produce electricity or something,
- 1:08:34to make a nuclear reactor you don’t have to be smarter than evolution.
- 1:08:36You’re just operating in a different regime from evolution.
- 1:08:38You’re able to do things that wouldn’t have been a good idea for evolution.
- 1:08:41So I’m very happy to dig in more on this, but it’s probably not going to
- 1:08:44be the most compelling argument for most people. It’s more compelling to look at the status quo and
- 1:08:47say, "Well, we’ve had 10 years of progress. You only have to extrapolate another 10
- 1:08:50years to get to the crazy superintelligence." But I do think we have a pretty good indicator
- 1:08:53that it does not top out anywhere near humans. Just to repeat your last point, what you’re
- 1:09:00saying is, look, humans developed on the savanna under this tight constraint,
- 1:09:05where evolution was really just trying to get something that is good at propagating its genes.
- 1:09:08One of the ways that a chimpanzee, or our ancestors, can get better at propagating
- 1:09:11its genes is by having these general planning and reasoning capabilities.
- 1:09:17But they’re not worth so much that you are willing to, say, give up digestive functions
- 1:09:22or immune functions and surrender more of your total 100-watt budget to the brain.
- 1:09:27You have this extremely tight power constraint for the brain, and an extremely
- 1:09:30tight size constraint as a result not only of that, but of other evolutionary
- 1:09:34reasons as well, like how your head can literally fit through the birth canal.
- 1:09:38That will not be true of AI systems. We’re not saying that hardware will be
- 1:09:42more power-efficient than the human brain, but simply that it’s not as big a constraint for us
- 1:09:46to have enormous amounts of hardware consuming a lot of power as it was for evolution when it
- 1:09:52was designing the human brain. This cuts in both directions.
- 1:09:54It’s also the case that you’ll have extremely power-efficient systems.
- 1:09:57If you’re a human, you can’t say, "Well, I’m going to make 10,000 simple agents, each of which uses
- 1:10:02only 1% of a brain or 0.01% of a brain, and is better than a human at doing cyber operations."
- 1:10:06It’s just this flexibility about form factor that is going to be huge.
- 1:10:10If you were no smarter than evolution and just did what evolution did, you would end
- 1:10:14up with agent swarms that are completely beyond teams of humans, really radically beyond them.
- 1:10:21Let's say that an agent is vibe coding a web app and it needs a database.
- 1:10:25There's one database provider that needs a human to go through the whole sign-up flow, verifying
- 1:10:29their account, adding billing details, and then generating an API key to hand back to the agent.
- 1:10:34And then there's another provider that lets the agent make an account all on its own.
- 1:10:38The agent will probably choose the second one. In general, agents tend to pick the tools that
- 1:10:43they can use right away versus the ones that they had to wait around on.
- 1:10:46So as agents do more and more work, giving them their own way to sign up becomes
- 1:10:50an important way to win new users. My sponsor, WorkOS, makes this easy.
- 1:10:55If you use WorkOS for login, you can give agents their own sign-up flow
- 1:11:00just by toggling on Agent Registration. You can then decide what agents should
- 1:11:04be able to do on their own, and what, if anything, should wait until the person
- 1:11:08they're working for gets involved. Going back to the database example,
- 1:11:12maybe you want to set things up so that agents can create databases all on their own,
- 1:11:16but they can't share them with anybody or upgrade to a paid plan without a human.
- 1:11:20You can do all of this with permissions that you already set up in WorkOS, instead of having
- 1:11:24to build a separate system for agents. To learn more, go to workos.com/authkit.
- 1:11:32There’s a bunch of objections people have to the idea that automating AI research will
- 1:11:39lead to crazy fast progress along this more general cluster of capabilities.
- 1:11:44You talk about experiments and data and everything else, but I think what it comes
- 1:11:49down to at the end of the day, if you dig in deep on any of these, is the sim-to-real gap.
- 1:11:55Is it possible that you could have a process which occurs entirely within a data center,
- 1:12:01which results in minds that are not only as capable as humans at learning in a new
- 1:12:05domain — especially a domain that is a real-world thing, like making a new company or negotiating a
- 1:12:13deal with China — but even better than humans? How do we get AIs that are capable in this
- 1:12:23more general way you’re suggesting from the within-data-center training
- 1:12:26that will be happening over the next year? I do want to start by reiterating that if
- 1:12:30you take progress in the real world — again, granting that it is slower on messy real-world
- 1:12:34tasks than it has been in math or coding — and just extrapolate it forward, I do think
- 1:12:40it gets us to systems which are powerful enough that they would be able to outcompete humans.
- 1:12:46We could talk about what people are doing to drive that and how you can scale it into the future.
- 1:12:52One, you can train on a bunch of related well-posed tasks.
- 1:12:56These systems we’re talking about will have been trained on a huge amount of predicting
- 1:12:59how humans will respond to something. When we talk about these AIs coming
- 1:13:02out of the data center, they’re not AIs who have never seen the world.
- 1:13:04They are AIs who will be radically, radically superhuman at predicting things like, given what
- 1:13:08just happened with that person’s face, how should you update on what they’re going to do tomorrow?
- 1:13:12Or given what this company has done over the last three weeks, what do you most
- 1:13:15expect this company to do in the next week? Or given observations of this laboratory
- 1:13:19environment, what is the best prediction for the results of this experiment?
- 1:13:22So when we talk about sim-to-real, we should be clear that we actually are including a bunch
- 1:13:26of these tasks which people train on now. That even includes things like, given the
- 1:13:30data so far from this laboratory, do as many computational experiments as you can, and as
- 1:13:35much hypothesis generation and exploration as you can, in order to come up with a hypothesis to make
- 1:13:39a good prediction about what you’ll see next. So these systems will also be very superhuman
- 1:13:42at many parts of science, at reasoning about the real world, at planning and hypothesis
- 1:13:48generation and so on about the real world. That’s part of the story for how even the
- 1:13:52kind of thing we’re doing now can produce very powerful systems.
- 1:13:54A second part of the story is this. One regime is that you train on
- 1:13:58exactly the thing you care about. You say, "I care about writing code that passes
- 1:14:02this type of unit test, so I’m just going to train systems that are incredibly good at that."
- 1:14:06You can also do a second thing, which is a large part of what we observe in performance:
- 1:14:09train on a broad distribution of tasks, either in a general domain or across several domains,
- 1:14:14and then deploy those systems. They acquire general skills relevant
- 1:14:18to that domain, which transfer to new tasks. I can say, "I’m not going to try and train a
- 1:14:22system which is good at the exact problem of going and talking to
- 1:14:26Congresspeople in order to influence them." But I can train a system on an extremely broad
- 1:14:30range of different tasks where there will be a lot of common skills, including these
- 1:14:33general skills: How do I adapt on the fly? How do I learn? How do I plan? How do I
- 1:14:39set up reasonable experiments that I can learn from and tell how well I’m doing?
- 1:14:42I can create very broad distributions of tasks, some of which could be coding tasks,
- 1:14:45but some of which can be tasks like, you’re dropped into a new, very messy virtual
- 1:14:49environment and need to try and navigate it. Or you’re given a new type of task with 20
- 1:14:54examples of it, and you need to learn how to do a really good job on the next example quickly.
- 1:14:59Or I have to interact with other AI systems, or with humans, in environments where I can.
- 1:15:03In the data center, there is a fair amount of human data available.
- 1:15:06You just go through the numbers and ask, "How much interaction with humans can you actually get?"
- 1:15:09It’s not like these systems have never seen humans.
- 1:15:10They will have seen vastly more human interaction than any human would.
- 1:15:14So they can also deal with a bunch of reinforcement learning
- 1:15:16environments that involve other humans or involve collaboration with humans.
- 1:15:19Because people have written in how a human would respond to something?
- 1:15:23How is that human data in there? You can also just literally engage
- 1:15:26with humans as part of the training process. People do spend large amounts of money saying,
- 1:15:29"Great, I’m going to have some humans engage with the system."
- 1:15:34There’s a question of how efficiently you squeeze information out of that data.
- 1:15:37How quickly do you learn? If you have these very general skills, how
- 1:15:39quickly do you learn about the particular domains? But again, the dotted line from what we’ve seen so
- 1:15:44far, together with a ton of capability to push that forward and to do a bunch of automation,
- 1:15:48like the construction of many more environments, just looks very likely to lead to systems
- 1:15:55that can quite quickly rise to at least human-level, and likely way past human-level,
- 1:16:00including on very messy real-world tasks. The thing I’m curious about is why you
- 1:16:05think this happens over the next year. It seems in principle possible to me that
- 1:16:12there’s enough data available even within a data center that if you have a decent amount
- 1:16:17of transfer, you have a very capable system. If there’s enough transfer from evolution
- 1:16:23training us in the savanna to a human being now able to make a semiconductor, then maybe
- 1:16:27there is enough transfer from what’s happening inside the data center to going to Congress and
- 1:16:32trying to persuade them of something. But the things you’re mentioning,
- 1:16:36like this endless backtesting or this RL on fuzzy tasks, seem like things we’ve
- 1:16:45already been doing for many years now. It doesn’t seem like we’re a year away,
- 1:16:48or a further year of scaling away, from this big a jump.
- 1:16:53Am I correct in understanding that you’re imagining something like what we’ve already done,
- 1:16:56but with a much bigger qualitative jump along those same lines?
- 1:17:00Yeah, much faster. If you asked me, "Hey, Paul, if you’re extrapolating the dotted line and looking
- 1:17:05at the mechanics of what people are doing, how long would it take to produce systems that are,
- 1:17:08in fact, capable enough at all these tasks that they would be able to reliably outcompete humans?"
- 1:17:12I would probably say it seems like it might happen in four years.
- 1:17:16Maybe it could happen faster than that. I feel more confident within eight
- 1:17:20years or something. Those are the kinds of
- 1:17:22numbers I would be throwing around if we were just asking, on the current trend,
- 1:17:25how long will it take, doing exactly the things we’re currently doing, to get these capabilities?
- 1:17:30There’s lots of room for debate about those numbers.
- 1:17:32But the thing that I am most worried about, that seems like an extremely live possibility, probably
- 1:17:37more likely than not at this point, is that instead the work that people are doing in machine
- 1:17:43learning to drive all those improvements we’ve seen over the last 4-8 years will be automated
- 1:17:47to such an extent that the pace of that progress is accelerated by at least 10x, and likely more.
- 1:17:54We are maybe already seeing acceleration in the ballpark of 2x.
- 1:17:57I think it is very likely, on the current margin, that the first thing that acceleration
- 1:18:01does is continue sustaining itself. Regardless of this broader discussion
- 1:18:06of how good AI systems are at acting in messy real-world environments, AI systems are very
- 1:18:09good at acting in virtual environments when they are trying to deliver on verifiable tasks.
- 1:18:14I think the thing that is most robust is that AI systems are improving extremely rapidly and
- 1:18:17are starting to cross human-level on the kinds of tasks that are involved
- 1:18:20in making your AI systems better at learning in environments you can simulate in the data center.
- 1:18:26Slightly more broadly, that includes the kinds of software engineering tasks that
- 1:18:28are involved in creating new environments that stress-test hard skills, and even many
- 1:18:32of the software engineering skills involved in orchestrating data collection from the real world.
- 1:18:36We could talk about what the bottlenecks are in that process
- 1:18:40which cannot be accelerated by automation. But if you look across the core things
- 1:18:44humans have been doing to drive the last 4-8 years of progress, those are things
- 1:18:47we are on an extremely good pace to automate. We are fairly likely to have the self-sustaining
- 1:18:51acceleration where, in eight months, you double the pace, and in the next four months,
- 1:18:56you double the pace again, and in two months, you double the pace again.
- 1:19:01At the beginning of that process, it will be focused primarily on these virtual tasks,
- 1:19:05because your main priority is making the self-sustaining loop faster and faster.
- 1:19:09If you’re a person who just wants to get to the endpoint as quickly as possible, you do not care
- 1:19:12how good a lawyer your AI system is for a while. You just care how good it is at driving the
- 1:19:16development of better architectures and better optimizers and better learning algorithms,
- 1:19:21or even constructing new environments. Maybe you care about one slightly messy task,
- 1:19:27which is constructing environments that help you build the skills that
- 1:19:30are relevant in the real world. But you don’t care about almost
- 1:19:32any of the real-world tasks where we’re seeing the slowest progress.
- 1:19:35You’re focused on that. You’re going through this process of seeing doublings in eight months
- 1:19:38and four months and two months and one month. Then, as that process goes, it becomes cheaper and
- 1:19:42cheaper, faster and faster, to develop algorithms and models that perform very effectively in the
- 1:19:48real-world domains that ultimately matter. That is really the concern.
- 1:19:52The first question is, is that process going to occur?
- 1:19:56Are you going to see the self-sustaining acceleration with faster and faster progress?
- 1:20:00Honestly, to me, that is probably the bigger open question.
- 1:20:03I think that looks more likely than not right now. But if it occurs, I think things are almost
- 1:20:06certain to be extremely crazy. There is then a second question
- 1:20:09about the particular way in which they get crazy, which is that it looks very likely
- 1:20:12those models will transfer to the real world, to systems that are able to outcompete humans.
- 1:20:17But that does happen very back-loaded in the year. If you’re imagining a year where very disruptive
- 1:20:21things happen, and this is part of why it’s so hard from a policy perspective,
- 1:20:24for the first six months of that year, you look at the systems and say, "Well,
- 1:20:30they were previously superhuman at math, and now they’re incredibly superhuman at math."
- 1:20:34But if you go and talk to someone, there’s no real ability to say,
- 1:20:37"Therefore, they are a threat to humanity." AI labs are currently saying things like,
- 1:20:42"Every month, we’re doubling the amount of internal use of AI, and the degree to which
- 1:20:45it’s contributing to our progress." Or, "Over the last six months,
- 1:20:48we’ve seen a doubling of the net pace of progress, even including diminishing returns."
- 1:20:53You can have more statements like that. But you could be in a world where the
- 1:20:59link from that to very severe harms does not become obvious until quite late in the process.
- 1:21:04To some extent, I think it is getting increasingly obvious now.
- 1:21:06If you look at cyber capabilities of models in particular, but also their capabilities in
- 1:21:10some scientific domains, they are approaching the point where there are very severe risks.
- 1:21:17We’re seeing that in a couple of domains, so it’s tempting to say, "Okay, we have two problems.
- 1:21:20There are just a few specific domains in which AI systems have dangerous capabilities."
- 1:21:25But if this process of acceleration continues, that
- 1:21:27is not going to be localized to cyber or to bio. That is going to be a thing that occurs across all
- 1:21:31of science, across all of social engineering. If you’re thinking about the risk,
- 1:21:36you really need to understand whether this is possible: whether you can have the self-sustaining
- 1:21:41acceleration, then leading to very, very rapid progress in dangerous real-world capabilities,
- 1:21:45both in specific domains and in these very general learning faculties we’ve talked about.
- 1:21:49This goes back to the thing you were saying, that you should imagine the systems in the
- 1:21:52future are less about fine-tuning GPT-6 to do something, and more like GPT-12 being able to
- 1:21:58do it out of the box with very little experience. GPT-12 will also be great at fine-tuning GPT-12.
- 1:22:03If GPT-12 is like, "I find that my chemistry intuition is not very good," GPT-12 will be
- 1:22:07great at saying, "Okay, I’m going to direct 1,000 humans, and here’s the data I think I most need.
- 1:22:11We’re going to do one day of data collection of that type.
- 1:22:14We’re going to run these experiments and do the de-risking."
- 1:22:16Within four days, GPT-12 has the new tiny model which has incredible
- 1:22:20chemistry intuition, and it’s doing this in parallel across all domains.
- 1:22:26You just answered a question which I meant to ask you.
- 1:22:28It seems like a big thing that has driven your concern over the last few days is this new result
- 1:22:32from METR, where they find that Anthropic’s pace of AI progress on well-scoped tasks has doubled.
- 1:22:40These are tasks as captured by benchmarks like the Epoch Capabilities Index, which
- 1:22:45amalgamates a bunch of different benchmarks. Now, it could be that there’s other evidence
- 1:22:49relevant to METR’s assessment that AI progress has doubled, but that evidence
- 1:22:52has not been made public. That’s one of the problems
- 1:22:54you have with the current setup. But just consider that evidence on its own.
- 1:23:00Progress on verifiable tasks has doubled. But we have this effect where we’re searching
- 1:23:05for our keys under the lamplight. It is also the verifiable tasks
- 1:23:09which we’re training on. So you can ask the question,
- 1:23:12has progress on non-verifiable tasks also doubled over the last year?
- 1:23:19As you may have noticed, over the last week the stock market took a big tumble.
- 1:23:22Part of that was almost certainly the midterms. Another part of the reason is that the growth of
- 1:23:30Anthropic’s and OpenAI’s revenue numbers did not keep track with what the market was expecting.
- 1:23:34In Q3, Anthropic’s revenue grew something like $5.4 billion.
- 1:23:39That was less in absolute terms than Anthropic’s revenue grew in Q2, $6.9 billion.
- 1:23:45So not only has the rate of growth slowed down, which people expected,
- 1:23:48but the absolute growth has also slowed down. That maybe suggests that measuring progress on
- 1:23:55well-scoped tasks is not actually telling us about the pace of overall AI progress.
- 1:23:59Because how can it be the case that overall progress is 2x faster,
- 1:24:03and yet absolute revenue is not even growing at the same rate it was previously growing?
- 1:24:08Maybe the explanation is what you just suggested: these AI companies are really developing better
- 1:24:14and better RSI models, and only incidentally releasing Claude for Law or Claude for Finance as
- 1:24:21a stopgap to keep this underlying process going. This hasn’t been a priority,
- 1:24:25and they haven’t worked that hard on it. You should really be concentrating on this
- 1:24:30inner exponential of the RSI capabilities of the model, and that will then generalize.
- 1:24:35But how would you interpret this disconnect between the pace of improvement in the end
- 1:24:41usefulness of these AI systems and the pace of improvement on well-scoped tasks?
- 1:24:47Zooming out slightly, if we talk about not just the last six weeks, but in general what
- 1:24:51has happened over the last 18 months, I think we have seen extraordinarily rapid growth in revenue.
- 1:24:57We can talk about what the story is for the last four weeks.
- 1:25:00But we are talking about revenue more than doubling every six months.
- 1:25:06That is very, very fast growth for the industry overall.
- 1:25:12If we want to match up these internal results on progress to external results,
- 1:25:17the period in question has also corresponded to a period of incredibly fast revenue growth.
- 1:25:21It’s very hard to say whether that revenue growth corresponds to capabilities progress that is
- 1:25:24twice as fast as previous capabilities progress. I think the revenue numbers are noisy enough, and
- 1:25:28the staggering of model releases is noisy enough, that it is really hard to draw that judgment.
- 1:25:34There are just a lot of problems in trying to make that inference.
- 1:25:39At a high level, I think revenue is a reasonable indicator, and revenue
- 1:25:42progress is extremely rapid. But very short-term trends in
- 1:25:44revenue, particularly at a single company level, are extremely noisy.
- 1:25:49To me, it feels like there’s a discourse where, periodically,
- 1:25:52the length of the AI winters is shortening. Now it’s been two months, and AI revenue
- 1:25:58over the last two months has not been on track to grow as explosively as before.
- 1:26:03That doesn’t mean it’s all over. My guess is that you’re just seeing
- 1:26:07some amount of noise in these processes. Just to take the other side of that, you
- 1:26:13might think it’s because Dario released his pacing letter, and they haven’t
- 1:26:16released their best models or something. But if you look at the METR graph for
- 1:26:21the capabilities index, Opus 5.5 doesn’t seem that far below Model 4, which seems
- 1:26:28to be Anthropic’s latest internal model. This might be the result of the staggering
- 1:26:35of AI capabilities, but it’s not clear this is the staggering of model releases.
- 1:26:39The main thing I mean about model releases is that the rate of revenue growth, again,
- 1:26:42specifically at the company level over very short time periods, is extremely noisy, because you’re
- 1:26:46looking at a small number of model releases. You’re looking at these market
- 1:26:49dynamics over very short time periods. My main opinion on this, and I am happy to dig
- 1:26:55more into it, is that it is extremely hard to draw inferences from extremely short time series data.
- 1:27:02This is true to a reasonable extent for AI capabilities too.
- 1:27:06The more downstream you go, the noisier things get.
- 1:27:08But part of the issue on the RSI side is that I also don’t really think we can draw
- 1:27:12inferences from the last couple months of data very well in terms of benchmark performance.
- 1:27:17We have the same thing there, where there’s enough noise that we have no idea.
- 1:27:22We can tell, over the period of years, whether things have accelerated.
- 1:27:25It’s very easy to look back over the last four years and
- 1:27:27say roughly how fast this thing’s been going. It just gets much harder as you zoom into small
- 1:27:30time periods because of the amount of noise. If you look at the details of where models are
- 1:27:35coming from, you have these pre-training runs that occur at some point in time.
- 1:27:38When do pre-training runs finish? Then you have the exact sequence of post-training,
- 1:27:42and which model checkpoints get released when. You have random variability in model quality.
- 1:27:48You have those dynamics, which make things very noisy over short timescales.
- 1:27:52Then those just get filtered out to even noisier stuff as you move to the market more broadly.
- 1:27:58If we’re going to be in a position where within a year we have superintelligences that could
- 1:28:02overthrow the government, what exactly is the plan to make them such that they don’t?
- 1:28:07Because we’re asking for a level of saintliness that we should just
- 1:28:12not expect out of even most humans. Sometimes people say that the goal of
- 1:28:15alignment is to give AIs human values. This doesn’t make sense to me.
- 1:28:19Of course, different humans have different values. But also, if there’s one lesson that history
- 1:28:24has given us, it’s that humans can’t be trusted with arbitrary amounts of power.
- 1:28:28Cortés was a human. Stalin was a human. A logical conclusion of believing the
- 1:28:39technological and forecasting model that you have laid out is just that we will have systems
- 1:28:44that we will have to trust with an overwhelming amount of power, if not realized then latent,
- 1:28:50because they will have the capability of overthrowing the government if they want to.
- 1:28:58Then you just think about why they would overthrow the government.
- 1:29:01Either they have secret goals as a result of the training process, or, as a direct result
- 1:29:05of the training process, they care about how they’re graded, or a million other things.
- 1:29:10This is the basic argument about instrumental convergence.
- 1:29:12There are many different reasons somebody could want power.
- 1:29:15So what is the argument? If this happens in a year,
- 1:29:18how do things turn out okay? What is the actual plan here?
- 1:29:21I think there are a lot of different things someone could hope for.
- 1:29:24They are all at the stage of "we hope something will happen over that year."
- 1:29:28We’re producing very smart AI systems. They’re going to do a bunch of research.
- 1:29:31It is plausible they will think of something over that year that we have not thought of.
- 1:29:34That is mostly where we’re at right now. If they do not think of something clever
- 1:29:38that we haven’t thought of, it is probably going to end badly.
- 1:29:43In terms of the kinds of hopes you could have, there is one
- 1:29:47hope that’s just: try and make them good guys. They’re nice. They share human values. They like
- 1:29:53flourishing. They’re in a position to take over. We have very indirect control over what they want.
- 1:29:57Then we just take the leap of faith. I think this is a pretty unappealing option.
- 1:30:02In general, I hope people will not go for that option.
- 1:30:05Even if it works out, even if the AIs are good people in some sense and now they call the shots,
- 1:30:11it is not a way that humanity as a whole should be like, "We are transferring our sovereignty
- 1:30:15to this new species we’ve built," because that was the fastest way for some person to
- 1:30:18build this particular product. I don’t like that option,
- 1:30:22but it is an option you could take. There’s another option: we are going
- 1:30:27to understand scientifically more about how the distribution of training environments
- 1:30:32shapes the motivations of these systems. We’d get to such a point where, instead of the
- 1:30:35conversations we’re having now about maybe they’ll want X or maybe they’ll want Y,
- 1:30:38we’ll have a good scientific basis for saying, "Hey, if we train them in the following way,
- 1:30:41they do in fact end up wanting reward or whatever. They don’t want some other thing.
- 1:30:45They actually just want to get high scores. And maybe even moreover, if we train them
- 1:30:49in environments where they didn’t have good opportunities to hack, maybe they want score but
- 1:30:52in a nice way where they won’t hack to get it. They’ll just do what we want or
- 1:30:55what we’re trying to incentivize." You could try and have an understanding like that.
- 1:30:59That is probably the closest thing to what the industry is trying to bank on right now.
- 1:31:03It’s very dicey. It’s very dicey to not have any opportunities to
- 1:31:07hack in training, which is basically a precondition for this approach working.
- 1:31:10And then it’s very dicey because we don’t actually have a good scientific
- 1:31:13understanding of how this works. Our best expectation is that models
- 1:31:16will learn to hack at deployment time, even if they never hack in training.
- 1:31:19I think that is the best scientific guess now. So you could try and deal with that.
- 1:31:25You could try and say, "We’re going to understand really well how models work internally,
- 1:31:28and then we can train them based not just on what they do, but on how they think."
- 1:31:33Do good interpretability or something. Be like, "We understand here’s how it thinks about
- 1:31:36reward, and here’s how it thinks about hacking. And we’re not just going to select for
- 1:31:39whether it behaved well in training. We’re going to look and intervene directly
- 1:31:41on how it thinks, and either notice that something is going bad or prevent something from going bad."
- 1:31:46As an aside, I run this organization, ARC, which is working on not the interpretability version,
- 1:31:50but the mathematical version of that hope. Can we write algorithms which do structural
- 1:31:55analysis of the weights of a neural network in order to understand something about its behavior,
- 1:32:00even if it’s not something we could select for in training?
- 1:32:02You’re literally having to understand the whole computational graph of superintelligent minds?
- 1:32:06That’s correct. Or we don’t understand it, but we are writing algorithms which are
- 1:32:10able to understand and make deductions about the behavior of some superintelligent mind.
- 1:32:14I would describe it as simultaneously a bank shot, which is probably not going
- 1:32:19to end up relevant to this discussion. I think there’s a 10% chance or 5% chance
- 1:32:23that we succeed in a game-changing way. But also, we don’t have that long a
- 1:32:27list of things we’re trying. I do think, pound for pound,
- 1:32:29it’s probably the most promising thing. Wait. You’re like, "Okay, here’s the thing.
- 1:32:34It has a 10% chance of working, but it’s also the most promising thing."
- 1:32:38Most promising pound for pound. If there are 20 people working on
- 1:32:43that thing, that can be a good bet. But probably what we’re going to do,
- 1:32:47if we do not slow down, is try and be like, "We fussed with the training environments.
- 1:32:53We did some experiments that give some kind of rough sense of what’s happening.
- 1:32:55We made the training environments really careful. We made the graders really good, so if you try and
- 1:32:59hack, it’s normally going to get penalized." That’s probably what we’re going to do.
- 1:33:02But you don’t think it will work. I think it will work for some amount
- 1:33:06of time, and then it will break. If you talk about this year,
- 1:33:08it works for the first nine months of the year, or some number of months of the year.
- 1:33:11We don’t know exactly when it breaks down. Sorry, it’s in the next year,
- 1:33:14so basically halfway into next year? If we go to superintelligence, at some
- 1:33:18point along the path to superintelligence, the existing methods are probably going to break down.
- 1:33:21Sorry, we’re not even on track to get it so that all the RL environments,
- 1:33:25for example, don’t have reward hacks and don’t actively encourage the AIs to hack.
- 1:33:29Not the case today. But suppose we did.
- 1:33:31Suppose we did that. What you’re claiming is that—
- 1:33:35I think eventually, basically, we’d still have a problem.
- 1:33:38The plan is basically that we just don’t know. Obviously, if you think about it,
- 1:33:44it’s a hard problem. Superintelligence is this alien mind,
- 1:33:47and you won’t have a guarantee that you understand why it is doing the things that it’s doing,
- 1:33:52even when it is thinking in alien ways many times faster than you would, in huge population sizes.
- 1:33:57That’s a goal. We don’t know how to solve it. There’s no single approach where we think there’s
- 1:34:04a 50% chance that that approach will work. That’s definitely the case. There’s a
- 1:34:07bunch of things people are trying. So the hope is just that three months
- 1:34:11before we get that superintelligence that could overthrow the government, we let the AIs loose
- 1:34:18on all these different approaches that have a couple percent chance of potentially working.
- 1:34:22Hopefully the AIs figure out one of these approaches.
- 1:34:24They give us a solution. Hopefully, at this point, they
- 1:34:27are still on our side, and they want this problem solved, and solved only in a way that benefits us,
- 1:34:34or whatever "us" means in this context. Then the hope is that we adopt it and
- 1:34:41build crazy amounts of superintelligent minds that are fundamentally on our side.
- 1:34:47Of course, I’ve obviously glossed over a very important discussion when
- 1:34:49I’ve talked about "on our side," because what does being on our side really mean?
- 1:34:52Aligned to whom? That is obviously a real discussion that can’t be glossed over.
- 1:34:56That is the plan. I agree that that is the plan. We can talk about whether they have a
- 1:35:00couple percent or tens of percent. I’m not saying this as if there’s a
- 1:35:04better plan, but I think people are not aware that this is the state of the situation.
- 1:35:13You’re sending the soldier on an expedition, and you’re hoping he finds a gun along the way.
- 1:35:19You’re building the plane as you fly it. You hope that you have the machinery to
- 1:35:23tell if something very bad is about to happen, and that you will stop then,
- 1:35:26although we’ve talked about security and why that can be challenging, and so on.
- 1:35:30Then you hope that if you do stop, you’ll be able to come up with something quickly, or you’re
- 1:35:33constantly trying to come up with something. I think interpretability is a good example of
- 1:35:36the kind of bank shot people have in mind. Right now we’re fairly far from
- 1:35:39understanding how these models work. But you could imagine that with a bunch
- 1:35:41of AI assistants, we’re able to really understand how individual models think,
- 1:35:45well enough that we can intervene in order to make them not want to do bad stuff.
- 1:35:50Another thing I want to emphasize is that I’m skeptical of whether things will happen this fast.
- 1:35:56Certainly, it is reasonable for somebody listening to be skeptical.
- 1:35:58You are skeptical, right? It’s 2/3 of worlds where, even if we went full steam ahead,
- 1:36:02this might not happen so fast. But it is a live possibility.
- 1:36:07I agree. It seems logically consistent that this could happen,
- 1:36:11and it should not be that surprising. I think people would just be shocked
- 1:36:16if they heard the kinds of discussions. This is not something where Paul is on
- 1:36:20some crazy end of the spectrum compared to how people inside the AI labs are thinking.
- 1:36:25It’s just crazy how many conversations and dinners I have with researchers at
- 1:36:29the labs where, again, you could believe the whole town has taken crazy pills.
- 1:36:33There’s certainly a chance of that. We hope. We really hope.
- 1:36:36But this is just not out of the realm of how people at the labs are thinking.
- 1:36:40This is truly how they see the situation. There’s some resignation where they
- 1:36:47accept that because this is not common knowledge, they are stuck in this dynamic.
- 1:36:51Of course, a way to solve that is to establish that common knowledge.
- 1:36:55I just wanted to really emphasize that people listening might be shocked at the way these
- 1:36:59researchers talk when they’re amongst themselves, or at dinner or something.
- 1:37:02This is really how they see the situation. A lot of people are worried to varying extents.
- 1:37:08I’m definitely more worried than the median, but less worried than the 90th percentile.
- 1:37:14If you had a dinner of researchers, I think it would be uncommon for someone to say,
- 1:37:17"There’s a 99% chance that this goes well." The discussion is more like, "Is it 95 or 75?
- 1:37:22What are we talking about for the probability of navigating this?"
- 1:37:26So why do you think it might go well? Is it just that the superintelligent AIs
- 1:37:31might give us some solution to alignment, and also that politically we are well organized
- 1:37:35enough to have a really good document… I think we basically have two saving throws.
- 1:37:39One saving throw is that there’s a reasonable chance we’re going
- 1:37:41to figure out something that works. I do think all the current things break
- 1:37:45down, but there are a number of irons in the fire. You could imagine things like
- 1:37:49the current methods that are going to work. That’s one. Maybe there’s a 50/50 chance
- 1:37:53that we find something that works in time, before there’s a disaster.
- 1:37:56You have a second saving throw: we could stop. If we discover, either preemptively or after
- 1:38:01the fact, that things are bad, we might say, "We cannot adequately control these systems,
- 1:38:04so we’re going to stop building them." I am very worried that we’re going
- 1:38:07into this process without the infrastructure that would be necessary to actually do that.
- 1:38:10Like we talked about, you stop, your adversary steals the model, and they’re like, "I’m willing
- 1:38:13to take the 50% risk," or whatever. But conceptually, you could stop.
- 1:38:17Those are my two saving throws. Each of them is a 50% chance, and bam,
- 1:38:20we’re at like a 75% chance things are okay. But that does involve being able to
- 1:38:24stop if things are not okay. That is not where we are now.
- 1:38:27So it does involve us fairly urgently doing something now in order to be in a
- 1:38:32better position to stop if necessary. I think it’s a good place to close.
- 1:38:38Paul, thanks so much. Thanks for having me. Great talking.
About this transcript
This page contains the full transcript of We could be 12 months away from AIs that can overthrow the government by Dwarkesh Patel, generated from the public captions YouTube serves with the video. The transcript has 19,506 words across 1,355 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.