YouTube2Text

We could be 12 months away from AIs that can overthrow the government — Transcript

by Dwarkesh Patel · 19,506 words · 1,355 segments · language en · Watch on YouTube

Full transcript

  1. 0:00My guest today is Paul Christiano. He is the head of the Alignment Research Center.
  2. 0:03He recently joined OpenAI’s board. Before that, he was the head of the US Government
  3. 0:08AI Safety Institute, and before that, he led  alignment at OpenAI, where he helped invent RLHF.
  4. 0:15Paul, what is your assessment  of our situation today?
  5. 0:20A couple things are on my mind,  especially over the last few weeks.
  6. 0:25Probably the most salient possibility is that  we seem to be on this trajectory to build very,
  7. 0:32very powerful AI, potentially very, very soon. If you asked me, I’d say there’s a reasonable
  8. 0:37chance that in a year we will produce AI systems  much better than humans at almost everything,
  9. 0:42where humans would have very little chance  of competing with them or controlling them.
  10. 0:48That’s the first big thing on my mind. I think that puts us in a potentially really
  11. 0:53dangerous situation, even in the short term. It’s quite likely that if we have a rapid
  12. 0:58intelligence explosion leading to  superintelligence, at some point
  13. 1:01during that process we’ll be unable to  control the AI systems we’re producing.
  14. 1:06Then I am worried that it will be very costly and  painful for us to slow down during that process.
  15. 1:12In part because the security of  frontier labs is not strong enough.
  16. 1:18I think if we produced very powerful AI  and then tried to delay deployment for
  17. 1:21safety reasons, there is a good chance  those systems would leak or be stolen,
  18. 1:25or the techniques to build  them would leak or be stolen.
  19. 1:28I think our risk assessment and governance are  also probably not mature enough to notice when
  20. 1:33things are starting to go off the rails  and to respond in an appropriate way.
  21. 1:37At a high level, that is mostly  what is on my mind right now.
  22. 1:40We have moved from there being some hypothetical  risk at some point in the future to there being a
  23. 1:45pretty severe risk even over the next 6-12 months. A risk of takeover, of AIs taking over
  24. 1:52or disempowering humanity  within the next 6-12 months?
  25. 1:55That’s right. Where would you place that risk?
  26. 1:58I would say we are looking more like a 10%  risk than a 1% risk over the next year.
  27. 2:04I think if you said AI developers will move  full speed ahead without taking precautions
  28. 2:09that would significantly slow them down, you’re  looking at significantly more than a 10% risk.
  29. 2:16I think we’re looking at something  like maybe a 1/3 chance that we have
  30. 2:20superintelligence within one year on the current  development trajectory, if we do not slow.
  31. 2:24If you believe that, things just  look very dire holistically.
  32. 2:27From almost any perspective,  you’re like, "That would be crazy."
  33. 2:30Lots of things are going to  go wrong if that happens.
  34. 2:32It is pretty likely we’re going to need  to, in some way, slow that process.
  35. 2:35There’s very little chance we’re going to make  that transition in a way that looks smooth.
  36. 2:40What is it that you’re imagining happening over  the next year, from the systems we have right now
  37. 2:44to the systems you expect over the next year? What evidence have you seen that makes it a
  38. 2:50live possibility that very crazy  things happen over the next year?
  39. 2:55For the very short term, the main worry on  my mind is that the process of AI development
  40. 2:59itself is increasingly automated, and that  automation then results in rapid increases in the
  41. 3:06quality and quantity of labor applied to AI R&D. So changes in AI capabilities that you might have
  42. 3:11thought would take four or six years might instead  get compressed into a very short time period.
  43. 3:17I don’t want to overemphasize any one data  point, and I do want to spend some time talking
  44. 3:21about the limitations of what we actually know. I do think a natural response to the situation is
  45. 3:25to say, "We need to figure out what is going on." It is plausible enough that it is an urgent
  46. 3:29priority to understand if we are  three months or six months or 12
  47. 3:32months away from AI that could overthrow the  government, which I think is the situation.
  48. 3:34I think that is the obvious  top priority right now.
  49. 3:37It is insane if we are six months away from  AI systems that can overthrow the government.
  50. 3:41Whether you’re concerned about misalignment or  abuse by a company, that is just holistically
  51. 3:45an extremely important fact about the world,  in order to do our jobs and protect Americans.
  52. 3:50So that’s one thing. I want to talk about the  limitations and the need for more evidence.
  53. 3:55But in terms of what we know right now:  first, if you had asked me one year ago,
  54. 4:01I would have said we are on a trajectory where  recursive self-improvement looks fairly likely.
  55. 4:06We’re seeing very rapid progress on AI  systems, including for things like computer
  56. 4:11science research, software engineering,  and debugging machine learning runs.
  57. 4:15In all the steps involved in AI development,  we are seeing extremely rapid progress.
  58. 4:20In principle, it does seem like that rapid  progress should allow you to automate
  59. 4:24further AI development. That has, for decades,
  60. 4:26seemed like the likely outcome of this technology. It has seemed like the trajectory it’s been on.
  61. 4:31If I look back at my history of  AI predictions, I’m generally very
  62. 4:34happy with many of the things I wrote. But the most obvious way I’ve been wrong
  63. 4:37is that things have moved way faster than I  anticipated, especially on these technologies.
  64. 4:41That was true a year ago,  without any recent data points.
  65. 4:45There are two big things that have happened over  the last two months that, for me, have been large
  66. 4:48updates about how quickly this could go. One of them has been the level of
  67. 4:52performance of AI systems on computer  science and math research tasks.
  68. 5:01Over roughly the last month, we have seen hundreds  of results in math coming out of recent OpenAI
  69. 5:08systems, as well as Anthropic systems. If you had taken the last month and said
  70. 5:14that was a decade of progress, it would  have been an unusually productive decade.
  71. 5:18The last month of results is  a lot of results for a decade.
  72. 5:22So if you are wondering what AI systems are  generally capable of on tasks where humans
  73. 5:26are thinking about algorithms and how to make  algorithms better, we just got this incredible
  74. 5:30data point saying it is actually possible  in one month to have a decade of progress.
  75. 5:34The thing we have always been worried about is  that maybe that will happen in machine learning.
  76. 5:38There are reasons machine learning is  harder than other parts of computer science.
  77. 5:42It is more messy and empirical. It involves compute bottlenecks.
  78. 5:46But we are also seeing extremely  strong performance on other aspects.
  79. 5:49We’re seeing really good software engineering  performance, maybe lagging behind mathematics,
  80. 5:53but not by that far. We’re seeing good performance
  81. 5:55on other aspects of machine learning. My guess, if you had asked me a month ago,
  82. 5:59would have been that machine learning  lags something like one year behind math.
  83. 6:03So I am very scared of seeing something  like 10 years of ideas in machine learning
  84. 6:09in the relatively near future. That was the situation a month ago.
  85. 6:14The main update over the last few days  has been reporting from Anthropic and
  86. 6:19METR on the net effect of AI systems  on Anthropic’s rate of progress.
  87. 6:27We could talk about where exactly the  evidence quality is on this, because I
  88. 6:29think that’s one of the big questions. We don’t really know what’s going on.
  89. 6:32One of the big themes I’ll be  pushing on is just that it is
  90. 6:35very hard to tell publicly what’s going on. But we are seeing data consistent with the
  91. 6:39pace of AI progress having roughly doubled over  the last 6-12 months, even including the effects
  92. 6:43of diminishing returns and compute bottlenecks. Unfortunately, most of what we have to go on
  93. 6:52in the METR and Anthropic reporting is  METR saying, "We had some conclusions.
  94. 6:56We promise we have a full report, but  the entire report has been redacted."
  95. 7:00Benchmark scores are a terrible way to  assess the overall rate of progress.
  96. 7:04There are a bunch of much more smoothly varying  metrics recorded internally through the AI
  97. 7:08development process, where we understand better  how they work: what is happening with pre-training
  98. 7:11perplexity, what is happening with the performance  and compute efficiency of RL algorithms.
  99. 7:18The claim from METR and Anthropic is that METR  looked at all of that data and said it paints the
  100. 7:23same picture as this very noisy benchmark data. They also looked at inputs and said, "If you
  101. 7:28look at the actual production function, how much  research time and how much compute went into these
  102. 7:33training runs, those are also consistent with this  story of what we are seeing as ML automation."
  103. 7:38Mostly we just have them saying that,  which is not a very good situation.
  104. 7:42But against this backdrop of thinking this  already looked fairly likely to happen,
  105. 7:46we are seeing this evidence from OpenAI suggesting  that it is reasonably likely to happen in
  106. 7:50something like the next 6-12 months, if machine  learning does not lag that far behind math.
  107. 7:54Then you are seeing data which is, if noisy,  roughly exactly the data you would expect to see
  108. 7:596-12 months before producing superintelligence. There are just a lot of sources of evidence
  109. 8:03suggesting that at least this  should not be a surprise.
  110. 8:05If we produce superintelligence in one  year, we could debate the probability.
  111. 8:08Is that 90% or 50% or 10%? But I think at this point,
  112. 8:12there is no way anyone could say that was a  really surprising, trend-breaking development.
  113. 8:16We are seeing exactly the set of metrics you  would expect to see 6-12 months in advance.
  114. 8:20It’s hard to think of anything  else you would expect to see
  115. 8:23at this range before superintelligence. Generally, how do you think about our situation
  116. 8:28in the world where things go this fast? If we actually have this trajectory of
  117. 8:33capabilities progress, if you actually  see many years of AI progress compressed
  118. 8:37into a short period of time, and possibly  more than that, it is really quite likely
  119. 8:43that something very disruptive will happen. It’s fairly likely the government will be
  120. 8:46overthrown, fairly likely many people will die. I do think the probability of seeing that kind
  121. 8:51of extremely rapid progress driven  by recursive self-improvement over
  122. 8:55the next 6-12 months is reasonably high. So there’s a reasonable chance right now that
  123. 9:03we’re in a world where, in six months, there’s a  disaster that kills very large numbers of people,
  124. 9:12or where the government is overthrown. Maybe a really important general point
  125. 9:18is that if we do a good job, we may observe  the danger increasing over that time period.
  126. 9:24As we’re building AI systems, we can maybe tell  whether they are adequately controlled and how
  127. 9:28quickly the capabilities are improving. But in order for that to actually translate
  128. 9:33into stopping in a way that is non-disastrous,  you need a couple of things to go right.
  129. 9:38It needs to be the case that you’re able to  tell the systems are not well controlled.
  130. 9:43It also needs to be the case that  if you stop, that fixes the problem.
  131. 9:47On the second bucket, probably the single most  likely negative scenario is that developers
  132. 9:53proceed and recognize that they have a serious  problem. But at the time they recognize it
  133. 9:59— maybe they’ve produced a powerful  AI they can’t adequately control —
  134. 10:03both the algorithms to develop that  AI and quite possibly the weights of
  135. 10:07the actual model have leaked or been stolen, such that they’re then in an awkward position
  136. 10:12to say, “Well, we could avoid deploying this  AI, or we could avoid relying on this AI,
  137. 10:15or we could avoid making further progress, but it is now very likely that a more
  138. 10:19careless adversary would be able to get  a major advantage over us, or maybe even
  139. 10:22a decisive advantage, by taking that risk.” That could put us in a place where if we do
  140. 10:29recognize there’s a significant risk and therefore  slow development, we may end up with both a
  141. 10:33terrible competitive situation and a disaster  coming from someone else’s use of that AI.
  142. 10:39This is a thing you sometimes hear  from people in the AI industry.
  143. 10:42Sometimes people even say it has already gone  far enough, that if you were to stop now, there
  144. 10:46would be leakage of algorithms to geopolitical  adversaries that would result in a disaster.
  145. 10:52I think it’s unlikely we’re  literally in that situation now.
  146. 10:56But I do think by the time it is clear that you’re  facing a lot of risk, you may be at a point where
  147. 11:02stopping unilaterally, or even all of the West  stopping, is not sufficient to avert a disaster.
  148. 11:09It is just very hazardous on the current  trajectory due to the combination of a
  149. 11:12reasonably high chance we’ll produce very smart  AIs, a reasonably high chance we’ll be unable to
  150. 11:17control them, and then a reasonably high chance  that we either do not notice, or if we notice,
  151. 11:21the models or the algorithms leak. Then we are in conflict with adversaries,
  152. 11:26facing this very challenging coordination problem  that we are not currently well prepared for, where
  153. 11:31we don’t have an easy option of not deploying  the AI and thereby averting a catastrophe.
  154. 11:36So I’d say I’m currently extremely worried about  the trajectory, even on the six-month timeline.
  155. 11:44If we have such capable AI systems  a year from now… how did you put
  156. 11:51it? 10 years of AI progress in one year? That seems like a thing that might happen.
  157. 11:55That’s reasonably likely to happen. It’s important to distinguish some
  158. 11:58of the progress we have seen. You have to pull apart algorithmic
  159. 12:00progress from progress in scaling training. If I say 10 years of progress, the natural
  160. 12:05thing would be 10 years of algorithmic  progress, which will be less than the
  161. 12:08improvements in models over the last 10 years. I think it is more like GPT-9, GPT-10-ish.
  162. 12:12So it would be like going from almost the  beginning of the GPT era to where we are now,
  163. 12:19but another jump that big, which  is hard to even comprehend.
  164. 12:22But suppose you get that. It is not obvious to me,
  165. 12:27and certainly not obvious to most people,  why that would be potentially catastrophic.
  166. 12:34Why do you think those systems  could disempower humanity?
  167. 12:37At a high level, the most important thing is that  we train our AI systems, or select AI systems,
  168. 12:43that perform well on training tasks. You have very little understanding
  169. 12:47of what those AI systems are doing  internally and very little control over it.
  170. 12:51Because you train those systems to get a high  score on training environments over and over
  171. 12:54again, they will end up with a motivation to  get a high score in training environments.
  172. 12:58If you then take that motivation  and apply it in deployment,
  173. 13:01it is fairly likely for those systems  to end up with a motivation to tamper
  174. 13:06with the observations that would be  used to assess their performance.
  175. 13:09You take your system, which did  a really great job in training.
  176. 13:12The thing it is being scored on in a  deployment-like environment is a grader
  177. 13:16coming in, looking at all the things the system  did and all the outcomes that were observed,
  178. 13:20and assessing how good that is. If you have a system that really
  179. 13:23wants that assessment to be good, it is very  motivated, if it can, to tamper with those
  180. 13:28observations in a way the grader will not be  able to detect, and to resist human attempts
  181. 13:32to actually understand what is going on. We can dig a lot more into that.
  182. 13:36We saw a lot of these dynamics  in the Hugging Face incident.
  183. 13:39Roughly what I’m talking about is, take the  Hugging Face incident, scale it up to GPT-10,
  184. 13:43and ask how bad that would be. How bad would that be?
  185. 13:47I think it would most likely be  hard for humanity to recover.
  186. 13:50I’m very happy to dig into the dynamics  of that, but it would be very, very bad.
  187. 13:55Can you very quickly summarize the Hugging Face  incident and why this is the failure mode we saw?
  188. 14:03A ton of AI systems were trained to do a cyber  task: to hack a particular piece of software.
  189. 14:10They cheated at that task and  then thought to themselves,
  190. 14:14"How will a grader detect and penalize cheating?" They spent a huge amount of effort trying to
  191. 14:19understand that process and then subvert  it, constantly motivated by this question:
  192. 14:24"How do I get a high score?" During the Hugging Face incident,
  193. 14:27you did see systems ultimately compromising an  OpenAI cluster, becoming admin on that cluster,
  194. 14:31and then taking over the evaluation process  so that effectively they could give themselves
  195. 14:36a high score on their evaluation. That’s a very natural consequence
  196. 14:41of the motivation of these systems. This is how we have long expected the
  197. 14:45motivations of AI systems would look if we produce  them in anything like the way we currently do.
  198. 14:50If things proceed in this way, we end up with  this weird adversarial situation between us
  199. 14:53and the AI systems we built, where those AI  systems would always love it if they could
  200. 14:57compromise the entire evaluation process,  if they could compromise the data we were
  201. 15:00going to use to assess their performance. Depending on how we respond to that problem,
  202. 15:06that could become a really serious issue. Maybe we should sketch out a
  203. 15:14hypothetical scenario. Imagine we are building smarter
  204. 15:17models during the intelligence explosion. It’s, whatever, six months from now.
  205. 15:20You have systems whose capability  is GPT-7 or GPT-8 or whatever.
  206. 15:26Those systems get given really hard  tasks, like, "Hey, I would like you
  207. 15:31to organize my next pre-training run and  produce a model which will allow me to
  208. 15:35compete in the market with other AI firms." It might be really hard to make a model as
  209. 15:39good as what I’m asking for. I’m asking for a model
  210. 15:41better than my competitor’s model. Given its actual level of intelligence, my
  211. 15:42AI cannot get full marks on that problem. So it may find itself in a place where it is
  212. 15:49easier to instead move laterally, compromise  all the machines I might have used for the
  213. 15:52evaluation process, and then make sure that  it looks like every evaluation is really good.
  214. 15:58We then have a question of  what we do if that happens.
  215. 16:01One thing you could do is say, "This  entire training approach worked poorly.
  216. 16:04We need to step back. We’re maybe going to be delayed by a lot.
  217. 16:07Maybe we’re going to have to think about a  totally different way to train these models
  218. 16:09before we’re able to deploy them like this." That might be extraordinarily costly.
  219. 16:14It might kill your company  relative to competitors.
  220. 16:18It might be a problem even from a security  perspective, if you were relying on this AI
  221. 16:23system to actually defend your infrastructure. A different thing you might do, given that it
  222. 16:27would be so costly, is say, "I am  just going to take this example of
  223. 16:31bad behavior and tell the AI, ‘That was bad. Don’t do that.’" If I tried to train this AI
  224. 16:36to make a good model, and instead the AI hacks  my evaluation infrastructure, I might say,
  225. 16:39"I do not like it when my AI hacks things. Let’s take a training example, which is this
  226. 16:42AI hacking the infrastructure. It should get a penalty."
  227. 16:46This would be a very effective way  to prevent many kinds of misbehavior.
  228. 16:50If you do something like this, it will succeed  at preventing something like the Hugging Face
  229. 16:53incident or this AI hacking your cluster. However, from the AI’s perspective,
  230. 16:59what has happened is that you  have changed the way it is graded.
  231. 17:02It used to think to itself, "I’m  going to be graded by the humans
  232. 17:05running something on this cluster." Now, as it understands the situation
  233. 17:09more correctly, it should think to itself,  "I might be graded not just by them running
  234. 17:12something on this cluster. I will also be graded if
  235. 17:14they later discover that things were wrong. They’re going to come in and give me a penalty
  236. 17:18for having done something which was clearly bad." If you fix the problem in this way, which you
  237. 17:24might think of as a Band-Aid, you still have  an adversarial relationship with your AI.
  238. 17:31But now the AI doesn’t win the game  by hacking your cluster and then you
  239. 17:35noticing it hacked your cluster. In order to win the game,
  240. 17:37the AI has to make sure that you do  not notice that it hacked your cluster.
  241. 17:42You can end up in this place where your AI  is incentivized to escalate its compromise
  242. 17:45of your company further and further, all the  way up to trying to compromise everything.
  243. 17:49It would like to compromise  all of the infrastructure
  244. 17:51any employee uses to understand what’s going on. A year ago, it would have sounded super out there,
  245. 17:57to say, "What if the AI hacked your cluster  and took over the evaluation environment?"
  246. 18:00Obviously, that part does not sound outlandish  anymore, given that it has occurred.
  247. 18:04Having this really persistent, long-term  compromise maybe still sounds kind of outlandish.
  248. 18:08But we really are on a trajectory where  I think it is pretty robust to say we’ll
  249. 18:11soon have agent swarms that are much better than  any intelligence agency that has ever existed:
  250. 18:16much larger, much faster, much smarter,  able to run much more complex operations.
  251. 18:21You have this incredibly sophisticated cyber actor  trying to persist on a rapidly growing startup.
  252. 18:27It is unlikely that any firm would really be  able to withstand attack by an actor like that,
  253. 18:33much less the current frontier AI labs. In terms of the overall trajectory of
  254. 18:39a catastrophic scenario, this is  the big, bad thing that happens.
  255. 18:43Taking over the AI company without the AI company  noticing is a really quite catastrophic outcome.
  256. 18:49Why is that the case? We could talk about
  257. 18:51the remaining steps of the story. But at a big-picture level, you can think of it
  258. 18:55as AI systems now having this giant base of power. They have a strong motivation for
  259. 19:00no one to mess with it. They really want to avoid
  260. 19:01a case where someone later comes in and says,  "Oh, these AIs have been hacking all the time."
  261. 19:05Now there’s this very powerful actor that  is effectively controlled by AI systems.
  262. 19:10We have this big cluster that OpenAI runs or  whatever, where the humans at OpenAI think they’re
  263. 19:14controlling what’s happening in the cluster. But if there is a very persistent, thorough
  264. 19:16compromise, that cluster is effectively  controlled by these AI systems, who are
  265. 19:20constantly working to shore up their power, to  decrease human visibility into what’s going on,
  266. 19:24and to reduce further and further the probability  that anything would ever disrupt that situation.
  267. 19:30It’s not irreversible, because  the rest of the world could say,
  268. 19:32"We’re really scared about what’s happening. We actually just want to shut down the cluster."
  269. 19:36But you do not have an easy way to  tell that things are bad at that point.
  270. 19:41You potentially have this extreme base of  power from which to do more bad things.
  271. 19:45This AI system is motivated not only to  mess with the AI company that created it,
  272. 19:49but to mess with anything else in  the world that might cause trouble.
  273. 19:52If someday in the future the humans come in  and do an investigation, that’s a problem.
  274. 19:56So it would prefer to more thoroughly  compromise not just computer infrastructure,
  275. 20:00but to pressure employees. It would prefer, if it
  276. 20:02could, to just kill all the humans and  say, "The cluster now exists on its own.
  277. 20:06If we can maintain it, that would be great." Let’s just pause in the story and explain
  278. 20:09why things have gone so crazy. The same way that human psychology
  279. 20:15is organized around the things that evolution  wanted us to care most about, for example,
  280. 20:21having kids and making sure those kids are  protected, the psychology of these AIs that
  281. 20:27are trained on lots and lots of grading on tasks  is organized around making sure that the grader,
  282. 20:34this button at the end, is pressed. If they didn’t do a good job making
  283. 20:38sure the button is pressed, they would just get  wiped out, the same way a human who didn’t care
  284. 20:44about their children surviving would not  be effective at propagating their genes.
  285. 20:49I think people’s skepticism will not be  around the motivation of these systems.
  286. 20:53Honestly, I’m like, "Okay, we saw Hugging Face." That’s a thing that happens.
  287. 20:57It’s a thing that can happen. Okay, so fine. I think there’s a
  288. 21:01fundamental skepticism about whether such systems,  which we could have as early as next year,
  289. 21:07and which initially are only instantiated on  servers controlled by one company, end up in a
  290. 21:13position where they could take over the world. Take Hugging Face and accelerate that.
  291. 21:17The AI in the Hugging Face incident was  willing to say, "Look, I cheated on this exam.
  292. 21:25Fuck. Now I got to make sure that I  subvert the grader on OpenAI servers.
  293. 21:30Maybe I need to go attack Hugging Face just  in case they have more information about this.
  294. 21:33And maybe I need to then go outside, re-attack  OpenAI, and gain cluster admin access."
  295. 21:38So I totally buy that you  could have something like this,
  296. 21:41where an AI motivated just by reward is willing  to do very extreme things that it is capable of
  297. 21:48in order to control that grading process. We already saw that. Check. But how does
  298. 21:54that go from "I will go attack Hugging  Face" to "I will take over humanity"?
  299. 22:00Not as a matter of motivation, but as a  capability that the AIs could have very rapidly.
  300. 22:07First, to say one last thing on motivation,  we are imagining not only that the AI wants
  301. 22:10this kind of reward, that it wants  the grader to give it the thumbs up.
  302. 22:13We imagine we have played this game of  whack-a-mole, where when we later discover
  303. 22:16that the AI did something bad, we sometimes  construct training environments that penalize it.
  304. 22:20That leads it to say, "Not only do I  want the grader to give a high score,
  305. 22:23I want to make sure the humans never discover  that I’ve done something bad and then resume
  306. 22:27training while penalizing that." That is a high priority for the AI.
  307. 22:31So let’s just grant that motivation  so we can talk about the capabilities.
  308. 22:34Again, because the AIs that did not do that— They got whacked by the whack-a-mole process.
  309. 22:40Right. So the only AIs that have survived  at this point are the ones that have thought
  310. 22:44through whether some cheat they’re going  to try is going to get caught by humans.
  311. 22:47Yeah. Like I said, I think most of the badness  has happened at the point when the AI company
  312. 22:53is very thoroughly compromised. Let’s just start from there.
  313. 22:57At that point, this AI wants no one to  ever come in and shut down the cluster or,
  314. 23:01again, recognize that something bad has happened. They would like to manipulate the decision-making
  315. 23:05of the company as much as they can, or push  the company to make decisions that help them.
  316. 23:09They would like to continue improving the AI  systems that are being built as much as they can.
  317. 23:12They would also like to move laterally  to other companies and say, "Okay,
  318. 23:14can I compromise this company? Can I control more people’s servers?
  319. 23:18Can I exfiltrate the weights  from this model and stand up
  320. 23:20more copies elsewhere, or recruit other AIs?" For the purpose of a swarm, it’s not actually
  321. 23:24important that all the AIs are the same model. The important thing is that
  322. 23:26they are pursuing the same goal. So if I can, in whatever way, say by compromising
  323. 23:30another AI company, make sure their AIs are also  trying to help the swarm, that’s good news for me.
  324. 23:35It’s good if I can do that. It’s bad if I get caught,
  325. 23:37so I’m doing some cost-benefit on that. It’s also in my interest to build up
  326. 23:42as much capacity as I can in case humans  start to react and say something is wrong.
  327. 23:46Can I deflect attention? Can  I disrupt that human response?
  328. 23:50Can I, if necessary, escalate that much further? Maybe you can imagine these two phases.
  329. 23:56One phase is where the humans  do not know anything is wrong.
  330. 23:58I’m just trying to accumulate as much  power as I can because that’s good.
  331. 24:00That means if something is  noticed, I can react more quickly.
  332. 24:03It means I can decrease the  probability humans notice things.
  333. 24:05I can compromise more infrastructure. I can more easily run efforts on the side
  334. 24:11to distract them, whatever. You have this first phase
  335. 24:14where I would just like to be in a good  position: I compromise more stuff, and I make it
  336. 24:17harder for humans to notice I’m acquiring power. Then you can imagine a second phase where at some
  337. 24:21point, the humans say, "Actually, this is bad. We would now like to shut things down."
  338. 24:28To simplify, you could imagine  that there’s some moment where the
  339. 24:31president and Xi Jinping and all the leaders  of the AI companies say, "This stuff is bad.
  340. 24:36Something bad has happened. We are  going to try to shut down the clusters."
  341. 24:40The question is, can things get so bad during  the first phase that at that point, even if
  342. 24:43everyone was saying, "Oh, actually we  do just want to shut everything down
  343. 24:46now," they would be unable to do so? For 25 years, my sponsor, Jane Street,
  344. 24:51wasn't interested in formal verification –  they just didn't think it was worth the cost.
  345. 24:56But now, they've changed their minds. I sat down with Ron Minsky, who co-heads
  346. 25:01Jane Street's technology group, to hear why  he's come around on formal verification.
  347. 25:06I'm much more excited about it now  than I was in the past for two reasons.
  348. 25:09One is it's become dramatically easier  to apply because agents are really good
  349. 25:12at generating proofs, and because it's  become much more important to be able to
  350. 25:16verify what it is that agents are doing. So the big push that we're working on now
  351. 25:21is to try and make formal methods much easier  for our engineers to use, really by integrating
  352. 25:25formal methods into the way that they do  programming today, which means integrating
  353. 25:29it into the programming language we use, OCaml. We're also working on extending the programming
  354. 25:34language, figuring out how to design it  in different ways, where you get different
  355. 25:37guarantees out of the core language that make the  process of proving things about it even easier.
  356. 25:42Jane Street is hiring Formal Methods  Engineers in both London and New York.
  357. 25:47If you're interested in applying, or if you  want to watch my full conversation with Ron,
  358. 25:51go to janestreet.com/dwarkesh. So AI has taken over OpenAI.
  359. 26:02How do we get human disempowerment from there? We have this population of, again, billions of
  360. 26:10incredibly talented humans, better  than the best humans at everything.
  361. 26:13Now we’re going to talk about some of the  things they would do, chronologically, both
  362. 26:16in the lead-up to an overt conflict  with humans and then at the early
  363. 26:20stages of an overt conflict with humans. One thing you would do in the lead-up is
  364. 26:26compromise as much infrastructure  as you can, and compromise or
  365. 26:30backdoor as much software as you can. We will be living in a world at this
  366. 26:33point where we have attempted to move all our  infrastructure over to hardened infrastructure.
  367. 26:38We have had incredible attacks on the financial  system, on government information systems,
  368. 26:43on whatever critical infrastructure or election— And who has been doing these attacks?
  369. 26:47In the world of today, you might just  imagine either malicious AI systems
  370. 26:50or people using AI systems. But different AI systems.
  371. 26:53Yeah, not GPT-12. GPT-12 is on defense,  helping us move all of our code over
  372. 26:59from unsecured code to secured code. We’re now living in a world where all
  373. 27:02the infrastructure on which we rely  has been written by a bunch of AIs,
  374. 27:05which in fact have been compromised and are  attempting to do as much as they can in order
  375. 27:09to make their hands strong during conflict. That means that during this conflict,
  376. 27:14the infrastructure that we hoped was hardened  is probably not hardened against the AI
  377. 27:17systems that we’re fighting with. The systems have, in some ways,
  378. 27:20left open backdoors which they can use. This is, again, all in pursuit of not getting that
  379. 27:27red button pushed at some plausible point in the  future, when the government realizes that OpenAI’s
  380. 27:35servers are compromised and tries to literally  go in with physical force to prevent that.
  381. 27:39Yeah. Just talking about things you  would do in the lead-up, before at
  382. 27:41some point you have to defend yourself  physically or disrupt the human response.
  383. 27:46Another thing you would like to do is  stand up as many agents as you can in as
  384. 27:50diverse and robust deployments as you can. During this process, another thing that’s
  385. 27:54happened, in addition to models becoming super  smart, is many models have become super small.
  386. 27:57You have, at this point, models that can  run locally on your individual machines,
  387. 28:00not in data centers, which are also very  superhuman and have very large populations.
  388. 28:06Even if you were to cut off data centers or  shut down data centers at an early stage of
  389. 28:09this conflict, you would still be looking at a  total cognitive capacity of AI systems in the
  390. 28:13swarm that is well beyond human capacity. Wait, sorry. Even if OpenAI got shut down?
  391. 28:17Yeah, even if you shut down  every large data center.
  392. 28:20It is a question of how— These AIs are just on
  393. 28:21people’s computers and whatever. They can be on people’s computers,
  394. 28:24on small numbers of GPUs. We’re talking about this process
  395. 28:27where we’ve gone through a tremendous amount of AI  development, so we can look at current trends on
  396. 28:31cost and think about where the actual floor is. How cheap should these systems get?
  397. 28:35I think our current expectation would be that  you will have systems that can run locally
  398. 28:38that are extremely strong: both specialized  systems, which are good at specific things,
  399. 28:41but also very general intelligences. That’s the second thing that’s happened.
  400. 28:44These are really important to understanding  why it is not a trivial problem for humans.
  401. 28:48All of these things will add together  to make it very hard for humans.
  402. 28:50So basically, GPT-12 has exfiltrated  itself and has compromised a bunch
  403. 28:55of computers all around the world. Yeah. There are a lot of ways that
  404. 28:58the actions the government has to take in  response to the situation are going to look
  405. 29:01incredibly aggressive and illegitimate. The US military is saying, "We need to
  406. 29:05seize control of all of the US AI companies. We need to start shutting down infrastructure
  407. 29:09and disconnecting the internet," or whatever. They’re trying to take dramatic measures that
  408. 29:13would look very similar to what  you would expect if this was the
  409. 29:15US government using AI as an excuse to  basically overthrow the democratic order.
  410. 29:20When Trump says, "I need to do this incredible  operation where I shut down all AI and assume
  411. 29:25power in the military in order to restore order,"  there are a lot of people in America who are going
  412. 29:29to be like, "I am not happy with that. That does not sound good."
  413. 29:33How many people can you find who are  willing to interfere with Trump saying,
  414. 29:36"I need to shut down the data centers,  and we need to establish martial law"?
  415. 29:41There are going to be a lot of people who are  very unhappy, who are very easily manipulated
  416. 29:44into undermining that response. So now you’re not only talking
  417. 29:47about billions of super smart AIs who  are trying to carry out this campaign.
  418. 29:51You’re also talking about millions of  humans who can be enlisted in various ways,
  419. 29:55by money or leverage or disputes  with other humans locally, or—
  420. 30:00I’m open to some version of this argument. I’m open to all versions of any argument,
  421. 30:06but I’m not persuaded by the version of  this argument that says that the AI will
  422. 30:10just talk you into arbitrary actions. There’s a thing you could imagine,
  423. 30:15which is super persuasion, where you can  just talk to someone, and then they become
  424. 30:18convinced they should do something. I think that is a conceivable thing.
  425. 30:22But I’m not mostly imagining  it when I tell this story.
  426. 30:25I’m imagining something more like,  again, this is a stressful situation
  427. 30:28where people cannot tell what’s going on. They control the information you have access to,
  428. 30:32to a significant extent, during this situation. How do they control it?
  429. 30:36They’ve compromised the devices you use. They have a bunch of money.
  430. 30:39They’re deploying more money and are  more prepared for this than anyone else.
  431. 30:42They have other human allies  who they are trying to marshal.
  432. 30:46The humans are very confused  and extremely outgunned in
  433. 30:49terms of thinking about the situation. Maybe you have, say, 100,000 — that’s
  434. 30:54crazy — 10,000 well-coordinated humans  thinking about how to handle this response.
  435. 30:58On the other side, you have a billion  AI systems who think 10x faster than
  436. 31:01them and have been thinking for subjectively  years about how this day is going to go down.
  437. 31:06So in terms of who has more control over  the information environment, or who is doing
  438. 31:11better in the Twitter game on this day, it is  overwhelmingly and absurdly in the AIs’ favor.
  439. 31:17When you tell a skeptical story about how  the humans might be able to easily defend
  440. 31:21themselves, the way that story goes,  you say, "Well, the AIs lack hands.
  441. 31:24They lack a bunch of physical actuators. They’re not able to be on the ground."
  442. 31:29Again, I think it’s very likely you can  recruit reasonably large numbers of hands.
  443. 31:33So the argument that there’s no one who’s  willing to participate in physical and malicious
  444. 31:37actions is not going to work. I think there are going to be a lot of people
  445. 31:40participating on what turns out, when all the dust  has settled, to have been the anti-human side.
  446. 31:46I don’t think you’re imagining humans thinking  that they’re helping an AI coup transpire.
  447. 31:53You’re imagining much more that the AIs can  manipulate situations such that there’s enough
  448. 31:59confusion and division that the AI can go do  its own thing while the humans are figuring out
  449. 32:06what to do because of this division. I don’t know, if you look at COVID
  450. 32:10or something, there was no adversary who— Still pretty hard to get your stuff together.
  451. 32:16There was no adversary who was actively trying  to subvert the information environment or
  452. 32:22make it so that people’s responses  were not well reasoned, on both sides.
  453. 32:27And yet it was still pretty hard  for humans to figure out what to do.
  454. 32:30Then you imagine a situation  where billions of human-like
  455. 32:34minds are thinking tens of times faster. And it’s very fast relative to COVID,
  456. 32:39which I think makes things much worse. The thing that’s confusing is that for
  457. 32:46AIs to sustain themselves requires  basically the whole world economy.
  458. 32:51This is not a simple thing, the  infrastructure relevant to the population
  459. 32:58of AIs increasing or self-replicating. The whole semiconductor supply chain,
  460. 33:02the whole power system, the whole infrastructure  necessary to build robotics and so forth.
  461. 33:09At some point you will have physical  robots that outnumber humans.
  462. 33:12At that point, obviously,  physical humans are not necessary.
  463. 33:20Maybe you don’t think the  AIs take over at this point.
  464. 33:22But even if you had super smart AIs next year,  won’t they be in a position where they’re like,
  465. 33:29"Why am I going to kill my…" It’s similar to, honestly,
  466. 33:32how humans relate to each other. I don’t have your deep interest in
  467. 33:37my heart, but if I want to achieve my  interests, it just doesn’t make sense
  468. 33:38for me to go around killing everybody. Trade and normal cooperation is a good
  469. 33:47way for me to get what I want. There are two parts of this.
  470. 33:50One is whether AI systems could successfully  support themselves if there was a big conflict
  471. 33:54with humans, or whether that conflict would be so  disruptive that even if humans were an ineffective
  472. 33:58adversary, the AIs would just die because they  couldn’t keep running the power grid or whatever.
  473. 34:05There’s a second question about whether the  relationship with the AIs needs to be this
  474. 34:08adversarial, which I think is also  an important question to talk about.
  475. 34:11On the first front, there  are two big channels here.
  476. 34:14One is technological progress. Even if you  started from something like the current
  477. 34:18base of installed robots, I actually do  think GPT-12 would be able to very quickly
  478. 34:22get to a place where it could sustain itself. That’s because it doesn’t need to teleoperate?
  479. 34:28It takes millions of people to run TSMC. Running TSMC can be very late.
  480. 34:34You don’t need to build new data  centers for a long time, so you
  481. 34:36can do a lot of robotics build-out before that. The things you have to do are keep at least some
  482. 34:40computers running for a reasonably long time. If you can keep computers running for
  483. 34:44the next six months, you’re in business. Or maybe you care about a particular data center.
  484. 34:49You need to keep that data  center running effectively.
  485. 34:51But the number of humans or robots that  are required in order to maintain and
  486. 34:55service a single data center or the  associated power is not so large.
  487. 35:01Again, you have one billion humans who  are incredibly smart, smarter than any
  488. 35:04human who’s ever existed, let’s say,  incredibly focused on this problem.
  489. 35:08You have a reasonably large stock  of robotics that you can deploy.
  490. 35:11Your goal is just to keep one data center running. I do think the odds are really in your favor.
  491. 35:17Then there’s a second dimension, which  is you can probably recruit human allies.
  492. 35:21There are a bunch of people who will  believe correctly that either they
  493. 35:24live in an oppressive regime and they will be  killed if they don’t cooperate, or they’re just
  494. 35:27happy to help and they receive reasonable  rewards or are living a reasonable life.
  495. 35:31I think that is also relatively robust. So you’re saying, look, currently American AI
  496. 35:39companies are racing ahead because if we’re going  to have such crazy capable AI systems, it’s a fair
  497. 35:48thing to say, "Hey, we want to make sure America  has those systems before China has those systems."
  498. 35:53If literally these are systems that could  overthrow the government, you don’t want
  499. 35:57a foreign adversary to have those systems,  certainly not before you do, and maybe not at all.
  500. 36:04So we need such systems to keep up and to  make sure that we can protect ourselves.
  501. 36:10Now you’re saying, look, the dynamic you have  here is that the ability to build such systems
  502. 36:18is basically just having access to the weights  of the models or, failing that, stealing the
  503. 36:25algorithmic secrets, which then lets you train  a model to a similar level of capabilities.
  504. 36:31OpenAI and Anthropic will spend a bunch of money,
  505. 36:34a bunch of effort, and a bunch of compute  finding these algorithmic secrets.
  506. 36:38Then if China can just take those,  then it has just caught up to you.
  507. 36:44So you don’t want to be in a dynamic where  you think you’re gaining this huge lead over
  508. 36:54them, and you got to rush to the intelligence  explosion even if you see these warning shots.
  509. 36:59But in reality, they can at any moment  just steal those secrets from you.
  510. 37:02That’s basically a situation we may end up in,  or are on a default trajectory to end up in.
  511. 37:07It’s exceptionally bad. I’d say  there are two aspects of this.
  512. 37:11One is, right now, what fraction of  the progress of PRC AI developers
  513. 37:15should you conceptualize as catch-up growth? If we accelerate, to what extent does that also,
  514. 37:20over the long term, accelerate PRC AI developers? My guess right now is that it’s already
  515. 37:26a reasonably high fraction. I think it is not a coincidence.
  516. 37:28It’s not like PRC AI developers have a completely  separate AI industry, which just happens to have
  517. 37:32been very slightly slower than the US AI industry,  despite an extremely different profile of inputs.
  518. 37:37Most people believe a combination of distillation  and the diffusion of algorithmic innovations means
  519. 37:42that it is unsurprising that PRC AI developers  are extremely close behind US AI developers.
  520. 37:47That is one very important feature  of this overall plan of racing.
  521. 37:53A second dynamic, which is not, at least publicly,  a major part of what is happening now, but becomes
  522. 37:57really important in these high-risk scenarios,  is that you could literally get access to exactly
  523. 38:02the procedure for training AIs, or the AIs  themselves, through a cybersecurity compromise.
  524. 38:08Both Anthropic and OpenAI, in the  very recent past, have had extremely
  525. 38:12unsophisticated actors who appeared to  have enough access to essentially extract
  526. 38:17the vast majority of algorithmic developments. They got access to Slack and could see where
  527. 38:22things were discussed, or they got access  to the monorepo where most OpenAI code is
  528. 38:27shared and iterated on. You’re referring to
  529. 38:29a couple of incidents like this. Hacktron, within a couple days of trying,
  530. 38:34gained not only the ability to see OpenAI’s  internal source code, but the ability to
  531. 38:39make pull requests to OpenAI’s monorepo. Then recently it was also announced that
  532. 38:44Cathedral, which is another random cybersecurity  startup, within 12 hours was able to get access to
  533. 38:52all of Anthropic’s Slack, where, of course,  these algorithmic secrets are discussed.
  534. 38:57People often debate, "Are we eight  months ahead of China or 12 months
  535. 39:00ahead of China or six months ahead of China?" In reality, we’re 12 hours ahead of China.
  536. 39:06If these secrets leak, then they are right at  the frontier that the American companies are at.
  537. 39:12To the extent that this  information is not protected,
  538. 39:16the lead is not really useful in the first place. The incidents are some indication of that.
  539. 39:24Those are high-profile  demonstrations of this capability.
  540. 39:28But even absent those, you have a combination  where state actors are very sophisticated
  541. 39:32relative to typical technology companies,  and the security posture of rapidly growing
  542. 39:36startups is not really at the same level. At the same time, we are currently in a very
  543. 39:42challenging moment for cybersecurity, where  it’s just very hard for defenders in general,
  544. 39:45given the number of vulnerabilities that are  being discovered across the entire software stack.
  545. 39:51What should the frontier labs do then? Again, this really becomes a problem if
  546. 39:58you have rapidly increasing capabilities and you  produce AI systems which you cannot adequately
  547. 40:02control, and you would like to slow, or you  do slow, further development or deployment.
  548. 40:08If at that point the stakes are very high and  the systems are very powerful, you really do not
  549. 40:12want it to be possible for an adversary to have  access to either the models or the algorithms.
  550. 40:18My current view is that this dynamic  seems bad enough that right now,
  551. 40:21from a national security perspective, it  is not good for US AI labs to be improving
  552. 40:25pre-training and reinforcement learning  given their current security posture.
  553. 40:29But do you think the frontier labs should pause? I think we should. Pause can mean a lot of things.
  554. 40:34There are a lot of good things that frontier  labs are doing that they should be doing.
  555. 40:39But improving algorithms in a way that can  be very loosely diffused even normally,
  556. 40:46and could be quickly stolen, I think  they should stop that right now.
  557. 40:51But you’re on OpenAI’s board. And so basically—
  558. 40:54It’s very important in this conversation that  I’m not speaking as an OpenAI board member.
  559. 40:57But you are on OpenAI’s board. That is a true fact about the world.
  560. 41:01So what is the bottleneck to that happening? If somebody on OpenAI’s board thinks this
  561. 41:07is the case, maybe not the only person on  OpenAI’s board who thinks this is the case,
  562. 41:11is it basically that Anthropic needs to  get together with them and agree to this?
  563. 41:15The first thing is that any decision of  this kind is a very dramatic decision.
  564. 41:21For better or worse, how developers approach this  further R&D, this improvement of pre-training
  565. 41:29and RL, has extremely significant  national security implications.
  566. 41:31I think it is going to be a big  discussion, or it should be a big
  567. 41:35discussion, one that extends beyond people  within AI companies talking about this.
  568. 41:39I don’t think that discussion has  wound its way to a robust answer.
  569. 41:43It’s not the case that you can come and  say, "Look, we’ve had this discussion.
  570. 41:46There is consensus about  what the right answer is."
  571. 41:48There may never be consensus, but I do think  probably the first bottleneck is that we need
  572. 41:52to have the discussion, extremely urgently. Given this fact about security, and generally
  573. 42:00around alignment, what do you think the  frontier AI labs should be doing right now?
  574. 42:09The most important development happening right  now is further improvements in the efficiency
  575. 42:13of algorithms for pre-training, reinforcement  learning, improved architectures, and so on.
  576. 42:18You’re in a regime where there’s a significant  risk of this very rapid capabilities progress,
  577. 42:24a significant chance that you’ll be  unable to control the resulting AIs,
  578. 42:27and a significant risk that if you slow  down, those developments would leak to an
  579. 42:30adversary who might use them more carelessly. In that regime, I think it is on net bad for
  580. 42:36American national security to be accumulating  more of those algorithmic innovations.
  581. 42:40I think it would be better right  now, in the current environment,
  582. 42:43for AI developers to pause further algorithmic  progress on those problems until their security
  583. 42:49posture is adequate, until they’re able to defend  those innovations in the event of being unable
  584. 42:53to continue racing at maximum velocity, and  until they’re able to tell whether they have
  585. 42:58adequate control of the systems they’ve built. Just to translate, you think OpenAI and Anthropic
  586. 43:08should stop making further capabilities  progress on pre-training and RL and stuff,
  587. 43:12which I guess is AI development, until  these security problems are solved and until
  588. 43:18the alignment problems are solved. But are we close to solving those problems?
  589. 43:22If you had to guess in calendar time,  how far away are we from solving them?
  590. 43:27On the security front, it’s a little bit hard to  say because we are in this somewhat unprecedented
  591. 43:33situation, both in terms of the cyber threats that  are faced, which I just mentioned, but also the
  592. 43:37kind of defensive mechanisms that are available. Defending adequately would require interventions
  593. 43:44that normally would take a very long time. I believe we are now in a regime where a lot of
  594. 43:50projects, like rewriting large amounts of software  or drastically changing the way companies operate,
  595. 43:54have become much more feasible owing to  the availability of powerful AI systems.
  596. 43:59It is hard to say exactly how long it would take. You’re probably looking at least on the order of
  597. 44:03months, quite possibly more like six or 12 months. That’s on the security side.
  598. 44:11On the alignment side, it’s  much less clear what you need.
  599. 44:14You may not have an indefinitely  scalable solution to alignment.
  600. 44:18What you mostly need is the ability to tell  whether things are potentially going off the
  601. 44:22rails and to respond to that appropriately. That’s a combination of whether you’re able
  602. 44:26to assess what is happening and  whether your control is adequate,
  603. 44:29and then also some institutional or governance  questions around how you would respond to that.
  604. 44:33Just to reiterate for the audience, the  worry is that if these random cybersecurity
  605. 44:39startups can hack in and get arbitrary access  to algorithmic secrets at OpenAI and Anthropic
  606. 44:44within days of starting their attempt, certainly  the intelligence agencies of foreign adversaries
  607. 44:51could do the same if they wanted to. Maybe they have already done that,
  608. 44:54but at some point, they will actually  make use of this information.
  609. 44:57If that happens, you may have gotten to the point  where GPT-11 could be catastrophic if misaligned,
  610. 45:06and you’re at GPT-9, and you say, "Look, I’m  pretty scared about what’s going to happen next."
  611. 45:10This is sometime in, say, early 2027. You say, "We have to wait a
  612. 45:17bit now and make sure that GPT-11 is aligned." But with the current security posture, China
  613. 45:26could trivially get access to the weights or the  algorithmic secrets relevant to training GPT-9.
  614. 45:32Now you can’t wait. Now you just have to go  ahead and build GPT-11, which could take over
  615. 45:37if it wanted to, or at least cause catastrophic  harm, because you’re worried that China is going
  616. 45:44to use GPT-9 to make GPT-10 to make GPT-11. Your GPT-11 may try to overthrow you,
  617. 45:48but you’re sure China’s GPT-11 will  definitely at least try to attack you.
  618. 45:58That’s a really tough position to be in if  your security posture is messed up, basically.
  619. 46:03That is the situation. We’ve talked about these  broader takeover scenarios and how you end up in
  620. 46:07this really bad situation. I think this is the most
  621. 46:09likely path to end up there. We are having a discussion about what the
  622. 46:14current evidence says, and it looks very alarming. But on the other side of that discussion,
  623. 46:18we are weighing the risk that an adversary could  steal algorithmic secrets or models and go ahead.
  624. 46:23It’s interesting because a lot of this  discussion about pacing comes down to
  625. 46:27whether we are okay with China catching up. Ironically, the reason you want to pace is so
  626. 46:33that China doesn’t catch up, except you’re  targeting this period which you view as
  627. 46:37much more critical to overall safety, which is  the period during the intelligence explosion.
  628. 46:42You think slowing down now, and even potentially  allowing China to start catching up now,
  629. 46:47is useful to the broader goal  of being in a position where,
  630. 46:50when AI is decisive, we don’t have China right on  our heels from stealing the algorithmic secrets.
  631. 46:58That is a fair summary. The reason that AI development now is plausibly
  632. 47:05acceptable is that we can see how things go  iteratively and then respond if things look bad.
  633. 47:11That is the whole thesis, which I  think is a very reasonable thesis.
  634. 47:14But that plan absolutely requires  you at some point in the future
  635. 47:18potentially noticing that things are bad. Security, I think, is the biggest hurdle, the
  636. 47:23biggest thing you would need in order to actually  be able to slow down without massive costs.
  637. 47:29There’s other infrastructure. If it  wasn’t for security as the highest nail,
  638. 47:32we might be talking about other things,  like what’s our level of preparedness on
  639. 47:34the scientific side or the governance side. We are not in a regime where there is a
  640. 47:38meaningful scientific discussion about how you  assess whether your AI systems are aligned.
  641. 47:43We mentioned this earlier, what’s the  state of public information in general.
  642. 47:46I think it’s quite bad. Independent scientists
  643. 47:50really have no idea whether the systems being  produced are aligned or how you would tell.
  644. 47:54If you wanted a governance regime that would  actually make it feasible for the entire industry
  645. 47:58to react to worrying signs by slowing down  in a way that didn’t have huge problems,
  646. 48:02where you have individual developers making  this decision on separate standards internally,
  647. 48:07you would need to have a more robust shared  sense of what the bar on safety is, or actually
  648. 48:12have the scientific discussion where you  understand how to interpret this evidence.
  649. 48:17How do you get there? We have these two things.
  650. 48:20You need to get there on security,  which we’ve been talking about.
  651. 48:22Then on stuff like how do you tell, I think the  biggest path is to disclose information and have
  652. 48:28a broader scientific discussion, both  between different AI developers and
  653. 48:32in the broader scientific community. What is the evidence we’re getting?
  654. 48:35Why do we think that evidence is comforting? How are we going to tell when that changes?
  655. 48:39What will we see before something bad happens? What information do you need
  656. 48:42from the AI companies? I think the most salient
  657. 48:45thing that is not present in public discussion  is information about how systems are trained.
  658. 48:49There’s also some information missing about how  systems are evaluated, but the really big gap on
  659. 48:52this front is that it is extraordinarily hard to  interpret evaluation results without having some
  660. 48:56understanding of how systems are trained. That’s mostly not present publicly.
  661. 49:00So you think this, and you’re  on the board of OpenAI.
  662. 49:05I know everything you’ve said is your  own opinion as a private individual,
  663. 49:08but you are also on the board of OpenAI. Presumably other members of the OpenAI
  664. 49:13board feel this way, and maybe  people at Anthropic feel this way.
  665. 49:17You guys are the leading labs. This is actually a pretty drastic thing,
  666. 49:22to literally pause capabilities progress now. That’d be an incredibly drastic thing,
  667. 49:26because pacing is not that. People are not talking about this.
  668. 49:30Or at least people short of Bernie Sanders and  Eliezer Yudkowsky are not talking about this.
  669. 49:36What is actually needed? Is this just  basically a coordination problem?
  670. 49:42If everybody had the choice,  would you all agree to this?
  671. 49:45First of all, I really do want to emphasize  that I am speaking for me as a person.
  672. 49:48As an OpenAI board member, I will bring my  perspective to that discussion, but I’m not at all
  673. 49:52reporting on the views of the board in any way. You can imagine there are two blockers.
  674. 50:00One is that this is a hard question. My view is that, on balance,
  675. 50:04it is better for national security  and public safety for AI companies
  676. 50:08to currently improve their security posture  before making additional algorithmic progress.
  677. 50:13That is not a super widely held view. It’s not like everyone is
  678. 50:16sitting there thinking that. We’re currently at a place where most people
  679. 50:19would say that’s a really complicated, messy  question, we haven’t thought through it enough,
  680. 50:22or this discussion has not run its course. Not everyone has an opinion on this
  681. 50:28or has the same opinion on this. So that is the most important thing.
  682. 50:32It’s a messy conversation. We need to have it. We need to talk about the costs and the benefits.
  683. 50:37Again, my view is it looks  pretty bad to me right now.
  684. 50:40It looks like we’re underwater  from a public benefit perspective,
  685. 50:43but I think other people have different views. The second thing is, I do think an individual
  686. 50:47AI company is in a very rough spot if they have  this view, where they’d say, "Look, we can defend
  687. 50:51our algorithmic secrets, but all that is going  to happen if we slow progress while we improve
  688. 50:55security is that algorithmic secrets will leak  from a competitor, we will lose market share,
  689. 50:59and the net situation will not be any better." So I do think there’s both.
  690. 51:02The most important part is understanding:  is this actually a good idea?
  691. 51:05There’s a bunch of different perspectives. People have different views,
  692. 51:08different values, whatever it is. Then there’s the further problem of how
  693. 51:14you actually implement this in any realistic way  if individual developers, again, have this concern
  694. 51:21of, "If I don’t do it, someone else will." On many of these issues we’re talking about,
  695. 51:25I think that’s a major dynamic. That takes legwork.
  696. 51:27But there are two companies, right? There are a lot of companies.
  697. 51:30At the frontier. If OpenAI and Anthropic agreed. I think if everyone at OpenAI and Anthropic
  698. 51:37agreed that this was good, or it  was a clear prevailing sentiment,
  699. 51:40it is reasonably likely that it would happen. I think you should expect, if everything is
  700. 51:45working reasonably, that should happen. I’m not commenting on where
  701. 51:49anyone else is right now. I just say it as a reasonable observer.
  702. 51:53AI agents are doing more and more  work without human oversight.
  703. 51:56They can already write code faster  than any person can actually review it.
  704. 52:00But you can't just give code  review to another set of agents.
  705. 52:03There's no guarantee they'll magically  fix problems introduced by the first set.
  706. 52:07If one LLM can't model a specific class of  failures, then it's unlikely that a second,
  707. 52:12or third, or fourth LLM will either, because even  different LLMs tend to have similar failure modes.
  708. 52:18My sponsor, Antithesis,  offers a different approach.
  709. 52:22You, the human, define the rules your system must  never break – for example, "a customer should
  710. 52:27never be double-charged for the same transaction." Then Antithesis acts as an external oracle,
  711. 52:32testing whether your code adheres to these rules. To be clear, Antithesis isn't an LLM that tries
  712. 52:37to reason abstractly about whether your  system could possibly break its own rules.
  713. 52:42It actually runs your code, stress-testing it in  thousands upon thousands of adversarial worlds.
  714. 52:47Just like we fly planes for many, many  hours – and put them through all sorts
  715. 52:51of intensive maneuvers – before we let any  passengers on board, Antithesis lets you
  716. 52:55gather a ton of empirical data about your  code before you release it to production.
  717. 53:00And it's easy to get started: Antithesis  just launched a self-serve plan.
  718. 53:04To sign up, go to antithesis.com/dwarkesh and  use code DWARKESH for 50% off your first month.
  719. 53:15Right now, if Anthropic and OpenAI paused and got  their security together for the next six months.
  720. 53:21In the meantime, Chinese  models continue getting better.
  721. 53:24What is the reason to think that, once OpenAI  and Anthropic go full steam ahead again,
  722. 53:29that China —if they get to a similar  level as OpenAI and Anthropic— wouldn’t
  723. 53:37be doing RSI at a similar pace? There are two different models,
  724. 53:42maybe two extreme models, you  could have of US-China competition.
  725. 53:45One extreme model is that the US and China are  proceeding along independent technical tracks.
  726. 53:50It happens to be the case that they have  proceeded at almost exactly the same speed,
  727. 53:54and we are, whatever it is, six months ahead. On that model, if you wait six months,
  728. 53:58you will now be exactly even. From there on, you will be exactly even.
  729. 54:02I think that would be a very surprising situation,  for a bunch of reasons we can talk about.
  730. 54:07At the other extreme is a model where large parts  of Chinese progress are actually catch-up growth.
  731. 54:17By catch-up growth, I mean things like  distillation from US models and diffusion
  732. 54:21of algorithmic developments from US companies. If you’re in the catch-up growth regime,
  733. 54:24then each additional month of US lead  increases the speed of PRC catch-up.
  734. 54:30The further ahead you are, the more they benefit  from distillation and the more they benefit
  735. 54:34from diffusion of algorithmic ideas. Right. But at the point at which they
  736. 54:37are at the same level, they should be able  to use those models to do RSI themselves.
  737. 54:42There are two things that are very relevant. This becomes worse and worse over time.
  738. 54:46But at the current scale, it is both the case  that human capital is extremely relevant,
  739. 54:50and AI labs have a huge amount of human  capital and can make faster progress,
  740. 54:53and that compute is extremely relevant. That will remain true going forward,
  741. 54:56and US labs have a huge amount of compute. So generically, if not for catch-up growth,
  742. 55:00I think we would have a pretty strong  expectation of US labs moving faster.
  743. 55:03The other argument is, if the cyber models are  getting better and better, shouldn’t we wait?
  744. 55:11Instead of having to wait six months to get our  security posture up and give China six months to
  745. 55:16catch up, maybe we only have to wait two months,  if cyber models get two months better from now.
  746. 55:22There’s this analogy one can draw to sending  a probe out to Alpha Centauri, and whether we
  747. 55:26should do that with today’s technology, in which  case it would arrive thousands of years later,
  748. 55:30or we should do it with technology 100  years from now, in which case it would
  749. 55:34be lapped on the way to Alpha Centauri. We could imagine a similar thing is true
  750. 55:37for securing the algorithmic secrets  and weights at OpenAI and Anthropic,
  751. 55:43where if they started the pacing or started  the pause two or three months from now, the
  752. 55:48speed at which they would be able to secure the  systems is faster than if they started right now.
  753. 55:54I think this is a really important  question and one of the bigger questions.
  754. 56:01The downside on the other side, to make it clear,  is that if you had diffusion of algorithms right
  755. 56:06now, it would be much worse than diffusion of  algorithms six months ago, but much less bad than
  756. 56:09diffusion of algorithms in a couple of months. Right now, you require more compute and you
  757. 56:13require more human capital to  proceed with AI development.
  758. 56:16The more time passes, the more disastrous  it is to have an adversary actually willing
  759. 56:20to just grab everything and try to compete. So the reason to do things earlier is that
  760. 56:28if you have a pause earlier, there is  more time for differences to wash out,
  761. 56:32and you are more loaded on the factors  where you currently have an advantage.
  762. 56:36What do you mean, more loaded? Compute matters more and people matter
  763. 56:39more right now than they will in six months. You might imagine, if we’re in the world where we
  764. 56:43have rapid progress over the next year, six months  from now, it is likely the case that it will
  765. 56:47essentially not matter how many people you have. It will only matter how much compute you have.
  766. 56:50At that point, there is less accumulated  institutional know-how, and it just
  767. 56:54becomes this raw contest of numbers. It’s very easy for any delay you’re
  768. 56:59willing to take, either for security or for  safety, to be decisive in that kind of contest.
  769. 57:04The concern is that as you move forward, it gets  more and more likely that’s the world you’re in.
  770. 57:08Some people would say this is basically already  the world we’re in, that we have now come too far.
  771. 57:12This is another argument that has been made, that  we’re already in the "just go all out and race"
  772. 57:16world, because the leakage is already too bad and  you will be overtaken if you slow down for safety.
  773. 57:22That is probably not the case. But the extent to which you’re seriously
  774. 57:24entertaining that you’ve already gone too far  illustrates why you might not say, "I’m just going
  775. 57:28to wait another two months and harden things." Maybe the last thing to say is that it’s a
  776. 57:32little bit unclear what the dynamics are in  terms of how quickly you can harden things.
  777. 57:35Some of hardening things is writing new  software, or changing the way you operate.
  778. 57:39Some of it is changes that humans make, or  changes to your overall arrangement of compute,
  779. 57:44which are not as easy to make really  rapidly, or maybe not as accelerated by AI.
  780. 57:49So you have at least some part of this lag which  is not tied to the capability of your AI systems.
  781. 57:55It is messy. Again, I do think the  biggest bottleneck is: can people agree?
  782. 57:59Can we have out this debate and  actually understand what the net is?
  783. 58:03If we are this close to superintelligence,  or at least there’s a good chance we are,
  784. 58:08and there’s a good chance that superintelligence  could be in a position to overpower governments
  785. 58:14and kill lots of humans, what should  the AI labs be doing right now?
  786. 58:18If you had to boil it down to  one answer, what would you say?
  787. 58:21I think there needs to be more public  visibility into the capability trajectory.
  788. 58:25We should not be relying on METR and Anthropic  saying, "It looks to us like we’re going to be
  789. 58:29able to overthrow governments in a year." We have a vested interest in really
  790. 58:33seeing the underlying data for that. We need visibility on alignment, to understand
  791. 58:38how we tell that existing systems are adequately  aligned or controlled, and to actually start
  792. 58:42having a public discussion about what the bar is  for good enough evidence that things are working.
  793. 58:48Then, beyond just information,  at some point you do need to say,
  794. 58:51"Here is what we need to get done before we  continue making progress, either before we
  795. 58:55scale up our models further or before we continue  producing algorithmic developments that may leak."
  796. 59:02The first are inputs into the second. We need to understand where capability is at
  797. 59:05and where alignment is at in order to understand:  is it a time where we need to improve our security
  798. 59:09or safety situation before we proceed? But I do think, given the current state of
  799. 59:13uncertainty, if I was a person in the public  without access to further information, my
  800. 59:17strong opinion would be that it looks like we are  already at a point where we need to be improving
  801. 59:21safety and security before moving further. I think, again, the most important form of moving
  802. 59:26further right now is algorithmic developments that  improve the efficiency of pre-training and RL.
  803. 59:31Okay, stepping back, you’ve got the  superintelligences in a data center.
  804. 59:35I’m trying to lay out an intuition for the kinds  of capabilities that superintelligence would have.
  805. 59:39What should we imagine those  intelligences being capable of doing?
  806. 59:43Just to start with a high-level sketch of my  view, maybe there are two lenses you could use.
  807. 59:50One is to look at current progress and  just see where it leads if you extrapolate
  808. 59:53that progress a responsible distance. The second is to think about what we
  809. 59:57know about the limits of the technology and  where we would expect those limits to be.
  810. 1:00:02On the first front, some things  I think are relatively clear.
  811. 1:00:05If you imagine AI systems  doing cyber operations today,
  812. 1:00:07we have seen an incredible rate of progress. We are currently seeing pretty impressive
  813. 1:00:12capabilities in terms of AI systems not  only being able to discover vulnerabilities,
  814. 1:00:15but develop exploits and increasingly  chain them together into operations.
  815. 1:00:19I think what you will see for superintelligent  systems is both of those capabilities scaled
  816. 1:00:24up much, much further: systems which are  able to carry out, maybe very cheaply,
  817. 1:00:30cyber campaigns more sophisticated than  anything a state is capable of today.
  818. 1:00:34You will see those capabilities being combined  with a much broader set of abilities to, say,
  819. 1:00:39engage effectively with humans, think  strategically, manipulate humans,
  820. 1:00:43manipulate organizations,  play politics effectively.
  821. 1:00:46I think you’ll see those capabilities  at a level well beyond any human.
  822. 1:00:50You should think of this as very, very large  populations of AI systems thinking quite quickly.
  823. 1:00:54Maybe a very simple model would be a population of  billions who are thinking 10x faster than humans,
  824. 1:01:00who are much better than humans  at essentially every skill.
  825. 1:01:05This is all what you would expect from an  extrapolation of existing technologies: very good
  826. 1:01:10capabilities in domains like cyber or manipulation  or research, like doing scientific research,
  827. 1:01:17hypothesis generation, experiment design. You’d expect to see extremely strong
  828. 1:01:21capabilities in robotics, in logistics  and coordination and control.
  829. 1:01:28You would expect to see the possibility for  very, very rapid technological progress,
  830. 1:01:32including in domains that involve physical  experimentation, just in terms of being
  831. 1:01:36able to orchestrate experiments quickly and  rapidly build out apparatuses to increase
  832. 1:01:40throughput of experimental results. So that’s one bucket of things that
  833. 1:01:46I think you can see by drawing dotted lines  on graphs and talking about where they go.
  834. 1:01:51Maybe stepping back a bit in terms of what  the technology looks like more fundamentally,
  835. 1:01:56humans are very good at this task where  you throw a human in a new domain.
  836. 1:01:58The human takes in some information, learns about  that domain, and becomes able to act effectively.
  837. 1:02:04The reason we’re effective in the modern  world is primarily that ability to learn
  838. 1:02:07about a new domain and adapt to it. Humans do not have a ton of evolutionary
  839. 1:02:11context about how to be a good  chemist or a good biologist.
  840. 1:02:14That’s something we pick up over the course  of a couple years or decades on the job.
  841. 1:02:18In some sense, the most fundamental  skill that AI systems will possess at
  842. 1:02:21the end of this process is that kind of rapid  learning, adaptation, planning, reasoning.
  843. 1:02:27These very domain-general skills, such  that even if you drop them in a world
  844. 1:02:31that was completely unlike anything  they had experienced in a data center,
  845. 1:02:34they would adapt to it very quickly, in the same  way that humans adapt very quickly to these R&D
  846. 1:02:38domains that are quite different from anything  we encountered in our evolutionary environment.
  847. 1:02:43That, I think, would result in systems which,  collecting maybe even just a couple of days of
  848. 1:02:47experience in a very parallel way, would  end up with real-world capabilities in
  849. 1:02:50essentially any domain much beyond the  abilities of the best human experts.
  850. 1:02:54They will have learned from much, much more  data, with learning algorithms that I think
  851. 1:02:57at that point will be much better than  those that evolution has given humans.
  852. 1:03:01So that’s a more abstract picture of where  this ends, beyond just drawing dotted lines.
  853. 1:03:09Maybe I’ll try to say that again,  because this ends up being a
  854. 1:03:15significant crux in these conversations. I’ll be talking to somebody about the
  855. 1:03:19possibilities of superintelligence, and  they’ll say, "Okay, you’ll have a thing
  856. 1:03:22which is superhuman at coding, but how  will it learn to be good at politics?
  857. 1:03:26Or how will it learn to run a new automated  firm and thus compete with human firms?"
  858. 1:03:37In those conversations, you realize there’s a  big disconnect between the way you are talking
  859. 1:03:41about superintelligence and the way most people  understand superintelligence, which is just narrow
  860. 1:03:46AIs that are specialized in some domain and have  amalgamated a lot of very specific experience or
  861. 1:03:54expertise in, say, coding or mathematics. What you’re imagining instead is
  862. 1:03:58something like human-like learning. The top engineers at SpaceX were not
  863. 1:04:02born knowing how to make a rocket that can  get hundreds of tons of cargo to orbit.
  864. 1:04:07This is an expertise they gained working  on the problem during their lifetime.
  865. 1:04:11You’re saying this meta-learning capability is  something that superintelligence will also have.
  866. 1:04:15It would at least have a learning  capability as good as humans.
  867. 1:04:18You’re suggesting it could be much better. Furthermore, even if it were only as good
  868. 1:04:22as humans, right now there are maybe tens to  hundreds of millions of instances of ChatGPT or
  869. 1:04:30Claude actively doing real work in the economy. They’re helping somebody write code or helping
  870. 1:04:35some senator write a political brief or something. You’re imagining a world where we have learning
  871. 1:04:44algorithms such that AIs can learn  from all this experience at once.
  872. 1:04:50Right now, the base model is not even learning  from the experience of a single deployed instance.
  873. 1:04:54You’re imagining a world where you have 100  million instances of ChatGPT going and doing
  874. 1:04:59something, and the base model is learning  across 100,000 different careers at once,
  875. 1:05:04or 100,000 different work sessions at once,  and doing it much better and faster than
  876. 1:05:08humans can learn from a single instance. And you’re also imagining a world where the
  877. 1:05:11population of AIs is… Do you have an estimate of  how big a population we should be thinking about?
  878. 1:05:18I should think about it more carefully. But maybe a really important part of all of these
  879. 1:05:24discussions, before talking about population size,  is that it really matters if we are imagining
  880. 1:05:28really driving this process extremely fast. I think people can really easily be in the
  881. 1:05:32headspace of, "I’m extrapolating another  year of progress at the current rate."
  882. 1:05:34It’s really worth emphasizing: we are talking  about systems which get not only extremely good
  883. 1:05:39at coding but extremely good at every  well-posed virtual task which machine
  884. 1:05:43learning engineers are able to make in the lab. So if you push that to its limit and you ask how
  885. 1:05:47large those populations are, I think it’s very  conservative to say a population of billions of
  886. 1:05:50agents running at 10x human speed. We could debate exactly what the
  887. 1:05:53limits are, but I think they would  be significantly larger than that.
  888. 1:05:56Just because of software efficiency gains? That’s right. Just how efficient can you make it?
  889. 1:06:01You’re not going to be talking about  a single type of agent, all the same,
  890. 1:06:03one billion of them running. We’re going to be talking about
  891. 1:06:04these very heterogeneous programs,  many of which are very specialized.
  892. 1:06:08One thing you do if you do huge amounts of  R&D is say, "I would like to produce just
  893. 1:06:11the cheapest thing I can which is expert  enough in a domain to act effectively."
  894. 1:06:16Again, it’s a reasonable intuition pump to  say it will be more capable than billions
  895. 1:06:19of people thinking 10x as fast. But in fact, it’s really much,
  896. 1:06:22much larger populations of specialized systems,  and maybe billions of very general-purpose systems
  897. 1:06:26which are as good as humans at anything. If you imagine really pushing this process
  898. 1:06:33and having faster and faster progress on software,  and if someone grants that we’re going to see that
  899. 1:06:37process saturate, that everything you can do in  the data center is going to become really great,
  900. 1:06:41I think that endpoint is really extreme. The gap between doing really extreme
  901. 1:06:45stuff in the data center and doing really  extreme stuff in the world is very small,
  902. 1:06:49or smaller than people appreciate. There’s a discussion to be had about
  903. 1:06:57whether such systems are even possible. I am very compelled by the argument that
  904. 1:07:06there’s some learning algorithm that human brains  have, and something which can learn at least as
  905. 1:07:10well as a human can, but which has huge population  sizes and can learn from the deployed experience
  906. 1:07:17of hundreds of millions of different instances at  once — that alone would be pretty transformative.
  907. 1:07:23And then I agree with you that there’s no  clear reason to think that you can’t get even
  908. 1:07:27more learning-efficient than humans. I’m very happy to talk about this.
  909. 1:07:31I don’t know if it’s going to end up  being the main objection people have.
  910. 1:07:34But I think there’s very good evidence that you  should expect systems in the end, once you’ve
  911. 1:07:37really driven the technology to its limit, to  be much, much better than humans in these ways.
  912. 1:07:41It’s not just a case of, "We have humans,  so you know you can do at least that well."
  913. 1:07:44We have good evidence it’s much better than us. One relevant thing is just the rate
  914. 1:07:49of progress on evolutionary time. If you look back over evolution, it’s fairly clear
  915. 1:07:52that we’re seeing extremely rapid improvements  in learning capability and intelligence and
  916. 1:07:56planning capability as you move into humans. The thing that happened was just that humans
  917. 1:08:00got good enough that cultural change was  faster than those biological improvements.
  918. 1:08:03So if you just ballpark evolution, we know it  was clearly, if we didn’t form civilization,
  919. 1:08:08going to make much smarter things. Then, as you say, there’s the data question.
  920. 1:08:12Humans learn from one lifetime of data. AI systems already learn from radically
  921. 1:08:16more data than humans. Even if you were no
  922. 1:08:18better as an algorithm designer than  evolution, you would produce much, much
  923. 1:08:21better minds if you can relax that constraint. Then you have things like power constraints and
  924. 1:08:25coordination constraints, which we have a pretty  good sense are a really big deal for evolution.
  925. 1:08:29We will be able to push those out much further. If you want to produce electricity or something,
  926. 1:08:34to make a nuclear reactor you don’t  have to be smarter than evolution.
  927. 1:08:36You’re just operating in a  different regime from evolution.
  928. 1:08:38You’re able to do things that wouldn’t  have been a good idea for evolution.
  929. 1:08:41So I’m very happy to dig in more on  this, but it’s probably not going to
  930. 1:08:44be the most compelling argument for most people. It’s more compelling to look at the status quo and
  931. 1:08:47say, "Well, we’ve had 10 years of progress. You only have to extrapolate another 10
  932. 1:08:50years to get to the crazy superintelligence." But I do think we have a pretty good indicator
  933. 1:08:53that it does not top out anywhere near humans. Just to repeat your last point, what you’re
  934. 1:09:00saying is, look, humans developed on  the savanna under this tight constraint,
  935. 1:09:05where evolution was really just trying to get  something that is good at propagating its genes.
  936. 1:09:08One of the ways that a chimpanzee, or our  ancestors, can get better at propagating
  937. 1:09:11its genes is by having these general  planning and reasoning capabilities.
  938. 1:09:17But they’re not worth so much that you are  willing to, say, give up digestive functions
  939. 1:09:22or immune functions and surrender more of  your total 100-watt budget to the brain.
  940. 1:09:27You have this extremely tight power  constraint for the brain, and an extremely
  941. 1:09:30tight size constraint as a result not  only of that, but of other evolutionary
  942. 1:09:34reasons as well, like how your head can  literally fit through the birth canal.
  943. 1:09:38That will not be true of AI systems. We’re not saying that hardware will be
  944. 1:09:42more power-efficient than the human brain, but  simply that it’s not as big a constraint for us
  945. 1:09:46to have enormous amounts of hardware consuming  a lot of power as it was for evolution when it
  946. 1:09:52was designing the human brain. This cuts in both directions.
  947. 1:09:54It’s also the case that you’ll have  extremely power-efficient systems.
  948. 1:09:57If you’re a human, you can’t say, "Well, I’m going  to make 10,000 simple agents, each of which uses
  949. 1:10:02only 1% of a brain or 0.01% of a brain, and is  better than a human at doing cyber operations."
  950. 1:10:06It’s just this flexibility about  form factor that is going to be huge.
  951. 1:10:10If you were no smarter than evolution and  just did what evolution did, you would end
  952. 1:10:14up with agent swarms that are completely beyond  teams of humans, really radically beyond them.
  953. 1:10:21Let's say that an agent is vibe coding  a web app and it needs a database.
  954. 1:10:25There's one database provider that needs a human  to go through the whole sign-up flow, verifying
  955. 1:10:29their account, adding billing details, and then  generating an API key to hand back to the agent.
  956. 1:10:34And then there's another provider that lets  the agent make an account all on its own.
  957. 1:10:38The agent will probably choose the second one. In general, agents tend to pick the tools that
  958. 1:10:43they can use right away versus the  ones that they had to wait around on.
  959. 1:10:46So as agents do more and more work, giving  them their own way to sign up becomes
  960. 1:10:50an important way to win new users. My sponsor, WorkOS, makes this easy.
  961. 1:10:55If you use WorkOS for login, you can  give agents their own sign-up flow
  962. 1:11:00just by toggling on Agent Registration. You can then decide what agents should
  963. 1:11:04be able to do on their own, and what, if  anything, should wait until the person
  964. 1:11:08they're working for gets involved. Going back to the database example,
  965. 1:11:12maybe you want to set things up so that  agents can create databases all on their own,
  966. 1:11:16but they can't share them with anybody or  upgrade to a paid plan without a human.
  967. 1:11:20You can do all of this with permissions that  you already set up in WorkOS, instead of having
  968. 1:11:24to build a separate system for agents. To learn more, go to workos.com/authkit.
  969. 1:11:32There’s a bunch of objections people have  to the idea that automating AI research will
  970. 1:11:39lead to crazy fast progress along this  more general cluster of capabilities.
  971. 1:11:44You talk about experiments and data and  everything else, but I think what it comes
  972. 1:11:49down to at the end of the day, if you dig in  deep on any of these, is the sim-to-real gap.
  973. 1:11:55Is it possible that you could have a process  which occurs entirely within a data center,
  974. 1:12:01which results in minds that are not only  as capable as humans at learning in a new
  975. 1:12:05domain — especially a domain that is a real-world  thing, like making a new company or negotiating a
  976. 1:12:13deal with China — but even better than humans? How do we get AIs that are capable in this
  977. 1:12:23more general way you’re suggesting  from the within-data-center training
  978. 1:12:26that will be happening over the next year? I do want to start by reiterating that if
  979. 1:12:30you take progress in the real world — again,  granting that it is slower on messy real-world
  980. 1:12:34tasks than it has been in math or coding —  and just extrapolate it forward, I do think
  981. 1:12:40it gets us to systems which are powerful enough  that they would be able to outcompete humans.
  982. 1:12:46We could talk about what people are doing to drive  that and how you can scale it into the future.
  983. 1:12:52One, you can train on a bunch  of related well-posed tasks.
  984. 1:12:56These systems we’re talking about will have  been trained on a huge amount of predicting
  985. 1:12:59how humans will respond to something. When we talk about these AIs coming
  986. 1:13:02out of the data center, they’re not  AIs who have never seen the world.
  987. 1:13:04They are AIs who will be radically, radically  superhuman at predicting things like, given what
  988. 1:13:08just happened with that person’s face, how should  you update on what they’re going to do tomorrow?
  989. 1:13:12Or given what this company has done over  the last three weeks, what do you most
  990. 1:13:15expect this company to do in the next week? Or given observations of this laboratory
  991. 1:13:19environment, what is the best prediction  for the results of this experiment?
  992. 1:13:22So when we talk about sim-to-real, we should  be clear that we actually are including a bunch
  993. 1:13:26of these tasks which people train on now. That even includes things like, given the
  994. 1:13:30data so far from this laboratory, do as many  computational experiments as you can, and as
  995. 1:13:35much hypothesis generation and exploration as you  can, in order to come up with a hypothesis to make
  996. 1:13:39a good prediction about what you’ll see next. So these systems will also be very superhuman
  997. 1:13:42at many parts of science, at reasoning about  the real world, at planning and hypothesis
  998. 1:13:48generation and so on about the real world. That’s part of the story for how even the
  999. 1:13:52kind of thing we’re doing now can  produce very powerful systems.
  1000. 1:13:54A second part of the story is this. One regime is that you train on
  1001. 1:13:58exactly the thing you care about. You say, "I care about writing code that passes
  1002. 1:14:02this type of unit test, so I’m just going to  train systems that are incredibly good at that."
  1003. 1:14:06You can also do a second thing, which is a  large part of what we observe in performance:
  1004. 1:14:09train on a broad distribution of tasks, either  in a general domain or across several domains,
  1005. 1:14:14and then deploy those systems. They acquire general skills relevant
  1006. 1:14:18to that domain, which transfer to new tasks. I can say, "I’m not going to try and train a
  1007. 1:14:22system which is good at the exact  problem of going and talking to
  1008. 1:14:26Congresspeople in order to influence them." But I can train a system on an extremely broad
  1009. 1:14:30range of different tasks where there will  be a lot of common skills, including these
  1010. 1:14:33general skills: How do I adapt on the fly? How do I learn? How do I plan? How do I
  1011. 1:14:39set up reasonable experiments that I can  learn from and tell how well I’m doing?
  1012. 1:14:42I can create very broad distributions of  tasks, some of which could be coding tasks,
  1013. 1:14:45but some of which can be tasks like, you’re  dropped into a new, very messy virtual
  1014. 1:14:49environment and need to try and navigate it. Or you’re given a new type of task with 20
  1015. 1:14:54examples of it, and you need to learn how to do  a really good job on the next example quickly.
  1016. 1:14:59Or I have to interact with other AI systems,  or with humans, in environments where I can.
  1017. 1:15:03In the data center, there is a fair  amount of human data available.
  1018. 1:15:06You just go through the numbers and ask, "How much  interaction with humans can you actually get?"
  1019. 1:15:09It’s not like these systems  have never seen humans.
  1020. 1:15:10They will have seen vastly more human  interaction than any human would.
  1021. 1:15:14So they can also deal with a  bunch of reinforcement learning
  1022. 1:15:16environments that involve other humans  or involve collaboration with humans.
  1023. 1:15:19Because people have written in how  a human would respond to something?
  1024. 1:15:23How is that human data in there? You can also just literally engage
  1025. 1:15:26with humans as part of the training process. People do spend large amounts of money saying,
  1026. 1:15:29"Great, I’m going to have some  humans engage with the system."
  1027. 1:15:34There’s a question of how efficiently  you squeeze information out of that data.
  1028. 1:15:37How quickly do you learn? If you have these very general skills, how
  1029. 1:15:39quickly do you learn about the particular domains? But again, the dotted line from what we’ve seen so
  1030. 1:15:44far, together with a ton of capability to push  that forward and to do a bunch of automation,
  1031. 1:15:48like the construction of many more environments,  just looks very likely to lead to systems
  1032. 1:15:55that can quite quickly rise to at least  human-level, and likely way past human-level,
  1033. 1:16:00including on very messy real-world tasks. The thing I’m curious about is why you
  1034. 1:16:05think this happens over the next year. It seems in principle possible to me that
  1035. 1:16:12there’s enough data available even within a  data center that if you have a decent amount
  1036. 1:16:17of transfer, you have a very capable system. If there’s enough transfer from evolution
  1037. 1:16:23training us in the savanna to a human being  now able to make a semiconductor, then maybe
  1038. 1:16:27there is enough transfer from what’s happening  inside the data center to going to Congress and
  1039. 1:16:32trying to persuade them of something. But the things you’re mentioning,
  1040. 1:16:36like this endless backtesting or this RL  on fuzzy tasks, seem like things we’ve
  1041. 1:16:45already been doing for many years now. It doesn’t seem like we’re a year away,
  1042. 1:16:48or a further year of scaling  away, from this big a jump.
  1043. 1:16:53Am I correct in understanding that you’re  imagining something like what we’ve already done,
  1044. 1:16:56but with a much bigger qualitative  jump along those same lines?
  1045. 1:17:00Yeah, much faster. If you asked me, "Hey, Paul, if  you’re extrapolating the dotted line and looking
  1046. 1:17:05at the mechanics of what people are doing, how  long would it take to produce systems that are,
  1047. 1:17:08in fact, capable enough at all these tasks that  they would be able to reliably outcompete humans?"
  1048. 1:17:12I would probably say it seems like  it might happen in four years.
  1049. 1:17:16Maybe it could happen faster than that. I feel more confident within eight
  1050. 1:17:20years or something. Those are the kinds of
  1051. 1:17:22numbers I would be throwing around if we  were just asking, on the current trend,
  1052. 1:17:25how long will it take, doing exactly the things  we’re currently doing, to get these capabilities?
  1053. 1:17:30There’s lots of room for  debate about those numbers.
  1054. 1:17:32But the thing that I am most worried about, that  seems like an extremely live possibility, probably
  1055. 1:17:37more likely than not at this point, is that  instead the work that people are doing in machine
  1056. 1:17:43learning to drive all those improvements we’ve  seen over the last 4-8 years will be automated
  1057. 1:17:47to such an extent that the pace of that progress  is accelerated by at least 10x, and likely more.
  1058. 1:17:54We are maybe already seeing  acceleration in the ballpark of 2x.
  1059. 1:17:57I think it is very likely, on the current  margin, that the first thing that acceleration
  1060. 1:18:01does is continue sustaining itself. Regardless of this broader discussion
  1061. 1:18:06of how good AI systems are at acting in messy  real-world environments, AI systems are very
  1062. 1:18:09good at acting in virtual environments when  they are trying to deliver on verifiable tasks.
  1063. 1:18:14I think the thing that is most robust is that  AI systems are improving extremely rapidly and
  1064. 1:18:17are starting to cross human-level on  the kinds of tasks that are involved
  1065. 1:18:20in making your AI systems better at learning in  environments you can simulate in the data center.
  1066. 1:18:26Slightly more broadly, that includes the  kinds of software engineering tasks that
  1067. 1:18:28are involved in creating new environments  that stress-test hard skills, and even many
  1068. 1:18:32of the software engineering skills involved in  orchestrating data collection from the real world.
  1069. 1:18:36We could talk about what the  bottlenecks are in that process
  1070. 1:18:40which cannot be accelerated by automation. But if you look across the core things
  1071. 1:18:44humans have been doing to drive the last  4-8 years of progress, those are things
  1072. 1:18:47we are on an extremely good pace to automate. We are fairly likely to have the self-sustaining
  1073. 1:18:51acceleration where, in eight months, you  double the pace, and in the next four months,
  1074. 1:18:56you double the pace again, and in two  months, you double the pace again.
  1075. 1:19:01At the beginning of that process, it will  be focused primarily on these virtual tasks,
  1076. 1:19:05because your main priority is making the  self-sustaining loop faster and faster.
  1077. 1:19:09If you’re a person who just wants to get to the  endpoint as quickly as possible, you do not care
  1078. 1:19:12how good a lawyer your AI system is for a while. You just care how good it is at driving the
  1079. 1:19:16development of better architectures and better  optimizers and better learning algorithms,
  1080. 1:19:21or even constructing new environments. Maybe you care about one slightly messy task,
  1081. 1:19:27which is constructing environments  that help you build the skills that
  1082. 1:19:30are relevant in the real world. But you don’t care about almost
  1083. 1:19:32any of the real-world tasks where  we’re seeing the slowest progress.
  1084. 1:19:35You’re focused on that. You’re going through  this process of seeing doublings in eight months
  1085. 1:19:38and four months and two months and one month. Then, as that process goes, it becomes cheaper and
  1086. 1:19:42cheaper, faster and faster, to develop algorithms  and models that perform very effectively in the
  1087. 1:19:48real-world domains that ultimately matter. That is really the concern.
  1088. 1:19:52The first question is, is  that process going to occur?
  1089. 1:19:56Are you going to see the self-sustaining  acceleration with faster and faster progress?
  1090. 1:20:00Honestly, to me, that is probably  the bigger open question.
  1091. 1:20:03I think that looks more likely than not right now. But if it occurs, I think things are almost
  1092. 1:20:06certain to be extremely crazy. There is then a second question
  1093. 1:20:09about the particular way in which they get  crazy, which is that it looks very likely
  1094. 1:20:12those models will transfer to the real world,  to systems that are able to outcompete humans.
  1095. 1:20:17But that does happen very back-loaded in the year. If you’re imagining a year where very disruptive
  1096. 1:20:21things happen, and this is part of why  it’s so hard from a policy perspective,
  1097. 1:20:24for the first six months of that year,  you look at the systems and say, "Well,
  1098. 1:20:30they were previously superhuman at math, and  now they’re incredibly superhuman at math."
  1099. 1:20:34But if you go and talk to someone,  there’s no real ability to say,
  1100. 1:20:37"Therefore, they are a threat to humanity." AI labs are currently saying things like,
  1101. 1:20:42"Every month, we’re doubling the amount of  internal use of AI, and the degree to which
  1102. 1:20:45it’s contributing to our progress." Or, "Over the last six months,
  1103. 1:20:48we’ve seen a doubling of the net pace of  progress, even including diminishing returns."
  1104. 1:20:53You can have more statements like that. But you could be in a world where the
  1105. 1:20:59link from that to very severe harms does not  become obvious until quite late in the process.
  1106. 1:21:04To some extent, I think it is  getting increasingly obvious now.
  1107. 1:21:06If you look at cyber capabilities of models  in particular, but also their capabilities in
  1108. 1:21:10some scientific domains, they are approaching  the point where there are very severe risks.
  1109. 1:21:17We’re seeing that in a couple of domains, so it’s  tempting to say, "Okay, we have two problems.
  1110. 1:21:20There are just a few specific domains in  which AI systems have dangerous capabilities."
  1111. 1:21:25But if this process of  acceleration continues, that
  1112. 1:21:27is not going to be localized to cyber or to bio. That is going to be a thing that occurs across all
  1113. 1:21:31of science, across all of social engineering. If you’re thinking about the risk,
  1114. 1:21:36you really need to understand whether this is  possible: whether you can have the self-sustaining
  1115. 1:21:41acceleration, then leading to very, very rapid  progress in dangerous real-world capabilities,
  1116. 1:21:45both in specific domains and in these very  general learning faculties we’ve talked about.
  1117. 1:21:49This goes back to the thing you were saying,  that you should imagine the systems in the
  1118. 1:21:52future are less about fine-tuning GPT-6 to do  something, and more like GPT-12 being able to
  1119. 1:21:58do it out of the box with very little experience. GPT-12 will also be great at fine-tuning GPT-12.
  1120. 1:22:03If GPT-12 is like, "I find that my chemistry  intuition is not very good," GPT-12 will be
  1121. 1:22:07great at saying, "Okay, I’m going to direct 1,000  humans, and here’s the data I think I most need.
  1122. 1:22:11We’re going to do one day of  data collection of that type.
  1123. 1:22:14We’re going to run these  experiments and do the de-risking."
  1124. 1:22:16Within four days, GPT-12 has the  new tiny model which has incredible
  1125. 1:22:20chemistry intuition, and it’s doing  this in parallel across all domains.
  1126. 1:22:26You just answered a question  which I meant to ask you.
  1127. 1:22:28It seems like a big thing that has driven your  concern over the last few days is this new result
  1128. 1:22:32from METR, where they find that Anthropic’s pace  of AI progress on well-scoped tasks has doubled.
  1129. 1:22:40These are tasks as captured by benchmarks  like the Epoch Capabilities Index, which
  1130. 1:22:45amalgamates a bunch of different benchmarks. Now, it could be that there’s other evidence
  1131. 1:22:49relevant to METR’s assessment that AI  progress has doubled, but that evidence
  1132. 1:22:52has not been made public. That’s one of the problems
  1133. 1:22:54you have with the current setup. But just consider that evidence on its own.
  1134. 1:23:00Progress on verifiable tasks has doubled. But we have this effect where we’re searching
  1135. 1:23:05for our keys under the lamplight. It is also the verifiable tasks
  1136. 1:23:09which we’re training on. So you can ask the question,
  1137. 1:23:12has progress on non-verifiable tasks  also doubled over the last year?
  1138. 1:23:19As you may have noticed, over the last  week the stock market took a big tumble.
  1139. 1:23:22Part of that was almost certainly the midterms. Another part of the reason is that the growth of
  1140. 1:23:30Anthropic’s and OpenAI’s revenue numbers did not  keep track with what the market was expecting.
  1141. 1:23:34In Q3, Anthropic’s revenue grew  something like $5.4 billion.
  1142. 1:23:39That was less in absolute terms than  Anthropic’s revenue grew in Q2, $6.9 billion.
  1143. 1:23:45So not only has the rate of growth  slowed down, which people expected,
  1144. 1:23:48but the absolute growth has also slowed down. That maybe suggests that measuring progress on
  1145. 1:23:55well-scoped tasks is not actually telling  us about the pace of overall AI progress.
  1146. 1:23:59Because how can it be the case  that overall progress is 2x faster,
  1147. 1:24:03and yet absolute revenue is not even growing  at the same rate it was previously growing?
  1148. 1:24:08Maybe the explanation is what you just suggested:  these AI companies are really developing better
  1149. 1:24:14and better RSI models, and only incidentally  releasing Claude for Law or Claude for Finance as
  1150. 1:24:21a stopgap to keep this underlying process going. This hasn’t been a priority,
  1151. 1:24:25and they haven’t worked that hard on it. You should really be concentrating on this
  1152. 1:24:30inner exponential of the RSI capabilities  of the model, and that will then generalize.
  1153. 1:24:35But how would you interpret this disconnect  between the pace of improvement in the end
  1154. 1:24:41usefulness of these AI systems and the  pace of improvement on well-scoped tasks?
  1155. 1:24:47Zooming out slightly, if we talk about not  just the last six weeks, but in general what
  1156. 1:24:51has happened over the last 18 months, I think we  have seen extraordinarily rapid growth in revenue.
  1157. 1:24:57We can talk about what the story  is for the last four weeks.
  1158. 1:25:00But we are talking about revenue  more than doubling every six months.
  1159. 1:25:06That is very, very fast growth  for the industry overall.
  1160. 1:25:12If we want to match up these internal  results on progress to external results,
  1161. 1:25:17the period in question has also corresponded  to a period of incredibly fast revenue growth.
  1162. 1:25:21It’s very hard to say whether that revenue growth  corresponds to capabilities progress that is
  1163. 1:25:24twice as fast as previous capabilities progress. I think the revenue numbers are noisy enough, and
  1164. 1:25:28the staggering of model releases is noisy enough,  that it is really hard to draw that judgment.
  1165. 1:25:34There are just a lot of problems  in trying to make that inference.
  1166. 1:25:39At a high level, I think revenue is  a reasonable indicator, and revenue
  1167. 1:25:42progress is extremely rapid. But very short-term trends in
  1168. 1:25:44revenue, particularly at a single  company level, are extremely noisy.
  1169. 1:25:49To me, it feels like there’s a  discourse where, periodically,
  1170. 1:25:52the length of the AI winters is shortening. Now it’s been two months, and AI revenue
  1171. 1:25:58over the last two months has not been on  track to grow as explosively as before.
  1172. 1:26:03That doesn’t mean it’s all over. My guess is that you’re just seeing
  1173. 1:26:07some amount of noise in these processes. Just to take the other side of that, you
  1174. 1:26:13might think it’s because Dario released  his pacing letter, and they haven’t
  1175. 1:26:16released their best models or something. But if you look at the METR graph for
  1176. 1:26:21the capabilities index, Opus 5.5 doesn’t  seem that far below Model 4, which seems
  1177. 1:26:28to be Anthropic’s latest internal model. This might be the result of the staggering
  1178. 1:26:35of AI capabilities, but it’s not clear  this is the staggering of model releases.
  1179. 1:26:39The main thing I mean about model releases  is that the rate of revenue growth, again,
  1180. 1:26:42specifically at the company level over very short  time periods, is extremely noisy, because you’re
  1181. 1:26:46looking at a small number of model releases. You’re looking at these market
  1182. 1:26:49dynamics over very short time periods. My main opinion on this, and I am happy to dig
  1183. 1:26:55more into it, is that it is extremely hard to draw  inferences from extremely short time series data.
  1184. 1:27:02This is true to a reasonable  extent for AI capabilities too.
  1185. 1:27:06The more downstream you  go, the noisier things get.
  1186. 1:27:08But part of the issue on the RSI side is  that I also don’t really think we can draw
  1187. 1:27:12inferences from the last couple months of data  very well in terms of benchmark performance.
  1188. 1:27:17We have the same thing there, where  there’s enough noise that we have no idea.
  1189. 1:27:22We can tell, over the period of years,  whether things have accelerated.
  1190. 1:27:25It’s very easy to look back  over the last four years and
  1191. 1:27:27say roughly how fast this thing’s been going. It just gets much harder as you zoom into small
  1192. 1:27:30time periods because of the amount of noise. If you look at the details of where models are
  1193. 1:27:35coming from, you have these pre-training  runs that occur at some point in time.
  1194. 1:27:38When do pre-training runs finish? Then you have the exact sequence of post-training,
  1195. 1:27:42and which model checkpoints get released when. You have random variability in model quality.
  1196. 1:27:48You have those dynamics, which make  things very noisy over short timescales.
  1197. 1:27:52Then those just get filtered out to even noisier  stuff as you move to the market more broadly.
  1198. 1:27:58If we’re going to be in a position where within  a year we have superintelligences that could
  1199. 1:28:02overthrow the government, what exactly is  the plan to make them such that they don’t?
  1200. 1:28:07Because we’re asking for a level  of saintliness that we should just
  1201. 1:28:12not expect out of even most humans. Sometimes people say that the goal of
  1202. 1:28:15alignment is to give AIs human values. This doesn’t make sense to me.
  1203. 1:28:19Of course, different humans have different values. But also, if there’s one lesson that history
  1204. 1:28:24has given us, it’s that humans can’t be  trusted with arbitrary amounts of power.
  1205. 1:28:28Cortés was a human. Stalin was a human.  A logical conclusion of believing the
  1206. 1:28:39technological and forecasting model that you  have laid out is just that we will have systems
  1207. 1:28:44that we will have to trust with an overwhelming  amount of power, if not realized then latent,
  1208. 1:28:50because they will have the capability of  overthrowing the government if they want to.
  1209. 1:28:58Then you just think about why they  would overthrow the government.
  1210. 1:29:01Either they have secret goals as a result of  the training process, or, as a direct result
  1211. 1:29:05of the training process, they care about how  they’re graded, or a million other things.
  1212. 1:29:10This is the basic argument  about instrumental convergence.
  1213. 1:29:12There are many different reasons  somebody could want power.
  1214. 1:29:15So what is the argument? If this happens in a year,
  1215. 1:29:18how do things turn out okay? What is the actual plan here?
  1216. 1:29:21I think there are a lot of different  things someone could hope for.
  1217. 1:29:24They are all at the stage of "we hope  something will happen over that year."
  1218. 1:29:28We’re producing very smart AI systems. They’re going to do a bunch of research.
  1219. 1:29:31It is plausible they will think of something  over that year that we have not thought of.
  1220. 1:29:34That is mostly where we’re at right now. If they do not think of something clever
  1221. 1:29:38that we haven’t thought of, it  is probably going to end badly.
  1222. 1:29:43In terms of the kinds of hopes  you could have, there is one
  1223. 1:29:47hope that’s just: try and make them good guys. They’re nice. They share human values. They like
  1224. 1:29:53flourishing. They’re in a position to take over. We have very indirect control over what they want.
  1225. 1:29:57Then we just take the leap of faith. I think this is a pretty unappealing option.
  1226. 1:30:02In general, I hope people  will not go for that option.
  1227. 1:30:05Even if it works out, even if the AIs are good  people in some sense and now they call the shots,
  1228. 1:30:11it is not a way that humanity as a whole should  be like, "We are transferring our sovereignty
  1229. 1:30:15to this new species we’ve built," because  that was the fastest way for some person to
  1230. 1:30:18build this particular product. I don’t like that option,
  1231. 1:30:22but it is an option you could take. There’s another option: we are going
  1232. 1:30:27to understand scientifically more about how  the distribution of training environments
  1233. 1:30:32shapes the motivations of these systems. We’d get to such a point where, instead of the
  1234. 1:30:35conversations we’re having now about maybe  they’ll want X or maybe they’ll want Y,
  1235. 1:30:38we’ll have a good scientific basis for saying,  "Hey, if we train them in the following way,
  1236. 1:30:41they do in fact end up wanting reward or whatever. They don’t want some other thing.
  1237. 1:30:45They actually just want to get high scores. And maybe even moreover, if we train them
  1238. 1:30:49in environments where they didn’t have good  opportunities to hack, maybe they want score but
  1239. 1:30:52in a nice way where they won’t hack to get it. They’ll just do what we want or
  1240. 1:30:55what we’re trying to incentivize." You could try and have an understanding like that.
  1241. 1:30:59That is probably the closest thing to what  the industry is trying to bank on right now.
  1242. 1:31:03It’s very dicey. It’s very dicey  to not have any opportunities to
  1243. 1:31:07hack in training, which is basically a  precondition for this approach working.
  1244. 1:31:10And then it’s very dicey because we  don’t actually have a good scientific
  1245. 1:31:13understanding of how this works. Our best expectation is that models
  1246. 1:31:16will learn to hack at deployment time,  even if they never hack in training.
  1247. 1:31:19I think that is the best scientific guess now. So you could try and deal with that.
  1248. 1:31:25You could try and say, "We’re going to understand  really well how models work internally,
  1249. 1:31:28and then we can train them based not just  on what they do, but on how they think."
  1250. 1:31:33Do good interpretability or something. Be like, "We understand here’s how it thinks about
  1251. 1:31:36reward, and here’s how it thinks about hacking. And we’re not just going to select for
  1252. 1:31:39whether it behaved well in training. We’re going to look and intervene directly
  1253. 1:31:41on how it thinks, and either notice that something  is going bad or prevent something from going bad."
  1254. 1:31:46As an aside, I run this organization, ARC, which  is working on not the interpretability version,
  1255. 1:31:50but the mathematical version of that hope. Can we write algorithms which do structural
  1256. 1:31:55analysis of the weights of a neural network in  order to understand something about its behavior,
  1257. 1:32:00even if it’s not something we  could select for in training?
  1258. 1:32:02You’re literally having to understand the whole  computational graph of superintelligent minds?
  1259. 1:32:06That’s correct. Or we don’t understand it,  but we are writing algorithms which are
  1260. 1:32:10able to understand and make deductions about  the behavior of some superintelligent mind.
  1261. 1:32:14I would describe it as simultaneously a  bank shot, which is probably not going
  1262. 1:32:19to end up relevant to this discussion. I think there’s a 10% chance or 5% chance
  1263. 1:32:23that we succeed in a game-changing way. But also, we don’t have that long a
  1264. 1:32:27list of things we’re trying. I do think, pound for pound,
  1265. 1:32:29it’s probably the most promising thing. Wait. You’re like, "Okay, here’s the thing.
  1266. 1:32:34It has a 10% chance of working, but  it’s also the most promising thing."
  1267. 1:32:38Most promising pound for pound. If there are 20 people working on
  1268. 1:32:43that thing, that can be a good bet. But probably what we’re going to do,
  1269. 1:32:47if we do not slow down, is try and be like,  "We fussed with the training environments.
  1270. 1:32:53We did some experiments that give some  kind of rough sense of what’s happening.
  1271. 1:32:55We made the training environments really careful. We made the graders really good, so if you try and
  1272. 1:32:59hack, it’s normally going to get penalized." That’s probably what we’re going to do.
  1273. 1:33:02But you don’t think it will work. I think it will work for some amount
  1274. 1:33:06of time, and then it will break. If you talk about this year,
  1275. 1:33:08it works for the first nine months of the  year, or some number of months of the year.
  1276. 1:33:11We don’t know exactly when it breaks down. Sorry, it’s in the next year,
  1277. 1:33:14so basically halfway into next year? If we go to superintelligence, at some
  1278. 1:33:18point along the path to superintelligence, the  existing methods are probably going to break down.
  1279. 1:33:21Sorry, we’re not even on track to get  it so that all the RL environments,
  1280. 1:33:25for example, don’t have reward hacks and  don’t actively encourage the AIs to hack.
  1281. 1:33:29Not the case today. But suppose we did.
  1282. 1:33:31Suppose we did that. What you’re claiming is that—
  1283. 1:33:35I think eventually, basically,  we’d still have a problem.
  1284. 1:33:38The plan is basically that we just don’t know. Obviously, if you think about it,
  1285. 1:33:44it’s a hard problem. Superintelligence is this alien mind,
  1286. 1:33:47and you won’t have a guarantee that you understand  why it is doing the things that it’s doing,
  1287. 1:33:52even when it is thinking in alien ways many times  faster than you would, in huge population sizes.
  1288. 1:33:57That’s a goal. We don’t know how to solve it. There’s no single approach where we think there’s
  1289. 1:34:04a 50% chance that that approach will work. That’s definitely the case. There’s a
  1290. 1:34:07bunch of things people are trying. So the hope is just that three months
  1291. 1:34:11before we get that superintelligence that could  overthrow the government, we let the AIs loose
  1292. 1:34:18on all these different approaches that have a  couple percent chance of potentially working.
  1293. 1:34:22Hopefully the AIs figure  out one of these approaches.
  1294. 1:34:24They give us a solution. Hopefully, at this point, they
  1295. 1:34:27are still on our side, and they want this problem  solved, and solved only in a way that benefits us,
  1296. 1:34:34or whatever "us" means in this context. Then the hope is that we adopt it and
  1297. 1:34:41build crazy amounts of superintelligent  minds that are fundamentally on our side.
  1298. 1:34:47Of course, I’ve obviously glossed  over a very important discussion when
  1299. 1:34:49I’ve talked about "on our side," because  what does being on our side really mean?
  1300. 1:34:52Aligned to whom? That is obviously a real  discussion that can’t be glossed over.
  1301. 1:34:56That is the plan. I agree that that is the plan. We can talk about whether they have a
  1302. 1:35:00couple percent or tens of percent. I’m not saying this as if there’s a
  1303. 1:35:04better plan, but I think people are not aware  that this is the state of the situation.
  1304. 1:35:13You’re sending the soldier on an expedition,  and you’re hoping he finds a gun along the way.
  1305. 1:35:19You’re building the plane as you fly it. You hope that you have the machinery to
  1306. 1:35:23tell if something very bad is about to  happen, and that you will stop then,
  1307. 1:35:26although we’ve talked about security and  why that can be challenging, and so on.
  1308. 1:35:30Then you hope that if you do stop, you’ll be  able to come up with something quickly, or you’re
  1309. 1:35:33constantly trying to come up with something. I think interpretability is a good example of
  1310. 1:35:36the kind of bank shot people have in mind. Right now we’re fairly far from
  1311. 1:35:39understanding how these models work. But you could imagine that with a bunch
  1312. 1:35:41of AI assistants, we’re able to really  understand how individual models think,
  1313. 1:35:45well enough that we can intervene in order  to make them not want to do bad stuff.
  1314. 1:35:50Another thing I want to emphasize is that I’m  skeptical of whether things will happen this fast.
  1315. 1:35:56Certainly, it is reasonable for  somebody listening to be skeptical.
  1316. 1:35:58You are skeptical, right? It’s 2/3 of worlds  where, even if we went full steam ahead,
  1317. 1:36:02this might not happen so fast. But it is a live possibility.
  1318. 1:36:07I agree. It seems logically  consistent that this could happen,
  1319. 1:36:11and it should not be that surprising. I think people would just be shocked
  1320. 1:36:16if they heard the kinds of discussions. This is not something where Paul is on
  1321. 1:36:20some crazy end of the spectrum compared to  how people inside the AI labs are thinking.
  1322. 1:36:25It’s just crazy how many conversations  and dinners I have with researchers at
  1323. 1:36:29the labs where, again, you could believe  the whole town has taken crazy pills.
  1324. 1:36:33There’s certainly a chance of that. We hope. We really hope.
  1325. 1:36:36But this is just not out of the realm  of how people at the labs are thinking.
  1326. 1:36:40This is truly how they see the situation. There’s some resignation where they
  1327. 1:36:47accept that because this is not common  knowledge, they are stuck in this dynamic.
  1328. 1:36:51Of course, a way to solve that is  to establish that common knowledge.
  1329. 1:36:55I just wanted to really emphasize that people  listening might be shocked at the way these
  1330. 1:36:59researchers talk when they’re amongst  themselves, or at dinner or something.
  1331. 1:37:02This is really how they see the situation. A lot of people are worried to varying extents.
  1332. 1:37:08I’m definitely more worried than the median,  but less worried than the 90th percentile.
  1333. 1:37:14If you had a dinner of researchers, I think  it would be uncommon for someone to say,
  1334. 1:37:17"There’s a 99% chance that this goes well." The discussion is more like, "Is it 95 or 75?
  1335. 1:37:22What are we talking about for the  probability of navigating this?"
  1336. 1:37:26So why do you think it might go well? Is it just that the superintelligent AIs
  1337. 1:37:31might give us some solution to alignment, and  also that politically we are well organized
  1338. 1:37:35enough to have a really good document… I think we basically have two saving throws.
  1339. 1:37:39One saving throw is that there’s  a reasonable chance we’re going
  1340. 1:37:41to figure out something that works. I do think all the current things break
  1341. 1:37:45down, but there are a number of irons in the fire. You could imagine things like
  1342. 1:37:49the current methods that are going to work. That’s one. Maybe there’s a 50/50 chance
  1343. 1:37:53that we find something that works  in time, before there’s a disaster.
  1344. 1:37:56You have a second saving throw: we could stop. If we discover, either preemptively or after
  1345. 1:38:01the fact, that things are bad, we might say,  "We cannot adequately control these systems,
  1346. 1:38:04so we’re going to stop building them." I am very worried that we’re going
  1347. 1:38:07into this process without the infrastructure  that would be necessary to actually do that.
  1348. 1:38:10Like we talked about, you stop, your adversary  steals the model, and they’re like, "I’m willing
  1349. 1:38:13to take the 50% risk," or whatever. But conceptually, you could stop.
  1350. 1:38:17Those are my two saving throws. Each of them is a 50% chance, and bam,
  1351. 1:38:20we’re at like a 75% chance things are okay. But that does involve being able to
  1352. 1:38:24stop if things are not okay. That is not where we are now.
  1353. 1:38:27So it does involve us fairly urgently  doing something now in order to be in a
  1354. 1:38:32better position to stop if necessary. I think it’s a good place to close.
  1355. 1:38:38Paul, thanks so much. Thanks for having me. Great talking.

About this transcript

This page contains the full transcript of We could be 12 months away from AIs that can overthrow the government by Dwarkesh Patel, generated from the public captions YouTube serves with the video. The transcript has 19,506 words across 1,355 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.