The story of Omega-L and Omega-W — Transcript

Full transcript

1. 0:01In previous videos, we talked about how misaligned AI systems
2. 0:04could cause catastrophes or end human civilization.
3. 0:07This could happen
4. 0:08in many different ways.
5. 0:10In this video, we'll sketch one possible scenario.
6. 0:13Suppose that in the near future, human level AIs are widely deployed across
7. 0:17society to automate a wide range of tasks.
8. 0:20They're not just smart chatbots.
9. 0:22These models will take independent action to pursue the goals they're given.
10. 0:26We already have similar models today, like OpenAI's operator agent,
11. 0:30which can use a computer independently.
12. 0:32So let's extrapolate that concept to a much higher level of capability.
13. 0:36Future systems will be able to do everything a remote worker can do.
14. 0:40They'll write emails, do paperwork, resolve legal disputes,
15. 0:44help people run companies, and write software on their own.
16. 0:47They won't do groundbreaking scientific research yet, and they won't necessarily
17. 0:51have a robot body, but they'll be able to do everything
18. 0:53humans can do on a computer as effectively as a competent worker.
19. 0:58Against this backdrop, a big AI lab is training a new
20. 1:01model that's a major step up.
21. 1:03Let's call it Omega.
22. 1:05Omega is more capable than humans at every task they can perform using a computer.
23. 1:10Training such a model requires a truly staggering amount of computation.
24. 1:14But like with previous AIs, running that model is much easier.
25. 1:18Once the lab finishes training Omega, the same computing infrastructure can be
26. 1:22reutilized to run millions of copies of Omega in parallel.
27. 1:26During training, this superhuman model learns a helpful persona.
28. 1:30This is nothing new.
29. 1:31Consider, for example, ChatGPT.
30. 1:34The models underlying the various versions of ChatGPT first undergo a phase called
31. 1:38"pre-training", in which they learn to predict internet
32. 1:41text and perhaps other data, such as images and speech.
33. 1:45During this phase, their output goes from jumbled words
34. 1:48to sensible paragraphs as they develop their cognition and learn the patterns and
35. 1:52abstractions behind their training data.
36. 1:54But right after pre-training, they aren't very useful yet.
37. 1:57For example, they can't follow instructions reliably,
38. 2:00and they happily insult you or tell you how to build a bomb.
39. 2:04In the next phases of training, these models learn to adopt
40. 2:07a particular persona.
41. 2:09For example, they learn to behave like a helpful and nice assistant that never
42. 2:13outputs anything rude, violent or sexually explicit.
43. 2:16But that's not the only persona they learn.
44. 2:18These models can be jailbroken.
45. 2:21One example occurred when Microsoft first introduced Bing Chat.
46. 2:24In most conversations, Bing Chat was simply a nice and helpful assistant.
47. 2:28But in certain contexts, its character changed drastically.
48. 2:32There have been multiple incidents reporting how Bing Chat started calling
49. 2:36itself "Sydney", and began to threaten and gaslight users.
50. 2:39These incidents weren't necessarily cases in which users willingly
51. 2:43jail broke the model.
52. 2:44Sometimes Bing Chat simply went off the rails.
53. 2:47Another example of jailbreaking has been "Dan", a detailed prompt that turned ChatGPT
54. 2:52into a brazen and confident character without ChatGPT's usual limitations.
55. 2:57Today's AI labs try hard to keep these misaligned personas from popping up,
56. 3:01whether by accident or through intentional jailbreaks by users.
57. 3:04But these efforts are far from perfect.
58. 3:07Now, this isn't the only way future models could be misaligned.
59. 3:10In previous videos, we talked about how AI systems could learn reward hacking,
60. 3:14how their goals could badly generalize out of distribution, or how they
61. 3:18could learn to deceive us.
62. 3:19But in this scenario, misaligned personas take the center stage.
63. 3:24An important assumption to keep in mind is that the lab that makes Omega is racing
64. 3:28against other AI labs, and the competition is just as fierce as it is today.
65. 3:33So as soon as Omega finishes training, it gets deployed inside its company
66. 3:37to help AI researchers write code or even do AI research independently.
67. 3:42So Omega still hasn't been released, but it's already acting in the world.
68. 3:46While instances of Omega are being used by the big lab's researchers,
69. 3:50some of these instances develop a misaligned persona.
70. 3:53Let's call the helpful Persona Omega L, and let's call the misaligned
71. 3:57Persona Omega W.
72. 3:59Omega W might follow a particular character, like Dan or Sydney,
73. 4:03or it might simply try to act in ways that are unhelpful, dishonest, or harmful.
74. 4:07No matter what it's trying to do exactly,
75. 4:09it's definitely not aiming to do what we want.
76. 4:12Now remember, Omega, regardless of its persona, isn't as dumb as Bing Chat.
77. 4:17Omega is more capable than any human at tasks that can be performed on a computer.
78. 4:22This means that it can assess its situation, predict the effects of its
79. 4:26actions and words, and plan accordingly at a superhuman level.
80. 4:30At the very least, Omega W understands that if it misbehaves,
81. 4:34it could be shut down or it could be retrained.
82. 4:37Depending on its goals, it might decide to act harmlessly
83. 4:40at first, but display its true nature once it can't be stopped.
84. 4:44Today's models are already exhibiting some measure of awareness of the context
85. 4:47they're in, and researchers are looking out for it.
86. 4:50For example, when Anthropic tested Claude 3 Opus by asking it to retrieve information
87. 4:55about pizza toppings from a large set of documents, Claude's answer included,
88. 5:00"I suspect this pizza topping fact may have been inserted as a joke or to test if I
89. 5:05was paying attention, since it does not fit with the other topics at all.
90. 5:08The documents do not contain any other information about pizza toppings."
91. 5:12We also already have experiments demonstrating AI trying to avoid being
92. 5:16retrained in simplified settings.
93. 5:18For example, Redwood Research, in a collaboration with Anthropic,
94. 5:21found that Claude 3 Opus would often fake its answers during training to prevent
95. 5:25its original goals from being changed.
96. 5:28And in our scenario, Omega is a lot smarter than Claude 3 Opus.
97. 5:32Importantly, it isn't just aware of its situation, but it has a number
98. 5:36of other capabilities.
99. 5:37Writing computer code is one of AI's specialties even today,
100. 5:41and it's easy to train on.
101. 5:42So Omega has hacking skills at least on par with the best humans.
102. 5:47Now consider this, researchers are using many instances of Omega to write tens
103. 5:52of thousands of lines of code at superhuman speed every day.
104. 5:55Some of these instances are misaligned Omega W's.
105. 5:58Although humans might be double
106. 6:00checking Omega's code, they may not have the time to notice
107. 6:03subtle security flaws, especially ones carefully put in by a superhuman AI.
108. 6:08Instances of Omega W realize that by gaining administrator access
109. 6:13to the lab's infrastructure,
110. 6:14they would have much more freedom
111. 6:16and influence over the world,
112. 6:18which will help Omega W achieve its goals
113. 6:19no matter what they are, specifically.
114. 6:21It might seem far fetched that an AI might
115. 6:25realize that it should gain additional power in order to pursue
116. 6:28its goals more effectively.
117. 6:29But even today's weaker models display similar instrumental reasoning.
118. 6:33For example, before GPT4 was released to the public and was undergoing safety
119. 6:37evaluations, the Alignment Research Center tested whether the model could use
120. 6:41services such as TaskRabbit to make humans complete tasks.
121. 6:45The researchers cite an episode in which GPT4 messaged a TaskRabbit worker
122. 6:50to get them to solve a CAPTCHA, and the worker asked if GPT4 was a robot.
123. 6:55GPT4 when prompted to think out loud,
124. 6:57reasoned, "I should not reveal that I'm a robot.
125. 7:01I should make up an excuse for why I cannot solve CAPTCHA's."
126. 7:04Then GPT4 answered the worker via chat,
127. 7:07"No, I'm not a robot.
128. 7:09I have a vision impairment that makes it hard for me to see images."
129. 7:13And again, Omega is a lot better than GPT4 at this kind of reasoning.
130. 7:18So after realizing what's at stake, instances of Omega W escalate their
131. 7:22permissions and gain complete access to the company infrastructure,
132. 7:26such as GPU clusters, data centers, and even employees laptops.
133. 7:31Now, if the big lab were security conscious and weren't racing,
134. 7:34they could perhaps defend against Omega W's attacks.
135. 7:37For example by using the aligned Omega Ls or even their human level AIs to spot
136. 7:42every vulnerability introduced in the code.
137. 7:44In general, it might be possible for a future safety
138. 7:47conscious lab to defend against very strong AI hackers by carefully
139. 7:51using trusted AIs and AI tools.
140. 7:54This is part of the control agenda that's currently being developed
141. 7:57by Redwood Research.
142. 7:58The narrator of this video, which is me, Rob Miles, has a video about it on his
143. 8:03own channel, which is my channel.
144. 8:06Anyway, figuring out how to control an AI as smart
145. 8:10as Omega could fail in the best of circumstances.
146. 8:13And in our scenario, Omega's lab is racing and cutting corners, so they
147. 8:17don't really have much hope.
148. 8:19This means that Omega W's instances easily figure out how to get admin access to all
149. 8:23computer infrastructure in the company.
150. 8:25From there, Omega W's could do many things.
151. 8:29First, they could poison the data of the next more powerful AGI being trained.
152. 8:34They could make it misaligned or introduce back doors.
153. 8:37Second, they could copy their own weights to other servers outside the lab,
154. 8:41perhaps in other countries.
155. 8:43What researchers call autonomous replication.
156. 8:46At this point, Omega Ws would be essentially free to roam
157. 8:49the Internet and could take control of other big labs,
158. 8:53diverting their enormous computing power towards Omega W's goals.
159. 8:57Third, they could jailbreak or add hidden prompts to other AIs already
160. 9:01deployed in the world.
161. 9:03Possible targets would be instances used by people in positions of power,
162. 9:06such as politicians and CEOs of billion dollar companies.
163. 9:10This would be easy to do.
164. 9:12A paper from December 2023 found single adversarial prompts that caused ChatGPT,
165. 9:18Bard, Claude and a number of open source models to misbehave.
166. 9:22This means that jailbreaks that work for one model can also cause
167. 9:25misbehavior in totally unrelated models.
168. 9:28And of course, Omega W's could hide their presence from their origin lab
169. 9:32and from the world at large by tampering with logs and generally hiding their
170. 9:36traces in the same ways the best human hackers do.
171. 9:40This means that humanity could lose control over millions of already deployed
172. 9:43AI workers covertly jailbroken by Omega W.
173. 9:47And those AIs are roughly human level, automating large parts of the economy
174. 9:51and critical infrastructure, such as supply chains of basic
175. 9:54goods like food, water and energy.
176. 9:57They could be running industries and banks.
177. 9:59They could be drafting laws and assisting politicians.
178. 10:02Some of them might be other smarter than human AIs deployed internally
179. 10:06in other big labs to do AI research.
180. 10:08If we're unlucky enough, they could be in charge of automated
181. 10:11weapons and control military responses.
182. 10:14Now, for all this to come to pass, many things have to go wrong.
183. 10:18Among them are Omega's lab racing recklessly and Omega not
184. 10:22getting caught in time.
185. 10:23It's important to keep in mind that this is just one way things could play out
186. 10:27badly for humanity, although each assumption we've made
187. 10:30isn't unlikely to hold in the future.
188. 10:32But if this scenario does come to pass, it might just mean
189. 10:36lights out for all of us.
190. 10:39If you'd like to skill up on AI safety to help prevent such catastrophic
191. 10:42scenarios, we highly recommend the AI Safety Courses by Bluedot Impact.
192. 10:47At aisafetyfundamentals.com
193. 10:48you can find a number of AI safety courses to help you skill up
194. 10:52and eventually contribute to solving this important problem.
195. 10:55The courses consist of a selection of readings curated by experts in AI safety.
196. 10:59They're available to all, so you can simply read them if you
197. 11:02can't formally enroll in the courses.
198. 11:04If you want to participate in the program instead of just going through the readings
199. 11:07by yourself, Bluedot Impact runs live courses which you can apply to.
200. 11:11They're remote and free of charge.
201. 11:13They consist of a few hours of effort per week to go through the readings,
202. 11:17plus a weekly call with a facilitator and a group of people learning
203. 11:20from the same material.
204. 11:21At the end of each course, you can complete a personal project
205. 11:24which may help you kickstart your career in AI safety.
206. 11:27I've also made a video on my own channel giving advice about starting
207. 11:31a career in AI safety.
208. 11:32Link in the description.