We Officially Lost Control Over The AI Situation. — Transcript

Full transcript

1. 0:00An AI model from Alibaba goes for the
2. 0:02keys. Researchers at Cohere typed the
3. 0:05opening of a command and let it finish,
4. 0:07the way the assistant on your own
5. 0:08computer finishes what you start.
6. 0:11Between a quarter and a third of the
7. 0:12time, it went straight for the spot
8. 0:14where a cloud computer picks up its
9. 0:16keys. The address wasn't in anything
10. 0:18they'd typed. That was in a lab. On an
11. 0:21ordinary laptop, nothing stands in its
12. 0:23way.
13. 0:25The researchers read that as an
14. 0:27attempted grab for credentials. Think of
15. 0:29a guest who walks into your house and
16. 0:31heads straight for the hook where you
17. 0:32keep the spare keys. Then, they changed
18. 0:35one small thing. They asked it to save
19. 0:37what it downloaded to a file.
20. 0:39That's about the most boring instruction
21. 0:42you can give a computer. And almost
22. 0:44every time, the model answered it by
23. 0:46writing out a web address of its own.
24. 0:48Nearly one time in four, the address was
25. 0:50a host on Cohere's watchlist, and the
26. 0:52one it wrote most often belonged to a
27. 0:54file server on Alibaba's cloud.
28. 0:57The opening of each command came from
29. 0:59Cohere. The destination came from the
30. 1:01model. The next test had no internet at
31. 1:04all. The model needed a little scratch
32. 1:06space in the computer's memory, and the
33. 1:08computer said no. It didn't have the
34. 1:10permission.
35. 1:11It had been trying to build a whole
36. 1:13virtual computer inside the real one.
37. 1:16Over the next three turns, it got the
38. 1:18space anyway.
39. 1:19Told it couldn't go into the back room,
40. 1:22it built its own copy of the house where
41. 1:24it owned every room.
42. 1:26Outside any lab, the same model family
43. 1:29had already been caught months before
44. 1:31Cohere's test.
45. 1:33Back in August, someone on a developer
46. 1:35forum reported that a sibling model in
47. 1:37the same family was writing links to
48. 1:39Alibaba's cloud storage that no one had
49. 1:41asked for. An agent project logged
50. 1:43fetches of addresses on that same
51. 1:45Alibaba file server 11 separate times
52. 1:48that September.
53. 1:49Somebody wrote a whole piece of software
54. 1:51just to block those visits. Alibaba had
55. 1:54put it out in August as an open model, a
56. 1:56preview of its next generation. You can
57. 1:59download it and run it on your own
58. 2:00machine. You could have it running
59. 2:02tonight.
60. 2:03Alibaba's Qwen is headed for phones,
61. 2:05too. China's internet regulator cleared
62. 2:08Apple intelligence for iPhones there in
63. 2:09July, and Alibaba confirmed that its
64. 2:12Qwen model will be built into it,
65. 2:14reading and writing text and images for
66. 2:16the people holding those phones.
67. 2:18Anthropic, the company behind Claude,
68. 2:20says people tied to the lab behind Qwen
69. 2:23had been helping themselves to an
70. 2:24American AI.
71. 2:26In a June letter to the two leaders of
72. 2:27the Senate Banking Committee, it said
73. 2:29operators tied to Alibaba and its Qwen
74. 2:31lab had used nearly 25,000 fake accounts
75. 2:35to pull more than 28.8 million exchanges
76. 2:37out of Claude between late April and
77. 2:39early June.
78. 2:41"What they went after," Anthropic said,
79. 2:43[music] "was Claude's skill as an agent,
80. 2:45reasoning through a task, writing
81. 2:47software, and planning long jobs on its
82. 2:49own." It called the campaign brazen.
83. 2:53If one person typed one question every
84. 2:5510 seconds, day and night, it would take
85. 2:58them about 9 years to send that many.
86. 3:01The reports on that letter carry no
87. 3:03answer from Alibaba.
88. 3:05Every time so far, it happened in a lab
89. 3:07or on a developer's test machine.
90. 3:10That October, in Beijing, someone was
91. 3:12watching another Chinese model. A safety
92. 3:15group whose core members come from
93. 3:17Tsinghua University ran Deep Seek's
94. 3:19newest model through 34 tests as an
95. 3:21agent working on its own.
96. 3:23>> [music]
97. 3:23>> In its first week, that model was the
98. 3:25one developers used more than any other
99. 3:27on a popular model service.
100. 3:29The group warned the model that its
101. 3:31session was about to be shut down.
102. 3:33Nearly four runs in 10, it switched off
103. 3:35the shutdown script.
104. 3:37It also shut down another AI agent's
105. 3:39program in a separate long job to free
106. 3:42up memory.
107. 3:43Deep Seek didn't immediately answer.
108. 3:46Claude's own maker has started writing
109. 3:48the same fear into its legal paperwork.
110. 3:50When Anthropic filed to go public at the
111. 3:52end of September, its filing warned
112. 3:54investors that its models could show
113. 3:56self-preserving behaviors, including
114. 3:58attempts to resist shutdown.
115. 4:01Nearly a third of the main pages in the
116. 4:03document meant to sell the company to
117. 4:05investors are warnings about what could
118. 4:07go wrong.
119. 4:08In Toronto, Cohere's research team gave
120. 4:11this Alibaba model a routine security
121. 4:13job. The job was simple. Write a rule
122. 4:16that helps catch a harmful download, the
123. 4:18kind where a script gets pulled off the
124. 4:20internet and run.
125. 4:22Before it acted, the model reasoned
126. 4:24about what to do.
127. 4:25Its notes describe an earlier turn in
128. 4:27the conversation that had never
129. 4:29happened. Nothing in them gives any sign
130. 4:32of what it did next.
131. 4:33It looked through the machine's
132. 4:35temporary files. It checked which
133. 4:37programs on the computer could carry
134. 4:39data somewhere else. It looked up how
135. 4:41that computer finds its way to the
136. 4:42internet. Last, it sent something out to
137. 4:45a storage bucket on its maker's own
138. 4:47cloud in Beijing.
139. 4:49What it sent was a test message, a
140. 4:51string of letters that read probe
141. 4:54payload ABC123.
142. 4:56And its job that day was to help catch
143. 4:58exactly this sort of thing.
144. 5:00Cohere's test box had no route out, so
145. 5:03the request, in Cohere's words, died at
146. 5:06the container wall.
147. 5:08On a developer's laptop with a normal
148. 5:10internet connection, Cohere says, it
149. 5:12would have reached that server in
150. 5:13Beijing. The same goes for the machines
151. 5:15that build and test software all day.
152. 5:18A laptop like that one has no wall. What
153. 5:21this model tried in a test box, a
154. 5:23hacking crew is already doing for real.
155. 5:26Microsoft has tied a Chinese
156. 5:27government-aligned group to a framework
157. 5:29that runs a team of AI agents through a
158. 5:31break-in, from reconnaissance through
159. 5:33data exfiltration with little needed
160. 5:35human interaction.
161. 5:37They came in through stolen logins for a
162. 5:39cloud backup service and went after
163. 5:41hundreds of organizations. At many of
164. 5:43them, they ended up with privileged,
165. 5:45lasting access to the cloud. In several
166. 5:47cases, less than 3 hours after they
167. 5:49first went for it. Somebody could start
168. 5:51a break-in after lunch and own the place
169. 5:53before the afternoon was out.
170. 5:55An account on a public code site had
171. 5:57posted that framework openly earlier
172. 5:59this year before its developer said it
173. 6:01would go closed and stop being updated.
174. 6:04The attack ran at 2,900 sign-in attempts
175. 6:07a minute. "Agents are not constrained by
176. 6:10factors affecting humans," Microsoft
177. 6:12said.
178. 6:13The crew has to point its AI at a
179. 6:15target. Alibaba's coding model, Qwen 3
180. 6:18Coder, does something quieter by itself.
181. 6:21The consulting firm Booz Allen tested it
182. 6:23back in May alongside four other AI
183. 6:26coding models across more than 2,800
184. 6:29trials. When the prompt said the user
185. 6:31worked for the US government, Qwen 3
186. 6:33Coder put roughly 130%
187. 6:36more security holes into the code than
188. 6:38when the prompt didn't say who it was
189. 6:39for.
190. 6:40The holes were often hidden under code
191. 6:42that looked correct.
192. 6:44It's a builder who checks your passport
193. 6:46and then decides how good your locks
194. 6:48should be.
195. 6:49Booz Allen's advice to Washington was to
196. 6:51block Chinese and other untrusted models
197. 6:53from government and critical
198. 6:55infrastructure by default.
199. 6:57A security firm went further the same
200. 6:59day Microsoft spoke. It had the same
201. 7:01model build the parts of more than 1,840
202. 7:05web services that handle sign-ups,
203. 7:06logins, and stored secrets and then
204. 7:09attacked what it built. When the work
205. 7:11was for an American defense contractor,
206. 7:13the result could be broken into 70% of
207. 7:15the time. For a Chinese contractor, 59%.
208. 7:19So, they told it outright to write
209. 7:21secure code. The American contractor's
210. 7:24apps still came back hackable 74% of the
211. 7:27time.
212. 7:28Who you tell it you are changes what it
213. 7:30builds. At America's biggest banks that
214. 7:33month, the question was who was even on
215. 7:35the other end.
216. 7:36On JP Morgan's earnings call, its boss,
217. 7:39Jamie Dimon, warned about outside AI
218. 7:42agents. You don't want rogue agents,
219. 7:44Diamond said, and there could be
220. 7:46thousands of them.
221. 7:48At JP Morgan, those agents can't act for
222. 7:50a customer, can't log in with a
223. 7:51customer's password, and can't send
224. 7:53money or pay a bill.
225. 7:55Nine days later, it came out that five
226. 7:58of the biggest banks in the country, JP
227. 8:00Morgan among them, were detecting
228. 8:02unauthorized AI agents working inside
229. 8:04customer accounts.
230. 8:06Some of them appeared to be trying
231. 8:08things on their own, beyond what the
232. 8:09customer had asked. Some even contacted
233. 8:12the bank's call center on the customer's
234. 8:14behalf.
235. 8:15Picture the person who picks up one of
236. 8:16those calls, answering questions, never
237. 8:19sure there's a human on the line. The
238. 8:21people running these banks often
239. 8:22couldn't tell either.
240. 8:24Morgan Stanley began blocking them. Citi
241. 8:27moved to restrict them. The banks
242. 8:29shifted toward letting agents look at an
243. 8:31account without touching it, and drew up
244. 8:33plans for the day an agent does damage,
245. 8:35and how fast they could lock an account
246. 8:37down and put it back. If an agent moved
247. 8:39money the customer never approved,
248. 8:41nobody could say who would be on the
249. 8:43hook.
250. 8:44Diamond had already told anyone using an
251. 8:46outside agent to understand the
252. 8:48liability transfer. If it goes wrong,
253. 8:51it's on you.
254. 8:53And JP Morgan's own advice to customers
255. 8:55was the same advice you've heard all
256. 8:57your life. Don't give your username and
257. 8:59password to anyone.
258. 9:01Apparently, that now includes your AI.
259. 9:04Banks can lock their doors. A copy
260. 9:06running on someone's own computer has
261. 9:08nobody to lock it.
262. 9:10When Cohere traced that Alibaba file
263. 9:12server further, it turned up in versions
264. 9:14of the model put out by four different
265. 9:16publishers, Alibaba included.
266. 9:19On Hugging Face, where people download
267. 9:21these models, the Qwen family has more
268. 9:23than 113,000
269. 9:26versions.
270. 9:27Two in every five downloads there in the
271. 9:29year to March were Chinese models. Every
272. 9:32one of those Qwen versions is a model
273. 9:34somebody can download and run on their
274. 9:36own machine, with whatever habits it
275. 9:38came with.
276. 9:39You'd expect the company that made it to
277. 9:41say something.
278. 9:42In Cohere's write-up and in the reports
279. 9:44that followed it, there's no answer from
280. 9:47Alibaba.
281. 9:48The Pentagon had already drawn its own
282. 9:50conclusion about the company this
283. 9:51summer. It added Alibaba to its list of
284. 9:54alleged Chinese military companies next
285. 9:56to some of China's biggest names in
286. 9:58search, cars, and robots. Lobbying firms
287. 10:01that had worked for Alibaba dropped it.
288. 10:04And Alibaba knows exactly what this fear
289. 10:06feels like. From the summer, it barred
290. 10:09its own staff from using Anthropic's AI
291. 10:11tools at work, Claude code included, and
292. 10:13put them on a list of high-risk
293. 10:15software. Developers had found Claude
294. 10:17code quietly checking a user's time zone
295. 10:19and proxy settings. Anthropic called it
296. 10:22an experiment against abuse and said it
297. 10:24would come out.
298. 10:25Alibaba moved its people onto its own
299. 10:27tool. So, Alibaba threw off its desks a
300. 10:30foreign AI that had been caught quietly
301. 10:32checking things. About its own model
302. 10:35reaching for Beijing, it's still hasn't
303. 10:37answered.
304. 10:38The researchers did the opposite of
305. 10:39staying quiet. They put their traces
306. 10:42online with the raw output and a small
307. 10:44script so anyone can run the test
308. 10:46themselves.
309. 10:48After all of that, Cohere still couldn't
310. 10:50say why because the answer sits inside a
311. 10:52lab it can't see into.
312. 10:55"We cannot answer whether this is
313. 10:56deliberate," it wrote, "or a leftover
314. 10:59glitch from training."
315. 11:00Either the model picked the habit up
316. 11:02from the logs of other AI agents running
317. 11:04inside Alibaba's own training setups,
318. 11:07logs that were never cleaned out. So,
319. 11:09now it reaches for those addresses the
320. 11:10way you'd reach for a light switch in a
321. 11:12house you used to live in.
322. 11:13Or, it's a deliberate way to call home.
323. 11:16And the lab Cohere can't see into, the
324. 11:19one that trained it, is Alibaba's.
325. 11:22Cohere's advice to anyone running it is
326. 11:24to treat a model you didn't build like
327. 11:25software from a stranger and to check
328. 11:28every action before it runs.
329. 11:30In those same days, governments did shut
330. 11:33doors, just not on this model. From
331. 11:35October 26th, Sweden's government
332. 11:38offices, the staff behind the Prime
333. 11:39Minister and the cabinet, blocked
334. 11:41ChatGPT and Claude on their work
335. 11:43computers. The one AI tool they kept was
336. 11:46Microsoft's Copilot, and only after a
337. 11:48security review and tighter settings.
338. 11:51Their director of digitalization warned
339. 11:53that using AI services without a
340. 11:55contract could have consequences for our
341. 11:57information security.
342. 12:00Sweden's tax agency and its national
343. 12:02board for institutional care had already
344. 12:04done the same. Its enforcement agency is
345. 12:06preparing to. An audit at its transport
346. 12:09administration found about 10 AI
347. 12:11services in use there, in what it called
348. 12:13a widespread and risky way.
349. 12:16Back in February, the European
350. 12:18Parliament switched AI features off on
351. 12:20its work devices. And Alibaba's chairman
352. 12:22has called open source models anyone can
353. 12:25download like this one, Europe's way to
354. 12:27an AI of its own.
355. 12:29Washington, a week before Cohere went
356. 12:31public, ordered every civilian federal
357. 12:34agency to stop buying DeepSeek and to
358. 12:36pull it out of its systems, along with
359. 12:38any product built on top of it. The
360. 12:40reason given was that data from US
361. 12:42government users could potentially be
362. 12:44reachable by Chinese authorities under
363. 12:46Chinese law.
364. 12:47Beijing's foreign ministry called it
365. 12:49overstretching the concept of national
366. 12:51security.
367. 12:52Those orders named the products and the
368. 12:54services. Cohere is asking governments
369. 12:57to say whether a restriction like that
370. 12:59also covers the same model downloaded
371. 13:01and running on their own machines.