Two AIs Learn to Collaborate to Fight Zombies — Transcript

Full transcript

1. 0:00In the previous episode, we met Gary,
2. 0:02our AI gladiator. He was able to fight
3. 0:05up to 16 zombies at the same time thanks
4. 0:08to a fearsome technique, a perfect parry
5. 0:10followed by a backstab.
6. 0:12>> [groaning]
7. 0:13>> But despite his prowess, Gary ended up
8. 0:15overwhelmed by the horde.
9. 0:21So today, he won't be fighting alone in
10. 0:23the arena anymore.
11. 0:25It's time to get a little help.
12. 0:27And that help is Larry.
13. 0:35Thanks to everyone for all the name
14. 0:37ideas in the comments.
15. 0:39It was very inspiring. Now, the only
16. 0:42question is whether they can work
17. 0:44together to take on the horde.
18. 0:47What brilliant strategies could possibly
19. 0:49emerge from these two incredible
20. 0:51fighters?
21. 0:56For deeper immersion, I completely
22. 0:58reworked the arena's ground system. The
23. 1:00sand is made up of hundreds of thousands
24. 1:02of particles that can be moved and
25. 1:04tinted. I also fully overhauled the
26. 1:07physics of the virus projections.
27. 1:13A diffusion mechanism lets all the
28. 1:15traces dissipate naturally.
29. 1:18What I really like is that the arena
30. 1:20floor tells a story all on its own.
31. 1:23Now, let's see how our two heroes do in
32. 1:25this all new environment. There's
33. 1:27nothing like a good zombie fight to
34. 1:29break the ice.
35. 1:33After 2 minutes, progress is looking
36. 1:35pretty good. Let's see how the boys are
37. 1:37doing.
38. 1:45>> [groaning]
39. 1:46>> Unfortunately, Gary has never had a
40. 1:47friend and he's not quite sure how to
41. 1:50handle it. Even worse, his brain isn't
42. 1:53wired for friendship.
43. 1:55Gary will need to learn trust and love
44. 1:57if he's going to succeed.
45. 1:59I had to rewrite the entire simulation
46. 2:01to introduce the concept of allies and
47. 2:04make the whole system more generic.
48. 2:07Until then, I was using ray casting for
49. 2:09perception. The advantage is that the
50. 2:12number of observations is fixed, but
51. 2:14there are several drawbacks.
52. 2:16The number of inputs is very high with
53. 2:18either many missing values or a huge
54. 2:20amount of redundancy. On top of that,
55. 2:23[music]
56. 2:24detection is also quite imprecise.
57. 2:27The new system is based on the proximity
58. 2:29of surrounding entities. The 10 nearest
59. 2:32ones are included and fed into the
60. 2:34neural network. Perhaps a mix of the two
61. 2:36systems could be used. This is something
62. 2:39I'm still exploring.
63. 2:40Of course, the rewards also need to be
64. 2:43adjusted to explicitly penalize attacks
65. 2:45between allies. Let's run the training
66. 2:47again hoping that we can observe the
67. 2:49early signs of collaboration. AI is
68. 2:52playing a growing role in my projects,
69. 2:54especially when it comes to reviewing my
70. 2:56code or researching deep technical
71. 2:58topics like how to adapt PPO to a
72. 3:01multi-agent problem. The challenge is
73. 3:03that different AI models excel at
74. 3:05different tasks and subscriptions are
75. 3:07not cheap. That's why I use Mammoth AI,
76. 3:10which brings a wide range of models
77. 3:12together under one subscription. It
78. 3:14gives me much more flexibility in how I
79. 3:16use my credits.
80. 3:18For my projects, I mainly use deep
81. 3:20search to quickly get up to speed on a
82. 3:22topic or exploit in more depth.
83. 3:24Perplexity is great for this, delivering
84. 3:26detailed well-sourced reports in just a
85. 3:29few minutes.
86. 3:30Opus 5.5 excels at coding questions and
87. 3:33explaining research papers, while GPT-6
88. 3:36is my go-to [music] for lighter, quicker
89. 3:38research. Mammoth gives me access to all
90. 3:41of these services for $10 a month.
91. 3:44You can also create custom Mammoths for
92. 3:46recurring tasks. For example, I've set
93. 3:48up a code reviewer, a researcher, and an
94. 3:50English translator.
95. 3:52If you want to cut your subscription
96. 3:54costs while accessing all the frontier
97. 3:56models, try Mammoth starting at $10 a
98. 3:59month when billed annually. Thanks again
99. 4:01to Mammoth AI for their support.
100. 4:04Let's get back to training and see
101. 4:06whether the new reward improves the
102. 4:08situation. This interface let me monitor
103. 4:10training progress, especially through
104. 4:12the event logger, which provides a
105. 4:14glimpse into the agent's capabilities.
106. 4:17>> [music]
107. 4:17>> After 100 iterations, let's quickly
108. 4:20check whether our two companions are at
109. 4:22each other's throats before we continue
110. 4:24training.
111. 4:28>> [groaning]
112. 4:31>> It's obviously still far from perfect,
113. 4:34but things between them already seem
114. 4:36much calmer. We can also see that
115. 4:38they're managing to pull off a few
116. 4:39perfect parries. Let's hope it stays
117. 4:42that way and get back to training.
118. 4:56The perfect parries seem to be pretty
119. 4:58well mastered and we're starting to see
120. 5:00a few backstabs from time to time. Let's
121. 5:02take a closer look.
122. 5:22It's definitely much better already.
123. 5:25Gary and Larry are far less reckless and
124. 5:27manage to block many more hits. However,
125. 5:30they are quite uncomfortable when they
126. 5:31get close. It feels like they are
127. 5:33getting in each other's way.
128. 5:35Clearly,
129. 5:36basic safety practices are not being
130. 5:39followed at all and inevitably an
131. 5:41accident eventually happens.
132. 5:48Despite a few minor mistakes and thanks
133. 5:50to Larry's courageous sacrifice,
134. 5:52they make it through the wave. That's a
135. 5:54good start, but there's still a long way
136. 5:57to go.
137. 6:15>> [groaning]
138. 6:22>> After 1,200 iterations, progress is
139. 6:25significant. Barriers are becoming more
140. 6:27frequent, and our gladiators clear the
141. 6:29wave without any trouble. However, I
142. 6:32think there's a problem with the
143. 6:33rewards. Nothing really compensate for
144. 6:35the prospect of hurting an ally. Staying
145. 6:38close doesn't do much for the agents
146. 6:40other than giving them a chance to try
147. 6:42to steal each other's points.
148. 6:44>> [groaning]
149. 6:45>> To counterbalance this, [music] I
150. 6:47decided to add an assist mechanic. Two
151. 6:49situations qualify as an assist.
152. 6:51Attacking a zombie that has been stunned
153. 6:53by an ally, or attacking an enemy that
154. 6:56is targeting an ally. In addition, any
155. 6:59elimination achieved during an assist
156. 7:00grants the same rewards to both agents.
157. 7:03To force Gary and Larry to target the
158. 7:05same enemy, I'll start by having them
159. 7:07face a single zombie. That will prevent
160. 7:09them from splitting up at the start of
161. 7:11the fight if the zombies are not grouped
162. 7:13together.
163. 7:15Let's take a quick look at where things
164. 7:17stand after 200 iterations.
165. 7:32The agents have turned into a pair of
166. 7:34zombie-bashing bullies.
167. 7:36It's an original strategy, though
168. 7:38unfortunately, it also involves beating
169. 7:40up their allies. And it isn't very
170. 7:43effective at racking up eliminations.
171. 7:45When it come to proximity, however, it's
172. 7:48a resounding success.
173. 7:50Let's see whether we can develop a
174. 7:51slightly more refined approach going
175. 7:53forward.
176. 7:57Soon after, eliminations start chaining
177. 7:59together, and the score literally
178. 8:01skyrockets.
179. 8:17>> [groaning]
180. 8:26>> The progress really is impressive.
181. 8:29The agents are far more effective at
182. 8:30eliminating threats, and we can see that
183. 8:33they show a great deal of restraint
184. 8:35toward one another.
185. 8:53Progress will continue more slowly until
186. 8:56it reaches a plateau after 600
187. 8:57iterations. Let's see how that goes.
188. 9:14The method is now finally honed, and the
189. 9:17zombies barely get a chance to do
190. 9:19anything. There is the occasional bit of
191. 9:21friendly fire between Gary and Larry,
192. 9:24but that's surely just good camaraderie.
193. 9:26It's almost become too easy. Let's bump
194. 9:28up the number of zombies they're facing.
195. 9:33Let's see how they fare against four
196. 9:36zombies.
197. 9:37They do attempt to cover each other, but
198. 9:40the two fighters are quickly overwhelmed
199. 9:42and eliminated within seconds.
200. 9:45It's a rather tough start for them, but
201. 9:47it's also a situation they've never
202. 9:49faced before.
203. 9:51Let's give them a little time to get
204. 9:53used to this new setup.
205. 9:57Fairly quickly, the agents adapt and the
206. 9:59score shoots back up. However, what's
207. 10:02happening here is a textbook example of
208. 10:05the deviousness of optimization
209. 10:07algorithms. They're a bit like evil
210. 10:09genies.
211. 10:10We ask them for something and they grant
212. 10:13it in the absolute worst way possible.
213. 10:16Here's what Gary does when asked not to
214. 10:18hurt his teammate during fights.
215. 10:24>> [groaning]
216. 10:27>> Uh
217. 10:28no ally
218. 10:30no damage to ally.
219. 10:32And this is not a one-off. It's
220. 10:35systematic.
221. 10:38On top of that, he's actually pretty
222. 10:39good. He's quite capable of controlling
223. 10:41the group of zombies and then stabbing
224. 10:43them one by one. Once we bumped it up to
225. 10:46four zombies, paying the price of
226. 10:48betrayal became more profitable. He
227. 10:50could rack up points four times as fast
228. 10:53rather than share them and risk getting
229. 10:55constantly punished whenever an accident
230. 10:57happened.
231. 10:58At that point, I was a bit stuck because
232. 11:01I couldn't find a way to get the agents
233. 11:03to cooperate. My attempts had even had
234. 11:06the exact opposite effect. The missing
235. 11:08piece came from an OpenAI paper about
236. 11:10their famous Dota playing AI.
237. 11:13Of course, the agents had to work
238. 11:15together and one of the things OpenAI
239. 11:17tried was giving them a shared reward.
240. 11:20That should solve my problem. If Gary
241. 11:22takes out an ally, he misses out on a
242. 11:24big reward bonus.
243. 11:26With this new system, after 200
244. 11:28iterations, there were no dubious
245. 11:31strategies. They seem to be playing by
246. 11:33the rules.
247. 11:35However, there is no real attacks or
248. 11:37signs of cooperation, but it's still
249. 11:40very early.
250. 11:41>> [groaning]
251. 11:50>> Very quickly, the attacks begins to
252. 11:52reappear.
253. 11:53But the most interesting part is yet to
254. 11:55come.
255. 12:04Something very cool is happening here.
256. 12:06Gary pushes an enemy towards Larry, who
257. 12:09can then eliminate it instantly.
258. 12:12In fact, this is the first glimmer of
259. 12:15teamwork. Finally.
260. 12:30This pattern clearly didn't escape them,
261. 12:33and even though it's still quite
262. 12:34hesitant, the intention is now
263. 12:37unmistakable.
264. 12:56After 1,500 iterations, their mastery is
265. 13:00much better, and they clear the wave
266. 13:02with ease.
267. 13:03However, the agents are clearly not very
268. 13:06economical with their attacks. They just
269. 13:08strike widely and continuously. To force
270. 13:11them to be more precise, I added an
271. 13:13energy pool that every action draws
272. 13:15from. This new addition clearly didn't
273. 13:18sit well with our two heroes, who
274. 13:20quickly ran out of energy and were swept
275. 13:23aside.
276. 13:31>> [groaning]
277. 13:33>> Adaptation is very quick, however, and
278. 13:35they are becoming much more economical
279. 13:37with their attacks.
280. 13:55A thousand iterations later, they have
281. 13:58become killing machines.
282. 14:02To make things more challenging, I'll
283. 14:04double the energy cost of attacks and
284. 14:06half the regeneration rate. Successfully
285. 14:09performing a perfect parry restores 20%
286. 14:12of the energy.
287. 14:18Clearly, we're going to have to learn to
288. 14:20parry a bit more.
289. 14:27This time, adaptation takes longer.
290. 14:29After 600 iterations, they are many
291. 14:32learning to buy their time and operate
292. 14:34more efficiently.
293. 14:35>> [groaning]
294. 14:44[groaning]
295. 14:47>> Gary, however, pulls off a super perfect
296. 14:49parry at the end, which is an excellent
297. 14:52signal.
298. 14:55>> [groaning]
299. 15:01>> A little later, they clearly grasped the
300. 15:04value of the perfect parry and use it
301. 15:05whenever they run low on energy. The
302. 15:08execution is still a bit hesitant, but
303. 15:10it's an excellent direction.
304. 15:21>> [groaning]
305. 15:26[groaning and screaming]
306. 15:32>> 13 minutes later, the strategy is nearly
307. 15:35flawless. The two cooperate seamlessly
308. 15:38and dispatch eight zombies with ease.
309. 15:41From here on, I'll disable all secondary
310. 15:43rewards and keep only eliminations. So,
311. 15:47they stay focused on what matters most.
312. 15:49I'll also gradually add more zombies to
313. 15:51see just how far Gary and Larry can go.
314. 16:38>> [groaning]
315. 17:17>> Uh!
316. 17:19Huh!
317. 17:22Uh! Huh!
318. 17:23Uh!
319. 17:25Huh!
320. 17:28Huh!
321. 17:29Huh!
322. 17:30Uh! Huh!
323. 17:34>> [groaning]
324. 17:34>> Huh!
325. 17:36Huh!
326. 17:36Huh!
327. 17:40Huh! Uh!
328. 17:43Uh!
329. 17:46>> [groaning]
330. 17:46>> Huh!
331. 17:48Uh!
332. 17:50Uh!
333. 17:52>> [groaning]
334. 17:53>> Uh!
335. 17:57Huh!
336. 18:00Huh!
337. 18:00>> [groaning and screaming]
338. 18:03>> Huh!
339. 18:05Huh!
340. 18:08Uh!
341. 18:09Huh! Huh!
342. 18:12Huh!
343. 18:15Huh!
344. 18:18Uh!
345. 18:21Uh!
346. 18:22Uh!
347. 18:25Uh!
348. 18:27Huh!
349. 18:28>> [groaning]
350. 18:28>> Huh!
351. 18:30Uh!
352. 18:32Uh!
353. 18:35Uh!
354. 18:45Uh!
355. 18:48Uh!
356. 18:51Uh!
357. 18:53HUH!
358. 18:56HUH!
359. 19:01>> [groaning]
360. 19:01>> HUH!
361. 19:02UH!
362. 19:05HUH!
363. 19:12HUH!
364. 19:14HUH!
365. 19:15>> [groaning]
366. 19:20[groaning]
367. 19:29[groaning]
368. 19:39[groaning]
369. 19:45[groaning]
370. 19:58[groaning]
371. 20:46[groaning]
372. 20:52>> GETTING THE TWO AGENTS TO collaborate
373. 20:54was no small feat. Still, I'm very
374. 20:56pleased with the result.
375. 20:58And I would like to try it with far more
376. 21:00than just two fighters.
377. 21:02Also, for the next stage of the project,
378. 21:04I'm looking for someone to create
379. 21:06sprites. Anyone with skills in sprite
380. 21:08animation, pixel art or otherwise is
381. 21:11welcome to contact me on Discord or by
382. 21:13email. All of the details are in the
383. 21:15description.
384. 21:16A huge thanks to all the Patreon members
385. 21:19for their support and to Mammoth AI.