Gemini 4 Argon — Transcript

Full transcript

1. 0:00Okay, so Google has finally announced
2. 0:03Gemini 4. So, this is actually called
3. 0:05Gemini 4 Argon. I should preface this by
4. 0:08saying this is not out yet. This is
5. 0:09still in testing, but since Google's
6. 0:11actually announced this, I'm going to go
7. 0:13through some of the key things that make
8. 0:15this a really interesting model. So,
9. 0:17first off, contrary to what a lot of
10. 0:18people are saying, this is definitely a
11. 0:20frontier model in its intelligence. And
12. 0:22I'll go through some of the benchmarks
13. 0:23in a second and talk about that. For me,
14. 0:26the probably most interesting thing
15. 0:27about this is that this can actually
16. 0:29generate a million tokens in a single
17. 0:32response. So, that's roughly eight times
18. 0:34what any of the other frontier models
19. 0:36will let you output right now. But, that
20. 0:39million token ceiling there isn't the
21. 0:41only number I think you should be paying
22. 0:43attention to. I think the biggest thing
23. 0:45with this model and the number that
24. 0:46matters most is how many tokens it
25. 0:49actually uses to get the task done. And
26. 0:52really, that's where this model gets
27. 0:53interesting. We know, for example, a lot
28. 0:56of the recent models, including things
29. 0:58like Gemini 3.6 Flash, 3.7, had issues
30. 1:02where they were just outputting very
31. 1:05large amounts of tokens to actually get
32. 1:07a task done. And that's one of the
33. 1:09things that's actually measured by an
34. 1:11independent party with the artificial
35. 1:13analysis's benchmarks. So, I'll go into
36. 1:15those in a second. But, first, let me
37. 1:17set the context of what's actually going
38. 1:19on here. So, Argon is the first Gemini 4
39. 1:22model. And really, Google is pitching
40. 1:24this as a model that can do long horizon
41. 1:26coding, knowledge work, and be a sort of
42. 1:30key model for cybersecurity defense
43. 1:32there. Currently, this is rolling out
44. 1:33just to selected users, but you should
45. 1:35expect broader availability in the
46. 1:37not-too-distant future. Now, artificial
47. 1:40analysis have already run this through
48. 1:42their intelligence index, and it scores
49. 1:4453 there. So, that's basically the same
50. 1:46as sort of GPT Astra at max reasoning
51. 1:49level. And just slightly 1. ahead of the
52. 1:52GPT 6.1 Sol, which which announced at
53. 1:55DevDay yesterday. But, I guess in some
54. 1:57ways for Google most importantly that
55. 1:59this is 23 points ahead of Gemini 3.1
56. 2:04Pro preview, which was their last
57. 2:06frontier Pro model. And like I mentioned
58. 2:08in other videos, that's 7 months that
59. 2:10they didn't actually release a sort of
60. 2:12pro or above standard model. Now, we've
61. 2:15seen them release multiple flash models,
62. 2:18and to be really honest, in many ways
63. 2:19that's their bread and butter. Industry
64. 2:22tends to like the much cheaper, much
65. 2:24faster flash models that still come with
66. 2:27the Google and the Gemini pedigree. And
67. 2:29I guess the other big thing with Argon
68. 2:31is that this finally puts Google back in
69. 2:34the top three labs on intelligence. All
70. 2:37right, first off, let me go back to the
71. 2:38million token output. So, Gemini 3.8
72. 2:41flash topped out at around 64K tokens
73. 2:45out. Opus 5.5, Fable 5.1, Astra, they
74. 2:50all cap out at 128K.
75. 2:52So, Argon goes to a million tokens out
76. 2:55in one shot. Now, why does that matter?
77. 2:58The biggest reason I think is that it
78. 3:00just means that you stop losing things
79. 3:02at the seam. When the model sort of hits
80. 3:04the output cap halfway through a hard
81. 3:06task, your harness has to sort of stop,
82. 3:09then often summarize or compact
83. 3:11something, and then kick it off again.
84. 3:13And every time you do anything like
85. 3:15that, you're throwing away a lot of
86. 3:16detail. Now, often there's benefit for
87. 3:18doing it in that you're saving money, et
88. 3:20cetera. But, as we're starting to see
89. 3:22caching prices of models go way down,
90. 3:25then the temptation is actually to be
91. 3:27able to generate more out in one shot.
92. 3:29So, the idea with a million tokens is
93. 3:31that everything that the model has
94. 3:33generated stays in view for a whole run.
95. 3:37The second advantage of this is it now
96. 3:38means that some of the thinking tokens
97. 3:41or the long chain of thought can
98. 3:43actually be a lot longer in what they're
99. 3:46generating out. And in the past, that
100. 3:48meant that models with a lower cap were
101. 3:50tending to sort of rush their thinking,
102. 3:52or it was getting cut off in the
103. 3:53generation. Now, of course, we know that
104. 3:55that doesn't mean you're always going to
105. 3:56get to see that thinking, but you do
106. 3:59hope that you'll eventually benefit from
107. 4:00the output of that thinking. The third
108. 4:03advantage of having the million tokens
109. 4:05out is that simply the model can now
110. 4:07generate much bigger artifacts in one
111. 4:10place. So, if you need a full module
112. 4:12rewrite, you need some kind of migration
113. 4:15done, you need to try and generate your
114. 4:16Minecraft game in one pass, the model's
115. 4:19going to be better at doing those kinds
116. 4:21of things because it's going to be able
117. 4:23to retain variable names and other key
118. 4:25bits in that pass as it goes along. Now,
119. 4:29Google actually gave an example of this
120. 4:30kind of work with a video decoder, where
121. 4:33Argon Agents ran many rounds of
122. 4:35experiments to rewrite a code base fully
123. 4:38new in Rust to make it 2.7 times faster.
124. 4:41Now, on the practical side, a run like
125. 4:43this is obviously going to take a while.
126. 4:45Google hasn't published generation
127. 4:47speeds for this, but if it's something
128. 4:49like 100 tokens a second, a million
129. 4:52tokens is going to be close on 3 hours
130. 4:56of decoding. So, perhaps that even hints
131. 4:58that we're going to get faster decoding
132. 5:00from this model than we've seen from the
133. 5:02previous Pro models. We know that the
134. 5:043.8 Flash is doing around 300 tokens per
135. 5:07second. And obviously with OpenAI's
136. 5:10announcement of Ultrafast, there's
137. 5:12definitely going to be a movement for
138. 5:13people wanting faster decoding. Another
139. 5:16big advantage of that is that as you're
140. 5:17getting smarter models, your harness can
141. 5:20actually get simpler in how it actually
142. 5:23does things. Because if the model can do
143. 5:25more in sort of one shot, it needs to
144. 5:27have less loops of actually going
145. 5:29through the model. Now, of course, there
146. 5:31are a few caveats here. If you're
147. 5:32building sort of tool-heavy agents, this
148. 5:35is going to matter less than it sounds
149. 5:36because those usually work across a lot
150. 5:38of separate calls anyway. All right, the
151. 5:40next thing up comes from the Artificial
152. 5:42Intelligence Index. And this is not just
153. 5:45about the raw intelligence of the model.
154. 5:47Over the last 6 to 9 months, pretty much
155. 5:50all of the labs have been working on
156. 5:52shorter chains of thought that can do
157. 5:55the same amount of intelligence or do
158. 5:57better. Getting models to reach the same
159. 5:59answer within a few tokens is starting
160. 6:01to become more important than just
161. 6:04purely having the best model with the
162. 6:05most amount of intelligence. And that
163. 6:07makes sense because if you're running
164. 6:08these models in production or for doing
165. 6:11certain tasks, you really don't care
166. 6:13about how many tokens the model outputs.
167. 6:16You care about how quickly can it
168. 6:17actually do my task? In fact, if you
169. 6:19think about it, you're kind of paying
170. 6:21per task. And so the big win here is
171. 6:24that fewer tokens per tasks means lower
172. 6:26cost and lower latency. And this is my
173. 6:29one big issue with 3.6 flash was that
174. 6:32that while the model was very smart, it
175. 6:35just chewed up so many tokens to get to
176. 6:38its answers. So you can kind of think of
177. 6:40this cost per intelligence index here is
178. 6:44like the hidden pricing of a model. So
179. 6:46Igon itself is $2.00 in and $10.00 out
180. 6:50per million tokens at launch. So that's
181. 6:52a 50% launch discount for this. And the
182. 6:55cached input is 95% off. And I would say
183. 6:59that unless things dramatically change,
184. 7:01you could think of that is probably
185. 7:02going to be the price of this model. In
186. 7:05that by the time that launch discount
187. 7:07runs out, there will be newer models
188. 7:09that are coming out to replace this. And
189. 7:11you could think of that really as
190. 7:12penalizing companies that stay on a
191. 7:14model and not continually updating to
192. 7:16the latest model. Anyway, at those
193. 7:18prices, Artificial Analysis puts this at
194. 7:21$1.99 per task, about 60% of Astra's
195. 7:25$3.26.
196. 7:27And they're pretty clear that that
197. 7:28advantage comes from the lower token
198. 7:30price, not from using fewer tokens. If
199. 7:33it's just purely on the number of
200. 7:35tokens, then this model is roughly 1.2
201. 7:37times Astra. And I guess I should point
202. 7:40out that while it's coming in cheaper
203. 7:41than GPT-6 Astra there, it's still about
204. 7:452.7 times the cost of GPT-6.1 Soul at
205. 7:48max. All right, now the other really
206. 7:50interesting thing that Artificial
207. 7:51Analysis results show is hallucinations.
208. 7:55So, according to the eval for this,
209. 7:57Argon has a 15% hallucination rate.
210. 8:00Astra is at 51% and GPT-6.1 Soul is at
211. 8:0454%. So, basically it shows that
212. 8:06DeepMind has done a lot of work in
213. 8:08getting Argon to just be much more
214. 8:10likely to tell you that it doesn't know
215. 8:13than to make something up. Now, the
216. 8:14trade-off at the moment in that
217. 8:15benchmark is that actually the accuracy
218. 8:18is lower than Astra on that benchmark.
219. 8:20So, if you look at the overall
220. 8:22omniscience scores, you end up at
221. 8:24roughly about the same. But, I do think
222. 8:26for a lot of production work, it's
223. 8:27actually much more useful to have a
224. 8:29model that says, "I don't know." than to
225. 8:30be confident about a wrong answer.
226. 8:33Another interesting thing that
227. 8:34Artificial Analysis points out here is
228. 8:36that this model is clearly better at
229. 8:37agentic work. It's number one on the
230. 8:39automation bench at 77.5%.
231. 8:42Unfortunately, on terminal bench four,
232. 8:44it's at 57%,
233. 8:46which while it's still a massive jump up
234. 8:48from 3.1 Pro, it's still a bit behind
235. 8:51Sonic 5.5 and Opus 5.5 and Astra there.
236. 8:54All right, so just to give this video
237. 8:56short, I'll finish up by saying Argon is
238. 8:58a strong model and the million tokens of
239. 9:00output opens up some genuinely new ways
240. 9:03to think about long-running tasks. A lot
241. 9:06of the tasks that you've seen people
242. 9:08doing demos with on Opus 5.5 and some of
243. 9:11the OpenAI models, Argon is definitely
244. 9:13at a similar level to that kind of
245. 9:15model. And I do think a lot of people
246. 9:17are going to enjoy using this model.
247. 9:18From what I'm hearing is not too far off
248. 9:20into the distant future. So, when it's
249. 9:22out and I am allowed to, I will
250. 9:24certainly be showing some demos of this
251. 9:26and showing you some of the testing of
252. 9:27the really long runs and how long they
253. 9:29take and what cost per task, etc. that
254. 9:32we're seeing on real-world tasks.
255. 9:34Anyway, if If to find out more about
256. 9:36this model, I've got links in the
257. 9:37description to the blog post about it.
258. 9:39They've got Google's own benchmarks in
259. 9:41there, which I didn't cover in this
260. 9:43video. And let me know in the comments
261. 9:44how you feel about this model coming
262. 9:46out, what you think are going to be the
263. 9:47most interesting use cases, etc. For me
264. 9:49personally, I think just having another
265. 9:51company that's pushing both OpenAI and
266. 9:53Anthropic constantly is always going to
267. 9:56be a big win. And even if Google's not
268. 9:58going to be the most intelligent model
269. 9:59out there, it's definitely good to know
270. 10:01that they've got a model that's back in
271. 10:03the game here. So anyway, as always, if
272. 10:05you found the video useful, click like
273. 10:07and subscribe, and I will talk to you in
274. 10:08the next video. Bye for now.