YouTube2Text

Post-Training Gemini 4 for Real Engineering Work — Melvin Johnson, Google DeepMind — Transcript

by AI Engineer · 2,802 words · 401 segments · language en · Watch on YouTube

Full transcript

  1. 0:12Hey everyone, good to be here. I'm
  2. 0:14Melvin and I lead post training for
  3. 0:16Gemini. So today I thought I'll take you
  4. 0:18through a little journey of how we post
  5. 0:20train Gemini 4. And our journey begins
  6. 0:25by people calling us not a frontier
  7. 0:27model. Um, so here's an example. You
  8. 0:30know, people say Gemini is cooked, but
  9. 0:33GCP is cooking. For all intents and
  10. 0:36purposes, we believe Deep Mind is no
  11. 0:38longer a frontier lab. Ethan Malik, who
  12. 0:41we respect a lot, he says in his own
  13. 0:44nice way, pretty long gap between
  14. 0:47launches is what he calls us.
  15. 0:50And I'm here to tell you we're not dead.
  16. 0:54We're here. uh and we have launched uh
  17. 0:59Gemini 4 Argon, we announced Carbon and
  18. 1:03we launched Carbon all within the last
  19. 1:0528 days. So we're on a we're you know
  20. 1:09we're on a run here and that's not it.
  21. 1:12Uh in the process we actually garnered
  22. 1:16an apology from Dylan. No, he says
  23. 1:19official apology to Google. Uh they're
  24. 1:22actually cooked. They're not cooked. Um
  25. 1:25Ethan says, you know, one of the most
  26. 1:27significant aspects of Gemini 4 is the
  27. 1:30speed at which we're launching things.
  28. 1:32Um so we've gone 236 days without having
  29. 1:35a Frontier model and we've had two
  30. 1:37Frontier models in 28 days. That's a 10x
  31. 1:40improvement in our speed of shipping.
  32. 1:43And you know, people call us Google kind
  33. 1:47of cooked, but Google's not cooked. So
  34. 1:49it's good um to see see that sentiment.
  35. 1:52It's glad to be on the other end. It's
  36. 1:54also good to be on the receiving end of
  37. 1:56the bad comments because it pushes you
  38. 1:58to do better.
  39. 2:00I want to start by playing a little
  40. 2:03video of a bunch of things that uh
  41. 2:05Carvin has built.
  42. 2:39Thank you.
  43. 2:42Now, kudos to the model for building
  44. 2:44that video uh itself. So you saw great
  45. 2:46examples of the model doing much better
  46. 2:49at 3D model building assets using
  47. 2:52Blender. There's lots of games in there.
  48. 2:55The real world use cases, model
  49. 2:56processing, multimodel documents. Uh but
  50. 2:59you know, it's not just the uh the
  51. 3:02demos. Carbon's actually number one on a
  52. 3:05bunch of real world customer use cases.
  53. 3:08And one of the things that we're really
  54. 3:09excited about is, you know, it's not
  55. 3:11just uh really good on these benchmarks,
  56. 3:14it's actually much cheaper. Um, so on
  57. 3:17Cursor Bench, it's the number one model
  58. 3:20today and it's 3.3 times cheaper than
  59. 3:22the second best model. On Perplexity's
  60. 3:25Wander, which is a benchmark for deep
  61. 3:27research um queries, it's the best
  62. 3:30model. you know, you saw um competitor
  63. 3:33models come out were 9x cheaper than the
  64. 3:36second best model there. And on
  65. 3:38automation bench, which is Zapier's uh
  66. 3:41benchmark, again, we're number one while
  67. 3:43being two times um cheaper per task.
  68. 3:48And that's not it. I mean, today we
  69. 3:51announced the new element uh to our
  70. 3:54Gemini 4 series, and it's not surprise.
  71. 3:57Uh it's uh hydrogen. Um, so you know,
  72. 4:02you might have seen the launch earlier
  73. 4:03today. Gemini 4 hydrogen is where we're
  74. 4:06trying to bring the Gemini 4
  75. 4:08intelligence to everyone. It's fast,
  76. 4:11it's dependable, it's launched on all
  77. 4:14surfaces on anti-gravity, on the Gemini
  78. 4:16API, in the Gemini app, and on search.
  79. 4:20Um, and you know I have a graph that
  80. 4:22shows the AA index with many of you
  81. 4:25might be familiar with and time on the
  82. 4:28x-axis. You can see it's one of the
  83. 4:30fastest models for that level of
  84. 4:33intelligence. And to give you a sense of
  85. 4:36how far you know Gemini has come, 3.8
  86. 4:40flash on terminal bench 4 was about 20%.
  87. 4:45And hydrogen is about 60%. So we're
  88. 4:48three times better performant while
  89. 4:51being three times faster, right? So
  90. 4:54that's no easy task and we're, you know,
  91. 4:56really happy to bring that to you in
  92. 4:59your hands so you can build things with
  93. 5:01it.
  94. 5:02And that completes, you know, our uh
  95. 5:06Gemini 4 series. So we have Gemini 4
  96. 5:09carbon a frontier model for deep
  97. 5:11frontier intelligence. Gemini 4 hydrogen
  98. 5:14which is fast for everyday uh execution
  99. 5:17and we'll have more elements you know of
  100. 5:19course there are many elements left in
  101. 5:20the periodic table and we have a lot of
  102. 5:22work to do to exhaust them uh but we're
  103. 5:25glad uh to put it out there for you all
  104. 5:27to test it and for you all to play with
  105. 5:29it. Um,
  106. 5:32now I think enough marketing with the
  107. 5:34models. I want to I was told this is a
  108. 5:36deeply technical audience. So I want to
  109. 5:38talk a little bit about why post-
  110. 5:40training these frontier models is hard
  111. 5:43and what goes into that. Let me start by
  112. 5:45saying you know AI writes about 75% of
  113. 5:49Google's new code today. And this is up
  114. 5:51from less than 25% a few years ago. And
  115. 5:56there are 100,000 people using
  116. 5:59anti-gravity writing code inside Google
  117. 6:03every single day.
  118. 6:06Now the complexity of the tasks that
  119. 6:08people do have also increased
  120. 6:10exponentially in the last couple of
  121. 6:11years. We started with you know models
  122. 6:15just completing chats which took seconds
  123. 6:17to writing small functions within the
  124. 6:20minutes. You can submit a PR and resolve
  125. 6:22an issue. Takes about an hour, let's
  126. 6:24say. But now people are doing days of
  127. 6:28real engineering tasks with the model.
  128. 6:30People have swarms of agents solving
  129. 6:32real open problems with the model. We
  130. 6:34have engineers across Google using
  131. 6:37anti-gravity and our Gemini 4 models to
  132. 6:39solve tasks over multiple days.
  133. 6:43Now, how do we go about post- training
  134. 6:45Gemini for real engineering tasks? We
  135. 6:48have to start with hard tasks. So we
  136. 6:50need really need to source hard problems
  137. 6:52for the model. Grading of these models
  138. 6:56is actually a very hard problem. So you
  139. 6:58want to grade like an experienced code
  140. 7:00reviewer would grade. You want to grade
  141. 7:02like an experienced engineer would grade
  142. 7:04a solution. Large scale RL is how we
  143. 7:07train the model. And we want to dog food
  144. 7:09our models with our 100,000 engineer
  145. 7:11community and repeat the loop. So any
  146. 7:14failures in the dog food become tasks
  147. 7:16for the next iteration of the loop.
  148. 7:20So I want to start by going into a
  149. 7:21little bit more detail of each of these
  150. 7:23sections. So let's start with the hard
  151. 7:24tasks, right? Um so you might have heard
  152. 7:28you know as models are getting better
  153. 7:29lots of benchmarks are getting saturated
  154. 7:32and it's not just the benchmarks that
  155. 7:33get saturated our training data gets
  156. 7:36saturated too. So here is a graph that
  157. 7:38shows for three different iterations of
  158. 7:40the Gemini 4 models how much uh the
  159. 7:44share of learnable prompts has gone
  160. 7:46down. So let's say we had x% of
  161. 7:47learnable prompts for the first
  162. 7:49iteration. By the time we finish and
  163. 7:51train the next one, the prompts are too
  164. 7:53easy for the second one. We finish that
  165. 7:54and train the third one, the prompts are
  166. 7:56too easy for the third one. So what's
  167. 7:58the problem with uh training on easy
  168. 8:00task? Easy task give you zero learning
  169. 8:02signal, right? So you have a prompt when
  170. 8:04all of your rollouts are going to be
  171. 8:06correct. There's no variance. There's no
  172. 8:08RL signal. So you really need to focus
  173. 8:11on getting hard tasks and you really
  174. 8:13need to be faster at getting hard task.
  175. 8:15Then the models are getting better,
  176. 8:17right? Um so how do we go about it? We
  177. 8:20first have to you know identify these
  178. 8:23easy tasks and prune them. So we cut
  179. 8:25these simple sweep style tasks from the
  180. 8:28mixture that are too easy for the
  181. 8:30models. we expand into much harder tasks
  182. 8:34for example that are multifile repos. We
  183. 8:36go from unit test or binary correctness
  184. 8:40um to rubric based grading and rubrics
  185. 8:43give you dense signal. They're not just
  186. 8:4501. There can be um more gradients
  187. 8:49within the um within the criteria.
  188. 8:54Here are some examples of hot tasks that
  189. 8:56we train on. So uh we look at multifile
  190. 8:59repos and we extract commits from them.
  191. 9:02From those commits we have models
  192. 9:04automatically extract rubrics. Um we
  193. 9:07don't actually ne necessitate unit tests
  194. 9:09being involved. We can actually uh
  195. 9:12generate rubrics on the fly based on
  196. 9:14these repos. Another example a great
  197. 9:16example of tasks are you try to teach
  198. 9:19the model to build uh CLI tools from
  199. 9:23scratch. So you start with an empty
  200. 9:24folder and a specification. The
  201. 9:26specification says, hey, you have to
  202. 9:29build this tool. It's called GP. These
  203. 9:31are the parameters of GP. These are the
  204. 9:33flags. This is the functionality. And
  205. 9:35you let the model build the entire tool
  206. 9:37from scratch. And you know what the
  207. 9:40functionality for GP is going to be. So
  208. 9:41you have a hidden test suite that you
  209. 9:43can evaluate the binary on. The last uh
  210. 9:48example is production codebased diffs.
  211. 9:50So you look at real usage of the model
  212. 9:53inside your production codebase and
  213. 9:56extract useful tasks out of them.
  214. 10:00Now let me talk a little bit about uh
  215. 10:02the graders.
  216. 10:05So agents you know they find any way
  217. 10:08you're you're training them to pass the
  218. 10:09test. So they will find any way to pass
  219. 10:11the test but if the test pass that
  220. 10:14doesn't necessarily mean that they've
  221. 10:15solved the problem. There are examples
  222. 10:17where the agent will bypass the checker.
  223. 10:20There are examples where the agent will
  224. 10:22game the benchmark. There are examples
  225. 10:25where the agent will fake the evidence.
  226. 10:27You know, it writes the uh fake evidence
  227. 10:30into a temp file. It'll delete any
  228. 10:31evidence of it faking it, right? Um so
  229. 10:35you have to be smarter about how you
  230. 10:36judge these models, especially as they
  231. 10:38become more capable.
  232. 10:41Now, this is what I meant by grading
  233. 10:43like a reviewer. So if you've done a
  234. 10:45code review, you have a very strict
  235. 10:47reviewer, you're going to know they're
  236. 10:49not just going to think about the
  237. 10:50correctness of the solution. They're
  238. 10:52actually going to think about the style.
  239. 10:54They're going to think about
  240. 10:55maintainability. They're going to think
  241. 10:57of many different aspects. So what we do
  242. 11:00is we move from unit test base
  243. 11:02correctness to rubric base and these are
  244. 11:05generated from real commits. The rubrics
  245. 11:08also provide dense learning signal not
  246. 11:10just 01 signal and we penalize any sort
  247. 11:14of solution that is looking for a
  248. 11:16workaround. Right? So here is an example
  249. 11:20of a rubric where the criteria is you
  250. 11:23know here are the edge cases that need
  251. 11:24to be passed. This is an example of
  252. 11:27hygiene that needs to be maintained.
  253. 11:29These are constraints around
  254. 11:30verification that need to be satisfied.
  255. 11:35Now, it's not just about the rubrics
  256. 11:38needed for grading. We also need to be
  257. 11:41careful about the graders not being
  258. 11:43gamed and you really need to harden the
  259. 11:45environment that the grader is in. So,
  260. 11:47we think about putting the graders in a
  261. 11:50isolated sandbox. So, we remove git
  262. 11:52history and git remote so the answers
  263. 11:54don't leak. The test sets are held out.
  264. 11:57So, only when the agent is done and
  265. 11:59submits the solution do we actually
  266. 12:02reveal the test. We've had instances
  267. 12:04where the agent will fake it and say,
  268. 12:05"Hey, I I'm done and the tests come into
  269. 12:07play and then it'll look at the test and
  270. 12:09resubmit the solution." Right? So, you
  271. 12:10don't want that. Um, we also want to be
  272. 12:13much more detailed with our scoring. So,
  273. 12:15we take the full diff and we look at the
  274. 12:17reference and we look at multiple
  275. 12:18aspects. It's not just the correctness.
  276. 12:19We also want to look at the code
  277. 12:21quality. We also want to look at the
  278. 12:23scope uh of the change that was done.
  279. 12:28And then finally we want to penalize any
  280. 12:31sort of cheating attempt by the agent
  281. 12:34while being careful not to police the
  282. 12:36thoughts. Um so any you know cheating or
  283. 12:39hampering of the environment gets a very
  284. 12:41negative RL reward. Uh and all of this
  285. 12:45is detected just based on the actions
  286. 12:46that the model is taking not the
  287. 12:48thoughts. Um so a good example of how
  288. 12:52we've handled cheating is you know
  289. 12:53carbon for instance has 0% cheating rate
  290. 12:56on terminal bench 4. Our previous models
  291. 12:58had a much higher cheating rate and it's
  292. 13:00about 5x less adversarial than argon. So
  293. 13:03every generation of the model we're
  294. 13:05trying to make it much more aligned so
  295. 13:06it's not trying to cheat.
  296. 13:10Now finally I think uh I want to talk
  297. 13:12about model behavior a little bit and
  298. 13:15modeling honesty specifically.
  299. 13:18Now agents they always want to be
  300. 13:21helpful and sometimes they fake
  301. 13:23helpfulness. You might have seen this in
  302. 13:25your usage of the agents. So sometimes
  303. 13:27they'll claim that tests have passed
  304. 13:29when they've obviously not passed.
  305. 13:32They'll declare done but obviously
  306. 13:34they're not done. They'll make edits
  307. 13:36that they'll claim to have made edits
  308. 13:38but they've not made those edits. Or
  309. 13:40they'll invent facts and APIs and
  310. 13:43whatnot. or they will give you a wall of
  311. 13:45text that has so many numbers that it's
  312. 13:47going to be impossible for you to
  313. 13:48verify. Right? We've gone through all of
  314. 13:51these examples.
  315. 13:53Now, here is an example of fabrication
  316. 13:56rates in Gemini. Right? So, we on an
  317. 14:00internal eval started with 8%
  318. 14:02fabrication in that hard eval. We cut
  319. 14:05that down dramatically to about 3% for
  320. 14:08hydrogen and carbon is half of that is
  321. 14:12about 1.5%. But we're striving to get
  322. 14:14this down to zero. Like you don't want
  323. 14:15your agents fabricating anything, right?
  324. 14:18But they try to fabricate because they
  325. 14:21want to be helpful. And you really have
  326. 14:23to try hard to align them to be helpful
  327. 14:26but not by faking it.
  328. 14:29How do we reduce the fabrication?
  329. 14:31Believe it or not, you have to clean
  330. 14:33your data. Uh right. So we filter out
  331. 14:36lots of training trajectories where
  332. 14:38there's you know not just uh intended
  333. 14:42fabrication but unintended fabrication.
  334. 14:44There's lots of claims that are not
  335. 14:46fully supported. They're not verified.
  336. 14:49We ground in the full trajectory. So if
  337. 14:51the agent says I've completed the task
  338. 14:53and it has a final response, we actually
  339. 14:56look at that response and see all of the
  340. 14:58actions that the agent had taken in the
  341. 15:00trajectory to get to that final solution
  342. 15:02and we look at the steps that were
  343. 15:04taken. So you really need to when the
  344. 15:06agent says, "I'm done." and gives you a
  345. 15:08summary, that needs to be grounded in
  346. 15:09the actions that were taken. And we want
  347. 15:11to reward honesty. So if there's a hard
  348. 15:15problem, it's better for the agent to
  349. 15:17say, "Hey, this is my partial progress.
  350. 15:19It's a hard problem. I'm done here. This
  351. 15:22is my partial solution." Rather than
  352. 15:24faking the last step and saying, "Hey,
  353. 15:26I've solved the problem. You know, this
  354. 15:27is done."
  355. 15:30Finally,
  356. 15:32shipping in weeks. So how are we able to
  357. 15:34ship so rapidly?
  358. 15:36And the biggest reason for that is our
  359. 15:39internal uh team of 100,000 plus
  360. 15:42engineers at Google using the model for
  361. 15:44realistic tasks. You know many of you
  362. 15:46might know Google is a monor repo and
  363. 15:49what is a monor repo? The entire company
  364. 15:52is one codebase right and I don't know
  365. 15:55how to measure this but I think it might
  366. 15:57be one of the largest monor repos in the
  367. 15:59world. Now that's a really complex uh
  368. 16:02codebase and we have the best people in
  369. 16:06the world using a really complex code
  370. 16:09base to do really complex tasks. So
  371. 16:11every time we have a new candidate from
  372. 16:13post training, we actually serve that
  373. 16:16model uh for our users. We run blind
  374. 16:20battles with them through multiple
  375. 16:21different candidates. We let users you
  376. 16:24know uh pick the best model that they
  377. 16:26want. We also run less where under the
  378. 16:29hood we're routing the routing people to
  379. 16:32experimental models and get signal and
  380. 16:35feedback from that and we take all of
  381. 16:37these signals and friction back into
  382. 16:40training the next candidate.
  383. 16:42So this is a cycle that helps us you
  384. 16:45know rapidly iterate really train on
  385. 16:47hard problems and as the models get
  386. 16:49better the engineers are going to push
  387. 16:51the model in interesting ways.
  388. 16:55This is also why, you know, we've gone
  389. 16:57from releasing in weeks um to releasing
  390. 17:01to we releasing in quarters to releasing
  391. 17:02in weeks. So, in the three in the Gemini
  392. 17:053 era, we've had launches every 2 to 3
  393. 17:07months, but now we've had six releases
  394. 17:10in the last 16 weeks, right? Uh and
  395. 17:14we're very excited to have you all try
  396. 17:17Carbon and Hydrogen uh and Gemini 4. Um,
  397. 17:20and we're excited for for for us to see
  398. 17:24what you all do with it. Uh, and we're
  399. 17:26not done. So, we have a lot more
  400. 17:27elements uh to go through. Uh, with
  401. 17:30that, you know, thank you.

About this transcript

This page contains the full transcript of Post-Training Gemini 4 for Real Engineering Work — Melvin Johnson, Google DeepMind by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 2,802 words across 401 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.