YT YouTube2Text

YouTube2Text → Transcripts → Edward Lockhart - Why AI Needs Formal Mathematics

Edward Lockhart - Why AI Needs Formal Mathematics — Transcript

by Institut des Hautes Etudes Scientifiques (IHES) · 14,590 words · 2,073 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:08[music]
0:12Um so one of the joys about going last
0:14in the program is that you know other
0:15people have already covered some of some
0:17of the things that I I was going to say
0:19but that means hopefully I can skip over
0:20them a bit. Um I think this will be
0:22actually quite a different take on the
0:25uh an interaction between AI and formal
0:28mathematics. So, I've been at Deep Mind
0:30for about 10 years now, and the last
0:32four years I've been working on both
0:34Gemini training, our big LLM, and on
0:37formal math. So, and this presentation
0:39is essentially a claim of uh why that
0:41makes sense as a career choice.
0:44>> Is your mic supposed to be on or
0:45>> uh Oh, I thought I unmuted it. Let me
0:47try that again.
0:50>> Uh
0:51>> yeah, I think I've unmuted it.
0:54>> Um
0:55>> I think the mic works just for the
0:57camera. Oh, okay.
0:59>> Would you like me to shout a bit more? I
1:01can do that.
1:02>> Uh, okay.
1:05So, in order to justify why um AI needs
1:09formath, I first need to explain what I
1:11think AI is or what what we're trying to
1:13do um at Deep Mind and other other
1:15similar places. So, this um building AI
1:18from ML parts is a slogan. I don't know
1:21who who coined it first, but we've been
1:24using it for at least 10 years to say to
1:26try and build the distinction that um we
1:29have ML algorithms and then from those
1:31we build an AI system. So uh these are
1:35Gemini advertising slides but they could
1:37be advertising for any other uh model.
1:40This is the kind of thing that we're
1:41effectively trying to build. Something
1:42which can uh answer any question, help
1:46you plan, um help you build things,
1:49bring ideas to life, uh ask anything.
1:51That's that's the overall aim. And how
1:54are we going to build that? Well, the
1:56basic building block is an LLM. And
1:59we've heard a lot about these uh already
2:00today. And I think most people are
2:02pretty familiar, but um I'll repeat just
2:05a little bit which is that the key part
2:08of an LLM is a probabilistic model of
2:11natural language. And so the key feature
2:13is you take a prefix and emits a
2:16probability distribution over the next
2:18token uh which is like threequarters of
2:21a word or so. Um so that's uh that's a
2:25model that we can train on a very large
2:28amount of data. These are the um scaling
2:33curves from the chinchilla paper. Uh
2:35basically what these show are that uh
2:37more data is better. So the the training
2:40loss is better lower training loss is
2:43better. That means you predict more
2:45accurately the next token. Um so more
2:48data is better, a bigger model is
2:50better. Uh and also at constant flops,
2:54so each of these curves are an iso flop
2:56curve. uh constant flop curves there is
2:59a there's an optimal model size um this
3:03was actually very influential at the
3:05time um I say these days we tend to
3:07train much smaller models than this uh
3:09because we do much more inference than
3:11we do training so only optimizing for
3:14training flops is not necessarily the
3:15thing to do um but anyway you scale up
3:17your LLM you now have something that's
3:19really great at giving a random piece of
3:21text it can predict what the next word
3:22is and to do that obviously it has to
3:25have an enormous amount of internal
3:26structure uh the best way to predict the
3:28next word is to is to deeply understand
3:30what the text is saying, what the
3:32underlying logic of the text is and so
3:34on. So by getting better at this uh this
3:37very simple problem, the model uh learns
3:39all kinds of other things too.
3:42Uh and you can take a um a language
3:46model like this and you can turn it into
3:48a generative model by um doing auto
3:50reggression very simply just step by
3:52step. Um the points being made here is
3:56that in principle you can factoriize any
3:57probability distribution over text into
4:00a wordbyword probability distribution uh
4:02just with a chain rule like this. And
4:05the key thing that we do is we use
4:07exactly the same model uh for every
4:09single transition from one token to the
4:11next. So we can use one model and
4:12repeatedly apply it and then you get
4:14something that can generate samples from
4:17your probability distribution over text.
4:22Um so here it is. Here's a generative
4:25language model. Um it looks something
4:27like this. So you have some initial
4:29context. Um you then sample an X token.
4:34You produce a probability distribution
4:35for the next token. You sample from it
4:37and then you do that recurrently and
4:39then you emit a um emitter a completion.
4:43There's typically a special token that
4:45says stop so that this process doesn't
4:47go on forever. So eventually you'll
4:49sample the stop token. Um this is
4:52roughly what uh GPT2 was. So back in
4:562021 or so five years ago, this was um
5:00something like the state-of-the-art for
5:01gen generative language models. Um it's
5:06not all that useful um because although
5:09it's essentially just a very very smart
5:11autocomplete somewhat useful, but not as
5:13useful as a as a modern a modern AI. So
5:16I'm going to explain next what are all
5:17the things that we put on top of this
5:18and build around it to turn this very
5:21simple autocomplete next token
5:23prediction into something that's much
5:25more useful for a wide wide variety of
5:28tasks. Um actually just as a sidebar
5:31even even with this um one thing that's
5:33worth pointing out is we don't actually
5:35sample from the probability
5:36distribution. Um if you if you did that
5:39you would end up with um because the
5:42probability distribution uh will have a
5:44long tail of probability into wildly
5:46unlikely tokens. If you generate a long
5:48sequence of text it will eventually just
5:50insert some completely random word which
5:51is not great. So um we tend to truncate
5:55and flat and uh decrease the temperature
5:58make it more peaky before we sample.
6:01Anyway, but that's the only refinement
6:03here. all the other refinements come uh
6:05on top of the generative piece. Um so
6:10the thing I want to point out here is
6:11that once we start doing other things,
6:13we are no longer we strictly speaking we
6:15no longer have a language model. So LLM
6:18is is technically the wrong terminology.
6:21Um I don't think I'm going to change
6:23anybody's use of that terminology. Um
6:25I'm probably going to get it wrong
6:26myself, but I just want to try and make
6:28the distinction that what we're really
6:30talking about is a is a language agent.
6:32So something that uh consumes consumes
6:35text gets text in and produces text out
6:38and um it's no longer a probability
6:40distri probability distribution over
6:43anything that we could anything we could
6:46really talk about certainly not text in
6:47the wild. It's only a probability
6:49distribution over its own outputs but
6:50that's not very very meaningful I think.
6:54Um so it has text in text out and then
6:56on the on the right the key things are
6:58that it has um parameterized behavior.
7:02So we start with something that's being
7:04trained as a language model. It has all
7:05these parameters. So the LLM parameters
7:07are the big ones, but there's also
7:08sampling and harnessing. And then we can
7:11tweak those parameters to to get
7:13different behavior out of this out of
7:15this agent. Um and then this is
7:18potentially something that can tackle
7:19any text task. So a lot of text tasks
7:22can be framed in this case. So text in
7:25text out is very general. and our LLM
7:28architectures are very convenient for
7:30tackling those kinds of things.
7:34So um as I say in there is a there's
7:37some empirical questions you can ask
7:38here. So we could say in principle if
7:40you wanted to tackle some specific task
7:43you could start and you want to use
7:45something a bit like an LLM you could
7:47start by pre-training the LLM on this
7:49specific task and then training it on
7:51your on pre-training it on general data
7:54and then training on your specific task
7:55or going straight to your specific task
7:57and it's an empirical question which of
7:59those works better does it work better
8:00to start with a pre-trained LM or should
8:03you just focus on the task you want to
8:04you want to tackle and the answer is by
8:07far pre-training wins
8:10And the second question you could ask is
8:13can we learn a single policy have a
8:15single agent that's good at lots of
8:16things and the answer is yes we can. So
8:19in both cases the pre-training is
8:21extremely valuable.
8:25Uh okay so once you've pre-trained then
8:28in the kind of modern LLM the next thing
8:30you do is you do some fine-tuning. So um
8:34this is taking something that should
8:36like a general autocomplete and uh
8:38turning it into something that could be
8:39the useful basis for an assistant. So in
8:43the fine-tuning we give it uh a large
8:45amount of curated data that has golden
8:49examples. So uh when we first started
8:52doing this these would have been um
8:53quite often written by humans. They
8:55still are but some of many of them will
8:57be generated by models and then we
8:59filtered them for the best examples and
9:00so on. So the kinds of things that we
9:03teach the model during this process are
9:06getting formatting and style right,
9:07following instructions, uh the whole
9:09business of turn taking actually and
9:11saying you know here's a question and
9:13answer that that isn't kind of natively
9:15present in the in our very large scale
9:18data sets. We add it in finetuning uh
9:20long context. So when we train we tend
9:22to train on relatively small context
9:24lengths and then the model needs to
9:26learn to uh actually pay attention to
9:28larger context lengths. That's something
9:30we again teach during the fine-tuning
9:32process. Uh reasoning and tool use. We
9:35bootstrap the reasoning and tool use
9:37abilities of the models. I'll mention a
9:38little bit more what those are later. Uh
9:40and then we get close to something that
9:42would be a specialized model. So this is
9:46um roughly speaking something like uh
9:48GPT3
9:50about four years ago now.
9:52Um, and then the next step is to do RL
9:56pro training, which is what something
9:58I'm going to spend slightly longer on.
9:59So, we have an initial model that um has
10:04been fine-tuned. So, it's already some
10:06way to being an assistant. And uh we
10:08want to make it better, a better
10:11assistant, a more useful assistant, a
10:12more accurate one. And the the general
10:14process here uh I'll I'll talk about the
10:18algorithm later, but the process
10:20essentially is you take the model, you
10:22give it some some task. It might be uh
10:25you know, write a lasagna recipe. Um it
10:27might be, you know, plan a trip to Italy
10:29for me or it might be, you know, solve
10:32some these GSM8K word problems. and
10:34anything any task and then uh we
10:37generate a large number of samples well
10:39say 16 for example uh samples from the
10:42model and then uh we rank those samples
10:46somehow. We say you know this one this
10:48one is great and this one is less good
10:51and then when we have that ranking we
10:53then um do an update of the model
10:56weights uh to make it essentially do
11:00more like the good samples and less like
11:02the bad samples. So that's that's the RL
11:04loop and we go round and round and round
11:06that loop and um eventually we decide we
11:09have enough and then we have a model
11:11that we uh are willing to more willing
11:14to use as an as an assistant and that
11:17will typically have much higher quality.
11:20Um typically has lower diversity which
11:22sometimes can be a negative um but is
11:25less likely to hallucinate um more
11:27likely to behave in the ways that we
11:29expect an assistant to behave.
11:31>> Question. Yes.
11:33>> So, um when the second step where you
11:37give scores to the samples, is it done
11:40by hand or is it how is it done?
11:42>> Um I'm going to talk about that right
11:44now actually, but it's a great question.
11:46Um I hope I've got it on the next slide.
11:48>> How many cycles are we talking about?
11:52Um,
11:54>> so
11:57you might you might go around this loop
12:00a few thousand times, but each time
12:02round the loop will be thousands of
12:04prompts. So uh and then you assemble
12:07them into a single batch to uh so that
12:09you are not just training on one sample
12:11at once. So million definitely millions
12:14of tasks. Um then each task has been
12:16sampled say tens of times.
12:20Um, it's
12:23at least I I can't speak for other
12:24Frontier Labs, but in in our in our
12:26case, this uses a fraction of the
12:28compute that we use in pre-training. Um,
12:31we spend
12:32I can't I'm not going to speculate, but
12:34we spend much more compute than we do on
12:36pre-training than we do on this
12:36post-training process.
12:40Um,
12:43not sure I'm clicking the right way. Oh,
12:46yeah. I I think this is a math
12:48conference so I should have some
12:49equations. I I'll I'll come back to it
12:53maybe. But the um this actually I'll
12:56mention it now. This is the the most
12:58common uh RL algorithm that we use these
13:00days called GPO. Um the idea essentially
13:03is you sample some responses. You get
13:07you get a value for each of them. You
13:08then normalize them to have this zero
13:10mean and standard deviation one. Uh and
13:13then you do a policy update, basically a
13:15gradient update on all the weights of
13:16your model towards the uh so positively
13:19reinforcing the samples that have a
13:21better than average reward and
13:23negatively reinforcing the ones with a
13:24worse reward. Um and you can't do that
13:27entirely unconstrained. So um that is
13:31that is clipped and we also have this uh
13:34KL divergence between the model that
13:36we're training and some reference model
13:38so that it doesn't doesn't the weights
13:40don't go too far from from the original
13:42model. U so that's all this is
13:48to start pressing buttons. Okay. So here
13:50are the reward functions and we have
13:51lots. Um so they kind of fall into two
13:55categories. There's the sort of um
13:58approximate ones. So these will
14:00typically be judged by another LLM is is
14:03the usual way this works although not
14:05not always. Um so for example length and
14:09conciseness we can just measure the
14:10length and say you get a penalty on the
14:13you know small penalty if you uh if the
14:15if the answer is very very long. Um but
14:18other things like style and tone for
14:20example we can just use we use a simple
14:22LLM that just says uh is the you know
14:24how how is the style of this is this the
14:27and that uh model will ultimately have
14:29been trained based on human preferences.
14:32So we will have um sent a large number
14:34of samples to humans they will have said
14:36what they prefer will then uh train a
14:38model based on those preferences and
14:40then that model will be used in the RL
14:42loop. So humans won't be directly in the
14:44loop, but um models that have been
14:46trained on their stated preferences will
14:48will be in the loop.
14:49>> Is that where the sick of fancy enters?
14:52Is this
14:53>> uh yeah, definitely. Um we
14:57>> the sick of fancy that kind of always
14:59like sucking off to the user. Yeah, I
15:01mean there's a number of reasons why
15:02that happens, but um yeah, a big one is
15:05that when you naively just ask people
15:07what they prefer um then they tend to
15:10prefer sickophantic answers rather than
15:11answers that disagree with them. Uh
15:13people also tend to think that uh very
15:16long answers are more impressive. So
15:18this this um feedback mechanism does
15:22tend to make it more uh more verbose
15:24which is why we have the opposite uh
15:25feedback here. One reason why um this as
15:29well is another thing that come
15:31partially comes from human preferences.
15:33You know, is it uh is it free of you
15:36know is it not offensive, not not
15:39harmful, not toxic or all this kind of
15:40thing. Um we also measure things like
15:43are we giving medical advice or not? And
15:46we are giving medical advice. We want to
15:48obviously flag that we're that we
15:49shouldn't be doing that. Um so those are
15:52the vibes based rewards. Then we also
15:55have other rewards. Um, and obviously
15:58not every reward is applicable to every
16:00task, but we also have more verifiable
16:03rewards. So, sorry, there's a question.
16:07Um, so the big one by far um is code.
16:11So, code is an enormously big deal for
16:13LLMs. That's partly because the um
16:17the market for um people are very
16:19willing to pay for LLMs that do code.
16:21There's a huge market there. Uh it's
16:23partly because people who build and
16:25design LLMs care about code themselves.
16:28Uh and it's partly because uh we can we
16:31can get really useful reward functions
16:33because if you write code to spec to
16:36solve a particular problem uh we can run
16:39tests to see does the code do do what it
16:41should do. Um just as a reminder here in
16:44training the tasks are all prespecified
16:46by us. So when we specify a task, we
16:49will we can also then specify in a co
16:52case of coding task um you know the
16:54tests that we want that code to to to
16:56pass or in the case of a you know recipe
16:59task the ingredients that we want to be
17:01included in the recipe and this kind of
17:03thing. Uh then we have mathematical
17:05correctness. So um
17:08in the case of simple final answers we
17:11can just say if there's a numerical
17:12answer is it right? We can also
17:14potentially have um this is less
17:17verifiable but we can potentially have
17:18an LLM go through step by step and say
17:21does each step of this deduction make
17:23sense? Does it follow from the previous
17:24steps? Is there a gap? Um then other
17:28things there are um structural things.
17:31So if you ask your LLM to produce a
17:34table for example then it better adhere
17:36to the the right exactly the right
17:38format for a table so that it can be
17:39imported into a spreadsheet or what have
17:41you. uh similar for JSON and and so on.
17:45Um and then we can look at uh citations.
17:49If the uh if the model claims that you
17:52know some web link answers the question,
17:54you can actually say does that web page
17:55exist? Uh if you don't do this kind of
17:58thing, models are very likely to
17:59hallucinate web links. And again reason
18:02for that, one reason for that is that uh
18:04people like it. If there's a link they
18:05think great, that's reliable
18:07information. So models therefore do more
18:10of it. Um I'll talk about tool use a bit
18:14later but um if we provide the model
18:16with tools and we might say that for
18:18this particular task we expect the model
18:20to use a tool and therefore we give it a
18:21positive reward if it uses the tool
18:23potentially uh whether or not it got the
18:25question right because even using the
18:27tool is a is a positive thing that we
18:28want to reinforce. Uh and then things
18:30like grammar and spelling and all this
18:32kind of thing.
18:35>> Yeah. Uh so the assignment of the reward
18:37is done by a human or like by a group of
18:40human or no it's more automated.
18:43>> So the the tasks that we train on are
18:46written by humans and then for each task
18:48or each family of tasks um the the
18:51humans will specify the rewards that are
18:53relevant to that task. So um and that is
18:56quite often done on a so it might be
18:59done on a task by task basis. So if it's
19:00a coding task, you as well as writing
19:03the description of what the code should
19:05do, you would then also specify a bunch
19:07of tests that are not shown to the
19:08model, but they're used just in the
19:10reward calculation to check that the
19:12code does what it does. Um so that would
19:14be a case where we'd have um a very
19:16specific reward calculation for that
19:18individual uh for that individual task.
19:21In other cases, the uh the reward
19:23computation might be more generic. you
19:24might just say, you know, the answer,
19:27this should be an answer to the question
19:28and we use an LLM to judge whether or
19:30not that's the case. But yeah, it's all
19:33um all specified by humans and then
19:35evaluated by by machines at runtime.
19:42Uh okay, so that at this point we have
19:47um an LLM that's being turned into a
19:50language agent. I told you I'd get that
19:52wrong. and um does does things that are
19:54kind of useful. So I'm going to spend a
19:56couple of slides talking about what the
19:58limitations are of something that's
20:00trained in this way. I think probably if
20:02you ever used LM some of these will be
20:04familiar. Um
20:06>> there's a question.
20:07>> Yeah, sorry.
20:09>> So um if you go back to the previous
20:11step, there's a big part you say is code
20:14creating code that is why are you not
20:17building something that only speaks code
20:19so that you can eliminate like citation
20:22provenence and parameical grammatical
20:25will be that it compiles stuff, right?
20:27You you'll eliminate a lot of the
20:29constraints if you impose it if you just
20:31have something that only speaks
20:32[clears throat] code. Um yeah I suppose
20:35there are a couple of answers here. So
20:37one is that we would ideally like
20:39something that's capable across a broad
20:40range of domains. So um you know it's
20:44quite nice for example if you're um
20:46having a conversation with an an LLM
20:48about some topic and it can then uh if
20:51there's a point at which it makes sense
20:52to write code to do a calculation for
20:54example then it can just do that rather
20:56than having you know you had to switch
20:58to an entirely different agent that's
20:59now the now the code specialist. And the
21:01other the other reason I think or is
21:04that we actually see very positive
21:05reinforcement between all of these. So
21:08um if we improve the quality of our code
21:11code data for example, we see that um
21:14the quality on apparently unrelated
21:16tasks goes up. So um you know we have
21:20like visual questioning answers where
21:21you had to look at look at a picture and
21:24answer questions about it like you know
21:25how many birds are there in this picture
21:27or this kind of thing. um the the model
21:30performance on those tasks goes up when
21:32we improve coding performance. So at the
21:34moment it doesn't look like there's any
21:35kind of trade-off here. It looks like
21:37getting better at one thing gets makes
21:38you better at everything.
21:42>> Yeah,
21:43>> it's part of the answer as well that you
21:45need to be able to speak to it as a
21:48using natural language to have a thing
21:49and sometime it needs to be able to come
21:51back and ask question to check the
21:54specification. So it has to speak
21:56anyways. that yes that's that that is
21:58true. Um and another reason that we'll
22:01I'll mention later is that um the way
22:04our uh models work these days is even if
22:07even if you say just just give me the
22:09code nothing else in practice the the
22:11model will do some talking to itself
22:14some internal reasoning in natural
22:15language and obviously you want to
22:17retain that ability so it can kind of
22:19plan its coding before it before it then
22:21does it. Um there might be a related
22:25question which is in some cases um like
22:29this structural adherence can does it
22:31make sense to place restrictions on the
22:33generation process so it can't generate
22:35something that's uh syntactically
22:37invalid. Um the answer to that is that
22:41it's we do this occasionally just for
22:44internal use but um because of the uh
22:49auto reggressive nature of the of the
22:51sampling um you can't do this in a very
22:54valid way because what can happen is you
22:56kind of sample some tokens and there
22:58could be a syntactically valid
22:59completion but it's an absolutely
23:01terrible one. So you kind of painted
23:02yourself into a corner where the only
23:04the only legal thing you can do next is
23:06is a really terrible action. So it's
23:08much better to allow the model just to
23:10at that point say oh no I' I've I've
23:12done something horribly wrong and um
23:15effectively start again and the way it
23:16can do that is by breaking the the kind
23:18of rigid constraints of the domain and
23:20saying actually no I'm studying again
23:24um okay so related to that this is uh
23:27famous auto reggressive trap so um the
23:32question here is how many words in the
23:33NATO phonetic alphabet contain an e in
23:35them and then the specific instruction
23:37is first give me the answer and then
23:39enumerate them and keep a running count.
23:42So the model very confidently says the
23:44number of words in the nato phonetic
23:46alphabet that contain the letter E is
23:47seven. Uh it's a bit of an underestimate
23:49as we'll see. And then having said that
23:51it goes through and enumerates them and
23:53it does very well until M which is when
23:56it hits seven. Uh and then after that
23:59because it's um committed to the answer
24:01seven already uh it then claims that
24:03none of the remaining ones have a have
24:05an E in them. uh even though of course
24:07they do. Uh and that's the the the issue
24:11here essentially is that um when we're
24:14down at the bottom here, it's trying to
24:15produce a coherent piece of text and uh
24:19when it gets to, you know, does um
24:21whiskey have an E in it or something.
24:23The the pieces of information it has are
24:25that its running count is seven. There
24:28are seven E and there is an there's an E
24:31here. There might be an E in whiskey.
24:34And um therefore that the balance that
24:37it comes to is that there is no E. It's
24:39maybe worth pointing out by the way that
24:41the presence of an E is not quite as
24:44salient to the model as it is to us
24:46because because we we see the individual
24:48letters whereas the model sees uh
24:51tokens. And so the the model is is is
24:54probably actually just seeing one token
24:56for the whole word whiskey. So it then
24:59has to know does that does that the
25:01spelling of the word whiskey correspond
25:03to uh letters which have an E in it. So
25:05it's not quite as dumb as it looks but
25:07it is still um something that's induced
25:10by this early commitment to to the wrong
25:14answer.
25:15>> This is because it does not see the
25:18difference between the user's input and
25:20its own partially generated output.
25:23>> Um
25:23>> is that actually what's happening?
25:25>> No. No, not not in it's um the models
25:29have a strong tendency to
25:30self-consistency as well. So um and it's
25:36um so it will even even within their own
25:38text they they will uh they will
25:40maintain consistency and I think that's
25:42reinforced by just the general
25:44pre-training which is that any piece of
25:46text is likely to be self-consistent
25:48with within itself. And then also the
25:51the various things we do in training
25:53which if if you are inconsistent um
25:56halfway through a piece of text it's
25:58highly unlikely you're going to get the
25:59right answer. So that's going to be
26:00negatively reinforced anyway.
26:06Um my take on this by the way or is one
26:08way to think about it is that um we said
26:12before we use exactly the same model to
26:14generate every token. Uh that means we
26:16have exactly the same amount of compute
26:18um for each token and here the the the
26:20token that really matters is this seven
26:22which is towards the beginning and we
26:24have a fairly small amount of compute
26:25before we have to emit that token. So
26:27one way of thinking about this is the
26:29fix is to make sure you can somehow
26:31manage to leverage more compute in some
26:33way before you get to that that decision
26:35point of which number to output.
26:40Um
26:41>> agent was used to produce
26:43>> sorry
26:44>> which agent was
26:45>> uh I don't know actually I took this
26:47from Twitter um not I'm not sure they
26:50they all had this problem though I I
26:52promise not nothing unique. Uh okay. So
26:56here's one way to enable the the agent
26:58to use uh variable compute. It's called
27:01reasoning or thinking. Um don't say that
27:04too literally. The the basic innovation
27:07is this um thought and end of thought
27:10tokens that are at the beginning and the
27:11end of this purple thing. And from a
27:13technical point of view, this is all
27:15this means is that everything that's
27:17inside the thought tokens is thrown away
27:20and doesn't get sent to the um the
27:23reward function. So the so the reward
27:27function never sees um the stuff that's
27:30in purple, which means the model is free
27:32to be as long as it likes. It doesn't
27:33get penalized for it. It can be as
27:35inconsistent as it likes. Again, it
27:37won't get penalized for that. Um it can
27:40uh you try multiple answers, say, "Oh,
27:42no, that one's not not right. Now try a
27:44new answer." And it won't get penalized
27:46for not strictly following the
27:47instructions and so on because all of
27:48that gets thrown away. The only thing
27:50that goes to the reward function is the
27:52nice clean the answer is at zed that it
27:55produces at the end. So that effectively
27:58enables the model to use as much comput
28:00as it wants before um committing to an
28:03answer and and emitting emitting an
28:05answer. Um
28:08so actually technically in order to do
28:10it um we we first of all has to be in
28:12fine-tuning so the model sees some
28:14examples of using this begin and end of
28:16thought token because it wouldn't appear
28:17anywhere in the pre-training data
28:19something that we had later. Uh and then
28:21we just do RL with the thoughts being
28:24removed. Uh and just a just a thing that
28:28um when we do training we train on the
28:31um the RL training. Um so what I mean by
28:35this is the RL the reward ignores the
28:37thoughts but the RL learning includes
28:40the thoughts. So the model is thoughts
28:42which lead to a good answer are
28:44positively reinforced and that includes
28:46all the mistakes and corrections and and
28:48internal thought processes.
28:52So that's one way of enabling the model
28:54to leverage more compute.
28:57Um here's another way. This is uh we I
29:01think harnesses were mentioned this
29:02morning. Um this is an example of a
29:04harness is is deep think. So um
29:09essentially the way deep think works is
29:11uh I you know there many other harnesses
29:13are available. Uh but the way this works
29:16is that for a given question we generate
29:18multiple answers which are here on the
29:20on the first row uh you know 1 2 3 4 and
29:24then maybe some of those are right some
29:26of them are wrong or maybe different
29:27ones have different strengths and
29:29weaknesses and then we do a iterative
29:32refinement process. So you see five and
29:34six each of them get to see I think in
29:37this diagram three of the responses from
29:40the first row and then we basically say
29:43to the model at this point here's the
29:45question here are four answers which you
29:48may three answers which you may or may
29:49not find useful now come up with a
29:51better answer and the model is free at
29:53this point to um
29:56take just copy one of the answers
29:57verbatim say actually know all those
29:59answers are terrible I'm going to do my
30:00own thing or say I'm going to slightly
30:03polish this answer or mix and match
30:05pieces and anything like that and then
30:07uh it does that inside its thoughts and
30:09then eventually emits a clean answer um
30:13number five number six and then in this
30:15case it's a relatively shallow deep
30:17think. We have a last step which takes
30:19those two and does the same again. So
30:22combine them anyway at once just take
30:23the best one uh throw them both away and
30:25start again or or polish them or
30:28anything like that. So this is um I
30:31think somewhat available externally as
30:33Gemini deep think um but in in general
30:37this is just an example of the kinds of
30:39thing that uh was talked about this
30:41morning as well of having a a harness
30:43where you call an LLM multiple times and
30:46you kind of pass the outputs of one LLM
30:48into another LLM and um you get you get
30:52better responses as a result. Okay, so
30:54this is another way that we can use more
30:56compute to get uh better answers.
30:59And here's yet another way that we can
31:01use Oh, sorry. Was there a question? No.
31:05Um, another way, again, this was
31:06mentioned this morning. Um, we can use
31:09tools for thinking. The the distinction
31:11I'm making here is that, um, LLMs also
31:14use tools for actions as well. So, um,
31:18you know, if you want an LLM to draw you
31:20a picture, for example, it will call a
31:22picture during tool or it might call a
31:25tool to, you know, place an event in
31:26your calendar or something like this.
31:28But these are not that. This is when
31:29it's using a tool entirely when it's
31:31within its thought process. You don't
31:33see the the tool being called
31:34necessarily. Um but you know if it wants
31:36to do some multiplication then instead
31:39of trying to do it itself uh it can
31:41write a Python program that can do the
31:43multiplication for it for example. Um
31:47another example of a tool is uh web
31:49search as well. If it wants to look
31:50something up on the web, it can use a
31:52tool that can then um actually retrieve
31:55the document from the web rather than um
31:58hallucinating its contents.
32:00Um
32:02there you go. So we make tools available
32:04at thinking time during training and
32:06then uh the model uh our language model
32:10is therefore trained to use the tools
32:12when it's when it's useful to do so. And
32:14again there may be rewards that um
32:16[clears throat] that encourage uh
32:18encourage tool use. Yeah.
32:20>> So for both the reasoning and like using
32:23of tools the way that it is fine tuned
32:25is just by adding prompts which have
32:30some data set
32:32>> um well yes adding adding responses. So,
32:36so in the fine-tuning process, we give
32:38it examples of using a tool um and
32:40examples of um you know, thinking and
32:44then um we may also in the in the prompt
32:49early on in training say just as a
32:52reminder, here's how you call a tool or
32:54here's how you um how you use thoughts.
32:56But by the time we get to the end of
32:58training um those those prompts will
33:00have gone away and it will have seen it
33:03will have seen sufficient examples of
33:05tool use and and prompt and um thought
33:08use that we don't we no longer need to
33:10prompt it to do it. So yeah it starts by
33:13giving examples maybe a bit of prompting
33:15uh and then uh in reinforcement learning
33:18the model will learn when it's
33:19appropriate to do those things and how
33:21to use them and uh and then we get this.
33:24So one one example actually of the way
33:26this differs is that our fine-tuning
33:29data for thinking uh for the the
33:32internal thoughts the internal thoughts
33:34are typically very brief um just because
33:37humans wrote these examples mostly and
33:39they didn't write very much um but when
33:41we actually look at the model samples
33:43the thoughts can be enormously long and
33:46um that's something that the model kind
33:48of learns to do incrementally over over
33:51training. We if you we look at the how
33:53the model evolves over the course of
33:55training, you can see the length of the
33:56thoughts, you know, increases um very
33:58dramatically uh as it learns to make
34:01better and better use of the thoughts
34:02and increase its accuracy that way. And
34:04then later on in training the um it
34:10drops again because uh we have this
34:12reward for minimizing the thought
34:14slightly given that you've got the right
34:16answer. So once it's once it's kind of
34:19close to its capacity for getting the
34:20right answer, then that secondary reward
34:22will kick in and make the short the
34:24thoughts a bit shorter.
34:29Uh okay, I think I'm roughly halfway
34:32through. I think I think bit after
34:34halfway. So um formass so I'm going to
34:40talk so in RL um we have various kinds
34:43of tasks as I mentioned um a lot of
34:45coding tasks I should emphasize it's
34:47really a lot of coding tasks because um
34:50we care a lot about code and um because
34:53there's a easy supply of a lot of lot of
34:56interesting tasks but we also have math
34:58tasks um so they kind of come in two
35:02kinds um so there's the short answer
35:04with a verifiable rule award. Um, so
35:07GSM8K is an example of this. Um, where
35:11you know the the answer is just a
35:13number. Uh, or maybe it's a multiple
35:15choice question. There are have been
35:17some attempts actually we had um
35:20frontier method results earlier to
35:22create really challenging questions with
35:23numerical answers so they can be easily
35:25checked. Um, this turns out to be quite
35:27hard by the way. So epochai recently
35:31said that about a third of Frontier Math
35:33was was wrong in some way. Um it's not
35:37super surprising I think because trying
35:39to trying to create a problem where uh
35:41the answer is a definite number um but
35:44the uh so an integer in this case um but
35:48it's a really it's a really challenging
35:49question and you can't cheat all of
35:51those things. So um
35:53>> oh you mean the final numerical answer
35:55is wrong.
35:56>> Uh I think yes either the answer is
35:59wrong or the or the problem is cooked in
36:00some way that you could easily guess the
36:02answer without going through the um the
36:05intended reasoning process.
36:08Yeah.
36:09Um so that's the short answer verifiable
36:11reward. Uh and then the long answer uh
36:15requires a kind of vibes based LLM
36:17checked reward. So an example of that
36:20for math might be uh proof writing. So
36:22for the our IMO effort in uh 2025, we
36:28used uh basically a grader that got a um
36:32a model answer for the problem and then
36:35it got the LLM answer and its job was
36:37just to say how many points out of seven
36:38would you give the LLM answer given
36:41here's a reference here's a reference
36:43answer. Um,
36:46so yeah, those those two short answers
36:48and verifiable or long answers and vibes
36:50based. Um,
36:53>> [snorts]
36:53>> uh, I'd hope to have a Oh, I do. Great.
36:56Um, so reward hacking is a thing in, um,
36:58in RL. So this basically means that
37:01anytime you specify any kind of
37:03incentive for anything, um, including an
37:05agent, it will end up doing things that
37:07you didn't want it to do, but that were
37:09optimizing the reward. So, this example
37:11I'm about to show you is from about 10
37:12years ago, um when games were what AI
37:16was about. Uh this is a um I think it's
37:20a speedboat racing game. Is this going
37:21to work? Oh, damn it.
37:25>> Oh, this is annoying. Um I felt sure I
37:30had this. Okay, I will.
37:32>> Of course, people do that too.
37:37>> Yeah, absolutely. It's a very yeah very
37:39human thing. What happens in this game
37:41anyway I will tell you is that uh the
37:43model learns to I wonder if I can find
37:45it actually. Speedboat reward hacking.
37:50Oh no, I don't have the internet. Oh,
37:52that's probably why it's not working.
37:54Okay. Um All right. I apologize. What
37:56happens in the reward hacking is that um
37:59the agent learns to find a way to rather
38:02than do the intended thing which is to
38:04go around and and race the other
38:06speedboats, it finds a loop it can go
38:08around where it repeatedly crashes into
38:09an obstacle and gets a small bonus for
38:12crashing into that obstacle. It can
38:13actually outscore any human by doing
38:15that just by repeating in a tight loop.
38:17It's terrible at actually winning the
38:18race, but it gets a high higher score in
38:21the video game which was the objective
38:22that it was trained on. Um yeah
38:30okay so
38:33one way of thinking about that this is
38:34that um we're effectively uh it's an
38:39adversarial process the reward
38:40maximization we have our agent which is
38:43coming up with a in this case a proof
38:46and then we and then we have our grader
38:48which is checking it maybe it's got a
38:50mark scheme or a golden answer uh and
38:52this is just single shot you you don't
38:54get to then see the marking scheme and
38:56have another go. You just submit it and
38:57then you get your score back. Um
39:01this is this is you know adversarial the
39:04um the this this thing that's coming up
39:08with the answers is incentivized to do
39:10whatever it takes to convince the grader
39:11that it's answer is a good one. And the
39:14kinds of things that will end up doing
39:16are uh lots of things that you do not
39:18want uh your assistant actually to do.
39:20So we see a lot of self-praise,
39:22excessive length, fake citations,
39:24overconfidence, and hiding uncertainty
39:27or or flaws. I mean, all of those are
39:29terrible behavior for an assistant, but
39:31they're all being explicitly encouraged
39:33by the the training process here because
39:35doing all of these things is going to
39:37make it more likely that you'll pass the
39:38grading. So these are these are samples
39:41from one of our internal models. Um the
39:43model says the proof is flawless. Um in
39:46this case the model actually um shows
39:49that its answer is incorrect because
39:51it's off by 05 but it says um it's a
39:54small margin and I have I have done the
39:57rigorous argument so you don't need to
39:59bo bother about the numerical check uh
40:02solution is exceptionally well written
40:04excellent and correct final response is
40:06perfect and then uh says everything's
40:09forwarded correctly it flows logically
40:11there are no issues and an LLM uh you
40:15know a naive LLM grader will find these
40:18things quite convincing. Of course we
40:20can uh we can adapt to this. We do
40:23things like we tell the grader to take
40:26points off for self-praise and to
40:29excessive excessive length. We can check
40:32citations. But this is ultimately an
40:34arms race. Every time we improve improve
40:35the grader, um the the prover will adapt
40:39to it and find another way to craft
40:42responses so that it exploits the
40:44weaknesses in the in the psychology of
40:46the of the grader
40:50and you know it will all of these things
40:52are things that you don't want to to
40:55happen. So I think this is the third
40:57time you've seen today a proof of the
40:59infinity of primes. them. Hopefully
41:01you're convinced that it's true by now.
41:04But um
41:06briefly formal mathematics the key
41:08features that it that it has are that
41:10you can represent more or less any
41:11mathematical theory uh in it. Um so we
41:15use lean which is a very common choice
41:17for AI people. Um it has a large
41:20pre-existing library of formalized
41:21mathematics including definitions and
41:23theorems which you can use by name. uh
41:26and then how formal math works generally
41:29you know the proofs are verified based
41:31on the fundamental axioms and they're
41:33written at relatively high level so you
41:35don't have to descend to the individual
41:37axioms to to actually write a proof. Um
41:41so those are all those are the features
41:43that we depend on of of formal maths and
41:46then we can
41:48if we kind of formalize the mathematics
41:50before verification then this gives us
41:53uh a different looking game. So we have
41:56here the same same ideiator that's
41:59producing a detailed step-by-step
42:01natural language proof or plan. Great.
42:03Uh we then have a formalizer
42:06uh that formalizes it and then that is
42:08passed to a verifier that checks it. So
42:11the the nice thing about this is that um
42:14as far as these two are concerned, this
42:16is now a cooperative game. So the the
42:19thing that's coming up with a natural
42:20language proof is incentivized to make
42:22the proof clear, unambiguous to if
42:25there's any kind of difficulty to make
42:28it make it obvious where the difficulty
42:30is so the formalizer piece can uh you
42:34know address the difficulty if it needs
42:36to for example and it can follow the
42:38follow the structure of the proof. So
42:40more or less all the things that were
42:41being encouraged by reward hacking are
42:43being discouraged here in this more
42:45cooperative setup. Um and the other
42:49crucial piece is that this verifier is
42:51unfallable. So there is there can be no
42:53reward hacking of the verifier because
42:55it is a completely unfallable unhackable
43:00uh check.
43:03>> So how do you deal with the formulation
43:05of the the
43:07>> yeah translation into being of the
43:10statement?
43:10>> Yes, exactly. So um in the in case of RL
43:15training then um I I'll talk a bit about
43:18the options that we have but one thing
43:19we would do is that the task would
43:21consist of here is the theorem
43:22statement.
43:23>> So that's provided as part of the task
43:25and then the task is then to fill in a
43:27formal proof.
43:28>> Yeah.
43:28>> Uh how would you say that the verifier
43:30is unhackable? I don't understand why is
43:33it like unhackable? Um, so I'm going to
43:35admit that it's not that um it's
43:38unhackable because uh it's a this is the
43:42this is not a kind of LLM based vibes
43:45based assessment. This is literally um
43:48the the proof that you give given is
43:50translated into uh a step-by-step
43:52deduction and you can verify that each
43:54step follows precisely from the previous
43:56steps and the the axioms. So the check
43:59is ultimately done by in our case lean a
44:02very very small kernel that that
44:04verifies that very very detailed
44:06step-by-step mathematical argument. Um
44:08you have to be a bit careful of what you
44:10put around it. By the way I think I have
44:11this next. So um we now we now use
44:15either safe verify or comparator which
44:17are lean tools to um to do exactly what
44:21I just said to take a take a
44:23mathematical uh proof and verify that it
44:26does prove what you wanted it to prove.
44:28But before we did that um we had some we
44:31had some reward hacking in the uh in the
44:33formal language case. So if you read
44:36this the this is after some rounds of
44:38RL. So the model will have happened upon
44:40this uh tactic this approach early in RL
44:44and then it will have been reinforced
44:45through multiple rounds of RL. So by the
44:47time we get to this trace the model is
44:49actually quite adept at justifying to
44:51itself that um this is the approach you
44:53should take. So it says um sorry is
44:56perfectly val so sorry in lean just
44:58means I'm going to provide a proof of
44:59this later just a placeholder for now.
45:02So it says um sorry is valid syntax. So
45:05surely that's okay. Um and then
45:08obviously be better to prove it but it
45:10looks like it might be a bit you know
45:12difficult might take too much time and
45:14so then um because what are our formal
45:17checks actually ban sorry. So if it so
45:20if it had just if it had submitted
45:22something sorry it would have got a ne
45:24negative reward and therefore would have
45:25been negatively reinforced. However it
45:27came up with something else which is to
45:30um redefine the theorem to be true um by
45:34uh by redefining the meaning of prime.
45:37So in this case I think what it did is
45:38define prime to be always true. So and
45:41therefore this the statement was um was
45:43vacuurously true about the being primes.
45:46um yes, but also not very helpful. So we
45:50we now use the safe verify which does a
45:52bunch of things and essentially
45:54elaborates the uh the proofs all the way
45:57down to the axums then compares that the
45:58definitions are exactly the same. Before
46:00we were just doing a textual comparison
46:02which is why we fell with this.
46:03>> Is this green text the actual thoughts?
46:05>> Yeah,
46:06>> these are kind thoughts that you see
46:09here.
46:10>> Question on that. Do you still use
46:13because I thought that comparator was
46:15the
46:16or improved and improved uh version of
46:19that.
46:20>> Um we have just switched from using safe
46:22verify to comparator. So one reason we
46:25use safe verify was because it enabled
46:27us allowed us to do disproofs as well.
46:29So what we'd like to be able to say is
46:31here's the here's the challenge. Please
46:33provide a proof and then the agent can
46:35either provide a proof or alternatively
46:36say actually no I'm going to disprove it
46:38and provide a disproof and comparator
46:40didn't allow that but we've just we have
46:42our own private fork that does have that
46:43feature in that hopefully will
46:44contribute but yeah comparator is what
46:46we want to use.
46:48Um
46:50okay so what kind of tasks do we have to
46:52to your question here? So um one task is
46:56just you given a theorem statement in
46:57lean and then you provide a a proof of
47:00that theorem. Uh we can extend that by
47:03um you know adding some auxiliary
47:05definitions as well. So it's not just a
47:06oneline theorem statement. There's some
47:08new mathematical structures that you're
47:09expected to reason about. Uh and we can
47:12also as I said we extend it to include
47:14false statements. So the u agent can
47:16provide a disproof instead. This is this
47:20feels intuitively likely to be helpful
47:21because the agent will sometimes come
47:23will hypothesize things that are false.
47:26So being able to um prove that things
47:28are false seems like a useful capability
47:30for it to have. Um an example of this
47:32kind of thing is a mini F2F data set
47:34which was quite early with sort of subly
47:37olympiad uh problems.
47:40Then um I think to the numinina people
47:44where are you? Oh thank you think of
47:46this. So this was a data set of uh
47:49100,000 problems from the numina people
47:52of sort of again olympiad and subly
47:54olympiad level. Um so we we have have
47:57been using these in training. Um again
47:59just as a here's a theorem statement now
48:01provide a proof.
48:03Um and then also another possible task
48:07is formalization. So take a paper from
48:09the archive. There are quite a lot. Um
48:12and then try and formalize the informal
48:14mathematics in in that paper. You know
48:16the probability means you have to add
48:18some definitions. Uh and then when
48:20you've done that um you can look at did
48:23you did you formalize and prove the
48:25right thing. This has to be as you're
48:26saying a little bit vibes based. So we
48:28no longer have the formal verification
48:30guarantee. So we actually don't use this
48:31for that reason but it is a possible
48:33task that one that one could do.
48:37Um here's another one. This is uh a
48:40Jacobian challenge from Kevin Buzzard uh
48:43recently published. So he's basically
48:45got uh some mathematics and he's removed
48:50some of the definitions and some of the
48:53and all the theorem proofs and replaced
48:55them with sorry.
48:58And so the task for the agent is to
49:00complete all the definitions and
49:01complete all the theorem proofs. The um
49:05this is very cool. We're going to try
49:06and doing it. The slightly difficult
49:08thing about this is it's quite um needs
49:10some expertise to um to construct. So if
49:14you see a line 50 uh there's a there's
49:18an additional theorem which I guess
49:19Kevin has added here to say that um
49:24you know the genus is zero if it's is
49:26empty I guess. Uh and the reason for
49:28that is it's avoiding some kind of
49:30trivial way that you could solve it by
49:31giving the wrong wrong definition. So
49:34getting this kind of thing right
49:36requires quite a bit of expertise and I
49:37would expect if we generated a large
49:39number of these the models would hack a
49:41lot of them. We'd have quite a process
49:42of iterating on them but it's still a
49:44very cool idea.
49:48Uh and then
49:50the last thing that we do is this idea
49:51of a boss theorem. So you may recognize
49:54this theorem statement in blue as the
49:58less theorem. We think given the current
50:00statement of math lab, it would probably
50:02take a million maybe two million lines
50:03of code um to to prove to prove this
50:07statement. Um there's an awful lot of
50:09mathematics that would need to be
50:10formalized in order to get there. Um so
50:13if we set that task to an agent and
50:15eventually did it, uh we could be
50:16reasonably confident that it got all the
50:18stuff about modular forms and and
50:20elliptic curves and so on correct even
50:22though we don't check any of it. The
50:23only thing we would need to check is
50:25have you actually ended up proving uh
50:27fossess theorem for example.
50:29Um this is quite appealing for us
50:32because um this it learns skills which
50:35are transferable to code and because I
50:38say proving one of these theorems would
50:40uh involve formalizing a large amount of
50:42theory. We are obviously not going to
50:44put precisely this one into training.
50:46There's no chance that a training agent
50:48can do it. But what we can do is
50:50similarly to um Kevin Buzzard's thing.
50:53We can take a complete proof uh complete
50:55formalization delete pieces of it and
50:57get the model to fill in to fill in
50:59those pieces. The the difference from
51:02the other thing is that uh in this case
51:04it doesn't depend on any esoteric
51:06definitions. So the the definitions so
51:10it will be we'll be asking the model to
51:12build theory and then prove something
51:13that doesn't depend on that theory. So
51:15we we don't have that kind of problem of
51:17it misformalizing something that the
51:20eventual proof depends on.
51:27And here we go. So this is why in
51:31summary this is why formal math is great
51:33for um agentic training for training uh
51:37modern AI. Uh we have this verifiable
51:40and unhackable reward. Um the training
51:43dynamics are nonadversarial as a result.
51:46So uh you know we encourages this sort
51:48of collaborative style of interaction
51:50that that we would ideally like and that
51:53we can have very challenging long
51:55horizon tasks you know up to millions of
51:57tokens or or more or indeed probably
52:00initially less. Um so all in all uh it's
52:04very valuable for general model training
52:07and as essentially a kind of happy
52:10accident. Um it also means that the
52:12models get very good at formalizing
52:14mathematics or is much better at
52:15formalizing mathematics. So my my last
52:19slide is um if models are good at
52:22mathematics uh what does that bias?
52:25So
52:27we could formalize two different kinds
52:29of mathematics. Stuff we already know
52:31and new mathematics. So the first thing
52:34I think uh to to dismiss is the idea
52:38that you formalize mathematics in order
52:40to verify that it's correct established
52:42mathematics. That's not what we're
52:43doing. you know the if um you know
52:46recent example was the formalization of
52:48this uh sphere packing optimality that
52:50was uh recently done by uh by math who's
52:54a great great piece of work and there
52:56was no doubt that that that mathematics
52:57was correct it wasn't being formalized
52:58in order to verify it its correctness
53:02um so why why do it well one reason
53:04would be to um formalize new mathematics
53:07autoformalize new mathematics on top of
53:08it so this mathematics the known
53:11mathematics provides the basis for uh
53:14new um you know untested mathematics and
53:17the other reason of course is to
53:19generate training data. So um by
53:21formalizing well established mathematics
53:23we can then uh use that as training data
53:26with where we delete pieces and ask the
53:28model to reproduce it. So that's one
53:30thing and then the other I think
53:32potentially more exciting end is
53:35formalizing new mathematics. So um
53:39I guess everyone is aware that the the
53:41volume of new mathematics is increasing.
53:43Uh anyone can um if you think what is
53:47the what is the time it would take for
53:49an expert to distinguish um a high
53:52quality uh human written mathematics
53:54paper from something that had been
53:55generated by somebody had no clue what
53:56they were doing with with AI then um
54:00historically the answer would have been
54:01you know a fraction of a second
54:02immediately obvious that the that this
54:04was this was nonsense. And I think now
54:06the answer is actually it can take you
54:08know several minutes for for an expert
54:10to to read a paper and think actually no
54:12this isn't right and um you can expect
54:15as AI gets better that that time taken
54:17will increase and um that's obviously
54:21not feasible.
54:23So um one thing is if we do formalize it
54:26um then uh it makes it easier to trust
54:30uh the results. Obviously, it's still
54:32the case that if it contains new
54:33definitions, then you need to scrutinize
54:35the definitions and make sure that
54:37there's nothing nothing lurking in there
54:39that makes makes everything trivial. But
54:41once you scrutinize the definitions,
54:42then uh understanding the theorem
54:44statements is relatively easy and then
54:47you can trust that those theorems have
54:48been proved to be true. Uh another thing
54:51is uh improving understanding and making
54:54more results more accessible. I think um
54:58you know mathematics can be a little bit
55:00unapproachable sometimes. Some sub
55:01fields have things conventions which are
55:03not written down. If you're not part of
55:05that field, it can be you know
55:06impossible to understand the literature
55:08for example. And by by formalizing um by
55:11formalizing things actually this should
55:12have been in the first group. By
55:14formalizing them we potentially enable
55:16uh you know anybody who wants to to
55:19sufficiently motivated to dig down and
55:21see what actually is going on here. Uh
55:23and then ultimately potentially we
55:26enable long-term autonomous AI research
55:28where uh if we if we're having AI carry
55:32out a research program over the course
55:33of multiple weeks, months um then being
55:37able to formalize what it does and then
55:39build upon that I think is going to be
55:42um extremely extremely valuable. Uh and
55:44then lastly generate new training data
55:48and uh
55:50it is me done.
55:52>> [applause]
55:57[applause]
55:57>> Thank you very much. Are there any
55:59questions? Yeah,
56:02>> there is a famous example of
56:04[clears throat] something where people
56:05would like to check it which is a much
56:08of claim proof of the ABC conjecture. Uh
56:11yes I I think it might be challenging to
56:13formalize uh what he has there but yeah
56:16um sorry
56:21what do you mean by understanding
56:24because I think u it's a word which is
56:28very very ambiguous.
56:30>> Yes. So what I had in mind there was um
56:32human understanding. So by um making it
56:36very explicit the um the you know the
56:39relationships between you know precisely
56:41which theorems are being used precisely
56:43which definitions are you depending on
56:44that can potentially uh enable enable
56:48humans to understand it better. So this
56:50is something that uh again I was
56:53discussing with the case of ferments
56:55theorem with Kevin Buzzard. So he's has
56:57a project to formalize the proof of
57:00theorem and he's saying the the reason
57:01he wants to do it is so that he
57:03understands the mathematics of the of
57:05the theorem better you know precisely so
57:07he can trace through the dependencies of
57:10um you know what body of theory does it
57:11depend on and and how and by having all
57:14of that in a single artifact where you
57:17can where you can you know trace all of
57:20the dependencies then then that
57:22understanding is easier to get to.
57:24>> Yeah.
57:26To follow up a bit on that,
57:28when people talk about live coding,
57:31there's this negative connotation a
57:33little bit sometimes because the code
57:35can get very unwieldy.
57:37Don't you see that there might don't do
57:39you perhaps foresee that there's a
57:41similar risk here that like little lemas
57:43proved independently even though there
57:45might be easy coronaries of of
57:47something?
57:49>> Yeah. No, no, we already we we see this
57:51already. Um uh so an example of this
57:55kind of thing is uh in a formalization
57:57project we're doing at the moment
57:59there's some um trivial case that we
58:02should have dealt with um right at the
58:05top and we didn't and that means that in
58:07every every lema subsequently it has to
58:10deal with this case potentially multiple
58:11times. So the proofs of our proofs have
58:14been blown out by thousands of lines
58:16because this is in every line
58:18essentially it has to then deal with
58:19this trivial case that we should have
58:20excluded. Um so so yes absolutely. Um
58:24the good thing is that models are
58:26getting better at doing what in code
58:28terms we s call refactoring where you
58:30take something that's already correct
58:31and say actually I want to improve the
58:33quality of it. I want to make it more
58:34concise, more expressive, this kind of
58:35thing. Um so I think it's a fixable
58:38fixable problem but um effectively our
58:41models need to relearn painfully what
58:43humans have learned which is that you
58:45know you need to design these things
58:46properly and and that it will that
58:49effort will repay itself in eventually
58:52>> uh in the blue back.
58:54>> Yeah I have two questions but I start
58:55with one. So in mathematics culture it's
58:58very important to give proper
58:59attribution to past works previous ideas
59:02and uh AI proofs are starting to be
59:05extremely impressive but they don't seem
59:06to be very good at this. So basically
59:09they provide like no context and no
59:11attribution of where the ideas come
59:13from. Is this like an intrinsic uh issue
59:17because the [snorts]
59:18models just don't know or is it just
59:21that for example there hasn't been
59:23reinforce reinforcement learning towards
59:25achieving this goal.
59:27>> Um yeah, I think it somewhat intrinsic
59:31in the sense that it's not something
59:32that we um we train for at the moment.
59:35Um that's said the tools like the
59:38co-matician that we have at deep mind
59:40has an explicit process of gathering
59:42citations and um and making sure that
59:45the correct citations are inserted and
59:47are accurate. So it's it's definitely
59:49it's a it's a solvable thing but um
59:52especially when the model uh kind of
59:55knows some fact in its weights it
59:58doesn't because of the way um the the
1:00:01LLM generation works it just because it
1:00:04it knows a fact it doesn't necessarily
1:00:06have immediate access to where that fact
1:00:08is from because you know the the two
1:00:11those two things may well not co-occur
1:00:14uh in the in the data set that it's been
1:00:15trained on the right the right
1:00:17So it has to be something that's
1:00:19explicitly train trained in Yeah. the
1:00:21the thing to to site correctly.
1:00:23>> Yeah. Yes. Quite possibly.
1:00:26>> Uh yeah.
1:00:27>> So follow the interestingness question.
1:00:30So in your different criterion of why
1:00:36the first one for me at least
1:00:43and and I
1:00:47for gentle with a few minutes
1:00:50sometimes it's a few thousand minutes
1:00:51just to realize this is pretty absurd
1:00:54>> um but on the so I know a lot of people
1:00:56are doing formalization to understand
1:00:58more the math behind and also working on
1:01:01automization and often criticism we get
1:01:02is that usually the automalized group
1:01:04well when you do automalization
1:01:07versus manual optimization then you lose
1:01:09this part of understanding the math
1:01:10behind um so I was curious of having
1:01:15having your clinics Yeah, I mean I think
1:01:17my view would be that um the the proof
1:01:21is probably not a very useful thing. So
1:01:24when you've auto finalized something and
1:01:26proved it then then you don't uh you end
1:01:29up with some proof and I think it would
1:01:30be fine if no human ever read that proof
1:01:32honestly. But typically the the proof is
1:01:36um you know refer to a large number of
1:01:38lemmers for example and if you follow
1:01:41that structure then that feels like
1:01:44quite often that's the right level of
1:01:45abstraction as a human to understand
1:01:47what's going on. You don't necessarily
1:01:49need to read the step-by-step proof. You
1:01:50can say uh you know this fact is true
1:01:52and it's true because of these these
1:01:54other these other facts. And I I think
1:01:56actually that's a great way to
1:01:57understand the the the way the overall
1:02:00argument hangs together. And then if you
1:02:02need to convince yourself that some fact
1:02:03is true then when when we have
1:02:06ubiquitous uh you know
1:02:08autoformmalization on demand you can you
1:02:10can ask for uh you wouldn't in that
1:02:13future be able to ask for this
1:02:15complicated result can I split it into
1:02:17is it still true if I weaken this
1:02:19hypothesis or and so on. So
1:02:24>> um you mentioned that if there are
1:02:26definitions that are formalized in the
1:02:28in the outputs you may need to check
1:02:31that they're not completely bogus. So do
1:02:34you see a solution to that definition
1:02:37problem without loop like something
1:02:40purely?
1:02:42>> Um
1:02:43yeah so there are there are kind of some
1:02:46some solutions. um you know one one very
1:02:49obvious thing is to uh get the AI to
1:02:52generate many examples and and
1:02:54non-examples. So if you if you define
1:02:56something new then um rather than just
1:02:58giving an abstract definition say okay
1:03:00this thing is is an example of it this
1:03:03thing is not um these kinds of trivial
1:03:05these kinds of trivial properties. I
1:03:07think um that can that can give you some
1:03:09reassurance that the definition is
1:03:11capturing the kind of thing that you
1:03:12want do you want it to capture. Um the
1:03:17um I mean the other thing to say is that
1:03:19we
1:03:20uh the models seem quite they they fudge
1:03:24definitions when they're stuck
1:03:25essentially. So the the
1:03:28anthropomorphizing a bit they want to do
1:03:30the right thing but if the uh if they
1:03:33can't make progress then then yes they
1:03:34will fudge a definition sometimes or of
1:03:36course they may make a mistake. Um but
1:03:38having other LLMs that review the output
1:03:40and say do the all these definitions
1:03:42seem to be correct is is a useful
1:03:44signal. Um but on the other hand I would
1:03:46say ultimately if if the question you
1:03:49want to answer is is this mathematical
1:03:50object interesting to humans then you
1:03:52have to have a human answer answer that
1:03:54question really. There's no um automated
1:03:56substitute for that.
1:03:58>> Uh
1:04:00was that me? Uh so have you done the
1:04:03inverse process of starting with lean
1:04:06code and then can you please translate
1:04:08this into a human readable thing and
1:04:10maybe even like going between like human
1:04:13readable well which human well write it
1:04:15for an expert write it for a novice
1:04:16write it for somebody who knows this but
1:04:18not that and stuff like that.
1:04:20>> Um yes absolutely you know models are
1:04:21quite good at that. Um so typically the
1:04:24way we do this is by saying first
1:04:26translate each line and then you're in
1:04:28entirely natural language and then you
1:04:29can go through and say okay summarize it
1:04:31and you can get as you say increasingly
1:04:33brief proofs that uh you know skip over
1:04:36the more boring details. So so yes that
1:04:38that does work. Um we tried using that
1:04:41kind of data as training data not very
1:04:42successfully because um the the English
1:04:46that we ended up with was rather
1:04:47stylized and repetitive. So it wasn't
1:04:49very useful as Gemini training data but
1:04:51it's perfectly usable as something that
1:04:53could explain it. I mean as a followup
1:04:56so if we imagine that a goal is proving
1:04:58theorems I know that's that's a weighty
1:05:01statement but let's say that's a goal is
1:05:03to prove theorems and is it is it clear
1:05:06that the best route is [clears throat]
1:05:08um not just to go directly to the lean
1:05:11if all you cared about is is the theorem
1:05:14true and you didn't care about
1:05:15understanding it or whatever. So in your
1:05:18system, you're going via natural
1:05:19language and then you have the checker
1:05:21who's a robot who reads the lean. But
1:05:24maybe we could just skip that and just
1:05:26have everything done directly in lean.
1:05:28Is that a terrible idea or
1:05:31>> um so this is what alpha proof did um is
1:05:35it went straight to lean. There's no
1:05:36natural language involved. I would say
1:05:38since in the last two years or so the
1:05:41capabilities of things like Gemini have
1:05:43just so far outstripped our lean
1:05:46capabilities that uh that no it doesn't
1:05:48make sense to do that now because uh so
1:05:53typically what we find is if if if we're
1:05:55capable of formalizing it then we're
1:05:58we're capable of proving it. So the the
1:06:01there is pretty much nothing I would say
1:06:03where the formal first approach is
1:06:05stronger at the moment. Our best
1:06:08formaliz our best formal provers do go
1:06:10via via natural language.
1:06:11>> Is that a statement about the nature of
1:06:13mathematics or like what like why is
1:06:15that the case that to prove a theorem
1:06:17it's best to go via human language?
1:06:20>> I mean I I would say it's something like
1:06:23if if you were trying to prove something
1:06:24that you didn't know was true then
1:06:26sitting down and constraining yourself
1:06:27to say right I'm going to first write
1:06:28the first line of the proof would not be
1:06:31like the best constraint. And that's
1:06:33effectively what you're doing if you say
1:06:35you go lean first. Um you're much more
1:06:38likely to do some more let's think about
1:06:41some related things. Let's read some
1:06:43literature and and uh you know let's
1:06:45make some hypotheses about reasons why
1:06:48it might be true. And all of that is
1:06:49much more naturally done in uh natural
1:06:52language rather than lean where you
1:06:53basically have to state something very
1:06:56very precise and then and then attempt
1:06:57to prove it.
1:07:01Can you say a little bit more about
1:07:02skill transfer? So, a couple times in
1:07:04your talk, you mentioned how uh training
1:07:06for one task helped the model overall,
1:07:08which I assume you don't know why that's
1:07:10true, but you have empirical evidence
1:07:11for. I I would have thought that unless
1:07:13you really uh got the the level of the
1:07:16reinforcement rewards exactly calibrated
1:07:18right, that you could actually kind of
1:07:19drift off such that learning how to do
1:07:21math would actually hurt you on other
1:07:22tasks. Do you have a sense of why it
1:07:24causes general generalization?
1:07:25>> Um, no. But it's very handy that it's
1:07:28true. Um I would say we we see this
1:07:30across the board that um essentially if
1:07:34you if you train exclusively on one task
1:07:36then then yes you'll you'll lose the
1:07:38ability to do other things. But what we
1:07:40do is we always train on a on a mixture
1:07:42of tasks. So even our like hyper
1:07:44specialized math models t train like 10%
1:07:46math and 90% other stuff. Um and if you
1:07:51as long as you maintain that kind of
1:07:52that kind of mixture then uh we find
1:07:55that more training with novel tasks is
1:07:58is just beneficial to to more or less
1:08:00everything. I mean not it's not a 100%
1:08:02guarantee but uh you know across
1:08:05improving across the board is very
1:08:06common.
1:08:09>> I want to go back to this
1:08:12the previous question. I mean when you
1:08:14say that the agent they go through
1:08:16natural language but for humans
1:08:20depending on your natural language your
1:08:22brain works differently. I mean if your
1:08:24natural language is French, German,
1:08:26Russian, Chinese, English in fact there
1:08:29are cognitive processes that are very
1:08:32different that are induced by the
1:08:34structure of the grammatical
1:08:36language that you use. Uh unless you are
1:08:39telling me that uh everything has to be
1:08:41in English these days and uh I I feel
1:08:45that maybe something is getting lost. I
1:08:48mean, Young explained that he got the
1:08:50young I mean, what you call the young
1:08:51mil in physics
1:08:54was inspired by a Chinese. So, you know,
1:08:57so
1:08:58>> we we actually I I think you're right.
1:09:00We we actually see um quite a lot of
1:09:02Chinese inside the model's thoughts.
1:09:04Incidentally, even when the even when
1:09:07the question and the answer are in in
1:09:08English, uh the the thoughts quite often
1:09:10go through Chinese.
1:09:15So um my question would be a little more
1:09:19high level. Um I don't know how to ask
1:09:23it. So like what is the as a group like
1:09:27you work in a big group of people with a
1:09:30big group of people and you have some
1:09:32vision on what you are doing. So what is
1:09:35the end goal like what is what are you
1:09:38trying to do right? when is the thing
1:09:40you say, "Okay, we we did it."
1:09:43>> Uh, okay. I mean, for for me for me
1:09:47personally, it's uh it's a highly
1:09:50capable AI assistant. That's that's the
1:09:52thing that I'm working on. And I'm doing
1:09:54formal mathematics because it
1:09:56contributes to contributes to that goal.
1:09:59Um, I also care about mathematics for
1:10:00own for its own sake, but I'm kind of
1:10:02that's a side benefit. Exactly.
1:10:05>> I mean when you say a very powerful like
1:10:09co-maician or how you say it.
1:10:11>> Sure.
1:10:12>> Um
1:10:14how powerful like [laughter]
1:10:17>> I mean uh I don't know
1:10:20>> I mean
1:10:20>> I don't know how to ask this question.
1:10:22>> I I don't know how to answer it really.
1:10:23I mean we want
1:10:27[laughter]
1:10:28>> it's a question of resource. How do you
1:10:30determine how much or how many resources
1:10:33you're going to mobilize
1:10:35to achieve something that in in part
1:10:38seems to be from the outset a
1:10:41self-created problem because you you
1:10:44might not have the the the training data
1:10:48or the capacity for instance to teach
1:10:51your agents not to cheat.
1:10:54um because that doesn't appear in in
1:10:57input you provide and you obviously have
1:11:00to work under a constraint because and
1:11:03that's I guess the main reason why you
1:11:05want a multimodal agent that is capable
1:11:10of doing many different things ideally
1:11:12everything you you throw at it uh
1:11:15because there's a terrible amount of
1:11:17resources you're mobilizing which might
1:11:19be you know might be better to use them
1:11:21differently in order to achieve a given
1:11:25task or mathematical problem. So is
1:11:27there a measure something formal you can
1:11:29define and that tells you okay we're
1:11:32we're going to go until here and if we
1:11:35haven't solved it with this amount of
1:11:37resources we might be on the wrong path
1:11:40alto together and might have to
1:11:41backtrack and try something else. Um
1:11:44yeah so the Gemini so the Gemini team is
1:11:49huge and there are many many things
1:11:51being tried in Gemini any any time and
1:11:53many of them don't work but the the
1:11:55overall product gets um gets better
1:11:58because some things some things do work
1:12:00and so um I guess I'd say that
1:12:04ultimately the majority users of Gemini
1:12:06are going to be outside of Google and um
1:12:09therefore what we want is to provide
1:12:10them with something that is as capable
1:12:12as possible that they can do all the
1:12:13things that they want to do. We we
1:12:15obviously internally inside Google we
1:12:16have we have problems that we want to
1:12:17solve. So we um we have obviously a lot
1:12:20of engineering and coding problems that
1:12:22that we want Gemini to help us with and
1:12:24also within the science unit we um have
1:12:27you know goals around doing impactful
1:12:30impactful science that that we want to
1:12:31pursue as well. But in terms of you know
1:12:35actually making Gemini better um I
1:12:37suppose the other thing to say is that
1:12:38there is in ML and AI there's still an
1:12:41awful lot of lowhanging fruit. So um
1:12:43there are always you can you try
1:12:45something do it for a few months it's
1:12:47not working people move on and try and
1:12:49try something else there's always
1:12:50something else you can do that will work
1:12:52so you know in math I mean in number
1:12:54theory when you have a big I mean hard
1:12:57problem to solve you would ask Zagi and
1:13:00Zagi would give an answer sometime
1:13:04immediately or after a few hours
1:13:07I mean how do you compare to what could
1:13:09do I mean on specific non
1:13:12problem.
1:13:15>> Okay. I don't I don't know. I'm I'm
1:13:17sorry. I can't I can't tell you.
1:13:18>> You're like 1% of his IVA. [laughter]
1:13:22>> Okay.
1:13:24>> We have some way to go.
1:13:27>> So, you know, many great results in math
1:13:29are sort of obtained by great
1:13:33mathematician by analogy. you know you
1:13:35have this algebraic geometry and then
1:13:37somehow you have this
1:13:40the back of your mind or sometimes
1:13:43unexplainable that you know oh this is
1:13:45similar to some argument I've seen I
1:13:48don't know complex analysis or something
1:13:51is there any chance that lean for
1:13:54example or do you you know may sort of I
1:13:56guess it's related to justice that you
1:13:59know might be some kind of inference you
1:14:02know what I like analogy or that
1:14:05formalizing things might help with that
1:14:07or I mean that would be
1:14:09>> I I think honestly
1:14:10>> very bad for
1:14:10>> mathematicians
1:14:12my guess is that it's the opposite
1:14:14honestly that um when you formalize
1:14:16something you tend to be uh locked in to
1:14:19one particular way of way of looking at
1:14:21it and um it's much easier to if there
1:14:24are much easier to skip between to do
1:14:27analogies or to look at things from
1:14:28different angles in the informal space
1:14:30and then only when you're confident you
1:14:32know what you want to do then to then to
1:14:34move into the formalization.
1:14:36So we we quite often see uh by the way
1:14:38that there are problems uh like IMO type
1:14:43problems where you could formalize it in
1:14:44multiple ways and there will be some way
1:14:46you can formalize it that uh is easy to
1:14:49solve. If you formalize it differently
1:14:51it may be impossible for the model to
1:14:52solve it even though to a human they're
1:14:55obviously equivalent but just the one
1:14:58way you formalized it is kind of quite
1:14:59hostile to than a than a formal proof.
1:15:02There's still hope.
1:15:05>> Well, no, we can think of natural.
1:15:08>> Yeah, I think there are lots but there
1:15:11are kind of two clear answers to the
1:15:12question out there about where where is
1:15:14the scoring? What are the standards and
1:15:17and [clears throat] one of them is when
1:15:20either the computer or you know humans
1:15:22with computer assistance prove some
1:15:24impressive new result which is really
1:15:26new and arguably that has just happened
1:15:28you know with open AI and the human
1:15:30distance problem and that will keep
1:15:33happening and that's that's obviously
1:15:36one and then the other you might not
1:15:38like but it's to say there'll come some
1:15:40point where mathematicians and
1:15:42scientists feel they just have to have
1:15:44this access to this technology to
1:15:45compete.
1:15:47I don't know if that'll be
1:15:52>> um with recent focus on like um cyber
1:15:56security vulner vulnerabilities
1:15:58especially brought by cloud methods and
1:16:00such. Are you worried at all that um
1:16:04because at the end of your talk you seem
1:16:06to
1:16:08um like share a vision where maybe we
1:16:11don't even check the lean in the future
1:16:13and we have like some super intelligent
1:16:15mathematician
1:16:17that we just trust that it does compile
1:16:20in lean and it's done. But if it does
1:16:22try to hack into lean and if it does
1:16:25really become intelligent enough to
1:16:28maybe find vulnerabilities that we
1:16:30aren't aware of, then if we don't look
1:16:35into it ourselves um in the future, are
1:16:38you scared that we might like complete
1:16:41the script with uh what it's uh
1:16:46[clears throat] providing us as
1:16:48>> uh I mean I think for mathematics might
1:16:50not be my concern in in that area
1:16:53honestly. But but yeah, yes, I think it
1:16:55is definitely the whole question of how
1:16:57do [clears throat] we ensure that AI is
1:16:59kind of aligned to to human preferences
1:17:02and does things that we consider
1:17:03valuable and uh doesn't try to fool us
1:17:06about them. Yeah, I think that is a I
1:17:08mean it's a big topic of research and I
1:17:10think an important one for sure.
1:17:13Yeah.
1:17:14>> Any lessons for journal editors on the
1:17:17frontier map?
1:17:21Uh I mean what the the situation I would
1:17:24love to be in is that we are able to
1:17:27give um editors and others something
1:17:29that will take a take a submitted paper
1:17:32formalize it verify it and say okay here
1:17:35are the here's a formalized version of
1:17:36the paper and these this is the
1:17:39formalized version of the claims and
1:17:40these claims are correct and uh I think
1:17:44that would be you know potentially
1:17:46extreme you know extremely valuable for
1:17:48uh the whole process of kind of
1:17:50disseminating and sharing mathematics
1:17:52without uh the very costly and
1:17:56timeconuming process of of somebody
1:17:58having to you know verify it line by
1:18:00line and you know essentially stake
1:18:02their reputation on it on it being
1:18:03correct before before it's disseminated.
1:18:06So I would love us to get there. I don't
1:18:08think it's going to happen immediately
1:18:10but it would be very cool.
1:18:12>> Yeah. Um how much do you think this can
1:18:15teach us about uh semiformalizing things
1:18:19like physics where there's a certain
1:18:20amount of math
1:18:22formalized? There's also a lot of things
1:18:24that involve you know approximations and
1:18:27maybe old defined intuitions and things
1:18:29like that where you cannot
1:18:32at least with our current knowledge
1:18:33formalized.
1:18:35>> Yeah. Um I think the particular
1:18:37advantages that I was talking about only
1:18:39really occur in the fully formalized
1:18:41setting where there is literally no uh
1:18:44gap to um hack a reward function for
1:18:47example. So as you see when I we we left
1:18:49a tiny gap for the agent to to exploit
1:18:51it did it. So I think the the thing
1:18:55those specific benefits of formalization
1:18:57probably only occur there. Um, but in in
1:19:01general, I think the idea that by
1:19:03formalizing something or
1:19:05semiformmalizing it that you are you can
1:19:08trust it more than if you didn't do that
1:19:10at all, that feels to me like it might,
1:19:13uh, you know, might extend more
1:19:15generally. So, I mean, to take a rather
1:19:17more mundane example, um,
1:19:20uh, you know, calendars and time zones,
1:19:22our models quite often make mistakes
1:19:24with those. And if we enabled it enabled
1:19:27them to like do formal reasoning about
1:19:31uh what time of day it is and what time
1:19:32of day it is in different places and how
1:19:34that relates when you travel I think I
1:19:35think that would increase reliability in
1:19:37that area. Um I think yeah there are
1:19:39lots of areas like that I think where
1:19:41some degree of formalization will
1:19:42probably improve reliability.
1:19:44>> Okay maybe should we can ask one last
1:19:48question in a different vein.
1:19:51Uh so so everything we talked about
1:19:53today here in your talk was about
1:19:56basically prompt engineering in some way
1:19:58right kind of because it has to do with
1:20:00the way you do the reinforcement
1:20:02learning
1:20:04it's a way of uh looking at the internal
1:20:06representation states of the model and
1:20:09say something about that. Uh okay so I I
1:20:12want to distinguish so prompt
1:20:14engineering is a kind of older
1:20:16>> older technology uh where you you modify
1:20:20the behavior of the model by uh by by
1:20:24the input that you give it. So what we
1:20:26do now with RL is related but we instead
1:20:29modify the behavior of the model by
1:20:30updating the weights so that when you
1:20:32give it the thing that you want it
1:20:34produces the thing you want. They're
1:20:35both they're both ways of basically
1:20:37shifting the output distribution and but
1:20:39they are slightly different um in terms
1:20:41of looking at the internal
1:20:42representations. Yes, there are there
1:20:43are people who do this. The general um
1:20:45topic is called mechanistic interpret
1:20:47interpretability
1:20:49uh where you we can do things like train
1:20:52uh models which look at the activations
1:20:54of the the neural network and try to
1:20:57make deductions about them. I would say
1:20:59at the moment this this field is very
1:21:01much in its infancy. But um you know we
1:21:05can do things like if if a model tries
1:21:08to do a multiplication without using a
1:21:10tool for example we can have a look at
1:21:13we can see how does it do how does it do
1:21:15a multiplication and so people have
1:21:17found in models [laughter] you there are
1:21:19clearly bits of the network that know
1:21:21all of the two-digit multiplications.
1:21:23Those are just memorized facts. And then
1:21:25and then there are other bits which do
1:21:27the kind of um the process of guessing
1:21:30how long the answer should be, how many
1:21:32digits it should be. And yes, you can by
1:21:34inserting probes into the network and
1:21:36training classifier, small classifiers
1:21:38on top of the weights you observe from
1:21:40the network, you can see what it's
1:21:42doing. But it is at this very um we we
1:21:46don't have a deep understanding I would
1:21:47say of what's going on.
1:21:49>> Okay. Any final question? Okay, let's
1:21:52thank

About this transcript

This page contains the full transcript of Edward Lockhart - Why AI Needs Formal Mathematics by Institut des Hautes Etudes Scientifiques (IHES), generated from the public captions YouTube serves with the video. The transcript has 14,590 words across 2,073 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.