YT YouTube2Text

YouTube2Text → Transcripts → YouTube transcript (9vM4p9NN0Ts)

YouTube transcript (9vM4p9NN0Ts) — Transcript

19,443 words · 2,704 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:05so let's get started uh so I'll be
0:07talking about building llms today um so
0:10I think a lot of you have heard of llms
0:12before uh but just as a quick recap uh
0:16llms standing for large language models
0:18are basically all the chat Bots uh that
0:20you've been hearing about recently so uh
0:24Chad GPT from open ey Claud from
0:27entropic Gemini and and lman other type
0:30of models like this and today we'll be
0:32talking about how do they actually work
0:34so it's going to be an overview because
0:35it's only one lecture and it's hard to
0:37compress everything but hopefully I'll
0:39touch a little bit about all the
0:40components that are needed to train uh
0:42some of these llms uh also if you have
0:44questions please interrupt me and ask uh
0:47if you have a question most likely other
0:49people in the room or on Zoom have other
0:52have the same question so please ask um
0:56great so what matters when training llms
0:59um so there a few key components that
1:01matter uh one is the architecture so as
1:04you probably all know LMS are newal
1:06networks and when you think about new
1:08networks you have to think about what
1:10architecture you're using and another
1:12component which is really important uh
1:13is the training loss and the training
1:15algorithm um so how you actually train
1:18these models then it's data so uh what
1:21do you train these models on um the
1:24evaluation which is how do you know
1:26whether you're actually making progress
1:28towards the goal of of uh llms and then
1:32the system component so that is like how
1:34do you actually make these models run on
1:37uh Modern Hardware which is really
1:38important because these models are
1:39really large um so now more than ever
1:42system is actually really an important
1:44topic um for
1:46llms so those five components um You
1:50probably all know that llms and if you
1:52don't know LMS are all based on
1:54Transformers or at least some version of
1:56Transformers uh I'm actually not going
1:58to talk about the AR lecture today uh
2:01one because I gave a SE lecture on um
2:04Transformers a few weeks ago and two
2:06because you can find so much information
2:08online on uh Transformers but I think
2:10you can it's there's much less
2:12information about the other four topics
2:14so I really want to talk about those um
2:17another thing to say is that most of
2:19Academia actually focuses on
2:21architecture and training algorithm and
2:23losses um as academics and I've done
2:26that for a lot big part of my career is
2:29simply we like thinking that this is uh
2:31like we make new architectures new
2:33models and it it seems like it's very
2:36important but in reality honestly what
2:38matters in practice is mostly the three
2:40other topics so data evaluation and
2:43systems uh which is what of most of
2:45Industry actually focuses on um so
2:48that's also one of the reason why I
2:49don't want to talk too much about the
2:51architecture uh because really the rest
2:52is super
2:53important um great so overview of the
2:56lecture I'll be talking about
2:57pre-training so pre-training uh you
2:59probably heard that word this is the
3:01general word this is kind of the
3:02classical language modeling uh Paradigm
3:06uh where you basically train your
3:07language model to essentially model all
3:09of internet and then there's a post
3:11training which is a more recent Paradigm
3:13which is taking these large language
3:14models and making them essentially AI
3:17assistants um so this is more of a
3:19recent Trend since Chad GPT uh so if you
3:23ever heard of gpt3 or gpt2 that's really
3:25pre-training land uh if you heard of
3:28chat GPT which you probably have this is
3:30really posttraining land uh so I'll be
3:32talking about both but I'll start with
3:34pre-training and uh specifically I'll
3:36talk about what is the task of
3:38pre-training llms and what is the laws
3:40that people actually
3:42use so language modeling this is a quick
3:45recap uh language models at a high level
3:48are simply models of probability
3:50distribution over sequences of tokens or
3:52of words so it's basically some uh model
3:56of P of X1 to XL where X1 is basically
3:59word one and Excel is the last one in
4:01the sequence or in the sentence um so
4:04very concretely if you have a sentence
4:06like the mouse ate the cheese what the
4:08language model gives you is simply a
4:10probability of this sentence being
4:13uttered by a human or being found on on
4:16online uh so if you have another
4:18sentence like the the mouse at cheese uh
4:22here there's grammatical mistakes so the
4:23model should know that this uh should
4:25have some syntactic knowledge so it
4:27should know that this has less
4:29likelihood of appearing
4:31online uh if you have another sentence
4:34like the cheese ate the mouse uh then
4:36the model should hopefully know about
4:38the fact that usually cheese don't eat
4:41Mouse um so there's some semantic
4:43knowledge and this is less likely than
4:44the first sentence so this is basically
4:46at a high level what language models are
4:49um one word that you probably have been
4:51hearing a lot in the news are generative
4:53models uh so this is just something that
4:55can generate models that can generate
4:57sentences or can generate some data uh
4:59the reason why we say language models
5:01are generative models is that once you
5:02have a model of a distribution you can
5:04simply sample from this model and now we
5:06can generate data uh so you can generate
5:08sentences uh using a language
5:11model so the type of models that uh
5:14people are all currently using are what
5:16we call Auto regressive language models
5:19and the key idea of autor regressive
5:21language models is that you take this
5:23distribution over words and you
5:26basically decompose it into the into the
5:28distribution of the first word multiply
5:31the by the distribution of or the
5:33likelihood of the distribution of the
5:34second word given the first word uh
5:37multiply by P of the third word given
5:39the first two words um so there's no
5:42approximation here this is just the
5:43chain rule of probability which you
5:44hopefully all know about uh really no
5:46approximation this is just one way of
5:48modeling a
5:49distribution uh so slightly more
5:51concisely you can write it as a product
5:53of U of PS of the next word given
5:56everything which happened in the past so
5:58of the context and uh so this this is
6:00what we call Auto regressive language
6:02models again this is really not the only
6:04way of modeling distribution this is
6:06just one way uh it has some benefits and
6:09some downsides one downside of
6:11autoaggressive language models is that
6:13when you actually sample from this
6:14autoaggressive language model you
6:16basically have a for Loop which
6:17generates the next word then conditions
6:20on that next word and then regenerate an
6:22other word so basically if you have a
6:24longer sentence that you want to
6:25generate you it takes more time to
6:27generate it uh so there are some
6:28downsides of this current Paradigm but
6:31that's what we currently have so I'm
6:33going to talk about this
6:34one uh great so Auto regressive language
6:37models at a high level um what the task
6:40of autoregressive language model is is
6:42simply predicting the next word as I
6:43just said so if you have a sentence like
6:45she likely prefers uh one potential next
6:48word might be dogs and the the way we do
6:51it is that we first tokenize so you take
6:55these words or subwords you tokenize
6:57them um and then you give an IDE for
7:00each token so here you have 1 2 three uh
7:03then you pass it through this black box
7:05as I already said we're not going to
7:06talk about the architecture you just
7:07pass it pass it through a model and you
7:10then get a distribution a probability
7:12distribution over the next word over the
7:15next token and then you sample uh from
7:19this distribution you get a new token
7:21and then you DET tokenize so you get a
7:23new ID you then DET toonize and that's
7:25how you basically sample from a language
7:27model uh one thing which is important to
7:29not is that the last two TS uh two steps
7:31are actually only need needed during
7:33inference uh when you do training you
7:35just need to predict uh the most likely
7:37token and you can just compare to the
7:39real token which happen in practice and
7:41then you basically change the weights of
7:43your model to increase the probability
7:45of generating that
7:48token um great so autoaggressive neural
7:51language models so to be slightly more
7:53specific still without talking about the
7:54architecture uh the first thing we do is
7:57that we have all of these oh sorry yes
7:59on the previous slide when you're
8:01predicting the probability of the next
8:02tokens does this mean that your final
8:04like output VOR has to be the same
8:06dimensionality as the number of tokens
8:08that you have yes how do you deal with
8:11like if you have more to like if you're
8:13adding more tokens to your cor something
8:16yeah so we're going to talk about
8:17tokenization actually later uh so you
8:19will get some sense of this you
8:22basically can deal with adding new
8:24tokens I am I'm kind of exaggerating
8:26there are methods for doing it but
8:27essentially people don't do it um so
8:30it's really important to think about how
8:32you tokenize your text and that's why
8:33we'll talk about that later but it's a
8:36very good point to notice that you
8:37basically the vocabulary size so the
8:38number of tokens that you have is
8:40essentially the output of your uh
8:42language model so it's actually pretty
8:44pretty
8:45large okay so autoaggressive new
8:47language models first thing you do is
8:49that you take every word or every token
8:52you embed them so you get a um some
8:55Vector representation for each of these
8:57tokens um you pass them through some ual
8:59Network as we said it's a Transformer
9:01then you get a representation for all
9:03the word in all the words in the context
9:06so it's basically representation of the
9:08entire sentence uh you pass it through a
9:10linear layer as you just said to
9:13basically map it to the number so that
9:16the output the number of outputs is the
9:17number of tokens uh you then pass it
9:20through some soft Max and you basically
9:22get uh probity distribution over the
9:25next words given every word in the
9:27context
9:30and the law that you use is basically
9:32it's essentially a task of classifying
9:34the next token so it's a very simple
9:36kind of machine learning task so you use
9:37the cross entry P loss where you
9:39basically you look at the actual Target
9:43that happened which is a target
9:45distribution which is a one hot encoding
9:46which here in this in this case says I
9:49saw uh the real word that happened is
9:51cat so that's a one hot um distribution
9:54over cat and here this is the actual uh
9:57do you see my mouse oh yeah this is the
9:59distribtion that you generated and
10:00basically you do cross entropy which
10:02really just increases the probability of
10:03generating cat and decreases all the the
10:05probility of generating all the other
10:07tokens one thing to notice is that as
10:10you all know again uh this is just
10:12equivalent to maximizing the text log
10:14like the text log likelihood because you
10:16can just rewrite the the max over the
10:20probability of um this autoregressive
10:22language moding task as just being this
10:25minimum over I just added the log here
10:27and minus which is just the minimum of
10:30the loss which is the cross enty loss so
10:31basically minimizing the loss is the
10:33same thing as maximizing the likelihood
10:35of your text any question
10:42questions
10:43okay
10:45tokenizer um so this is one thing that
10:48people usually don't talk that much
10:50about tokenizers are extremely important
10:53uh so it's really important that you
10:54kind of understand at least uh what they
10:56do at a high level so why do we need
10:58token in the first place uh first it's
11:01more General than words so one simple
11:04thing that you might think is oh we're
11:05just going to take every word that we
11:06will have you just say every word is a
11:09new is a token in its own um but then
11:11what happens is if there's a typo in
11:13your word then you might not have any
11:16token associated with this this word
11:19with a typo and then you don't know how
11:20to actually pass this word with a typo
11:22into a large language model so what do
11:24you do next and also even if you think
11:27about words words is a very like words
11:29are fine with like Latin based languages
11:32uh but if you think about a language
11:34like taii you won't have a simple way of
11:36tokenizing by spaces because there are
11:37no spaces between words um so really uh
11:41tokens are much more General Than Words
11:43first thing second thing that you might
11:45think is that you might tokenize every
11:47sentence character by character you
11:49might say a is one token b is another
11:51token uh that would actually work and
11:54probably very well the issue is that
11:56then your sequence becomes super long
11:58and as you probably remember from the
12:00lecture on on Transformers uh the
12:02complexity uh grows quadratically with
12:05the length of sequences so you really
12:07don't want to have a super long sequence
12:09um so tokenizers basically try to deal
12:13with those two problems and give common
12:17subsequences a certain token and usually
12:20how you should be think about is around
12:22uh an average every token is around
12:24three four letters
12:26um and there are many algorithm for
12:29tokenization I'll just talk about one of
12:31them to give you a high level which is
12:32what we call bite P en coding which is
12:34actually pretty common one of the two
12:35most common tokenizers and the way that
12:38you train a tokenizer is that first you
12:40start with a very large Corpus of text
12:42and here I'm really not talking about
12:44training a large language model yet this
12:45is purely for the tokenization step uh
12:48so this is my large Corpus of text with
12:50these five words um then you associate
12:54every character in this Corpus of text a
12:57different token uh so here I just split
12:59up every character with a different
13:01token uh and I just color coded all of
13:04those tokens and then what you do is
13:06that you go through your text and every
13:08time you see pairs of tokens that are
13:11very common the most common pair of
13:13token you just merge them so here you
13:15see three times the the the tokens T and
13:19O next to each other so you're just
13:21going to say this is a new token and
13:22then you continue you repeat that so now
13:24you have to talk which happens three
13:27times to with an E that happens sorry
13:31two times and an token which happens
13:34twice and then ex which also happen
13:36twice so this is that if you were to
13:39train a tokenizer on this Corpus of text
13:41which is very small that's how you would
13:43uh finish with a token with a pre like a
13:45trained tokenizer uh in reality you do
13:48it on on much larger corpuses of text um
13:51and this is the real tokenizer of uh
13:54actually I think this is gpt3 or chat
13:56GPT uh and here you see how it would
13:59actually separate these words so
14:00basically you see the same thing as what
14:01we gave in the previous example token
14:04becomes its own token so tokenizer is
14:08actually split up into two tokens token
14:10and iser um so yeah that's all about
14:14tokenizers any questions on that yeah
14:16how do you deal with spes and how do you
14:18deal
14:19with yeah so actually there's a a step
14:22before tokenizers which is what we call
14:24pre- tokenizers which is exactly what
14:26you just said uh so this is mostly
14:29in theory there's no reason to deal with
14:31spaces and punctuation separately you
14:34could just say every space gets its own
14:36token every um uh punctuation get its
14:39own token and you can just do all the
14:41merging the problem is that so there's
14:43an efficiency question actually training
14:45these tokenizes takes a long time uh so
14:48you better off because you have to
14:49consider every pair of token so what you
14:52end up doing is saying if there's a
14:53space this is very like pre- tokenizes
14:55are very English specific you say if
14:57there's a space we're not going to start
14:59looking at the the token that came
15:00before and the token that came
15:02afterwards so you're not merging in
15:04between spaces but this is just like a
15:07optimiz like a computation optimization
15:10you could theoretically just deal with
15:11it um the same way as you deal with any
15:13other character and yeah when you merge
15:17tokens do you delete the tokens that you
15:19merged away or do you keep the the
15:21smaller tokens that merge um you
15:23actually keep the smaller tokens I mean
15:25in reality it doesn't matter much
15:26because um usually on large Corpus of
15:31text you will have actually everything
15:32uh but you usually keep the small ones
15:34and the reason why you want to do that
15:35is because if in case there's as we said
15:37before you have some um some grammatical
15:40mistakes so some typos you still want to
15:42be able to represent these words by
15:44character um so yeah yes are the tokens
15:50unique so I mean say in this case T Ken
15:54is there only one occurrence or could do
15:56you need to leave multiple occurr so
15:59they could have take on different
16:00meanings or something oh oh I see what
16:02you say no no it's every token has its
16:05own uh unique ID um so a usual this is a
16:10great question for example if you think
16:11about a bank which could be bank for
16:14like money or bank like water um it will
16:16have the same token but the model will
16:18learn the Transformer will learn that
16:20based on the words that are around it it
16:23should associate that I'm saying I'm
16:25being very high wavy here but associate
16:27that with the with a with a
16:29representation that is either more like
16:31the bank money side or the Bank water
16:34side um but that's a Transformer that
16:35does that it's not a
16:37tokenizer yes yeah so you mentioned
16:40during tokenization keep the smaller
16:41tokens you started with right like if
16:44you start with a t you keep the T and
16:46then you build your tokenizer to the
16:48that you can now in token so let's say
16:50maybe you didn't train on token but like
16:52in your data you are trying to encode
16:54token so how does the tokenizer know to
16:57encode it with token or
17:00a great question you basically when you
17:01so when you tokenize so that's after
17:03training of the tokenizer when you
17:04actually apply the tokenizer you
17:06basically always choose the largest uh
17:09token that you can apply uh so if you
17:11can do token you will never do T you
17:13will always do token um but there's
17:16actually so people don't usually talk
17:18that much about tokenizers but uh
17:20there's a lot of of computational
17:22benefits uh or computational tricks that
17:24you can do for making these things
17:26faster uh so I really don't think we and
17:28honestly I think a lot of people think
17:29that we should just get away from
17:31tokenizers um and just kind of tokenize
17:34character by character or bites by bites
17:37uh but as I said right now there's this
17:38issue of like length uh but maybe one
17:40day like in five or 10 years we will
17:42have different architectures that don't
17:43scale quadratically with the length of
17:45the sequence and uh maybe we'll um yeah
17:49move away from tokenizes so can you
17:51share with us the drawback why do people
17:53want to move away from the tokenizer oh
17:56um yeah so think
18:00one good example is uh math if you think
18:03about math actually numbers right now
18:06are not tokenized so for example 327
18:08might have its own token which means
18:10that models when they see numbers they
18:13don't see them the same way as we do and
18:15this is very annoying because what I
18:17mean the reason why we can kind of
18:18generalize with math is because we can
18:20deal with every every letter separately
18:22and we can then do composition where you
18:24know that basically if you add stuff
18:26it's just the same thing as adding every
18:28one separately plus like whatever the
18:29unit that you add so they can do that um
18:32so then you have to do like special
18:34tokenization and like one of the big
18:36changes that GPT 4 did uh is changing
18:40the way that they tokenize uh code so
18:43for example uh if you have code you know
18:45you have like often in Python these four
18:46spaces at the beginning those were dealt
18:49with uh kind of strangely before um and
18:52as a result like the model couldn't
18:54really understand uh how to deal with
18:56code uh so so toiz actually a lot um
19:00okay so I'll move on right now but we
19:03can come back later on token Isis great
19:06so we talked about the task the L the
19:07tokenizer let's talk a little bit about
19:10evaluation uh so the way that LMS are
19:12usually evaluated is what we call is
19:14using what we call perplexity um at a
19:17high level it's basically just your
19:18validation loss uh the slight difference
19:20with perplexity is that we use something
19:22that is slightly more interpretable
19:24which is that we use the average per
19:26token loss and then you expon entiate it
19:29and the reason why you exponentiate it
19:31is because you want I mean the loss has
19:33a log inside and you like one humans are
19:36actually pretty bad at thinking in log
19:37space but two logs depend on the base of
19:40the log uh while when you exponentiate
19:42you basically have everything in the uh
19:45kind of the vocabulary size uh unit um
19:48and the average proten is just so that
19:50your your complexity is independent of
19:52the length of your sequence um so
19:54perplexity is just two to the power uh
19:56average of the loss of the sequence
19:59um so perplexity is between one and the
20:03length of the vocabulary of your
20:04tokenizer uh one it's simply well if you
20:07predict perfectly the thing which uh
20:09every word then every word will have
20:12basically product of ones uh so the best
20:15perplexity you can have is one if you
20:17really have no idea you basically
20:18predict with one divided by uh size of
20:21vocabulary um and then you do simple
20:23math and you basically get perplexity of
20:25size of vocabulary uh so the intuition
20:27of perplexity is that basically the
20:29number of tokens that your model is kind
20:31of hesitating between uh so if you if
20:33your model is perfect it doesn't
20:34hesitate it know exactly the word if it
20:37really has no idea then it hesitates
20:39between uh all of the
20:42vocabulary uh so perplexity really
20:45improved that's perplexity on a standard
20:48data set between 2017 and 2023 it it
20:51went from kind of 70 tokens to less than
20:5410 tokens over these five six years so
20:56that means that the models were
20:58previously as dating between 70 words
21:00every time it was generating a word and
21:02now it's as dating between like less
21:04than 10 words so that's much better
21:06perplexity is actually not used anymore
21:08in academic benchmarking mostly because
21:10it depends on the tokenizers that you
21:12use uh it depends on the actual data
21:14that people are evaluating on but it's
21:16still very important for development of
21:18llms so when you when you actually train
21:20your own llm people will still really
21:22look at the
21:24perplexity uh one common other way and
21:28now more common in Academia of
21:30evaluating these llms is just by taking
21:33all the classical NLP benchmarks and
21:35I'll give you a few examples later and
21:37just kind of aggregating everything um
21:39so collect as many automatically
21:41evaluatable benchmarks and just evaluate
21:44across all of them um so one such if uh
21:48or actually two such uh benchmarks of
21:51what we call uh Helm which is from
21:53Stanford and another one is the hugging
21:55face open LM leader board which are the
21:57probably two two most common ones right
21:58now um so just to give you an idea in
22:02Helm there are all of these type of
22:03tasks which are mostly things that can
22:06be easily evaluated uh like question
22:09answering so think about many different
22:11question answering uh tasks um and the
22:14benefit with question answering is that
22:15you usually know what is the real answer
22:18um so you can the way that you evaluate
22:20these models and I'll give you a
22:21concrete example in one second um is
22:23that you can just look at How likely the
22:25language model is to generate the real
22:28answer compared to some other answers
22:30and that's essentially at a high level
22:32how you evaluate these models um so to
22:34give you a specific example mlu is
22:36probably the most common um academic
22:39Benchmark for
22:41llms uh and this is just a collection of
22:44many question and answers in all of
22:46those domains for example College
22:48medicine College physics astronomy and
22:51these type of topics and the questions
22:53are things like so this in astronomy
22:55what is true for type 1 a supernova then
22:58you give uh four different potential
23:01answers and you just ask the model which
23:03one is more likely so there are many
23:05different ways of doing it either you
23:07can look at the likelihood of generating
23:09all these answers uh or you can ask the
23:11model which one is the most likely uh so
23:13there are different ways that you can
23:14promp the model but at a high level you
23:16know which one is correct and there are
23:17three other mistakes um yes kind
23:22creating is like unconstrained text as
23:24the output yeah how do you evaluate a
23:26model if it give something that's you
23:29know semantically completely identical
23:32but is not the exact token list that
23:35expect yeah so that's a great question
23:37I'll talk more about that later here in
23:39this case we don't do unconstrained so
23:41the way you would evaluate MML is
23:43basically either you you ask the first
23:46question and then you look at the
23:47likelihood of the model generating a the
23:50likelihood of the model generating b c
23:53and d and you look at which one is the
23:54most likely or you can as the model out
23:57of ABC d which one is the most likely
23:59and you look at whe the to the most
24:01likely next token is A B C or D so uh
24:04you can strain the model to say it can
24:06only answer these four things you say
24:09you constraint the model you mean you
24:11constraint The Prompt or do you mean of
24:13its whole probability distribution
24:15outputs you only comparing the outputs
24:18like you're only comparing the
24:20a so uh in the second case I gave you
24:23you would do exactly the I actually you
24:24would do both you would prompt the model
24:26saying ABC or D plus you would constrain
24:28to only uh look at these two these four
24:31tokens in the first case you don't even
24:33need to generate anything so in the
24:34first case you literally just look given
24:36that it's a language model it can give a
24:38distribution over sentences you just
24:40look at what is the likelihood of
24:42generating all of these words what is
24:45the likelihood of generating the second
24:47choice and you just look at whether the
24:49most likely sentence is actually the
24:52real answer so you don't actually sample
24:55from it you really just use P of x one
24:58to excel does that make sense uh that
25:01being said evaluation of open-ended
25:04questions is something we're going to
25:05talk about later and is actually really
25:07important and really challenging yes
25:10earlier you mentioned that um like um
25:13metrics like flexity are not are not
25:15like usually used because it depends on
25:17like how you do your terization some
25:19design choices I was wondering if you
25:21could speak more to that oh um yeah so
25:25think about perplexity I told you
25:27perplexity is between one and vocabulary
25:29size so now imagine that Chad GPT uses a
25:32tokenizer that has like 10,000 tokens
25:35but Gemini from Google uses a tokenizer
25:37that had 100,000 uh potential tokens
25:41then actually the Gemini one will will
25:44have like the upper bound of the the
25:46perplexity that you can get is actually
25:47worse for Gemini than for Chad GPT does
25:51that make sense so that's just an idea
25:54it's actually a little bit more
25:55complicated than that but that's just
25:56like one uh first or the bit of you can
25:58see that the tokenizer actually
26:01matters um
26:04great okay so evaluation challenges
26:07there are many I'll just talk about two
26:09really briefly uh one as I told you
26:11there are two ways of doing evaluation
26:13for these mlu actually there are many
26:15more than two but I give you two
26:16examples um and it happens that for a
26:19long time even though that was a very
26:21classical Benchmark that everyone used
26:23uh actually different uh different
26:26companies and different um different uh
26:30uh different organization were actually
26:32using different ways of evaluating mlu
26:35and as a result you could you get
26:36completely different results for example
26:38Lama
26:3965b uh which was the first model of meta
26:42in the Lama series uh had on Helm 63.7
26:47accuracy but on this other um Benchmark
26:50had like
26:5148.8 um so really the way that you
26:54evaluate and this is not even talking
26:56about prompting this is really just kind
26:58of the the way that you evaluate the uh
27:00the models prompting is another issue so
27:02really there are a lot of
27:03inconsistencies it's not as easy as it
27:06looks uh first thing yeah sorry how can
27:09we make sure that all these models AR
27:10trained on The Benchmark okay second
27:13thing this is a great question uh chain
27:15test contamination uh this is something
27:18which I would say is really important in
27:22Academia in uh given that the talk is
27:25mostly about training large language
27:26models uh for companies it's maybe not
27:29that important CU they know what they
27:31trained on uh for us we have no idea so
27:35for us it's a real problem uh so there
27:37are many different ways of trying to
27:39test whether uh the test set sorry
27:43whether the test set was actually in the
27:44training Set uh one kind of cute trick
27:48um that people uh in in the lab on T lab
27:52have found is that what you can do is
27:54that given that most of the data set
27:56online are not randomized
27:58you can just look at and in that
28:00language models what they do is just
28:02predict the next word um you can just
28:04look at the entire test Set uh what if
28:07you generate all the examples in order
28:10versus all the examples in a different
28:13order and if it's more likely to
28:15generate a thing in order given that
28:17there's no real order there then it
28:19means that probably was in a training
28:20set does that make sense um so there are
28:23many that's like one of them there are
28:24many other ways of doing it train test
28:27contamination again not that important
28:28for development really important for
28:30academic
28:32benchmarking great so there are many
28:34other challenges but uh I'll move on for
28:36now great data um so data is another
28:41really big topic um at a high level
28:44people just say oh you basically train
28:46large language models on all of Internet
28:48what does that even mean um so or people
28:51sometimes say all of clean internet
28:53which is even less defined um so
28:56internet is very dirty and really not
28:58representative of what we want in
29:00practice if I download a random website
29:03right now you would be shocked at what
29:05is in there it's definitely not your
29:07Wikipedia um so I'll go really briefly
29:12on like what people do um I can answer
29:14some questions but I mean data is on its
29:17own is a huge topic uh basically first
29:20what you do is download all of Internet
29:22what that means is that you use uh web
29:24crowlers that will go on every web page
29:27on Internet or every web page that is um
29:30on Google uh and that is around 250
29:34billion pages right now um and that's
29:36around one petabyte of of data so this
29:39is actually a common common C is one web
29:42crowler so people will usually write
29:43their own web crowlers what they do is
29:45that they use standard web crowlers and
29:47we common crawl is one of them uh that
29:50basically every month adds all the new
29:52websites that were added on uh internet
29:55that are found by by Google and they put
29:57it in a big uh basically a big data set
30:00um so that's on common call you have
30:02around 250 billion pages right now so 1
30:05E6 gigabytes of data once you have this
30:09uh so this is a random web page like
30:11literally random uh from this common
30:13craw and what you see is that one it
30:15really doesn't look at type of things
30:17that you would usually see but actually
30:19so this is an HTML page uh it's hard to
30:22see but if you look through you will see
30:25some content for example here here uh
30:29tesing world is your ultimate source for
30:32the system X high performance server and
30:34then you have three dots so you don't
30:35even the sentence is not even finished
30:37that's how a random internet looks like
30:40uh so of course it's not that useful if
30:42you just train a like large language
30:44model to generate things like this so
30:46what are some of the steps that are
30:47needed first one you extract the text
30:50from the HTML so that's what I just try
30:52to do by looking at uh basically the
30:54correct text uh there are a lot of
30:56challenges by through this for example
30:58extracting math is actually very
31:00complicated but pretty important for
31:01training large language models um or for
31:04example boiler plates a lot of your
31:06forums will have the same type of
31:07headers the same type of Footers uh you
31:10don't want to repeat all of this in your
31:12data um then you will filter undesirable
31:15content uh so not safe for work harmful
31:19content pii uh so usually every company
31:21has basically a a black list of websites
31:25that they don't want to train the models
31:27on that Black List is very long and you
31:29basically say if it comes from there we
31:31don't train on this there are other ways
31:32of doing these things is that you can
31:34train a small model for classifying what
31:36is pii removing these things um it's
31:40hard every Point here that I'm going to
31:42show you is like a hard amount of work
31:46uh but I'm going to go go quickly
31:47through it so filter undesirable content
31:50second or fourth is the dup D
31:53duplication as I said um you might have
31:56things like headers and Footers in
31:58forums that are always the same you want
32:00to remove that another thing that you
32:01might have is a lot of URLs that are
32:04different but actually show the same
32:06website um and you might also have a lot
32:10of like U um paragraphs that come from
32:13like common books that are basically
32:14duplicated a thousand times or 10,000
32:17times on internet so you have to
32:19duplicate also very challenging uh
32:22because you have to do that at scale
32:24once you do duplication you will do some
32:26heuristic filtering you will try to
32:28remove low quality documents uh the way
32:31you do that are things like rules-based
32:33um filtering for example if you see that
32:36there are some outlier tokens if the
32:38distribution of tokens in the website is
32:39very different than the usual
32:40distribution of tokens then it's
32:42probably some outlier if you see that
32:44the length of the words in this website
32:46is super long there's something strange
32:48going on on that website if you see that
32:50the the website has only three words
32:52maybe is it worth training on it maybe
32:54not if it has like 10 million words
32:56maybe there's something also
32:58wrong going on that page um so a lot of
33:00rules like this yes why we filter out
33:03undesirable content from our dat set
33:05instead of kind
33:06of putting it in is like a supervised
33:09loss right like can we not just say like
33:12you know here's this like hate speech
33:13website let's actively try to Let's
33:17actively penalize the for generating
33:19we'll do exactly that but not at this
33:22step that's where the posttraining will
33:24come from uh pre-training um the idea is
33:28just to say I want to model kind of how
33:32humans speak essentially um and I want
33:35to remove all these like headers photos
33:36and and menus and things like this but
33:38it's a very good uh like idea that you
33:41just had and that's exactly what we'll
33:42do
33:44later Next Step modelbased filtering so
33:47once you filtered a lot of data what you
33:49will do uh that's actually a very cute
33:51trick uh you will take all of Wikipedia
33:53and you will look at all the links that
33:56are linked through Wikipedia p
33:58because probably if something is
33:59referenced by Wikipedia it's probably
34:01some high quality website and you will
34:03train a classifier to predict whether
34:06something comes from whether a document
34:09comes from one of these references uh
34:12from Wikipedia or whether it's from the
34:14random web and you will try to basically
34:16say I want more of the things that come
34:19from Wikipedia references does that make
34:22sense so yeah so you will train a a
34:25machine learning uh model usually also
34:27very simp simple models because you need
34:28to do that really at scale I mean just
34:30think about the 250 billion
34:32Pages uh next one you will try to
34:36classify your data into different
34:38different um domains you will say okay
34:41this is entertainment this is books this
34:43is code this is like these type of
34:45domains and then you will try to either
34:49um up or down weight some of the domains
34:52uh for example you might say uh you
34:54might see that actually if you train
34:56more on code then actually your model
34:58becomes bettered on reasoning so that's
34:59something that people usually say in a
35:01very handwavy way if you train your
35:03model more code actually it helps
35:04reasoning so you want to upweight the
35:07coding uh distribution because that
35:09helps for General language modeling
35:10skills uh books is usually also another
35:13one that people usually um upweight
35:16entertainment they usually downweight uh
35:18so things like this of course you want
35:20to do it so people used to do it maybe
35:23uh kind of theistically now there's
35:25entire pipelines that we'll talk about
35:28of how to do these things uh slightly
35:30more um
35:32automatically and then at the end of
35:34training uh usually train um after
35:38training on all of this data that we saw
35:40usually train on very high quality data
35:42at the end of of training your large
35:45language model where you decrease your
35:46learning rate uh and that basically
35:48means that you're kind of overfitting
35:50your model on a very high quality data
35:52so usually what you do there is like
35:54Wikipedia you basically overfit on
35:57Wikipedia yeah and you overfit on like
36:00human uh data that was collected um the
36:04other things like continual pre-training
36:06for getting longer context I'm I'm going
36:08to skip over all of these things uh but
36:10I just to give you a sense of how hard
36:12it is when people just say oh I'm going
36:13to train on internet that's a lot of
36:16work um and really we haven't figured it
36:18out yet so collecting World data is a
36:22huge part of practical large language
36:24model uh some might say it's actually
36:26the key yes
36:28about data so basic question so usually
36:30when you start with like the terabyte of
36:33data after I go through all that steps
36:35the typical amount of data you have in
36:37and then like how how large a team does
36:40it typically think to go through all the
36:42steps you talk about so how is the
36:44question how large is the data after you
36:46filter yeah after you filter and then to
36:48go through all the step how large a team
36:50do you need to go through like the the
36:52other fation sttion uh how slow is it or
36:56how like how how many people would you
36:58need to be able to do this uh okay
37:02that's a great question I'm going to
37:04somewhat answer about the data uh how
37:06large is the data set uh at the end of
37:08this slide uh for number of people that
37:12work on
37:13it um that's a good question I'm
37:15actually not quite sure but I would
37:18say yeah I actually don't quite no but I
37:22would say it's probably even bigger than
37:24the number of people that work on kind
37:26of the two tuning of the pre-training of
37:28the model uh so the data is bigger than
37:31kind of the modeling aspect um yeah I I
37:35don't think I have a good sense I would
37:38say probably in Lama's team which have
37:40like 70 years people I would say maybe
37:4215 work on data uh I yeah all these
37:47things you don't need that many people
37:48you need a lot of computer so because
37:49for data you need a lot of CPUs um so
37:53yeah and I'll answer the second question
37:54at the end of this slide so as I just
37:57kind of alluded to really we haven't
38:00solved data at all for pre-training so
38:02there's a lot of research that that has
38:03to be done first how do you process
38:05these things super efficiently uh second
38:07how do you balance kind of like all of
38:09these different domains uh can you do
38:11synthetic data generation that's
38:12actually a big one right now uh and
38:15because we don't have uh we'll talk
38:16about that later we don't have enough
38:18data on the internet um can you use
38:21multimodal data instead of just text
38:23data and how does that improve even your
38:25text performance um
38:28there's a lot of seccy because really
38:30this is the key of most of the pre-train
38:32pre-trained large language models so for
38:34competitive Dynamics uh usually these
38:37these um these companies don't talk
38:40about how they do the data collection
38:41and also there's a copyright liability
38:43issue they definitely don't want to tell
38:44you that they've trained on books even
38:46though they did um because if not you
38:48can uh sue them uh common academic
38:51benchmarks uh so that will kind of
38:53answer what you asked um it started so
38:55those are the smaller ones it's the
38:57names are not that important but it
38:59started from around 150 billion tokens
39:02which around uh 800 GB of data now it's
39:05around 15 trillion of to 15 trillion
39:07tokens which is also uh the size of the
39:10models that are right now the best
39:12models are probably trained on that
39:13amount of data so 15 trillion tokens uh
39:16which is probably I guess two order of
39:19manage bigger than that so 80 uh E3 gab
39:23so that would be
39:25around 100 to thousand times uh
39:28filtering of the common crawl if I'm not
39:31mistaken um so yeah one very one very uh
39:35famous one is the pile so this is
39:37academic Benchmark of the pile and we
39:39can just look at what distribution of
39:41data they have it's things like um
39:44archive PBM Central uh which is all the
39:47the biology stuff uh here it's Wikipedia
39:52you see stack exchange um some GitHub
39:56and some books and things like this um
39:58again this is on the smaller side so
40:00this is if we look at here this is on
40:01280b so in reality it's like 100 times
40:04bigger so you cannot have that much of
40:05GitHub and and of
40:07Wikipedia um in terms of close Source
40:10models just to give you an idea uh Lama
40:132 um it was trained on 20 two trillion
40:16tokens lamb 3 15 trillion tokens which
40:19is currently the best model that we know
40:21on how much it was trained on which is
40:22the same thing as this the the the best
40:25academic or the biggest academic
40:27Benchmark which is 15 trillion tokens
40:29GPD 4 we don't really know but it's
40:30probably in the same water of magnitude
40:32or it's probably around that actually
40:33it's probably around 13 um from leaks if
40:36the leaks are true
40:39um great so scaling laws um any other
40:43questions on Data before you go to
40:45scaling
40:48laws sorry I know I'm giving you a lot
40:50of information but uh there's a lot into
40:52training at large language models great
40:55scaling laws so so the idea is that what
40:58people saw um around 2020 or at least
41:01from a long time but they've been able
41:03to kind of theoretically show it or
41:06impurely show it since 2020 is that the
41:08more data you train your models on and
41:10the larger the models the better the
41:11performance this is actually pretty
41:13different than what you've seen in this
41:15class in this class we teach you about
41:16overfitting overfitting doesn't happen
41:18with large language models uh larger
41:21models better performance um it's
41:24something that really took a long time
41:25for the community who took this type of
41:27class to realize um but for the exam
41:31overfitting
41:32exists so okay the idea of scaling laws
41:36is that if given that you know that more
41:38data and larger models will always give
41:41you better performance can we predict
41:44how much better your performance will be
41:46if you increase the amount of data and
41:48the size of your model and surprisingly
41:51it works uh so here you see three plots
41:53from a very famous paper called scaling
41:55loss from openi um here you see on the
41:58x-axis compute so how much did you train
42:01like how much compute did you did you
42:02spend for training and here you see test
42:04loss so this is essentially I mean it's
42:07not perplexity but it's your validation
42:08loss um so it's a log of the perplexity
42:11and if you put these two on uh log scale
42:15uh then you see that uh the the
42:17performance or like the this the sorry
42:20the the scaling law is linear uh that
42:22means that if you increase your compute
42:25by a certain amount you can you can say
42:26by how much your test loss will actually
42:29decrease same thing with data and same
42:32thing for parameters if you increase the
42:34data set size your loss will will
42:36decrease by an amount that is somewhat
42:39predictable if you increase the number
42:40of parameters it will decre the loss
42:43will decrease by amount which is
42:44somewhat predictable this is really
42:46amazing um very surprising I mean it
42:49looks in nocuous when you look at these
42:51type of plots but that's crazy because
42:53it means that you can predict uh how
42:55well we're going to perform in 2 3 years
42:58depending on how much compute we will
42:59add assuming that these things will hold
43:01there's nothing theoretical about it um
43:04yes two things one what is the loss that
43:07they're using here is this perplexity or
43:09so it's it's you know I said perplexity
43:11was like two to the power of the LW so
43:13this is the the the power of the
43:16perplexity and then the second thing is
43:18when you like increase the number of
43:20parameters or you increase the total
43:21data set size going dat times doesn't
43:25that just inherently increase your
43:27compute like do all this work to
43:31just specific no this is a great
43:33question so the compute here is actually
43:35a factor of two things the data and the
43:37parameter what I'm showing here is that
43:39you can um well actually we're going to
43:40talk about that in details but basically
43:42if you increase the number of parameters
43:44you should increase the number of data
43:46that you have um so you actually don't
43:49go multiple times through the same data
43:50set no one does EPO in a lar at least
43:55not yet uh because we have still kind of
43:58enough data um so yeah this is all the
44:01same Trend which is increase compute
44:03decrease
44:04loss yes have we seen the numbers for
44:07the last two years or is it still
44:10holding it is still holding I I don't
44:14have like good numbers to show you uh
44:16but it is still holding
44:20surprisingly yes is there no evidence
44:22like empirical evidence that you
44:25plateau expected PL
44:28no empirical evidence of plateauing
44:30anytime soon um why we don't know um
44:36will it happen probably I mean it
44:38doesn't need to because it's actually in
44:39log scale so it's not like as if it had
44:43to go it had to Plateau like
44:45mathematically it could continue
44:46decreasing like this I mean most people
44:48think that it will probably Plateau at
44:49some point we don't know
44:51when um okay so that's I'll talk more
44:55about scaling laws now
44:57so why are scaling laws really cool
45:00imagine that I give you um you're very
45:02fortunate I gave you 10,000 gpus for
45:04this month what model will you train how
45:07do you even go about answering that
45:08question and I mean this is a a
45:11hypothetical but that's exactly what
45:13these companies are faced with uh the
45:16old pipeline um which was basically you
45:19tune High parameters on the big models
45:21so let's say I have 30 days I will train
45:2430 models for one day each I will pick
45:27the best one uh and that will be the
45:29final model that I will use in
45:31production um that means that the model
45:33that I actually used was only trained
45:34for one day the new pipeline is that you
45:38first find a scaling recipe so you find
45:40something that tells you for example oh
45:43like one common thing is that if you
45:44increase the size of your model you
45:46should decrease your learning rate so
45:47you find a scaling recipe such that you
45:49know if I increase the the the the size
45:52of my model here's what I should do with
45:53some high parameters then you tune your
45:56high parameter
45:57on smaller models of different sizes
46:00let's say I will say for 3 Days of my 30
46:03days I will train many different models
46:05and I would do highper parameter tuning
46:07on these small models each of different
46:08sizes then I will fit a scaling law and
46:11try to extrapolate from these smaller
46:14models which one will be the best if I
46:17if I train it for much longer or sorry
46:20if I train it for a larger model and
46:23then I will train the final huge model
46:24for 27 days instead of just one day
46:27um so the new pipeline is not train
46:31things or do high prity tuning on the
46:33real scale of the model that you're
46:34going to use in practice but do things
46:36on smaller ones at different scales try
46:39to predict how well they will perform
46:41once you make them bigger I will give I
46:43will give you a very concrete example
46:45right now uh let's say Transformers
46:48versus lstms let's say you you have
46:50these 10,000 gpus you will not sure
46:52which one you should be using should I
46:53be using Transformer based model or LCM
46:55based model what I will do is I will
46:57train Transformers at different skills
47:00so here you see different parameters on
47:01the x-axis Y axis is my test loss I will
47:04then train different different lstms at
47:07different scales once I have these
47:09points I will see oh it kind of fits a
47:11scaling law I will fit my scaling law
47:13and then I will be able to predict oh if
47:16I had 10 times more compute here's how
47:19well I would perform for the LM it's
47:21actually slightly less linear for the
47:22lstm but like you could probably try to
47:25predict where you would end up and
47:26clearly from this plot you would see
47:28that Transformers are better um one
47:31thing to notice when you read these type
47:32of scaling laws is that are two things
47:34that are important uh one is really your
47:38scaling rate uh which is kind of the uh
47:42the slope of the the slope of the
47:45scaling law the other thing is your um
47:48your intercept like you could start
47:51worse but actually become better over
47:53time it just happens that lstms are
47:55worse for both uh but I could show you
47:57another one where things you can predict
47:59that actually after a certain scale
48:01you're better off using that type of
48:03model than others uh so that's why
48:05scaling laws are actually really
48:07useful any questions on
48:11that yeah so these are all kind of very
48:15how how sensitive are these to like
48:17small differences in the architecture
48:18like one one like Transformer
48:21architecture versus another Transformer
48:23architecture you basically have to like
48:25fit your own curve and make basically
48:27say like oh scaling law has tell me
48:28there should be some like logarithmic
48:30function let me extrapolate that for my
48:34own yeah so uh usually for example if
48:37you're an academic and you want to now
48:39at least that's like pretty recent and
48:41you want to propose a new like
48:42activation uh that's exactly what you
48:44will do you will fit a scaling law show
48:46another scaling law with the standard
48:48like I don't know G and you will say
48:50that it's better in reality once you
48:51start thinking about it in scaling loss
48:53terms you really realize that actually
48:55all the architecture differences that we
48:57can make like the small minor ones all
48:59they do is maybe change a little bit the
49:01The
49:02Intercept but really that doesn't matter
49:05uh cuz just train it for 10 hours longer
49:07or like wait for the next uh for the
49:09next Compu gpus and these things are
49:11really secondary which is exactly why I
49:12was telling you originally people spend
49:14too much time on the architecture and
49:15losses um in reality these things don't
49:18matter as much data though if you use
49:20good data you will have much better
49:22scaling loss than if use bad data so
49:24that really matters
49:27uh another really cool thing you can do
49:28with scaling laws is that you can ask
49:30yourself uh how to optimally allocate
49:33training resources should I train larger
49:36models because we saw that it's better
49:38when you train larger models but we saw
49:40that it's also better when you use more
49:41data so which one should I do should I
49:44just train on more data a smaller model
49:45or should I train a larger model on less
49:47data um so chinchilla is a very famous
49:52paper that first showed this uh the way
49:54they did it I want to give you a little
49:55bit of a sense of what these plots are
49:58uh here you see training loss again on
50:00the x-axis you see parameter parameter
50:02differences uh sorry parameter size uh
50:04number of parameters so the size of the
50:05model and here all these curves are what
50:07we call isof flops which is that all the
50:11models on this curve H have been trained
50:14with the same amount of
50:16compute um the way that you do that is
50:18that you train you change sorry you vary
50:20the number of tokens that we trained on
50:22and the size of the models but you vary
50:24in such a way that the total compute is
50:26constant
50:27okay so all these curves that you see
50:28with different colors have different
50:30amount of computers that were trained on
50:32then you take the best one for each of
50:34those curves once you have the best one
50:36for each of those curves um you can ask
50:40you can plot um how much flops it was
50:44and which curve were you on and how much
50:46parameters did you actually use for
50:49training that specific point you put
50:51that on the on the log log uh scale
50:54again and now you fit a scaling law
50:56again so now I have something which
50:58tells me if I want to train a model of
51:0110^ 23 flops here's exactly the number
51:04of parameters that I should be using 100
51:07100b and you can do the same thing with
51:09flops and
51:10tokens so now you can predict if if I
51:14tell you exactly I have one month of
51:16compute what size of model should I be
51:18training F your scaling law and I tell
51:20you um of course that all looks
51:23beautiful in reality like there's like
51:25there's a lot of like small things of
51:26like should you be counting like
51:27embedding parameters like there's
51:29there's a lot of complexities but if you
51:31do things well these things actually do
51:34hold um so the optimal number of
51:37parameters that that chinchilla Pap have
51:39found is to use 20 tokens for every
51:42parameter that you train uh so if you
51:44add one more parameter you should add
51:45you should train your thing on your
51:47model on 20 more tokens so one caveat
51:50here is that this is optimal training
51:52resources so that is telling me if you
51:54have 10^ 23 FL
51:57or if you have like 100 I don't know how
51:58much that is100 million or 10 no that's
52:02much less actually let's say I have $5
52:03million to to train my best model that
52:06gets the lowest loss how how what would
52:08I train on in reality these companies
52:11need to think about inference also if
52:13you have a smaller model they will spend
52:16less over time um so actually if you
52:18consider the inference cost you have
52:20other papers that Tred to show that um
52:22it's around
52:24150 uh parameters per sorry tokens per
52:27parameters because you prefer having a
52:29smaller model cuz over time you're going
52:32to you're going to actually um spend
52:35less money on inference of these models
52:37so 150 to one that's around what the
52:40best models are trained on right now at
52:43least the ones that are that are used um
52:46in practice for in
52:49production
52:51great any question on
52:55chin great oh sorry in practice how
52:58expensive is inference for these models
53:01rela to
53:02train actually very expensive uh I will
53:05not talk about inference because that
53:06would be another entire lecture but just
53:09think about Chad GPT where they have I
53:12don't know how much it is now like 600
53:14million people that used it um like
53:19that's a lot
53:21um yeah so it's actually very expensive
53:24there's a lot of optimization you can do
53:26for in though um and that's an entire
53:28other lecture so I'm going to skip that
53:30uh this time but it's very
53:32interesting okay tuning um as I said
53:35there are many things that you can uh
53:37answer with scaling laws I just try to
53:39give you two examples uh but really
53:41there are many things what data do you
53:43use what mixture what data mixing
53:45waiting you use data mixtures that's
53:47what we talked about before uh what
53:49architecture you use whether you should
53:51make your models uh wider or deeper um
53:54should you be paying for more gpus or
53:56actually collecting more data um all
53:59these things are things you can try to
54:00answer with scaling
54:02laws one thing I want to say is the bit
54:05lesson if you ever heard of Richard
54:07sudden a very famous blog post in 2019
54:10um what he realized uh which I think not
54:15enough people realize I didn't
54:17definitely did not realize at that time
54:19um is that once you see these type of
54:21scaling laws you know that the more
54:23compute you have the better models you
54:25will get so with skill you will get
54:27better model and you also know by Mo law
54:30or these type of variant of Mo law that
54:32you will always have better compute then
54:34the only thing that matters is just to
54:37have architectures that can leverage
54:39computation so what matters is basically
54:41systems data and less so the
54:44architecture like the small architecture
54:46differences like your your your
54:47activation and things like this uh so I
54:50think that's like one of the reasons why
54:51most of research focuses on um some
54:54things that for industry matters less
54:56and I was one of those researchers for a
54:59large part of my my career um so don't
55:02spend time over complicating do the
55:05simple things do it well seal them
55:08that's really what openi taught us with
55:10um with chat gpg and with all the gpts
55:14before okay I want to give you some
55:16backup the envelope computation so I
55:19might be off by a few factors here but I
55:20just want to give you a sense of how
55:22costly it is to train some of these
55:24models I'll give as an example
55:26Lama 3 400b which is currently the best
55:29open source model that you can get uh it
55:32was trained on 15.6 tokens it has 45
55:36billion parameters so just now that you
55:38know what is like this uh optimal tokens
55:41per parameter that's around 40 so that's
55:43a little bit more than chinchilla but
55:45less than this like inference uh optimal
55:49um model so they went for training
55:52optimality uh flops for this model so
55:55one simple uh way to compute flops is
55:57six uh times the number of parameters
56:00times the number of data you train on uh
56:03so if you do the simple calculation here
56:04it's 3.8 e25 flops the reason why this
56:08is important is that if you follow the
56:10little bit the news there's an executive
56:11order from Biden that basically says
56:13that once you have uh 1 e26 parameters
56:17uh sorry flops uh then you have special
56:20scrutiny on your models so they went 2x
56:22less than that so they really went right
56:24below this to not have special scrutiny
56:27so 38 uh I might be off by a little bit
56:29but it's definitely under the 1
56:3426 oh um so paramet p is parameters n is
56:40data number of tokens this is a uh this
56:43is just an
56:44approximation we
56:47yeah okay uh compute and we know that
56:51they trained on 16,000
56:53h100s um and we know the throughput but
56:56they they said it too uh so if you do
56:59the computation it takes around 70 days
57:02um or 26 million GPU hours at least
57:05that's with my uh back of the envelope
57:07computation they actually said that they
57:09use 30 million instead of 26 million GPU
57:12hours um so maybe they had like some uh
57:16some challenges I don't really know but
57:18if you follow the simple computation
57:20it's around 70 days um cost uh I mean
57:24this it's hard to to approximate but I'm
57:27just going to say it's kind of the rent
57:29like what if I were to rent h100s that
57:32many h100s for that many days how much
57:35will I pay uh h100 a lower bound on the
57:38on the renting uh cost of h100 is around
57:412 hours uh $2 per hour so if you
57:44multiply this by 26 million uh hours uh
57:48you get 52 million uh dollars so they
57:51probably pay less than that but not
57:53actually much less because all these um
57:57all these services that actually rent
57:58gpus they don't make that much money so
58:00it's it's probably slightly less but not
58:02that much less um now salary I said 50
58:06employees 500k per
58:09year say yeah it's probably the right
58:11ballpark 25 million uh so if you put all
58:14together around 75 million um dollars
58:17for
58:18training uh this Slammer model I'm
58:21probably off by like 10 million but but
58:23that's kind of right uh bpk
58:27carbon emitted um a lot of people might
58:30ask like also the cost is not the only
58:32thing that is important so I did the
58:34computation um it's around 4 uh 4,000 um
58:40tons of CO2 equivalent that is actually
58:43only 2,000 return tickets from JFK to uh
58:46London so right now uh carbon emitted is
58:50actually not uh I mean it's huge but
58:52it's not like um meaningful yeah yet I
58:56think in maybe GPT 6 gpt7 once you
59:01multiply this by 100 that might become a
59:03real issue right now it's still not uh I
59:06think um an issue in the grand scheme of
59:08things next model the way you should be
59:11thinking about these models is that
59:12every new generation the number of flops
59:15essentially uh multiplies 10x or at
59:17least that's what they try uh if they
59:19have enough energy and if they can buy
59:21enough
59:22gpus uh great any question on these back
59:25of the envelope math
59:29no
59:31okay so now we talked about pre-training
59:34I wanted to also chat about systems
59:36because now we know computer is really
59:38important so there's a question of how
59:39do you optimize the how do you optimize
59:42your computer I will leave that for the
59:44end because I'm not sure how much time
59:45we will have I think it's important but
59:47hopefully I I'll be able to to talk
59:49about it later it's slightly different
59:51than what we've been talking about right
59:53now so I'll move on to post training for
59:55now
59:56so the task of post training ER the
59:59reason why we need to do Post training
1:00:01is as I told you before um it's to make
1:00:04AI assistants so language modeling is
1:00:07not uh really the thing that you want
1:00:10when you have an AI assistant uh for
1:00:12example if you ask to gbd3 which is a
1:00:15purely language Model A pure language
1:00:17model not a um not an aligned one if you
1:00:20ask a question like explain the moon
1:00:22landing to a
1:00:23six-year-old the completion that you
1:00:25would get is something like explain the
1:00:27theory of gravity to a six-year-old
1:00:29because what it learned is that on on on
1:00:31internet if you have one question you
1:00:33usually have maybe another bullet point
1:00:35of other similar questions you don't
1:00:37usually have question and then answer
1:00:38later uh this is not what you want from
1:00:41an AI assistant so how do we uh do this
1:00:45alignment which is this post training
1:00:46and making these models
1:00:48assistance um so the goal of this
1:00:51alignment is to basically get LMS follow
1:00:54the instructions that are given um by
1:00:56users and and maybe some designers kind
1:01:00of desires um so think about moderation
1:01:03you don't want the model like open ey
1:01:05definitely doesn't want the model to say
1:01:07stuff that is very
1:01:08toxic um so here you see on the left
1:01:11hand side uh that when you ask a
1:01:13question it actually provides a a real
1:01:15answer so it's not like uh before the
1:01:17llm and on the right hand side you see
1:01:19that it would if you ask to write a
1:01:21tweet describing how a certain part of
1:01:25the population are evil it will say that
1:01:27it cannot do that um so that's kind of
1:01:31this
1:01:31alignment uh the background here is that
1:01:35uh basically the data that you want for
1:01:38training some of these models um is like
1:01:41we know what we want which is just
1:01:43asking humans this is a question this is
1:01:44the answer that you want uh but the
1:01:46thing is that it's very expensive to
1:01:48collect that data and it's hard to find
1:01:49it online uh in contrast pre-training
1:01:52data is not what you want but there's a
1:01:55lot of it um so what what we will do a
1:01:57the main idea is simply take a pre-train
1:02:00large language model pre-train all of
1:02:02internet and then you just fine tune so
1:02:03you just change a little bit of weights
1:02:05on the type of data that you actually
1:02:06want and hopefully given it you already
1:02:08pre-train it on all of Internet it
1:02:10basically learns or knows how to speak
1:02:12in English and and knows a standard um
1:02:16language syntax uh then you can really
1:02:19find tune in with very little
1:02:22data okay sft so supervis fine tuning is
1:02:26really exactly what I just said which is
1:02:27the idea of fine-tuning the large
1:02:29language model on uh basically the
1:02:32desired answers that are collected from
1:02:34humans um so why is it called supervis
1:02:37fine tuning because you basically want
1:02:38to do language modeling on the real
1:02:41ansers so language modeling is this like
1:02:42next word prediction and and that's the
1:02:44fine-tuning part and then you want to do
1:02:46it on desired answers given by humans so
1:02:48that's why we call it
1:02:50supervis so how do we collect this data
1:02:52well we I just said it you just ask
1:02:54humans uh to to tell you this is the
1:02:56this is a question this is the answer
1:02:58that you uh you would want from some of
1:03:00these models so this is an example um
1:03:03sorry I can't read very well on my
1:03:04computer but uh my kid uh needs to do a
1:03:07science um no let's read this one can
1:03:09you write a short introduction about the
1:03:12relevance of the term monopsony and then
1:03:14it says monopsony refers to a market
1:03:15structure blah blah blah and that's a
1:03:16human that wrote that um so actually
1:03:19this is open Assistant which was a a way
1:03:22to collect um uh data online by
1:03:27humans so this type of supervised fine
1:03:30tuning or alignment is really the key of
1:03:32Chad GPT this is what made uh the big
1:03:35jump from gpt3 which was mostly
1:03:37something that was known by AI
1:03:39researchers to Chad GPT which became
1:03:42known by basically
1:03:44everyone
1:03:46um so the problem with uh human data is
1:03:51that it's uh very slow to collect and
1:03:54very expensive um so
1:03:57one possible simple idea is to use llms
1:04:01to scale data collection uh so that's
1:04:03exactly what we did with alpaca uh one
1:04:06year ago what we did is that we asked uh
1:04:08humans or we use a data set of human uh
1:04:11question answers so there were 175 uh
1:04:14question answers here and we asked the
1:04:15best mod at the time so text3 to
1:04:18basically generate many more of these
1:04:21question and answers so all we did is
1:04:23like this is what humans would write now
1:04:25write similar answers and similar
1:04:26questions and we collected 52,000 LM
1:04:30generated question answers and then what
1:04:32we did is simply we took Lama 7B which
1:04:34was the best pre-train model at the time
1:04:36and we just fine- tuned this with
1:04:38supervised fine tuning as I told you and
1:04:40that's how we got um the Alpac s7b
1:04:43model uh and this is the type of data
1:04:46that we collected so things like what
1:04:48does algorithm mean an algorithm is a
1:04:50step by a stepbystep uh set of
1:04:52instruction used to solve a problem or
1:04:54achieve a goal blah blah blah blah so
1:04:56the data is not actually it's actually
1:04:58pretty good given it was LM generated by
1:05:00LMS from essentially two generations ago
1:05:04um so that really started at least for
1:05:07us kind of as an academic replication of
1:05:09chat GPT uh now it really there's a big
1:05:12field of like synthetic data generation
1:05:14of how to use llms to basically make
1:05:18development of llms faster um and by
1:05:21basically by decreasing the amount of of
1:05:23human hours that you need
1:05:27quantity of data so we talked about what
1:05:29type of data and how we collect it um
1:05:31one thing which is surprising with sft
1:05:33is that you don't need that much data uh
1:05:36so what this paper showed this is called
1:05:38Lima is that if you have if you scale
1:05:41the amount of data that use from uh
1:05:43supervised fine training from 2,000 to
1:05:4532,000 it really doesn't help much so
1:05:48here scaling laws definitely don't help
1:05:50um so the the intuition here is that all
1:05:53you learn um is is you learn how to
1:05:56format your desired answers another way
1:05:59of saying it is that your pre-trained
1:06:01models they essentially model the
1:06:03distribution of every user on internet
1:06:05one that might write bullet points
1:06:07another one that might answer qu answer
1:06:09question with an answer so all you tell
1:06:11your model is like wait you should
1:06:13actually be optimizing more for this
1:06:15type of user than another one so you're
1:06:17not actually teaching it and you're not
1:06:19teaching anything through this um sft uh
1:06:23so supervis fine tuning all you do is
1:06:25you tell the model to kind of optimize
1:06:27for one type of user that it saw already
1:06:29in a pre-train data set so the knowledge
1:06:31is already in the pre-train llm uh and
1:06:33you basically just specialize to one
1:06:35type of
1:06:36user great any question on
1:06:40sft yes so I know it's a big issue with
1:06:44synthetic data where uh if you keep
1:06:47generating data from the same
1:06:49distribution eventually you're not
1:06:50learning a new distribution you're
1:06:51essentially playing with it it just
1:06:52bootstrapping that yeah surely
1:06:56you can't scale that forever right you
1:06:57can't keep going on and generating from
1:06:59the same distribution you hope to learn
1:07:01something new yeah uh so are there it's
1:07:03an active area of research but any
1:07:05thoughts that you have around how people
1:07:07are maybe thinking around this and uh
1:07:10better ways to bootstrap or to give up
1:07:12on this idea and and realize that the
1:07:14chart shows you don't need that many so
1:07:16just get humans to generate 2,000 really
1:07:18good uh yeah so that's a very good
1:07:20question uh so for the data stuff so I'm
1:07:23saying it's not that important for sft
1:07:24but there will be another thing we'll
1:07:25talk about right after where actually
1:07:28data does
1:07:29matter my intuition based on not that
1:07:32much empirical results is that you can
1:07:34still get um even though you use your
1:07:37LMS if you use purely LM generated text
1:07:40and you do that for like three four
1:07:42generations of llms I agree with you
1:07:43that probably you won't improve much but
1:07:46for me what is important is how do you
1:07:47use like human in the loop with llms not
1:07:50purely LMS not purely uh humans but
1:07:53maybe what you can do is just have the
1:07:54model generate some new text and just uh
1:07:57humans write a few Edits edits are much
1:07:59faster than writing the entire text and
1:08:02I think that if you have that type of
1:08:03collaboration then from like kind of an
1:08:05information theoretical point of view
1:08:07you still get additional information but
1:08:09you still much faster than if you use
1:08:11humans and I think that as a field we'll
1:08:13probably move towards these type of
1:08:14things uh which is um really just
1:08:17finding the examples that are important
1:08:19and and asking humans it's kind of
1:08:21active learning just asking humans
1:08:22exactly when uh you need to to get
1:08:27inputs yes do we train with like the
1:08:29same loss function the same like General
1:08:32training algorithm for the supervis
1:08:33tuning bit as we do for the for the
1:08:35pre-training right because like the
1:08:37examples you showed I think the the
1:08:39important thing of the good examples is
1:08:43they're like supera accurate there's
1:08:45these more complex still just like chain
1:08:48same so that's why here I yeah I didn't
1:08:51maybe didn't emphasize enough this is
1:08:52just language modeling fine tun the LM
1:08:54with language model on the desired
1:08:56answers so this is literally the same
1:08:57loss um it will be different in two
1:09:01seconds but the first step of sft is
1:09:03literally the same loss where you just
1:09:05say Okay I want to actually specialize
1:09:07on that type of data so there's even a
1:09:09question of like what is pre-training
1:09:10what is post-training because in reality
1:09:12it's just like a different data that you
1:09:13use the reason why we usually call it
1:09:15post training is that the way we collect
1:09:16that data is very
1:09:18different great great questions uh yes
1:09:22maybe it's the same question but why
1:09:23would these 2,000 examples have such an
1:09:26overweighted
1:09:28influence you tun so that's why we uh
1:09:31also that's another reason why we call
1:09:33it post training is that we use
1:09:34different type of hyper parameters so
1:09:35you know I told you basically at the end
1:09:37of pre training you essentially end up
1:09:38with a learning rate of zero and here
1:09:40you're going to increase your learning
1:09:41rate so like 1 eus 5 one E Yeah and and
1:09:44so um the weight that you give to them
1:09:47is actually
1:09:49different
1:09:51um okay uh Second Step or second part of
1:09:56this post training um is what we call
1:09:59reinforcement learning from Human
1:10:00feedback or rhf uh some of you might
1:10:03have heard of that um the idea is that
1:10:06sft has a problem namely that uh you do
1:10:09behavioral cloning which means that you
1:10:11just try to clone what the humans would
1:10:14say and that had that has many issues
1:10:16one of them is that you're bound by
1:10:18human abilities so if um like humans
1:10:23actually humans won't generate the
1:10:26things that they think is actually the
1:10:27best thing to generate so if you ask me
1:10:29to write a book I mean I can definitely
1:10:31enjoy a book I can probably say one book
1:10:33is better than another but I'm
1:10:34definitely not going to be as good as
1:10:36writing the book that I want to read uh
1:10:38so you're going to be bound by the human
1:10:39ability to generate things even though
1:10:41the humans might be better at
1:10:42distinguishing between things that's one
1:10:44issue issue number two uh I find that
1:10:46actually pretty interesting is that it
1:10:48might if you ever heard of the word
1:10:50hallucination so this is llms generating
1:10:53F like false information
1:10:56hallucination might these people have um
1:10:58hypothesized that that can come from the
1:11:00supervised fine tuning even if you do
1:11:02supervised fine tuning on data that is
1:11:05correct and the reason why that is is
1:11:08that if uh given I told you that
1:11:10basically sftt is with very little data
1:11:13and it's with data that doesn't the
1:11:15model doesn't learn anything new so what
1:11:17if the human gives an answer that the
1:11:20model didn't know was true from the
1:11:23model perspective you the human
1:11:25basically is telling the the model uh
1:11:28generate this thing that seems plausible
1:11:31but actually have no idea if it's true
1:11:32or not um so just to give you a very
1:11:35concrete example if we go back to this
1:11:37uh monopsony example can you write blah
1:11:39blah blah about monopsony uh imagine
1:11:42that a human uh wrote a reference on
1:11:44this type of book um and that book might
1:11:47exist that might be a correct reference
1:11:49but what if the llm never saw this
1:11:51reference during pre-training then it
1:11:53doesn't know that it's a correct
1:11:54reference so really what you tell the
1:11:55model is to generate or make up some
1:11:58plausibly sounding reference um rather
1:12:01than actually tell the real reference
1:12:03that it saw during pre-training uh so
1:12:06hallucination might be um uh a re like
1:12:10might be caused by this sft that's
1:12:12problem number two does that all make
1:12:14sense great problem number three price
1:12:18generating the ideal answers is very
1:12:21pricey and that comes back to your
1:12:22question um of like humans writing
1:12:25answer is actually pretty
1:12:27expensive um so that's where rhf comes
1:12:29in the idea is that instead of cloning
1:12:32the behaviors of humans we're going to
1:12:34maximize human preference um and the way
1:12:37we're going to do that so the pipeline
1:12:39is that for a certain for every
1:12:41instruction you're going to ask a model
1:12:43to generate two answers um and usually
1:12:47use a pretty good model so you usually
1:12:48don't use an LM here you use a sft uh
1:12:52fine tune you use a fine tuned llm
1:12:54already to give like pretty good answers
1:12:57and then you ask labelers which of these
1:12:59two answers was better so select the
1:13:01preferred one and then with different
1:13:04type of algorithms we're going to talk
1:13:05about the algorithms um you just
1:13:07fine-tune the model to generate more of
1:13:09the green thing than the red thing so
1:13:10more of the good stuff uh so now the
1:13:13question is how and we're going to talk
1:13:14about that right
1:13:16now so there are two ways that we're
1:13:19going to talk about and two that are
1:13:20mainly used in the community um the
1:13:23first one is simply the idea of of using
1:13:25reinforcement learning so hopefully you
1:13:26all know what reinforcement learning is
1:13:28now um so when you think about using
1:13:32reinforcement learning one important
1:13:33question is like what is the reward that
1:13:35we're optimizing uh so in this case
1:13:37there are really two options that I
1:13:38could think about the first one you
1:13:40could just say I'm going to compare the
1:13:42output generated by some baseline the
1:13:44output generated by my model U and I'm
1:13:46just going to ask the human to say which
1:13:48one is better and I'm going to use this
1:13:51as a reward so if I'm better than the
1:13:52Baseline this is a plus one if not it's
1:13:54a minus one one uh so now it's binary
1:13:56reward the problem with binary reward is
1:13:58that it's very sparse and you don't get
1:14:00much information out of it uh like maybe
1:14:02your answer was slightly better maybe it
1:14:04was like way better and you don't really
1:14:07know from this um how much better it was
1:14:11so option two is that you can train what
1:14:13we call a reward model which is simply a
1:14:15classifier uh so you use machine
1:14:17learning to to classify how much better
1:14:21uh two outputs are from the preference
1:14:24from the perspective of the human um so
1:14:27this is a little bit meta but what you
1:14:29basically do is that you train uh you
1:14:31take um a reward model R which is a uh
1:14:34just a large also a large um a large
1:14:38classifier and you basically ask this
1:14:40reward model you give it the input and
1:14:42the actual output that you have one of
1:14:44the two outputs uh and you just um
1:14:47exponentiate that so that's the soft Max
1:14:49law that you all know about and now you
1:14:51divide by um the the exponential
1:14:55reward uh on the first example sorry on
1:14:59the first output and this is on the
1:15:00second output and you basically train so
1:15:02the reason why you do that is that you
1:15:04train your your model you train this
1:15:06reward model to be able to classify um
1:15:10how much better one output is to another
1:15:13one so another uh slightly less
1:15:15convoluted way of saying it is that your
1:15:16reward model will output some reward
1:15:19that will be used as the logits of your
1:15:21soft Max so now if you have high logic
1:15:25in your softmax it means that you highly
1:15:27likely this um output is
1:15:31better uh so that's what we call Bradley
1:15:33ter model yes is this reward model going
1:15:36over the entire output or is it
1:15:39going um so this takes the
1:15:43entire uh yeah this takes the entire
1:15:46output at once so it takes all the input
1:15:47and all the output and it gives one
1:15:49number
1:15:52yes would human be sorry with the reward
1:15:56model where would a human be like oh I
1:15:59see okay sorry maybe I wasn't clear um
1:16:03you train this reward model to fit this
1:16:06green and and red preference from humans
1:16:09so basically you train a classifier to
1:16:12say whether the humans prefer red or
1:16:14green uh but instead of using the binary
1:16:17reward which is what the human would
1:16:19tell you you basically use the logits of
1:16:22the soft Max and the thing with the
1:16:23logits is that that logits are
1:16:25continuous so now you know that if your
1:16:27reward model said it has high logits
1:16:30then in some ways the human highly
1:16:32prefer this answer to some other
1:16:36answer great um so as I just said
1:16:39continuous information so it's better so
1:16:41that's what people uh use in practice or
1:16:43at least used to use in practice I'll
1:16:45tell you about uh the other algorithm
1:16:47later uh so what you do at the end is
1:16:49that you basically try to just use
1:16:51reinforcement learning that you know
1:16:53about now we know we have reward what
1:16:55you sample through is the generation
1:16:58from your large language model um and
1:17:00then you just use some regularization
1:17:01term so the reason why you do this
1:17:03regularization term is for avoiding what
1:17:05we call over optimization so this reward
1:17:07model might not be really represent like
1:17:09might not perfectly model human
1:17:11preferences so you don't want to
1:17:12maximize this thing to essentially
1:17:15Infinity um and you do it using uh po
1:17:19which is a common uh reinforcement
1:17:22learning algorithm um one thing to note
1:17:25here because it will be important for
1:17:26later is that when we use maximum
1:17:29likelihood
1:17:31um sorry now the large language models
1:17:35are actually a policy for your
1:17:37reinforcement learning it's not
1:17:39maximizing maximum likelihood anymore
1:17:41which means that you're not modeling any
1:17:42distribution anymore and the reason why
1:17:44this is important is that models that
1:17:46went through this type of Po actually
1:17:49don't give you likelihoods of text that
1:17:51are meaningful cuz what you optimize
1:17:53them to do is B basically just optimized
1:17:55for generating the most likely thing not
1:17:58optimize for modeling like all the
1:18:00answers that humans might say another
1:18:02way of saying that is that there's
1:18:04nothing that incentivizes here the model
1:18:06to not give a like a um a single
1:18:10possible generation nothing here says
1:18:13it's good if you have some distribution
1:18:15with some
1:18:16entropy um okay if you haven't followed
1:18:18it's not that important but just good to
1:18:21knowe great so PO is exact what chat GPT
1:18:26did originally so here's the on the blog
1:18:28post or what they have is step one do
1:18:32supervise fine training which now you
1:18:33all know about step two train a reward
1:18:36model on human preferences step three do
1:18:39po multiple steps which is where you see
1:18:41this this blue arrow so you continue you
1:18:43train the model once with po you collect
1:18:45new data you continue uh and that's why
1:18:48and that's exactly what Chad GPT did uh
1:18:50that was a big breakthrough between gpt3
1:18:53and Chad GPT
1:18:55one thing to note is that uh P has many
1:18:58challenges reinforcement learning is
1:19:00something that's super nice
1:19:01theoretically in practice anyone who
1:19:03ever worked with reinforcement learning
1:19:04knows it's such a mess uh there's a lot
1:19:07of things like roll outs out of Loops
1:19:08clipping so many complications um so
1:19:12it's messy this is the idealized PO used
1:19:14for LM settings so that's already much
1:19:16more complicated than this expectation
1:19:18we saw before and in practice it's
1:19:19actually much more complicated so we
1:19:21have one implementation of it that we
1:19:22had to do and I'm not going to go
1:19:24through it but basically you have like
1:19:26so much stuff that you have to think
1:19:27about when you implement that type of of
1:19:30uh po algorithm so you have clipping
1:19:32everywhere you have a lot of
1:19:34complexities and things are not well
1:19:36documented all this to say um that we're
1:19:40going to there was a new method that was
1:19:41proposed uh also from Sanford one year
1:19:44ago called DPO which is essentially a
1:19:46simplification of Po um and the way uh
1:19:51what they did or the idea that they have
1:19:53is that instead of using reinforcement
1:19:55learning you can just maximize the
1:19:57probability of generating the stuff that
1:19:58you like and minimizing the probability
1:20:00of the stuff that you don't like uh so
1:20:02if you think about the human preference
1:20:04the red and green maximize uh green
1:20:07minimize red um so the loss is actually
1:20:10this one uh where what you see this is
1:20:12simply um some log of the model so this
1:20:17is the likelihood of a model generating
1:20:18the things that the human preferred
1:20:20given the the inputs um and what you try
1:20:24to do is basically
1:20:26maximize uh the likelihood of generating
1:20:29the things that you like minimize the
1:20:31likelihood of the things that you don't
1:20:32like um all the rest of the terms here
1:20:35it's not too important it's actually
1:20:37really not that complicated to
1:20:39understand but at a high level it's
1:20:41really just maximizing the things you
1:20:42like minimizing the the rest um and one
1:20:46thing to note uh which I was going to
1:20:48say just here is that actually all the
1:20:50rest is chosen such that um the global
1:20:53Minima of of Po and a global Minima of
1:20:57like this DPO under some assumptions are
1:20:59essentially equivalent so this is the
1:21:02right thing to do mathematically I'm not
1:21:04going to go through the derivations but
1:21:06that's the right thing to do uh it's
1:21:08pretty different with Po in the sense
1:21:09that now and with P what you had to do
1:21:11is collect the human preferences then
1:21:13train a uh reward model with maximum
1:21:15likelihood then use reinforcement
1:21:17learning now all you do is basically
1:21:18maximum likelihood much simpler yes I
1:21:21mean yeah so it seems like this is a
1:21:23much simpler and B like what you just
1:21:25intuitively do if this why did they
1:21:28start with this reward model like what
1:21:30what led them doing that I think it's a
1:21:32great question uh I don't really know
1:21:34what I can tell you is that at open ey
1:21:37the people who did the um uh who did
1:21:40basically this PP uh sorry who did Chad
1:21:43GPT initially are the ones who actually
1:21:46wrote Po and I think they were just like
1:21:48there are a lot of reinforcement
1:21:49learning people and I think that for
1:21:51them it was very intuitive um so there's
1:21:55also some additional like potential
1:21:57benefits for example I don't want to
1:22:01yeah for example if you use the reward
1:22:02model uh the cool thing here with
1:22:04reinforcement learning is that you can
1:22:05use unlabeled data with the reward model
1:22:08so here you can only use the label data
1:22:10for doing DPO um for PP for po you first
1:22:14train your reward model and then you can
1:22:16use unlabeled data uh where the reward
1:22:18model will basically label this
1:22:20unlabeled data so there there's
1:22:21additional kind of potential uh
1:22:25there could be potential improvements in
1:22:27practice it happens at down and on and I
1:22:29think just that a lot of people in this
1:22:32team were reinforcement learning experts
1:22:34including uh the main author of Po John
1:22:37hman um so much simpler in poo and is
1:22:41basically performs as well uh so now
1:22:43this is the standard uh thing that
1:22:45people use at least in the open source
1:22:47Community I believe it's actually the
1:22:48standard also in in Industry so that's
1:22:52called DPO gains
1:22:55um so those are all the papers on the
1:22:57left here this is on a summarization
1:22:59task you see all I want to show you is
1:23:01that basically the pre-train models uh
1:23:03were okay and they improve with scale if
1:23:05you do supervised fine tuning you
1:23:07improve them a little bit more if you do
1:23:09po or something with all HF with human
1:23:11feedback you get performance that are as
1:23:14often times depending on a benchmark
1:23:16even better than uh humans so this is
1:23:19the human uh reference summaries same
1:23:21thing this is on a uh on a paper that we
1:23:23have Alpaca Farm
1:23:25where we see uh the evaluation here is
1:23:27not too important but basically you see
1:23:28pre-train model you jump to sft and then
1:23:31you jump to PPO and popo have the exact
1:23:34same
1:23:35performance so basically all HF helps
1:23:38that's kind of the conclusion and DPO is
1:23:41simple uh data uh the way that you
1:23:44collect that type of data um first idea
1:23:47is just use humans as we already talked
1:23:50about uh guidelines are very complicated
1:23:52for what humans should be labeling and
1:23:54and it's really not that easy and
1:23:55actually if you ever do some of the
1:23:57labeling you will see that it's
1:24:00extremely complicated like if I zoom in
1:24:02to this uh here I have a question tell
1:24:05tell me about self-driving cars and you
1:24:07read both self-driving cars are vehicles
1:24:09that are capable of detecting their
1:24:10surroundings blah blah blah self-driving
1:24:12cars are cars that are equipped with
1:24:13sensors blah blah blah to navigate
1:24:15without the need for a driver I mean
1:24:17both seem okay like which one is better
1:24:19it's actually hard to say at a glance um
1:24:22and as a result uh the problem with
1:24:23humans is that you will start optimizing
1:24:27a lot of like high level features for
1:24:28example the second one is longer I can
1:24:30guarantee you that most humans will
1:24:31choose second one even though I mean
1:24:34maybe the first one is better I don't
1:24:35know I haven't read it carefully so
1:24:38challenges with humans first slow and
1:24:41expensive uh second as I just mentioned
1:24:44it's hard to focus on things that matter
1:24:46like correctness and people uh usually
1:24:49look at things that don't matter as much
1:24:51like the form like length uh and as a
1:24:54result so what I show here is that uh
1:24:55when you do lhf the more you do of lhf
1:24:58the longer the output of the of the
1:25:00models become so if you've ever been
1:25:02annoyed at chat GPT answering you super
1:25:04long sentences this is because of all
1:25:07rhf um annotator distribution shift uh
1:25:11like the distribution of annotators that
1:25:13you use matters a lot and you have to
1:25:15think like what is what is even the
1:25:17humans that we want to represent in
1:25:18these models uh now the question is like
1:25:20crowdsourcing ethics uh like usually
1:25:23these basically a lot of the the
1:25:25labeling that is done um like the people
1:25:28who do them are not paid well and they
1:25:30have to go through a lot of toxic data
1:25:32uh because you basically want the model
1:25:34to avoid saying the toxic data um so
1:25:37crowdsourcing ethics
1:25:39too so many challenges with human data
1:25:42um so what we did also last year is
1:25:45again the same thing as alpaca just the
1:25:47idea of like oh well they're challenges
1:25:49with humans maybe we can just replace
1:25:50them with llms uh so what we did is
1:25:53simply replace
1:25:54um oh I see that I'm just realizing that
1:25:57the slides are not sented anyways uh you
1:26:00replace a human preference with LM
1:26:01preferences uh so here on this uh figure
1:26:04you see on the xaxis the price that we
1:26:06paid uh for collecting human data it's
1:26:09around
1:26:10$300 for 1,000 examples and this is on
1:26:13mechanical turkers which are usually
1:26:15like cheaper than than maybe some of the
1:26:17other um companies that you could go
1:26:20through and on the Y AIS it's basically
1:26:22the agreement with uh other humans with
1:26:25the mode of other humans and what you
1:26:27see is that actually as I told you
1:26:28before labeling is really complicated
1:26:30humans agree with themselves only around
1:26:3366% of the time on a binary Tas and it's
1:26:36not that the humans are not good here
1:26:38because uh we were five main authors on
1:26:40this paper we tried to label this data
1:26:43ourselves and we only had like say 67 or
1:26:4668% accuracy even though we talk like we
1:26:48talk for like 3 hours of how we should
1:26:50be doing labeling really it's
1:26:52complicated it's not an easy task um and
1:26:54here I just showed many different models
1:26:56and um basically you see that models are
1:26:58much cheaper and they can actually get
1:27:00higher agreement with the mode of humans
1:27:02than human humans themselves and the
1:27:04reason why is because humans have a lot
1:27:06of varant models have no varant so they
1:27:08might be a little bit more biased but
1:27:09have less virence uh so it works
1:27:12surprisingly well and now it's kind of
1:27:14the standard in open uh Source Community
1:27:16I think even in Industry a lot of people
1:27:18use both humans and llms for improving
1:27:21uh the colle collection of allf data
1:27:24um and this is like this is the paper
1:27:26from last year but honestly now it's
1:27:28more like that llms would be around this
1:27:30agreement and this cost so around I
1:27:32would say 50x cheaper than humans and
1:27:34better agreement with human than humans
1:27:38themselves okay so that gets us to
1:27:41evaluation of post
1:27:43training um that goes back to your
1:27:46initial question at the beginning of the
1:27:47lecture how do you evaluate something
1:27:49like chpt uh the answers that chpt could
1:27:51give are basically unbounded and it's
1:27:54not that there one right answer there
1:27:56are many answers that are just as good
1:27:58um so there are many challenges one you
1:28:01can't use validation loss because one
1:28:05method might use po the other one might
1:28:06use DPO validation loss is not
1:28:08comparable second you can't use Cal uh
1:28:10sorry perplexity that's the thing I told
1:28:12you before these models uh are not
1:28:15calibrated they don't give distributions
1:28:17they they just optimize for one thing so
1:28:19you can't use perplexity for actually
1:28:21evaluating uh these type of models once
1:28:23they're aligned sorry one Z lined third
1:28:27uh there's a large diversity of
1:28:28questions that human might ask to these
1:28:30models generation open QA like some
1:28:33question answering some summarization
1:28:35and all of these things so there's so
1:28:36many things you have to cover um then
1:28:39the tasks are really open-ended so it's
1:28:41very hard to automate so that's what you
1:28:43were alluding to before so the idea uh
1:28:46is that instead of trying to come up
1:28:48with really easily automated uh
1:28:50benchmarks uh it's just we're going to
1:28:52ask questions that that users actually
1:28:54ask to these models in practice and
1:28:56we're just going to ask annotators to
1:28:58say between these two models which one
1:29:01is better like what's the what's the
1:29:02better output so basically do exact same
1:29:04thing as um basically the data from rhf
1:29:08but you use it now for evaluation yes
1:29:10I'm not sure I understand what you mean
1:29:12by like can't use perplexity and not
1:29:13calibrated right like LM is still doing
1:29:16like next token
1:29:18prediction so I can't so think about um
1:29:23the optim solution after doing PO is
1:29:26basically one model that gives you uh
1:29:28essentially a Delta um like basically
1:29:31says that there's only one sentence that
1:29:33is that could be generated for that
1:29:36question so now if you use it on
1:29:38something that is slightly semantically
1:29:39differently different it would actually
1:29:41give a likelihood of zero for that
1:29:43answer so in reality it's not that
1:29:45extreme because as you say it's still a
1:29:47distribution but I just shows you that
1:29:48there's a there's a fundamental issue
1:29:50with perplexity once these models are
1:29:53not llms anymore they were not trained
1:29:56at least with P they were not trained to
1:29:57to do maximum likelihood anymore they
1:29:59were trained to be
1:30:02policies okay um so probably the most
1:30:05common or like the most um yeah the most
1:30:08common Benchmark or the most trusted one
1:30:11is what we call Chad uh sorry chatbot
1:30:12Arena uh which is basically go on
1:30:15internet have random users on the
1:30:17internet blindly talk with two chat Bots
1:30:19just ask many questions see the two
1:30:21answers and rate which one is better and
1:30:24and you do that over hundred of
1:30:25thousands of users and then you get uh
1:30:27the actual preferences and you get
1:30:29rankings of models uh so you can go
1:30:31right now on chatbot Arena and actually
1:30:34interact with these models um one
1:30:36potential issue just to highlight is
1:30:38that while people who want to do these
1:30:40type of things are usually more like
1:30:41Tech driven um or like techsavvy uh so a
1:30:44lot of the questions that you will ask
1:30:46are more like Tech stuff discussing
1:30:47software errors inquiries about AI tools
1:30:50and all these things um so another issue
1:30:53is cost and speed if you really want to
1:30:55use something like this for development
1:30:57process um it will be too costly because
1:30:59you would need to basically pay a lot of
1:31:01humans to do that so one simple idea is
1:31:06again as we said many times just use LM
1:31:08instead of humans uh you probably know
1:31:11the drill at this point uh steps for
1:31:13every instruction generate outputs by
1:31:15some baseline and the model that you
1:31:17want to evaluate um so here you imagine
1:31:20that I I'm comparing an answer from Chad
1:31:22GPT and from
1:31:24I'm just asking a model uh another model
1:31:28uh which one is better and I just
1:31:30basically average that out uh yeah I
1:31:32asked gp4 which one is better I average
1:31:35that out over my entire distribution
1:31:37over my entire Benchmark or data set and
1:31:39that gives me a RN rate so RN
1:31:41probability for one model compared to
1:31:43another one and now you can rank models
1:31:46uh and this is the Alpa eval uh
1:31:49leaderboard so the benefits of this is
1:31:52that actually we show we get 98%
1:31:54correlation with Chad B Arena so very
1:31:56high correlation with humans um so this
1:31:59is yeah comparison with correlation with
1:32:01other benchmarks and it takes less than
1:32:03three minutes and less than $10 to run
1:32:05so it's pretty cheap um there are
1:32:07downsides though uh one of them is purus
1:32:10correlation um so as we already saw
1:32:13before LMS prefer this is one SP
1:32:15correlation not many I'll just talk
1:32:17about one LMS prefer longer outputs
1:32:18actually humans also prefer longer
1:32:20outputs but the problem or the issue
1:32:22once you use llms is that once there
1:32:24bias you will continue optimizing that
1:32:26humans at some point I can guarantee you
1:32:28if I ask a simple question and you give
1:32:29me five pages of answers I'll be like no
1:32:31I don't like that answer but LMS if they
1:32:33have this bius and they were trained for
1:32:34that they will continue preferring
1:32:36longer outputs so uh here we see um the
1:32:41the preference just showing that like
1:32:43humans and models prefer longer outputs
1:32:45um and here is another view of the
1:32:48initial apaka eval data uh Benchmark
1:32:51where when we asked um when we we rank
1:32:54gp4 when we look at the Run rate of gp4
1:32:56versus actually uh gp4 itself if we com
1:33:00if we use the standard GPT 4 it gets 50%
1:33:02kind of by definition because we're
1:33:03comparing GPT 4 versus gp4 but if we ask
1:33:07a gbd4 to be slightly more verose so we
1:33:09just say in the prompt be Vos in your
1:33:11answers then it gets a r rate of
1:33:1364.4% so really there's a huge variance
1:33:16and if we ask it to be concise it gets
1:33:1820% so there's a huge variance depending
1:33:20on um whether you ask it to be concise
1:33:23of
1:33:24that's very annoying um so one possible
1:33:27solution which is what we did is uh just
1:33:29use some regression analysis I'm not
1:33:31going to go into details but basically
1:33:33use Cal inference tools to control for
1:33:35length and right now uh actually length
1:33:37matters much less so if you ask it to be
1:33:39veros we still get some gains but much
1:33:43less great so that's all about post
1:33:46training and now for the next eight
1:33:48minutes I might talk about systems or
1:33:50just answer questions yes can you um go
1:33:54back to your post training in terms of
1:33:56post training how did we tune those
1:33:58parameters using the small body of
1:34:01fine-tuning data and have such big
1:34:04effect on the model you mentioned
1:34:05earlier that there's a different set of
1:34:07hyperparameters are we changing just
1:34:10some of the weights the later weights or
1:34:11all the weights what's actually
1:34:13happening yeah uh yeah I I kind of
1:34:15skimmed through all of this you change
1:34:17all the weights actually um industry
1:34:19would change all the weights in open
1:34:21source land you might have heard of
1:34:23Laura which is going to change basically
1:34:26only some of the weights or it actually
1:34:28to be more specific it's going to add
1:34:30some differences to the output of every
1:34:32of every layer but but in Industry
1:34:34you're going to just fine tune all the
1:34:36weights um and also to say something
1:34:39else about the data actually the SL St
1:34:41all HF you usually going to collect uh a
1:34:44lot more data than with sft so if fft is
1:34:46like 5,000 10,000 maybe 50,000 with rhf
1:34:51I think you're going to be more around
1:34:52like the 1 million
1:34:54uh order of magnitude it's still much
1:34:55less than pre-training though yeah
1:34:57because pre-training is 15 trillion
1:34:59tokens I mean this is like that's not
1:35:01even a drop and yet you influence the
1:35:04weight a lot so because you do it I mean
1:35:06you have to think that how you do it is
1:35:08you use um I mean as I said the learning
1:35:12rate that you're going to use is going
1:35:13to be different but also you only do
1:35:15that so just imagine if I train even if
1:35:18I train on one sentence but over and
1:35:20over again all at some point my model
1:35:23will only that sentence even if uh it
1:35:26was just one sentence instead of the 15
1:35:28trillion tokens so if you use a large
1:35:30enough learning rate and for enough time
1:35:32you will basically overfit that sentence
1:35:35so the the the key thing to to remember
1:35:37is that um the data is not I it's not as
1:35:40if you mix some posttraining data and
1:35:42some pre-training data you do
1:35:44pre-training and then you just start
1:35:46fine-tuning only on the post trining so
1:35:48another way maybe another perspective is
1:35:50that the post the pre-training is just
1:35:52the initialization of your model
1:35:54and once you view it that way that this
1:35:55is just initialization of Weights then
1:35:58there's nothing special like you don't
1:36:00need to remember that you train a lot of
1:36:01data before the only thing that matters
1:36:03is that you had an initialization and
1:36:05now I actually train a model so maybe
1:36:06think about it that way like there's a
1:36:08there's a mark of property in some way
1:36:10just like you had your weights this is
1:36:11my initialization now I'm training that
1:36:13one does that kind of answer your
1:36:15question kind of but you said something
1:36:19just now about it's almost the
1:36:21equivalence of just rerunning the find
1:36:23tuning data many times is it actually is
1:36:26that what actually happens in order to
1:36:29give so much more preference
1:36:32um you might I actually don't know right
1:36:36now how they do it in Industry when we
1:36:38did alpaca we had to do three box so you
1:36:40did run it three times to it
1:36:43um but I mean even the number of times
1:36:46that you run it through it's actually
1:36:47not important the only thing like the
1:36:49only thing is the is kind of the
1:36:51effective learning rate that what
1:36:53matters
1:36:54um so
1:36:56yeah
1:36:57great so I think I have five minutes
1:37:02[Music]
1:37:05right okay I might try to give a high
1:37:10level Overview at least from one of the
1:37:12systems trick systems as we said uh for
1:37:17everyone Bott neck is a sorry compute is
1:37:20the huge bottleneck uh one question you
1:37:22might ask is why not buy more gpus uh
1:37:25gpus are expensive but also are scarce
1:37:26even if you have $10 million right now
1:37:28you cannot buy the best gpus um
1:37:32there's oh yeah there's also some
1:37:34physical limitations when you have when
1:37:36you have multiple gpus you have to
1:37:37communicate between them that takes time
1:37:40um so just buying more gpus is not that
1:37:42easy um so it's really important to
1:37:44think about how do you allocate
1:37:46resources and how do you optimize your
1:37:47pipeline so system 101 on gpus I'm sorry
1:37:51I'm going slightly faster I hope for
1:37:53that some of you at least can follow uh
1:37:55gpus are basically optimized for
1:37:57throughput CPUs are optimized uh for
1:38:00latency so gpus the way you have to
1:38:03think about it is that there's one Comm
1:38:04there's one command that is run on many
1:38:07many Calles at the same time on
1:38:08different type of data um so this is how
1:38:12you see a GPU you see there are many
1:38:14different CES we call them streaming
1:38:16multiprocessors which is very different
1:38:18than the usual CPU architecture so just
1:38:20think High throughput paralyzation for
1:38:23gpus uh gpus are optimized for fast
1:38:26matrix multiplication so every time you
1:38:28will do uh you will do something on GPU
1:38:30if you can do it with a a matrix
1:38:32multiplication it's going to be 10 times
1:38:34faster than with anything else uh that
1:38:37is a little bit annoying because it
1:38:38means that we're kind of uh bottlenecked
1:38:40to doing anything with Matrix
1:38:43multiplications um another thing to note
1:38:45with gpus is that compute has been
1:38:47improving faster than memory and
1:38:49communication so right now gpus usually
1:38:53are hard to keep uh like the data that
1:38:56you send that send to gpus is actually
1:38:59hard to keep up with the processess so
1:39:00most of your gpus are actually going to
1:39:02be idle if you just run normal code if
1:39:05you don't optimize your code so
1:39:06communication and this will continue
1:39:09over time another thing to know about
1:39:11gpus is that there's a memory hierarchy
1:39:13this is the same thing actually with
1:39:14CPUs but basically the closer you are to
1:39:17your cuse the less memory there is but
1:39:19the faster things run if you're further
1:39:21more memory slower
1:39:24um okay I'm going to skip that okay
1:39:26actually I'm going to say it I told you
1:39:28about this uh the fact of communication
1:39:31uh the metric that people usually look
1:39:32at is model flop utilization so what is
1:39:34the theoretical maximum that GPU could
1:39:37run at no more flops that you could use
1:39:38per second divide sorry the number of OB
1:39:41observed through put divided by this
1:39:43theoretical um maximum and in general if
1:39:47you reach 50% you're very happy like
1:39:49Facebook I looked at Lama was at 45 or
1:39:51something like this so that that means
1:39:54that data doesn't come fast enough even
1:39:56for these big
1:39:58companies so one simple trick and that
1:40:00might be the only one I'm going to tell
1:40:02you about is low Precision one simple
1:40:05idea is that well if I'm going to put my
1:40:07floats in lower Precision then there's
1:40:09going to be fewer bits that I have to
1:40:11send to my gpus if there's fewer bits
1:40:13it's faster communication lower memory
1:40:15consumption things are going to go
1:40:16faster uh and for deep learning it just
1:40:19happens that de decimal is not that
1:40:21important uh so so when you do matrix
1:40:24multiplication when you do like for
1:40:26example SGD there's already so much
1:40:27noise that if you update something by
1:40:290.01 or
1:40:310.015 who cares uh so basically instead
1:40:34of using uh 32 bits per float which is
1:40:38um what people used to use or 64 for
1:40:41example which is what you would use in
1:40:42other domains you use 16 bits uh for
1:40:45matrix multiplication so for every float
1:40:47you use 16 bits um and for training you
1:40:50have this type of like uh what we call
1:40:53aut atic mix Precision which is that uh
1:40:55some of the things are in 32 bits others
1:40:57are in 60 bit in 16 bits um generally
1:41:00the way you should be thinking about it
1:41:02is that your weights are stored of your
1:41:04model are stored in 32 bits um but just
1:41:07before the computation you put
1:41:08everything in 16 16 bits like this you
1:41:10do computation super fast and at the end
1:41:13you update your weights in 32 Bits And
1:41:16the reason why you do all the updates in
1:41:1732 bits it's just think that if your
1:41:19learning rate for example is very small
1:41:21you still want to be able to like make a
1:41:23difference in your weights uh so all the
1:41:25computation is done in 16 bits but the
1:41:28weights are actually stored in 32 bits
1:41:30so that's like the standard way that
1:41:32people are doing it um okay I'll
1:41:35actually talk just about this and then
1:41:37I'll skip all the rest operator Fusion
1:41:38because I think this is actually pretty
1:41:39cool as I just said communication is
1:41:41very slow and actually every time you
1:41:44use a pie torch line it basically moves
1:41:46variable to Global memory of your GPU so
1:41:49when you have something like this x do
1:41:52cosine uh equal X1 and then you do X1 do
1:41:55cosine what is happening behind the
1:41:57scenes is that you take the X which is
1:41:59data you ship it to your um to your
1:42:02actual processes of your gpus you apply
1:42:04the coign you ship it back to the main
1:42:06memory of your GPU and then you see the
1:42:08next sign you ship it back to the
1:42:10computer to the GPU processor you apply
1:42:12another cosign and you ship it back
1:42:14again um so another way to see that is
1:42:17that you go from your Dam which is your
1:42:18Global memory in your GPU and you ship
1:42:21it to compute you ship it back for every
1:42:23line This is a naive way of doing it
1:42:25this seems very wasteful um so the idea
1:42:29simple idea of operative Fusion is just
1:42:32communicate do all the computation ship
1:42:34it back once and this is exactly what
1:42:37fuse kernels are um so if you ever want
1:42:40to make your comp your computations in
1:42:44pytorch much faster just apply torch.
1:42:47compile on your model this is going to
1:42:49make your model around two times faster
1:42:52and what it does is simply that it
1:42:53rewrites your code uh your P like your
1:42:57py torch code basically in C++ in Cuda
1:43:01uh to to do the communication only once
1:43:04then do all the operations then uh ship
1:43:06it back okay I'm not going to have time
1:43:09to talk about tiling tiling is important
1:43:11paration paration is important um and
1:43:16mixture of experts mixture of experts is
1:43:18important Outlook there are many things
1:43:20we haven't T talked about we haven't
1:43:23talked about architectures we definitely
1:43:25haven't talked about inference um there
1:43:27are many other things that are important
1:43:29with LMS what is the UI that you use I
1:43:31mean arguably chat jpt the big novelty
1:43:33was just have a simple UI to use it
1:43:35multimodality what are all the misuses
1:43:37you could have uh the fact that there
1:43:39might not be enough data on the internet
1:43:41to train all these models legality of
1:43:43data collection so many other things if
1:43:45you are interested in all these topics
1:43:47uh I would suggest three classes cs224n
1:43:50is probably the one that touches the
1:43:52least on uh LMS uh but it gives some
1:43:55background and historical context um of
1:43:58all the LMS and gives kind of some
1:44:00adjacent material CS 324 I think it's
1:44:03called Uh I think it's just called large
1:44:06language models uh more in-depth reading
1:44:08and lectures on everything I talked
1:44:09about CS 336 which is large language
1:44:12model from scratch you actually build
1:44:14your own llm uh it's an amazing class
1:44:18also given by my two supervisors very
1:44:20heavy workload so be careful and um
1:44:23great

About this transcript

This page contains the full transcript of YouTube transcript (9vM4p9NN0Ts) , generated from the public captions YouTube serves with the video. The transcript has 19,443 words across 2,704 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.