YT YouTube2Text

YouTube2Text → Transcripts → Gradient Boost Part 1 (of 4): Regression Main Ideas

Gradient Boost Part 1 (of 4): Regression Main Ideas — Transcript

by StatQuest with Josh Starmer · 2,029 words · 378 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:01Gradient Boost, part one.
0:04Regression main ideas.
0:07StatQuest.
0:10Hello, I'm Josh Starmer and welcome to
0:13StatQuest. Today, we're going to talk
0:15about the gradient boost machine
0:17learning algorithm. Specifically, we're
0:19going to focus on how gradient boost is
0:21used for regression.
0:24Note, this StatQuest assumes you already
0:27understand decision trees. So, if you're
0:29not already down with those, check out
0:32the quest.
0:33This StatQuest also assumes that you are
0:36familiar with AdaBoost and the trade-off
0:38between bias and variance. If not, check
0:42out the quests. The links are in the
0:44description below.
0:46This StatQuest is the first part in a
0:49series that explains how the gradient
0:51boost machine learning algorithm works.
0:54Specifically, we'll use this data
0:57where we have the height measurements
0:59from six people, their favorite colors,
1:02their genders,
1:04and their weights.
1:07And we'll walk through, step-by-step,
1:09the most common way that gradient boost
1:11fits a model to this training data.
1:14Note, when gradient boost is used to
1:17predict a continuous value, like weight,
1:20we say that we are using gradient boost
1:22for regression.
1:24Using gradient boost for regression is
1:27different from doing linear regression.
1:29So, while the two methods are related,
1:31don't get them confused with each other.
1:34Part two in this series will dive deep
1:37into the math behind the gradient boost
1:39algorithm for regression, walking
1:41through it step-by-step and proving that
1:43what we covered today is correct.
1:47Part three in this series shows how
1:49gradient boost can be used for
1:51classification.
1:53Specifically, we'll walk through,
1:55step-by-step, the most common way
1:58gradient boost can classify someone as
2:00either loving the movie Troll 2 or not
2:03loving Troll 2.
2:05Part four will return to the math behind
2:08gradient boost, this time focusing on
2:10classification, walking through it
2:12step-by-step.
2:14Note, the gradient boost algorithm looks
2:17complicated because it was designed to
2:19be configured in a wide variety of ways.
2:24But, the reality is that 99% of the time
2:27only one configuration is used to
2:29predict continuous values like weight,
2:32and one configuration is used to
2:34classify samples into different
2:36categories.
2:38This StatQuest focuses on showing you
2:40the most common way gradient boost is
2:42used to predict a continuous value like
2:45weight.
2:47If you are familiar with AdaBoost, then
2:49a lot of gradient boost will seem very
2:51similar.
2:53So, let's briefly compare and contrast
2:55AdaBoost and gradient boost.
2:59If we want to use these measurements to
3:00predict weight,
3:02then AdaBoost starts by building a very
3:05short tree called a stump from the
3:07training data.
3:09And then the amount of say that the new
3:11stump has on the final output is based
3:14on how well it compensated for those
3:16previous errors.
3:19Then AdaBoost builds the next stump
3:21based on errors that the previous stump
3:23made.
3:25In this example, the new stump did a
3:27poor job compensating for the previous
3:30stump's errors, and its size reflects
3:32its reduced amount of say.
3:35Then AdaBoost builds another stump based
3:38on the errors made by the previous
3:40stump.
3:41And this stump did a little better than
3:43the last stump, so it's a little larger.
3:47Then AdaBoost continues to make stumps
3:49in this fashion until it has made the
3:51number of stumps you asked for or it has
3:54a perfect fit.
3:56In contrast, gradient boost starts by
3:59making a single leaf instead of a tree
4:01or stump.
4:04This leaf represents an initial guess
4:06for the weights for all of the samples.
4:10When trying to predict a continuous
4:12value like weight, the first guess is
4:14the average value.
4:17Then gradient boost builds a tree.
4:20Like AdaBoost, this tree is based on the
4:23errors made by the previous tree.
4:26But unlike AdaBoost, this tree is
4:28usually larger than a stump.
4:31That said, gradient boost still
4:33restricts the size of the tree.
4:36In the simple example that we will go
4:38through in this stat quest, we will
4:40build trees with up to four leaves, but
4:43no larger.
4:45However, in practice, people often set
4:48the maximum number of leaves to be
4:50between eight and 32.
4:53Thus, like AdaBoost, gradient boost
4:56builds fixed-size trees based on the
4:58previous trees' errors, but unlike
5:00AdaBoost, each tree can be larger than a
5:03stump.
5:05Also, like AdaBoost, gradient boost
5:08scales the trees. However, gradient
5:11boost scales all trees by the same
5:14amount.
5:16Then gradient boost builds another tree
5:18based on the errors made by the previous
5:20tree.
5:22And then it scales the tree.
5:25And gradient boost continues to build
5:27trees in this fashion until it has made
5:29the number of trees you asked for or
5:32additional trees fail to improve the
5:34fit.
5:36Now that we know the main similarities
5:38and differences between gradient boost
5:40and AdaBoost,
5:42let's see how the most common gradient
5:44boost configuration would use this
5:47training data to predict weight.
5:50The first thing we do is calculate the
5:52average weight.
5:54This is the first attempt at predicting
5:57everyone's weight.
5:59In other words, if we stopped right now,
6:02we would predict that everyone weighed
6:0471.2 kg.
6:08However, gradient boost doesn't stop
6:10here.
6:12The next thing we do is build a tree
6:15based on the errors from the first tree.
6:18The errors that the previous tree made
6:20are the differences between the observed
6:23weights and the predicted weight, 71.2.
6:28So, let's start by plugging in 71.2
6:31for the predicted weight.
6:34And then plug in the first observed
6:36weight.
6:38And do the math.
6:41And save the difference, which is called
6:43a pseudo residual, in a new column.
6:47Note, the term pseudo residual is based
6:50on linear regression, where the
6:52difference between the observed values
6:54and the predicted values results in
6:56residuals.
6:59The pseudo part of pseudo residual is a
7:02reminder that we are doing gradient
7:03boost, not linear regression, and is
7:06something I'll talk more about in part
7:08two of this series when we go through
7:10the math.
7:12Now, we do the same thing for the
7:14remaining weights.
7:17Now, we will build a tree using height,
7:20favorite color, and gender
7:22to predict the residuals.
7:25If it seems strange to predict the
7:27residuals instead of the original
7:29weights, just bear with me, and soon all
7:32will become clear.
7:35So, setting aside the reason why we are
7:38building a tree to predict the residuals
7:40for the time being, here's the tree.
7:44Remember, in this example, we are only
7:47allowing up to four leaves.
7:49But when using a larger data set, it is
7:52common to allow anywhere from eight to
7:5432.
7:57By restricting the total number of
7:58leaves, we get fewer leaves than
8:01residuals.
8:03As a result, these two rows of data go
8:06to the same leaf.
8:08So, we replace these residuals with
8:11average.
8:13And these two rows of data go to the
8:15same leaf.
8:17So, we replace these residuals with
8:20their average.
8:22Now, we can combine the original leaf
8:25with the new tree
8:27to make a new prediction of an
8:29individual's weight from the training
8:31data.
8:33We start with the initial prediction,
8:3571.2.
8:38Then, we run the data down the tree,
8:41and we get 16.8.
8:45So, the predicted weight equals 71.2
8:49+ 16.8,
8:51which equals 88.
8:54Which is the same as the observed
8:56weight.
8:57Is this awesome?
9:00No. The model fits the training data too
9:03well.
9:05In other words, we have low bias, but
9:07probably very high variance.
9:11Gradient boost deals with this problem
9:13by using a learning rate to scale the
9:15contribution from the new tree.
9:19The learning rate is a value between
9:21zero and one.
9:24In this case, we'll set the learning
9:26rate to 0.1.
9:29Now, the predicted weight equals 71.2
9:33+ 0.1 * 16.8,
9:37which equals 72.9.
9:41With the learning rate set to 0.1,
9:44the new prediction isn't as good as it
9:46was before,
9:48but it's a little better than the
9:49prediction made with just the original
9:51leaf, which predicted that all samples
9:54would weigh 71.2.
9:57In other words, scaling the tree by the
9:59learning rate results in a small step in
10:02the right direction.
10:04According to the dude that invented
10:06gradient boost, Jerome Friedman,
10:08empirical evidence shows that taking
10:10lots of small steps in the right
10:12direction results in better predictions
10:14with a testing data set, i.e. lower
10:17variance.
10:19Bam!
10:21So, let's build another tree so we can
10:24take another small step in the right
10:26direction.
10:28Just like before, we calculate the
10:30pseudo residuals, the difference between
10:32the observed weights and our latest
10:34predictions.
10:36So, we plug in the observed weight
10:40and the new predicted weight
10:43and we get 15.1.
10:46And we save that in the column for
10:48pseudo residuals.
10:51Then we repeat for all of the other
10:53individuals in the training data set.
10:58Small bam.
11:00Note, these are the original residuals
11:03from when our prediction was simply the
11:05average overall weight.
11:08And these are the residuals after adding
11:10the new tree scaled by the learning
11:12rate.
11:14The new residuals are all smaller than
11:16before, so we've taken a small step in
11:19the right direction.
11:21Double bam!
11:24Now, let's build a new tree to predict
11:27the new residuals.
11:30And here's the new tree.
11:32Note, in this simple example, the
11:35branches are the same as before.
11:37However, in practice, the trees can be
11:40different each time.
11:43Just like before, since multiple samples
11:45ended up in these leaves, we just
11:47replace the residuals with their
11:49averages.
11:51Now we combine the new tree with the
11:54previous tree and the initial leaf.
11:58Note, we scale all of the trees by the
12:01learning rate, which we set to 0.1.
12:05And add everything together.
12:08Now we're ready to make a new prediction
12:10from the training data.
12:13Just like before, we start with the
12:15initial prediction.
12:17Then add the scaled amount from the
12:19first tree
12:21and the scaled amount from the second
12:23tree.
12:25That gives us 71.2
12:29+ 0.1 * 16.8
12:32+ 0.1 * 15.1
12:36which equals 74.4.
12:40Which is another small step closer to
12:43the observed weight.
12:45Now we use the initial leaf
12:48plus the scaled values from the first
12:50tree
12:51plus the scaled values from the second
12:53tree
12:55to calculate new residuals.
12:59Remember, these were the residuals from
13:01when we just used a single leaf to
13:04predict weight.
13:06And these were the residuals after we
13:08added the first tree to the prediction.
13:11And these are the residuals after we
13:14added the second tree to the prediction.
13:17Each time we add a tree to the
13:19prediction, the residuals get smaller.
13:23So we've taken another small step
13:25towards making good predictions.
13:28Now we build another tree to predict the
13:31new residuals.
13:34And add it to the chain of trees that we
13:36have already created.
13:39And we keep making trees until we reach
13:41the maximum specified or adding
13:43additional trees does not significantly
13:46reduce the size of the residuals.
13:49Bam.
13:50Then, when we get some new measurements,
13:54we can predict weight
13:56by starting with the initial prediction,
14:00then adding the scaled value from the
14:02first tree,
14:04and the second tree,
14:06and the third tree,
14:09et cetera, et cetera, et cetera.
14:13And once the math is all done, we are
14:15left with the predicted weight.
14:18In this case, we predicted that this
14:21person weighed 70 kg.
14:24Triple bam.
14:28In summary, when gradient boost is used
14:31for regression,
14:33we start with a leaf that is the average
14:35value of the variable we want to
14:37predict.
14:39In this case, we wanted to predict
14:42weight.
14:44Then we add a tree based on the
14:46residuals, the difference between the
14:48observed values and the predicted
14:50values.
14:52And we scale the tree's contribution to
14:55the final prediction with a learning
14:57rate.
14:59Then we add another tree based on the
15:01new residuals.
15:04And we keep adding trees based on the
15:06errors made by the previous tree.
15:09That's all there is to it.
15:12Bam.
15:14Tune in for part two in this series when
15:16we dive deep into the math behind the
15:18gradient boost algorithm for regression,
15:21walking through it step by step, and
15:23proving that it really is this simple.
15:27Hooray! We've made it to the end of
15:29another exciting StatQuest. If you like
15:32this StatQuest and want to see more,
15:34please subscribe. And if you want to
15:36support StatQuest, consider buying one
15:38of my original songs, or buying a
15:40StatQuest t-shirt or hoodie. The links
15:43are in the description below.
15:45All right. Until next time, quest on.

About this transcript

This page contains the full transcript of Gradient Boost Part 1 (of 4): Regression Main Ideas by StatQuest with Josh Starmer, generated from the public captions YouTube serves with the video. The transcript has 2,029 words across 378 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.