YT YouTube2Text

YouTube2Text → Transcripts → Linear Regression, Clearly Explained!!!

Linear Regression, Clearly Explained!!! — Transcript

by StatQuest with Josh Starmer · 3,687 words · 647 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Sailing on a boat headed towards
0:03StatQuest.
0:06Join me on this boat, let's go to
0:08StatQuest.
0:10It's super cool.
0:13Hello, and welcome to StatQuest.
0:16StatQuest is brought to you by the
0:18friendly folks in the genetics
0:19department at the University of North
0:21Carolina at Chapel Hill.
0:23Today, we're going to be talking about
0:25linear regression, aka general linear
0:28models part one. There's a lot of parts
0:31to linear models, but it's a really cool
0:33and powerful concept. So, let's get
0:36right down to it.
0:38I promise you, I have lots and lots of
0:40slides that talk about all the
0:42nitty-gritty details behind linear
0:44regression. But first, let's talk about
0:46the main ideas behind it.
0:49The first thing you do in linear
0:50regression is use least squares to fit a
0:53line to the data.
0:55The second thing you do is calculate R
0:58squared.
0:59Lastly, calculate a P value for R
1:02squared.
1:04There are lots of other little things
1:05that come up along the way, but these
1:08are the three most important concepts
1:10behind linear regression.
1:13In the StatQuest fitting a line to data,
1:16we talked about
1:18fitting a line to data. Duh.
1:21But let's do a quick review.
1:24I'm going to introduce some new
1:25terminology in this part of the video,
1:27so it's worth watching even if you've
1:29already seen the earlier StatQuest.
1:32That said, if you need more details,
1:34check that StatQuest out.
1:37For this review, we're going to be
1:39talking about a data set where we took a
1:41bunch of mice, and we measured their
1:43size, and we measured their weight.
1:46Our goal is to use mouse weight as a way
1:50to predict mouse size.
1:53First, draw a a through the data.
1:57Second, measure the distance from the
1:59line to the data. Square each distance,
2:02and then add them up.
2:05Terminology alert.
2:07The distance from the line to the data
2:08point is called a residual.
2:12Third, rotate the line a little bit.
2:16With the new line, measure the
2:18residuals, square them, and then sum up
2:21the squares.
2:23Now, rotate the line a little bit more.
2:27Sum up the squared residuals, etc.,
2:30etc., etc. We rotate, and then sum up
2:33the squared residuals. Rotate, then sum
2:35up the squared residuals. Just keep
2:37doing that.
2:39After a bunch of rotations, you can plot
2:42the sum of squared residuals and
2:44corresponding rotation.
2:46So, in this graph, we have the sum of
2:48squared residuals on the Y axis, and the
2:51different rotations on the X axis.
2:55Lastly, you find the rotation that has
2:58the least sum of squares.
3:00More details about how this is actually
3:03done in practice are provided in the
3:05StatQuest on fitting a line to data.
3:09So, we see that this rotation is the one
3:12with the least squares. So, it will be
3:15the one to fit to the data.
3:18This is our least squares rotation
3:21superimposed on the original data.
3:24Bam!
3:25Now, we know why the method for fitting
3:27a line is called least squares.
3:31Now, we have fit a line to the data.
3:33This is awesome.
3:36Here's the equation for the line.
3:38Least squares estimated two parameters.
3:43A Y axis intercept
3:46and a slope.
3:48Since the slope is not zero, it means
3:51that knowing a mouse's weight will help
3:53us make a guess about that mouse's size.
3:57How good is that guess?
4:00Calculating R squared is the first step
4:03in determining how good that guess will
4:06be.
4:07The StatQuest R squared explained talks
4:11about
4:12you got it R squared.
4:15Let's do a quick review.
4:17I'm also going to introduce some
4:19additional terminology. So, it's worth
4:21watching this part of the video even if
4:23you've seen the original StatQuest on R
4:25squared.
4:27First, calculate the average mouse size.
4:31Okay, I've just shifted all the data
4:34points to the Y axis to emphasize that
4:37at this point we are only interested in
4:40mouse size.
4:42Here, I've drawn a black line to show
4:45the average mouse size.
4:48Bam!
4:50Now, sum the squared residuals.
4:53Just like in least squares, we measure
4:56the distance from the mean to the data
4:58point and square it and then add those
5:01squares together.
5:04Terminology alert. We'll call this SS
5:07mean for sum of squares around the mean.
5:12Note,
5:13the sum of squares around the mean
5:15equals the data minus the mean squared.
5:20The variation around the mean equals the
5:22data minus the mean squared divided by
5:25n.
5:27N is the sample size. In this case, n
5:30equals 9.
5:32The shorthand notation is the variation
5:35around the mean equals the sum of
5:37squares around the mean divided by n,
5:40the sample size.
5:42Another way to think about variance is
5:45as the average sum of squares per mouse.
5:49Now, go back to the original plot and
5:52sum up the squared residuals around our
5:55least squares fit.
5:57We'll call this SS fit for the sum of
6:01squares around the least squares fit.
6:04The sum of squares around the least
6:06squares fit is the sum of the distances
6:10between the data and the line squared.
6:14Just like with the mean, the variance
6:17around the fit
6:18is the distance between the line and the
6:20data squared divided by n, the sample
6:24size.
6:25The shorthand is the variation around
6:28the fitted line equals the sum of
6:30squares around the fitted line divided
6:33by n, the sample size.
6:36Again, we can think of the variation
6:38around the fit as the average of the sum
6:41of squares around the fit for each
6:43mouse.
6:45In general, the variance of something
6:47equals the sum of squares divided by the
6:50number of those things.
6:52In other words, it's an average of sum
6:55of squares.
6:56I mentioned this because it's going to
6:58come in handy in a little bit, so keep
7:00it in the back of your mind.
7:03Okay, let's step back a little bit.
7:05This is the raw variation in mouse size.
7:10And this is the variation around the
7:12least squares line.
7:14There is less variation around the line
7:16that we fit by least squares. That is to
7:19say, the residuals are smaller.
7:22As a result, we say that some of the
7:25variation in mouse size is explained by
7:28taking mouse weight into account.
7:31In other words, heavier mice are bigger,
7:35lighter mice are smaller.
7:37R squared tells us how much of the
7:40variation in mouse size can be explained
7:43by taking mouse weight into account.
7:47This is the formula for R squared. It's
7:50the variation around the mean minus the
7:52variation around the fit divided by the
7:55variation around the mean.
7:58Let's look at an example.
8:00In this example,
8:02the variation around the mean equals
8:0411.1
8:06and the variation around the fit equals
8:084.4.
8:10So, we plug those numbers into the
8:12equation.
8:13The result is that R squared equals 0.6,
8:18which is the same thing as saying 60%.
8:22This means there is a 60% reduction in
8:26the variance when we take the mouse
8:28weight into account.
8:30Alternatively,
8:32we can say that mouse weight explains
8:3460% of the variation in mouse size.
8:39We can also use the sum of squares to
8:41make the same calculation.
8:43This is because when we're talking about
8:45variation, everything's divided by n,
8:48the sample size. Since everything's
8:50scaled by n, we can pull that term out
8:53and just use the raw sum of squares.
8:56In this case, the sum of squares around
8:58the mean equals 100
9:01and the sum of squares around the fit
9:03equals 40.
9:05Plugging those numbers into the equation
9:07gives us the same value we had before. R
9:10squared equals 0.6, which equals 60%.
9:1660% of the sums of squares of the mouse
9:19size can be explained by mouse weight.
9:23Here's another example. We're also going
9:25to go back to using variation in the
9:27calculation, since that's more common.
9:30In this case, knowing mouse weight means
9:33you can make a perfect prediction of
9:36mouse size.
9:38The variation around the mean is the
9:40same as it was before, 11.1.
9:44But now the variation around the fitted
9:46line equals zero because there are no
9:48residuals.
9:50Plugging the numbers in gives us an R
9:52squared equal to one, which equals 100%.
9:57In this case, mouse weight explains 100%
10:02of the variation in mouse size.
10:05Okay. One last example.
10:09In this case, knowing mouse weight
10:12doesn't help us predict mouse size.
10:15If someone tells us they have a heavy
10:17mouse, well, that mouse could either be
10:19small or large with equal probability.
10:23Similarly, if someone said they had a
10:25light mouse, well, again, we wouldn't
10:28know if it was a big mouse or a small
10:30mouse because each of those options is
10:33equally likely.
10:35Just like the other two examples, the
10:37variation around the mean is equal 11.1.
10:41However, in this case, the variation
10:43around the fit is also equal 11.1.
10:47So, we plug those numbers in and we get
10:50R squared equals zero, which equals zero
10:52percent.
10:54In this case, mouse weight doesn't
10:56explain any of the variation around the
10:59mean.
11:00When calculating the sum of squares
11:02around the mean, we collapse the points
11:05onto the Y axis just to emphasize the
11:08fact that we were ignoring mouse weight.
11:12But we could just as easily draw a line
11:15Y equals the mean mouse size
11:18and calculate the sum of squares around
11:20the mean around that.
11:24In this example, we applied R squared to
11:27a simple equation for a line Y equals
11:310.1
11:32+ 0.78
11:34* x.
11:36This gave us an r-squared of 60% meaning
11:4060% of the variation in mouse size could
11:43be explained by mouse weight.
11:46But the concept applies to any equation,
11:49no matter how complicated.
11:51First, you measure, square, and sum the
11:54distance from the data to the mean.
11:57Then measure, square, and sum the
12:00distance from the data to the
12:02complicated equation.
12:04Once you've got those two sums of
12:05squares, just plug them in and you've
12:07got r-squared.
12:10Let's look at a slightly more
12:11complicated example.
12:14Imagine we wanted to know if mouse
12:17weight and tail length did a good job
12:19predicting the length of the mouse's
12:21body.
12:23So we measure a bunch of mice.
12:26To plot this data, we need a
12:28three-dimensional graph.
12:31We want to know how well weight and tail
12:34length
12:35predict body length.
12:37The first mouse we measured had weight =
12:402.1,
12:42tail length = 1.3,
12:45and body length = 2.5.
12:49So that's how we plot this data on this
12:513D graph.
12:53Here's all the data in the graph.
12:56The larger circles are points that are
12:58closer to us and represent mice that
13:01have shorter tails.
13:03The smaller circles are points that are
13:05further from us and represent mice with
13:08longer tails.
13:10Now we do a least squares fit.
13:13Since we have the extra term in the
13:15equation representing an extra
13:17dimension, we fit a plane instead of a
13:20line.
13:22Here's the equation for the plane.
13:25The y value represents body length.
13:29Least squares estimates three different
13:32parameters.
13:34The first is the Y intercept. That's
13:37when both tail length and mouse weight
13:39are equal to zero.
13:41The second parameter, 0.7,
13:44is for the mouse weight.
13:47The last term,
13:480.5,
13:50is for the tail length.
13:52If we know a mouse's weight and tail
13:55length, we can use the equation to guess
13:58the body length.
14:00For example, given the weight and tail
14:02length for this mouse,
14:05the equation predicts this body length.
14:09Just like before, we can measure the
14:11residuals, square them, and then add
14:13them up to calculate R squared.
14:17Now,
14:18if the tail length, or the Z axis, is
14:21useless and doesn't make the sum of
14:23squares fit any smaller, then least
14:26squares will ignore it by making that
14:28parameter equal to zero.
14:31In this case, plugging the tail length
14:33into the equation would have no effect
14:36on predicting the mouse size.
14:39This means equations with more
14:41parameters will never make the sum of
14:44squares around the fit worse than
14:46equations with fewer parameters.
14:49In other words,
14:50this equation,
14:52mouse size = 0.3
14:55+ mouse weight + flip of a coin +
14:59favorite color + astrological sign +
15:02extra stuff, will never perform worse
15:05than this equation,
15:07mouse size = 0.3 + mouse weight.
15:11This is because least squares will cause
15:13any term that makes sum of squares
15:15around the fit worse to be multiplied by
15:18zero, and in a sense, no longer exist.
15:23Now, due to random chance, there is a
15:26small probability that the small mice in
15:29the data set might get heads more
15:31frequently than large mice.
15:34If this happened, then we'd get a
15:36smaller sum of squares fit
15:38and a better R squared.
15:41Womp womp. Here's the frowny face of sad
15:45times.
15:47The more silly parameters we add to the
15:49equation, the more opportunities we have
15:52for random events to reduce sum of
15:55squares fit and result in a better R
15:57squared.
15:59Thus, people report an adjusted R
16:01squared value that, in essence, scales R
16:05squared by the number of parameters.
16:09R squared is awesome,
16:11but it's missing something.
16:13What if all we had were two
16:15measurements?
16:17We'd calculate the sum of squares around
16:19the mean. In this case, that would be
16:2110.
16:23Then we'd calculate the sum of squares
16:25around the fit,
16:26which equals zero.
16:29The sum of squares around the fit equals
16:31zero because you can always draw a
16:33straight line to connect any two points.
16:37What this means is when we calculate R
16:40squared by plugging the numbers in,
16:42we're going to get 100%.
16:45100% is a great number. We've explained
16:48all the variation, but any two random
16:51points will give us the exact same
16:53thing. It doesn't actually mean
16:55anything.
16:57We need a way to determine if the R
17:00squared value is statistically
17:02significant.
17:04We need a P value.
17:06Before we calculate the P value, let's
17:09review the main concepts behind R
17:11squared one last time.
17:14The general equation for R squared is
17:17the variance around the mean minus the
17:19variance around the fit divided by the
17:22variance around the mean.
17:24In our example, this means the variation
17:27in the mouse size minus the variation
17:30after taking weight into account divided
17:33by the variation in mouse size.
17:36In other words, r squared equals the
17:38variation in mouse size explained by
17:41weight divided by the variation in mouse
17:44size without taking weight into account.
17:48In this particular example, r squared
17:51equals 0.6,
17:53meaning we saw a 60% reduction in
17:55variation once we took mouse weight into
17:58account.
18:00Now that we have a thorough
18:01understanding of the ideas behind r
18:04squared, let's talk about the main ideas
18:07behind calculating a p value for it.
18:10The p value for r squared comes from
18:13something called f.
18:16F is equal to the variation in mouse
18:19size explained by weight divided by the
18:22variation mouse size not explained by
18:25weight.
18:27The numerators for r squared and for f
18:31are the same.
18:33That is to say, it's the reduction in
18:35variance when we take the weight into
18:37account.
18:39The denominator is a little different.
18:42These dotted lines, the residuals,
18:45represent the variation that remains
18:48after fitting the line.
18:50This is the variation that is not
18:52explained by weight.
18:54So together,
18:55we have the variation in mouse size
18:57explained by weight divided by the
19:00variation in mouse size not explained by
19:03weight.
19:05Now let's look at the underlying
19:07mathematics.
19:08Just as a reminder, here's the equation
19:11for R squared.
19:13This is the general equation that will
19:16tell us if R squared is significant.
19:19The meat of these two equations are very
19:22similar and rely on the same sums of
19:24squares.
19:26Like we said before, the numerators are
19:29the same.
19:30In our mouse size and weight example,
19:33the numerator is the variation in mouse
19:36size explained by weight.
19:39And the sum of squares around the fit is
19:41just the residuals squared and summed up
19:45around the fitted line. So, that's the
19:47variation that the fit does not explain.
19:51These numbers over here are the degrees
19:53of freedom.
19:55They turn the sums of squares into
19:57variances.
19:59I'm going to dedicate a whole StatQuest
20:02to degrees of freedom, but for now,
20:05let's see if we can get an intuitive
20:06feel for what they're doing here.
20:10Let's start with these.
20:13P fit is the number of parameters in the
20:16fit line.
20:17Here's the equation for the fit line in
20:20a general format. We just have the Y
20:22intercept plus the slope times X.
20:26The Y intercept and the slope are two
20:29separate parameters.
20:31That means P fit equals two.
20:35P mean is the number of parameters in
20:38the mean line.
20:40In general, that equation is Y equals
20:43the Y intercept. That's what gives us a
20:46horizontal line that cuts through the
20:48data.
20:49In this case, the Y intercept is the
20:52mean value.
20:54This equation just has one parameter.
20:58Thus, P mean equals one.
21:02Both equations have a parameter for the
21:04Y intercept.
21:07However, the fit line has one extra
21:10parameter, the slope.
21:12In our example,
21:13this slope is the relationship between
21:16weight and size.
21:19In this example,
21:20P fit minus P mean equals two minus one,
21:25which equals one.
21:27The fit has one extra parameter, mouse
21:30weight.
21:32Thus,
21:33the numerator is the variance explained
21:36by the extra parameter.
21:38In our example, that's the variance in
21:40mouse size explained by mouse weight.
21:44If we had used mouse weight and tail
21:47length to explain variation in size,
21:50then we would end up with an equation
21:52that had three parameters and P fit
21:55would equal three.
21:57Thus, P fit minus P mean would equal
22:01three minus one, which equals two.
22:04Now, the fit has two extra parameters,
22:07mouse weight and tail length.
22:11With the fancier equation for the fit,
22:14the numerator is the variance in mouse
22:16size explained by mouse weight and tail
22:19length.
22:21Now, let's talk about the denominator
22:23for our equation for F.
22:26The denominator is the variation in
22:29mouse size not explained by the fit.
22:33That is to say,
22:35it's the sum of squares of the residuals
22:37that remain after we fit our new line to
22:40the data.
22:42Why divide sum of squares fit by N minus
22:46P fit instead of just N?
22:50Intuitively,
22:51the more parameters you have in your
22:53equation,
22:54the more data you need to estimate them.
22:57For example, you only need two points to
23:00estimate a line,
23:01but you need three points to estimate a
23:03plane.
23:05If the fit is good,
23:07then the variation explained by the
23:09extra parameters in the fit will be a
23:11large number, and the variation not
23:14explained by the extra parameters in the
23:16fit will be a small number.
23:18That makes F a really large number.
23:22Now, that question we've all been dying
23:24to know the answer to,
23:26how do we turn this number into a P
23:28value?
23:30Conceptually,
23:31generate a set of random data.
23:34Calculate the mean and the sum of
23:36squares around the mean.
23:39Calculate the fit and the sum of squares
23:41around the fit.
23:44Now, plug all those values into our
23:46equation for F,
23:48and that will give us a number. In this
23:50case, that number is two.
23:52Now, plot that number in a histogram.
23:56Now, generate another set of random
23:58data.
23:59Calculate the mean and the sum of
24:01squares around the mean.
24:03Then calculate the fit and the sum of
24:06squares around the fit.
24:08Plug those values into our equation for
24:10F,
24:12and in this case, we get F equals three.
24:15So, we then plug that value into our
24:17histogram.
24:19And then we repeat with yet another set
24:22of random data.
24:23In this case, we got F equals one.
24:26That's plotted on our histogram.
24:28And we just keep generating more and
24:30more random data sets, calculating the
24:33sums of squares, plugging them into our
24:35equation for F, and plotting the results
24:38on our histogram.
24:40Now, imagine we did that hundreds, if
24:43not millions, of times.
24:46When we're all done with our random data
24:47sets, we return to our original data
24:50set.
24:52We then plug the numbers into our
24:54equation for F. In this case, we got F =
24:576.
24:59The P value is the number of more
25:02extreme values divided by all of the
25:05values.
25:06So, in this case, we have the value at F
25:09= 6 and the value at F = 7 divided by
25:13all the other randomizations that we
25:15created originally.
25:17If this concept is confusing to you, I
25:20have a StatQuest that explains P values,
25:22so check that one out.
25:25Bam!
25:27You can approximate the histogram with a
25:29line.
25:30In practice, rather than generating tons
25:33of random data sets, people use the line
25:35to calculate the P value.
25:38Here's an example of a one standard F
25:41distribution that people use to
25:43calculate P values.
25:45The degrees of freedom determine the
25:47shape.
25:49The red line represents another standard
25:52F distribution that people use to
25:54calculate P values.
25:56In this case, the sample size used to
25:59draw the red line is smaller than the
26:01sample size used to draw the blue line.
26:05Notice that when N minus P fit equals
26:0810, the distribution tapers off faster.
26:13This means that the P value will be
26:16smaller when there are more samples
26:18relative to the number of parameters in
26:20the fit equation.
26:22Triple bam!
26:24Hooray! We finally got our P value. Now,
26:27let's review the main ideas.
26:31Given some data that you think are
26:32related,
26:34linear regression quantifies the
26:37relationship in the data.
26:39This is R squared.
26:41This needs to be large.
26:44It also determines how reliable that
26:46relationship is.
26:48This is the P value that we calculated
26:50with F.
26:51This needs be small.
26:53You need both to have an interesting
26:55result.
26:57Hooray! We've made it to the end of
27:00another exciting StatQuest. Wow, this
27:03was a long one.
27:04I hope you had a good time.
27:06If you like this and want to see more
27:08StatQuests like it, why don't you
27:10subscribe to my channel? It's real easy,
27:12just click the red button.
27:14And if you have any ideas of StatQuests
27:17that you'd like me to create, just put
27:19them in the comments below. That's all
27:21there is to it. All right. Tune in next
27:23time for another really exciting
27:25StatQuest.

About this transcript

This page contains the full transcript of Linear Regression, Clearly Explained!!! by StatQuest with Josh Starmer, generated from the public captions YouTube serves with the video. The transcript has 3,687 words across 647 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.