Machine Learning Fundamentals: Bias and Variance — Transcript

Full transcript

1. 0:00>> Hurricane Florence came
2. 0:03by while I was working on StatQuest.
3. 0:07Dark clouds filled the sky,
4. 0:10but that didn't stop StatQuest.
5. 0:14StatQuest. Hello/ I'm Josh Starmer
6. 0:19and welcome to StatQuest.
7. 0:21Today we're going to be talking about
8. 0:22some machine learning fundamentals,
9. 0:24bias, and variance
10. 0:26and they're going to be clearly explained.
11. 0:28Imagine we measured the weight and height of
12. 0:32a bunch of mice and
13. 0:33plotted the data on a graph.
14. 0:35Light mice tend to be
15. 0:37short and heavier mice tend to be taller.
16. 0:42But after a certain weight,
17. 0:44mice don't get any taller, just more obese.
18. 0:47Given this data, we would like to
19. 0:50predict mouse height given its weight.
20. 0:52For example, if you
21. 0:54told me your mouse weighed this much,
22. 0:57then we might predict that
23. 0:59the mouse is this tall.
24. 1:01Ideally, we would know
25. 1:03the exact mathematical formula that
26. 1:05describes the relationship
27. 1:06between weight and height.
28. 1:08But in this case,
29. 1:10we don't know the formula,
30. 1:12so we're going to use
31. 1:13two machine learning methods
32. 1:14to approximate this relationship.
33. 1:17However, I'll leave
34. 1:19the true relationship curve
35. 1:21in the figure for reference.
36. 1:23The first thing we do is
37. 1:25split the data into two sets,
38. 1:26one for training
39. 1:27the machine learning algorithms
40. 1:29and one for testing them.
41. 1:31The blue dots are the training set
42. 1:34and the green dots are the testing set.
43. 1:37Here is just the training set.
44. 1:40The first machine learning algorithm that
45. 1:43we will use is linear regression,
46. 1:45AKA, least squares.
47. 1:48Linear regression fits
48. 1:50a straight line to the training set.
49. 1:52Note. The straight line
50. 1:55doesn't have the flexibility
51. 1:56to accurately replicate the arc
52. 1:58in the true relationship.
53. 2:00No matter how we try to fit
54. 2:02the line, it will never curve.
55. 2:06Thus, the straight line will never
56. 2:09capture the true relationship
57. 2:11between weight and height,
58. 2:12no matter how well we
59. 2:13fit it to the training set.
60. 2:15The inability for a
61. 2:17machine learning method like
62. 2:19linear regression to capture
63. 2:21the true relationship is called bias.
64. 2:24Because the straight line can't
65. 2:26be curved like the true relationship,
66. 2:29it has a relatively large amount of bias.
67. 2:32Another machine learning method might
68. 2:35fit a squiggly line to the training set.
69. 2:38The squiggly line is super flexible and
70. 2:41hugs the training set along
71. 2:42the arc of the true relationship.
72. 2:44Because the squiggly line can handle
73. 2:47the arc in the true relationship
74. 2:49between weight and height,
75. 2:50it has very little bias.
76. 2:52We can compare how well the straight line and
77. 2:56the squiggly line fit
78. 2:57the training set by
79. 2:58calculating their sums of squares.
80. 3:01In other words, we measure
81. 3:03the distances from the fit lines to the data,
82. 3:06square them, and add them up.
83. 3:09They are squared so that
84. 3:11negative distances do not
85. 3:13cancel out positive distances.
86. 3:15Notice how the squiggly line
87. 3:18fits the data so well
88. 3:19that the distances between
89. 3:21the line and the data are all zero.
90. 3:24In the contest to see
91. 3:26whether the straight line
92. 3:27fits the training set
93. 3:29better than the squiggly line,
94. 3:30the squiggly line wins.
95. 3:33But remember, so far we've only
96. 3:36calculated the sums of
97. 3:38squares for the training set.
98. 3:40We also have a testing set.
99. 3:43Now let's calculate the sums
100. 3:45of squares for the testing set.
101. 3:47In the contest to
102. 3:49see whether the straight line
103. 3:50fits the testing set
104. 3:52better than the squiggly line,
105. 3:54the straight line wins.
106. 3:57Even though the squiggly line did
107. 3:59a great job fitting the training set,
108. 4:02it did a terrible job
109. 4:04fitting the testing set.
110. 4:06In machine learning lingo,
111. 4:08the difference in fits between
112. 4:10datasets is called variance.
113. 4:12The squiggly line has low bias,
114. 4:15since it is flexible and can adapt to
115. 4:17the curve in the relationship
116. 4:19between weight and height.
117. 4:21But the squiggly line has
118. 4:23high variability because it
119. 4:25results in vastly different sums
120. 4:26of squares for different datasets.
121. 4:29In other words, it's hard to predict how well
122. 4:32the squiggly line will perform
123. 4:34with future datasets.
124. 4:35It might do well sometimes
125. 4:37and other times it might do terribly.
126. 4:41In contrast, the straight line
127. 4:43has relatively high bias,
128. 4:45since it cannot capture
129. 4:47the curve in the relationship
130. 4:48between weight and height.
131. 4:50But the straight line
132. 4:52has relatively low variance
133. 4:54because the sums of squares are
134. 4:55very similar for different datasets.
135. 4:58In other words, the straight line might only
136. 5:01give good predictions and
137. 5:03not great predictions,
138. 5:04but they will be consistently
139. 5:06good predictions.
140. 5:08Bam. Oh, no.
141. 5:11Terminology alert.
142. 5:13Because the squiggly line
143. 5:14fits the training set really well,
144. 5:16but not the testing set,
145. 5:18we say that the squiggly line is overfit.
146. 5:22In machine learning, the ideal algorithm has
147. 5:25low bias and can
148. 5:26accurately model the true relationship.
149. 5:29And it has low variability by
150. 5:32producing consistent predictions
151. 5:34across different datasets.
152. 5:36This is done by finding
153. 5:38the sweet spot between
154. 5:40a simple model and a complex model.
155. 5:44Oh, no. Another terminology alert.
156. 5:47Three commonly used methods
157. 5:49for finding the sweet spot between
158. 5:51simple and complicated models
159. 5:53are regularization,
160. 5:55boosting, and bagging.
161. 5:58The StatQuest on a random forest show
162. 6:01an example of bagging in action.
163. 6:03And we'll talk about regularization
164. 6:06and boosting in future StatQuests.
165. 6:08Double bam. Hooray. We've made
166. 6:13it to the end of another exciting StatQuest.
167. 6:15If you like this StatQuest and
168. 6:17want to see more, please subscribe.
169. 6:19And if you want to support StatQuest,
170. 6:21please consider buying one
171. 6:23or two of my original songs.
172. 6:25All right. Until next time, quest on.