YouTube2Text

How Random Forest Performs So Well? Bias Variance Trade-Off in Random Forest — Transcript

by CampusX · 2,077 words · 309 segments · language en · Watch on YouTube

Full transcript

  1. 0:00So guys, so far we have taken a basic
  2. 0:02intuition of Random Forest and how the
  3. 0:04algorithm works. I also showed you a
  4. 0:07step-by-step example. In this video, we
  5. 0:10will see why Random Forest is able to
  6. 0:12work so well. Why its performance is
  7. 0:15better compared to other algorithms.
  8. 0:18The reason for that lies in the
  9. 0:19bias-variance tradeoff. I hope you have
  10. 0:22read about this topic by now. If you
  11. 0:24haven't, I will give you a short
  12. 0:26summary of the entire topic. There are
  13. 0:28two very important things in machine
  14. 0:30learning: bias and variance. Basically,
  15. 0:33in machine learning, you get errors.
  16. 0:35Your machine learning models produce
  17. 0:38errors. Right? So, that error is
  18. 0:40composed of two things. One is bias and
  19. 0:42the other is variance. Right? What is
  20. 0:44bias? Bias is what we call it when a
  21. 0:47machine learning model cannot provide
  22. 0:49good results on training data. Right?
  23. 0:52Meaning, you have a machine learning
  24. 0:54model. It didn't learn very well during
  25. 0:56the training process itself. Meaning,
  26. 0:58it won't give good results on the
  27. 1:00training data. This means its bias will
  28. 1:02be high. Right? The second is variance.
  29. 1:05What is variance? Sometimes your
  30. 1:08machine learning model learns so well
  31. 1:10on the training data that it memorizes
  32. 1:12it. Basically, it memorizes the
  33. 1:14patterns of the training data. Due to
  34. 1:16which, when you give it new data, its
  35. 1:18performance isn't that good. So
  36. 1:20basically, if you make even slight
  37. 1:22changes to the data, your machine
  38. 1:23learning model's outcome won't be good.
  39. 1:25This is called high variance. Right? So
  40. 1:28, in machine learning, you must always
  41. 1:31try to keep your bias low. I will write
  42. 1:33it here. You always want low bias and
  43. 1:37low variance. But there is a problem.
  44. 1:40The problem is that there is an inverse
  45. 1:43relationship between these two, between
  46. 1:47bias and variance. Meaning, if you try
  47. 1:50to reduce one, the other increases.
  48. 1:52Right? And that is why most machine
  49. 1:55learning algorithms struggle with the
  50. 1:57bias-variance tradeoff. When you go to
  51. 1:59reduce one error, the other one
  52. 2:00increases. That is why most of the
  53. 2:02machine learning algorithms you have
  54. 2:04are defined into two classes. Right?
  55. 2:07The first class is low-bias,
  56. 2:09high-variance algorithms. Meaning, an
  57. 2:13algorithm that is able to give very
  58. 2:15good results on the training data. But
  59. 2:18is not able to provide as good results
  60. 2:20on new test data. An example of these
  61. 2:23algorithms is a fully grown decision
  62. 2:25tree. A decision tree where you don't
  63. 2:28cap the maximum depth. Right? A fully
  64. 2:31grown decision tree. Second, SVMs
  65. 2:35sometimes show this behavior, as do
  66. 2:38KNNs when the number of nearest
  67. 2:40neighbors is very low. Then there is a
  68. 2:43second class of algorithms where you
  69. 2:46have high bias but low variance. A very
  70. 2:48good example of this is linear
  71. 2:50regression and, in certain cases,
  72. 2:52logistic regression as well. But you
  73. 2:55never really get the combination you
  74. 2:57want. Where both the bias and the
  75. 3:00variance are low. This is where Random
  76. 3:03Forest helps you. The specialty of
  77. 3:07Random Forest is that if you give it a
  78. 3:11low-bias, high-variance algorithm, it
  79. 3:14converts it into a low-bias,
  80. 3:17low-variance algorithm. This means that
  81. 3:23for such algorithms, Random Forest
  82. 3:25keeps the bias as it is while reducing
  83. 3:28the variance. Right? Now let's
  84. 3:31understand how this happens. Right? So
  85. 3:34the answer lies in its working. Right?
  86. 3:37I had explained to you how it works.
  87. 3:39What do you do? You take some base
  88. 3:42models, and all these base models are
  89. 3:44actually decision trees, and these are
  90. 3:45fully grown decision trees. Since they
  91. 3:48are fully grown decision trees, each
  92. 3:51one has low bias and high variance.
  93. 3:53Here, every model has low bias. High
  94. 3:55variance. It won't make any mistakes on
  95. 3:57the training data. It will make
  96. 3:58mistakes on new data. Right? Now what
  97. 4:01do you do? You take your data. This is
  98. 4:04your data, and you feed a subset of it
  99. 4:08to each algorithm. Suppose if there
  100. 4:13were 10k rows, 10,000 rows, then maybe
  101. 4:16you are giving 1k rows to everyone.
  102. 4:19Meaning you are giving 1000 rows. Now
  103. 4:21these models are trained. Now, since
  104. 4:24these are internally decision trees.
  105. 4:27The outcome will still remain low bias.
  106. 4:32But let's think about what happens to
  107. 4:35the variance. So something interesting
  108. 4:37happens with the variance. I will give
  109. 4:39you an example. Suppose what did you do
  110. 4:40? You replaced 100 points in these 10k
  111. 4:45points. And how did you replace the
  112. 4:49points? With points that are very
  113. 4:51different, very distinct, noise,
  114. 4:53outliers. Right? Meaning you made
  115. 4:55changes to the data. Now obviously, if
  116. 4:58you had made changes to the data and
  117. 5:00had a single decision tree, the
  118. 5:02variance would have remained high. But
  119. 5:04since you are using Random Forest, you
  120. 5:06are using a collection of trees. Now
  121. 5:08what will happen is, since you are
  122. 5:11sending the data randomly, it will
  123. 5:13never happen that all 100 noisy points
  124. 5:16go to a single decision tree. Because
  125. 5:19you are drawing them randomly. So it is
  126. 5:21possible that the first one gets 10
  127. 5:23noisy points. The second one gets five.
  128. 5:26The third one gets none. The next one
  129. 5:29gets 15. So, the impact of 100 points
  130. 5:32that a single decision tree had to
  131. 5:34endure earlier. Now, no decision tree
  132. 5:37has to endure that. So, the impact of
  133. 5:40those outliers, the impact of noisy
  134. 5:42points, is now distributed and has
  135. 5:43become weak due to that distribution.
  136. 5:46So, that is why there won't be much
  137. 5:48difference between the result we were
  138. 5:50getting earlier and the result we are
  139. 5:53getting after introducing these points.
  140. 5:55And not having much of a difference
  141. 5:57means you have reduced the variance,
  142. 6:00reduced the variety. And because of
  143. 6:03this, the overall model you create now
  144. 6:07will have low bias—sorry—low
  145. 6:10variance. So, in short, Random Forest
  146. 6:14performs well because it solves a major
  147. 6:16problem for you, the bias-variance
  148. 6:19tradeoff, which says that you cannot
  149. 6:21reduce both bias and variance because
  150. 6:23there is an inverse relationship
  151. 6:25between them. If you reduce bias,
  152. 6:28variance will increase. If you reduce
  153. 6:29variance, bias will increase. But by
  154. 6:31using this technique, you are able to
  155. 6:34keep the bias low while also lowering
  156. 6:37the variance. I hope you are
  157. 6:38understanding this entire thing. All
  158. 6:40right? Now I will show you an exact
  159. 6:43example of this. By plotting it through
  160. 6:46a graph. I have a dataset that I have
  161. 6:50generated myself using the make_circles
  162. 6:55function. And Let me show you first.
  163. 6:59This dataset has 500 rows and two
  164. 7:03columns. And this dataset looks like
  165. 7:05this. First of all, it is a
  166. 7:07classification dataset. Meaning it has
  167. 7:09two classes. Yellow class, purple class
  168. 7:11. And this data is somewhat like this,
  169. 7:13of a concentric circle type. All right?
  170. 7:16Now what will I do? I will first apply
  171. 7:18a decision tree to it, and I will give
  172. 7:20the decision tree the task to classify
  173. 7:22between both labels. To create a
  174. 7:24decision boundary. And then I will have
  175. 7:26the same task done by Random Forest and
  176. 7:28show you the difference. All right? So,
  177. 7:30first of all, we imported a decision
  178. 7:33tree algorithm. Created its object, and
  179. 7:36after creating the object, what did I
  180. 7:39do? Plotted the decision boundary. And
  181. 7:42this is a fully grown decision tree.
  182. 7:44This is a fully grown decision tree.
  183. 7:47And you can see by looking at this
  184. 7:49graph that this algorithm here has low
  185. 7:52bias and high variance. How? Look, the
  186. 7:56regions you can see where these long
  187. 7:59yellow lines are forming, that shows
  188. 8:01that 'this algorithm is overfitting'.
  189. 8:05Overfitting in the sense that it has
  190. 8:06low bias and high variance. Now look at
  191. 8:09this, this yellow region here, it is
  192. 8:11yellow only because there was a single
  193. 8:14yellow point here. Similarly, if you
  194. 8:16look at this yellow region, this yellow
  195. 8:18region exists only because there is a
  196. 8:20yellow point here. A single yellow
  197. 8:23point, but you can clearly see that the
  198. 8:26region should have been purple because
  199. 8:28all the other points in that region are
  200. 8:30purple. Okay? So, this algorithm will
  201. 8:34give very good results on the training
  202. 8:37data. Meaning, the bias will be low.
  203. 8:39But change this data just a little bit.
  204. 8:41Bring in new data that follows a
  205. 8:43similar pattern. Your results will
  206. 8:46start to become very poor. On new data,
  207. 8:47your results will start to vary.
  208. 8:49Meaning, the variance is high. Okay?
  209. 8:52And that is the problem with fully
  210. 8:54grown decision trees. They overfit.
  211. 8:56This is overfitting on your screen.
  212. 8:58Okay? Now let's apply Random Forest
  213. 9:00here and see what changes occur. So, I
  214. 9:03went below and imported the Random
  215. 9:05Forest Classifier. I trained 500
  216. 9:08decision trees inside the Random Forest
  217. 9:11Classifier. And as soon as I plotted
  218. 9:14its decision boundary, you would see
  219. 9:16the change. Look at this guys, this is
  220. 9:19the change. You can see now that by
  221. 9:22looking at this graph, you can
  222. 9:24understand that it has low bias and low
  223. 9:27variance. Actually, the bias has
  224. 9:29increased slightly. The error on the
  225. 9:31training data has increased a bit. You
  226. 9:33are now labeling these yellow points as
  227. 9:34purple as well. The training error will
  228. 9:36increase slightly. Not a lot, just a
  229. 9:38little. But your testing error, the
  230. 9:41variance, will be significantly reduced
  231. 9:44. These unnecessary yellow regions that
  232. 9:47had formed are all gone, and your model
  233. 9:50is now able to capture the true nature
  234. 9:52of the data. What is the true nature of
  235. 9:55the data? That yellow points are within
  236. 9:56purple points. It is showing exactly
  237. 9:58that. And that is why Random Forest is
  238. 10:01such a great algorithm. Because it
  239. 10:03converts a low-bias, high-variance
  240. 10:05algorithm into a low-bias, low-variance
  241. 10:08algorithm. Okay? I can show you this
  242. 10:11same example for regression as well. So
  243. 10:13, I generated a data point. This is the
  244. 10:19data, by the way. So, you can see these
  245. 10:23are the points. There is only one input
  246. 10:25column here and one output column. This
  247. 10:27is a regression problem. The red curve
  248. 10:29that you can see here. That is the true
  249. 10:32pattern of this data. The true nature.
  250. 10:34Using this very curve, I have generated
  251. 10:36this data. Basically, by adding random
  252. 10:38noise. Okay? Now, what do I need to do?
  253. 10:40I need to apply regression. So, first,
  254. 10:43let me show you by applying a Decision
  255. 10:45Tree Regressor. And this is the
  256. 10:47Decision Tree Regressor I applied. And
  257. 10:50you can see that the green curve here,
  258. 10:53which is fluctuating a lot. This is our
  259. 10:57Decision Tree Regressor trying to find
  260. 10:59the true nature of this data. But you
  261. 11:02can see it's not performing very well.
  262. 11:05I mean, you can clearly see that it's
  263. 11:07trying to touch every single point.
  264. 11:09Meaning, it is trying to reduce the
  265. 11:11training error. Meaning, it is trying
  266. 11:13to keep the bias low. But along the way
  267. 11:15, the mistake it is making is that it
  268. 11:17is going to touch the extreme points
  269. 11:19too. And because of that, when new
  270. 11:20points arrive, it will start making
  271. 11:22mistakes. Meaning, the variance will
  272. 11:24become high. Like, look at this point.
  273. 11:26Now, this point is clearly an outlier.
  274. 11:28It's a noisy point. But your model went
  275. 11:31to touch that too. It went passing
  276. 11:33right by it. Similarly, look at all
  277. 11:35these points above. Look at this point.
  278. 11:37It missed this point, but look at the
  279. 11:39points below. So, in short, the
  280. 11:41training error is low. The testing
  281. 11:43error is high. Okay? Now, if we do the
  282. 11:46same thing with Random Forest, and here
  283. 11:48I have trained 1000 decision trees
  284. 11:50together. And look at this, guys, there
  285. 11:53is some smoothing. Okay? And now your
  286. 11:55green curve is passing much closer to
  287. 11:58the red one. Whereas if you look here,
  288. 12:00your green curve is deviating quite a
  289. 12:03lot from the red. Because of which,
  290. 12:05your mean squared error is also quite
  291. 12:07high, around 22.7. Okay? Whereas here,
  292. 12:10notice that your green curve, first of
  293. 12:12all, is not touching the extreme points
  294. 12:14. It stops. Okay? And it's trying to
  295. 12:17stay around the second red curve. So,
  296. 12:20clearly you can see Random Forest is
  297. 12:22much better in comparison to Decision
  298. 12:25Trees. In fact, you can even quantify
  299. 12:27it. Here you can see that your mean
  300. 12:30squared error is only 15 in comparison
  301. 12:32to 22, which was the case for the
  302. 12:34Decision Tree. So, the lower the mean
  303. 12:36squared error, the better. Right? So
  304. 12:39yeah guys, this is what I wanted to
  305. 12:41tell you in this video: why Random
  306. 12:44Forests work so well. What is the
  307. 12:46reason for them working so well? I hope
  308. 12:48I was able to explain it to you.
  309. 12:49Alright? Thanks for watching.

About this transcript

This page contains the full transcript of How Random Forest Performs So Well? Bias Variance Trade-Off in Random Forest by CampusX, generated from the public captions YouTube serves with the video. The transcript has 2,077 words across 309 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.