How Random Forest Performs So Well? Bias Variance Trade-Off in Random Forest — Transcript
Full transcript
- 0:00So guys, so far we have taken a basic
- 0:02intuition of Random Forest and how the
- 0:04algorithm works. I also showed you a
- 0:07step-by-step example. In this video, we
- 0:10will see why Random Forest is able to
- 0:12work so well. Why its performance is
- 0:15better compared to other algorithms.
- 0:18The reason for that lies in the
- 0:19bias-variance tradeoff. I hope you have
- 0:22read about this topic by now. If you
- 0:24haven't, I will give you a short
- 0:26summary of the entire topic. There are
- 0:28two very important things in machine
- 0:30learning: bias and variance. Basically,
- 0:33in machine learning, you get errors.
- 0:35Your machine learning models produce
- 0:38errors. Right? So, that error is
- 0:40composed of two things. One is bias and
- 0:42the other is variance. Right? What is
- 0:44bias? Bias is what we call it when a
- 0:47machine learning model cannot provide
- 0:49good results on training data. Right?
- 0:52Meaning, you have a machine learning
- 0:54model. It didn't learn very well during
- 0:56the training process itself. Meaning,
- 0:58it won't give good results on the
- 1:00training data. This means its bias will
- 1:02be high. Right? The second is variance.
- 1:05What is variance? Sometimes your
- 1:08machine learning model learns so well
- 1:10on the training data that it memorizes
- 1:12it. Basically, it memorizes the
- 1:14patterns of the training data. Due to
- 1:16which, when you give it new data, its
- 1:18performance isn't that good. So
- 1:20basically, if you make even slight
- 1:22changes to the data, your machine
- 1:23learning model's outcome won't be good.
- 1:25This is called high variance. Right? So
- 1:28, in machine learning, you must always
- 1:31try to keep your bias low. I will write
- 1:33it here. You always want low bias and
- 1:37low variance. But there is a problem.
- 1:40The problem is that there is an inverse
- 1:43relationship between these two, between
- 1:47bias and variance. Meaning, if you try
- 1:50to reduce one, the other increases.
- 1:52Right? And that is why most machine
- 1:55learning algorithms struggle with the
- 1:57bias-variance tradeoff. When you go to
- 1:59reduce one error, the other one
- 2:00increases. That is why most of the
- 2:02machine learning algorithms you have
- 2:04are defined into two classes. Right?
- 2:07The first class is low-bias,
- 2:09high-variance algorithms. Meaning, an
- 2:13algorithm that is able to give very
- 2:15good results on the training data. But
- 2:18is not able to provide as good results
- 2:20on new test data. An example of these
- 2:23algorithms is a fully grown decision
- 2:25tree. A decision tree where you don't
- 2:28cap the maximum depth. Right? A fully
- 2:31grown decision tree. Second, SVMs
- 2:35sometimes show this behavior, as do
- 2:38KNNs when the number of nearest
- 2:40neighbors is very low. Then there is a
- 2:43second class of algorithms where you
- 2:46have high bias but low variance. A very
- 2:48good example of this is linear
- 2:50regression and, in certain cases,
- 2:52logistic regression as well. But you
- 2:55never really get the combination you
- 2:57want. Where both the bias and the
- 3:00variance are low. This is where Random
- 3:03Forest helps you. The specialty of
- 3:07Random Forest is that if you give it a
- 3:11low-bias, high-variance algorithm, it
- 3:14converts it into a low-bias,
- 3:17low-variance algorithm. This means that
- 3:23for such algorithms, Random Forest
- 3:25keeps the bias as it is while reducing
- 3:28the variance. Right? Now let's
- 3:31understand how this happens. Right? So
- 3:34the answer lies in its working. Right?
- 3:37I had explained to you how it works.
- 3:39What do you do? You take some base
- 3:42models, and all these base models are
- 3:44actually decision trees, and these are
- 3:45fully grown decision trees. Since they
- 3:48are fully grown decision trees, each
- 3:51one has low bias and high variance.
- 3:53Here, every model has low bias. High
- 3:55variance. It won't make any mistakes on
- 3:57the training data. It will make
- 3:58mistakes on new data. Right? Now what
- 4:01do you do? You take your data. This is
- 4:04your data, and you feed a subset of it
- 4:08to each algorithm. Suppose if there
- 4:13were 10k rows, 10,000 rows, then maybe
- 4:16you are giving 1k rows to everyone.
- 4:19Meaning you are giving 1000 rows. Now
- 4:21these models are trained. Now, since
- 4:24these are internally decision trees.
- 4:27The outcome will still remain low bias.
- 4:32But let's think about what happens to
- 4:35the variance. So something interesting
- 4:37happens with the variance. I will give
- 4:39you an example. Suppose what did you do
- 4:40? You replaced 100 points in these 10k
- 4:45points. And how did you replace the
- 4:49points? With points that are very
- 4:51different, very distinct, noise,
- 4:53outliers. Right? Meaning you made
- 4:55changes to the data. Now obviously, if
- 4:58you had made changes to the data and
- 5:00had a single decision tree, the
- 5:02variance would have remained high. But
- 5:04since you are using Random Forest, you
- 5:06are using a collection of trees. Now
- 5:08what will happen is, since you are
- 5:11sending the data randomly, it will
- 5:13never happen that all 100 noisy points
- 5:16go to a single decision tree. Because
- 5:19you are drawing them randomly. So it is
- 5:21possible that the first one gets 10
- 5:23noisy points. The second one gets five.
- 5:26The third one gets none. The next one
- 5:29gets 15. So, the impact of 100 points
- 5:32that a single decision tree had to
- 5:34endure earlier. Now, no decision tree
- 5:37has to endure that. So, the impact of
- 5:40those outliers, the impact of noisy
- 5:42points, is now distributed and has
- 5:43become weak due to that distribution.
- 5:46So, that is why there won't be much
- 5:48difference between the result we were
- 5:50getting earlier and the result we are
- 5:53getting after introducing these points.
- 5:55And not having much of a difference
- 5:57means you have reduced the variance,
- 6:00reduced the variety. And because of
- 6:03this, the overall model you create now
- 6:07will have low bias—sorry—low
- 6:10variance. So, in short, Random Forest
- 6:14performs well because it solves a major
- 6:16problem for you, the bias-variance
- 6:19tradeoff, which says that you cannot
- 6:21reduce both bias and variance because
- 6:23there is an inverse relationship
- 6:25between them. If you reduce bias,
- 6:28variance will increase. If you reduce
- 6:29variance, bias will increase. But by
- 6:31using this technique, you are able to
- 6:34keep the bias low while also lowering
- 6:37the variance. I hope you are
- 6:38understanding this entire thing. All
- 6:40right? Now I will show you an exact
- 6:43example of this. By plotting it through
- 6:46a graph. I have a dataset that I have
- 6:50generated myself using the make_circles
- 6:55function. And Let me show you first.
- 6:59This dataset has 500 rows and two
- 7:03columns. And this dataset looks like
- 7:05this. First of all, it is a
- 7:07classification dataset. Meaning it has
- 7:09two classes. Yellow class, purple class
- 7:11. And this data is somewhat like this,
- 7:13of a concentric circle type. All right?
- 7:16Now what will I do? I will first apply
- 7:18a decision tree to it, and I will give
- 7:20the decision tree the task to classify
- 7:22between both labels. To create a
- 7:24decision boundary. And then I will have
- 7:26the same task done by Random Forest and
- 7:28show you the difference. All right? So,
- 7:30first of all, we imported a decision
- 7:33tree algorithm. Created its object, and
- 7:36after creating the object, what did I
- 7:39do? Plotted the decision boundary. And
- 7:42this is a fully grown decision tree.
- 7:44This is a fully grown decision tree.
- 7:47And you can see by looking at this
- 7:49graph that this algorithm here has low
- 7:52bias and high variance. How? Look, the
- 7:56regions you can see where these long
- 7:59yellow lines are forming, that shows
- 8:01that 'this algorithm is overfitting'.
- 8:05Overfitting in the sense that it has
- 8:06low bias and high variance. Now look at
- 8:09this, this yellow region here, it is
- 8:11yellow only because there was a single
- 8:14yellow point here. Similarly, if you
- 8:16look at this yellow region, this yellow
- 8:18region exists only because there is a
- 8:20yellow point here. A single yellow
- 8:23point, but you can clearly see that the
- 8:26region should have been purple because
- 8:28all the other points in that region are
- 8:30purple. Okay? So, this algorithm will
- 8:34give very good results on the training
- 8:37data. Meaning, the bias will be low.
- 8:39But change this data just a little bit.
- 8:41Bring in new data that follows a
- 8:43similar pattern. Your results will
- 8:46start to become very poor. On new data,
- 8:47your results will start to vary.
- 8:49Meaning, the variance is high. Okay?
- 8:52And that is the problem with fully
- 8:54grown decision trees. They overfit.
- 8:56This is overfitting on your screen.
- 8:58Okay? Now let's apply Random Forest
- 9:00here and see what changes occur. So, I
- 9:03went below and imported the Random
- 9:05Forest Classifier. I trained 500
- 9:08decision trees inside the Random Forest
- 9:11Classifier. And as soon as I plotted
- 9:14its decision boundary, you would see
- 9:16the change. Look at this guys, this is
- 9:19the change. You can see now that by
- 9:22looking at this graph, you can
- 9:24understand that it has low bias and low
- 9:27variance. Actually, the bias has
- 9:29increased slightly. The error on the
- 9:31training data has increased a bit. You
- 9:33are now labeling these yellow points as
- 9:34purple as well. The training error will
- 9:36increase slightly. Not a lot, just a
- 9:38little. But your testing error, the
- 9:41variance, will be significantly reduced
- 9:44. These unnecessary yellow regions that
- 9:47had formed are all gone, and your model
- 9:50is now able to capture the true nature
- 9:52of the data. What is the true nature of
- 9:55the data? That yellow points are within
- 9:56purple points. It is showing exactly
- 9:58that. And that is why Random Forest is
- 10:01such a great algorithm. Because it
- 10:03converts a low-bias, high-variance
- 10:05algorithm into a low-bias, low-variance
- 10:08algorithm. Okay? I can show you this
- 10:11same example for regression as well. So
- 10:13, I generated a data point. This is the
- 10:19data, by the way. So, you can see these
- 10:23are the points. There is only one input
- 10:25column here and one output column. This
- 10:27is a regression problem. The red curve
- 10:29that you can see here. That is the true
- 10:32pattern of this data. The true nature.
- 10:34Using this very curve, I have generated
- 10:36this data. Basically, by adding random
- 10:38noise. Okay? Now, what do I need to do?
- 10:40I need to apply regression. So, first,
- 10:43let me show you by applying a Decision
- 10:45Tree Regressor. And this is the
- 10:47Decision Tree Regressor I applied. And
- 10:50you can see that the green curve here,
- 10:53which is fluctuating a lot. This is our
- 10:57Decision Tree Regressor trying to find
- 10:59the true nature of this data. But you
- 11:02can see it's not performing very well.
- 11:05I mean, you can clearly see that it's
- 11:07trying to touch every single point.
- 11:09Meaning, it is trying to reduce the
- 11:11training error. Meaning, it is trying
- 11:13to keep the bias low. But along the way
- 11:15, the mistake it is making is that it
- 11:17is going to touch the extreme points
- 11:19too. And because of that, when new
- 11:20points arrive, it will start making
- 11:22mistakes. Meaning, the variance will
- 11:24become high. Like, look at this point.
- 11:26Now, this point is clearly an outlier.
- 11:28It's a noisy point. But your model went
- 11:31to touch that too. It went passing
- 11:33right by it. Similarly, look at all
- 11:35these points above. Look at this point.
- 11:37It missed this point, but look at the
- 11:39points below. So, in short, the
- 11:41training error is low. The testing
- 11:43error is high. Okay? Now, if we do the
- 11:46same thing with Random Forest, and here
- 11:48I have trained 1000 decision trees
- 11:50together. And look at this, guys, there
- 11:53is some smoothing. Okay? And now your
- 11:55green curve is passing much closer to
- 11:58the red one. Whereas if you look here,
- 12:00your green curve is deviating quite a
- 12:03lot from the red. Because of which,
- 12:05your mean squared error is also quite
- 12:07high, around 22.7. Okay? Whereas here,
- 12:10notice that your green curve, first of
- 12:12all, is not touching the extreme points
- 12:14. It stops. Okay? And it's trying to
- 12:17stay around the second red curve. So,
- 12:20clearly you can see Random Forest is
- 12:22much better in comparison to Decision
- 12:25Trees. In fact, you can even quantify
- 12:27it. Here you can see that your mean
- 12:30squared error is only 15 in comparison
- 12:32to 22, which was the case for the
- 12:34Decision Tree. So, the lower the mean
- 12:36squared error, the better. Right? So
- 12:39yeah guys, this is what I wanted to
- 12:41tell you in this video: why Random
- 12:44Forests work so well. What is the
- 12:46reason for them working so well? I hope
- 12:48I was able to explain it to you.
- 12:49Alright? Thanks for watching.
About this transcript
This page contains the full transcript of How Random Forest Performs So Well? Bias Variance Trade-Off in Random Forest by CampusX, generated from the public captions YouTube serves with the video. The transcript has 2,077 words across 309 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.