Introduction to Ensemble Learning | Ensemble Techniques in Machine Learning — Transcript
Full transcript
- 0:00Hello guys, welcome to my YouTube
- 0:02channel. We studied Decision Trees last
- 0:04time, and now we are going to start
- 0:06something very important in Machine
- 0:08Learning called Ensemble Learning. It
- 0:11is so useful that whenever you do any
- 0:13machine learning project in a company
- 0:16or a Kaggle competition, it is
- 0:18mandatory that you try Ensemble
- 0:20Learning there as well. Okay? Now,
- 0:24there are many algorithms in Ensemble
- 0:25Learning. We will cover all these
- 0:27algorithms one by one. But in this
- 0:29video, I will just tell you about
- 0:31Ensemble Learning. I will provide an
- 0:33introduction to it. I will tell you
- 0:35which algorithms are included in it.
- 0:37And I will also give an introduction to
- 0:38how those algorithms work roughly. Okay
- 0:41? So let's start with the word '
- 0:43Ensemble' first. What does the word '
- 0:45Ensemble' mean? If you Google the
- 0:48meaning of 'Ensemble', you will get the
- 0:51answer that an ensemble is a group of
- 0:53artists or a group of something. In
- 0:57machine learning, when you talk about
- 1:00ensemble learning, it signifies a
- 1:02collection of multiple machine learning
- 1:05models. So if you combine different
- 1:08machine learning models and then create
- 1:11a bigger model from them, that
- 1:13technique is called Ensemble Learning.
- 1:16Okay? So in this video, we will first
- 1:19understand how Ensemble Learning works.
- 1:22Why does it work? Meaning, why does it
- 1:24perform so well? Where is it useful and
- 1:27things like that, okay; and in fact,
- 1:30the types of Ensemble Learning. But
- 1:33first, I want to explain a concept that
- 1:35we call the 'Wisdom of the Crowd'. If
- 1:39you want to understand Ensemble
- 1:40Learning, you must first have an idea
- 1:42of what is meant by the 'Wisdom of the
- 1:44Crowd'. So let's start with that topic.
- 1:47I guess you might have understood a
- 1:48little bit from the name itself what
- 1:50the 'Wisdom of the Crowd' means. In
- 1:51Hindi, it would be 'the intelligence of
- 1:53the crowd'. Okay? So let me tell you
- 1:56from my personal experience when I felt
- 1:58the 'Wisdom of the Crowd'. When I was
- 2:01young, I used to watch KBC (Kaun Banega
- 2:03Crorepati) and if you remember, there
- 2:06used to be a lifeline called 'Audience
- 2:08Poll', right? And I don't know,
- 2:12whenever I noticed it, whenever I
- 2:14watched KBC, most of the time when
- 2:16participants took the audience poll,
- 2:18they got the right answer. It meant
- 2:21that the audience somehow always knew
- 2:24the right answer, and it was very
- 2:26surprising to me how it could be that
- 2:29the audience always knows the answer.
- 2:32Guess what? This isn't just in KBC.
- 2:35It’s spread everywhere around us.
- 2:38Wisdom of the crowd is a widespread
- 2:40phenomenon that our lives depend on.
- 2:43Let me give you some examples. Another
- 2:46great example that you might be able to
- 2:48relate to is product reviews. So,
- 2:52nowadays when we shop on Amazon or
- 2:54Flipkart, whenever we are going to buy
- 2:56a product, how do we get the
- 2:58reliability that we should buy it or
- 3:00not? Because when buying online, you
- 3:04can't touch and see the product. You
- 3:06can't hold it in your hands. You only
- 3:08have a photo. So where does that trust
- 3:10come from? That is why websites like
- 3:13Flipkart and Amazon do what they do.
- 3:15They take reviews from customers like
- 3:17you, and all those reviews are shown to
- 3:19you when you go to buy that product. So
- 3:22let's say there is a product that
- 3:2415,000 people have reviewed and rated,
- 3:27and its rating is 4.5 out of 5. So what
- 3:30comes to my mind? It comes to my mind
- 3:33that since so many people have said
- 3:35this product is good, it must be good.
- 3:39If the rating were 4.5 and only one
- 3:41customer had reviewed it, meaning only
- 3:43one customer gave it 4.5. And no one
- 3:46else gave any review, would I feel as
- 3:48comfortable and confident about buying
- 3:51that product? The answer is no. Right?
- 3:54So, in a way, I am relying on the
- 3:56wisdom of the crowd. When one person
- 3:59said it's 4.5. My internal alarm didn't
- 4:01ring. But as soon as 15,000 people said
- 4:04it's 4.5, I thought that if so many
- 4:06people are saying it, it must be great,
- 4:08and I relied on that and bought it.
- 4:10Okay? You will find this same thing in
- 4:12other places too. When you want to
- 4:15watch a movie, you search for it on
- 4:18IMDb, and you decide whether or not to
- 4:20watch the movie based on the IMDb
- 4:23rating. But where is that rating coming
- 4:25from? Many other viewers like you went
- 4:29and watched the movie and rated it, and
- 4:31when you look at it collectively, you
- 4:34get a number like 8.5, 8.9, 6.1, and
- 4:36that number tells you how people
- 4:39collectively feel about that movie.
- 4:42Again, an example of the wisdom of the
- 4:44crowd. Okay? Furthermore, the biggest
- 4:47example is democracy. India is a
- 4:49democracy. Right? How do elections
- 4:51happen in a democracy? Everyone has the
- 4:54right to vote. Everyone's vote has
- 4:56equal value. And in the end, the party
- 4:59that gets the most votes wins. Why?
- 5:02Because we believe that when most
- 5:04people are supporting a party, then
- 5:06that party must be right. And the
- 5:09fundamental of democracy also lies in
- 5:11this very thing. Wisdom of the crowd. I
- 5:13will give you a very good example. I
- 5:15was reading a recent example in a blog.
- 5:20It was written there that some 200-300
- 5:23years ago, this incident actually
- 5:25happened where there was a fair, an
- 5:28animal was put on stage, and the host
- 5:31asked the whole crowd to guess how much
- 5:34that animal weighed. Whoever had the
- 5:38correct answer would get a prize. So,
- 5:41many people started guessing. Obviously
- 5:44, no one had the perfectly correct
- 5:46answer. But the host took all those
- 5:50answers and calculated their mean, and
- 5:53as soon as the mean was calculated, you
- 5:55would be surprised to know that the
- 5:58mean was almost, down to the decimal
- 6:00value, very close to the actual weight
- 6:03of that animal. So somehow, the crowd
- 6:07always knows the answer. And this
- 6:09phenomenon is called the wisdom of the
- 6:11crowd. If you want to read more about
- 6:14this topic, you can check out this book
- 6:18. There is a book on Amazon called "The
- 6:20Wisdom of Crowds". Although I haven't
- 6:22read the whole thing. I read the first
- 6:24couple of chapters, which I found very
- 6:26interesting. And it deals exactly with
- 6:29this topic: how people in a crowd know
- 6:32the answer. And guess what? Ensemble
- 6:35learning is based on this very thing.
- 6:38The entire fundamental concept of
- 6:39ensemble learning in machine learning
- 6:41is based on the fact of the wisdom of
- 6:43the crowd. That a crowd knows more than
- 6:47an individual person. Okay? So now that
- 6:49we know what the wisdom of the crowd is
- 6:51. Now let's move towards ensemble
- 6:53learning and understand its basics. So
- 6:56guys, now we will understand the core
- 6:58idea of ensemble learning. The good
- 7:00part is that the core idea is very
- 7:02simple. Okay? So, there are two steps
- 7:05related to any machine learning
- 7:07algorithm. I guess you already know
- 7:08this. One is training and one is
- 7:13prediction. Training is where it looks
- 7:18at the data to find patterns within it,
- 7:20and prediction happens when you give it
- 7:22a new query point and it has to answer
- 7:25based on what it has learned. Right? I
- 7:28am explaining the prediction stage to
- 7:30you. I will explain the training stage
- 7:32to you later. When we study the types
- 7:34of ensemble learning. Okay? Now look
- 7:36closely. In the prediction stage,
- 7:38suppose you have an ensemble learning
- 7:41algorithm. An ensemble learning
- 7:43algorithm means you have a collection
- 7:45of smaller base models. Suppose you
- 7:48have different algorithms or different
- 7:51base models. Like this. These are
- 7:55different base models. Basically, they
- 7:58are algorithms. Okay? It could be a
- 8:01decision tree, SVM, KNN, linear
- 8:02regression, it could be anything. These
- 8:05are different machine learning models.
- 8:08And collectively, this collective model
- 8:12is your ensemble learning model. By
- 8:16combining all of these, you are
- 8:18creating a big model that you call an
- 8:20ensemble learning model. Okay? Now, one
- 8:25thing you always have to ensure is that
- 8:28these models are different. These
- 8:33models are different. This will always
- 8:35be an important point. You must always
- 8:38make sure that your models are very
- 8:40different. I can give you an intuition
- 8:43for this as well. Suppose you are
- 8:45sitting in KBC and you have to use an
- 8:48audience poll, and you find out that
- 8:51all the people sitting in your audience
- 8:54are software developers. So, is this a
- 8:57good thing or a bad thing for you?
- 8:59Obviously, it is a bad thing because if
- 9:01everyone is a software developer, then
- 9:03their knowledge is in the software
- 9:05domain. So, if a question about
- 9:07computer science is asked of me
- 9:08tomorrow and I don't know it, these
- 9:10people will tell me because they are
- 9:12all software developers. But they won't
- 9:15know anything else. Now imagine a
- 9:16second scenario where the audience
- 9:18sitting in front of me is from
- 9:19different backgrounds. One is a CA, one
- 9:21is a computer science engineer, one is
- 9:23a civil engineer, one is a doctor, one
- 9:25is a politician, one is a philosopher.
- 9:27Right? So now I have a variety of crowd
- 9:30. Right? So now if I ever get stuck,
- 9:32it's possible that if the engineer
- 9:34doesn't know, the politician might know
- 9:37. If the politician doesn't know, then
- 9:38the literature person might know. Right
- 9:40? So that is why when you build
- 9:43ensemble learning algorithms, you try
- 9:45to ensure that the base models that
- 9:47make up the ensemble model come from
- 9:49different backgrounds. They should be
- 9:52different. So we need experts who
- 9:54possess expertise in different domains.
- 9:56All right? So going by that logic, I am
- 9:59telling you that you need different
- 10:01models. Now, what exactly does it mean
- 10:03to have different models? It has two
- 10:05meanings. One meaning is that all these
- 10:07algorithms themselves are different.
- 10:09For example, let the first one be
- 10:10Linear Regression. The second one be
- 10:13SVM. The third one be a Decision Tree.
- 10:15In this way, there should be different
- 10:17algorithms. So it automatically became
- 10:19different. There is a second way too;
- 10:22what can you do? Fix the algorithm. All
- 10:25models are Linear Regression algorithms
- 10:28. All of them are just Linear
- 10:30Regression algorithms. But when you
- 10:33were training these models, you gave
- 10:35them different data. You gave them
- 10:38different data. The algorithm is the
- 10:40same. You changed the data. And because
- 10:43you changed the data, these models will
- 10:45train differently, and since they train
- 10:48differently, you have created that
- 10:50variety. You have created a difference.
- 10:52So always remember point one: in
- 10:53ensemble learning, the base models must
- 10:55be different. They should be different,
- 10:58you need variety. And how will that
- 10:59difference come about? There are two
- 11:01ways. First, using different algorithms
- 11:04. Second, using the same algorithm but
- 11:06showing different data. All right? And
- 11:08the third is a combination of these two
- 11:10. You are using different algorithms
- 11:12and also showing different data. All
- 11:13right? So I guess you are understanding
- 11:15up to this point. So now all these
- 11:17models have been trained. All these
- 11:19models are trained. Now I have a new
- 11:22query point. You can take the same
- 11:24example we have been using until today.
- 11:27CGPA, IQ, and whether the placement
- 11:31happened or not. We are talking about a
- 11:35classification problem. So now a new
- 11:37student has come, and I know their CGPA
- 11:41and their IQ, and I need to predict
- 11:43whether they will get placed or not. So
- 11:47what will I do? I will pass this query
- 11:50point to every algorithm. Since every
- 11:52algorithm is a machine learning model
- 11:54in itself. It can make a prediction.
- 11:57Right? So it turns out the first one
- 11:59said they will get placed. The second
- 12:01one said they will not. The third one
- 12:03said they will, they won't, and so on.
- 12:06Now suppose we had five models. Right
- 12:09now, I have written 'n' here. So, let's
- 12:12assume we have five models, where three
- 12:15people said it will happen and two said
- 12:17it won't. Then the democracy logic
- 12:21applies. That logic is called majority
- 12:24count. So, whichever class the majority
- 12:28count favors, that is what your
- 12:30ensemble model will say. So here, if
- 12:33more people are saying placement will
- 12:35happen, your ensemble model's output
- 12:37will be that placement will happen. If
- 12:40more people had said it won't happen,
- 12:42your ensemble model would say it won't
- 12:44happen. Right? I hope you are
- 12:46understanding how prediction works in
- 12:48ensemble learning. Right? We will talk
- 12:50about the training part a little later.
- 12:52Don't worry about that. Now, this was
- 12:54the classification setting. What about
- 12:56the regression setting? What will
- 12:58happen in the regression setting?
- 12:59That's the question. Nothing will be
- 13:01different. It will be exactly the same
- 13:03thing. You will follow two things again
- 13:05. First, all the models you have must
- 13:08be different. There should be variety.
- 13:11And the second thing is that your
- 13:13trained models will either be the same
- 13:15models or the same models trained on
- 13:17different data. Different, I'm sorry, I
- 13:20got confused. Either they will be
- 13:22different models or the same models
- 13:23trained on different data. I hope
- 13:25there's no problem. Sorry about that.
- 13:28Now, if you have a regression setting.
- 13:31Suppose your problem statement is to
- 13:33predict the LPA based on CGPA and IQ.
- 13:36How many lakhs will the placement be?
- 13:38So in this case too, what will you do?
- 13:40You will send your input query point to
- 13:43all the models. Now, these models are
- 13:46internally regression models. So, they
- 13:49will output a number for you like 8.1,
- 13:533.2, 4.1, 7.5, and the ensemble model,
- 13:57the overall model, what will it do? It
- 14:01will add all these up and divide by
- 14:04four. Or in short, I could say it will
- 14:06calculate the mean. And as soon as you
- 14:09calculate the mean, you will get your
- 14:11answer. Right? So I guess you are
- 14:15understanding. It is a very simple
- 14:17thing. It is a very simple thing and
- 14:19very effective as well. Right? So the
- 14:21core idea is just that you bring in a
- 14:24lot of models and ask them all one
- 14:26question. You take all their answers
- 14:29and then make a decision based on those
- 14:31answers. Right? That is the whole idea.
- 14:34So, I have explained the prediction
- 14:36stage quite well here. I didn't explain
- 14:39the training stage because training
- 14:41happens in many different ways.
- 14:43Depending on which type of ensemble
- 14:45learning you are using. So that is why
- 14:47what will we study next? Types of
- 14:49ensemble learning. Where I will tell
- 14:51you the different types and how
- 14:53training happens in each type? I will
- 14:55explain that to you as well. Okay? That
- 14:57is what we are going to study next.
- 14:58Okay guys? So now let's discuss the
- 15:01different types of ensemble learning
- 15:03methods in machine learning. Okay? So
- 15:06first, let me create a sort of chart
- 15:07for you and then we will have a more
- 15:09detailed discussion about it. So if you
- 15:12talk about ensemble learning, precisely
- 15:15we have four different types that are
- 15:18used the most. Actually, there are more
- 15:20types than that. But these are the four
- 15:23most important types. The easiest and
- 15:26most useful type is voting. This is
- 15:31called voting ensemble. Okay? The
- 15:36second one is called bagging. This is
- 15:42also used very extensively. The third
- 15:45one, which is perhaps the most used
- 15:47ensemble technique, is called boosting.
- 15:52And the last one is called stacking.
- 15:59Okay? These are the four most important
- 16:01techniques you need to study and use in
- 16:03ensemble learning. Okay? Now here, if
- 16:06you have ever heard the names of some
- 16:08famous algorithms. Then within bagging,
- 16:11there is a famous algorithm we call
- 16:13Random Forest. It is a very powerful
- 16:16and useful machine learning technique
- 16:19and you will be using it a lot later.
- 16:22Okay? Some very famous techniques also
- 16:24come under boosting. Such as AdaBoost.
- 16:30It is called Adaptive Boosting. Next is
- 16:33Gradient Boosting. This is also a very
- 16:36useful technique. Very useful. And the
- 16:41last one is what is used in every
- 16:43machine learning competition on Kaggle,
- 16:46called XGBoost, which stands for
- 16:48Extreme Gradient Boosting. Okay? So in
- 16:52these next four or five videos, we are
- 16:55going to do exactly that, we will study
- 16:57all these types in great detail and
- 17:00learn to use them. Okay? Now a little
- 17:04discussion on why these different types
- 17:07exist? These different types exist only
- 17:12because, as I told you a while ago, the
- 17:14most important thing in ensemble
- 17:16learning is that your base models
- 17:17should be different. Okay? So how will
- 17:21that difference come about? We studied
- 17:24two different things. First, you use
- 17:26different algorithms. And second, you
- 17:29use the same algorithm but provide
- 17:31different data. So, because of these
- 17:34two techniques, there are four
- 17:36different types. All right? I will give
- 17:40you a rough, quick, brief introduction
- 17:43to these four techniques. So that you
- 17:45are prepared. So when we study in
- 17:47detail later, you have some idea about
- 17:50it. All right? So let's start with
- 17:52voting. Voting ensemble depends solely
- 17:55on the fact that your base models are
- 17:58different. All right? Base models are
- 18:05different algorithms. So, what do you
- 18:08do here? Suppose you pick three
- 18:11different algorithms. One is SVM, one
- 18:15is logistic regression, and one is a
- 18:19decision tree. And you have a dataset.
- 18:23Let's call it D. And you provide this
- 18:26same dataset to all three and train all
- 18:29three models. As soon as all three
- 18:32models are trained, and now I have a
- 18:34new query point. What will I do now? I
- 18:38will send this same query point to
- 18:40these three trained models. And if it's
- 18:45a classification problem, I will find
- 18:47the result through majority voting
- 18:49count, and if it's a regression problem
- 18:52, I will calculate the mean. So,
- 18:54basically, voting is very simple. The
- 18:58difference or variety you are creating
- 19:00here comes from the fact that all your
- 19:03base models are different algorithms.
- 19:06They are all getting the same data;
- 19:08they are different algorithms. And
- 19:10since there are different algorithms,
- 19:12there is variety in your ensemble, and
- 19:14it gives you better results. All right?
- 19:16So the logic here is very simple. Now,
- 19:19the next is stacking. I am teaching
- 19:22stacking first because stacking is
- 19:24conceptually very close to voting. So,
- 19:27what do you do in voting? In voting,
- 19:29you take three or any number of
- 19:32algorithms of different types and
- 19:34provide them with the same data. And
- 19:37when you have to make a prediction, how
- 19:40do you make it? You give a new query
- 19:43point to everyone and then calculate
- 19:45the majority count accordingly. In the
- 19:48second one, you just do one thing
- 19:50differently; you have the same setup
- 19:52here. This is the same setup. But here,
- 19:54you introduce one more machine learning
- 19:56algorithm. Here, you introduce one more
- 19:59machine learning algorithm. Let's say
- 20:01you introduced a KNN. Let's say you
- 20:04placed KNN here. All right? Now, what
- 20:07will this do is that this KNN model
- 20:12will also be trained. How will it be
- 20:14trained? Suppose in the training phase,
- 20:18when you sent the first data to these
- 20:21three, it predicted one, it predicted
- 20:24zero, and it predicted one. Right? So
- 20:28what was the output? One. So now the
- 20:31input KN will get is this. The data for
- 20:34KN, just like the data for these three
- 20:36models, is this data set. But the data
- 20:39for KN will be this. What did SVM say?
- 20:42What did LOR say? As in, what did
- 20:44Logistic Regression say? What did the
- 20:46Decision Tree say? And what was the
- 20:48result in the end? So here it was one.
- 20:50Here it was zero. Here it was one and
- 20:51the result was also one. Now for the
- 20:53next training point, suppose SVM said
- 20:55zero, LOR said zero, Decision Tree said
- 20:57one. But the result was still one. So
- 21:00now this data will be given to this
- 21:02model and this model will be trained on
- 21:05this data, and after training on this
- 21:07data, what will it do? Its main job is
- 21:10to assign a weightage to these base
- 21:13models. So it will give W1 to this, W2
- 21:16to this, and W3 to this. So basically,
- 21:19what you are doing now is you just made
- 21:21a small change in voting. I mean, what
- 21:24happens in democracy? Everyone's voting
- 21:26rights are equal. My vote will count as
- 21:28one and Mukesh Ambani's vote will also
- 21:30count as one. Right? But in stacking,
- 21:33you are just creating a difference
- 21:36where you analyze which model is
- 21:38talking more sense and you give it more
- 21:41weightage. Which model is making a few
- 21:44mistakes, you give it slightly less
- 21:46weightage. So imagine that in a
- 21:48different democracy, Mukesh Ambani's
- 21:51vote weightage would be higher than my
- 21:54vote's weightage. Meaning his say would
- 21:57be more important than my say. Because
- 21:59for whatever reasons, he is a rich guy
- 22:01or whatever. Right? So the same factor
- 22:04is being applied here. This model's job
- 22:07is to decide the weight of these models
- 22:09. Why decide the weight? Because it is
- 22:11not necessary that all three models
- 22:13will be equal. One model might be
- 22:14better, one model might be worse. So
- 22:16you also want to see who is good and
- 22:18who is bad. So this is called stacking.
- 22:20Right? Stacking and voting are
- 22:22conceptually the same. There is just
- 22:25one small difference in stacking where
- 22:27you give additional weightage to the
- 22:28models, which you don't do in voting.
- 22:31In voting, all your models 'weight, say
- 22:33, or output are considered equal. Right
- 22:36? I hope you have understood packing
- 22:38and voting. Now let's talk about
- 22:40bagging. Bagging is a variant of the
- 22:43ensemble learning technique. Let's talk
- 22:46about this quickly as well. So, first
- 22:49of all, the full form of bagging is
- 22:54Bagging stands for bootstrap Bootstrap
- 23:01aggregation. Okay? The concept is very
- 23:07simple, let me explain. Here, the
- 23:10variety you create is not achieved by
- 23:13choosing different base models or
- 23:15different algorithms; no, here you keep
- 23:17the same algorithm. The difference, or
- 23:21the variety, comes from the fact that
- 23:24you give each model different data.
- 23:27That is the main idea. So I will show
- 23:29you. Suppose you have three base models
- 23:35and all three are SVMs. Support Vector
- 23:39Machines or Logistic Regression. It's
- 23:42up to you. No problem. All three are
- 23:44the same models. So at this point,
- 23:45there is no difference. All three are
- 23:47the exact same individuals. They have
- 23:48the same education. They have the same
- 23:50knowledge. Now, when you receive data.
- 23:53When the data comes to you. This is D.
- 23:57Now, suppose you have data for 1000
- 24:00students and you need to create variety
- 24:03. What will you do? You will perform
- 24:06bootstrapping. What do you mean by
- 24:08bootstrapping? You will give each of
- 24:09them different data. You will give D1
- 24:11to this, D2 to this, and D3 to this.
- 24:13Now, how will D1 be created? You
- 24:15decided that I will show each model
- 24:17data of 500 students. So what did you
- 24:19do? You randomly picked data of 500
- 24:22students from it, which we call
- 24:25bootstrapping, and you are calling that
- 24:29D1, and you fed that same D1 here.
- 24:32Right? Then what did you do? You put
- 24:34the data of these 500 students back
- 24:36inside. Put it back into the original
- 24:38data. And you randomly picked 500
- 24:41students' data again. This is D2. Now
- 24:44there is no guarantee that D1 and D2
- 24:46will be the same. In fact, there is a
- 24:48very high probability that both will be
- 24:50very different data. Now you took D2
- 24:53and gave it to model two. Then you put
- 24:56the data of these 500 students back
- 24:59inside, and for the third time, you
- 25:01created D3 by randomly picking 500
- 25:03students 'data, and you gave this D3 to
- 25:06model three. Now, since they are the
- 25:08same machine learning models, if you
- 25:10give them different data, they will be
- 25:13trained differently, and if they are
- 25:15trained differently, the variety we
- 25:17wanted is achieved. The rest of the
- 25:19process is exactly the same. Now, if
- 25:21you have a new query point, you will
- 25:24give it to all three. You take the
- 25:26output of all three, calculate the
- 25:27majority count, or find the mean.
- 25:29Depending on the problem you are trying
- 25:30to solve. And that is how bagging works
- 25:33. Okay? A special case of this is
- 25:35Random Forest. Random Forest is a
- 25:38special case of bagging. Where the base
- 25:41models used are always decision trees.
- 25:44And that is why the name is forest. A
- 25:46collection of trees. Okay? If you are
- 25:49making an ensemble of any other
- 25:51algorithm, it is called bagging. But if
- 25:53you are building it with decision trees
- 25:55, it is called Random Forest. That is
- 25:57the biggest difference. Okay? So I hope
- 25:59you have understood bagging as well.
- 26:01The last one is boosting, which is the
- 26:03most powerful technique. And the most
- 26:07useful technique. The concept of
- 26:10boosting is a bit different. In
- 26:12boosting, you also take the same models
- 26:16. You take the same model. Suppose this
- 26:19is your logistic regression. This is
- 26:23also logistic regression, this is also
- 26:25logistic regression. Now, here you take
- 26:28your data and give it to the first
- 26:32model. The first model trains on it and
- 26:37notes down what mistakes it is making.
- 26:42Okay? What mistakes it is making.
- 26:44Basically, if I showed it data of 1000
- 26:46students, how many students did it
- 26:48predict incorrectly? It keeps that in
- 26:52mind. And then, the mistakes it made,
- 26:55it passes on to the next model. It says
- 26:58, "Look here, these are the students I
- 27:00made mistakes on.""You work carefully
- 27:02on these." And since the next model is
- 27:05specifically briefed about that data
- 27:07where the errors occurred. "You work
- 27:09carefully." So now, what will this
- 27:11model do? It starts giving correct
- 27:13results on those wrong points. It
- 27:15starts giving slightly improved results
- 27:16. But then it has its own mistakes. So
- 27:19it passes its own mistakes to the next
- 27:21one. So in a way, you are boosting,
- 27:24boosting your model performance. And
- 27:28you tell the next one every time, and
- 27:30doing this in a series, what does your
- 27:32model do? It starts making fewer
- 27:35mistakes. Okay? And that is how
- 27:36boosting works. Honestly, it's a bit of
- 27:38a complex topic, and I won't be able to
- 27:40explain it within five minutes. But
- 27:42this is the core idea. The second model
- 27:44reduces the mistakes of the first model
- 27:45. The third model reduces the mistakes
- 27:47of the second model. And you keep doing
- 27:48this in a series. And eventually, you
- 27:51get a model, a big model that is
- 27:53reducing overall errors. All right? So
- 27:55we will study boosting in great detail.
- 27:58It is the most powerful machine
- 27:59learning algorithm or technique. So we
- 28:02are going to study this thoroughly. So
- 28:03don't worry about that. But the core
- 28:05idea is exactly this. All right? So
- 28:06these are the different types of
- 28:08ensemble techniques you need to study,
- 28:10precisely four, and then different
- 28:12algorithms within them. But yeah, I
- 28:14hope I have given you a clear-cut
- 28:16overview. All right? Now, at this point
- 28:19, you might have some questions in your
- 28:21mind: why does ensemble learning
- 28:23actually work? What is the reason it
- 28:25works so well and gives good results?
- 28:27Why are multiple models able to provide
- 28:29better results than a single model? I
- 28:31will tell you that in the next section,
- 28:34where we discuss why ensemble learning
- 28:36works. All right? So guys, now I will
- 28:39try to explain it from a pure machine
- 28:41learning perspective, why ensemble
- 28:43learning is able to work. All right? I
- 28:46found a great image while searching on
- 28:48Google. First, I will explain it in
- 28:51terms of classification, and then I
- 28:52will explain it in terms of regression.
- 28:54So, if you talk about classification,
- 28:56let's assume you have this dataset. All
- 28:59right? So here, there are three
- 29:01different classes. Square is one class,
- 29:03triangle is one class, and circle is
- 29:05one class. And the black and white one
- 29:08is the training data. And the green one
- 29:11is the testing data. And this is your
- 29:14model one. This is model one. All right
- 29:17? It could be any model. And this is
- 29:20the decision boundary of this model.
- 29:23According to model one, this is the
- 29:25decision boundary. All three classes
- 29:28will be separated in this way. All
- 29:29right? Now you trained model two on the
- 29:33same data. And model two said that your
- 29:36decision boundary should be this. Based
- 29:39on its own learning, based on its
- 29:40training. This should be the decision
- 29:42boundary. And when you trained a third
- 29:44model, model three, on the same data,
- 29:47it said the decision boundary should be
- 29:49something like this. Right? Now, if you
- 29:52stack these three decision boundaries
- 29:54on top of each other, you will see
- 29:57something like this. Here are the
- 29:59decision boundaries of all three. And
- 30:00you can see that there is a slight
- 30:01difference. There is a slight
- 30:03difference in their opinions. Right?
- 30:05Now, what will ensemble learning do?
- 30:07Using majority count, it will use the
- 30:10concept of majority count. And by
- 30:13applying majority count, what is
- 30:15happening is that this decision
- 30:17boundary, as you can see, has become a
- 30:19bit smoother, or in short, it has
- 30:21become better than what it was. And
- 30:24that is how ensemble learning works in
- 30:26machine learning. Okay? You have to
- 30:29take data from multiple models, remove
- 30:32the noise from their decision-making,
- 30:34and extract a better model for yourself
- 30:37. And that is how it works in
- 30:39classification. Okay? You can imagine
- 30:41the same thing in regression as well.
- 30:42Suppose this is a dataset and you need
- 30:45a regression model. Now, different
- 30:48models are saying different things.
- 30:49This is one model which says it should
- 30:51be at this angle. The green one says it
- 30:53should be a bit higher. The red and
- 30:55orange ones are saying something else.
- 30:56But what are you doing? You are
- 30:58calculating the mean of all of them. As
- 31:00soon as you calculate the mean, you
- 31:02arrive somewhere in the middle here.
- 31:04This is the line. Okay? And it is
- 31:08neither too extreme in this direction
- 31:10nor too extreme in that direction. This
- 31:12is a model in the middle that will
- 31:13perform better than all those models.
- 31:16And that is the intuition. I mean,
- 31:18there is nothing much to explain here.
- 31:19Just by looking at these photos and
- 31:21images, an intuition will start forming
- 31:23in your mind about why ensemble
- 31:25learning works. Okay? So, I hope I was
- 31:27able to explain this to you. Now, let's
- 31:30discuss what the key benefits of using
- 31:32ensemble learning are. So guys, now
- 31:35let's discuss the benefits of using
- 31:37ensemble techniques. Now look, let's
- 31:40discuss a clear-cut disadvantage first
- 31:43because there is one big disadvantage.
- 31:45Pause the video for a second and think
- 31:47about what that big disadvantage is.
- 31:49That disadvantage is that if you want
- 31:51to use an ensemble technique. It means
- 31:54you will have to train multiple models.
- 31:56So, where earlier you used to train
- 31:58just one model to get your work done.
- 31:59Now, you are training multiple models
- 32:01there. Obviously, if you train more
- 32:03models, the computational complexity
- 32:05increases. Right? Which is not a good
- 32:08thing. So, that means there must be
- 32:09some solid advantage because of which
- 32:11we would use ensemble learning. Because
- 32:13if there is an obvious disadvantage,
- 32:16then there must be a big advantage to
- 32:18use it. So, there are actually
- 32:20advantages here. The very first
- 32:22advantage I have clearly told you until
- 32:24now. That is improvement in performance
- 32:29. Improvement in performance. That is
- 32:33the biggest one. The second one is that
- 32:39it reduces bias and variance. I guess
- 32:44you have already studied bias and
- 32:46variance. The job of a machine learning
- 32:48model is to make predictions. There are
- 32:50errors in predictions, and there are
- 32:52two types of errors. It is either bias
- 32:54or it is variance. Bias means the
- 32:56result is not good even on the training
- 32:58data. Variance means that your model is
- 33:01giving very different results on
- 33:03different data. It is varying. There is
- 33:05too much variance. These are the only
- 33:06two types of errors you have. Right?
- 33:08Now your goal in machine learning, the
- 33:11end goal in applied machine learning,
- 33:14is to have low bias, meaning not having
- 33:18high error on training data, plus low
- 33:21variance, meaning if you make changes
- 33:24in the data, the model's performance
- 33:27should not change significantly. These
- 33:31are your end goals. So, ensemble
- 33:34learning helps you achieve this. So, if
- 33:39you have a model that is a low bias,
- 33:45but a high variance model, it means it
- 33:47makes very few mistakes on training
- 33:50data. But if you change the data, its
- 33:52errors increase. So, a model like this,
- 33:55for example, a decision tree model. A
- 33:57decision tree is a low bias, high
- 33:58variance model. So, if you apply
- 34:00ensemble learning on top of it, what
- 34:02will it do? It will keep the low bias
- 34:05as it is, but convert the high variance
- 34:08into low variance. In short, it will
- 34:10give you what you want. Low bias, low
- 34:13variance, and similarly, if you have a
- 34:16high bias, low variance model, like
- 34:21logistic regression or linear
- 34:23regression, in that case, it will
- 34:26convert high bias to low bias and keep
- 34:29the low variance as it is. Again, in
- 34:31short, it will give you this result. So
- 34:34, the biggest challenge in applied
- 34:36machine learning is to reduce both bias
- 34:39and variance. That is generally not
- 34:41possible because increasing one
- 34:43decreases the other. It has a kind of
- 34:46inversely proportional or inverse
- 34:47relationship. Ensemble learning helps
- 34:51in achieving this difficult task. We
- 34:56will discuss this point in more detail
- 34:58when we study all these techniques one
- 35:00by one. But at this point, just
- 35:03understand that ensemble learning helps
- 35:05you achieve low bias and low variance.
- 35:07Which is quite a difficult thing to
- 35:09achieve. Okay? And the third most
- 35:12important thing is robustness. Your
- 35:18model becomes robust for many different
- 35:21types of data. Even if you make many
- 35:23changes to the data, your performance
- 35:25does not fluctuate much. Again, this is
- 35:27related to variance. But robustness is
- 35:30explained in a different way, used as a
- 35:32key point, a key benefit of using
- 35:35ensemble learning. Okay? I hope you are
- 35:37understanding everything up to this
- 35:38point. Okay? Now, the last point that
- 35:42we will discuss in this video is when
- 35:44one should use ensemble learning. I
- 35:48will give you the answer in short. The
- 35:50answer is always. There is no reason
- 35:54not to use ensemble learning. If you
- 35:57are a data scientist in a company, you
- 36:00must use ensemble learning in your
- 36:03projects. Generally, ensemble learning
- 36:05is tried at the end after doing
- 36:07everything else. When you have finished
- 36:09data cleaning, pre-processing, feature
- 36:11engineering, model building, and
- 36:13evaluation. Apply ensemble learning at
- 36:16the end, and you would see that your
- 36:18results will improve most of the time.
- 36:20Okay? So even if you are working in a
- 36:22company, you should use ensemble
- 36:24learning. Or if you are in a Kaggle
- 36:27competition, then you should apply
- 36:29ensemble learning there as well. In
- 36:32fact, Kaggle competitions became famous
- 36:34precisely because people were able to
- 36:37get great results by applying ensemble
- 36:39learning. Okay? XGBoost became very
- 36:42famous because of Kaggle. Okay? So in
- 36:44short, I have given you a complete
- 36:46introduction to ensemble learning. Now,
- 36:49what will we do? We will study all
- 36:50ensemble techniques one by one. We will
- 36:53start with voting. Then we will study
- 36:55bagging. Then random forest, after that
- 36:57boosting algorithms, and finally I will
- 37:00teach you stacking. The goal is that
- 37:02you should understand ensemble learning
- 37:05thoroughly. And be able to use it. I
- 37:07will teach you in such a way, and guess
- 37:09what? Once you have learned this, many
- 37:11times you won't even need deep learning
- 37:14. I mean, from what I have seen in my
- 37:17experience, if you have a small dataset
- 37:20, 95%of the time your ensemble
- 37:21technique algorithms will beat deep
- 37:24learning. But if the data is large,
- 37:27then deep learning is a different thing
- 37:29. So yeah, I hope I have given you a
- 37:32clear, bird's-eye view of all the
- 37:35ensemble techniques. If you liked the
- 37:38video, guys, please consider
- 37:39subscribing. Thanks for watching. Bye.
About this transcript
This page contains the full transcript of Introduction to Ensemble Learning | Ensemble Techniques in Machine Learning by CampusX, generated from the public captions YouTube serves with the video. The transcript has 6,065 words across 896 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.