How Adaboost Classifier Works? | Geometric Intuition — Transcript
Full transcript
- 0:00Hi guys, my name is Nitish and welcome
- 0:03to my YouTube channel. Our next machine
- 0:06learning algorithm is AdaBoost. So,
- 0:09AdaBoost is a boosting algorithm. You
- 0:11might have heard of bagging and
- 0:12boosting, which fall under ensemble
- 0:14learning. And this is a class of
- 0:16algorithms that always performs very
- 0:18well. We will study other algorithms in
- 0:20boosting as well. We will study
- 0:22gradient boosting. We will also study
- 0:24XGBoost. But the very first algorithm
- 0:25that we will cover in boosting. Its
- 0:27name is AdaBoost. AdaBoost came out
- 0:29quite a long time ago, almost 20-25
- 0:32years ago, and when it first arrived,
- 0:34it was used a lot in face detection.
- 0:36Okay? Now, deep learning methods have
- 0:39arrived, so it was replaced there, but
- 0:41even today in machine learning,
- 0:43AdaBoost gets a lot of importance
- 0:45because the results it gives are
- 0:47excellent. Honestly, based on my
- 0:50personal experience, I can tell you
- 0:51that AdaBoost is a good algorithm. Your
- 0:53results turn out well. But most of the
- 0:56time, it is beaten by Random Forest,
- 0:58Gradient Boost, and XGBoost, etc. So,
- 1:00in the end, you might never even use
- 1:03AdaBoost. So, a question might come to
- 1:05your mind: why should we study it then?
- 1:07The answer is very simple. If you want
- 1:09to learn boosting, it starts with
- 1:11AdaBoost. If you know AdaBoost, it
- 1:13means you know 50%of boosting, so then
- 1:15while studying Gradient Boosting and
- 1:17XGBoost, you won't face as much trouble
- 1:19as you would without studying this.
- 1:21Okay? So that is why it is a good idea
- 1:23that you study AdaBoost carefully. I
- 1:26will make a complete video series kind
- 1:28of on AdaBoost. I have decided that I
- 1:31will make four videos on AdaBoost. The
- 1:34first video, which is this video, we
- 1:36will look at the core idea of AdaBoost
- 1:38in a classification scenario. Now,
- 1:41let's start this video. Now, before
- 1:43starting this video, I would like to
- 1:45tell you about three different topics.
- 1:47Because if you don't know these three
- 1:49topics, you might face some problems.
- 1:51So, the first thing is weak learners.
- 1:54What are weak learners? A weak learner
- 1:57is a machine learning model whose
- 1:59accuracy is very low. Or, to put it
- 2:02another way, it is just slightly more
- 2:04than 50%. 50%means luck; if you are
- 2:06tossing a coin and guessing if it will
- 2:09be heads or tails, your accuracy will
- 2:11also be nearly 50%. Because you are
- 2:13guessing. Right? If you have a machine
- 2:16learning algorithm whose accuracy is
- 2:18just over 50%, then obviously its
- 2:20performance is not very good. We call
- 2:23that a weak learner. Opposite to this
- 2:25is a strong learner, which has very
- 2:27good accuracy. 95%, 99%—meaning it
- 2:29can predict very accurately what is
- 2:32going to happen. Now, why are weak
- 2:34learners important in the context of
- 2:36AdaBoost? Because what do you do in
- 2:38AdaBoost? You combine multiple weak
- 2:40learners to build a large, powerful
- 2:43model. Okay? So that is why you should
- 2:45know the concept of weak learners. The
- 2:47second thing that we are going to
- 2:49discuss is decision stumps. Now,
- 2:51decision stumps are actually weak
- 2:53learners. There are different types of
- 2:55weak learners. One type of that is
- 2:57decision stumps. Now, a decision stump
- 3:00is a decision tree that has a maximum
- 3:02depth of one. Meaning you can only
- 3:05perform a split on a single node. Okay?
- 3:09This means it should be clear to you
- 3:11that this is a machine learning
- 3:13algorithm that is a weak learner
- 3:14because its result is not very good. I
- 3:17will quickly show you with an example.
- 3:19We have this dataset of students. CGPA,
- 3:22IQ, and whether they got placed or not.
- 3:26And I have these three students. I have
- 3:28also plotted them here on the graph.
- 3:30This is the CGPA axis. This is the IQ
- 3:33axis. Now, if you remember decision
- 3:34trees, then in a decision tree, what do
- 3:36you do? You make multiple splits in the
- 3:39data in this way. In this way, right,
- 3:42but in a decision stump, you can only
- 3:45make one single split. Only one split.
- 3:48So that one split will be either
- 3:50parallel to the CGPA axis or parallel
- 3:53to the IQ axis. Okay? So what do you do
- 3:55? You try both. You try making a split
- 3:59like this. You try making a split like
- 4:02this. You try making a split like this.
- 4:04In this direction, these are the only
- 4:06three splits possible. Similarly, if
- 4:08you talk about parallel to the IQ, one
- 4:10split is possible here. Everything here
- 4:13or here, one split is possible. One
- 4:15split is possible here and one split is
- 4:17possible here. Okay? These are the only
- 4:19splits. Right? Now, which split will
- 4:21you choose from these? You will choose
- 4:24the split that reduces your entropy the
- 4:28most. The one with the maximum
- 4:30information gain. So I hope decision
- 4:32stumps are clear to you. Now, a
- 4:34question might come to your mind: why
- 4:36did we study decision stumps? The
- 4:38reason for this is that although you
- 4:41can use any weak learner in AdaBoost.
- 4:43You can use any algorithm. You can even
- 4:46use neural networks. But 99%of the time
- 4:49, whenever you see someone using
- 4:51AdaBoost, which weak learner will they
- 4:53use internally? A decision stump. In
- 4:56fact, we will also use a decision stump
- 4:58as our weak learner throughout this
- 5:00entire four-video series. Because it
- 5:03gives the best results, and even more
- 5:05importantly, we get immediate
- 5:07interpretation of what our algorithm is
- 5:09doing. It's immediately clear what is
- 5:11happening. So, for these two reasons,
- 5:13you will mostly use decision stumps
- 5:15within AdaBoost. Alright? Now, the last
- 5:18point before starting: +1 and -1.
- 5:21Generally, whenever you work with any
- 5:23machine learning algorithm in a
- 5:25classification scenario, your classes
- 5:28in binary classification are 1 and 0.
- 5:31But here in AdaBoost, those classes are
- 5:34+1 and -1. Positive points are +1 and
- 5:38negative points are -1. They are not 0.
- 5:44They are -1. Why is that? You will
- 5:46understand in a little while. So, it
- 5:48was very important to understand these
- 5:50three points before starting. First,
- 5:52what is a weak learner? A machine
- 5:54learning model whose accuracy is just
- 5:56over 50%. What is a decision stump? A
- 5:59decision tree that has a maximum depth
- 6:01of one. Which has only one node. Third,
- 6:05+1 and -1 instead of 1 and 0. Alright?
- 6:08Now we are ready to start AdaBoost. So,
- 6:10let's start AdaBoost. So guys, first of
- 6:12all, let's start with the core idea of
- 6:15this entire algorithm. I will give you
- 6:17a geometric intuition. You will
- 6:19understand the big idea behind it.
- 6:22Alright? So, we have taken an example.
- 6:26Again, the same example, you could say
- 6:29we have data for, I don't know, nine
- 6:31students regarding their CGPA, IQ, and
- 6:34whether they got placed or not. So here
- 6:38, the minus points are the students who
- 6:40didn't get placed, and the plus points
- 6:42are those who got placed. Alright? We
- 6:45want to perform classification on this
- 6:48using AdaBoost. Alright? How will we do
- 6:50it? You have to keep watching. Before
- 6:53moving forward, I will introduce an
- 6:56important term here; whenever you read
- 6:59about AdaBoost, you will find it
- 7:02written that AdaBoost is a stage-wise
- 7:04additive method. Let's understand this
- 7:09additive method first. What is it? So,
- 7:11what is AdaBoost? It is formed by
- 7:14adding multiple weak learners. That is
- 7:17why the term "additive method" is used.
- 7:19And what does "stage-wise" mean? That
- 7:22all those weak learners are added one
- 7:24by one, sequentially. So, that is why
- 7:28this term: "stage-wise additive method.
- 7:31" Okay? Now, put the technicalities
- 7:33aside. Let's understand the core idea
- 7:36from scratch. So, suppose you have this
- 7:38dataset. Now, what will you do? You
- 7:40will fit a decision stump on this
- 7:43dataset. Which decision stump? There
- 7:45can be different decision stumps. Like
- 7:47this one, this one, this could also be
- 7:50one, or this, it could be any of them.
- 7:53There are many possibilities. You will
- 7:56fit the decision stump that results in
- 7:59the maximum reduction in entropy when
- 8:02applied here. Okay? Let's assume it was
- 8:05this one. This one. So, you created a
- 8:10decision stump. Creating which caused
- 8:14your data to be split in this way. This
- 8:17is the split. I have drawn the same
- 8:19dataset here. And here you can see this
- 8:22is the line that is cutting through
- 8:23your data. This is the decision stump
- 8:25that is splitting your data. And these
- 8:28are your two regions. This smaller one
- 8:30is the positive region, and this larger
- 8:31one is the negative region. Alright,
- 8:33this first decision stump has done its
- 8:35job. But if you analyze its results,
- 8:38you will notice that there are three
- 8:40points here that are incorrectly
- 8:42classified. Basically, these three plus
- 8:45points that are in the red region,
- 8:47where should they have been? In the
- 8:49blue region. Right? So now, what you
- 8:52will do is, the first decision stump's
- 8:55work is done. Now you will tell the
- 8:59next stump, "Look, I made mistakes on
- 9:01these three points." So you should
- 9:05handle these three points a bit more
- 9:07carefully. Okay? So what happens is,
- 9:11when you send the data to the next
- 9:13stage, you increase the value of these
- 9:15points—value in the sense, this will
- 9:17be clearer in the next video. We just
- 9:20increase the focus on them. How is the
- 9:23focus increased? That is not our
- 9:25concern at this point. There is a
- 9:27technique called upsampling. We perform
- 9:32upsampling. In a way, we just increase
- 9:34the value or importance of these three
- 9:36points in the data by any means. So
- 9:39that the next decision stump focuses
- 9:42more on these three. That is the core
- 9:44idea. Okay? Now, as soon as the first
- 9:46decision stump was applied, it made
- 9:48some mistakes. Here is that decision
- 9:50stump. I have drawn it right here. At
- 9:52this stage, at this point, you do one
- 9:54more thing. You calculate a weight for
- 9:58this particular decision stump. What is
- 10:01calculating weight? That finally, when
- 10:05the whole model is ready, how much say
- 10:08will this particular model have when
- 10:10making predictions? How much say will
- 10:13this decision stump have? Here,
- 10:16boosting is a bit different from
- 10:18bagging. What happens in bagging is
- 10:21that all your base classifiers are
- 10:22equal. Meaning, I give everyone a
- 10:25weight of one. It's like a democracy;
- 10:27everyone is equal. No matter how rich
- 10:29or how poor, they are equal. That's how
- 10:31it is in bagging. But it is not like
- 10:33that in boosting. In boosting, what
- 10:34happens is that depending on your
- 10:36performance on the data set. Based on
- 10:38how many things you get right and how
- 10:40many mistakes you make, you are
- 10:41assigned a weight. And at the end, when
- 10:43you go to make a prediction, this model
- 10:45will have a say based on this weight
- 10:47during prediction. I hope you are
- 10:49understanding. If not, you will
- 10:51understand in a little while. Anyways,
- 10:53we are now starting stage two. In stage
- 10:55two, we have made a small change to the
- 10:57data. What change? These points have
- 11:00now been up-sampled. Meaning, we have
- 11:02increased their importance in the data.
- 11:04Increased the value. Increased it. Now
- 11:07in the second stage, we have applied a
- 11:09decision stump again. Again, different
- 11:12decision stumps could have been formed.
- 11:14But this time, the decision stump that
- 11:16is giving the most reduction in entropy
- 11:18is this one. This vertical split from
- 11:21this spot. Okay? By the way, these are
- 11:23red points here. So, this split is
- 11:25giving the best result. Okay? So, I
- 11:27have drawn it here. This is our second
- 11:29base model. This is our second weak
- 11:31learner. This is our second decision
- 11:33stump. Okay? Now, if you look carefully
- 11:37, this time this stump classified these
- 11:39three points correctly, which were
- 11:41misclassified here, but it made some of
- 11:44its own mistakes. Look, these two red
- 11:48points, instead of being in the red
- 11:49region, are now in the blue region, so
- 11:51this is also a mistake. So, you will do
- 11:54two things here again: the first thing
- 11:56you will do is up-sample these two
- 11:57points and send them to the next stage,
- 11:59okay? And the second point will be that
- 12:03depending on its performance, you will
- 12:04calculate alpha 2 for it. What is alpha
- 12:072? The weightage of the second decision
- 12:10stump in the overall model. At the end,
- 12:13when making a prediction, alpha 2 will
- 12:15determine how much say the second
- 12:17decision stump has. If it had made
- 12:19predictions very well here, then alpha
- 12:21would be very high. If its predictions
- 12:23are very poor, then this alpha will be
- 12:25very low. It is a simple matter. Right?
- 12:28Next, we will come to stage three. Now
- 12:31in stage three, you see we have
- 12:33up-sampled these two. Okay? Because of
- 12:37which, notice again that a decision
- 12:39stump was formed that correctly
- 12:41classified these two points but made
- 12:44three more mistakes. Right? So now
- 12:48again, you will do two things here.
- 12:50What is the first thing you will do?
- 12:52You will calculate alpha 3. Depending
- 12:54on the performance of this particular
- 12:56model, you will derive alpha 3. And
- 12:58then, if you have more stages, meaning
- 13:00if you have to use more base models,
- 13:02then again you will up-sample these
- 13:04three points and send them to the next
- 13:07stage. But we are not doing any more. I
- 13:09guess you have understood the idea. We
- 13:11are building an AdaBoost with only
- 13:13three models. So our work is finished
- 13:15here. Now, if you understand at this
- 13:17point, what all do you have? You have
- 13:20all three base models. You have all
- 13:23three decision stumps, and you have
- 13:26alpha 1, alpha 2, and alpha 3 for them.
- 13:29Now what do you have to do? You have to
- 13:31add these three together. Okay? And
- 13:33what is the formula for adding them?
- 13:35Very simple. That formula is this. You
- 13:38can call your first decision stump h1x.
- 13:43You can write it in this notation
- 13:44because, in the end, any machine
- 13:46learning algorithm is a mathematical
- 13:47function. We are calling this function
- 13:50h. Hypothesis function, and x is our
- 13:52input. + h2x + h3x And this is my
- 14:02overall AdaBoost function. Just one
- 14:06last thing here: you will multiply this
- 14:08function by alpha 1. This one by alpha
- 14:122, and this one by alpha 3, and one
- 14:15last thing: you will take the sign of
- 14:18this whole thing. Meaning, after adding
- 14:21this whole thing, if you get a positive
- 14:24result, your answer will be +1, and if
- 14:27you get a negative result, your answer
- 14:29will be -1. So basically, here you
- 14:32calculate the sign. And this is the
- 14:36function. This is the function. Okay?
- 14:40Now let me show you how to make a
- 14:42prediction. Okay? Now suppose someone
- 14:44asked me, here is a student whose CGPA
- 14:46is 7.5. And their IQ is 81. You have to
- 14:52tell whether they will get placed or
- 14:54not. So, what will you do now? You will
- 14:57first bring this 7.5 and 81 and plot it
- 14:59here. Let's assume it came here. Okay?
- 15:03So, according to this model, this
- 15:05person should not get placed. So, it
- 15:07will be -1. Then you plot this point on
- 15:12this decision tree, and it came here,
- 15:14so it says they will get placed. And
- 15:18then, according to the third one, it
- 15:20came here again, so it says they won't.
- 15:24Now, it turned out that our weight
- 15:27alpha 1 was two. Alpha 2 was 10 and
- 15:32alpha 3 was one. Okay? So, what will
- 15:36you do now? You will put these values
- 15:38into this formula. So, what does alpha
- 15:391 become? 2 times -1 plus, what does
- 15:43alpha 2 become? 10 times 1 + what is
- 15:49alpha 3? 1 times -1, and now when you
- 15:52calculate this, it will clearly be a
- 15:54positive number. And what is the sign
- 15:57of eight? Positive. So, the answer will
- 16:00be +1. Meaning, this student should get
- 16:03placed. That is how AdaBoost works.
- 16:06Prediction works exactly like this.
- 16:08Okay? Now, if you want to get a
- 16:10geometric intuition of why this works
- 16:12correctly, it is on your screen. If you
- 16:14stack these three on top of each other,
- 16:16don't you think you will see a decision
- 16:19boundary like this? Like this, and now
- 16:21look at this data. If you think here,
- 16:25you actually wanted a decision boundary
- 16:29similar to this. Where this would be
- 16:33the blue region. And this would be the
- 16:35red region. This would be your best
- 16:37classification. Right? And by using
- 16:39these three, exactly this thing is
- 16:41being formed. Obviously, the more
- 16:44models you keep adding, the more this
- 16:46algorithm will move towards perfection.
- 16:48But yeah, this is the entire geometric
- 16:51intuition of this algorithm. I hope I
- 16:53was able to explain it to you. Now, we
- 16:55will study this thing in detail, step
- 16:58by step. Like, how these weights are
- 17:00being calculated, we don't know that.
- 17:02How this up-sampling is happening, we
- 17:04don't know that either. So, some of
- 17:06these key questions that remain, we
- 17:08will discuss them shortly in the next
- 17:11video. Thanks for watching. Bye.
About this transcript
This page contains the full transcript of How Adaboost Classifier Works? | Geometric Intuition by CampusX, generated from the public captions YouTube serves with the video. The transcript has 2,838 words across 413 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.