AdaBoost - A Step by Step Explanation — Transcript
Full transcript
- 0:00Hi guys, my name is Nitish and welcome
- 0:02to my YouTube channel. In the last
- 0:04video, we studied the core intuition of
- 0:06AdaBoost. In this video, I am going to
- 0:09show you how AdaBoost works step by
- 0:11step. You can watch this video in two
- 0:13parts. In the first part, I will show
- 0:16you all the theory on the whiteboard,
- 0:18and then in the second part, I will
- 0:20demonstrate the practical
- 0:21implementation of that theory through
- 0:23code. So let's start the video. Also,
- 0:27if you haven't subscribed to my channel
- 0:30yet, please subscribe, and now we will
- 0:32start this whole thing. So, to
- 0:36understand AdaBoost step by step, I
- 0:38have first taken a toy dataset with two
- 0:40input columns, x1 and x2, and one
- 0:42output column, y. This means it is a
- 0:45classification problem because y only
- 0:48contains one and zero. So now, what we
- 0:51will do is, first of all, we know that
- 0:54n = 5 here. The number of rows in the
- 0:58dataset is five. This is a small toy
- 1:00dataset. Okay? So, speaking step by
- 1:04step, the very first step in AdaBoost
- 1:07is that you initially assign a weight
- 1:10to each of your rows. Okay? And
- 1:14initially, when you assign weights, the
- 1:16weight of all rows should be equal.
- 1:18Okay? So, what will we do? To calculate
- 1:21that weight, we will give each row a
- 1:24weight of 1/5, as in 0.2. Generally,
- 1:29whenever you start, what do you do? You
- 1:32do 1/n. What is n? The total number of
- 1:36rows. And this is your weight. The
- 1:38initial weight of every row will be 1/n
- 1:41. If you add all these 1/n, 1/n, 1/n, n
- 1:44times, you will get n. I am sorry, you
- 1:46will get one. Okay? So, what am I doing
- 1:49now? I am creating a column named '
- 1:52weight' here. And here I am writing 0.2
- 1:58, 0.2, 0.2, 0.2, and 0.2. Okay? Now,
- 2:04the main work starts from here. Now we
- 2:06will enter stage one. First of all, we
- 2:09will enter stage one. In stage one, the
- 2:13first thing you will do is train a
- 2:16decision tree on this dataset. The
- 2:20specialty of that decision tree will be
- 2:22that its max depth will be one. Meaning
- 2:24, it will be a decision stump. Okay?
- 2:27Now, how will it work on this? You will
- 2:30have to read the decision tree to know
- 2:31that. I can give you a rough intuition
- 2:35that you will make a cut on either x1
- 2:38or x2. So it is possible that the
- 2:41decision tree stump looks like this. x1
- 2:44> 5 This could be a stump. x2 < 10,
- 2:51this could also be a stump. So many
- 2:54such decision trees are possible.
- 2:57Decision stumps are possible on this
- 2:59dataset. But which one of these
- 3:01decision stumps will you choose? You
- 3:04will choose the decision stump that
- 3:06results in the maximum decrease in
- 3:07entropy or the highest information gain
- 3:09. Call it entropy or Gini entropy, it
- 3:13is the same thing. Right? So what is
- 3:15the first thing we did? Finalized a
- 3:18decision stump that was performing the
- 3:21best. Right? Now we have selected that
- 3:23model. Let's call that decision stump
- 3:26Model M1. Right? Now we have to do two
- 3:30things. The first thing we have to do
- 3:34is check the performance of this
- 3:36decision stump on this dataset. So what
- 3:40will we do? We will take these inputs,
- 3:42feed them to this decision stump, and
- 3:44get the output. Right? Or simply put,
- 3:47we will calculate y-pred. So I am
- 3:49creating another column here by the
- 3:53name of y-pred. Basically, what is this
- 3:56algorithm's prediction on the training
- 3:58data? Now let's assume the prediction
- 4:00is something like this. This is the
- 4:05prediction we got. Right? Obviously,
- 4:07this is not the real prediction. I am
- 4:08telling you this for the sake of an
- 4:10example. Now if you notice here, all
- 4:13the predictions are correct. Except for
- 4:16this one. This prediction came out
- 4:19wrong. Actually, it was one. Our
- 4:22algorithm says it is zero. Right? So
- 4:24now what do we have to do? We have to
- 4:26do two things. First, we have to
- 4:31calculate alpha 1. What is alpha 1? It
- 4:34is how much say Model 1 will have when
- 4:37making the final prediction. Right?
- 4:40That is alpha 1, or you can call it the
- 4:42weight of this model. Right? Now, what
- 4:45will this depend on? This alpha will
- 4:48depend on your error rate. What is the
- 4:53error rate? It is how many mistakes you
- 4:55made on this data. It will depend on
- 4:58how many mistakes your model made on
- 5:00this data. If your error is very high,
- 5:04then your alpha will be very low.
- 5:06Because think about it, it is logical.
- 5:09If your model is making too many
- 5:10mistakes, its say should be less when
- 5:13making the final prediction. If its
- 5:15performance is very good, its error
- 5:17rate will be low, so its alpha will be
- 5:19higher. In the final prediction, we
- 5:21will value its input more because it
- 5:23made more correct predictions. Okay?
- 5:26Now the question is, how do we
- 5:27calculate this alpha? We need a
- 5:30function that will calculate this alpha
- 5:32for us. Okay? Now, to calculate this,
- 5:35we will have to use our brains a little
- 5:37. What do we need to think about? I
- 5:38will tell you. Let me ask you a simple
- 5:40question. In the process of answering
- 5:42that, you will understand how alpha is
- 5:44calculated. Suppose we have three
- 5:47models. This is model A, this is model
- 5:51B, and this is model C. Okay? Now,
- 5:56model A is always correct. Meaning its
- 6:00error rate is zero. It never makes a
- 6:03mistake. Model B is never correct. It
- 6:08is always wrong. This means this
- 6:11model's error rate is 100%. See, error
- 6:13rate is basically one minus accuracy.
- 6:15If your accuracy is 90%, then your
- 6:17error rate will be 10%. I hope this
- 6:19makes sense. The third model is a bit
- 6:22weird. It has a 50%error rate. So, this
- 6:26model's accuracy is 100%, this one's is
- 6:280%, and this one's is 50%. The question
- 6:31is, which of these models is the most
- 6:34reliable? You have to answer this
- 6:37question. Pause the video for a moment
- 6:39and think about which of these three is
- 6:42the most reliable model. Now, I guess
- 6:45you can easily understand that the
- 6:47first model is the most reliable. Right
- 6:50? But if I ask you this question, if we
- 6:52remove that one, which of these two is
- 6:54more reliable? So think about which
- 6:56model is more reliable between these
- 6:58two. Between B and C. Now, if many of
- 7:01you thought it was C, then that is
- 7:03actually wrong. B is more reliable.
- 7:06Think about it. You have a friend who
- 7:09lies 50%of the time and another friend
- 7:12who lies 100%of the time. So whose
- 7:15words would you trust more? Obviously,
- 7:18you would trust the friend more who
- 7:20always lies. Whatever they say, you
- 7:23just take the opposite, and it becomes
- 7:25100%true. Right? But this one sometimes
- 7:28tells the truth and sometimes tells a
- 7:29lie. This means this model is risky for
- 7:32us. We can use this model's prediction
- 7:35by negating it. There won't be any
- 7:37problem. So both these models are
- 7:39actually the same. This one always
- 7:41tells the truth. This one always tells
- 7:42a lie. If you reverse its logic, this
- 7:44model and the one above become the same
- 7:46. This model here is dangerous. Okay?
- 7:49So that means if you plot a graph. If
- 7:52you plot a graph. Between alpha and
- 8:00error. Then I hope you understand what
- 8:04kind of graph you need. You need a
- 8:06graph such that when the error is close
- 8:09to zero, alpha should be high. Listen
- 8:13carefully. If the error rate is close
- 8:16to zero or zero, then you want a high
- 8:18alpha. Because if your model makes very
- 8:22few mistakes, you will give it a high
- 8:24weightage. If you have a model that
- 8:28makes a lot of mistakes. Let's assume
- 8:31this error rate is one. The error rate
- 8:34varies between zero and one. So if your
- 8:36error rate is very close to one.
- 8:39Meaning this model, then you need an
- 8:42alpha that is very large in the
- 8:45negative. Something like this. And if
- 8:50your error rate is 50%, i.e., .5, then
- 8:53what kind of alpha do you need? You
- 8:56don't trust it at all, so you should
- 8:58give it zero. Meaning you need
- 9:00something like this. This means you
- 9:02need a mathematical function that
- 9:04follows this graph. You need a
- 9:07mathematical function to calculate
- 9:09alpha that follows such a graph. And
- 9:13there is such a mathematical function,
- 9:18and it is 1/2 ln (1-error/error). And
- 9:25this is the function using which you
- 9:28calculate alpha. Okay? I hope you are
- 9:30understanding. How is the error
- 9:33calculated? I will tell you in a little
- 9:35while. But this is the formula using
- 9:38which you calculate the weightage or
- 9:41alpha of any decision stump. I hope I
- 9:44was able to give you the intuition as
- 9:46to why we use this function. This
- 9:47intuition is very important. So please
- 9:49make sure you have understood this once
- 9:51. So now comes the question, how is the
- 9:53error in this formula calculated? Or
- 9:56how do you determine the error of any
- 9:58decision stump? Okay? Now, this error
- 10:01is not exactly one minus accuracy. I
- 10:03will tell you how the error is
- 10:05calculated here. Before that, I am
- 10:07making a small change. Let's assume
- 10:09that until now we considered that the
- 10:10mistake was only at one point. To make
- 10:12the example even better, let's assume
- 10:14that mistakes are happening at two
- 10:15points. This is another point where a
- 10:17mistake is happening. Okay? Now, the
- 10:19error is very simple. The error is the
- 10:23sum of the weights of all those rows
- 10:25that are misclassified. It's simple.
- 10:29First, find out how many of your rows
- 10:31are misclassified. In our case, these
- 10:34two rows are misclassified. Take the
- 10:37weights of both and add them up. That
- 10:42is your error. So, for this first model
- 10:45, our first decision stump m1, the
- 10:49error is actually equal to .4 because
- 10:52it made mistakes on two rows. Now, if
- 10:56we put this value into the formula to
- 11:00calculate alpha 1, it will be 1/2 log 1
- 11:04-.4 /.4 If we simplify this, it becomes
- 11:111/2 log of .6 /.4, and this value comes
- 11:18to around .20. Okay? You can
- 11:21cross-check it once. But this is the
- 11:23value. So, we just calculated a very
- 11:27important thing: how much say our first
- 11:30weak learner, the decision stump, has
- 11:32in the final prediction. Alpha 1 will
- 11:37be equal to .20, okay? I hope you are
- 11:41following along so far. Now, the next
- 11:43step is also very important. In it, you
- 11:46have to do what I explained in the
- 11:48previous video: stage one is now
- 11:50complete. In stage one, we trained a
- 11:53weak learner and calculated its weight.
- 11:55It made some mistakes. Now we need to
- 11:58pass those mistakes to the next stage.
- 12:00We discussed this in the last video as
- 12:02well. You increase the importance of
- 12:06the points that are misclassified. How
- 12:09do you increase importance? By using a
- 12:11technique called up-sampling. So, the
- 12:13concept is very simple. You have to
- 12:16tell the next model, the next decision
- 12:18stump, that I made a mistake on these
- 12:19two points, so don't you make a mistake
- 12:21. And the way to tell it is by
- 12:25increasing the weight of these two rows
- 12:28in the data. Increase the weights. And
- 12:31decrease the weights of the correctly
- 12:34classified points. So, in a way, I am
- 12:37boosting the weight of the
- 12:38misclassified points. And 'boosting'
- 12:41comes from this boosting. Okay? So,
- 12:43what do we have to do now? We need to
- 12:46update the weights. Increase the
- 12:49weights of the misclassified points and
- 12:51decrease the weights of the correctly
- 12:53classified ones. But the question is,
- 12:55how much to increase and how much to
- 12:57decrease? And guys, the answer is on
- 12:59the whiteboard. I have written a
- 13:01formula. And this is the weight update
- 13:04formula. There are two different
- 13:05formulas. One formula is for
- 13:07misclassified points. Meaning, for
- 13:09these two points, and one for correctly
- 13:11classified points. Meaning, for these
- 13:13three points. There isn't much
- 13:14difference in the formulas. If we talk
- 13:17about the first misclassified point,
- 13:19the new weight value will be the
- 13:21current weight multiplied by e raised
- 13:24to the power of alpha one. If you
- 13:27remember, our alpha one was. 2 Right?
- 13:33So if I have to calculate the new
- 13:35weight for these two rows, it will
- 13:37actually be 2*e raised to the power of
- 13:422, and for correctly classified points
- 13:472*e raised to the power of -2. Now if
- 13:51you do this calculation, you will find
- 13:53that this value comes to 244 and this
- 13:55value comes to 166, and you can see,
- 13:57the weights of the incorrectly
- 13:59classified points have increased, and
- 14:01the weights of correctly classified
- 14:03points have decreased. So guys, I have
- 14:06added the results we got to a new
- 14:08column called updated weights, 6 for
- 14:11correctly classified points, 244 for
- 14:14misclassified points. There is only one
- 14:16problem. If you add all of these up,
- 14:19the number is 96. The thing is, you
- 14:24need to make sure that the sum of your
- 14:27weights is always one. Just like it was
- 14:31here. So what we will do very simply is
- 14:33? Divide every weight by 96. Basically,
- 14:37normalize them. And I have already
- 14:39performed this operation. So after
- 14:40doing this, the weights we get are
- 14:43these. And when you add these weights,
- 14:46you will get one. Okay? So now our
- 14:49dataset is prepared. Almost prepared.
- 14:53One last step remains to move to the
- 14:55next stage. Okay? So what I do is?
- 14:57First, rub this whole whiteboard and
- 14:59write this dataset in a more compact
- 15:01way. So guys, I have rewritten the
- 15:03entire dataset on the screen in a
- 15:05compact way. Here is my X1 column, X2
- 15:07column, Y column, and here are my new
- 15:10weights. Okay? Now I have to perform
- 15:13the most important step of this entire
- 15:15process. That is upsampling. So I told
- 15:17you that somehow I need to send the
- 15:19information to the following stages
- 15:21about what went wrong in this stage,
- 15:23and how do we do that? We kind of boost
- 15:27the misclassified points. And how does
- 15:29that boost happen? Using upsampling.
- 15:32How do we do upsampling? I will tell
- 15:34you in a little while. But to do
- 15:35upsampling, you will have to do one
- 15:37thing. And that is that you will have
- 15:39to create a range like this. So what I
- 15:42did? Depending on the weights, I
- 15:44created these ranges. So the first one
- 15:46was 0.166, so I created this range from
- 15:48zero. Then the next was 0.25. So I
- 15:51added 0.255 from here. Then I added 255
- 15:55to this, then added 16 to this, and
- 15:57then 16 again, so it reached one. So
- 15:59basically, I created the ranges. Now
- 16:02what will I do? Now, five times I will
- 16:07generate random numbers between zero
- 16:12and one. I will generate five random
- 16:17numbers between zero and one. Okay? Let
- 16:19me generate the numbers and show you.
- 16:22Suppose I have a number 0.13, one is
- 16:250.43, one is 0.62, one is 0.50, and one
- 16:29is 0.8. Suppose these were generated
- 16:33randomly. Now if you look, we need to
- 16:36find which range these numbers fall
- 16:38into. Whichever range it falls into, we
- 16:41will pick that column and that row. See
- 16:44, 0.13 falls into this range, so we
- 16:47will now pick column/row number one. 43
- 16:55Falls into this range. So we will pick
- 17:00row number three. Then 62 also falls
- 17:03into this range. We will pick three
- 17:06again. Then 50 also falls right here.
- 17:10We will pick row number three again.
- 17:12And then there is 8, which is this one.
- 17:14So we will pick four. So now, what am I
- 17:18doing? I am creating a completely new
- 17:20dataset that will only contain these
- 17:23numbers. These row numbers will remain.
- 17:25So one will be there, three will be
- 17:27there three times, and four will be
- 17:28there. Now because of this technique,
- 17:31you can see three is in my data more
- 17:34often. Meaning its weight has increased
- 17:36in the data. Okay? Now if you drew
- 17:38randomly, it could also be that one
- 17:40came, then three, then three, then two,
- 17:43then two. There are more errors in
- 17:44three and two, so you will see three
- 17:45and two more often. So basically, with
- 17:49this entire technique, when you do
- 17:51sampling, since we are picking rows
- 17:54based on this range, the row with the
- 17:56larger range has a higher chance of
- 17:59being picked. Right? And this technique
- 18:03is called upsampling. So now, the new
- 18:06dataset I have for stage two will not
- 18:08be this dataset. It will be this
- 18:11dataset. I will take the first column
- 18:13once. Then I will take the three column
- 18:17three times, and then the four column.
- 18:20I will remove this one and this one,
- 18:23both these rows. Okay? And I will work
- 18:25on these remaining ones and after that
- 18:27everything will repeat. What will I do
- 18:29again? I will take a new decision stump
- 18:32and train it on this data. Obviously,
- 18:35the data has changed. The behavior of
- 18:37the decision stump will also change. I
- 18:39will then calculate Alpha 2 for that
- 18:41new decision stump. And then I will
- 18:43keep performing this entire upsampling
- 18:45stage step again. How many times will
- 18:47we do this? As many decision stumps as
- 18:49you are going to keep. So, in the end,
- 18:51you will have values up to Alpha 1,
- 18:54Alpha 2, Alpha 3, dot dot dot Alpha n.
- 18:56And your final prediction will be Alpha
- 19:001 times h1x + Alpha 2 times h2x dot dot
- 19:04dot Alpha n times hnx, which I
- 19:06explained to you in the last video. Now
- 19:10I know showing it by repeating it again
- 19:12and again is a bit boring. So let's do
- 19:13one thing. Let's look at this whole
- 19:15thing step-by-step through some code.
- 19:17Okay? That will give you a great
- 19:19understanding of how this entire
- 19:20process works multiple times.
About this transcript
This page contains the full transcript of AdaBoost - A Step by Step Explanation by CampusX, generated from the public captions YouTube serves with the video. The transcript has 3,028 words across 425 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.