AdaBoost Algorithm | Code from Scratch — Transcript
Full transcript
- 0:00Hello guys, my name is Nitish. Welcome
- 0:02to my YouTube channel. We've been
- 0:04studying the Ad Boost algorithm. In the
- 0:07last video, I tried to explain
- 0:09step-by-step how Ad Boost works on a
- 0:11whiteboard. In this video, I'll give
- 0:14you a walk-through of the entire
- 0:16algorithm through the code. Okay? I
- 0:19wrote this entire code from scratch. So
- 0:21, you'll get a sense of how this
- 0:23algorithm could be built from scratch.
- 0:26So, let's start the video. I've already
- 0:29written the code. I won't rewrite the
- 0:31entire code. I'll just give you a
- 0:33walk-through. That's a fast and
- 0:35efficient way to explain the code. So,
- 0:37first, I installed a few libraries.
- 0:40I'll use a library called mlx10 to plot
- 0:42the decision boundaries. You know the
- 0:45other two libraries: Padas and NPI. I'm
- 0:48creating my own data frame and
- 0:49inserting my own data into that data
- 0:51frame, which I created myself. And
- 0:54here's my data frame. This is my data
- 0:56frame. There are 10 rows. There are
- 0:58three columns. Two columns are for
- 1:00inputs. One column is for outputs. And
- 1:03this is a classification problem. Now,
- 1:05let me plot it to show you what our
- 1:08data looks like. And you can see that
- 1:10this is our data, and it's essentially
- 1:12a slightly non-linear type of data. So,
- 1:15I deliberately chose this type of data.
- 1:18Now, we'll start applying the algorithm
- 1:20. So, the first thing you do in Boost
- 1:24is assign a weight to each row.
- 1:27Initially, those weights are equal. How
- 1:30that equality is calculated is very
- 1:32simple. You assign each row a weight
- 1:35divided by the total number of rows. In
- 1:38our case, we have 10 rows. So, each row
- 1:42gets one. So, I'm running this code.
- 1:45And now my data frame looks like this.
- 1:47You can see that everyone's weight is
- 1:49equal. Point one. Now the next thing
- 1:53you'll do is create a Decision Tree
- 1:55Classifier object, and you have to make
- 1:57sure that the max depth should be equal
- 1:59to one. Because the decision trees you
- 2:02use in Ada Boost are decision stumps.
- 2:06Okay? So I created this object.
- 2:08Extracted X and Y, and trained the
- 2:10model. Since the data is very small,
- 2:12the model trained very quickly. Let me
- 2:15show you the decision stump first. This
- 2:18is the decision stump. And this
- 2:20decision stump is being created above
- 2:22the second column, above X2. Here it is
- 2:24written X bracket one, meaning the
- 2:26second column, zero is the first column
- 2:28. And here is its decision boundary.
- 2:31Let's see how many mistakes this model
- 2:33is making? And you can see a, it's
- 2:35making three mistakes. The three blue
- 2:37points are in the orange region. Okay,
- 2:40now what we'll do is make predictions
- 2:43on the entire training data using this
- 2:45code. Now my data frame looks like this
- 2:48. You can compare them side by side to
- 2:51see how many mistakes our model made.
- 2:53So, there's no error in the first row
- 2:55because the label is one, and the
- 2:56prediction is one. There's no error in
- 2:57the second either. There's an error in
- 2:59the third. There's no error after that.
- 3:02There's one here, and there's one here.
- 3:04In total, our model is making three
- 3:06mistakes. Row two, row six, and row
- 3:10eight. Okay? Now you know what, next?
- 3:17Calculate a weight for this decision
- 3:20stump you trained the model using the
- 3:23formula 1/2 log 1-error divided by
- 3:26error. What is error? The sum of the
- 3:29weights for all the rows where a
- 3:31misclassification occurred. There were
- 3:34misclassifications on three rows. If
- 3:37the weights of all three are 1, then
- 3:39our total error will be 1 + 1 + 1,
- 3:40meaning 3. I've created a function here
- 3:43called "calculate model weights." If
- 3:46you provide the error, the formula will
- 3:48be applied and you'll get the model's
- 3:50weights. I'll run this formula once.
- 3:54We'll calculate alpha for our first
- 3:56model. So, I passed 3 into "calculate
- 3:59model weights," which is my total error
- 4:02, and the alpha is 0.42. Now, our next
- 4:07step is to increase the weights for
- 4:10rows with misclassifications and
- 4:12decrease the weights for rows with
- 4:15correct classifications. This way, you
- 4:19can report the current model's errors
- 4:21to the next model. Okay? The entire
- 4:24upsampling step was from the previous
- 4:26video. So, I've created a function here
- 4:29called "update rows," where you pass
- 4:32the rows and an alpha value. In our
- 4:35case, the alpha value is 0.423. And
- 4:39here I've put a check so that if that
- 4:42row is correctly classified, you
- 4:44multiply the current rate by e to the
- 4:46power of alpha. And if there's a
- 4:50misclassification, you multiply the
- 4:52current rate by e to the power of alpha
- 4:54. Okay? Let's run this code once. And
- 4:59we're creating a new column called
- 5:02Updated Weights and calling this code.
- 5:05And here are our updated weights, and
- 5:07you'll see that the weights for the
- 5:09rows where we made mistakes have
- 5:10increased. Okay? From point one, it's
- 5:150.15, and the weights for the rows
- 5:17where we made the right ones have
- 5:19decreased. From point one, it's 0.06.
- 5:22You're getting the idea. Now, there's
- 5:25one thing I want to discuss with you
- 5:27that I forgot to mention in the
- 5:29previous video: why do we multiply e to
- 5:32the power of alpha? Why do you multiply
- 5:36the current weights by e to the power
- 5:38alpha or e to the power alpha? Let me
- 5:41explain the reason. For that, we'll
- 5:43have to go into Desmos. And I'll show
- 5:47you this graph: y = e to the power x.
- 5:53This is a graph of e to the power x.
- 5:56Now, if you look at the behavior of
- 5:58this graph, as x increases, y increases
- 6:02rapidly. And if x is small, y also
- 6:06decreases. Right? So, x here is our
- 6:10alpha. So, this means that if a model
- 6:14has a high alpha, when will its alpha
- 6:16be high? Think about it. A model's
- 6:18alpha will be high when it is very
- 6:21accurate. It makes very few errors. A
- 6:23model with very accurate predictions
- 6:25will have a high alpha. Now, if you
- 6:28have a model with a very high alpha,
- 6:30then this whole thing will also be very
- 6:32high. And if this whole thing is very
- 6:35high, then when you multiply the
- 6:36current weight by a large number, the
- 6:38new weight will also be large. This
- 6:41means that if you have a very
- 6:44trust-uniform model, it will increase
- 6:47the weight of your incorrectly
- 6:49classified rows significantly. But if
- 6:54you don't have a very trust-uniform
- 6:56model, meaning alpha is small, then
- 6:58this number won't be very large; it
- 7:00will be small, and then the update will
- 7:02also be very small. And that's why we
- 7:05use the exponential function. I can
- 7:07also give you an intuition for-x.
- 7:09Here's a graph of-x. For very small x,
- 7:12y is large, and for very large x, y is
- 7:15small. Exactly the opposite. Okay? So
- 7:17if you have a model with a very high
- 7:20alpha, meaning your model is very
- 7:22trust-uniform, then this large number
- 7:25will become less than one, which will
- 7:28reduce your current weight. Okay? So
- 7:31please back up a bit and understand
- 7:33this whole idea. It is actual. Okay?
- 7:36Now we have our updated weights.
- 7:39There's only one problem, and that's
- 7:41that if you sum the updated weights,
- 7:50the total sum of all the days is only
- 7:550.91. And I told you that you should
- 7:58always update the weights in such a way
- 8:01that the sum of all the daily weights
- 8:03should always be equal to one. So we
- 8:05need to normalize. And here's the code
- 8:08for normalization. I'm dividing each
- 8:10weight by the sum of the weights
- 8:11multiplied by the sum of the weights.
- 8:13And storing it in a new column. And
- 8:16here's our output. Here are our
- 8:19normalized weights. So now all the
- 8:21incorrectly classified points are 16.6
- 8:24and all the correctly classified points
- 8:27are 0.7. Okay? And here are our
- 8:30normalized weights, exactly equal to
- 8:33one. Okay? What are we doing now? We're
- 8:35creating a range. So this is the code
- 8:37for creating a range. I'm creating an
- 8:38upper range. I'm creating a lower range
- 8:41. And here are my upper and lower
- 8:43ranges. Using these rates, I've created
- 8:46the upper and lower ranges. Okay? You
- 8:50can see this once. Now, our next task
- 8:53is to create a new data set from this
- 8:55same data set. That new data set will
- 8:58have the same number of rows as our
- 8:59current data set. And how will that new
- 9:02data set be created? We'll generate 10
- 9:05random numbers. And we'll check which
- 9:08range those random numbers fall into.
- 9:12We'll pick the particular row in which
- 9:14they fall. Okay? For this, I've created
- 9:18a function called "Create New Data Set"
- 9:19where I'll pass my old data set in.
- 9:22I'll calculate the indices. I'm running
- 9:25this loop 10 times. Each time, I'm
- 9:27generating a random number and checking
- 9:29which range it falls into. I'm taking
- 9:32the index of that range and putting it
- 9:34into a list of indexes and returning it
- 9:37to you. So here I called this function
- 9:41and you can see that two appears once,
- 9:44eight appears twice, and six appears
- 9:47once. So basically, the weights of
- 9:49those rows where mistakes were made
- 9:51have been increased. And this is what I
- 9:53told you in the last video, this step
- 9:55is called upsampling. So we've
- 9:57upsampled. Now I'm creating a second
- 10:00data frame from the old data frame and
- 10:03taking only these rows. Basically,
- 10:05taking the index values of these rows
- 10:08and taking only these columns: 0 1 2 3
- 10:10these four rows. Okay? And here is my
- 10:17new data frame. Okay? Always remember,
- 10:20as soon as you create a new data frame
- 10:22for the next step, you'll have the
- 10:24weights initialized from point one.
- 10:26You'll re-initialize with equal weights
- 10:28because you've already upsampled. Okay?
- 10:31What do you do now? We'll create a
- 10:32second decision tree. We'll drain it
- 10:36over the new data. Here's my second
- 10:40decision tree. It's drained over the
- 10:42first column. Okay? It's drained over
- 10:45x1. And then we'll plot its decision
- 10:49boundary. And you can see it's making a
- 10:52mistake. Above the blue point. So we're
- 10:55extracting the vibration again. And
- 10:58here's the second BF. So you can see
- 11:01it's making a mistake here. And there
- 11:04are no other mistakes. Let's
- 11:06cross-check again. One mistake, no more
- 11:10mistakes. Okay? We're calling calculate
- 11:13model weights again. And here we're
- 11:16doing one pass. And we're calculating
- 11:19alpha for our second decision tree. And
- 11:22its weights are coming out to be 1.09,
- 11:24which is higher than the previous
- 11:26weights because it made fewer mistakes.
- 11:29One more thing: sometimes you might end
- 11:30up with a model that doesn't make any
- 11:31mistakes, especially when you're
- 11:33working with a small data set. In that
- 11:35case, your error will become zero. Now,
- 11:38if the error becomes zero, this formula
- 11:40will start crying. This formula we saw
- 11:44above will start crying. Because you
- 11:47divide by the error, and if the error
- 11:49itself is zero, this entire term will
- 11:51explode. So what can you do in that
- 11:53case? You can put a small number in
- 11:55front of the error. That will prevent
- 11:59the term from becoming zero. Okay? I've
- 12:02already told you separately how to
- 12:04solve this problem if you ever run into
- 12:06it. This problem is solved internally
- 12:08in SciKitLearn, so you don't need to do
- 12:10it. Now coming back. So, we've reached
- 12:15this point. We calculated our alpha.
- 12:18Now we'll update our weights again. The
- 12:20weights of correctly classified points
- 12:22will be reduced. The weights of
- 12:24incorrectly classified points will be
- 12:25increased. So we're calling Update Row
- 12:28Weights again. This time, the value of
- 12:30alpha will be 1.09. Okay? Run it.
- 12:35Created the updated weights column, and
- 12:37here it is. And you can see there was
- 12:40only one error. So its weight is 29.
- 12:42All the others have a weight of 0.03.
- 12:44Again, we'll normalize it. And after
- 12:49normalizing, here it is. 49 is the only
- 12:53one, and all the others are five. Sum
- 12:58is equal to one. Okay? Again, I'm
- 13:00creating ranges. Here are my ranges.
- 13:07Okay? Again, I'm creating index values.
- 13:11And creating a third data frame. In the
- 13:14third data frame, you'll notice that
- 13:16you see zeros quite often, because this
- 13:19is the row with zeros where the error
- 13:21was occurring, so you'll see them quite
- 13:23often. Now the sample is done, okay,
- 13:26there's a lot of them. I'll train the
- 13:28third decision tree again, and here's
- 13:31its decision bound tree. It's also
- 13:34making some mistakes, precisely one
- 13:35mistake. We'll calculate the viability
- 13:44for this one as well and see how many
- 13:45mistakes it's making: one mistake, two
- 13:47mistakes, three mistakes, four, five,
- 13:49six, and seven. It's making a lot of
- 13:52mistakes, one mistake, two, three, four
- 13:55, five, six, and seven. Okay, so here
- 13:59we'll write the point 7, and its weight
- 14:05is negative, okay, it's logical too. Ah
- 14:08, so now here we have our three alphas,
- 14:14okay? Now we have to make a prediction.
- 14:16So what do we do when making a
- 14:18prediction? Let's pick any existing row
- 14:20. I'll show you this one first. One
- 14:24five should predict one, and 99 should
- 14:26predict zero. I already have these two,
- 14:29so I'm picking these. So look, this is
- 14:32my first query point. One five. I sent
- 14:35this to decision tree one first. The
- 14:38output is one. Then I sent it to
- 14:40decision tree two. The output is one.
- 14:43And then I sent it to three. The output
- 14:45is one. Now what do we do? We'll
- 14:47multiply alpha one by one. We'll
- 14:50multiply alpha two again by one, and
- 14:52we'll multiply alpha three by one. And
- 14:55this total is 1.09. Now I just need to
- 14:58take the sign of 1.09 and get one. So
- 15:04our output will be one. And if you
- 15:06remember, the output for this data
- 15:07point should also be one. Okay? Let's
- 15:10take the second point, nine, and nine.
- 15:11So, according to the first decision
- 15:13tree, the prediction should be one.
- 15:15According to the second, the prediction
- 15:17should be zero. And according to the
- 15:20third, the prediction should be zero.
- 15:22Okay? So, we'll multiply alpha by one.
- 15:26We'll multiply alpha two by -1, and
- 15:29alpha three by -1. I've already told
- 15:31you this. In the case of boosting, 0 is
- 15:34taken as -1. Okay? And now we're
- 15:36running this, and it's coming out as
- 15:39-25. If I put this number in the mp dot
- 15:44sign and run it, the output is -1,
- 15:47meaning the point is negative. And
- 15:51that's how this algorithm works. I ran
- 15:53it with all three decision stumps, but
- 15:55you could use 100 or 200. Now, what's
- 15:57your job after watching this video?
- 16:01I've written this code very simply. If
- 16:05you want, you can create a function in
- 16:07which you train as many decision stumps
- 16:10as you want, and you can create your
- 16:12own class to add a boost. Okay? So, I
- 16:15hope this is all clear to you. I'll
- 16:18give you the entire code. So, just
- 16:21spend some time with it, and that's it.
- 16:25Thanks for watching. Bye.
About this transcript
This page contains the full transcript of AdaBoost Algorithm | Code from Scratch by CampusX, generated from the public captions YouTube serves with the video. The transcript has 2,530 words across 363 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.