AdaBoost Hyperparameters | GridSearchCV in Adaboost — Transcript
Full transcript
- 0:00Hello guys, my name is Nitish. Welcome
- 0:02to my YouTube channel. We are studying
- 0:04the AdaBoost algorithm. In the last
- 0:07three videos, we discussed in detail
- 0:09how AdaBoost works. Now in this video,
- 0:12we will learn about AdaBoost's
- 0:15hyperparameters, how to implement
- 0:17AdaBoost in scikit-learn, and finally,
- 0:20we will also perform hyperparameter
- 0:23tuning for this algorithm. So, let's
- 0:26start. Unlike Gradient Boosting and
- 0:30XGBoost, where there are many
- 0:31hyperparameters. AdaBoost is a slightly
- 0:35simpler algorithm. You won't find that
- 0:37many hyperparameters here. Right? So,
- 0:39precisely there are only three to four
- 0:41hyperparameters that you need to study.
- 0:44I will give you a quick introduction to
- 0:45these hyperparameters. Then we will
- 0:48implement them practically. So here,
- 0:50you have the main four hyperparameters.
- 0:55I am not counting the last one, random
- 0:58state, because you only use random
- 1:01state for result consistency. The first
- 1:04one is the base estimator. It is a
- 1:07boosting algorithm. If you want, you
- 1:11can use an algorithm other than a
- 1:14decision tree or decision stump. But
- 1:19mostly, you will see the best results
- 1:22with a decision tree having a max depth
- 1:25equal to one. That is why it says here:
- 1:28the base estimator from which the
- 1:31boosted ensemble is built. Support for
- 1:34sample weighting is required, as well
- 1:36as proper classes and n_classes
- 1:38attributes. If None, then the base
- 1:41estimator is a DecisionTreeClassifier
- 1:44initialized with max_depth = 1. Right?
- 1:47You can use logistic regression here.
- 1:50You can use SVM here. KNN doesn't work
- 1:53here because it doesn't have the
- 1:54feature to take sample weights. So,
- 1:57because of that, KNN does not work here
- 1:59. But again, practically speaking, 99.9
- 2:03%of the time, you only need to use
- 2:05decision trees. So, this is a parameter
- 2:08that you don't really touch much. Right
- 2:11? The next parameter is perhaps the
- 2:13most important one. The n_estimators
- 2:16hyperparameter. Basically, how many
- 2:19base models you want to use. Right? So,
- 2:22it is written here: the maximum number
- 2:24of estimators at which boosting is
- 2:26terminated. In case of perfect fit, the
- 2:28learning procedure is stopped early.
- 2:31Right? If it finds a decision tree in
- 2:34between that makes zero mistakes, then
- 2:36your learning stops right there. But if
- 2:39it doesn't find such a thing, then this
- 2:42many base models will be trained. All
- 2:45right? I hope you are understanding
- 2:47this. Next is learning rate. I haven't
- 2:49taught you this yet. Learning rate is
- 2:52basically the rate applied to each
- 2:54classifier at each boosting iteration.
- 2:58A higher learning rate increases the
- 3:00contribution of each classifier. There
- 3:02is a trade-off between the learning
- 3:04rate and n_estimators parameter. So,
- 3:06the entire hyperparameter tuning here
- 3:08is based on these two. All right? I
- 3:11will tell you more about the learning
- 3:13rate in detail shortly. Next is
- 3:15algorithm. There are only two
- 3:17algorithms here. One is SAMME and one
- 3:19is SAMME.R. All right? By default, you
- 3:21should use SAMME.R. It is written here.
- 3:24It typically converges faster than
- 3:26SAMME and achieves a lower test error
- 3:28with fewer boosting iterations. So,
- 3:31most of the time it will help you.
- 3:34Otherwise, you can use GridSearchCV to
- 3:36check. All right? So, these are the
- 3:38three or four hyperparameters we have.
- 3:41Now, I will show you practically how to
- 3:43tune them. So, if you go here, I have
- 3:46created a notebook. Here I have created
- 3:48a dataset of my own. A classification
- 3:51dataset that looks like this. And what
- 3:54did I do on this? I applied an AdaBoost
- 3:57classifier without any hyperparameter
- 3:59tuning. I used the default values.
- 4:02Meaning decision tree max depth 1, 50
- 4:04estimators, learning rate value 1, and
- 4:07the algorithm value. All right? And
- 4:10when I trained it, my accuracy came out
- 4:14to be 78. All right? After that, I
- 4:17plotted it and after plotting, this is
- 4:21my output on that data. And you can see
- 4:24some mistakes are happening. Some
- 4:26yellow points are in the blue region
- 4:28and some blue points are in the yellow
- 4:29region. All right? Now, let's talk
- 4:32about these hyperparameters one by one.
- 4:35First of all, we will talk about
- 4:37n_estimators, what happens if you
- 4:39increase or decrease n_estimators? So,
- 4:43first of all, I will do one thing. I
- 4:45will set n_estimators equal to 1 here.
- 4:47Meaning, I am taking only one base
- 4:48model, one decision stump. I hope you
- 4:50can imagine what will happen. This is
- 4:52the output. This is your decision
- 4:55boundary. And obviously, it is way too
- 4:57simplistic and this is a clear-cut case
- 5:00of underfitting. All right? It is not
- 5:02working well. But as you increase it,
- 5:06you would see that it will show some
- 5:09complex behavior. This is 50, which we
- 5:12just saw earlier. If you set this to
- 5:15150, you will start seeing overfitting
- 5:18behavior. Small regions will start to
- 5:20form. Which indicates that you are
- 5:22overfitting. All right? So, if you set
- 5:25it to 500 here, it starts showing too
- 5:28much overfitting. All right? If you set
- 5:30it to 1500, it will be a clear-cut case
- 5:32of overfitting and it will take more
- 5:34time because you have to train that
- 5:36many decision stumps. Look, these are
- 5:39very tiny regions and this is not a
- 5:40good thing. So, you need to find an
- 5:43optimum value for n_estimators. Too low
- 5:45leads to underfitting. Too high leads
- 5:47to overfitting. You need to find a
- 5:49value in between. All right? Now, here
- 5:52comes an interesting thing, and that is
- 5:54the learning rate. So, if you remember,
- 5:58I told you that whenever you calculate
- 6:01alpha for any decision stump, its
- 6:04formula is 1/2*log ((1-error)/error).
- 6:09All right? Now, what is the learning
- 6:10rate? The learning rate is, if you
- 6:14remember, what did we used to do? When
- 6:19we calculated alpha, that alpha was 1/2
- 6:26*log ((1-error)/error). Now, what the
- 6:33learning rate does is that it gets
- 6:35multiplied by alpha. All right? It gets
- 6:39multiplied by alpha. So, by default,
- 6:42the value of your learning rate is one.
- 6:45That is why your alpha remains as it is
- 6:47. But you can increase or decrease the
- 6:49LR value if you want. Generally, you
- 6:51decrease it, making it less than one.
- 6:53All right? Suppose, what did you do?
- 6:54You set it to 0.1. Setting it to 0.1
- 6:59will cause your alpha to also be
- 7:01reduced 10 times. Now, think about what
- 7:04reducing alpha does. Reducing the
- 7:07alphas leads to something called
- 7:09shrinkage. You will read more about
- 7:11this in boosting later. What happens in
- 7:13shrinkage? A very good explanation is
- 7:15given here. See, it is written here
- 7:17that decreasing the learning rate makes
- 7:19the coefficients, alpha_m, smaller. All
- 7:22right? Just as I told you, the value of
- 7:24alpha will decrease. Now, what will
- 7:26happen because the value of alpha
- 7:27decreases? It will reduce the amplitude
- 7:30of sample weights at each stage. Now, I
- 7:34told you that what do you have to do?
- 7:36You have to use this alpha to update
- 7:38the weights. How? Current weight
- 7:42multiplied by e to the power minus
- 7:44alpha for your correctly classified
- 7:47points, and current weight multiplied
- 7:50by e to the power alpha for
- 7:52misclassified points. Now think about
- 7:55it. If this alpha decreases, if this
- 7:59alpha reduces, then the amplitude of
- 8:02your update will decrease. A decrease
- 8:06in amplitude means if it was growing in
- 8:09the negative, it will grow less in the
- 8:12negative. If it was growing in the
- 8:15positive, it will grow less in the
- 8:16positive as well. That is what a
- 8:18decrease in amplitude means. And that
- 8:21translates into smaller variations of
- 8:23weighted data points, and therefore
- 8:25fewer differences between the weak
- 8:27classifiers 'decision boundaries. So
- 8:31basically, if you keep adding more and
- 8:34more weak learners, its effect is
- 8:36reduced. Right? So in short, you can
- 8:40say, as written here, the idea is to
- 8:43slow down learning to dramatically
- 8:46reduce overfitting. So the core concept
- 8:49here is that you can increase the
- 8:52n_estimators. What happens by
- 8:54increasing them? Overfitting. But what
- 8:56can you do here? You can decrease the
- 8:58learning rate. By decreasing the
- 9:00learning rate to one, your overfitting
- 9:02will be reduced. Because as soon as you
- 9:04decrease the learning rate, the alpha
- 9:06decreases. With the decrease in alpha,
- 9:10the weights being updated are now
- 9:12updating less, and because their
- 9:13amplitude is reduced, learning is not
- 9:16happening as fast. The speed of
- 9:19learning decreased, so the speed of
- 9:21overfitting will also decrease, and the
- 9:23resulting output you get will look like
- 9:25this. Let me show you. Look here, there
- 9:30is some smoothing happening. Right? So
- 9:32this is called the shrinkage parameter.
- 9:34Right? You can use the learning rate
- 9:37very intelligently to avoid overfitting
- 9:39while using AdaBoost. Right? Many
- 9:42people don't use this parameter. But if
- 9:46you use n_estimators and the learning
- 9:48rate together, you can unleash the true
- 9:51power of AdaBoost. Right? Now, the last
- 9:54thing I want to show you quickly is how
- 9:56you can apply GridSearch CV on this. So
- 9:58what did I do? I created a grid
- 10:00dictionary where I put some values for
- 10:02n_estimators. You can try even more
- 10:05values. For the learning rate, I tried
- 10:07values from very low up to one and
- 10:09added both variations of the algorithm.
- 10:11Created a class called GridSearch CV.
- 10:13Passed the estimator value there.
- 10:15Passed my grid into the AdaBoost
- 10:17parameter grid. Set n_jobs to minus one
- 10:20. Because the process will become a bit
- 10:22faster. Cross-validation 10 and scoring
- 10:25accuracy, and now I am storing the grid
- 10:28results here. And this is my output,
- 10:31guys: 83. Before, when I did it
- 10:34manually, it was coming out to be 78,
- 10:37but by applying GridSearchCV, I was
- 10:40able to reach 83. The algorithm, in
- 10:43this case, is surprisingly the same,
- 10:45and the learning rate of one is the
- 10:47best, with 500 n-estimators. Okay? So,
- 10:50tuning and using AdaBoost is not that
- 10:53difficult. It is one of those easier
- 10:56boosting algorithms that, if you have a
- 10:58little understanding of these two
- 11:00parameters, you can use very easily.
- 11:03Okay? Now, you should also pick up a
- 11:05dataset and try it out. Apply
- 11:07everything you learned in this video. I
- 11:09hope you liked the video. Thanks for
- 11:11watching. Bye.
About this transcript
This page contains the full transcript of AdaBoost Hyperparameters | GridSearchCV in Adaboost by CampusX, generated from the public captions YouTube serves with the video. The transcript has 1,678 words across 257 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.