YouTube2Text

AdaBoost Hyperparameters | GridSearchCV in Adaboost — Transcript

by CampusX · 1,678 words · 257 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Hello guys, my name is Nitish. Welcome
  2. 0:02to my YouTube channel. We are studying
  3. 0:04the AdaBoost algorithm. In the last
  4. 0:07three videos, we discussed in detail
  5. 0:09how AdaBoost works. Now in this video,
  6. 0:12we will learn about AdaBoost's
  7. 0:15hyperparameters, how to implement
  8. 0:17AdaBoost in scikit-learn, and finally,
  9. 0:20we will also perform hyperparameter
  10. 0:23tuning for this algorithm. So, let's
  11. 0:26start. Unlike Gradient Boosting and
  12. 0:30XGBoost, where there are many
  13. 0:31hyperparameters. AdaBoost is a slightly
  14. 0:35simpler algorithm. You won't find that
  15. 0:37many hyperparameters here. Right? So,
  16. 0:39precisely there are only three to four
  17. 0:41hyperparameters that you need to study.
  18. 0:44I will give you a quick introduction to
  19. 0:45these hyperparameters. Then we will
  20. 0:48implement them practically. So here,
  21. 0:50you have the main four hyperparameters.
  22. 0:55I am not counting the last one, random
  23. 0:58state, because you only use random
  24. 1:01state for result consistency. The first
  25. 1:04one is the base estimator. It is a
  26. 1:07boosting algorithm. If you want, you
  27. 1:11can use an algorithm other than a
  28. 1:14decision tree or decision stump. But
  29. 1:19mostly, you will see the best results
  30. 1:22with a decision tree having a max depth
  31. 1:25equal to one. That is why it says here:
  32. 1:28the base estimator from which the
  33. 1:31boosted ensemble is built. Support for
  34. 1:34sample weighting is required, as well
  35. 1:36as proper classes and n_classes
  36. 1:38attributes. If None, then the base
  37. 1:41estimator is a DecisionTreeClassifier
  38. 1:44initialized with max_depth = 1. Right?
  39. 1:47You can use logistic regression here.
  40. 1:50You can use SVM here. KNN doesn't work
  41. 1:53here because it doesn't have the
  42. 1:54feature to take sample weights. So,
  43. 1:57because of that, KNN does not work here
  44. 1:59. But again, practically speaking, 99.9
  45. 2:03%of the time, you only need to use
  46. 2:05decision trees. So, this is a parameter
  47. 2:08that you don't really touch much. Right
  48. 2:11? The next parameter is perhaps the
  49. 2:13most important one. The n_estimators
  50. 2:16hyperparameter. Basically, how many
  51. 2:19base models you want to use. Right? So,
  52. 2:22it is written here: the maximum number
  53. 2:24of estimators at which boosting is
  54. 2:26terminated. In case of perfect fit, the
  55. 2:28learning procedure is stopped early.
  56. 2:31Right? If it finds a decision tree in
  57. 2:34between that makes zero mistakes, then
  58. 2:36your learning stops right there. But if
  59. 2:39it doesn't find such a thing, then this
  60. 2:42many base models will be trained. All
  61. 2:45right? I hope you are understanding
  62. 2:47this. Next is learning rate. I haven't
  63. 2:49taught you this yet. Learning rate is
  64. 2:52basically the rate applied to each
  65. 2:54classifier at each boosting iteration.
  66. 2:58A higher learning rate increases the
  67. 3:00contribution of each classifier. There
  68. 3:02is a trade-off between the learning
  69. 3:04rate and n_estimators parameter. So,
  70. 3:06the entire hyperparameter tuning here
  71. 3:08is based on these two. All right? I
  72. 3:11will tell you more about the learning
  73. 3:13rate in detail shortly. Next is
  74. 3:15algorithm. There are only two
  75. 3:17algorithms here. One is SAMME and one
  76. 3:19is SAMME.R. All right? By default, you
  77. 3:21should use SAMME.R. It is written here.
  78. 3:24It typically converges faster than
  79. 3:26SAMME and achieves a lower test error
  80. 3:28with fewer boosting iterations. So,
  81. 3:31most of the time it will help you.
  82. 3:34Otherwise, you can use GridSearchCV to
  83. 3:36check. All right? So, these are the
  84. 3:38three or four hyperparameters we have.
  85. 3:41Now, I will show you practically how to
  86. 3:43tune them. So, if you go here, I have
  87. 3:46created a notebook. Here I have created
  88. 3:48a dataset of my own. A classification
  89. 3:51dataset that looks like this. And what
  90. 3:54did I do on this? I applied an AdaBoost
  91. 3:57classifier without any hyperparameter
  92. 3:59tuning. I used the default values.
  93. 4:02Meaning decision tree max depth 1, 50
  94. 4:04estimators, learning rate value 1, and
  95. 4:07the algorithm value. All right? And
  96. 4:10when I trained it, my accuracy came out
  97. 4:14to be 78. All right? After that, I
  98. 4:17plotted it and after plotting, this is
  99. 4:21my output on that data. And you can see
  100. 4:24some mistakes are happening. Some
  101. 4:26yellow points are in the blue region
  102. 4:28and some blue points are in the yellow
  103. 4:29region. All right? Now, let's talk
  104. 4:32about these hyperparameters one by one.
  105. 4:35First of all, we will talk about
  106. 4:37n_estimators, what happens if you
  107. 4:39increase or decrease n_estimators? So,
  108. 4:43first of all, I will do one thing. I
  109. 4:45will set n_estimators equal to 1 here.
  110. 4:47Meaning, I am taking only one base
  111. 4:48model, one decision stump. I hope you
  112. 4:50can imagine what will happen. This is
  113. 4:52the output. This is your decision
  114. 4:55boundary. And obviously, it is way too
  115. 4:57simplistic and this is a clear-cut case
  116. 5:00of underfitting. All right? It is not
  117. 5:02working well. But as you increase it,
  118. 5:06you would see that it will show some
  119. 5:09complex behavior. This is 50, which we
  120. 5:12just saw earlier. If you set this to
  121. 5:15150, you will start seeing overfitting
  122. 5:18behavior. Small regions will start to
  123. 5:20form. Which indicates that you are
  124. 5:22overfitting. All right? So, if you set
  125. 5:25it to 500 here, it starts showing too
  126. 5:28much overfitting. All right? If you set
  127. 5:30it to 1500, it will be a clear-cut case
  128. 5:32of overfitting and it will take more
  129. 5:34time because you have to train that
  130. 5:36many decision stumps. Look, these are
  131. 5:39very tiny regions and this is not a
  132. 5:40good thing. So, you need to find an
  133. 5:43optimum value for n_estimators. Too low
  134. 5:45leads to underfitting. Too high leads
  135. 5:47to overfitting. You need to find a
  136. 5:49value in between. All right? Now, here
  137. 5:52comes an interesting thing, and that is
  138. 5:54the learning rate. So, if you remember,
  139. 5:58I told you that whenever you calculate
  140. 6:01alpha for any decision stump, its
  141. 6:04formula is 1/2*log ((1-error)/error).
  142. 6:09All right? Now, what is the learning
  143. 6:10rate? The learning rate is, if you
  144. 6:14remember, what did we used to do? When
  145. 6:19we calculated alpha, that alpha was 1/2
  146. 6:26*log ((1-error)/error). Now, what the
  147. 6:33learning rate does is that it gets
  148. 6:35multiplied by alpha. All right? It gets
  149. 6:39multiplied by alpha. So, by default,
  150. 6:42the value of your learning rate is one.
  151. 6:45That is why your alpha remains as it is
  152. 6:47. But you can increase or decrease the
  153. 6:49LR value if you want. Generally, you
  154. 6:51decrease it, making it less than one.
  155. 6:53All right? Suppose, what did you do?
  156. 6:54You set it to 0.1. Setting it to 0.1
  157. 6:59will cause your alpha to also be
  158. 7:01reduced 10 times. Now, think about what
  159. 7:04reducing alpha does. Reducing the
  160. 7:07alphas leads to something called
  161. 7:09shrinkage. You will read more about
  162. 7:11this in boosting later. What happens in
  163. 7:13shrinkage? A very good explanation is
  164. 7:15given here. See, it is written here
  165. 7:17that decreasing the learning rate makes
  166. 7:19the coefficients, alpha_m, smaller. All
  167. 7:22right? Just as I told you, the value of
  168. 7:24alpha will decrease. Now, what will
  169. 7:26happen because the value of alpha
  170. 7:27decreases? It will reduce the amplitude
  171. 7:30of sample weights at each stage. Now, I
  172. 7:34told you that what do you have to do?
  173. 7:36You have to use this alpha to update
  174. 7:38the weights. How? Current weight
  175. 7:42multiplied by e to the power minus
  176. 7:44alpha for your correctly classified
  177. 7:47points, and current weight multiplied
  178. 7:50by e to the power alpha for
  179. 7:52misclassified points. Now think about
  180. 7:55it. If this alpha decreases, if this
  181. 7:59alpha reduces, then the amplitude of
  182. 8:02your update will decrease. A decrease
  183. 8:06in amplitude means if it was growing in
  184. 8:09the negative, it will grow less in the
  185. 8:12negative. If it was growing in the
  186. 8:15positive, it will grow less in the
  187. 8:16positive as well. That is what a
  188. 8:18decrease in amplitude means. And that
  189. 8:21translates into smaller variations of
  190. 8:23weighted data points, and therefore
  191. 8:25fewer differences between the weak
  192. 8:27classifiers 'decision boundaries. So
  193. 8:31basically, if you keep adding more and
  194. 8:34more weak learners, its effect is
  195. 8:36reduced. Right? So in short, you can
  196. 8:40say, as written here, the idea is to
  197. 8:43slow down learning to dramatically
  198. 8:46reduce overfitting. So the core concept
  199. 8:49here is that you can increase the
  200. 8:52n_estimators. What happens by
  201. 8:54increasing them? Overfitting. But what
  202. 8:56can you do here? You can decrease the
  203. 8:58learning rate. By decreasing the
  204. 9:00learning rate to one, your overfitting
  205. 9:02will be reduced. Because as soon as you
  206. 9:04decrease the learning rate, the alpha
  207. 9:06decreases. With the decrease in alpha,
  208. 9:10the weights being updated are now
  209. 9:12updating less, and because their
  210. 9:13amplitude is reduced, learning is not
  211. 9:16happening as fast. The speed of
  212. 9:19learning decreased, so the speed of
  213. 9:21overfitting will also decrease, and the
  214. 9:23resulting output you get will look like
  215. 9:25this. Let me show you. Look here, there
  216. 9:30is some smoothing happening. Right? So
  217. 9:32this is called the shrinkage parameter.
  218. 9:34Right? You can use the learning rate
  219. 9:37very intelligently to avoid overfitting
  220. 9:39while using AdaBoost. Right? Many
  221. 9:42people don't use this parameter. But if
  222. 9:46you use n_estimators and the learning
  223. 9:48rate together, you can unleash the true
  224. 9:51power of AdaBoost. Right? Now, the last
  225. 9:54thing I want to show you quickly is how
  226. 9:56you can apply GridSearch CV on this. So
  227. 9:58what did I do? I created a grid
  228. 10:00dictionary where I put some values for
  229. 10:02n_estimators. You can try even more
  230. 10:05values. For the learning rate, I tried
  231. 10:07values from very low up to one and
  232. 10:09added both variations of the algorithm.
  233. 10:11Created a class called GridSearch CV.
  234. 10:13Passed the estimator value there.
  235. 10:15Passed my grid into the AdaBoost
  236. 10:17parameter grid. Set n_jobs to minus one
  237. 10:20. Because the process will become a bit
  238. 10:22faster. Cross-validation 10 and scoring
  239. 10:25accuracy, and now I am storing the grid
  240. 10:28results here. And this is my output,
  241. 10:31guys: 83. Before, when I did it
  242. 10:34manually, it was coming out to be 78,
  243. 10:37but by applying GridSearchCV, I was
  244. 10:40able to reach 83. The algorithm, in
  245. 10:43this case, is surprisingly the same,
  246. 10:45and the learning rate of one is the
  247. 10:47best, with 500 n-estimators. Okay? So,
  248. 10:50tuning and using AdaBoost is not that
  249. 10:53difficult. It is one of those easier
  250. 10:56boosting algorithms that, if you have a
  251. 10:58little understanding of these two
  252. 11:00parameters, you can use very easily.
  253. 11:03Okay? Now, you should also pick up a
  254. 11:05dataset and try it out. Apply
  255. 11:07everything you learned in this video. I
  256. 11:09hope you liked the video. Thanks for
  257. 11:11watching. Bye.

About this transcript

This page contains the full transcript of AdaBoost Hyperparameters | GridSearchCV in Adaboost by CampusX, generated from the public captions YouTube serves with the video. The transcript has 1,678 words across 257 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.