YouTube2Text

How Adaboost Classifier Works? | Geometric Intuition — Transcript

by CampusX · 2,838 words · 413 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Hi guys, my name is Nitish and welcome
  2. 0:03to my YouTube channel. Our next machine
  3. 0:06learning algorithm is AdaBoost. So,
  4. 0:09AdaBoost is a boosting algorithm. You
  5. 0:11might have heard of bagging and
  6. 0:12boosting, which fall under ensemble
  7. 0:14learning. And this is a class of
  8. 0:16algorithms that always performs very
  9. 0:18well. We will study other algorithms in
  10. 0:20boosting as well. We will study
  11. 0:22gradient boosting. We will also study
  12. 0:24XGBoost. But the very first algorithm
  13. 0:25that we will cover in boosting. Its
  14. 0:27name is AdaBoost. AdaBoost came out
  15. 0:29quite a long time ago, almost 20-25
  16. 0:32years ago, and when it first arrived,
  17. 0:34it was used a lot in face detection.
  18. 0:36Okay? Now, deep learning methods have
  19. 0:39arrived, so it was replaced there, but
  20. 0:41even today in machine learning,
  21. 0:43AdaBoost gets a lot of importance
  22. 0:45because the results it gives are
  23. 0:47excellent. Honestly, based on my
  24. 0:50personal experience, I can tell you
  25. 0:51that AdaBoost is a good algorithm. Your
  26. 0:53results turn out well. But most of the
  27. 0:56time, it is beaten by Random Forest,
  28. 0:58Gradient Boost, and XGBoost, etc. So,
  29. 1:00in the end, you might never even use
  30. 1:03AdaBoost. So, a question might come to
  31. 1:05your mind: why should we study it then?
  32. 1:07The answer is very simple. If you want
  33. 1:09to learn boosting, it starts with
  34. 1:11AdaBoost. If you know AdaBoost, it
  35. 1:13means you know 50%of boosting, so then
  36. 1:15while studying Gradient Boosting and
  37. 1:17XGBoost, you won't face as much trouble
  38. 1:19as you would without studying this.
  39. 1:21Okay? So that is why it is a good idea
  40. 1:23that you study AdaBoost carefully. I
  41. 1:26will make a complete video series kind
  42. 1:28of on AdaBoost. I have decided that I
  43. 1:31will make four videos on AdaBoost. The
  44. 1:34first video, which is this video, we
  45. 1:36will look at the core idea of AdaBoost
  46. 1:38in a classification scenario. Now,
  47. 1:41let's start this video. Now, before
  48. 1:43starting this video, I would like to
  49. 1:45tell you about three different topics.
  50. 1:47Because if you don't know these three
  51. 1:49topics, you might face some problems.
  52. 1:51So, the first thing is weak learners.
  53. 1:54What are weak learners? A weak learner
  54. 1:57is a machine learning model whose
  55. 1:59accuracy is very low. Or, to put it
  56. 2:02another way, it is just slightly more
  57. 2:04than 50%. 50%means luck; if you are
  58. 2:06tossing a coin and guessing if it will
  59. 2:09be heads or tails, your accuracy will
  60. 2:11also be nearly 50%. Because you are
  61. 2:13guessing. Right? If you have a machine
  62. 2:16learning algorithm whose accuracy is
  63. 2:18just over 50%, then obviously its
  64. 2:20performance is not very good. We call
  65. 2:23that a weak learner. Opposite to this
  66. 2:25is a strong learner, which has very
  67. 2:27good accuracy. 95%, 99%—meaning it
  68. 2:29can predict very accurately what is
  69. 2:32going to happen. Now, why are weak
  70. 2:34learners important in the context of
  71. 2:36AdaBoost? Because what do you do in
  72. 2:38AdaBoost? You combine multiple weak
  73. 2:40learners to build a large, powerful
  74. 2:43model. Okay? So that is why you should
  75. 2:45know the concept of weak learners. The
  76. 2:47second thing that we are going to
  77. 2:49discuss is decision stumps. Now,
  78. 2:51decision stumps are actually weak
  79. 2:53learners. There are different types of
  80. 2:55weak learners. One type of that is
  81. 2:57decision stumps. Now, a decision stump
  82. 3:00is a decision tree that has a maximum
  83. 3:02depth of one. Meaning you can only
  84. 3:05perform a split on a single node. Okay?
  85. 3:09This means it should be clear to you
  86. 3:11that this is a machine learning
  87. 3:13algorithm that is a weak learner
  88. 3:14because its result is not very good. I
  89. 3:17will quickly show you with an example.
  90. 3:19We have this dataset of students. CGPA,
  91. 3:22IQ, and whether they got placed or not.
  92. 3:26And I have these three students. I have
  93. 3:28also plotted them here on the graph.
  94. 3:30This is the CGPA axis. This is the IQ
  95. 3:33axis. Now, if you remember decision
  96. 3:34trees, then in a decision tree, what do
  97. 3:36you do? You make multiple splits in the
  98. 3:39data in this way. In this way, right,
  99. 3:42but in a decision stump, you can only
  100. 3:45make one single split. Only one split.
  101. 3:48So that one split will be either
  102. 3:50parallel to the CGPA axis or parallel
  103. 3:53to the IQ axis. Okay? So what do you do
  104. 3:55? You try both. You try making a split
  105. 3:59like this. You try making a split like
  106. 4:02this. You try making a split like this.
  107. 4:04In this direction, these are the only
  108. 4:06three splits possible. Similarly, if
  109. 4:08you talk about parallel to the IQ, one
  110. 4:10split is possible here. Everything here
  111. 4:13or here, one split is possible. One
  112. 4:15split is possible here and one split is
  113. 4:17possible here. Okay? These are the only
  114. 4:19splits. Right? Now, which split will
  115. 4:21you choose from these? You will choose
  116. 4:24the split that reduces your entropy the
  117. 4:28most. The one with the maximum
  118. 4:30information gain. So I hope decision
  119. 4:32stumps are clear to you. Now, a
  120. 4:34question might come to your mind: why
  121. 4:36did we study decision stumps? The
  122. 4:38reason for this is that although you
  123. 4:41can use any weak learner in AdaBoost.
  124. 4:43You can use any algorithm. You can even
  125. 4:46use neural networks. But 99%of the time
  126. 4:49, whenever you see someone using
  127. 4:51AdaBoost, which weak learner will they
  128. 4:53use internally? A decision stump. In
  129. 4:56fact, we will also use a decision stump
  130. 4:58as our weak learner throughout this
  131. 5:00entire four-video series. Because it
  132. 5:03gives the best results, and even more
  133. 5:05importantly, we get immediate
  134. 5:07interpretation of what our algorithm is
  135. 5:09doing. It's immediately clear what is
  136. 5:11happening. So, for these two reasons,
  137. 5:13you will mostly use decision stumps
  138. 5:15within AdaBoost. Alright? Now, the last
  139. 5:18point before starting: +1 and -1.
  140. 5:21Generally, whenever you work with any
  141. 5:23machine learning algorithm in a
  142. 5:25classification scenario, your classes
  143. 5:28in binary classification are 1 and 0.
  144. 5:31But here in AdaBoost, those classes are
  145. 5:34+1 and -1. Positive points are +1 and
  146. 5:38negative points are -1. They are not 0.
  147. 5:44They are -1. Why is that? You will
  148. 5:46understand in a little while. So, it
  149. 5:48was very important to understand these
  150. 5:50three points before starting. First,
  151. 5:52what is a weak learner? A machine
  152. 5:54learning model whose accuracy is just
  153. 5:56over 50%. What is a decision stump? A
  154. 5:59decision tree that has a maximum depth
  155. 6:01of one. Which has only one node. Third,
  156. 6:05+1 and -1 instead of 1 and 0. Alright?
  157. 6:08Now we are ready to start AdaBoost. So,
  158. 6:10let's start AdaBoost. So guys, first of
  159. 6:12all, let's start with the core idea of
  160. 6:15this entire algorithm. I will give you
  161. 6:17a geometric intuition. You will
  162. 6:19understand the big idea behind it.
  163. 6:22Alright? So, we have taken an example.
  164. 6:26Again, the same example, you could say
  165. 6:29we have data for, I don't know, nine
  166. 6:31students regarding their CGPA, IQ, and
  167. 6:34whether they got placed or not. So here
  168. 6:38, the minus points are the students who
  169. 6:40didn't get placed, and the plus points
  170. 6:42are those who got placed. Alright? We
  171. 6:45want to perform classification on this
  172. 6:48using AdaBoost. Alright? How will we do
  173. 6:50it? You have to keep watching. Before
  174. 6:53moving forward, I will introduce an
  175. 6:56important term here; whenever you read
  176. 6:59about AdaBoost, you will find it
  177. 7:02written that AdaBoost is a stage-wise
  178. 7:04additive method. Let's understand this
  179. 7:09additive method first. What is it? So,
  180. 7:11what is AdaBoost? It is formed by
  181. 7:14adding multiple weak learners. That is
  182. 7:17why the term "additive method" is used.
  183. 7:19And what does "stage-wise" mean? That
  184. 7:22all those weak learners are added one
  185. 7:24by one, sequentially. So, that is why
  186. 7:28this term: "stage-wise additive method.
  187. 7:31" Okay? Now, put the technicalities
  188. 7:33aside. Let's understand the core idea
  189. 7:36from scratch. So, suppose you have this
  190. 7:38dataset. Now, what will you do? You
  191. 7:40will fit a decision stump on this
  192. 7:43dataset. Which decision stump? There
  193. 7:45can be different decision stumps. Like
  194. 7:47this one, this one, this could also be
  195. 7:50one, or this, it could be any of them.
  196. 7:53There are many possibilities. You will
  197. 7:56fit the decision stump that results in
  198. 7:59the maximum reduction in entropy when
  199. 8:02applied here. Okay? Let's assume it was
  200. 8:05this one. This one. So, you created a
  201. 8:10decision stump. Creating which caused
  202. 8:14your data to be split in this way. This
  203. 8:17is the split. I have drawn the same
  204. 8:19dataset here. And here you can see this
  205. 8:22is the line that is cutting through
  206. 8:23your data. This is the decision stump
  207. 8:25that is splitting your data. And these
  208. 8:28are your two regions. This smaller one
  209. 8:30is the positive region, and this larger
  210. 8:31one is the negative region. Alright,
  211. 8:33this first decision stump has done its
  212. 8:35job. But if you analyze its results,
  213. 8:38you will notice that there are three
  214. 8:40points here that are incorrectly
  215. 8:42classified. Basically, these three plus
  216. 8:45points that are in the red region,
  217. 8:47where should they have been? In the
  218. 8:49blue region. Right? So now, what you
  219. 8:52will do is, the first decision stump's
  220. 8:55work is done. Now you will tell the
  221. 8:59next stump, "Look, I made mistakes on
  222. 9:01these three points." So you should
  223. 9:05handle these three points a bit more
  224. 9:07carefully. Okay? So what happens is,
  225. 9:11when you send the data to the next
  226. 9:13stage, you increase the value of these
  227. 9:15points—value in the sense, this will
  228. 9:17be clearer in the next video. We just
  229. 9:20increase the focus on them. How is the
  230. 9:23focus increased? That is not our
  231. 9:25concern at this point. There is a
  232. 9:27technique called upsampling. We perform
  233. 9:32upsampling. In a way, we just increase
  234. 9:34the value or importance of these three
  235. 9:36points in the data by any means. So
  236. 9:39that the next decision stump focuses
  237. 9:42more on these three. That is the core
  238. 9:44idea. Okay? Now, as soon as the first
  239. 9:46decision stump was applied, it made
  240. 9:48some mistakes. Here is that decision
  241. 9:50stump. I have drawn it right here. At
  242. 9:52this stage, at this point, you do one
  243. 9:54more thing. You calculate a weight for
  244. 9:58this particular decision stump. What is
  245. 10:01calculating weight? That finally, when
  246. 10:05the whole model is ready, how much say
  247. 10:08will this particular model have when
  248. 10:10making predictions? How much say will
  249. 10:13this decision stump have? Here,
  250. 10:16boosting is a bit different from
  251. 10:18bagging. What happens in bagging is
  252. 10:21that all your base classifiers are
  253. 10:22equal. Meaning, I give everyone a
  254. 10:25weight of one. It's like a democracy;
  255. 10:27everyone is equal. No matter how rich
  256. 10:29or how poor, they are equal. That's how
  257. 10:31it is in bagging. But it is not like
  258. 10:33that in boosting. In boosting, what
  259. 10:34happens is that depending on your
  260. 10:36performance on the data set. Based on
  261. 10:38how many things you get right and how
  262. 10:40many mistakes you make, you are
  263. 10:41assigned a weight. And at the end, when
  264. 10:43you go to make a prediction, this model
  265. 10:45will have a say based on this weight
  266. 10:47during prediction. I hope you are
  267. 10:49understanding. If not, you will
  268. 10:51understand in a little while. Anyways,
  269. 10:53we are now starting stage two. In stage
  270. 10:55two, we have made a small change to the
  271. 10:57data. What change? These points have
  272. 11:00now been up-sampled. Meaning, we have
  273. 11:02increased their importance in the data.
  274. 11:04Increased the value. Increased it. Now
  275. 11:07in the second stage, we have applied a
  276. 11:09decision stump again. Again, different
  277. 11:12decision stumps could have been formed.
  278. 11:14But this time, the decision stump that
  279. 11:16is giving the most reduction in entropy
  280. 11:18is this one. This vertical split from
  281. 11:21this spot. Okay? By the way, these are
  282. 11:23red points here. So, this split is
  283. 11:25giving the best result. Okay? So, I
  284. 11:27have drawn it here. This is our second
  285. 11:29base model. This is our second weak
  286. 11:31learner. This is our second decision
  287. 11:33stump. Okay? Now, if you look carefully
  288. 11:37, this time this stump classified these
  289. 11:39three points correctly, which were
  290. 11:41misclassified here, but it made some of
  291. 11:44its own mistakes. Look, these two red
  292. 11:48points, instead of being in the red
  293. 11:49region, are now in the blue region, so
  294. 11:51this is also a mistake. So, you will do
  295. 11:54two things here again: the first thing
  296. 11:56you will do is up-sample these two
  297. 11:57points and send them to the next stage,
  298. 11:59okay? And the second point will be that
  299. 12:03depending on its performance, you will
  300. 12:04calculate alpha 2 for it. What is alpha
  301. 12:072? The weightage of the second decision
  302. 12:10stump in the overall model. At the end,
  303. 12:13when making a prediction, alpha 2 will
  304. 12:15determine how much say the second
  305. 12:17decision stump has. If it had made
  306. 12:19predictions very well here, then alpha
  307. 12:21would be very high. If its predictions
  308. 12:23are very poor, then this alpha will be
  309. 12:25very low. It is a simple matter. Right?
  310. 12:28Next, we will come to stage three. Now
  311. 12:31in stage three, you see we have
  312. 12:33up-sampled these two. Okay? Because of
  313. 12:37which, notice again that a decision
  314. 12:39stump was formed that correctly
  315. 12:41classified these two points but made
  316. 12:44three more mistakes. Right? So now
  317. 12:48again, you will do two things here.
  318. 12:50What is the first thing you will do?
  319. 12:52You will calculate alpha 3. Depending
  320. 12:54on the performance of this particular
  321. 12:56model, you will derive alpha 3. And
  322. 12:58then, if you have more stages, meaning
  323. 13:00if you have to use more base models,
  324. 13:02then again you will up-sample these
  325. 13:04three points and send them to the next
  326. 13:07stage. But we are not doing any more. I
  327. 13:09guess you have understood the idea. We
  328. 13:11are building an AdaBoost with only
  329. 13:13three models. So our work is finished
  330. 13:15here. Now, if you understand at this
  331. 13:17point, what all do you have? You have
  332. 13:20all three base models. You have all
  333. 13:23three decision stumps, and you have
  334. 13:26alpha 1, alpha 2, and alpha 3 for them.
  335. 13:29Now what do you have to do? You have to
  336. 13:31add these three together. Okay? And
  337. 13:33what is the formula for adding them?
  338. 13:35Very simple. That formula is this. You
  339. 13:38can call your first decision stump h1x.
  340. 13:43You can write it in this notation
  341. 13:44because, in the end, any machine
  342. 13:46learning algorithm is a mathematical
  343. 13:47function. We are calling this function
  344. 13:50h. Hypothesis function, and x is our
  345. 13:52input. + h2x + h3x And this is my
  346. 14:02overall AdaBoost function. Just one
  347. 14:06last thing here: you will multiply this
  348. 14:08function by alpha 1. This one by alpha
  349. 14:122, and this one by alpha 3, and one
  350. 14:15last thing: you will take the sign of
  351. 14:18this whole thing. Meaning, after adding
  352. 14:21this whole thing, if you get a positive
  353. 14:24result, your answer will be +1, and if
  354. 14:27you get a negative result, your answer
  355. 14:29will be -1. So basically, here you
  356. 14:32calculate the sign. And this is the
  357. 14:36function. This is the function. Okay?
  358. 14:40Now let me show you how to make a
  359. 14:42prediction. Okay? Now suppose someone
  360. 14:44asked me, here is a student whose CGPA
  361. 14:46is 7.5. And their IQ is 81. You have to
  362. 14:52tell whether they will get placed or
  363. 14:54not. So, what will you do now? You will
  364. 14:57first bring this 7.5 and 81 and plot it
  365. 14:59here. Let's assume it came here. Okay?
  366. 15:03So, according to this model, this
  367. 15:05person should not get placed. So, it
  368. 15:07will be -1. Then you plot this point on
  369. 15:12this decision tree, and it came here,
  370. 15:14so it says they will get placed. And
  371. 15:18then, according to the third one, it
  372. 15:20came here again, so it says they won't.
  373. 15:24Now, it turned out that our weight
  374. 15:27alpha 1 was two. Alpha 2 was 10 and
  375. 15:32alpha 3 was one. Okay? So, what will
  376. 15:36you do now? You will put these values
  377. 15:38into this formula. So, what does alpha
  378. 15:391 become? 2 times -1 plus, what does
  379. 15:43alpha 2 become? 10 times 1 + what is
  380. 15:49alpha 3? 1 times -1, and now when you
  381. 15:52calculate this, it will clearly be a
  382. 15:54positive number. And what is the sign
  383. 15:57of eight? Positive. So, the answer will
  384. 16:00be +1. Meaning, this student should get
  385. 16:03placed. That is how AdaBoost works.
  386. 16:06Prediction works exactly like this.
  387. 16:08Okay? Now, if you want to get a
  388. 16:10geometric intuition of why this works
  389. 16:12correctly, it is on your screen. If you
  390. 16:14stack these three on top of each other,
  391. 16:16don't you think you will see a decision
  392. 16:19boundary like this? Like this, and now
  393. 16:21look at this data. If you think here,
  394. 16:25you actually wanted a decision boundary
  395. 16:29similar to this. Where this would be
  396. 16:33the blue region. And this would be the
  397. 16:35red region. This would be your best
  398. 16:37classification. Right? And by using
  399. 16:39these three, exactly this thing is
  400. 16:41being formed. Obviously, the more
  401. 16:44models you keep adding, the more this
  402. 16:46algorithm will move towards perfection.
  403. 16:48But yeah, this is the entire geometric
  404. 16:51intuition of this algorithm. I hope I
  405. 16:53was able to explain it to you. Now, we
  406. 16:55will study this thing in detail, step
  407. 16:58by step. Like, how these weights are
  408. 17:00being calculated, we don't know that.
  409. 17:02How this up-sampling is happening, we
  410. 17:04don't know that either. So, some of
  411. 17:06these key questions that remain, we
  412. 17:08will discuss them shortly in the next
  413. 17:11video. Thanks for watching. Bye.

About this transcript

This page contains the full transcript of How Adaboost Classifier Works? | Geometric Intuition by CampusX, generated from the public captions YouTube serves with the video. The transcript has 2,838 words across 413 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.