YouTube2Text

AdaBoost - A Step by Step Explanation — Transcript

by CampusX · 3,028 words · 425 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Hi guys, my name is Nitish and welcome
  2. 0:02to my YouTube channel. In the last
  3. 0:04video, we studied the core intuition of
  4. 0:06AdaBoost. In this video, I am going to
  5. 0:09show you how AdaBoost works step by
  6. 0:11step. You can watch this video in two
  7. 0:13parts. In the first part, I will show
  8. 0:16you all the theory on the whiteboard,
  9. 0:18and then in the second part, I will
  10. 0:20demonstrate the practical
  11. 0:21implementation of that theory through
  12. 0:23code. So let's start the video. Also,
  13. 0:27if you haven't subscribed to my channel
  14. 0:30yet, please subscribe, and now we will
  15. 0:32start this whole thing. So, to
  16. 0:36understand AdaBoost step by step, I
  17. 0:38have first taken a toy dataset with two
  18. 0:40input columns, x1 and x2, and one
  19. 0:42output column, y. This means it is a
  20. 0:45classification problem because y only
  21. 0:48contains one and zero. So now, what we
  22. 0:51will do is, first of all, we know that
  23. 0:54n = 5 here. The number of rows in the
  24. 0:58dataset is five. This is a small toy
  25. 1:00dataset. Okay? So, speaking step by
  26. 1:04step, the very first step in AdaBoost
  27. 1:07is that you initially assign a weight
  28. 1:10to each of your rows. Okay? And
  29. 1:14initially, when you assign weights, the
  30. 1:16weight of all rows should be equal.
  31. 1:18Okay? So, what will we do? To calculate
  32. 1:21that weight, we will give each row a
  33. 1:24weight of 1/5, as in 0.2. Generally,
  34. 1:29whenever you start, what do you do? You
  35. 1:32do 1/n. What is n? The total number of
  36. 1:36rows. And this is your weight. The
  37. 1:38initial weight of every row will be 1/n
  38. 1:41. If you add all these 1/n, 1/n, 1/n, n
  39. 1:44times, you will get n. I am sorry, you
  40. 1:46will get one. Okay? So, what am I doing
  41. 1:49now? I am creating a column named '
  42. 1:52weight' here. And here I am writing 0.2
  43. 1:58, 0.2, 0.2, 0.2, and 0.2. Okay? Now,
  44. 2:04the main work starts from here. Now we
  45. 2:06will enter stage one. First of all, we
  46. 2:09will enter stage one. In stage one, the
  47. 2:13first thing you will do is train a
  48. 2:16decision tree on this dataset. The
  49. 2:20specialty of that decision tree will be
  50. 2:22that its max depth will be one. Meaning
  51. 2:24, it will be a decision stump. Okay?
  52. 2:27Now, how will it work on this? You will
  53. 2:30have to read the decision tree to know
  54. 2:31that. I can give you a rough intuition
  55. 2:35that you will make a cut on either x1
  56. 2:38or x2. So it is possible that the
  57. 2:41decision tree stump looks like this. x1
  58. 2:44> 5 This could be a stump. x2 < 10,
  59. 2:51this could also be a stump. So many
  60. 2:54such decision trees are possible.
  61. 2:57Decision stumps are possible on this
  62. 2:59dataset. But which one of these
  63. 3:01decision stumps will you choose? You
  64. 3:04will choose the decision stump that
  65. 3:06results in the maximum decrease in
  66. 3:07entropy or the highest information gain
  67. 3:09. Call it entropy or Gini entropy, it
  68. 3:13is the same thing. Right? So what is
  69. 3:15the first thing we did? Finalized a
  70. 3:18decision stump that was performing the
  71. 3:21best. Right? Now we have selected that
  72. 3:23model. Let's call that decision stump
  73. 3:26Model M1. Right? Now we have to do two
  74. 3:30things. The first thing we have to do
  75. 3:34is check the performance of this
  76. 3:36decision stump on this dataset. So what
  77. 3:40will we do? We will take these inputs,
  78. 3:42feed them to this decision stump, and
  79. 3:44get the output. Right? Or simply put,
  80. 3:47we will calculate y-pred. So I am
  81. 3:49creating another column here by the
  82. 3:53name of y-pred. Basically, what is this
  83. 3:56algorithm's prediction on the training
  84. 3:58data? Now let's assume the prediction
  85. 4:00is something like this. This is the
  86. 4:05prediction we got. Right? Obviously,
  87. 4:07this is not the real prediction. I am
  88. 4:08telling you this for the sake of an
  89. 4:10example. Now if you notice here, all
  90. 4:13the predictions are correct. Except for
  91. 4:16this one. This prediction came out
  92. 4:19wrong. Actually, it was one. Our
  93. 4:22algorithm says it is zero. Right? So
  94. 4:24now what do we have to do? We have to
  95. 4:26do two things. First, we have to
  96. 4:31calculate alpha 1. What is alpha 1? It
  97. 4:34is how much say Model 1 will have when
  98. 4:37making the final prediction. Right?
  99. 4:40That is alpha 1, or you can call it the
  100. 4:42weight of this model. Right? Now, what
  101. 4:45will this depend on? This alpha will
  102. 4:48depend on your error rate. What is the
  103. 4:53error rate? It is how many mistakes you
  104. 4:55made on this data. It will depend on
  105. 4:58how many mistakes your model made on
  106. 5:00this data. If your error is very high,
  107. 5:04then your alpha will be very low.
  108. 5:06Because think about it, it is logical.
  109. 5:09If your model is making too many
  110. 5:10mistakes, its say should be less when
  111. 5:13making the final prediction. If its
  112. 5:15performance is very good, its error
  113. 5:17rate will be low, so its alpha will be
  114. 5:19higher. In the final prediction, we
  115. 5:21will value its input more because it
  116. 5:23made more correct predictions. Okay?
  117. 5:26Now the question is, how do we
  118. 5:27calculate this alpha? We need a
  119. 5:30function that will calculate this alpha
  120. 5:32for us. Okay? Now, to calculate this,
  121. 5:35we will have to use our brains a little
  122. 5:37. What do we need to think about? I
  123. 5:38will tell you. Let me ask you a simple
  124. 5:40question. In the process of answering
  125. 5:42that, you will understand how alpha is
  126. 5:44calculated. Suppose we have three
  127. 5:47models. This is model A, this is model
  128. 5:51B, and this is model C. Okay? Now,
  129. 5:56model A is always correct. Meaning its
  130. 6:00error rate is zero. It never makes a
  131. 6:03mistake. Model B is never correct. It
  132. 6:08is always wrong. This means this
  133. 6:11model's error rate is 100%. See, error
  134. 6:13rate is basically one minus accuracy.
  135. 6:15If your accuracy is 90%, then your
  136. 6:17error rate will be 10%. I hope this
  137. 6:19makes sense. The third model is a bit
  138. 6:22weird. It has a 50%error rate. So, this
  139. 6:26model's accuracy is 100%, this one's is
  140. 6:280%, and this one's is 50%. The question
  141. 6:31is, which of these models is the most
  142. 6:34reliable? You have to answer this
  143. 6:37question. Pause the video for a moment
  144. 6:39and think about which of these three is
  145. 6:42the most reliable model. Now, I guess
  146. 6:45you can easily understand that the
  147. 6:47first model is the most reliable. Right
  148. 6:50? But if I ask you this question, if we
  149. 6:52remove that one, which of these two is
  150. 6:54more reliable? So think about which
  151. 6:56model is more reliable between these
  152. 6:58two. Between B and C. Now, if many of
  153. 7:01you thought it was C, then that is
  154. 7:03actually wrong. B is more reliable.
  155. 7:06Think about it. You have a friend who
  156. 7:09lies 50%of the time and another friend
  157. 7:12who lies 100%of the time. So whose
  158. 7:15words would you trust more? Obviously,
  159. 7:18you would trust the friend more who
  160. 7:20always lies. Whatever they say, you
  161. 7:23just take the opposite, and it becomes
  162. 7:25100%true. Right? But this one sometimes
  163. 7:28tells the truth and sometimes tells a
  164. 7:29lie. This means this model is risky for
  165. 7:32us. We can use this model's prediction
  166. 7:35by negating it. There won't be any
  167. 7:37problem. So both these models are
  168. 7:39actually the same. This one always
  169. 7:41tells the truth. This one always tells
  170. 7:42a lie. If you reverse its logic, this
  171. 7:44model and the one above become the same
  172. 7:46. This model here is dangerous. Okay?
  173. 7:49So that means if you plot a graph. If
  174. 7:52you plot a graph. Between alpha and
  175. 8:00error. Then I hope you understand what
  176. 8:04kind of graph you need. You need a
  177. 8:06graph such that when the error is close
  178. 8:09to zero, alpha should be high. Listen
  179. 8:13carefully. If the error rate is close
  180. 8:16to zero or zero, then you want a high
  181. 8:18alpha. Because if your model makes very
  182. 8:22few mistakes, you will give it a high
  183. 8:24weightage. If you have a model that
  184. 8:28makes a lot of mistakes. Let's assume
  185. 8:31this error rate is one. The error rate
  186. 8:34varies between zero and one. So if your
  187. 8:36error rate is very close to one.
  188. 8:39Meaning this model, then you need an
  189. 8:42alpha that is very large in the
  190. 8:45negative. Something like this. And if
  191. 8:50your error rate is 50%, i.e., .5, then
  192. 8:53what kind of alpha do you need? You
  193. 8:56don't trust it at all, so you should
  194. 8:58give it zero. Meaning you need
  195. 9:00something like this. This means you
  196. 9:02need a mathematical function that
  197. 9:04follows this graph. You need a
  198. 9:07mathematical function to calculate
  199. 9:09alpha that follows such a graph. And
  200. 9:13there is such a mathematical function,
  201. 9:18and it is 1/2 ln (1-error/error). And
  202. 9:25this is the function using which you
  203. 9:28calculate alpha. Okay? I hope you are
  204. 9:30understanding. How is the error
  205. 9:33calculated? I will tell you in a little
  206. 9:35while. But this is the formula using
  207. 9:38which you calculate the weightage or
  208. 9:41alpha of any decision stump. I hope I
  209. 9:44was able to give you the intuition as
  210. 9:46to why we use this function. This
  211. 9:47intuition is very important. So please
  212. 9:49make sure you have understood this once
  213. 9:51. So now comes the question, how is the
  214. 9:53error in this formula calculated? Or
  215. 9:56how do you determine the error of any
  216. 9:58decision stump? Okay? Now, this error
  217. 10:01is not exactly one minus accuracy. I
  218. 10:03will tell you how the error is
  219. 10:05calculated here. Before that, I am
  220. 10:07making a small change. Let's assume
  221. 10:09that until now we considered that the
  222. 10:10mistake was only at one point. To make
  223. 10:12the example even better, let's assume
  224. 10:14that mistakes are happening at two
  225. 10:15points. This is another point where a
  226. 10:17mistake is happening. Okay? Now, the
  227. 10:19error is very simple. The error is the
  228. 10:23sum of the weights of all those rows
  229. 10:25that are misclassified. It's simple.
  230. 10:29First, find out how many of your rows
  231. 10:31are misclassified. In our case, these
  232. 10:34two rows are misclassified. Take the
  233. 10:37weights of both and add them up. That
  234. 10:42is your error. So, for this first model
  235. 10:45, our first decision stump m1, the
  236. 10:49error is actually equal to .4 because
  237. 10:52it made mistakes on two rows. Now, if
  238. 10:56we put this value into the formula to
  239. 11:00calculate alpha 1, it will be 1/2 log 1
  240. 11:04-.4 /.4 If we simplify this, it becomes
  241. 11:111/2 log of .6 /.4, and this value comes
  242. 11:18to around .20. Okay? You can
  243. 11:21cross-check it once. But this is the
  244. 11:23value. So, we just calculated a very
  245. 11:27important thing: how much say our first
  246. 11:30weak learner, the decision stump, has
  247. 11:32in the final prediction. Alpha 1 will
  248. 11:37be equal to .20, okay? I hope you are
  249. 11:41following along so far. Now, the next
  250. 11:43step is also very important. In it, you
  251. 11:46have to do what I explained in the
  252. 11:48previous video: stage one is now
  253. 11:50complete. In stage one, we trained a
  254. 11:53weak learner and calculated its weight.
  255. 11:55It made some mistakes. Now we need to
  256. 11:58pass those mistakes to the next stage.
  257. 12:00We discussed this in the last video as
  258. 12:02well. You increase the importance of
  259. 12:06the points that are misclassified. How
  260. 12:09do you increase importance? By using a
  261. 12:11technique called up-sampling. So, the
  262. 12:13concept is very simple. You have to
  263. 12:16tell the next model, the next decision
  264. 12:18stump, that I made a mistake on these
  265. 12:19two points, so don't you make a mistake
  266. 12:21. And the way to tell it is by
  267. 12:25increasing the weight of these two rows
  268. 12:28in the data. Increase the weights. And
  269. 12:31decrease the weights of the correctly
  270. 12:34classified points. So, in a way, I am
  271. 12:37boosting the weight of the
  272. 12:38misclassified points. And 'boosting'
  273. 12:41comes from this boosting. Okay? So,
  274. 12:43what do we have to do now? We need to
  275. 12:46update the weights. Increase the
  276. 12:49weights of the misclassified points and
  277. 12:51decrease the weights of the correctly
  278. 12:53classified ones. But the question is,
  279. 12:55how much to increase and how much to
  280. 12:57decrease? And guys, the answer is on
  281. 12:59the whiteboard. I have written a
  282. 13:01formula. And this is the weight update
  283. 13:04formula. There are two different
  284. 13:05formulas. One formula is for
  285. 13:07misclassified points. Meaning, for
  286. 13:09these two points, and one for correctly
  287. 13:11classified points. Meaning, for these
  288. 13:13three points. There isn't much
  289. 13:14difference in the formulas. If we talk
  290. 13:17about the first misclassified point,
  291. 13:19the new weight value will be the
  292. 13:21current weight multiplied by e raised
  293. 13:24to the power of alpha one. If you
  294. 13:27remember, our alpha one was. 2 Right?
  295. 13:33So if I have to calculate the new
  296. 13:35weight for these two rows, it will
  297. 13:37actually be 2*e raised to the power of
  298. 13:422, and for correctly classified points
  299. 13:472*e raised to the power of -2. Now if
  300. 13:51you do this calculation, you will find
  301. 13:53that this value comes to 244 and this
  302. 13:55value comes to 166, and you can see,
  303. 13:57the weights of the incorrectly
  304. 13:59classified points have increased, and
  305. 14:01the weights of correctly classified
  306. 14:03points have decreased. So guys, I have
  307. 14:06added the results we got to a new
  308. 14:08column called updated weights, 6 for
  309. 14:11correctly classified points, 244 for
  310. 14:14misclassified points. There is only one
  311. 14:16problem. If you add all of these up,
  312. 14:19the number is 96. The thing is, you
  313. 14:24need to make sure that the sum of your
  314. 14:27weights is always one. Just like it was
  315. 14:31here. So what we will do very simply is
  316. 14:33? Divide every weight by 96. Basically,
  317. 14:37normalize them. And I have already
  318. 14:39performed this operation. So after
  319. 14:40doing this, the weights we get are
  320. 14:43these. And when you add these weights,
  321. 14:46you will get one. Okay? So now our
  322. 14:49dataset is prepared. Almost prepared.
  323. 14:53One last step remains to move to the
  324. 14:55next stage. Okay? So what I do is?
  325. 14:57First, rub this whole whiteboard and
  326. 14:59write this dataset in a more compact
  327. 15:01way. So guys, I have rewritten the
  328. 15:03entire dataset on the screen in a
  329. 15:05compact way. Here is my X1 column, X2
  330. 15:07column, Y column, and here are my new
  331. 15:10weights. Okay? Now I have to perform
  332. 15:13the most important step of this entire
  333. 15:15process. That is upsampling. So I told
  334. 15:17you that somehow I need to send the
  335. 15:19information to the following stages
  336. 15:21about what went wrong in this stage,
  337. 15:23and how do we do that? We kind of boost
  338. 15:27the misclassified points. And how does
  339. 15:29that boost happen? Using upsampling.
  340. 15:32How do we do upsampling? I will tell
  341. 15:34you in a little while. But to do
  342. 15:35upsampling, you will have to do one
  343. 15:37thing. And that is that you will have
  344. 15:39to create a range like this. So what I
  345. 15:42did? Depending on the weights, I
  346. 15:44created these ranges. So the first one
  347. 15:46was 0.166, so I created this range from
  348. 15:48zero. Then the next was 0.25. So I
  349. 15:51added 0.255 from here. Then I added 255
  350. 15:55to this, then added 16 to this, and
  351. 15:57then 16 again, so it reached one. So
  352. 15:59basically, I created the ranges. Now
  353. 16:02what will I do? Now, five times I will
  354. 16:07generate random numbers between zero
  355. 16:12and one. I will generate five random
  356. 16:17numbers between zero and one. Okay? Let
  357. 16:19me generate the numbers and show you.
  358. 16:22Suppose I have a number 0.13, one is
  359. 16:250.43, one is 0.62, one is 0.50, and one
  360. 16:29is 0.8. Suppose these were generated
  361. 16:33randomly. Now if you look, we need to
  362. 16:36find which range these numbers fall
  363. 16:38into. Whichever range it falls into, we
  364. 16:41will pick that column and that row. See
  365. 16:44, 0.13 falls into this range, so we
  366. 16:47will now pick column/row number one. 43
  367. 16:55Falls into this range. So we will pick
  368. 17:00row number three. Then 62 also falls
  369. 17:03into this range. We will pick three
  370. 17:06again. Then 50 also falls right here.
  371. 17:10We will pick row number three again.
  372. 17:12And then there is 8, which is this one.
  373. 17:14So we will pick four. So now, what am I
  374. 17:18doing? I am creating a completely new
  375. 17:20dataset that will only contain these
  376. 17:23numbers. These row numbers will remain.
  377. 17:25So one will be there, three will be
  378. 17:27there three times, and four will be
  379. 17:28there. Now because of this technique,
  380. 17:31you can see three is in my data more
  381. 17:34often. Meaning its weight has increased
  382. 17:36in the data. Okay? Now if you drew
  383. 17:38randomly, it could also be that one
  384. 17:40came, then three, then three, then two,
  385. 17:43then two. There are more errors in
  386. 17:44three and two, so you will see three
  387. 17:45and two more often. So basically, with
  388. 17:49this entire technique, when you do
  389. 17:51sampling, since we are picking rows
  390. 17:54based on this range, the row with the
  391. 17:56larger range has a higher chance of
  392. 17:59being picked. Right? And this technique
  393. 18:03is called upsampling. So now, the new
  394. 18:06dataset I have for stage two will not
  395. 18:08be this dataset. It will be this
  396. 18:11dataset. I will take the first column
  397. 18:13once. Then I will take the three column
  398. 18:17three times, and then the four column.
  399. 18:20I will remove this one and this one,
  400. 18:23both these rows. Okay? And I will work
  401. 18:25on these remaining ones and after that
  402. 18:27everything will repeat. What will I do
  403. 18:29again? I will take a new decision stump
  404. 18:32and train it on this data. Obviously,
  405. 18:35the data has changed. The behavior of
  406. 18:37the decision stump will also change. I
  407. 18:39will then calculate Alpha 2 for that
  408. 18:41new decision stump. And then I will
  409. 18:43keep performing this entire upsampling
  410. 18:45stage step again. How many times will
  411. 18:47we do this? As many decision stumps as
  412. 18:49you are going to keep. So, in the end,
  413. 18:51you will have values up to Alpha 1,
  414. 18:54Alpha 2, Alpha 3, dot dot dot Alpha n.
  415. 18:56And your final prediction will be Alpha
  416. 19:001 times h1x + Alpha 2 times h2x dot dot
  417. 19:04dot Alpha n times hnx, which I
  418. 19:06explained to you in the last video. Now
  419. 19:10I know showing it by repeating it again
  420. 19:12and again is a bit boring. So let's do
  421. 19:13one thing. Let's look at this whole
  422. 19:15thing step-by-step through some code.
  423. 19:17Okay? That will give you a great
  424. 19:19understanding of how this entire
  425. 19:20process works multiple times.

About this transcript

This page contains the full transcript of AdaBoost - A Step by Step Explanation by CampusX, generated from the public captions YouTube serves with the video. The transcript has 3,028 words across 425 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.