YouTube2Text

AdaBoost Algorithm | Code from Scratch — Transcript

by CampusX · 2,530 words · 363 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Hello guys, my name is Nitish. Welcome
  2. 0:02to my YouTube channel. We've been
  3. 0:04studying the Ad Boost algorithm. In the
  4. 0:07last video, I tried to explain
  5. 0:09step-by-step how Ad Boost works on a
  6. 0:11whiteboard. In this video, I'll give
  7. 0:14you a walk-through of the entire
  8. 0:16algorithm through the code. Okay? I
  9. 0:19wrote this entire code from scratch. So
  10. 0:21, you'll get a sense of how this
  11. 0:23algorithm could be built from scratch.
  12. 0:26So, let's start the video. I've already
  13. 0:29written the code. I won't rewrite the
  14. 0:31entire code. I'll just give you a
  15. 0:33walk-through. That's a fast and
  16. 0:35efficient way to explain the code. So,
  17. 0:37first, I installed a few libraries.
  18. 0:40I'll use a library called mlx10 to plot
  19. 0:42the decision boundaries. You know the
  20. 0:45other two libraries: Padas and NPI. I'm
  21. 0:48creating my own data frame and
  22. 0:49inserting my own data into that data
  23. 0:51frame, which I created myself. And
  24. 0:54here's my data frame. This is my data
  25. 0:56frame. There are 10 rows. There are
  26. 0:58three columns. Two columns are for
  27. 1:00inputs. One column is for outputs. And
  28. 1:03this is a classification problem. Now,
  29. 1:05let me plot it to show you what our
  30. 1:08data looks like. And you can see that
  31. 1:10this is our data, and it's essentially
  32. 1:12a slightly non-linear type of data. So,
  33. 1:15I deliberately chose this type of data.
  34. 1:18Now, we'll start applying the algorithm
  35. 1:20. So, the first thing you do in Boost
  36. 1:24is assign a weight to each row.
  37. 1:27Initially, those weights are equal. How
  38. 1:30that equality is calculated is very
  39. 1:32simple. You assign each row a weight
  40. 1:35divided by the total number of rows. In
  41. 1:38our case, we have 10 rows. So, each row
  42. 1:42gets one. So, I'm running this code.
  43. 1:45And now my data frame looks like this.
  44. 1:47You can see that everyone's weight is
  45. 1:49equal. Point one. Now the next thing
  46. 1:53you'll do is create a Decision Tree
  47. 1:55Classifier object, and you have to make
  48. 1:57sure that the max depth should be equal
  49. 1:59to one. Because the decision trees you
  50. 2:02use in Ada Boost are decision stumps.
  51. 2:06Okay? So I created this object.
  52. 2:08Extracted X and Y, and trained the
  53. 2:10model. Since the data is very small,
  54. 2:12the model trained very quickly. Let me
  55. 2:15show you the decision stump first. This
  56. 2:18is the decision stump. And this
  57. 2:20decision stump is being created above
  58. 2:22the second column, above X2. Here it is
  59. 2:24written X bracket one, meaning the
  60. 2:26second column, zero is the first column
  61. 2:28. And here is its decision boundary.
  62. 2:31Let's see how many mistakes this model
  63. 2:33is making? And you can see a, it's
  64. 2:35making three mistakes. The three blue
  65. 2:37points are in the orange region. Okay,
  66. 2:40now what we'll do is make predictions
  67. 2:43on the entire training data using this
  68. 2:45code. Now my data frame looks like this
  69. 2:48. You can compare them side by side to
  70. 2:51see how many mistakes our model made.
  71. 2:53So, there's no error in the first row
  72. 2:55because the label is one, and the
  73. 2:56prediction is one. There's no error in
  74. 2:57the second either. There's an error in
  75. 2:59the third. There's no error after that.
  76. 3:02There's one here, and there's one here.
  77. 3:04In total, our model is making three
  78. 3:06mistakes. Row two, row six, and row
  79. 3:10eight. Okay? Now you know what, next?
  80. 3:17Calculate a weight for this decision
  81. 3:20stump you trained the model using the
  82. 3:23formula 1/2 log 1-error divided by
  83. 3:26error. What is error? The sum of the
  84. 3:29weights for all the rows where a
  85. 3:31misclassification occurred. There were
  86. 3:34misclassifications on three rows. If
  87. 3:37the weights of all three are 1, then
  88. 3:39our total error will be 1 + 1 + 1,
  89. 3:40meaning 3. I've created a function here
  90. 3:43called "calculate model weights." If
  91. 3:46you provide the error, the formula will
  92. 3:48be applied and you'll get the model's
  93. 3:50weights. I'll run this formula once.
  94. 3:54We'll calculate alpha for our first
  95. 3:56model. So, I passed 3 into "calculate
  96. 3:59model weights," which is my total error
  97. 4:02, and the alpha is 0.42. Now, our next
  98. 4:07step is to increase the weights for
  99. 4:10rows with misclassifications and
  100. 4:12decrease the weights for rows with
  101. 4:15correct classifications. This way, you
  102. 4:19can report the current model's errors
  103. 4:21to the next model. Okay? The entire
  104. 4:24upsampling step was from the previous
  105. 4:26video. So, I've created a function here
  106. 4:29called "update rows," where you pass
  107. 4:32the rows and an alpha value. In our
  108. 4:35case, the alpha value is 0.423. And
  109. 4:39here I've put a check so that if that
  110. 4:42row is correctly classified, you
  111. 4:44multiply the current rate by e to the
  112. 4:46power of alpha. And if there's a
  113. 4:50misclassification, you multiply the
  114. 4:52current rate by e to the power of alpha
  115. 4:54. Okay? Let's run this code once. And
  116. 4:59we're creating a new column called
  117. 5:02Updated Weights and calling this code.
  118. 5:05And here are our updated weights, and
  119. 5:07you'll see that the weights for the
  120. 5:09rows where we made mistakes have
  121. 5:10increased. Okay? From point one, it's
  122. 5:150.15, and the weights for the rows
  123. 5:17where we made the right ones have
  124. 5:19decreased. From point one, it's 0.06.
  125. 5:22You're getting the idea. Now, there's
  126. 5:25one thing I want to discuss with you
  127. 5:27that I forgot to mention in the
  128. 5:29previous video: why do we multiply e to
  129. 5:32the power of alpha? Why do you multiply
  130. 5:36the current weights by e to the power
  131. 5:38alpha or e to the power alpha? Let me
  132. 5:41explain the reason. For that, we'll
  133. 5:43have to go into Desmos. And I'll show
  134. 5:47you this graph: y = e to the power x.
  135. 5:53This is a graph of e to the power x.
  136. 5:56Now, if you look at the behavior of
  137. 5:58this graph, as x increases, y increases
  138. 6:02rapidly. And if x is small, y also
  139. 6:06decreases. Right? So, x here is our
  140. 6:10alpha. So, this means that if a model
  141. 6:14has a high alpha, when will its alpha
  142. 6:16be high? Think about it. A model's
  143. 6:18alpha will be high when it is very
  144. 6:21accurate. It makes very few errors. A
  145. 6:23model with very accurate predictions
  146. 6:25will have a high alpha. Now, if you
  147. 6:28have a model with a very high alpha,
  148. 6:30then this whole thing will also be very
  149. 6:32high. And if this whole thing is very
  150. 6:35high, then when you multiply the
  151. 6:36current weight by a large number, the
  152. 6:38new weight will also be large. This
  153. 6:41means that if you have a very
  154. 6:44trust-uniform model, it will increase
  155. 6:47the weight of your incorrectly
  156. 6:49classified rows significantly. But if
  157. 6:54you don't have a very trust-uniform
  158. 6:56model, meaning alpha is small, then
  159. 6:58this number won't be very large; it
  160. 7:00will be small, and then the update will
  161. 7:02also be very small. And that's why we
  162. 7:05use the exponential function. I can
  163. 7:07also give you an intuition for-x.
  164. 7:09Here's a graph of-x. For very small x,
  165. 7:12y is large, and for very large x, y is
  166. 7:15small. Exactly the opposite. Okay? So
  167. 7:17if you have a model with a very high
  168. 7:20alpha, meaning your model is very
  169. 7:22trust-uniform, then this large number
  170. 7:25will become less than one, which will
  171. 7:28reduce your current weight. Okay? So
  172. 7:31please back up a bit and understand
  173. 7:33this whole idea. It is actual. Okay?
  174. 7:36Now we have our updated weights.
  175. 7:39There's only one problem, and that's
  176. 7:41that if you sum the updated weights,
  177. 7:50the total sum of all the days is only
  178. 7:550.91. And I told you that you should
  179. 7:58always update the weights in such a way
  180. 8:01that the sum of all the daily weights
  181. 8:03should always be equal to one. So we
  182. 8:05need to normalize. And here's the code
  183. 8:08for normalization. I'm dividing each
  184. 8:10weight by the sum of the weights
  185. 8:11multiplied by the sum of the weights.
  186. 8:13And storing it in a new column. And
  187. 8:16here's our output. Here are our
  188. 8:19normalized weights. So now all the
  189. 8:21incorrectly classified points are 16.6
  190. 8:24and all the correctly classified points
  191. 8:27are 0.7. Okay? And here are our
  192. 8:30normalized weights, exactly equal to
  193. 8:33one. Okay? What are we doing now? We're
  194. 8:35creating a range. So this is the code
  195. 8:37for creating a range. I'm creating an
  196. 8:38upper range. I'm creating a lower range
  197. 8:41. And here are my upper and lower
  198. 8:43ranges. Using these rates, I've created
  199. 8:46the upper and lower ranges. Okay? You
  200. 8:50can see this once. Now, our next task
  201. 8:53is to create a new data set from this
  202. 8:55same data set. That new data set will
  203. 8:58have the same number of rows as our
  204. 8:59current data set. And how will that new
  205. 9:02data set be created? We'll generate 10
  206. 9:05random numbers. And we'll check which
  207. 9:08range those random numbers fall into.
  208. 9:12We'll pick the particular row in which
  209. 9:14they fall. Okay? For this, I've created
  210. 9:18a function called "Create New Data Set"
  211. 9:19where I'll pass my old data set in.
  212. 9:22I'll calculate the indices. I'm running
  213. 9:25this loop 10 times. Each time, I'm
  214. 9:27generating a random number and checking
  215. 9:29which range it falls into. I'm taking
  216. 9:32the index of that range and putting it
  217. 9:34into a list of indexes and returning it
  218. 9:37to you. So here I called this function
  219. 9:41and you can see that two appears once,
  220. 9:44eight appears twice, and six appears
  221. 9:47once. So basically, the weights of
  222. 9:49those rows where mistakes were made
  223. 9:51have been increased. And this is what I
  224. 9:53told you in the last video, this step
  225. 9:55is called upsampling. So we've
  226. 9:57upsampled. Now I'm creating a second
  227. 10:00data frame from the old data frame and
  228. 10:03taking only these rows. Basically,
  229. 10:05taking the index values of these rows
  230. 10:08and taking only these columns: 0 1 2 3
  231. 10:10these four rows. Okay? And here is my
  232. 10:17new data frame. Okay? Always remember,
  233. 10:20as soon as you create a new data frame
  234. 10:22for the next step, you'll have the
  235. 10:24weights initialized from point one.
  236. 10:26You'll re-initialize with equal weights
  237. 10:28because you've already upsampled. Okay?
  238. 10:31What do you do now? We'll create a
  239. 10:32second decision tree. We'll drain it
  240. 10:36over the new data. Here's my second
  241. 10:40decision tree. It's drained over the
  242. 10:42first column. Okay? It's drained over
  243. 10:45x1. And then we'll plot its decision
  244. 10:49boundary. And you can see it's making a
  245. 10:52mistake. Above the blue point. So we're
  246. 10:55extracting the vibration again. And
  247. 10:58here's the second BF. So you can see
  248. 11:01it's making a mistake here. And there
  249. 11:04are no other mistakes. Let's
  250. 11:06cross-check again. One mistake, no more
  251. 11:10mistakes. Okay? We're calling calculate
  252. 11:13model weights again. And here we're
  253. 11:16doing one pass. And we're calculating
  254. 11:19alpha for our second decision tree. And
  255. 11:22its weights are coming out to be 1.09,
  256. 11:24which is higher than the previous
  257. 11:26weights because it made fewer mistakes.
  258. 11:29One more thing: sometimes you might end
  259. 11:30up with a model that doesn't make any
  260. 11:31mistakes, especially when you're
  261. 11:33working with a small data set. In that
  262. 11:35case, your error will become zero. Now,
  263. 11:38if the error becomes zero, this formula
  264. 11:40will start crying. This formula we saw
  265. 11:44above will start crying. Because you
  266. 11:47divide by the error, and if the error
  267. 11:49itself is zero, this entire term will
  268. 11:51explode. So what can you do in that
  269. 11:53case? You can put a small number in
  270. 11:55front of the error. That will prevent
  271. 11:59the term from becoming zero. Okay? I've
  272. 12:02already told you separately how to
  273. 12:04solve this problem if you ever run into
  274. 12:06it. This problem is solved internally
  275. 12:08in SciKitLearn, so you don't need to do
  276. 12:10it. Now coming back. So, we've reached
  277. 12:15this point. We calculated our alpha.
  278. 12:18Now we'll update our weights again. The
  279. 12:20weights of correctly classified points
  280. 12:22will be reduced. The weights of
  281. 12:24incorrectly classified points will be
  282. 12:25increased. So we're calling Update Row
  283. 12:28Weights again. This time, the value of
  284. 12:30alpha will be 1.09. Okay? Run it.
  285. 12:35Created the updated weights column, and
  286. 12:37here it is. And you can see there was
  287. 12:40only one error. So its weight is 29.
  288. 12:42All the others have a weight of 0.03.
  289. 12:44Again, we'll normalize it. And after
  290. 12:49normalizing, here it is. 49 is the only
  291. 12:53one, and all the others are five. Sum
  292. 12:58is equal to one. Okay? Again, I'm
  293. 13:00creating ranges. Here are my ranges.
  294. 13:07Okay? Again, I'm creating index values.
  295. 13:11And creating a third data frame. In the
  296. 13:14third data frame, you'll notice that
  297. 13:16you see zeros quite often, because this
  298. 13:19is the row with zeros where the error
  299. 13:21was occurring, so you'll see them quite
  300. 13:23often. Now the sample is done, okay,
  301. 13:26there's a lot of them. I'll train the
  302. 13:28third decision tree again, and here's
  303. 13:31its decision bound tree. It's also
  304. 13:34making some mistakes, precisely one
  305. 13:35mistake. We'll calculate the viability
  306. 13:44for this one as well and see how many
  307. 13:45mistakes it's making: one mistake, two
  308. 13:47mistakes, three mistakes, four, five,
  309. 13:49six, and seven. It's making a lot of
  310. 13:52mistakes, one mistake, two, three, four
  311. 13:55, five, six, and seven. Okay, so here
  312. 13:59we'll write the point 7, and its weight
  313. 14:05is negative, okay, it's logical too. Ah
  314. 14:08, so now here we have our three alphas,
  315. 14:14okay? Now we have to make a prediction.
  316. 14:16So what do we do when making a
  317. 14:18prediction? Let's pick any existing row
  318. 14:20. I'll show you this one first. One
  319. 14:24five should predict one, and 99 should
  320. 14:26predict zero. I already have these two,
  321. 14:29so I'm picking these. So look, this is
  322. 14:32my first query point. One five. I sent
  323. 14:35this to decision tree one first. The
  324. 14:38output is one. Then I sent it to
  325. 14:40decision tree two. The output is one.
  326. 14:43And then I sent it to three. The output
  327. 14:45is one. Now what do we do? We'll
  328. 14:47multiply alpha one by one. We'll
  329. 14:50multiply alpha two again by one, and
  330. 14:52we'll multiply alpha three by one. And
  331. 14:55this total is 1.09. Now I just need to
  332. 14:58take the sign of 1.09 and get one. So
  333. 15:04our output will be one. And if you
  334. 15:06remember, the output for this data
  335. 15:07point should also be one. Okay? Let's
  336. 15:10take the second point, nine, and nine.
  337. 15:11So, according to the first decision
  338. 15:13tree, the prediction should be one.
  339. 15:15According to the second, the prediction
  340. 15:17should be zero. And according to the
  341. 15:20third, the prediction should be zero.
  342. 15:22Okay? So, we'll multiply alpha by one.
  343. 15:26We'll multiply alpha two by -1, and
  344. 15:29alpha three by -1. I've already told
  345. 15:31you this. In the case of boosting, 0 is
  346. 15:34taken as -1. Okay? And now we're
  347. 15:36running this, and it's coming out as
  348. 15:39-25. If I put this number in the mp dot
  349. 15:44sign and run it, the output is -1,
  350. 15:47meaning the point is negative. And
  351. 15:51that's how this algorithm works. I ran
  352. 15:53it with all three decision stumps, but
  353. 15:55you could use 100 or 200. Now, what's
  354. 15:57your job after watching this video?
  355. 16:01I've written this code very simply. If
  356. 16:05you want, you can create a function in
  357. 16:07which you train as many decision stumps
  358. 16:10as you want, and you can create your
  359. 16:12own class to add a boost. Okay? So, I
  360. 16:15hope this is all clear to you. I'll
  361. 16:18give you the entire code. So, just
  362. 16:21spend some time with it, and that's it.
  363. 16:25Thanks for watching. Bye.

About this transcript

This page contains the full transcript of AdaBoost Algorithm | Code from Scratch by CampusX, generated from the public captions YouTube serves with the video. The transcript has 2,530 words across 363 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.