YouTube2Text

Introduction to Ensemble Learning | Ensemble Techniques in Machine Learning — Transcript

by CampusX · 6,065 words · 896 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Hello guys, welcome to my YouTube
  2. 0:02channel. We studied Decision Trees last
  3. 0:04time, and now we are going to start
  4. 0:06something very important in Machine
  5. 0:08Learning called Ensemble Learning. It
  6. 0:11is so useful that whenever you do any
  7. 0:13machine learning project in a company
  8. 0:16or a Kaggle competition, it is
  9. 0:18mandatory that you try Ensemble
  10. 0:20Learning there as well. Okay? Now,
  11. 0:24there are many algorithms in Ensemble
  12. 0:25Learning. We will cover all these
  13. 0:27algorithms one by one. But in this
  14. 0:29video, I will just tell you about
  15. 0:31Ensemble Learning. I will provide an
  16. 0:33introduction to it. I will tell you
  17. 0:35which algorithms are included in it.
  18. 0:37And I will also give an introduction to
  19. 0:38how those algorithms work roughly. Okay
  20. 0:41? So let's start with the word '
  21. 0:43Ensemble' first. What does the word '
  22. 0:45Ensemble' mean? If you Google the
  23. 0:48meaning of 'Ensemble', you will get the
  24. 0:51answer that an ensemble is a group of
  25. 0:53artists or a group of something. In
  26. 0:57machine learning, when you talk about
  27. 1:00ensemble learning, it signifies a
  28. 1:02collection of multiple machine learning
  29. 1:05models. So if you combine different
  30. 1:08machine learning models and then create
  31. 1:11a bigger model from them, that
  32. 1:13technique is called Ensemble Learning.
  33. 1:16Okay? So in this video, we will first
  34. 1:19understand how Ensemble Learning works.
  35. 1:22Why does it work? Meaning, why does it
  36. 1:24perform so well? Where is it useful and
  37. 1:27things like that, okay; and in fact,
  38. 1:30the types of Ensemble Learning. But
  39. 1:33first, I want to explain a concept that
  40. 1:35we call the 'Wisdom of the Crowd'. If
  41. 1:39you want to understand Ensemble
  42. 1:40Learning, you must first have an idea
  43. 1:42of what is meant by the 'Wisdom of the
  44. 1:44Crowd'. So let's start with that topic.
  45. 1:47I guess you might have understood a
  46. 1:48little bit from the name itself what
  47. 1:50the 'Wisdom of the Crowd' means. In
  48. 1:51Hindi, it would be 'the intelligence of
  49. 1:53the crowd'. Okay? So let me tell you
  50. 1:56from my personal experience when I felt
  51. 1:58the 'Wisdom of the Crowd'. When I was
  52. 2:01young, I used to watch KBC (Kaun Banega
  53. 2:03Crorepati) and if you remember, there
  54. 2:06used to be a lifeline called 'Audience
  55. 2:08Poll', right? And I don't know,
  56. 2:12whenever I noticed it, whenever I
  57. 2:14watched KBC, most of the time when
  58. 2:16participants took the audience poll,
  59. 2:18they got the right answer. It meant
  60. 2:21that the audience somehow always knew
  61. 2:24the right answer, and it was very
  62. 2:26surprising to me how it could be that
  63. 2:29the audience always knows the answer.
  64. 2:32Guess what? This isn't just in KBC.
  65. 2:35It’s spread everywhere around us.
  66. 2:38Wisdom of the crowd is a widespread
  67. 2:40phenomenon that our lives depend on.
  68. 2:43Let me give you some examples. Another
  69. 2:46great example that you might be able to
  70. 2:48relate to is product reviews. So,
  71. 2:52nowadays when we shop on Amazon or
  72. 2:54Flipkart, whenever we are going to buy
  73. 2:56a product, how do we get the
  74. 2:58reliability that we should buy it or
  75. 3:00not? Because when buying online, you
  76. 3:04can't touch and see the product. You
  77. 3:06can't hold it in your hands. You only
  78. 3:08have a photo. So where does that trust
  79. 3:10come from? That is why websites like
  80. 3:13Flipkart and Amazon do what they do.
  81. 3:15They take reviews from customers like
  82. 3:17you, and all those reviews are shown to
  83. 3:19you when you go to buy that product. So
  84. 3:22let's say there is a product that
  85. 3:2415,000 people have reviewed and rated,
  86. 3:27and its rating is 4.5 out of 5. So what
  87. 3:30comes to my mind? It comes to my mind
  88. 3:33that since so many people have said
  89. 3:35this product is good, it must be good.
  90. 3:39If the rating were 4.5 and only one
  91. 3:41customer had reviewed it, meaning only
  92. 3:43one customer gave it 4.5. And no one
  93. 3:46else gave any review, would I feel as
  94. 3:48comfortable and confident about buying
  95. 3:51that product? The answer is no. Right?
  96. 3:54So, in a way, I am relying on the
  97. 3:56wisdom of the crowd. When one person
  98. 3:59said it's 4.5. My internal alarm didn't
  99. 4:01ring. But as soon as 15,000 people said
  100. 4:04it's 4.5, I thought that if so many
  101. 4:06people are saying it, it must be great,
  102. 4:08and I relied on that and bought it.
  103. 4:10Okay? You will find this same thing in
  104. 4:12other places too. When you want to
  105. 4:15watch a movie, you search for it on
  106. 4:18IMDb, and you decide whether or not to
  107. 4:20watch the movie based on the IMDb
  108. 4:23rating. But where is that rating coming
  109. 4:25from? Many other viewers like you went
  110. 4:29and watched the movie and rated it, and
  111. 4:31when you look at it collectively, you
  112. 4:34get a number like 8.5, 8.9, 6.1, and
  113. 4:36that number tells you how people
  114. 4:39collectively feel about that movie.
  115. 4:42Again, an example of the wisdom of the
  116. 4:44crowd. Okay? Furthermore, the biggest
  117. 4:47example is democracy. India is a
  118. 4:49democracy. Right? How do elections
  119. 4:51happen in a democracy? Everyone has the
  120. 4:54right to vote. Everyone's vote has
  121. 4:56equal value. And in the end, the party
  122. 4:59that gets the most votes wins. Why?
  123. 5:02Because we believe that when most
  124. 5:04people are supporting a party, then
  125. 5:06that party must be right. And the
  126. 5:09fundamental of democracy also lies in
  127. 5:11this very thing. Wisdom of the crowd. I
  128. 5:13will give you a very good example. I
  129. 5:15was reading a recent example in a blog.
  130. 5:20It was written there that some 200-300
  131. 5:23years ago, this incident actually
  132. 5:25happened where there was a fair, an
  133. 5:28animal was put on stage, and the host
  134. 5:31asked the whole crowd to guess how much
  135. 5:34that animal weighed. Whoever had the
  136. 5:38correct answer would get a prize. So,
  137. 5:41many people started guessing. Obviously
  138. 5:44, no one had the perfectly correct
  139. 5:46answer. But the host took all those
  140. 5:50answers and calculated their mean, and
  141. 5:53as soon as the mean was calculated, you
  142. 5:55would be surprised to know that the
  143. 5:58mean was almost, down to the decimal
  144. 6:00value, very close to the actual weight
  145. 6:03of that animal. So somehow, the crowd
  146. 6:07always knows the answer. And this
  147. 6:09phenomenon is called the wisdom of the
  148. 6:11crowd. If you want to read more about
  149. 6:14this topic, you can check out this book
  150. 6:18. There is a book on Amazon called "The
  151. 6:20Wisdom of Crowds". Although I haven't
  152. 6:22read the whole thing. I read the first
  153. 6:24couple of chapters, which I found very
  154. 6:26interesting. And it deals exactly with
  155. 6:29this topic: how people in a crowd know
  156. 6:32the answer. And guess what? Ensemble
  157. 6:35learning is based on this very thing.
  158. 6:38The entire fundamental concept of
  159. 6:39ensemble learning in machine learning
  160. 6:41is based on the fact of the wisdom of
  161. 6:43the crowd. That a crowd knows more than
  162. 6:47an individual person. Okay? So now that
  163. 6:49we know what the wisdom of the crowd is
  164. 6:51. Now let's move towards ensemble
  165. 6:53learning and understand its basics. So
  166. 6:56guys, now we will understand the core
  167. 6:58idea of ensemble learning. The good
  168. 7:00part is that the core idea is very
  169. 7:02simple. Okay? So, there are two steps
  170. 7:05related to any machine learning
  171. 7:07algorithm. I guess you already know
  172. 7:08this. One is training and one is
  173. 7:13prediction. Training is where it looks
  174. 7:18at the data to find patterns within it,
  175. 7:20and prediction happens when you give it
  176. 7:22a new query point and it has to answer
  177. 7:25based on what it has learned. Right? I
  178. 7:28am explaining the prediction stage to
  179. 7:30you. I will explain the training stage
  180. 7:32to you later. When we study the types
  181. 7:34of ensemble learning. Okay? Now look
  182. 7:36closely. In the prediction stage,
  183. 7:38suppose you have an ensemble learning
  184. 7:41algorithm. An ensemble learning
  185. 7:43algorithm means you have a collection
  186. 7:45of smaller base models. Suppose you
  187. 7:48have different algorithms or different
  188. 7:51base models. Like this. These are
  189. 7:55different base models. Basically, they
  190. 7:58are algorithms. Okay? It could be a
  191. 8:01decision tree, SVM, KNN, linear
  192. 8:02regression, it could be anything. These
  193. 8:05are different machine learning models.
  194. 8:08And collectively, this collective model
  195. 8:12is your ensemble learning model. By
  196. 8:16combining all of these, you are
  197. 8:18creating a big model that you call an
  198. 8:20ensemble learning model. Okay? Now, one
  199. 8:25thing you always have to ensure is that
  200. 8:28these models are different. These
  201. 8:33models are different. This will always
  202. 8:35be an important point. You must always
  203. 8:38make sure that your models are very
  204. 8:40different. I can give you an intuition
  205. 8:43for this as well. Suppose you are
  206. 8:45sitting in KBC and you have to use an
  207. 8:48audience poll, and you find out that
  208. 8:51all the people sitting in your audience
  209. 8:54are software developers. So, is this a
  210. 8:57good thing or a bad thing for you?
  211. 8:59Obviously, it is a bad thing because if
  212. 9:01everyone is a software developer, then
  213. 9:03their knowledge is in the software
  214. 9:05domain. So, if a question about
  215. 9:07computer science is asked of me
  216. 9:08tomorrow and I don't know it, these
  217. 9:10people will tell me because they are
  218. 9:12all software developers. But they won't
  219. 9:15know anything else. Now imagine a
  220. 9:16second scenario where the audience
  221. 9:18sitting in front of me is from
  222. 9:19different backgrounds. One is a CA, one
  223. 9:21is a computer science engineer, one is
  224. 9:23a civil engineer, one is a doctor, one
  225. 9:25is a politician, one is a philosopher.
  226. 9:27Right? So now I have a variety of crowd
  227. 9:30. Right? So now if I ever get stuck,
  228. 9:32it's possible that if the engineer
  229. 9:34doesn't know, the politician might know
  230. 9:37. If the politician doesn't know, then
  231. 9:38the literature person might know. Right
  232. 9:40? So that is why when you build
  233. 9:43ensemble learning algorithms, you try
  234. 9:45to ensure that the base models that
  235. 9:47make up the ensemble model come from
  236. 9:49different backgrounds. They should be
  237. 9:52different. So we need experts who
  238. 9:54possess expertise in different domains.
  239. 9:56All right? So going by that logic, I am
  240. 9:59telling you that you need different
  241. 10:01models. Now, what exactly does it mean
  242. 10:03to have different models? It has two
  243. 10:05meanings. One meaning is that all these
  244. 10:07algorithms themselves are different.
  245. 10:09For example, let the first one be
  246. 10:10Linear Regression. The second one be
  247. 10:13SVM. The third one be a Decision Tree.
  248. 10:15In this way, there should be different
  249. 10:17algorithms. So it automatically became
  250. 10:19different. There is a second way too;
  251. 10:22what can you do? Fix the algorithm. All
  252. 10:25models are Linear Regression algorithms
  253. 10:28. All of them are just Linear
  254. 10:30Regression algorithms. But when you
  255. 10:33were training these models, you gave
  256. 10:35them different data. You gave them
  257. 10:38different data. The algorithm is the
  258. 10:40same. You changed the data. And because
  259. 10:43you changed the data, these models will
  260. 10:45train differently, and since they train
  261. 10:48differently, you have created that
  262. 10:50variety. You have created a difference.
  263. 10:52So always remember point one: in
  264. 10:53ensemble learning, the base models must
  265. 10:55be different. They should be different,
  266. 10:58you need variety. And how will that
  267. 10:59difference come about? There are two
  268. 11:01ways. First, using different algorithms
  269. 11:04. Second, using the same algorithm but
  270. 11:06showing different data. All right? And
  271. 11:08the third is a combination of these two
  272. 11:10. You are using different algorithms
  273. 11:12and also showing different data. All
  274. 11:13right? So I guess you are understanding
  275. 11:15up to this point. So now all these
  276. 11:17models have been trained. All these
  277. 11:19models are trained. Now I have a new
  278. 11:22query point. You can take the same
  279. 11:24example we have been using until today.
  280. 11:27CGPA, IQ, and whether the placement
  281. 11:31happened or not. We are talking about a
  282. 11:35classification problem. So now a new
  283. 11:37student has come, and I know their CGPA
  284. 11:41and their IQ, and I need to predict
  285. 11:43whether they will get placed or not. So
  286. 11:47what will I do? I will pass this query
  287. 11:50point to every algorithm. Since every
  288. 11:52algorithm is a machine learning model
  289. 11:54in itself. It can make a prediction.
  290. 11:57Right? So it turns out the first one
  291. 11:59said they will get placed. The second
  292. 12:01one said they will not. The third one
  293. 12:03said they will, they won't, and so on.
  294. 12:06Now suppose we had five models. Right
  295. 12:09now, I have written 'n' here. So, let's
  296. 12:12assume we have five models, where three
  297. 12:15people said it will happen and two said
  298. 12:17it won't. Then the democracy logic
  299. 12:21applies. That logic is called majority
  300. 12:24count. So, whichever class the majority
  301. 12:28count favors, that is what your
  302. 12:30ensemble model will say. So here, if
  303. 12:33more people are saying placement will
  304. 12:35happen, your ensemble model's output
  305. 12:37will be that placement will happen. If
  306. 12:40more people had said it won't happen,
  307. 12:42your ensemble model would say it won't
  308. 12:44happen. Right? I hope you are
  309. 12:46understanding how prediction works in
  310. 12:48ensemble learning. Right? We will talk
  311. 12:50about the training part a little later.
  312. 12:52Don't worry about that. Now, this was
  313. 12:54the classification setting. What about
  314. 12:56the regression setting? What will
  315. 12:58happen in the regression setting?
  316. 12:59That's the question. Nothing will be
  317. 13:01different. It will be exactly the same
  318. 13:03thing. You will follow two things again
  319. 13:05. First, all the models you have must
  320. 13:08be different. There should be variety.
  321. 13:11And the second thing is that your
  322. 13:13trained models will either be the same
  323. 13:15models or the same models trained on
  324. 13:17different data. Different, I'm sorry, I
  325. 13:20got confused. Either they will be
  326. 13:22different models or the same models
  327. 13:23trained on different data. I hope
  328. 13:25there's no problem. Sorry about that.
  329. 13:28Now, if you have a regression setting.
  330. 13:31Suppose your problem statement is to
  331. 13:33predict the LPA based on CGPA and IQ.
  332. 13:36How many lakhs will the placement be?
  333. 13:38So in this case too, what will you do?
  334. 13:40You will send your input query point to
  335. 13:43all the models. Now, these models are
  336. 13:46internally regression models. So, they
  337. 13:49will output a number for you like 8.1,
  338. 13:533.2, 4.1, 7.5, and the ensemble model,
  339. 13:57the overall model, what will it do? It
  340. 14:01will add all these up and divide by
  341. 14:04four. Or in short, I could say it will
  342. 14:06calculate the mean. And as soon as you
  343. 14:09calculate the mean, you will get your
  344. 14:11answer. Right? So I guess you are
  345. 14:15understanding. It is a very simple
  346. 14:17thing. It is a very simple thing and
  347. 14:19very effective as well. Right? So the
  348. 14:21core idea is just that you bring in a
  349. 14:24lot of models and ask them all one
  350. 14:26question. You take all their answers
  351. 14:29and then make a decision based on those
  352. 14:31answers. Right? That is the whole idea.
  353. 14:34So, I have explained the prediction
  354. 14:36stage quite well here. I didn't explain
  355. 14:39the training stage because training
  356. 14:41happens in many different ways.
  357. 14:43Depending on which type of ensemble
  358. 14:45learning you are using. So that is why
  359. 14:47what will we study next? Types of
  360. 14:49ensemble learning. Where I will tell
  361. 14:51you the different types and how
  362. 14:53training happens in each type? I will
  363. 14:55explain that to you as well. Okay? That
  364. 14:57is what we are going to study next.
  365. 14:58Okay guys? So now let's discuss the
  366. 15:01different types of ensemble learning
  367. 15:03methods in machine learning. Okay? So
  368. 15:06first, let me create a sort of chart
  369. 15:07for you and then we will have a more
  370. 15:09detailed discussion about it. So if you
  371. 15:12talk about ensemble learning, precisely
  372. 15:15we have four different types that are
  373. 15:18used the most. Actually, there are more
  374. 15:20types than that. But these are the four
  375. 15:23most important types. The easiest and
  376. 15:26most useful type is voting. This is
  377. 15:31called voting ensemble. Okay? The
  378. 15:36second one is called bagging. This is
  379. 15:42also used very extensively. The third
  380. 15:45one, which is perhaps the most used
  381. 15:47ensemble technique, is called boosting.
  382. 15:52And the last one is called stacking.
  383. 15:59Okay? These are the four most important
  384. 16:01techniques you need to study and use in
  385. 16:03ensemble learning. Okay? Now here, if
  386. 16:06you have ever heard the names of some
  387. 16:08famous algorithms. Then within bagging,
  388. 16:11there is a famous algorithm we call
  389. 16:13Random Forest. It is a very powerful
  390. 16:16and useful machine learning technique
  391. 16:19and you will be using it a lot later.
  392. 16:22Okay? Some very famous techniques also
  393. 16:24come under boosting. Such as AdaBoost.
  394. 16:30It is called Adaptive Boosting. Next is
  395. 16:33Gradient Boosting. This is also a very
  396. 16:36useful technique. Very useful. And the
  397. 16:41last one is what is used in every
  398. 16:43machine learning competition on Kaggle,
  399. 16:46called XGBoost, which stands for
  400. 16:48Extreme Gradient Boosting. Okay? So in
  401. 16:52these next four or five videos, we are
  402. 16:55going to do exactly that, we will study
  403. 16:57all these types in great detail and
  404. 17:00learn to use them. Okay? Now a little
  405. 17:04discussion on why these different types
  406. 17:07exist? These different types exist only
  407. 17:12because, as I told you a while ago, the
  408. 17:14most important thing in ensemble
  409. 17:16learning is that your base models
  410. 17:17should be different. Okay? So how will
  411. 17:21that difference come about? We studied
  412. 17:24two different things. First, you use
  413. 17:26different algorithms. And second, you
  414. 17:29use the same algorithm but provide
  415. 17:31different data. So, because of these
  416. 17:34two techniques, there are four
  417. 17:36different types. All right? I will give
  418. 17:40you a rough, quick, brief introduction
  419. 17:43to these four techniques. So that you
  420. 17:45are prepared. So when we study in
  421. 17:47detail later, you have some idea about
  422. 17:50it. All right? So let's start with
  423. 17:52voting. Voting ensemble depends solely
  424. 17:55on the fact that your base models are
  425. 17:58different. All right? Base models are
  426. 18:05different algorithms. So, what do you
  427. 18:08do here? Suppose you pick three
  428. 18:11different algorithms. One is SVM, one
  429. 18:15is logistic regression, and one is a
  430. 18:19decision tree. And you have a dataset.
  431. 18:23Let's call it D. And you provide this
  432. 18:26same dataset to all three and train all
  433. 18:29three models. As soon as all three
  434. 18:32models are trained, and now I have a
  435. 18:34new query point. What will I do now? I
  436. 18:38will send this same query point to
  437. 18:40these three trained models. And if it's
  438. 18:45a classification problem, I will find
  439. 18:47the result through majority voting
  440. 18:49count, and if it's a regression problem
  441. 18:52, I will calculate the mean. So,
  442. 18:54basically, voting is very simple. The
  443. 18:58difference or variety you are creating
  444. 19:00here comes from the fact that all your
  445. 19:03base models are different algorithms.
  446. 19:06They are all getting the same data;
  447. 19:08they are different algorithms. And
  448. 19:10since there are different algorithms,
  449. 19:12there is variety in your ensemble, and
  450. 19:14it gives you better results. All right?
  451. 19:16So the logic here is very simple. Now,
  452. 19:19the next is stacking. I am teaching
  453. 19:22stacking first because stacking is
  454. 19:24conceptually very close to voting. So,
  455. 19:27what do you do in voting? In voting,
  456. 19:29you take three or any number of
  457. 19:32algorithms of different types and
  458. 19:34provide them with the same data. And
  459. 19:37when you have to make a prediction, how
  460. 19:40do you make it? You give a new query
  461. 19:43point to everyone and then calculate
  462. 19:45the majority count accordingly. In the
  463. 19:48second one, you just do one thing
  464. 19:50differently; you have the same setup
  465. 19:52here. This is the same setup. But here,
  466. 19:54you introduce one more machine learning
  467. 19:56algorithm. Here, you introduce one more
  468. 19:59machine learning algorithm. Let's say
  469. 20:01you introduced a KNN. Let's say you
  470. 20:04placed KNN here. All right? Now, what
  471. 20:07will this do is that this KNN model
  472. 20:12will also be trained. How will it be
  473. 20:14trained? Suppose in the training phase,
  474. 20:18when you sent the first data to these
  475. 20:21three, it predicted one, it predicted
  476. 20:24zero, and it predicted one. Right? So
  477. 20:28what was the output? One. So now the
  478. 20:31input KN will get is this. The data for
  479. 20:34KN, just like the data for these three
  480. 20:36models, is this data set. But the data
  481. 20:39for KN will be this. What did SVM say?
  482. 20:42What did LOR say? As in, what did
  483. 20:44Logistic Regression say? What did the
  484. 20:46Decision Tree say? And what was the
  485. 20:48result in the end? So here it was one.
  486. 20:50Here it was zero. Here it was one and
  487. 20:51the result was also one. Now for the
  488. 20:53next training point, suppose SVM said
  489. 20:55zero, LOR said zero, Decision Tree said
  490. 20:57one. But the result was still one. So
  491. 21:00now this data will be given to this
  492. 21:02model and this model will be trained on
  493. 21:05this data, and after training on this
  494. 21:07data, what will it do? Its main job is
  495. 21:10to assign a weightage to these base
  496. 21:13models. So it will give W1 to this, W2
  497. 21:16to this, and W3 to this. So basically,
  498. 21:19what you are doing now is you just made
  499. 21:21a small change in voting. I mean, what
  500. 21:24happens in democracy? Everyone's voting
  501. 21:26rights are equal. My vote will count as
  502. 21:28one and Mukesh Ambani's vote will also
  503. 21:30count as one. Right? But in stacking,
  504. 21:33you are just creating a difference
  505. 21:36where you analyze which model is
  506. 21:38talking more sense and you give it more
  507. 21:41weightage. Which model is making a few
  508. 21:44mistakes, you give it slightly less
  509. 21:46weightage. So imagine that in a
  510. 21:48different democracy, Mukesh Ambani's
  511. 21:51vote weightage would be higher than my
  512. 21:54vote's weightage. Meaning his say would
  513. 21:57be more important than my say. Because
  514. 21:59for whatever reasons, he is a rich guy
  515. 22:01or whatever. Right? So the same factor
  516. 22:04is being applied here. This model's job
  517. 22:07is to decide the weight of these models
  518. 22:09. Why decide the weight? Because it is
  519. 22:11not necessary that all three models
  520. 22:13will be equal. One model might be
  521. 22:14better, one model might be worse. So
  522. 22:16you also want to see who is good and
  523. 22:18who is bad. So this is called stacking.
  524. 22:20Right? Stacking and voting are
  525. 22:22conceptually the same. There is just
  526. 22:25one small difference in stacking where
  527. 22:27you give additional weightage to the
  528. 22:28models, which you don't do in voting.
  529. 22:31In voting, all your models 'weight, say
  530. 22:33, or output are considered equal. Right
  531. 22:36? I hope you have understood packing
  532. 22:38and voting. Now let's talk about
  533. 22:40bagging. Bagging is a variant of the
  534. 22:43ensemble learning technique. Let's talk
  535. 22:46about this quickly as well. So, first
  536. 22:49of all, the full form of bagging is
  537. 22:54Bagging stands for bootstrap Bootstrap
  538. 23:01aggregation. Okay? The concept is very
  539. 23:07simple, let me explain. Here, the
  540. 23:10variety you create is not achieved by
  541. 23:13choosing different base models or
  542. 23:15different algorithms; no, here you keep
  543. 23:17the same algorithm. The difference, or
  544. 23:21the variety, comes from the fact that
  545. 23:24you give each model different data.
  546. 23:27That is the main idea. So I will show
  547. 23:29you. Suppose you have three base models
  548. 23:35and all three are SVMs. Support Vector
  549. 23:39Machines or Logistic Regression. It's
  550. 23:42up to you. No problem. All three are
  551. 23:44the same models. So at this point,
  552. 23:45there is no difference. All three are
  553. 23:47the exact same individuals. They have
  554. 23:48the same education. They have the same
  555. 23:50knowledge. Now, when you receive data.
  556. 23:53When the data comes to you. This is D.
  557. 23:57Now, suppose you have data for 1000
  558. 24:00students and you need to create variety
  559. 24:03. What will you do? You will perform
  560. 24:06bootstrapping. What do you mean by
  561. 24:08bootstrapping? You will give each of
  562. 24:09them different data. You will give D1
  563. 24:11to this, D2 to this, and D3 to this.
  564. 24:13Now, how will D1 be created? You
  565. 24:15decided that I will show each model
  566. 24:17data of 500 students. So what did you
  567. 24:19do? You randomly picked data of 500
  568. 24:22students from it, which we call
  569. 24:25bootstrapping, and you are calling that
  570. 24:29D1, and you fed that same D1 here.
  571. 24:32Right? Then what did you do? You put
  572. 24:34the data of these 500 students back
  573. 24:36inside. Put it back into the original
  574. 24:38data. And you randomly picked 500
  575. 24:41students' data again. This is D2. Now
  576. 24:44there is no guarantee that D1 and D2
  577. 24:46will be the same. In fact, there is a
  578. 24:48very high probability that both will be
  579. 24:50very different data. Now you took D2
  580. 24:53and gave it to model two. Then you put
  581. 24:56the data of these 500 students back
  582. 24:59inside, and for the third time, you
  583. 25:01created D3 by randomly picking 500
  584. 25:03students 'data, and you gave this D3 to
  585. 25:06model three. Now, since they are the
  586. 25:08same machine learning models, if you
  587. 25:10give them different data, they will be
  588. 25:13trained differently, and if they are
  589. 25:15trained differently, the variety we
  590. 25:17wanted is achieved. The rest of the
  591. 25:19process is exactly the same. Now, if
  592. 25:21you have a new query point, you will
  593. 25:24give it to all three. You take the
  594. 25:26output of all three, calculate the
  595. 25:27majority count, or find the mean.
  596. 25:29Depending on the problem you are trying
  597. 25:30to solve. And that is how bagging works
  598. 25:33. Okay? A special case of this is
  599. 25:35Random Forest. Random Forest is a
  600. 25:38special case of bagging. Where the base
  601. 25:41models used are always decision trees.
  602. 25:44And that is why the name is forest. A
  603. 25:46collection of trees. Okay? If you are
  604. 25:49making an ensemble of any other
  605. 25:51algorithm, it is called bagging. But if
  606. 25:53you are building it with decision trees
  607. 25:55, it is called Random Forest. That is
  608. 25:57the biggest difference. Okay? So I hope
  609. 25:59you have understood bagging as well.
  610. 26:01The last one is boosting, which is the
  611. 26:03most powerful technique. And the most
  612. 26:07useful technique. The concept of
  613. 26:10boosting is a bit different. In
  614. 26:12boosting, you also take the same models
  615. 26:16. You take the same model. Suppose this
  616. 26:19is your logistic regression. This is
  617. 26:23also logistic regression, this is also
  618. 26:25logistic regression. Now, here you take
  619. 26:28your data and give it to the first
  620. 26:32model. The first model trains on it and
  621. 26:37notes down what mistakes it is making.
  622. 26:42Okay? What mistakes it is making.
  623. 26:44Basically, if I showed it data of 1000
  624. 26:46students, how many students did it
  625. 26:48predict incorrectly? It keeps that in
  626. 26:52mind. And then, the mistakes it made,
  627. 26:55it passes on to the next model. It says
  628. 26:58, "Look here, these are the students I
  629. 27:00made mistakes on.""You work carefully
  630. 27:02on these." And since the next model is
  631. 27:05specifically briefed about that data
  632. 27:07where the errors occurred. "You work
  633. 27:09carefully." So now, what will this
  634. 27:11model do? It starts giving correct
  635. 27:13results on those wrong points. It
  636. 27:15starts giving slightly improved results
  637. 27:16. But then it has its own mistakes. So
  638. 27:19it passes its own mistakes to the next
  639. 27:21one. So in a way, you are boosting,
  640. 27:24boosting your model performance. And
  641. 27:28you tell the next one every time, and
  642. 27:30doing this in a series, what does your
  643. 27:32model do? It starts making fewer
  644. 27:35mistakes. Okay? And that is how
  645. 27:36boosting works. Honestly, it's a bit of
  646. 27:38a complex topic, and I won't be able to
  647. 27:40explain it within five minutes. But
  648. 27:42this is the core idea. The second model
  649. 27:44reduces the mistakes of the first model
  650. 27:45. The third model reduces the mistakes
  651. 27:47of the second model. And you keep doing
  652. 27:48this in a series. And eventually, you
  653. 27:51get a model, a big model that is
  654. 27:53reducing overall errors. All right? So
  655. 27:55we will study boosting in great detail.
  656. 27:58It is the most powerful machine
  657. 27:59learning algorithm or technique. So we
  658. 28:02are going to study this thoroughly. So
  659. 28:03don't worry about that. But the core
  660. 28:05idea is exactly this. All right? So
  661. 28:06these are the different types of
  662. 28:08ensemble techniques you need to study,
  663. 28:10precisely four, and then different
  664. 28:12algorithms within them. But yeah, I
  665. 28:14hope I have given you a clear-cut
  666. 28:16overview. All right? Now, at this point
  667. 28:19, you might have some questions in your
  668. 28:21mind: why does ensemble learning
  669. 28:23actually work? What is the reason it
  670. 28:25works so well and gives good results?
  671. 28:27Why are multiple models able to provide
  672. 28:29better results than a single model? I
  673. 28:31will tell you that in the next section,
  674. 28:34where we discuss why ensemble learning
  675. 28:36works. All right? So guys, now I will
  676. 28:39try to explain it from a pure machine
  677. 28:41learning perspective, why ensemble
  678. 28:43learning is able to work. All right? I
  679. 28:46found a great image while searching on
  680. 28:48Google. First, I will explain it in
  681. 28:51terms of classification, and then I
  682. 28:52will explain it in terms of regression.
  683. 28:54So, if you talk about classification,
  684. 28:56let's assume you have this dataset. All
  685. 28:59right? So here, there are three
  686. 29:01different classes. Square is one class,
  687. 29:03triangle is one class, and circle is
  688. 29:05one class. And the black and white one
  689. 29:08is the training data. And the green one
  690. 29:11is the testing data. And this is your
  691. 29:14model one. This is model one. All right
  692. 29:17? It could be any model. And this is
  693. 29:20the decision boundary of this model.
  694. 29:23According to model one, this is the
  695. 29:25decision boundary. All three classes
  696. 29:28will be separated in this way. All
  697. 29:29right? Now you trained model two on the
  698. 29:33same data. And model two said that your
  699. 29:36decision boundary should be this. Based
  700. 29:39on its own learning, based on its
  701. 29:40training. This should be the decision
  702. 29:42boundary. And when you trained a third
  703. 29:44model, model three, on the same data,
  704. 29:47it said the decision boundary should be
  705. 29:49something like this. Right? Now, if you
  706. 29:52stack these three decision boundaries
  707. 29:54on top of each other, you will see
  708. 29:57something like this. Here are the
  709. 29:59decision boundaries of all three. And
  710. 30:00you can see that there is a slight
  711. 30:01difference. There is a slight
  712. 30:03difference in their opinions. Right?
  713. 30:05Now, what will ensemble learning do?
  714. 30:07Using majority count, it will use the
  715. 30:10concept of majority count. And by
  716. 30:13applying majority count, what is
  717. 30:15happening is that this decision
  718. 30:17boundary, as you can see, has become a
  719. 30:19bit smoother, or in short, it has
  720. 30:21become better than what it was. And
  721. 30:24that is how ensemble learning works in
  722. 30:26machine learning. Okay? You have to
  723. 30:29take data from multiple models, remove
  724. 30:32the noise from their decision-making,
  725. 30:34and extract a better model for yourself
  726. 30:37. And that is how it works in
  727. 30:39classification. Okay? You can imagine
  728. 30:41the same thing in regression as well.
  729. 30:42Suppose this is a dataset and you need
  730. 30:45a regression model. Now, different
  731. 30:48models are saying different things.
  732. 30:49This is one model which says it should
  733. 30:51be at this angle. The green one says it
  734. 30:53should be a bit higher. The red and
  735. 30:55orange ones are saying something else.
  736. 30:56But what are you doing? You are
  737. 30:58calculating the mean of all of them. As
  738. 31:00soon as you calculate the mean, you
  739. 31:02arrive somewhere in the middle here.
  740. 31:04This is the line. Okay? And it is
  741. 31:08neither too extreme in this direction
  742. 31:10nor too extreme in that direction. This
  743. 31:12is a model in the middle that will
  744. 31:13perform better than all those models.
  745. 31:16And that is the intuition. I mean,
  746. 31:18there is nothing much to explain here.
  747. 31:19Just by looking at these photos and
  748. 31:21images, an intuition will start forming
  749. 31:23in your mind about why ensemble
  750. 31:25learning works. Okay? So, I hope I was
  751. 31:27able to explain this to you. Now, let's
  752. 31:30discuss what the key benefits of using
  753. 31:32ensemble learning are. So guys, now
  754. 31:35let's discuss the benefits of using
  755. 31:37ensemble techniques. Now look, let's
  756. 31:40discuss a clear-cut disadvantage first
  757. 31:43because there is one big disadvantage.
  758. 31:45Pause the video for a second and think
  759. 31:47about what that big disadvantage is.
  760. 31:49That disadvantage is that if you want
  761. 31:51to use an ensemble technique. It means
  762. 31:54you will have to train multiple models.
  763. 31:56So, where earlier you used to train
  764. 31:58just one model to get your work done.
  765. 31:59Now, you are training multiple models
  766. 32:01there. Obviously, if you train more
  767. 32:03models, the computational complexity
  768. 32:05increases. Right? Which is not a good
  769. 32:08thing. So, that means there must be
  770. 32:09some solid advantage because of which
  771. 32:11we would use ensemble learning. Because
  772. 32:13if there is an obvious disadvantage,
  773. 32:16then there must be a big advantage to
  774. 32:18use it. So, there are actually
  775. 32:20advantages here. The very first
  776. 32:22advantage I have clearly told you until
  777. 32:24now. That is improvement in performance
  778. 32:29. Improvement in performance. That is
  779. 32:33the biggest one. The second one is that
  780. 32:39it reduces bias and variance. I guess
  781. 32:44you have already studied bias and
  782. 32:46variance. The job of a machine learning
  783. 32:48model is to make predictions. There are
  784. 32:50errors in predictions, and there are
  785. 32:52two types of errors. It is either bias
  786. 32:54or it is variance. Bias means the
  787. 32:56result is not good even on the training
  788. 32:58data. Variance means that your model is
  789. 33:01giving very different results on
  790. 33:03different data. It is varying. There is
  791. 33:05too much variance. These are the only
  792. 33:06two types of errors you have. Right?
  793. 33:08Now your goal in machine learning, the
  794. 33:11end goal in applied machine learning,
  795. 33:14is to have low bias, meaning not having
  796. 33:18high error on training data, plus low
  797. 33:21variance, meaning if you make changes
  798. 33:24in the data, the model's performance
  799. 33:27should not change significantly. These
  800. 33:31are your end goals. So, ensemble
  801. 33:34learning helps you achieve this. So, if
  802. 33:39you have a model that is a low bias,
  803. 33:45but a high variance model, it means it
  804. 33:47makes very few mistakes on training
  805. 33:50data. But if you change the data, its
  806. 33:52errors increase. So, a model like this,
  807. 33:55for example, a decision tree model. A
  808. 33:57decision tree is a low bias, high
  809. 33:58variance model. So, if you apply
  810. 34:00ensemble learning on top of it, what
  811. 34:02will it do? It will keep the low bias
  812. 34:05as it is, but convert the high variance
  813. 34:08into low variance. In short, it will
  814. 34:10give you what you want. Low bias, low
  815. 34:13variance, and similarly, if you have a
  816. 34:16high bias, low variance model, like
  817. 34:21logistic regression or linear
  818. 34:23regression, in that case, it will
  819. 34:26convert high bias to low bias and keep
  820. 34:29the low variance as it is. Again, in
  821. 34:31short, it will give you this result. So
  822. 34:34, the biggest challenge in applied
  823. 34:36machine learning is to reduce both bias
  824. 34:39and variance. That is generally not
  825. 34:41possible because increasing one
  826. 34:43decreases the other. It has a kind of
  827. 34:46inversely proportional or inverse
  828. 34:47relationship. Ensemble learning helps
  829. 34:51in achieving this difficult task. We
  830. 34:56will discuss this point in more detail
  831. 34:58when we study all these techniques one
  832. 35:00by one. But at this point, just
  833. 35:03understand that ensemble learning helps
  834. 35:05you achieve low bias and low variance.
  835. 35:07Which is quite a difficult thing to
  836. 35:09achieve. Okay? And the third most
  837. 35:12important thing is robustness. Your
  838. 35:18model becomes robust for many different
  839. 35:21types of data. Even if you make many
  840. 35:23changes to the data, your performance
  841. 35:25does not fluctuate much. Again, this is
  842. 35:27related to variance. But robustness is
  843. 35:30explained in a different way, used as a
  844. 35:32key point, a key benefit of using
  845. 35:35ensemble learning. Okay? I hope you are
  846. 35:37understanding everything up to this
  847. 35:38point. Okay? Now, the last point that
  848. 35:42we will discuss in this video is when
  849. 35:44one should use ensemble learning. I
  850. 35:48will give you the answer in short. The
  851. 35:50answer is always. There is no reason
  852. 35:54not to use ensemble learning. If you
  853. 35:57are a data scientist in a company, you
  854. 36:00must use ensemble learning in your
  855. 36:03projects. Generally, ensemble learning
  856. 36:05is tried at the end after doing
  857. 36:07everything else. When you have finished
  858. 36:09data cleaning, pre-processing, feature
  859. 36:11engineering, model building, and
  860. 36:13evaluation. Apply ensemble learning at
  861. 36:16the end, and you would see that your
  862. 36:18results will improve most of the time.
  863. 36:20Okay? So even if you are working in a
  864. 36:22company, you should use ensemble
  865. 36:24learning. Or if you are in a Kaggle
  866. 36:27competition, then you should apply
  867. 36:29ensemble learning there as well. In
  868. 36:32fact, Kaggle competitions became famous
  869. 36:34precisely because people were able to
  870. 36:37get great results by applying ensemble
  871. 36:39learning. Okay? XGBoost became very
  872. 36:42famous because of Kaggle. Okay? So in
  873. 36:44short, I have given you a complete
  874. 36:46introduction to ensemble learning. Now,
  875. 36:49what will we do? We will study all
  876. 36:50ensemble techniques one by one. We will
  877. 36:53start with voting. Then we will study
  878. 36:55bagging. Then random forest, after that
  879. 36:57boosting algorithms, and finally I will
  880. 37:00teach you stacking. The goal is that
  881. 37:02you should understand ensemble learning
  882. 37:05thoroughly. And be able to use it. I
  883. 37:07will teach you in such a way, and guess
  884. 37:09what? Once you have learned this, many
  885. 37:11times you won't even need deep learning
  886. 37:14. I mean, from what I have seen in my
  887. 37:17experience, if you have a small dataset
  888. 37:20, 95%of the time your ensemble
  889. 37:21technique algorithms will beat deep
  890. 37:24learning. But if the data is large,
  891. 37:27then deep learning is a different thing
  892. 37:29. So yeah, I hope I have given you a
  893. 37:32clear, bird's-eye view of all the
  894. 37:35ensemble techniques. If you liked the
  895. 37:38video, guys, please consider
  896. 37:39subscribing. Thanks for watching. Bye.

About this transcript

This page contains the full transcript of Introduction to Ensemble Learning | Ensemble Techniques in Machine Learning by CampusX, generated from the public captions YouTube serves with the video. The transcript has 6,065 words across 896 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.