YouTube2Text

Kernel Trick in SVM | Code Example — Transcript

by CampusX · 2,376 words · 333 segments · language en · Watch on YouTube

Full transcript

  1. 0:01So hi guys, we just discussed what the
  2. 0:04kernel trick is in SVM, and although we
  3. 0:07haven't studied the various kernel
  4. 0:10functions like RBF or polynomial in
  5. 0:12deep mathematical detail yet. But still
  6. 0:16, what I will do is I will show you one
  7. 0:19example, a live code, where you can
  8. 0:21actually see the power of the kernel
  9. 0:24trick. Okay? So, I will take a
  10. 0:26non-linear dataset and apply this
  11. 0:28kernel trick to it, and you will be
  12. 0:30able to see how you are able to
  13. 0:32classify non-linear data as well using
  14. 0:35linear surfaces. Okay, so the math is a
  15. 0:38bit difficult, so at this point, I will
  16. 0:41skip it and I just want to create this
  17. 0:43understanding so that you get a
  18. 0:45conceptual understanding of how this
  19. 0:48whole thing works. Okay? So let's start
  20. 0:51. I will share my screen. Yeah, so I
  21. 0:57have already created a notebook. I will
  22. 1:03just maximize the window. Yeah, so guys
  23. 1:05, I have already created the window. I
  24. 1:08am sorry, I have already created the
  25. 1:10notebook. So these are some of the
  26. 1:12imports, I will share this notebook. Uh
  27. 1:14, let me run this once, and after that,
  28. 1:18using a scikit-learn sample generator
  29. 1:21class, I have created a dataset which
  30. 1:24looks like this. Okay? It's a
  31. 1:27non-linear sort of dataset. Meaning if
  32. 1:30you try to draw a line anywhere in it.
  33. 1:33You won't get a good classifier. Right?
  34. 1:36Because you can see for obvious reasons
  35. 1:38. Okay? Now what do we do? The approach
  36. 1:41would be that first I will try applying
  37. 1:44a normal SVM on this. Where you will
  38. 1:47apply a linear SVM. And you would see
  39. 1:49that the results won't be good. And
  40. 1:52then we will replace it with a
  41. 1:53non-linear kernel, that is the RBF
  42. 1:55kernel or the polynomial kernel. Okay?
  43. 1:58So let me show you what I have done
  44. 2:00here. First of all, these are some
  45. 2:02obvious steps. I had 100 data points,
  46. 2:04by the way, if you notice above. So I
  47. 2:07have generated 100 data points. Okay?
  48. 2:09And they are stored in my X and y. In X
  49. 2:12, I have stored their x and y
  50. 2:14coordinates, and in y, I have stored
  51. 2:17their labels. Okay? So there are two
  52. 2:19labels, red and blue. Now if I go back,
  53. 2:23I have done this train-test split,
  54. 2:26right? After that, what we did was
  55. 2:30create a classifier object of the SVC
  56. 2:33class, okay? We created a Support
  57. 2:36Vector Classifier and explicitly
  58. 2:37mentioned that we want a linear kernel,
  59. 2:40right? Meaning, I want to perform a
  60. 2:43linear SVM, and then we fitted it and
  61. 2:46calculated the y-predict, okay? And
  62. 2:49after calculating the y-predict, I
  63. 2:51calculated the accuracy score, and it
  64. 2:54came out to be 55%, only 55%. And it
  65. 2:56was bound to happen, right? I mean, our
  66. 2:59data was totally non-linear, and we
  67. 3:00were applying a linear classifier on
  68. 3:02top of it, so the results were bound to
  69. 3:04be poor. In fact, what I have done is I
  70. 3:06have retained this function, plot
  71. 3:09decision boundary. Using this, you can
  72. 3:12actually see how our linear SVM tried
  73. 3:14to classify it. And yes, here it is.
  74. 3:18Here is my test dataset. I passed the
  75. 3:23test dataset and my classifier object,
  76. 3:26and it plotted it; in fact, let me give
  77. 3:30you, let me give the entire X and the
  78. 3:34entire y. And you can see what is
  79. 3:40happening. Okay? So, it misclassified
  80. 3:43almost all these blue points because,
  81. 3:45obviously, linear surfaces weren't
  82. 3:47going to work on this. Okay? Now, what
  83. 3:51can be done? What can be the solution?
  84. 3:53So, what did we do? We tried to
  85. 3:56transform our data. Meaning, my data is
  86. 4:00in 2D. But I will bring it into 3D.
  87. 4:03Okay? And first, I will show you what
  88. 4:05function I used, and then I will
  89. 4:07explain the logic behind how that
  90. 4:09function is able to do this. Okay? So,
  91. 4:12I created a function called plot 3D
  92. 4:15plot and took every X, every X point,
  93. 4:19and applied this transformation on it.
  94. 4:23Okay? So, this is called a Radial Basis
  95. 4:26Function. An RBF kernel is a Radial
  96. 4:29Basis Function. We will read a little
  97. 4:32more about this in detail in a while.
  98. 4:35But for now, just understand that for
  99. 4:37every point of mine, for every X point
  100. 4:39of mine—what was in X? My 100 points
  101. 4:42'coordinates, the X and Y coordinates.
  102. 4:44I calculated an R for every point. You
  103. 4:48can think of this R as that particular
  104. 4:50point's value on the Z-axis. Okay? So,
  105. 4:54I passed my X and Y coordinates into
  106. 4:56this formula and calculated R for every
  107. 4:58point. Okay? And using that R, I
  108. 5:02plotted a 3D scatter plot by giving my
  109. 5:06X coordinate, my Y coordinate, and my Z
  110. 5:09coordinate. Okay? And as soon as I ran
  111. 5:13this, guys, look at this. What did I do
  112. 5:16with that non-linear concentric circle
  113. 5:18type dataset of mine? I took the red
  114. 5:21points in the center and lifted them up
  115. 5:24. And the blue points are still right
  116. 5:25where they were. And now that this
  117. 5:27dataset is in 3D. What you can do is
  118. 5:30you can create a decision boundary with
  119. 5:34a plane. And you can very easily
  120. 5:36classify your points. So the idea we
  121. 5:39discussed in the last video is clearly
  122. 5:42visible here, that because we applied
  123. 5:45that function, your 2D dataset moved
  124. 5:47into 3D. And you are easily able to
  125. 5:51classify it using a plane. Okay? Now
  126. 5:54what I will do is I will explain how
  127. 5:56this function works. Okay? So first of
  128. 5:58all, it's a radial basis function. What
  129. 6:00is a radial basis function? You guys
  130. 6:02will study this in more detail. We will
  131. 6:04discuss it in the next video. But for
  132. 6:07now, understand that this function we
  133. 6:09used, e to the power-x², is a radial
  134. 6:12basis function, and what did it do? It
  135. 6:16transformed our data into 3D space.
  136. 6:19Let's discuss how. Okay? So if you
  137. 6:22think about it, when our data was
  138. 6:24initially here, looking at it, it was
  139. 6:27clear that if I want to classify it
  140. 6:29linearly in any way and take it to a
  141. 6:31higher dimension, one approach could be
  142. 6:34that I lift the center points up a bit,
  143. 6:37right? That could be one approach. So
  144. 6:40that means I have to apply a
  145. 6:41transformation on it. But what will
  146. 6:43that transformation function be? So
  147. 6:45there are only a few radial basis
  148. 6:46functions. There are only three or four
  149. 6:47. You try some of those, and one of
  150. 6:49them is this function. Let me show you.
  151. 6:52This Desmos we discussed earlier as
  152. 6:53well. So you used a function, y = e to
  153. 6:58the power-x². And guys, this is the
  154. 7:04function. Look, this is the function.
  155. 7:07If you have these data points and you
  156. 7:09apply this transformation to them, then
  157. 7:12automatically what will happen? Your
  158. 7:14center points will fly up a little.
  159. 7:17Right? In comparison to other points.
  160. 7:19And exactly this same transformation we
  161. 7:21applied to our dataset, so the
  162. 7:23z-coordinate of our center points
  163. 7:25lifted up a bit and the rest stayed
  164. 7:27down. Okay? Because of this
  165. 7:29transformation, we were able to create
  166. 7:31a new feature space where we could
  167. 7:33differentiate these points. Okay? Now
  168. 7:35this thing is very tricky. Right? The
  169. 7:37math behind this is very tricky. It's
  170. 7:39because, I mean, right now you can see
  171. 7:41that the data was in the center. But in
  172. 7:43high-dimensional space, you won't even
  173. 7:45be able to see it, and it becomes very
  174. 7:47important to determine what to consider
  175. 7:49the center to apply this transformation
  176. 7:51. So what does SVM do there? It takes
  177. 7:53every data point and performs this task
  178. 7:56once. How? I will show you. For example
  179. 7:58, suppose if your data was shifted
  180. 8:00slightly, if the red points were
  181. 8:01shifted a bit, you could apply this
  182. 8:03transformation. I will show you. Uh x-5
  183. 8:10And I will put it like this. Look guys,
  184. 8:13if I had applied this transformation,
  185. 8:14if our point was centered slightly to
  186. 8:16this side, it would have come up. Okay?
  187. 8:19If you need to tackle this side, you
  188. 8:21would apply this transformation. So
  189. 8:23what you will do is apply these
  190. 8:25transformations for every point. Okay?
  191. 8:28And then you will find out, I mean SVM
  192. 8:31will find out, which of these gives the
  193. 8:34best result. Okay? And obviously, this
  194. 8:36is computationally very, very expensive
  195. 8:37. Right? As the data keeps increasing,
  196. 8:40applying these transformations becomes
  197. 8:42very difficult. And this is where the
  198. 8:44best thing about SVM is, that you don't
  199. 8:46actually have to perform these
  200. 8:47transformations. If you think, uh, I
  201. 8:49will show you in a little while. You
  202. 8:51don't actually have to do these
  203. 8:53transformations. You don't actually
  204. 8:55need to create this third space. I made
  205. 8:57this just to show you. This 3D plot
  206. 9:00diagram. I don't have to give this to
  207. 9:03the algorithm. I just showed it to you
  208. 9:05to demonstrate that this is happening
  209. 9:07internally. What will happen is that
  210. 9:09you never have to create a separate
  211. 9:10feature or a separate column; it will
  212. 9:12work on the same number of columns, you
  213. 9:14just change the kernel and you get your
  214. 9:16result. Unlike polynomial regression,
  215. 9:18if you remember, where you had to
  216. 9:20create a new column for every
  217. 9:21polynomial degree. Here, you don't have
  218. 9:24to create extra features. You just fit
  219. 9:26your current coordinates into that
  220. 9:28formula and you start getting results.
  221. 9:30We will discuss this part in the math
  222. 9:31section. When we talk about it further
  223. 9:33ahead. But for now, understand that
  224. 9:36because of the kernel trick, instead of
  225. 9:37actually doing the kernel
  226. 9:39transformation, you just plug the
  227. 9:40values into that formula, calculate the
  228. 9:42dot product, and you start getting the
  229. 9:44values. So, this is a trick because of
  230. 9:47which SVM is so famous today. Okay? So
  231. 9:49now I have discussed a lot. Let me show
  232. 9:51you what you will do now. What will you
  233. 9:53do now? You will create a new SVC
  234. 9:55object called RBF classifier, and this
  235. 9:57time you will set the kernel to RBF.
  236. 9:59This is not linear. Okay? And then
  237. 10:01again you will train your data and
  238. 10:04predict. And guess what? This time your
  239. 10:06accuracy score is coming out to be 100%
  240. 10:09. Right? And if you call the plot
  241. 10:12decision boundary function and plot
  242. 10:14your decision boundaries, you can
  243. 10:16actually see this is your new decision
  244. 10:18boundary that has been created on your
  245. 10:212D graph. Okay? And guess what? Just as
  246. 10:25I told you, you don't have to create
  247. 10:27any extra features. No feature
  248. 10:29engineering to do. The columns you had,
  249. 10:31the work was done on those same two
  250. 10:33columns, but this kernel performed
  251. 10:35those calculations internally. Okay? So
  252. 10:37that's the benefit. Okay? Now, one last
  253. 10:40thing I will show you, and that is that
  254. 10:42you can also use a polynomial kernel if
  255. 10:43you want. Okay? So if you talk about
  256. 10:47polynomial kernel, I will show you; if
  257. 10:50you go to SVM scikit-learn, you get the
  258. 10:53documentation here. And This is the
  259. 10:58documentation, guys. I will show you.
  260. 11:01If you go down a little here, you can
  261. 11:04set the value of the kernel. We are
  262. 11:06already doing that. We have used linear
  263. 11:08and RBF. Now we are using poly. And
  264. 11:10there is a term called degree, whose
  265. 11:12default value is three. And the degree
  266. 11:15of the polynomial kernel function is
  267. 11:16ignored by all other kernels. This
  268. 11:18degree term is only applicable when
  269. 11:21your kernel is poly, as in polynomial.
  270. 11:24So what do we do now? We will do it for
  271. 11:26the default degree of three and run it
  272. 11:29once. Actually, I have already done it.
  273. 11:31So you can see that the accuracy is
  274. 11:33even worse, around 45%, and its
  275. 11:35decision boundary looks like this. This
  276. 11:38is a polynomial kernel with a degree of
  277. 11:40three because the default value is
  278. 11:41three. Okay? Now what can you do? Using
  279. 11:44a technique like Grid Search CV, you
  280. 11:46can try with different values. Like, if
  281. 11:49I set the degree value to two and run
  282. 11:53it again. You can see now my accuracy
  283. 11:56score is one. So for a degree value of
  284. 11:59three, it was coming out to be 45%, but
  285. 12:01for a degree value of two, it's coming
  286. 12:03out to be one. Okay? Now if I plot the
  287. 12:06decision classifier boundary, you can
  288. 12:11see we have the same boundary. Okay? So
  289. 12:16, guys, that's it. I am giving you one
  290. 12:18last revision on what exactly you were
  291. 12:21doing in that function. What did you do
  292. 12:23there with that RBF function? You have
  293. 12:25this x. This is my x, so I have 100
  294. 12:28coordinates. x and y coordinates. So,
  295. 12:31this is one point. This is the second
  296. 12:34point. This is the third point. And
  297. 12:36what did you do? You just put this
  298. 12:39entire x inside that function. Okay?
  299. 12:43What did that do? You calculated e to
  300. 12:45the power-x² for your x coordinate as
  301. 12:47well. You calculated e to the power-x²
  302. 12:49for your y coordinate as well. And
  303. 12:51after that, what did you do? You summed
  304. 12:54it up along axis one. Meaning, you
  305. 12:57added the x and y coordinates. Added
  306. 12:59the x and y coordinates. And by doing
  307. 13:01so, you got this 100-length z
  308. 13:04coordinate. Okay? And then, if you plot
  309. 13:08this, uh, what did I do? I just stored
  310. 13:13it in x_new before summing and plotted
  311. 13:16x_new. So look guys, actually in 2D,
  312. 13:18this transformation looks like this. It
  313. 13:20came up in 3D, but in 2D, this
  314. 13:22transformation is like this. For these
  315. 13:25red points, both your X and Y values
  316. 13:26are high. So obviously, when you add
  317. 13:29them, the value becomes higher and you
  318. 13:31move up in the Z coordinate, and for
  319. 13:33all these, either X is high or Y is
  320. 13:34high. Okay? And that is why these
  321. 13:36stayed at the bottom. Okay? So I know a
  322. 13:38lot happened here and it might be a bit
  323. 13:41difficult. But I am sharing this
  324. 13:43notebook with you. Play around with it
  325. 13:44a little. You can try with different
  326. 13:46functions. You can try this too, and
  327. 13:48just go through the documentation once.
  328. 13:50This will also help you understand some
  329. 13:52things. Otherwise, we are going to
  330. 13:54discuss these kernels in detail in
  331. 13:56future videos. Okay? So I think I was
  332. 13:58able to help. That's it for the video.
  333. 14:01Thanks for watching.

About this transcript

This page contains the full transcript of Kernel Trick in SVM | Code Example by CampusX, generated from the public captions YouTube serves with the video. The transcript has 2,376 words across 333 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.