Kernel Trick in SVM | Code Example — Transcript
Full transcript
- 0:01So hi guys, we just discussed what the
- 0:04kernel trick is in SVM, and although we
- 0:07haven't studied the various kernel
- 0:10functions like RBF or polynomial in
- 0:12deep mathematical detail yet. But still
- 0:16, what I will do is I will show you one
- 0:19example, a live code, where you can
- 0:21actually see the power of the kernel
- 0:24trick. Okay? So, I will take a
- 0:26non-linear dataset and apply this
- 0:28kernel trick to it, and you will be
- 0:30able to see how you are able to
- 0:32classify non-linear data as well using
- 0:35linear surfaces. Okay, so the math is a
- 0:38bit difficult, so at this point, I will
- 0:41skip it and I just want to create this
- 0:43understanding so that you get a
- 0:45conceptual understanding of how this
- 0:48whole thing works. Okay? So let's start
- 0:51. I will share my screen. Yeah, so I
- 0:57have already created a notebook. I will
- 1:03just maximize the window. Yeah, so guys
- 1:05, I have already created the window. I
- 1:08am sorry, I have already created the
- 1:10notebook. So these are some of the
- 1:12imports, I will share this notebook. Uh
- 1:14, let me run this once, and after that,
- 1:18using a scikit-learn sample generator
- 1:21class, I have created a dataset which
- 1:24looks like this. Okay? It's a
- 1:27non-linear sort of dataset. Meaning if
- 1:30you try to draw a line anywhere in it.
- 1:33You won't get a good classifier. Right?
- 1:36Because you can see for obvious reasons
- 1:38. Okay? Now what do we do? The approach
- 1:41would be that first I will try applying
- 1:44a normal SVM on this. Where you will
- 1:47apply a linear SVM. And you would see
- 1:49that the results won't be good. And
- 1:52then we will replace it with a
- 1:53non-linear kernel, that is the RBF
- 1:55kernel or the polynomial kernel. Okay?
- 1:58So let me show you what I have done
- 2:00here. First of all, these are some
- 2:02obvious steps. I had 100 data points,
- 2:04by the way, if you notice above. So I
- 2:07have generated 100 data points. Okay?
- 2:09And they are stored in my X and y. In X
- 2:12, I have stored their x and y
- 2:14coordinates, and in y, I have stored
- 2:17their labels. Okay? So there are two
- 2:19labels, red and blue. Now if I go back,
- 2:23I have done this train-test split,
- 2:26right? After that, what we did was
- 2:30create a classifier object of the SVC
- 2:33class, okay? We created a Support
- 2:36Vector Classifier and explicitly
- 2:37mentioned that we want a linear kernel,
- 2:40right? Meaning, I want to perform a
- 2:43linear SVM, and then we fitted it and
- 2:46calculated the y-predict, okay? And
- 2:49after calculating the y-predict, I
- 2:51calculated the accuracy score, and it
- 2:54came out to be 55%, only 55%. And it
- 2:56was bound to happen, right? I mean, our
- 2:59data was totally non-linear, and we
- 3:00were applying a linear classifier on
- 3:02top of it, so the results were bound to
- 3:04be poor. In fact, what I have done is I
- 3:06have retained this function, plot
- 3:09decision boundary. Using this, you can
- 3:12actually see how our linear SVM tried
- 3:14to classify it. And yes, here it is.
- 3:18Here is my test dataset. I passed the
- 3:23test dataset and my classifier object,
- 3:26and it plotted it; in fact, let me give
- 3:30you, let me give the entire X and the
- 3:34entire y. And you can see what is
- 3:40happening. Okay? So, it misclassified
- 3:43almost all these blue points because,
- 3:45obviously, linear surfaces weren't
- 3:47going to work on this. Okay? Now, what
- 3:51can be done? What can be the solution?
- 3:53So, what did we do? We tried to
- 3:56transform our data. Meaning, my data is
- 4:00in 2D. But I will bring it into 3D.
- 4:03Okay? And first, I will show you what
- 4:05function I used, and then I will
- 4:07explain the logic behind how that
- 4:09function is able to do this. Okay? So,
- 4:12I created a function called plot 3D
- 4:15plot and took every X, every X point,
- 4:19and applied this transformation on it.
- 4:23Okay? So, this is called a Radial Basis
- 4:26Function. An RBF kernel is a Radial
- 4:29Basis Function. We will read a little
- 4:32more about this in detail in a while.
- 4:35But for now, just understand that for
- 4:37every point of mine, for every X point
- 4:39of mine—what was in X? My 100 points
- 4:42'coordinates, the X and Y coordinates.
- 4:44I calculated an R for every point. You
- 4:48can think of this R as that particular
- 4:50point's value on the Z-axis. Okay? So,
- 4:54I passed my X and Y coordinates into
- 4:56this formula and calculated R for every
- 4:58point. Okay? And using that R, I
- 5:02plotted a 3D scatter plot by giving my
- 5:06X coordinate, my Y coordinate, and my Z
- 5:09coordinate. Okay? And as soon as I ran
- 5:13this, guys, look at this. What did I do
- 5:16with that non-linear concentric circle
- 5:18type dataset of mine? I took the red
- 5:21points in the center and lifted them up
- 5:24. And the blue points are still right
- 5:25where they were. And now that this
- 5:27dataset is in 3D. What you can do is
- 5:30you can create a decision boundary with
- 5:34a plane. And you can very easily
- 5:36classify your points. So the idea we
- 5:39discussed in the last video is clearly
- 5:42visible here, that because we applied
- 5:45that function, your 2D dataset moved
- 5:47into 3D. And you are easily able to
- 5:51classify it using a plane. Okay? Now
- 5:54what I will do is I will explain how
- 5:56this function works. Okay? So first of
- 5:58all, it's a radial basis function. What
- 6:00is a radial basis function? You guys
- 6:02will study this in more detail. We will
- 6:04discuss it in the next video. But for
- 6:07now, understand that this function we
- 6:09used, e to the power-x², is a radial
- 6:12basis function, and what did it do? It
- 6:16transformed our data into 3D space.
- 6:19Let's discuss how. Okay? So if you
- 6:22think about it, when our data was
- 6:24initially here, looking at it, it was
- 6:27clear that if I want to classify it
- 6:29linearly in any way and take it to a
- 6:31higher dimension, one approach could be
- 6:34that I lift the center points up a bit,
- 6:37right? That could be one approach. So
- 6:40that means I have to apply a
- 6:41transformation on it. But what will
- 6:43that transformation function be? So
- 6:45there are only a few radial basis
- 6:46functions. There are only three or four
- 6:47. You try some of those, and one of
- 6:49them is this function. Let me show you.
- 6:52This Desmos we discussed earlier as
- 6:53well. So you used a function, y = e to
- 6:58the power-x². And guys, this is the
- 7:04function. Look, this is the function.
- 7:07If you have these data points and you
- 7:09apply this transformation to them, then
- 7:12automatically what will happen? Your
- 7:14center points will fly up a little.
- 7:17Right? In comparison to other points.
- 7:19And exactly this same transformation we
- 7:21applied to our dataset, so the
- 7:23z-coordinate of our center points
- 7:25lifted up a bit and the rest stayed
- 7:27down. Okay? Because of this
- 7:29transformation, we were able to create
- 7:31a new feature space where we could
- 7:33differentiate these points. Okay? Now
- 7:35this thing is very tricky. Right? The
- 7:37math behind this is very tricky. It's
- 7:39because, I mean, right now you can see
- 7:41that the data was in the center. But in
- 7:43high-dimensional space, you won't even
- 7:45be able to see it, and it becomes very
- 7:47important to determine what to consider
- 7:49the center to apply this transformation
- 7:51. So what does SVM do there? It takes
- 7:53every data point and performs this task
- 7:56once. How? I will show you. For example
- 7:58, suppose if your data was shifted
- 8:00slightly, if the red points were
- 8:01shifted a bit, you could apply this
- 8:03transformation. I will show you. Uh x-5
- 8:10And I will put it like this. Look guys,
- 8:13if I had applied this transformation,
- 8:14if our point was centered slightly to
- 8:16this side, it would have come up. Okay?
- 8:19If you need to tackle this side, you
- 8:21would apply this transformation. So
- 8:23what you will do is apply these
- 8:25transformations for every point. Okay?
- 8:28And then you will find out, I mean SVM
- 8:31will find out, which of these gives the
- 8:34best result. Okay? And obviously, this
- 8:36is computationally very, very expensive
- 8:37. Right? As the data keeps increasing,
- 8:40applying these transformations becomes
- 8:42very difficult. And this is where the
- 8:44best thing about SVM is, that you don't
- 8:46actually have to perform these
- 8:47transformations. If you think, uh, I
- 8:49will show you in a little while. You
- 8:51don't actually have to do these
- 8:53transformations. You don't actually
- 8:55need to create this third space. I made
- 8:57this just to show you. This 3D plot
- 9:00diagram. I don't have to give this to
- 9:03the algorithm. I just showed it to you
- 9:05to demonstrate that this is happening
- 9:07internally. What will happen is that
- 9:09you never have to create a separate
- 9:10feature or a separate column; it will
- 9:12work on the same number of columns, you
- 9:14just change the kernel and you get your
- 9:16result. Unlike polynomial regression,
- 9:18if you remember, where you had to
- 9:20create a new column for every
- 9:21polynomial degree. Here, you don't have
- 9:24to create extra features. You just fit
- 9:26your current coordinates into that
- 9:28formula and you start getting results.
- 9:30We will discuss this part in the math
- 9:31section. When we talk about it further
- 9:33ahead. But for now, understand that
- 9:36because of the kernel trick, instead of
- 9:37actually doing the kernel
- 9:39transformation, you just plug the
- 9:40values into that formula, calculate the
- 9:42dot product, and you start getting the
- 9:44values. So, this is a trick because of
- 9:47which SVM is so famous today. Okay? So
- 9:49now I have discussed a lot. Let me show
- 9:51you what you will do now. What will you
- 9:53do now? You will create a new SVC
- 9:55object called RBF classifier, and this
- 9:57time you will set the kernel to RBF.
- 9:59This is not linear. Okay? And then
- 10:01again you will train your data and
- 10:04predict. And guess what? This time your
- 10:06accuracy score is coming out to be 100%
- 10:09. Right? And if you call the plot
- 10:12decision boundary function and plot
- 10:14your decision boundaries, you can
- 10:16actually see this is your new decision
- 10:18boundary that has been created on your
- 10:212D graph. Okay? And guess what? Just as
- 10:25I told you, you don't have to create
- 10:27any extra features. No feature
- 10:29engineering to do. The columns you had,
- 10:31the work was done on those same two
- 10:33columns, but this kernel performed
- 10:35those calculations internally. Okay? So
- 10:37that's the benefit. Okay? Now, one last
- 10:40thing I will show you, and that is that
- 10:42you can also use a polynomial kernel if
- 10:43you want. Okay? So if you talk about
- 10:47polynomial kernel, I will show you; if
- 10:50you go to SVM scikit-learn, you get the
- 10:53documentation here. And This is the
- 10:58documentation, guys. I will show you.
- 11:01If you go down a little here, you can
- 11:04set the value of the kernel. We are
- 11:06already doing that. We have used linear
- 11:08and RBF. Now we are using poly. And
- 11:10there is a term called degree, whose
- 11:12default value is three. And the degree
- 11:15of the polynomial kernel function is
- 11:16ignored by all other kernels. This
- 11:18degree term is only applicable when
- 11:21your kernel is poly, as in polynomial.
- 11:24So what do we do now? We will do it for
- 11:26the default degree of three and run it
- 11:29once. Actually, I have already done it.
- 11:31So you can see that the accuracy is
- 11:33even worse, around 45%, and its
- 11:35decision boundary looks like this. This
- 11:38is a polynomial kernel with a degree of
- 11:40three because the default value is
- 11:41three. Okay? Now what can you do? Using
- 11:44a technique like Grid Search CV, you
- 11:46can try with different values. Like, if
- 11:49I set the degree value to two and run
- 11:53it again. You can see now my accuracy
- 11:56score is one. So for a degree value of
- 11:59three, it was coming out to be 45%, but
- 12:01for a degree value of two, it's coming
- 12:03out to be one. Okay? Now if I plot the
- 12:06decision classifier boundary, you can
- 12:11see we have the same boundary. Okay? So
- 12:16, guys, that's it. I am giving you one
- 12:18last revision on what exactly you were
- 12:21doing in that function. What did you do
- 12:23there with that RBF function? You have
- 12:25this x. This is my x, so I have 100
- 12:28coordinates. x and y coordinates. So,
- 12:31this is one point. This is the second
- 12:34point. This is the third point. And
- 12:36what did you do? You just put this
- 12:39entire x inside that function. Okay?
- 12:43What did that do? You calculated e to
- 12:45the power-x² for your x coordinate as
- 12:47well. You calculated e to the power-x²
- 12:49for your y coordinate as well. And
- 12:51after that, what did you do? You summed
- 12:54it up along axis one. Meaning, you
- 12:57added the x and y coordinates. Added
- 12:59the x and y coordinates. And by doing
- 13:01so, you got this 100-length z
- 13:04coordinate. Okay? And then, if you plot
- 13:08this, uh, what did I do? I just stored
- 13:13it in x_new before summing and plotted
- 13:16x_new. So look guys, actually in 2D,
- 13:18this transformation looks like this. It
- 13:20came up in 3D, but in 2D, this
- 13:22transformation is like this. For these
- 13:25red points, both your X and Y values
- 13:26are high. So obviously, when you add
- 13:29them, the value becomes higher and you
- 13:31move up in the Z coordinate, and for
- 13:33all these, either X is high or Y is
- 13:34high. Okay? And that is why these
- 13:36stayed at the bottom. Okay? So I know a
- 13:38lot happened here and it might be a bit
- 13:41difficult. But I am sharing this
- 13:43notebook with you. Play around with it
- 13:44a little. You can try with different
- 13:46functions. You can try this too, and
- 13:48just go through the documentation once.
- 13:50This will also help you understand some
- 13:52things. Otherwise, we are going to
- 13:54discuss these kernels in detail in
- 13:56future videos. Okay? So I think I was
- 13:58able to help. That's it for the video.
- 14:01Thanks for watching.
About this transcript
This page contains the full transcript of Kernel Trick in SVM | Code Example by CampusX, generated from the public captions YouTube serves with the video. The transcript has 2,376 words across 333 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.