Mathematics of Support Vector Machine | Soft Margin SVM — Transcript
Full transcript
- 0:00Hi guys. So we have already done hard
- 0:04margin SVM and this was the formulation
- 0:06that we arrived upon. Okay? We got this
- 0:08formulation that if we want to maximize
- 0:11the margin. If you want to find out a
- 0:13maximizing margin maximizing
- 0:14hyper-plane then we will have to
- 0:16maximize this equation. We need to find
- 0:19such a W and B for which this value is
- 0:21maximum. Right? But there was a
- 0:24constraint, right? Right? There was a
- 0:26constraint and the constraint was this
- 0:29that for every point for every xi, this
- 0:32term yi*w transpose xi + b should
- 0:34always be greater than or equal to 1.
- 0:37This means that if you talk about this
- 0:39positive hyper-plane and if you talk
- 0:41about these negative hyper-this
- 0:43negative hyper-plane. If you talk about
- 0:46both of these, all the positive points
- 0:48should be on this side of the positive
- 0:51hyper-plane and all the negative points
- 0:53should be on the lower side of the
- 0:55negative hyper-plane. I guess this
- 0:57point is clear because if you think
- 0:59about this, suppose a point comes here.
- 1:02If a green point comes here, then this
- 1:03equation will not remain true for it. I
- 1:05can prove this; yi would be positive +1
- 1:08, but this entire thing because it came
- 1:10below this. This entire thing will be
- 1:12negative. And as soon as you multiply
- 1:15positive and negative, I am sorry, this
- 1:16should be one by the way, yeah this
- 1:18should be one, or you can see in the
- 1:20previous video, this should be one. So
- 1:21when you multiply +1 and negative, it
- 1:23would obviously be negative and less
- 1:25than equal to one. Similarly, if you
- 1:28talk about a red point in this area or
- 1:30anywhere here, then what will happen
- 1:32here? This will be -1 and this term
- 1:35will be positive, because it is on the
- 1:37upper side, so this term will become
- 1:39positive and, uh, yeah I am sorry, this
- 1:42term should be negative, right? Okay? I
- 1:44got confused. So let's do it again. If
- 1:47the point is here, its label will be -1
- 1:50only. Right? Because it is negative.
- 1:52This term will become positive. Because
- 1:54it came to the upper side. So this term
- 1:55will become positive. And again, if you
- 1:57multiply negative and positive, it
- 1:59would be less than one. Okay? So we
- 2:01call this type of problem a constrained
- 2:04optimization problem. Because here you
- 2:06have to do this, given this. So there
- 2:09are some constraints. So there are n
- 2:11constraints here. There are as many
- 2:12constraints as the number of points you
- 2:14have. And there is a big problem that
- 2:16if you want to do this, you are
- 2:18assuming that no green point will fall
- 2:20in this region or this region, and no
- 2:23red point will fall in this region or
- 2:25this region. Now this is not possible.
- 2:28Think about it. You will never find a
- 2:30data set in real life that is perfectly
- 2:33linearly separable. Like this. What you
- 2:40will get is a data set which is almost
- 2:42linearly separable. So there would be
- 2:45certain points which, you know, are
- 2:46misclassified, which you can call
- 2:48outliers. There will definitely be such
- 2:50points in the data. Okay? So what I am
- 2:52trying to tell you is that if you
- 2:53proceed with this formulation. This is
- 2:55not a general formulation and it will
- 2:58not work in most real-world scenarios.
- 3:01So what you need to do is you need to
- 3:03modify this formulation in such a way
- 3:05that it gives some outliers space to
- 3:07exist in the system. Okay? And that is
- 3:11why we are going to learn another
- 3:12variation of SVM, which is known as
- 3:14Soft Margin SVM. Okay? And in Soft
- 3:17Margin SVM, you would see that you will
- 3:19be able to handle outliers in your
- 3:20system and you will be able to deal
- 3:22with them. Okay? So let's see what Soft
- 3:25Margin SVM does and how its formulation
- 3:27is derived. But before going to Soft
- 3:29Margin SVM, I will modify certain
- 3:31things here; Soft Margin will be
- 3:32derived by making changes to the hard
- 3:34margin one itself. But before doing
- 3:36that, I am making some changes to the
- 3:37hard margin. The change is this. So you
- 3:40already know that if you are trying to
- 3:43maximize f (x), this is actually
- 3:46equivalent to minimizing 1/f (x), just
- 3:50think about it. If you are maximizing f
- 3:53(x), this is equivalent to minimizing 1
- 3:55/f (x). Right? Going by this logic,
- 3:58what you can do is you can write this
- 4:02entire thing as argmin w*, B*, or
- 4:05rather than writing this, you can write
- 4:09it like this. Right? What did I do? My
- 4:15formulation is still Hard Margin SVM,
- 4:17but I changed it to become this. Why
- 4:20did I do that? You will understand in a
- 4:21little while. This is mathematically
- 4:23more convenient and aligns with what we
- 4:25are about to do. Okay? So anyways, this
- 4:28is our term so far. Now, what am I
- 4:29saying? Now, I will add one more term
- 4:32here. So that this becomes a Soft
- 4:33Margin SVM. I will add this term sigma
- 4:37of i = 1 to n, basically for every
- 4:41point I am going to find out this term
- 4:45ξi. Sorry, I mean this is this is ξ,
- 4:48actually this is known as ξ. Okay, and
- 4:50I really don't know how to draw this,
- 4:52it's very complex. I will try it one
- 4:54more time, it turned into a noodle,
- 4:58Something like that. Okay, this is
- 5:01known as zeta, it's a Greek symbol zeta
- 5:03i. So what I'm saying is that now your
- 5:05error function, this is your error
- 5:07function. Right? You were solving this
- 5:09to find the values of W and B. You can
- 5:11call it a loss function, error function
- 5:13, or cost function. So you have added
- 5:16another term to your loss function. ξ
- 5:18i = 1 to N, I'm sorry i = 1 to N ξ,
- 5:22right? So what are you doing? You are
- 5:25calculating this zeta value for every
- 5:28point, for all the points in your
- 5:31system. And you are also trying to
- 5:34minimize that. You are also trying to
- 5:36minimize that. So now you have two
- 5:38terms in your error function or loss
- 5:39function. This is the first term which
- 5:41you already know. What are you doing
- 5:42with this? You are trying to increase
- 5:44the margin. And this is a separate term
- 5:47which we will discuss. So I need the W
- 5:49and B that minimize both of these. Okay
- 5:52? Now let's focus on what is this zeta?
- 5:55What is zeta? Okay? So actually it's a
- 5:58very simple thing and you will
- 5:59immediately understand what zeta is. So
- 6:01zeta is basically, first of all, let me
- 6:04tell you that if your points are
- 6:06correctly classified by these positive
- 6:08and negative hyperplanes, which means
- 6:10all the green points in this region are
- 6:12correctly classified and all the red
- 6:15points in this region are also
- 6:16correctly classified. For all the
- 6:19correctly classified points, your ξi
- 6:22would be equal to zero. I will write it
- 6:27here, uh, ξi = 0 for correctly
- 6:32classified points. Okay? Look, whenever
- 6:38you talk about SVM, or talk about
- 6:40training an SVM, you always talk in
- 6:42terms of positive and negative
- 6:43hyperplanes. When new data comes, it
- 6:45will be in terms of the hyperplane,
- 6:46that if it is on this side of the plane
- 6:47, it's positive, if on this side, it's
- 6:48negative. But when you are training,
- 6:50you view the positive points through
- 6:52this plane, and the negative points
- 6:54through this plane. All right? So, what
- 6:56did I say? That all those positive
- 6:59points that are above this, their ξ (
- 7:01xi) will all be zero. And all those
- 7:05negative points that are below the
- 7:08negative hyperplane, their ξi will
- 7:10also be zero. All right? Now, let's
- 7:13talk about the green points that are in
- 7:15this region, or I don't know, here, or
- 7:17somewhere around here. Wherever they
- 7:20are, as long as they are above it. All
- 7:22these green points are error points.
- 7:26Just think, my system was unable to
- 7:28classify them correctly. These are my
- 7:29error points. Right? Similarly, if my
- 7:32red points are in this region, or I
- 7:35don't know, in this region. Okay? Too
- 7:40many points in this region. These are
- 7:42all incorrectly classified points. They
- 7:45will have a ξ, and what will that ξ
- 7:48be? It's very simple. Let me tell you.
- 7:50So, for this point, if you go to find
- 7:53its ξ, it's basically its distance
- 7:56from the positive hyperplane. Similarly
- 7:59, to talk about this point, the ξ for
- 8:01this point, the ξ for this point is
- 8:04this distance. Similarly, for this
- 8:06point, it's this distance. The distance
- 8:09from the positive hyperplane in the
- 8:11wrong direction is its respective ξ.
- 8:14Okay? Similarly, if you talk about the
- 8:16red points. So, for this red point, the
- 8:18distance would be this. Right? For this
- 8:21red point, the distance would be this.
- 8:24So, what are you doing? You are
- 8:25calculating the distance of all your
- 8:27incorrectly classified points from
- 8:28their corresponding positive or
- 8:29negative hyperplanes. And you are
- 8:31summing them up. You are summing them
- 8:33up. All right? And now you are saying
- 8:35that I want to minimize these distances
- 8:38. Think about it, guys, what did you
- 8:39say? In general, if you were to state
- 8:41this, what are you trying to say? You
- 8:43are trying to minimize these errors,
- 8:45these errors that you have. Right?
- 8:48Don't you think that this whole term
- 8:50just means that you want to minimize
- 8:52these errors? All right? So, we have
- 8:54two terms now. One term is trying to
- 8:57increase the margin, and one term is
- 8:59trying to reduce these errors. All
- 9:02right? So, in short, you can say that
- 9:05your SVM's loss function, or the SVM's
- 9:08error, is basically a combination of
- 9:10two things. The first thing is the
- 9:14margin error, the margin error, which
- 9:18is the first term, this one, this term.
- 9:21All right? What are you trying to do
- 9:23through this? You want to increase its
- 9:25width, right? Margin error, meaning,
- 9:27margin error is inversely proportional
- 9:28to margin. The larger the margin, the
- 9:30lower the margin error will be. Okay?
- 9:32And the smaller the margin, the higher
- 9:34the margin error will be. Okay? So, the
- 9:37first thing is this term, which is
- 9:38margin error. And the second term is
- 9:40this one. This is known as
- 9:42classification error. Classification
- 9:45error. Okay? So, what are you doing
- 9:48here? You certainly want to reduce the
- 9:50classification error. You definitely
- 9:52want to minimize these distances. But
- 9:54at the same time, you also want to
- 9:55increase this distance. We are
- 9:57formulating exactly what we thought of
- 9:59initially in the first video. Okay? Now
- 10:02, what do you do? There is one last
- 10:04thing left here. I didn't explain it
- 10:06completely. What you do is you multiply
- 10:09this thing with a value of C. You
- 10:12multiply by something called C. Okay?
- 10:16And this C is actually a hyperparameter
- 10:18. This C is a hyperparameter.
- 10:21Hyperparameter means it's a tuning knob
- 10:24for my algorithm. Okay? Listen
- 10:27carefully. Now, if you increase the
- 10:30value of C a lot. Let's say C can be
- 10:33any number, any positive number. Let's
- 10:35say you increased the value of C a lot.
- 10:36Let's say you made it 10,000. 100,000.
- 10:39Then obviously, this term, the
- 10:40classification error term, will start
- 10:43getting much more importance compared
- 10:45to the margin error. So in short, what
- 10:48you are saying is, don't worry about
- 10:50increasing the line width right now.
- 10:53Just worry about ensuring my algorithm
- 10:56doesn't misclassify any points. Look at
- 10:59these two diagrams, guys. So, this is a
- 11:02hyperplane with a very small margin.
- 11:04And this is a hyperplane with a very
- 11:06large margin. But if you look carefully
- 11:08, you would see that although this
- 11:10hyperplane has a larger margin. Which
- 11:11is good in SVM. It is making some
- 11:13mistakes. Look, it is making two
- 11:14mistakes here. It is making two
- 11:16mistakes here. And this hyperplane,
- 11:18although its margin is a bit smaller,
- 11:20which is not preferred in SVM, it isn't
- 11:22making any mistakes. Okay? So, if
- 11:24someone asks you which of these two is
- 11:26better? Actually, there is no right
- 11:27answer. But by using this, you can
- 11:29justify it. As soon as you increase the
- 11:32value of C significantly. Right? If you
- 11:34increase it a lot, then you are saying
- 11:36don't focus on the margin error.
- 11:38Meaning, don't focus on maximizing the
- 11:40margin. What should you focus on? On
- 11:42reducing these distances. That is,
- 11:44focus on reducing the number of these
- 11:46misclassified points. So, in a way, I
- 11:49am choosing this model. Where the
- 11:50margin is small but there is no
- 11:52misclassification. No errors at all.
- 11:54Okay? And if you make C very small,
- 11:56like .1, .01, .001. Then what are you
- 12:00saying? To ignore classification error.
- 12:03Focus more on margin error. Meaning,
- 12:05even if you make some
- 12:06misclassifications, I'm okay with it.
- 12:09But you try to maximize this margin.
- 12:12Okay? And in that manner, this becomes
- 12:15a hyperparameter. Based on your data,
- 12:17what you do is you run the algorithm.
- 12:21You do cross-validation, sorry, grid
- 12:23search CV, you do cross-validation and
- 12:25you find the optimized value of C for
- 12:27which you understand how much value I
- 12:29should give to increasing the margin
- 12:31and how much to reducing errors. And
- 12:34it's a mix of both of them. Okay? I
- 12:36guess you guys are understanding this
- 12:37whole thing. Right? In fact, what you
- 12:40can do is you can directly compare it
- 12:41with the logistic regression loss
- 12:43function. If you remember, the logistic
- 12:48regression loss function was logistic
- 12:51loss plus lambda times this, this was
- 12:55kind of your L2. Okay? This was L2
- 13:00regularization. And this lambda was
- 13:02basically a hyperparameter. Right? Now
- 13:05you can directly compare it with this.
- 13:08And in this case, what you can see is,
- 13:10or what you can say is, that this term
- 13:12can be considered as your
- 13:13regularization term. This term can be
- 13:16considered as your regularization term,
- 13:18similar to that. Right? And just as
- 13:20there is logistic loss here, here this
- 13:23loss, where you are minimizing these
- 13:25distances, is called hinge loss. We
- 13:27will discuss this in a video. This is
- 13:29called hinge loss. Okay? So here, your
- 13:32SVM error or loss function is a
- 13:34combination of two things: your hinge
- 13:36loss plus your regularization term.
- 13:38Okay? And the C here is actually
- 13:41inversely proportional to lambda. Okay?
- 13:44And in fact, if you have coded in
- 13:46logistic regression, there isn't a
- 13:47lambda there. There, it is just C. And
- 13:49you are told that if you increase C,
- 13:51lambda decreases, and if you decrease C
- 13:53, lambda increases. So that C is
- 13:55actually coming through the SVM itself.
- 13:56In an attempt to create a general sort
- 13:58of algorithm, they included C instead
- 13:59of lambda. Okay? So guys, I think you
- 14:02understood this entire discussion. I am
- 14:06just thinking if anything was left out
- 14:08or not. Yeah, so guys, that was it for
- 14:11this video. Since we consider some
- 14:14errors here. That is why we call it
- 14:17soft margin SVM because we are
- 14:18softening our criteria. We are not hard
- 14:20. We are not saying that you cannot
- 14:22make even a single mistake. We are
- 14:23saying that you can make mistakes. So
- 14:25because of this formulation, we are now
- 14:27inducing some error, or rather we are
- 14:29allowing some errors, and that is why
- 14:31this is known as soft margin SVM. Okay?
- 14:34So I hope this entire video was clear.
- 14:36Thanks for watching.
About this transcript
This page contains the full transcript of Mathematics of Support Vector Machine | Soft Margin SVM by CampusX, generated from the public captions YouTube serves with the video. The transcript has 2,591 words across 373 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.