YouTube2Text

Mathematics of Support Vector Machine | Soft Margin SVM — Transcript

by CampusX · 2,591 words · 373 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Hi guys. So we have already done hard
  2. 0:04margin SVM and this was the formulation
  3. 0:06that we arrived upon. Okay? We got this
  4. 0:08formulation that if we want to maximize
  5. 0:11the margin. If you want to find out a
  6. 0:13maximizing margin maximizing
  7. 0:14hyper-plane then we will have to
  8. 0:16maximize this equation. We need to find
  9. 0:19such a W and B for which this value is
  10. 0:21maximum. Right? But there was a
  11. 0:24constraint, right? Right? There was a
  12. 0:26constraint and the constraint was this
  13. 0:29that for every point for every xi, this
  14. 0:32term yi*w transpose xi + b should
  15. 0:34always be greater than or equal to 1.
  16. 0:37This means that if you talk about this
  17. 0:39positive hyper-plane and if you talk
  18. 0:41about these negative hyper-this
  19. 0:43negative hyper-plane. If you talk about
  20. 0:46both of these, all the positive points
  21. 0:48should be on this side of the positive
  22. 0:51hyper-plane and all the negative points
  23. 0:53should be on the lower side of the
  24. 0:55negative hyper-plane. I guess this
  25. 0:57point is clear because if you think
  26. 0:59about this, suppose a point comes here.
  27. 1:02If a green point comes here, then this
  28. 1:03equation will not remain true for it. I
  29. 1:05can prove this; yi would be positive +1
  30. 1:08, but this entire thing because it came
  31. 1:10below this. This entire thing will be
  32. 1:12negative. And as soon as you multiply
  33. 1:15positive and negative, I am sorry, this
  34. 1:16should be one by the way, yeah this
  35. 1:18should be one, or you can see in the
  36. 1:20previous video, this should be one. So
  37. 1:21when you multiply +1 and negative, it
  38. 1:23would obviously be negative and less
  39. 1:25than equal to one. Similarly, if you
  40. 1:28talk about a red point in this area or
  41. 1:30anywhere here, then what will happen
  42. 1:32here? This will be -1 and this term
  43. 1:35will be positive, because it is on the
  44. 1:37upper side, so this term will become
  45. 1:39positive and, uh, yeah I am sorry, this
  46. 1:42term should be negative, right? Okay? I
  47. 1:44got confused. So let's do it again. If
  48. 1:47the point is here, its label will be -1
  49. 1:50only. Right? Because it is negative.
  50. 1:52This term will become positive. Because
  51. 1:54it came to the upper side. So this term
  52. 1:55will become positive. And again, if you
  53. 1:57multiply negative and positive, it
  54. 1:59would be less than one. Okay? So we
  55. 2:01call this type of problem a constrained
  56. 2:04optimization problem. Because here you
  57. 2:06have to do this, given this. So there
  58. 2:09are some constraints. So there are n
  59. 2:11constraints here. There are as many
  60. 2:12constraints as the number of points you
  61. 2:14have. And there is a big problem that
  62. 2:16if you want to do this, you are
  63. 2:18assuming that no green point will fall
  64. 2:20in this region or this region, and no
  65. 2:23red point will fall in this region or
  66. 2:25this region. Now this is not possible.
  67. 2:28Think about it. You will never find a
  68. 2:30data set in real life that is perfectly
  69. 2:33linearly separable. Like this. What you
  70. 2:40will get is a data set which is almost
  71. 2:42linearly separable. So there would be
  72. 2:45certain points which, you know, are
  73. 2:46misclassified, which you can call
  74. 2:48outliers. There will definitely be such
  75. 2:50points in the data. Okay? So what I am
  76. 2:52trying to tell you is that if you
  77. 2:53proceed with this formulation. This is
  78. 2:55not a general formulation and it will
  79. 2:58not work in most real-world scenarios.
  80. 3:01So what you need to do is you need to
  81. 3:03modify this formulation in such a way
  82. 3:05that it gives some outliers space to
  83. 3:07exist in the system. Okay? And that is
  84. 3:11why we are going to learn another
  85. 3:12variation of SVM, which is known as
  86. 3:14Soft Margin SVM. Okay? And in Soft
  87. 3:17Margin SVM, you would see that you will
  88. 3:19be able to handle outliers in your
  89. 3:20system and you will be able to deal
  90. 3:22with them. Okay? So let's see what Soft
  91. 3:25Margin SVM does and how its formulation
  92. 3:27is derived. But before going to Soft
  93. 3:29Margin SVM, I will modify certain
  94. 3:31things here; Soft Margin will be
  95. 3:32derived by making changes to the hard
  96. 3:34margin one itself. But before doing
  97. 3:36that, I am making some changes to the
  98. 3:37hard margin. The change is this. So you
  99. 3:40already know that if you are trying to
  100. 3:43maximize f (x), this is actually
  101. 3:46equivalent to minimizing 1/f (x), just
  102. 3:50think about it. If you are maximizing f
  103. 3:53(x), this is equivalent to minimizing 1
  104. 3:55/f (x). Right? Going by this logic,
  105. 3:58what you can do is you can write this
  106. 4:02entire thing as argmin w*, B*, or
  107. 4:05rather than writing this, you can write
  108. 4:09it like this. Right? What did I do? My
  109. 4:15formulation is still Hard Margin SVM,
  110. 4:17but I changed it to become this. Why
  111. 4:20did I do that? You will understand in a
  112. 4:21little while. This is mathematically
  113. 4:23more convenient and aligns with what we
  114. 4:25are about to do. Okay? So anyways, this
  115. 4:28is our term so far. Now, what am I
  116. 4:29saying? Now, I will add one more term
  117. 4:32here. So that this becomes a Soft
  118. 4:33Margin SVM. I will add this term sigma
  119. 4:37of i = 1 to n, basically for every
  120. 4:41point I am going to find out this term
  121. 4:45ξi. Sorry, I mean this is this is ξ,
  122. 4:48actually this is known as ξ. Okay, and
  123. 4:50I really don't know how to draw this,
  124. 4:52it's very complex. I will try it one
  125. 4:54more time, it turned into a noodle,
  126. 4:58Something like that. Okay, this is
  127. 5:01known as zeta, it's a Greek symbol zeta
  128. 5:03i. So what I'm saying is that now your
  129. 5:05error function, this is your error
  130. 5:07function. Right? You were solving this
  131. 5:09to find the values of W and B. You can
  132. 5:11call it a loss function, error function
  133. 5:13, or cost function. So you have added
  134. 5:16another term to your loss function. ξ
  135. 5:18i = 1 to N, I'm sorry i = 1 to N ξ,
  136. 5:22right? So what are you doing? You are
  137. 5:25calculating this zeta value for every
  138. 5:28point, for all the points in your
  139. 5:31system. And you are also trying to
  140. 5:34minimize that. You are also trying to
  141. 5:36minimize that. So now you have two
  142. 5:38terms in your error function or loss
  143. 5:39function. This is the first term which
  144. 5:41you already know. What are you doing
  145. 5:42with this? You are trying to increase
  146. 5:44the margin. And this is a separate term
  147. 5:47which we will discuss. So I need the W
  148. 5:49and B that minimize both of these. Okay
  149. 5:52? Now let's focus on what is this zeta?
  150. 5:55What is zeta? Okay? So actually it's a
  151. 5:58very simple thing and you will
  152. 5:59immediately understand what zeta is. So
  153. 6:01zeta is basically, first of all, let me
  154. 6:04tell you that if your points are
  155. 6:06correctly classified by these positive
  156. 6:08and negative hyperplanes, which means
  157. 6:10all the green points in this region are
  158. 6:12correctly classified and all the red
  159. 6:15points in this region are also
  160. 6:16correctly classified. For all the
  161. 6:19correctly classified points, your ξi
  162. 6:22would be equal to zero. I will write it
  163. 6:27here, uh, ξi = 0 for correctly
  164. 6:32classified points. Okay? Look, whenever
  165. 6:38you talk about SVM, or talk about
  166. 6:40training an SVM, you always talk in
  167. 6:42terms of positive and negative
  168. 6:43hyperplanes. When new data comes, it
  169. 6:45will be in terms of the hyperplane,
  170. 6:46that if it is on this side of the plane
  171. 6:47, it's positive, if on this side, it's
  172. 6:48negative. But when you are training,
  173. 6:50you view the positive points through
  174. 6:52this plane, and the negative points
  175. 6:54through this plane. All right? So, what
  176. 6:56did I say? That all those positive
  177. 6:59points that are above this, their ξ (
  178. 7:01xi) will all be zero. And all those
  179. 7:05negative points that are below the
  180. 7:08negative hyperplane, their ξi will
  181. 7:10also be zero. All right? Now, let's
  182. 7:13talk about the green points that are in
  183. 7:15this region, or I don't know, here, or
  184. 7:17somewhere around here. Wherever they
  185. 7:20are, as long as they are above it. All
  186. 7:22these green points are error points.
  187. 7:26Just think, my system was unable to
  188. 7:28classify them correctly. These are my
  189. 7:29error points. Right? Similarly, if my
  190. 7:32red points are in this region, or I
  191. 7:35don't know, in this region. Okay? Too
  192. 7:40many points in this region. These are
  193. 7:42all incorrectly classified points. They
  194. 7:45will have a ξ, and what will that ξ
  195. 7:48be? It's very simple. Let me tell you.
  196. 7:50So, for this point, if you go to find
  197. 7:53its ξ, it's basically its distance
  198. 7:56from the positive hyperplane. Similarly
  199. 7:59, to talk about this point, the ξ for
  200. 8:01this point, the ξ for this point is
  201. 8:04this distance. Similarly, for this
  202. 8:06point, it's this distance. The distance
  203. 8:09from the positive hyperplane in the
  204. 8:11wrong direction is its respective ξ.
  205. 8:14Okay? Similarly, if you talk about the
  206. 8:16red points. So, for this red point, the
  207. 8:18distance would be this. Right? For this
  208. 8:21red point, the distance would be this.
  209. 8:24So, what are you doing? You are
  210. 8:25calculating the distance of all your
  211. 8:27incorrectly classified points from
  212. 8:28their corresponding positive or
  213. 8:29negative hyperplanes. And you are
  214. 8:31summing them up. You are summing them
  215. 8:33up. All right? And now you are saying
  216. 8:35that I want to minimize these distances
  217. 8:38. Think about it, guys, what did you
  218. 8:39say? In general, if you were to state
  219. 8:41this, what are you trying to say? You
  220. 8:43are trying to minimize these errors,
  221. 8:45these errors that you have. Right?
  222. 8:48Don't you think that this whole term
  223. 8:50just means that you want to minimize
  224. 8:52these errors? All right? So, we have
  225. 8:54two terms now. One term is trying to
  226. 8:57increase the margin, and one term is
  227. 8:59trying to reduce these errors. All
  228. 9:02right? So, in short, you can say that
  229. 9:05your SVM's loss function, or the SVM's
  230. 9:08error, is basically a combination of
  231. 9:10two things. The first thing is the
  232. 9:14margin error, the margin error, which
  233. 9:18is the first term, this one, this term.
  234. 9:21All right? What are you trying to do
  235. 9:23through this? You want to increase its
  236. 9:25width, right? Margin error, meaning,
  237. 9:27margin error is inversely proportional
  238. 9:28to margin. The larger the margin, the
  239. 9:30lower the margin error will be. Okay?
  240. 9:32And the smaller the margin, the higher
  241. 9:34the margin error will be. Okay? So, the
  242. 9:37first thing is this term, which is
  243. 9:38margin error. And the second term is
  244. 9:40this one. This is known as
  245. 9:42classification error. Classification
  246. 9:45error. Okay? So, what are you doing
  247. 9:48here? You certainly want to reduce the
  248. 9:50classification error. You definitely
  249. 9:52want to minimize these distances. But
  250. 9:54at the same time, you also want to
  251. 9:55increase this distance. We are
  252. 9:57formulating exactly what we thought of
  253. 9:59initially in the first video. Okay? Now
  254. 10:02, what do you do? There is one last
  255. 10:04thing left here. I didn't explain it
  256. 10:06completely. What you do is you multiply
  257. 10:09this thing with a value of C. You
  258. 10:12multiply by something called C. Okay?
  259. 10:16And this C is actually a hyperparameter
  260. 10:18. This C is a hyperparameter.
  261. 10:21Hyperparameter means it's a tuning knob
  262. 10:24for my algorithm. Okay? Listen
  263. 10:27carefully. Now, if you increase the
  264. 10:30value of C a lot. Let's say C can be
  265. 10:33any number, any positive number. Let's
  266. 10:35say you increased the value of C a lot.
  267. 10:36Let's say you made it 10,000. 100,000.
  268. 10:39Then obviously, this term, the
  269. 10:40classification error term, will start
  270. 10:43getting much more importance compared
  271. 10:45to the margin error. So in short, what
  272. 10:48you are saying is, don't worry about
  273. 10:50increasing the line width right now.
  274. 10:53Just worry about ensuring my algorithm
  275. 10:56doesn't misclassify any points. Look at
  276. 10:59these two diagrams, guys. So, this is a
  277. 11:02hyperplane with a very small margin.
  278. 11:04And this is a hyperplane with a very
  279. 11:06large margin. But if you look carefully
  280. 11:08, you would see that although this
  281. 11:10hyperplane has a larger margin. Which
  282. 11:11is good in SVM. It is making some
  283. 11:13mistakes. Look, it is making two
  284. 11:14mistakes here. It is making two
  285. 11:16mistakes here. And this hyperplane,
  286. 11:18although its margin is a bit smaller,
  287. 11:20which is not preferred in SVM, it isn't
  288. 11:22making any mistakes. Okay? So, if
  289. 11:24someone asks you which of these two is
  290. 11:26better? Actually, there is no right
  291. 11:27answer. But by using this, you can
  292. 11:29justify it. As soon as you increase the
  293. 11:32value of C significantly. Right? If you
  294. 11:34increase it a lot, then you are saying
  295. 11:36don't focus on the margin error.
  296. 11:38Meaning, don't focus on maximizing the
  297. 11:40margin. What should you focus on? On
  298. 11:42reducing these distances. That is,
  299. 11:44focus on reducing the number of these
  300. 11:46misclassified points. So, in a way, I
  301. 11:49am choosing this model. Where the
  302. 11:50margin is small but there is no
  303. 11:52misclassification. No errors at all.
  304. 11:54Okay? And if you make C very small,
  305. 11:56like .1, .01, .001. Then what are you
  306. 12:00saying? To ignore classification error.
  307. 12:03Focus more on margin error. Meaning,
  308. 12:05even if you make some
  309. 12:06misclassifications, I'm okay with it.
  310. 12:09But you try to maximize this margin.
  311. 12:12Okay? And in that manner, this becomes
  312. 12:15a hyperparameter. Based on your data,
  313. 12:17what you do is you run the algorithm.
  314. 12:21You do cross-validation, sorry, grid
  315. 12:23search CV, you do cross-validation and
  316. 12:25you find the optimized value of C for
  317. 12:27which you understand how much value I
  318. 12:29should give to increasing the margin
  319. 12:31and how much to reducing errors. And
  320. 12:34it's a mix of both of them. Okay? I
  321. 12:36guess you guys are understanding this
  322. 12:37whole thing. Right? In fact, what you
  323. 12:40can do is you can directly compare it
  324. 12:41with the logistic regression loss
  325. 12:43function. If you remember, the logistic
  326. 12:48regression loss function was logistic
  327. 12:51loss plus lambda times this, this was
  328. 12:55kind of your L2. Okay? This was L2
  329. 13:00regularization. And this lambda was
  330. 13:02basically a hyperparameter. Right? Now
  331. 13:05you can directly compare it with this.
  332. 13:08And in this case, what you can see is,
  333. 13:10or what you can say is, that this term
  334. 13:12can be considered as your
  335. 13:13regularization term. This term can be
  336. 13:16considered as your regularization term,
  337. 13:18similar to that. Right? And just as
  338. 13:20there is logistic loss here, here this
  339. 13:23loss, where you are minimizing these
  340. 13:25distances, is called hinge loss. We
  341. 13:27will discuss this in a video. This is
  342. 13:29called hinge loss. Okay? So here, your
  343. 13:32SVM error or loss function is a
  344. 13:34combination of two things: your hinge
  345. 13:36loss plus your regularization term.
  346. 13:38Okay? And the C here is actually
  347. 13:41inversely proportional to lambda. Okay?
  348. 13:44And in fact, if you have coded in
  349. 13:46logistic regression, there isn't a
  350. 13:47lambda there. There, it is just C. And
  351. 13:49you are told that if you increase C,
  352. 13:51lambda decreases, and if you decrease C
  353. 13:53, lambda increases. So that C is
  354. 13:55actually coming through the SVM itself.
  355. 13:56In an attempt to create a general sort
  356. 13:58of algorithm, they included C instead
  357. 13:59of lambda. Okay? So guys, I think you
  358. 14:02understood this entire discussion. I am
  359. 14:06just thinking if anything was left out
  360. 14:08or not. Yeah, so guys, that was it for
  361. 14:11this video. Since we consider some
  362. 14:14errors here. That is why we call it
  363. 14:17soft margin SVM because we are
  364. 14:18softening our criteria. We are not hard
  365. 14:20. We are not saying that you cannot
  366. 14:22make even a single mistake. We are
  367. 14:23saying that you can make mistakes. So
  368. 14:25because of this formulation, we are now
  369. 14:27inducing some error, or rather we are
  370. 14:29allowing some errors, and that is why
  371. 14:31this is known as soft margin SVM. Okay?
  372. 14:34So I hope this entire video was clear.
  373. 14:36Thanks for watching.

About this transcript

This page contains the full transcript of Mathematics of Support Vector Machine | Soft Margin SVM by CampusX, generated from the public captions YouTube serves with the video. The transcript has 2,591 words across 373 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.