Skip to main content

Support Vector Machine

Support Vector Machine

How the algorithm works:








As you can see from the image above, SVM works by dividing different classes with a hyperplane. For the explanation of how this algorithm works, I will be using a 2D graph to simplify how this essentially works.



← 2D case


SVM works by finding the optimal weights and bias that separates two different classes. The linear line (when looking at 2D graph) has the equation: w*x - b =0. When w*x - b ≥ 0 it would predict class 1 and if w*x -b < 0 it would predict class 2.

What the SVM model tries to do is to maximize the margin between the support vectors(blue and green points on w*x - b =1 and w*x -b = -1; points that are on the boundary).

 Hyperplane Definition
  • w*x - b ≥ +1 (yi = +1) {where +1 = class 1}
  • w*x - b ≤ -1 (yi = -1) {where -1 = class 2}
The two could be combined to form: yi ( w*x - b ) ≥ 1

Finding the Separation of the Margin:
We know that w*x - b = +1 and w*x - b = -1  are the lines going through the support vectors. Let's say that x1 is the support vector for class +1 and x2 the support vector for class -1. Then the two lines would become w*x1 - b = +1 and w* x2- b = -1.

If we considered this a simulation equation problem, then we would get w*(x1 - x2 ) = 2 and if we divide both sides by the magnitude of w to get the total distance between the support vectors we get
(w*(x1 - x2 ))/⟪w⟫ = 2/⟪w⟫ 

 Margin:  2/⟪w⟫

As we are trying to maximize the margin, we would need to try and maximize:  2/⟪w⟫

 basically minimizing ⟪w⟫, which we can rewrite as 0.5⟪w⟫^2

To find the optimal value of w:

 (w, b): min(0.5⟪w⟫^2) + Ci x Summation of Z

where C = number of errors to be considered
where Summation of Z: total error

C is basically a regularization parameter that will allow the model to accept a certain number of outliers. As you can see from the image below, you can see 2 points on the wrong side of the hyperplane. However, the model will not try to tweak itself to perfectly divide the points depending on the value of C. This is imperative as it can prevent your model from overfitting. 

For Z you can consider it as the distance between the lines created by the support vectors (w*x-b=1 and w*x-b=-1). As you can see from the image below the magnitude of  vector Ƹ's would be the value of Z.   






Comments

Post a Comment

Popular posts from this blog

Overfitting vs Underfitting

Overfitting vs Underfitting Hello! In this post, we will be exploring the concepts of Overfitting and Underfitting. Overfitting: Overfitting is a modelling error that occurs when the model fits the training data too well.  As you can see above, the overfitted model is fit almost perfectly to the training data. The overall cost of the model to the training data be near 0, however, the accuracy of this model would be poor when used on testing data. Underfitting: Underfitting is when the model fits to the training data too simply; when the model isn't complex enough to adequately understand the trend/pattern of the training data.   As you can see from the image above, the model is just a linear line and does not fit to the training data very well; the model does not accurately understand the trend of the data and is fit too simply.
Post a Comment
Read more

Classification vs Regression

Classification vs Regression Classification: Classification algorithms attempt to predict a discrete class label. For example, a problem where you are trying to build a model(or algorithm) to interpret whether an image is a dog or a cat would be a classification problem.    Some Common Classification Machine Learning Algorithms/models : - K Nearest Neighbors - Support Vector Machine - Logistic Regression - Random Forest Classifier - Decision Trees - Neural Networks Regression: On the other hand, regression algorithms attempt to predict a continuous quantity. An example of a regression problem would be when you're trying to predict the prices of homes, where the prediction could be an integer, fraction, or decimal number.  Some Common Regression Machine Learning Algorithms/models: - Linear Regressor - Polynomial Regressor - Lasso Regressor - Ridge Regressor - Elastic Net Regressor - Support Vector Regressor - Regression Trees - Neural Networks ...
Post a Comment
Read more

K Nearest Neighbors

K Nearest Neighbors Hello! In this post, we will be exploring a very simple classification algorithm called 'K Nearest Neighbors'. How K Nearest Neighbors work: The green point above is the testing data and the blue and red points are training data in different classes. The way KNN works is by taking the Euclidean distance from your test data(green point) to each of your training data and classify your test data by the class of the nearest point to the testing data. Euclidean Distance: The Euclidean distance is basically the distance between two points on the Euclidean space. To put simply, the distance between two points.  Formula : K K: Parameter that takes a certain number of nearest points to use for the classification process.  K basically takes 'k' number of nearest points(training data) and classifies the test data by the class that has the highest vote. For example, if ...
Post a Comment
Read more