K Nearest Neighbors

Hello! In this post, we will be exploring a very simple classification algorithm called 'K Nearest Neighbors'.

How K Nearest Neighbors work:

The green point above is the testing data and the blue and red points are training data in different classes.

The way KNN works is by taking the Euclidean distance from your test data(green point) to each of your training data and classify your test data by the class of the nearest point to the testing data.

Euclidean Distance:

The Euclidean distance is basically the distance between two points on the Euclidean space. To put simply, the distance between two points.

Formula:

K

K: Parameter that takes a certain number of nearest points to use for the classification process.

K basically takes 'k' number of nearest points(training data) and classifies the test data by the class that has the highest vote. For example, if you set K=3 and you take 3 closest training data from your test data and get 2 points in class blue and 1 in class red, then you would classify the test data blue.

Comments

Overfitting vs Underfitting

Overfitting vs Underfitting Hello! In this post, we will be exploring the concepts of Overfitting and Underfitting. Overfitting: Overfitting is a modelling error that occurs when the model fits the training data too well. As you can see above, the overfitted model is fit almost perfectly to the training data. The overall cost of the model to the training data be near 0, however, the accuracy of this model would be poor when used on testing data. Underfitting: Underfitting is when the model fits to the training data too simply; when the model isn't complex enough to adequately understand the trend/pattern of the training data. As you can see from the image above, the model is just a linear line and does not fit to the training data very well; the model does not accurately understand the trend of the data and is fit too simply.

Linear Regression

Linear Regression As you can see from the image above, a line of best fit is drawn on a scatter plot. The line of best fit is essentially the Linear Regression model and the blue data points is the training data. What we need to do in order to implement the linear regression model to the data points/training data is to calculate the line of best fit. Linear Regression Model : output = weight * input + b (where "weight" and "input" are vectors) When looking at how the linear regression model works, you need to have an understanding of what the terms "weight" and "bias" mean. You can think of the "weight" vector and "bias" as terms that optimize the line to be a line of best fit on the training data points (like a gradient the y-intercept in ...

Lasso and Ridge Regression

Ridge and Lasso Regression Hello! Today we will be exploring two different regression algorithms called Ridge Regression and Lasso Regression which are similar to how Linear Regression works. Before we get into how Lasso and Ridge regression models work, you need to understand what is meant by the term 'overfitting'. Overfitting: Overfitting refers to training the model too well by the training data. For example, the indicator that your model is overfitting could be when the total cost of your model by the training data is zero and the cost for your test data is huge. As a result, this could negatively affect your model as it's unable to predict accurately for your testing data. I will be exploring more about the concept of overfitting and underfitting in future posts Ridge Regression: Like I said at the beginning of this post, the Lasso regression model is very similar to how the Linear regression model works. As some of you may already know...

Sam Jang's Blog

Search This Blog