Course : Artificial Intelligence

Effective Period : September 2018

K-Nearest Neighbor

Session 18

1
 Learning Outcomes
At the end of this session, students will be able to:

 LO 5: Apply various learning algorithms to solve the problems

 LO 6: Apply AI algorithms on various applications such as

Game AI, Natural Language Processing, and Computer Vision

2
 Outline
1. Non-Parametric Model

2. K-Nearest Neighbor

3. Distance Metric

3
 Non-Parametric Model
• Parametric model

– A learning model that summarizes data with a set of

parameters of fixed size

– Linear regression & artificial neural network

• Non-parametric model

– One that cannot be characterized by a bounded set of

parameters

– K-nearest neighbors

4
 Non-Parametric Model
• For example, suppose that each hypothesis we generate
simply retains within itself all of the training examples and uses
all of them to predict the next example

– It would be non-parametric solution (instance based

learning or memory based learning)

• The simplest instance based learning is

– Table lookup

5
 Table Lookup
• Take all the training examples, put them in a lookup table,
and then when asked for h(x), see if x is in the table

– If it is, return the corresponding y

• The problem with this method is that it does not generalize

well: when x is not in the table all it can do is return some
default value

6
 K-nearest Neighbors
• We can improve on table lookup with a slight variation: given a
query xq, find the k examples that are nearest to xq.

– This is called k-nearest neighbors lookup (NN(k,xq))

7
 K-nearest Neighbors
• To do classification, first find NN(k, xq), then take the plurality
vote of the neighbors (which is the majority vote in the case of
binary classification)

– To avoid ties, k is always chosen to be an odd number

• To do regression, we can take the mean or median of the k

neighbors, or we can solve a linear regression problem on the
neighbors.

8
 K-nearest Neighbors
• We can improve on table lookup with a slight variation: given a
query xq, find the k examples that are nearest to xq.

– This is called k-nearest neighbors lookup (NN(k,xq))

• To do classification, first find NN(k, xq), then take the plurality

vote of the neighbors (which is the majority vote in the case of
binary classification)

– To avoid ties, k is always chosen to be an odd number

9
 Decision Boundary of k=1

Might be overfitting
10
Decision Boundary of k=5

11
 Distance Metric
• The very word “nearest” implies a distance metric

• Typically, distances are measured with a Minkowski distance

or Lp norm

• P = 2  Euclidean distance

• P = 1  Manhattan distance

12
 Distance Metric
• Euclidean distance is used if the dimensions are measuring
similar properties, such as the width, height and depth of
parts on a conveyor belt

• Manhattan distance is used if they are dissimilar, such as

age, weight, and gender of a patient.

13
 Distance Metric
• Hamming distance?

• Mahalanobis distance?

14
 Distance Metric
• If we use the raw numbers from each dimension then the total
distance will be affected by a change in scale in any dimension

– To avoid this, it is common to apply normalization to the

measurements in each dimension

– Simple approach is to compute mean and standard

deviation

15
 Nearest Neighbor Classifier
• Assign label of nearest training data point to each test data
point

from Duda et al.

Voronoi partitioning of feature space

Source: D. Lowe for two-category 2D and 3D data 16
K-nearest neighbor

x
x
x o
x x
x
+ o
o x
x
o o+
o
o
x2

x1

17
1-nearest neighbor
x
x
x o
x x
x
+ o
o x
x
o o+
o
o
x2

x1
3-nearest neighbor
x
x
x o
x x
x
+ o
o x
x
o o+
o
o
x2

x1
5-nearest neighbor
x
x
x o
x x
x
+ o
o x
x
o o+
o
o
x2

x1
 Using K-NN

• Simple, a good one to try first

• With infinite examples, 1-NN provably has error that is at most

twice Bayes optimal error
 References
• Stuart Russell, Peter Norvig. 2010. Artificial Intelligence : A
Modern Approach. Pearson Education. New Jersey.
ISBN:9780132071482

• https://www.cc.gatech.edu/~hays/compvision/lectures/17.pdf

22