Perceptron

1
 Rosenblatt (1958) proposed the perceptron as the first
model for learning with a teacher

Perceptron – simplest form of neural network

• Can classify patterns that are linearly separable

• Consists of single neuron

• Converges for linearly separable 2 classes,
decision surface is a hyperplane.
• For multiple classes, multiple neurons are
required, but Perceptron (single layer) will
perform well only with linearly separable classes.

2
 Perceptron

[+1 -1]

Induced local field

Hard Limiter

[+1 -1] →

3
 m-dimensional input (signal / feature) space

Hyperplane:

4
 Perceptron Convergence Theorem

Error correction learning algorithm

5
 (particular n)
So the two classes must be linearly separable
(boundary A linear equation)

6
 Subspace of training vectors from class

Subspace of training vectors from class

Training involves adjustment of weight vectors such that the

classes linearly separable (best we can do with a perceptron)

Finding W such that

Boundary

7
 The Algo:
Rule 1:

Rule 2:

Positive

8
Updation for Type 1 error:

Updation for Type 2 error:

Considering the learning rate as fixed, it becomes unimportant if

its >0 (assume 1)

Initial condition: Till current

Suppose ∀ 𝑛𝑛 ∈
That is incorrect classification (of second type)

Iteratively

9
Recall our assumption
 Linearly separable 

A solution exists:

∀ 𝑛𝑛 ∈

Now let

Now from

10
 By

Euclidean!

RESULT 1

11
 ≤0

Adding across k

RESULT 2
12
 Cannot go beyond For the
error to
happen

real
So, the adaption (error correction) process terminates
(converges) at some value of n when there exists a solution
Assumed WLOG

Also true for the first type of incorrect classification

• Sufficient samples?!
13 • Finds boundary only for training samples!
 The theorem:

Smallest such that

Let
and we have

14
 So if sign of is wrong

So if sign of will be right

So
Will yield correct classification

If η(n) is not appropriate (very small)

Repeated presentation of

will yield correct classification

So the rate can be any positive value!

Convergence time might change, but guaranteed convergence
15
 If not satisfied

Only a positive or a negative value will be added to the sum,

changing the convergence time. But convergence is guaranteed.

Now if we write

Output:

Desired output:

like LMS
Error correction learning:
Stable weight estimate (smooth,
Usually: small step)
Fast adaptation (large step)
16
 Relation with Bayes Classifier
*
⊇

Risk

Costs: C11, C22, C21 and C12

17 Negative integrand lowers risk

 Decision to prefer class 1 over class 2:

Likelihood ratio Threshold

Gaussianity

For two classes: Means differ, but same covariance

18
Classes equally probable and correct classification yields zero error

Both wrong classifications have equal cost!!

So…

19
 Perceptron and Gauss Bayes CL,
similar yet different

• Perceptron assumes linear separability, Gauss Bayes CL

does not
• Gauss Bayes CL assumes distribution type, perceptron
does not
• Perceptron is non parametric, Gauss Bayes is
parametric
20
 Batch Perceptron Algorithm

Perceptron cost function: Batch

Set of misclassified samples

21