Documente Academic
Documente Profesional
Documente Cultură
Neural Networks
Myelin
Dendrites
sheaths
Axon
6.1.1 Perceptrons
A perceptron is the simplest kind of ANNs, invented in
1957 by Frank Rosenblatt. A perceptron is a neural network
Neural Networks ◾ 91
x1
w1
x2 w2
Processor y
w m−1
xm−1
wm
xm
∑w
j =1
j ⋅ x j = w1 ⋅ x 1 + + w m ⋅ x m
1 if x > 0
f ( x ) =
0 if x < 0
1 if x > 0
f ( x ) =
−1 if x < 0
92 ◾ Machine Learning
2.0 2.0
1.5 1.5
1.0
1.0 0.5
0.5 0.0
y
y
0.0 −0.5
−1.0
−0.5 −1.5
−1.0 −2.0
−1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1
(a) x (b) x
1.0 2.0
1.5
0.5
1.0
0.0 0.5
y
y
0.0
−0.5
−0.5
−1.0 −1.0
−1 −0.5 0 0.5 1 −5 0 5
(c) x (d) x
Figure 6.3 Activation functions: (a) step activation function, (b) sign
activation function, (c) linear activation function, and (d) sigmoid
activation function.
1
f (x ) =
1 + e−x
b
x1
w1
x2 w2
w m−1
y=f ( m
)
b + Σ wj . xj
j=1
y
xm−1
wm
xm
w1 a j1
w = , xj =
w m a jm
b 1
and generate random values for the (m + 1)-dimensional
vector w. We notice that
94 ◾ Machine Learning
w T ⋅ x j = w1a j 1 + + w m a jm + b
E j = y j − sign ( w T ⋅ x j )
The error E j could be either 2,0 or −2. Both the first and third
values of the error (nonzero values) indicate the occurrence of
an error in classifying the feature vector x j . Knowing the error
helps us in the adjustment of weights.
To readjust the weight, we define a learning rate parameter
α ∈ ( 0,1). This parameter determines how fast the weights are
changed, and hence, how fast the perceptron learns during the
training phase. Given the learning rate α, the correction in the
weight vector is given by
∆w = αx j E j
wnew = wold + ∆w
1.5
1.0
0.5
0.0
−0.5
−0.5 0 0.5 1 1.5
Figure 6.5 The XOR binary operation: True instances are plotted in
circles and false instances are plotted with squares.
98 ◾ Machine Learning
Hidden layers
the weights between the hidden layer and the input layer are
adjusted in a backward fashion. Another training instance is
passed to the neural network and to the process of evaluating
the error at the output layer, thereby correcting the weights
between the different layers from the output layer to the input
layer. Repeating this process for as many epochs will help in
learning the neural network.
1 (1 + e − x )
1. S1 ( x ) = /
2. S 2 ( x ) = (1 − e − x )/(1 + e − x )
dS1 ( x )
= S1 ( x ) (1 − S1 ( x ) )
dx
and
dS 2 ( x )
= 1 − S2 ( x )
dx
The second sigmoid function S 2 ( x ) is the hyperbolic tangent
function. Choosing sigmoid functions guarantees the continuity
and differentiability of the error function. The graphs of the two
sigmoid functions S1 ( x ) and S 2 ( x ) are explained in Figure 6.7.
S1 =
1 1 − e−x
S2 =
1 + e−x 1 + e−x
1.5 1.5
1.0
1.0
0.5
y 0.5 y 0.0
−0.5
0.0
−1.0
−0.5 −1.5
−4 −3 −2 −1 0 1 2 3 4 −4 −3 −2 −1 0 1 2 3 4
x x
∑
1 2
E= yj − p j
2 j =1
∂E
∂w1
∇E =
∂E
∂w L
In the gradient decent method, the update in the weights
vector is proportional to negative of the gradient. That is,
∂E
∆w j = −α , j = 1, …, L
∂w j
∂E ∂E ∂oi
= ⋅ = − ( u i − oi ) oi ( 1 − oi ) z j
∂wij ∂oi ∂wij
∆wij = α ( u i − oi ) oi (1 − oi ) z j
clear; clc;
A = load(‘EColi1.txt’); % Loading the ecoli data,
with the classes at the
% last column
%B = A(1:end, 2:end);
C = A(1:end, 1:end-1)’; % C is the matrix of the
feature vectors
T = A(:, end)’; % T is the vector of classes
net = feedforwardnet; % Initializing a neural
network ‘net’
net = configure(net, C, T);
hiddenLayerSize = 10; % Setting the number of
hidden layers to 10
Neural Networks ◾ 103
Figure 6.8 The output neural network model for classifying the
ecoli data.
104 ◾ Machine Learning
100
10−1
0 2 4 6 8 10 12 14 16
17 Epochs
120
100
Instances
80
60
40
20
0
−4.779
−4.246
−3.714
−3.181
−2.649
−2.116
−1.584
−1.051
−0.5186
0.01395
0.5465
1.079
1.612
2.144
2.677
3.209
3.742
4.274
4.807
5.339
References
1. Vapnik, V. N. The Nature of Statistical Learning Theory, 2nd Ed.,
New York: Springer-Verlag, 1999.
2. Negnevitsky, M. Artificial Intelligence: A Guide to Intelligent
Systems, 2nd Ed., London: Pearson Education Limited, 2005.
3. Manning, C. D., Raghavan, P., and Schutze, H. Introduction to
Information Retrieval. New York: Cambridge University Press,
2008.
4. Beale, M. H., Hagan, M. T., and Demuth, H. B. Neural Network
Toolbox User’s Guide. Natick, MA: The MathWorks, 2015.