Artificial Neural Network

Content
1. History of Neural Networks

2. What is an Artificial Neural network ?

3. Biological Neuron

4. Artificial Neuron

5. Basic Model of Artificial Neural Network

6. Classification Based on Interconnections

7. Learning Rules

8. Activation Functions

9. Training an Artificial Network Based on Supervised Learning

 History of Neural Network

First step toward harnessing the thinking power of the human brain was done in 1943
when Warren McCulloch, a neurophysiologist, and a young mathematician, Walter Pitts,
wrote a paper on how neurons might work and modeled a simple neuron using electrical
circuits.

In 1949 in book The Organization of Behavior, it was pointed out that neural
pathways strengthen each time as they are used.

In 1950s IBM laboratories first attempted to simulate a neural network but failed.

In 1956, the Dartmouth Summer Research Project on Artificial Intelligence provided a

boost to study of neural networks.

Frank Rosenblatt, a neurobiologist began work on perceptron and it is the oldest neural
network still in use.

In 1959, Bernard Widrow and Marcian Hoff developed the first neural network to be
applied to real world problems.
After that neural network research faced stunted growth till 1982 due to the fear
of effects of thinking machines on human race.

By 1985 the American Institute of Physics began an annual meeting Neural

Networks for Computing.

By 1987, the Institute of Electrical and Electronic Engineers (IEEE) first

International Conference on Neural Networks drew more than 1800 attendees.

And research is still going on in this field till date.

 Biological Neuron

Has 3 Parts
Soma or Cell body Cell nucleus is located
Dendrites:- nerve connected to cell body
Axon: carries impulses of the neuron
End of axon splits into fine strands.
Each strand terminates into a bulb-like organ called synapse.
Electric impulses are passed between the synapse and dendrites.
Synapses are of two types
Inhibitory:- impulses hinder the firing of the receiving cell
Excitatory:- impulses cause the firing of the receiving cell
The information transmission happens at the synapses.
 Artificial Neuron

The McCullogh-Pitts model of an artificial Neuron:

Spikes are interpreted as spike rates.
Synaptic strength are translated as synaptic weights.
Excitation means positive product between the incoming spike rate and the
corresponding synaptic weight.
Inhibition means negative product between the incoming spike rate and the
corresponding synaptic weight.
 What is an Artificial Neural Network ?

Artificial Neural Network are the electronic models based on neural structure
of brain.

It possess a large number of processing elements called nodes or neurons

which operate in parallel.

Neurons are connected with other by connection links.

Each link is associated with weights which contain information about the
input signal.

Each neuron or node has an internal state of its own which is a function of
inputs that a neuron receives.
Artificial Neural Network Basic Model
 Basic Model of Artificial Neural Network

Basic Models of
ANN

Activation
Interconnections Learning rules
function
Classification Based on Interconnections
Interconnections

Feed forward Feed Back

Single layer

Multilayer
 Learning Rules

Learning - Its a process by which a NN adapts itself to a stimulus by

making proper parameter adjustments, resulting in the production of
desired response

Two kinds of learning -

Parameter learning:- connection weights are updated

This is achieved through
Supervised Learning
Unsupervised Learning

Structure Learning:- change in network structure

 Activation Functions
The transfer function translates the input signals to output signals.
 Building a Neural Network
(Supervised)
Steps
01. Forward propagation
In forward propagation, we apply a set of weights to the input data and
calculate an output. For the first forward propagation, the set of weights is
selected randomly

02. Back propagation

In back propagation, we measure the margin of error of the output
and adjust the weights accordingly to decrease the error.

Neural networks repeats both forward and back propagation until the weights are
calibrated to accurately predict an output.
EX: Exclusive or (XOR) Inputs
Outputs

0 0 0
0 1 1
1 0 1
1 1 0
FORWARD PROPAGATION:

Using the last row data (1,1) -- 0 to construct the first forward propagation

Input Hidden Outpu

t

1
Choosing first forward propagation weights randomly

Input Hidden Output

0.8 0.
3
1 0.4
0.5 0
0.9
2
0.

1 9
0.

0.5 0.
3

Calculation of first values for the hidden layer

Hidden layersums

Application of activation function to the hidden layer sums

Activation function: Sigmoid function

Applying S(x) to the three hidden layer sums, we get:

S(1) = = 0.73105857863

S(1.3) = = 0.78583498304

S(0.8) = = 0.68997448112
Adding these values to our neural network as hidden layer results

Input Hidden Output

0.73

0.79 0.5 0

1 0.
3

0.69

Output sum is:

Applying the activation function to get the final output result

S(1.235) = = 0.7746924929
Full diagram:

Input Hidden Output

0.73

0.79 0.5 0.77

1
0.
3

0.69

Target = 0
Calculated = 0.77
BACK PROPAGATION

Calculating the incremental change to these weights happens in two steps

0.1 Finding the margin of error of the output result to back out the necessary
change in the output sum (delta output sum)

0.2 Extracting the change in weights by multiplying delta output sum by

hidden layer results

Output sum margin error = Target Calculated

= 0 0.77
= - 0.77
Calculation of delta output sum

This means that the change in the output sum is the same as the sigmoid prime of the output result

Delta output sum =
Delta output sum =
Delta output sum =
Calculation of new change in weights

= Hidden layer result

= Hidden to outer weights
= Output sum

(This relationship suggests that a greater change in

output sum yields a greater change in the weights)

Delta weights = delta output sum / hidden layer results

= -0.1344 / [0.73105, 0.78583, 0.69997]
= [-0.1838, -0.1710, -0.1920]

Old w7 = 0.3 New w7 = 0.3 0.1838 = 0.1162

Old w8 = 0.5 New w8 = 0.5 0.1710 = 0.329
Old w9 = 0.9 New w9 = 0.9 0.1920 = 0.708
Determination of the change in the weights between the input and hidden layers

We can also define

Delta hidden sum = (delta output sum / hidden to outer weights) (Hidden sum)
= (-0.1344 / [0.3, 0.5, 0.9])
= [-0.088, -0.0452, -0.0319]


Delta weights = delta hidden sum / input data

= [-0.088, -0.0452, -0.0319] / [1, 1]
= [-0.088, -0.0452, -0.0319, -0.088, -0.0452, -0.0319]

Old w1 = 0.8 New w1 = 0.8 0.088 = 0.712

Old w2 = 0.4 New w2 = 0.4 0.0452 = 0.3548
Old w3 = 0.3 New w3 = 0.3 0.0319 = 0.2681
Old w4 = 0.2 New w4 = 0.2 0.088 = 0.112
Old w5 = 0.9 New w5 = 0.9 0.0452 = 0.8548
Old w6 = 0.5 New w6 = 0.5 0.0319 = 0.4681
 Again performing forward propagation

Input Hidden Output

0.69

1
0. 0.77 0.329 0.69

1
26

Target = 0
8

0.68 Calculated = 0.69

After one iteration calculated value changed from 0.77 to 0.69 which is
more near to desired value

This training of network (forward and back propagation) is repeated thousands of times
to get the desired results
MATLAB Code to train for XOR

net = feedforwardnet([3]);
%Repeat 4 times
P = [0 0; 0 1; 1 0; 1 1; 0 0; 0 1; 1 0; 1 1; 0 0; 0 1; 1 0; 1 1; 0 0; 0 1; 1 0; 1 1]';
T = [0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0]; % desired output
net = configure(net, P, T);
net.trainParam.goal = 1e-8;
net.trainParam.epochs = 1000;
net = train(net, P, T);

sim(net, P)

ans =
Columns 1 through 8
0.0000 1.0000 1.0000 0.0000 0.0000 1.0000 1.0000 0.0000
Columns 9 through 16
0.0000 1.0000 1.0000 0.0000 0.0000 1.0000 1.0000 0.000