SOFT COMPUTING

Course Objectives
• To understand the features of neural network
and its applications
• To learn about the concept of fuzzy logic
components
• To expose the ideas about genetic algorithm
 Course Outcomes
• Ability to understand the basics of artificial
neural network and supervised learning
network
• Applying knowledge and understanding of
associative memory networks
• Applying knowledge and understanding of
unsupervised learning network
• Comprehend the fuzzy sets and the concept of
fuzziness involved in various systems
 Course Outcomes
• Understand the concepts of fuzzy logic,
knowledge representation using fuzzy rules,
approximate reasoning
• Understand fuzzy concepts and develop a
Fuzzy inference system to derive decisions
• Ability to understand the concepts of genetic
Algorithm
• Apply soft computing techniques for real life
applications
 RECOMMENDED BOOKS
Text Book
• S.N. Sivanandam & S.N.Deepa, “Principles of Soft Computing”, 2nd
Edition, Wiley India, 2011.
Reference Book
• Samir Roy and Udit Chakraborty, Introduction to Soft Computing,
Pearson.2013.
• Laurene Fausett, Fundamentals of Neural networks: architectures,
algorithms and applications , Pearson India, 2008
• Ross Timothy J, Fuzzy Logic with Engineering Applications, Wiley India Pvt
Ltd, New Delhi, 2010.
 Module-I
Topics
• Introduction to Soft computing
• Neural networks- Introduction, evolution, basic
models, terminologies of ANN,
• Pitts model
• Perceptron
• Adaline
• Back-propagation network
• RBF network
 Soft Computing
• Soft computing exploits the tolerance for imprecision,
uncertainty, and partial truth to achieve tractability,
robustness, low solution-cost, and better relationship with
reality
Soft Computing Main Components:
• Approximate Reasoning
• Search & Optimization
Neural Networks, Fuzzy Logic, Evolutionary Algorithms
 Hard computing
• Conventional computing
• It requires a precisely stated analytical model
and often a lot of computation time
• Binary logic, crisp systems, numerical analysis
 HARD COMPUTING SOFT COMPUTING

Conventional computing that are Non conventional approach that are

deterministic and has sharp boundary stochastic and has vague boundary

Precise, certain and has two valued Imprecise, uncertain and has multi
(Boolean) logic valued logic

Needs exact input Can handle ambiguous and noisy data

Not tractable Tractable solution

High computational cost Low computational cost

low Intelligence Quotient (MIQ) High Machine Intelligence Quotient (MIQ)

Precise Approximate reasoning

PROBLEM SOLVING TECHNIQUES

HARD COMPUTING SOFT COMPUTING

Precise Models Approximate Models

Traditional Functional
Symbolic Numerical Approximate Approximation
Logic Modeling and Reasoning and Randomized
Reasoning Search Search
 SOME APPLICATION AREAS OF
SOFT COMPUTING
• Data clustering
• Rule generation
• Image processing
• Medical diagnosis
• Pattern recognition
• Social networks
• Distributed computing
• Parallel processing
• Machine learning and
• Granular computing
 OVERVIEW OF TECHNIQUES IN SOFT
COMPUTING
• Neural Networks

• Fuzzy Logic

• Genetic Algorithm

• Hybrid Systems

12
 Neural Networks
• Neural network was inspired by the design and
functioning of human brain and components.
• Information processing model that is inspired by the
way biological nervous system (i.e) the brain, process
information.
• ANN is composed of large number of highly
interconnected processing elements(neurons)
working in unison to solve problems.
• It is configured for special application such as pattern
recognition and data classification through a learning
process.
• 85-90% accurate.
13
 Advantages of Neural Networks
• Adaptive learning
Ability to learn how to do tasks based on the data
given for training or initial experience.
• Self-organization
Creates its own organization or representation of
information it receives during learning time.
• Real time operation
Computations can be carried out in parallel.
• Fault tolerance via redundant information coding
Partial destruction of neural network cause
degradation of performance. In some cases, it can
be retained even after major network damage.
Multi-disciplinary point of view of
Neural Networks
 Application Scope of Neural Network
• Air traffic control
• Animal behaviour
• Appraisal and valuation of
property, etc.,
• Betting on horse races, stock
markets
• Criminal sentencing
• Complex physical and
chemical process
• Data mining, cleaning and
validation
• Direct mail advertisers
• Echo patterns
• Economic modeling
• Employee hiring
• Expert consultants
• Fraud detection
• Hand writing and typewriting
• Lake water levels
• Machinery controls
• Medical diagnosis
• Music composition
• Photos and finger prints
• Recipes and chemical formulation
• Traffic flows
• Weather prediction
 Fuzzy Logic
1. An organized method for dealing with imprecise data is called fuzzy
data
• Fuzzy logic includes 0 and 1 as extreme cases of truth (or "the state
of matters" or "fact") but also includes the various states of truth in
between so that, for example, the result of a comparison between
two things could be not "tall" or "short" but ".38 of tallness.“
• Allows partial membership
• Implemented in small, embedded micro controllers to large,
networked, multichannel PC or work station.
• Can be implemented in hardware, software or in both.
• Fuzzy logic provides a simple way to arrive at a definite conclusion
based upon vague, ambiguous, imprecise, noisy or missing input
information.
 Genetic Algorithm
• How genes of parents combine to form those of their children.
• Create an initial population of individuals representing possible solutions
to solve a problem
• Individual characters determine whether they are less or more fit to the
population
• The more fit members will take high probability.
• It is very effective in finding optimal or near optimal solutions.
• Generate and test strategy.
• Differ from normal optimization and search procedures in:
Work with coding of the parameter set
Work with multiple points
Search via sampling( a blind search)
Search using stochastic operators
• In business, scientific and engineering circles, etc.,
 Hybrid System
Neuro Fuzzy hybrid system
 Combination of fuzzy set theory and neural networks
 Fuzzy system deal with explicit knowledge that can be explained and
understood
 Neural network deal with implicit knowledge acquired by learning
Advantages
 Handle any kind of information (Numeric, Linguistic, logical)
 Manage imprecise, partial, vague or imperfect information
 Resolve conflicts by collaboration and aggregation.
 Self-learning, self-organizing and self-tuning capability
 No need for prior knowledge of relationship of data
 Mimic human decision making system
 Computation fast by using fuzzy number operations.
 Hybrid System
Neuron genetic hybrid system
Topology optimization
 Genetic algorithm used to select a topology for ANN , common one is back
propagation
Genetic algorithm training
 Learning of ANN is formulated ad weight optimization problem, usually
mean squared error as a fitness measure
Control parameter optimization
 Learning rate, momentum rate, tolerance level. etc., are optimized using
GA.
 Hybrid System
Fuzzy genetic hybrid system
 Optimization ability of GA are used to best set of rules to be
used for fuzzy inference engine
 Creating the classification rules for a fuzzy system where
objects are classified by linguistic terms.
 Find the appropriate set of rules
 Training data and randomly generated rules are combined to
create initial population
 Fitness function measures the strength of rules, balancing the
quality and diversity of the population.
 NEURAL NETWORKS
 Neural networks design is inspired by the design and
functioning of human brains and components
 It has the ability to learn by example
 It has made them very flexible and powerful tool
 The networks are also well suited for real-time systems
 They have fast response and less computational times
 They have a parallel architecture
 NEURAL NETWORKS
 Resembles the characteristic of biological neural network.
 Nodes – interconnected processing elements (units or
neurons)
 Neuron is connected to other by a connection link.
 Each connection link is associated with weight which has
information about the input signal.
 ANN processing elements are called as neurons or artificial
neurons , since they have the capability to model networks of
original neurons as found in brain.
 Internal state of neuron is called activation or activity level of
neuron, which is the function of the inputs the neurons
receives.
 Neuron can send only one signal at a time.
 ARCHITECTURE OF A SIMPLE ANN
y in  x1w1  x 2 w 2 y  f ( y i n ) f : Activation function
X1, X2 : Input neurons Y : Output neuron
X 1 , X 2 transmit signals,Y receives signal
x1
X1
w1

Y y

x2 w2
X2

x1, x2 : activations of the input neurons : output of input signals

w1, w2 : associated weights, whichcontain information about theinput signals
 Activation function
• The function to be applied over the net input
is called activation function.
• Weight involved in ANN is equal to the slope
of linear straight line (y=mx).
 TASKS PERFORMED BY ANN
• Pattern-matching
• Classification
• Optimization function
• Approximation
• Vector quantization
• Data Clustering
SCHEMATIC DIAGRAM OF A BIOLOGICAL
NEURON
Synapse : bulb like organ at the end of strands
Strands :The splits at the end of axons
Synapse

Nucleus Strands

Axon

Cell body (Soma)

Dendrites

Dendrites :Where the nerve is connected to the cell body

Axon :Which carries the impulses of theneuron
TERMINOLOGICAL RELATIONSHIP BETWEEN
BIOLOGICAL NN AND ANN
Biological Neuron Artificial Neuron
Cell Neuron

Dendrites Weights or Interconnections

Soma Net input

Axon Output
 BIOLOGICAL NN CONTD…
 In the human brain there are approximately 10000 synapses
per neuron

Mathematical representation of the above process in ANN is:

 Suppose there are n inputs from n neurons X 1 , X 2 , ... X n with
activations x 1 , x 2 , . . . x n respectively
 Let the weights of the interconnections between X 1 , X 2 , . . . X n
and the connecting neuron Y be w1, w2 ,...wn respectively
 ARTIFICIAL NEURAL NETWORKS
CONTD…
• The net input to the neuron Y is given by the formula:
y in  x 1 w 1  x 2 w 2  ...x n w n
• The activation function is applied to yin to compute the
output
 The weight represents the strength of synapse connecting the
input and the output neurons
 The weights may be positive or negative
 +ve weight means the synapse is excitatory
 -ve weight means the synapse is inhibitory
 Brain Vs computer
Term Brain
Speed Execution time is few
milliseconds
Processing Perform massive parallel
operations simultaneously
Size and complexity Number of Neuron is 1011 and
number of interconnections is
1015.
So complexity of brain is
higher than computer
Storage capacity i) Information is stored in
interconnections or in synapse
strength.
ii) New information is stored
without destroying old one.
iii) Sometimes fails to recollect
information
Computer
Execution time is few nano
seconds
Perform several parallel
operations simultaneously.
It is faster the biological
neuron
It depends on the chosen
application and network
designer.
i) Stored in continuous
memory location.
ii) Overloading may destroy
older locations.
iii) Can be easily retrieved
31
 Contd…

Tolerance Fault tolerant i) Information corrupted

Store and retrieve information if the network
even interconnections fails connections
Accept redundancies
disconnected.
ii) No redundancies
Control mechanism Depends on active CPU
chemicals and neuron Control mechanism is very
connections are strong or simple
weak

32
 Characteristics of ANN
 Neurally implemented mathematical model
 Large number of interconnected processing elements called neurons exists
here.
 Interconnections with weighted linkage hold informative knowledge.
 Input signals arrive at processing elements through connections and
connecting weights.
 Processing elements can learn, recall and generalize from the given data by
adjustment of weights
 Computational power is determined by the collective behaviour of neurons.
 ANN is a connection models, parallel distributed processing models, self-
organizing systems, neuro-computing systems and neuro-morphic system
 BASIC MODELS OF ARTIFICIAL NEURAL NETWORK

• The models are specified by the three basic entities

 The model’s synaptic (through synapses) interconnections

 The training or learning rules adopted for updating and
adjusting the connection weights
 The activation functions
 CONNECTIONS
 The neurons can be visualised for their arrangements in layers
 An ANN consists of a set of highly interconnected processing elements
called neurons
 Output of each processing element is found to be connected through
weights to the other processing elements or itself
 Delay lead and lag-free connections are allowed
 Arrangement of these processing elements and the geometry of their
interconnections are essential for an ANN
 The point where the connection originates and terminates
should be noted
 The function of each processing element in an ANN should be specified
 BASIC NEURON CONNECTION
ARCHITECTURES
There are five types of basic connections:
 SINGLE-LAYER FEED FORWARD NETWORK
 MULTI-LAYER FEED FORWARD NETWORK
 SINGLE NODE WITH ITS OWN FEEDBACK
 SINGLE-LAYER RECURRENT NETWORK
 MULTI-LAYER RECURRENT NETWORK
 SINGLE LAYER FEED FORWARD
NETWORK
• Architecture
X1 w11
Y1
w21
w12
X2 w22
Y2
Input wn1 Output
neurons w23 wn2 neurons

w13

Xn Ym
wn3

Input Layer Output Layer

 Single layer Feed- Forward Network
 Layer is formed by taking processing elements and
combining it with other processing elements.
 Input and output are linked with each other
 Inputs are connected to the processing nodes with
various weights, resulting in series of outputs one
per node.

38
 SINGLE LAYER FEED FORWARD
NETWORK

• This architecture is called the single-layer feed forward

network
• The input nodes are say X i , i  1, 2,...n
• The output nodes be say Y j , j  1, 2...m
•
The connections from ‘n’ input nodes to the ‘m’ output nodes
may be assigned with weights be given by
wij ,i 1, 2...n; j 1, 2...m
 Multilayer Feed-forward
Network
• Formed by the interconnection of
several layers.
• Input layer receives input and buffers
input signal.
• Output layer generated output.
• Layer between input and output is
called hidden layer.
• Hidden layer is internal to the
network.
• Zero to several hidden layers in a
network.
• More the hidden layer, more is the
complexity of network, but efficient
output is produced.

40
 Feedback Network
• If no neuron in the output layer is an input
to a node in the same layer / proceeding
layer – feed forward network.
• If outputs are directed back as input to the
processing elements in the same
layer/proceeding layer –feedback network.
• If the output are directed back to the input
of the same layer then it is lateral
feedback.
• Recurrent networks are networks with
feedback networks with closed loop.
• Fig 2.8 (A) –simple recurrent neural
network having a single neuron with
feedback to itself.
• Fig 2.9 – single layer network with
feedback from output can be directed to
processing element itself or to other
processing element/both.

41
SINGLE NODE WITH ITS OWN
FEEDBACK

Output
Input

Feedback
 SINGLE LAYER RECURRENT
NETWORK

X1 w11
Y1

X2 w22
Y2

wnm
Xn Ym

Processing element output can be directed back to the processing element itself or to the
other processing element or both.
Multilayer Recurrent network
• Processing element output
can be directed back to the
nodes in the preceding layer,
forming a multilayer
recurrent network.
• Processing element output
can be directed to
processing element itself or
to other processing element
in the same layer.

44
 Learning
Neural network adapts itself to a stimulus by making proper parameter
adjustment, resulting in the production of desired response.
Two broad kinds of learning in ANNs is
 Parameter learning – updates connecting weights in a neural net.
 Structure learning – focus on change in the networks structure
(Number of processing elements, types of
connection between nodes)
Apart from these, learning in ANN is classified into three categories as
 Supervised learning
 Unsupervised learning
 Reinforcement learning

45
 Supervised Learning
 Learning with the help of a teacher.
 In ANN, each input vector requires a
corresponding target vector, which
represents the desired output.
 The input vector along with target vector
is called training pair.
 The input vector results in output vector.
 The actual output vector is compared with
desired output vector.
 If there is a difference means an error
signal is generated by the network.
 It is used for adjustment of weights until
actual output matches desired output.
 Unsupervised learning
• Learning is performed without the help
of a teacher.
• Example: tadpole – learn to swim by
itself.
• In ANN, during training process,
network receives input patterns and
organize it to form clusters.
• From the Fig. it is observed that no
feedback is applied from environment to
inform what output should be or whether
they are correct.
• The network itself discover patterns,
regularities, features/ categories from the
input data and relations for the input data
over the output.
• Exact clusters are formed by discovering
similarities & dissimilarities so called as
self – organizing.
47
Reinforcement learning
 Similar to supervised learning.
 Learning based on critic information
is called reinforcement learning &
the feedback sent is called
reinforcement signal.
 The network receives some feedback
from the environment.
 Feedback is only evaluative.
 The external reinforcement signals
are processed in the critic signal
generator, and the obtained critic
signals are sent to the ANN for
adjustment of weights properly to get
critic feedback in future.
48
 Activation functions
 To make work more efficient and for exact output, some force or activation
is given.
 Like that, activation function is applied over the net input to calculate the
output of an ANN.
 Information processing of processing element has two major parts
input and output.
 An integration function (f) is associated with input of processing element.
 This function serves to combine activation, information or evidence from
an external source or other processing elements into a net input to the
processing element

49
 Activation functions
1. Identity function:
It is a linear function which is defined as
f(x) =x for all x
The output is same as the input. The input layer uses the identify
activation function

2. Binary step function

It is defined as

where θ represents thresh hold value.

It is used in single layer nets to convert the net input to an output that
is binary. ( 0 or 1)
 Activation functions
3. Bipolar step function:
It is defined as

where θ represents threshold value.

This function used in single layer nets to convert the net input to an output
that is bipolar (+1 or -1).

51
 Activation functions
3.Sigmoid function
Back propagation nets.
Two types:
a) binary sigmoid function
logistic sigmoid function or unipolar sigmoid function.

where λ – steepness parameter.

The derivative of this function is
f’(x) = λ f(x)[1-f(x)]. The range of sigmoid function is 0 to 1.
 Activation functions
b) Bipolar sigmoid function

where λ- steepness parameter and the sigmoid range is between -1 and +1.
5. Ramp function

53
54
 Important Terminologies
• Weight
– The weight contain information about the input signal.
– It is used by the net to solve the problem.
– It is represented in terms of matrix & called as
connection matrix.
– If weight matrix W contains all the elements of an
ANN, then the set of all W matrices will determine the
set of all possible information processing
configuration.
– The ANN can be realized by finding an appropriate
matrix W
55
 Important Terminologies
• Bias
– Bias has an impact in calculating net input.
– Bias is included by adding x0 to the input vector x.
– The net output is calculated by

– The bias is of two types

– Positive bias
» Increase the net input
– Negative bias
» Decrease the net input
56
 Important Terminologies
• Threshold
 It is a set value based upon which the final output is calculated.
 Calculated net input and threshold is compared to get the
network output.
 The activation function of threshold is defined as

where θ is the fixed threshold value

57
 Important Terminologies
• Learning rate
 Denoted by α.
 Control the amount of weight adjustment at each step of training.
 The learning rate range from 0 to 1.
 Determine the rate of learning at each step
• Momentum Factor
 Convergence is made faster if a momentum factor is added to the weight
updation process.
 Done in back propagation network.
• Vigilance parameter
 Denoted by ρ.
 Used in Adaptive Resonance Theory (ART) network.
 Used to control the degree of similarity.
 Ranges from 0.7 to 1 to perform useful work in controlling the number of
clusters.

58
 Problems -1
For network shown in figure, calculate the net input to the
output neuron.
 Problem -2
Calculate the net input for the network shown in figure
 Problem -3
• Obtain the output of the neuron Y for the network shown in
figure using activation function
• i. Binary sigmoid ii. Bipolar sigmoid
 Problem 3
Solution
𝑛
𝑦𝑖𝑛 = 𝑖 𝑥𝑖 𝑤𝑗
=b+x1W1+x2W2+x3W3 = 0.53
Binary sigmoid activation
= 0.625
Biplolar sigmoid activation function
= 0.259
Mcculloch-Pitts (M-P) neuron
 Mcculloch-Pitts (M-P) neuron
 Discovered in 1943.
 Usually called as M-P neuron.
 M-P neurons are connected by directed weighted paths.
 Activation of M-P neurons is binary (i.e) at any time step the neuron may
fire or may not fire.
 Weights associated with communication links may be excitatory(weights
are positive)/inhibitory(weights are negative).
 Fixed threshold for each neuron and if the net input to the neuron is
greater than the threshold then the neuron fires.
 They are widely used in logic functions.

64
Mcculloch-Pitts neuron
 A simple M-P neuron is shown in the
figure.
 It is excitatory with weight (w>0) /
inhibitory with weight –p (p<0).
 In the Fig., inputs from x1 to xn possess
excitatory weighted connection and
Xn+1 to xn+m has inhibitory weighted
interconnections.
 Since the firing of neuron is based on
threshold, activation function is
defined as

65
 Mcculloch-Pitts neuron (Contd…)
 For inhibition to be absolute, the threshold with the activation
function should satisfy the following condition:
θ >nw –p
 Output will fire if it receives “k” or more excitatory inputs but
no inhibitory inputs where
kw≥θ>(k-1) w
 The M-P neuron has no particular training algorithm.
 An analysis is performed to determine the weights and the
threshold.
 It is used as a building block where any function or
phenomenon is modeled based on a logic function.
66
 Problem -4
Implement AND function using McCulloch-pitts neuron
(take binary data).

x1 x2 y

1 1 1
1 0 0
0 1 0
0 0 0
Assume weights be w1 = 1 and w2 = 1.
yin=x1w1+x2w2
(1,1) = l x 1+1 x I =2
(l,0) =1x1 +0 X 1 = 1
(0, 1) = 0 X 1+ 1 X 1 = 1
(0,0) = 0 X 1 +OX 1 = 0
The net input is calculated as 2.
if the threshold value is greater than or equal
to 2 then the neuron fires, else it does nor fire.
So the threshold value is set equal to2((ϴ= 2).
θ ≥ nw –p
n=2, w=1 p=0
ϴ≥2x1-0 => ϴ≥2
Thus , the output of neuron Y can be written
as
ϴ=2, W1=1, W2=1