Jacek Mazurkiewicz, PhD

Softcomputing
Part 1: Introduction, Elementary ANNs
Formal Introduction
contact hours, room No. 225 building C-3:
Tuesday: 11-13, Wednesday: 11-13
slides: www.zsk.ict.pwr.wroc.pl
Professor Wiktor Zinn
test: 19.01.2012 during lecture
- softcomputing:
- lecture + laboratory
- laboratory mark 20% of final mark
- bonus question!

Program
Idea of intelligent processing
Fuzzy sets and approximate reasoning
Expert systems - knowledge base organization
Expert systems - reasoning rules creation
Expert systems: typical organization and applications
Artificial neural networks: learning and retrieving algorithms
Multilayer percetpron
Kohonen neural network
Hopfield neural network
Hamming neural network
Artificial neural networks: applications
Genetic algorithms: description and classification
Genetic algorithms: basic mechanisms and solutions

Literature
B. Bouchon Meunier, Fuzzy Logic and Soft Computing
O. Castilo, A. Bonarini, Soft Computing Applications
M. Caudill, Ch. Butler, Understanding Neural Networks
E. Damiani, Soft Computing in Software Engineering
R. Hecht-Nielsen, Neurocomputing
S. Y. Kung, Digital Neural Networks
D. K. Pratihar, Soft Computing
S. N. Sivanandam, S. N. Deepa, Principles of Soft Computing
A. K. Srivastava, Soft Computing
D. A. Waterman, A Guide to Expert Systems
D. Zhang, Parallel VLSI Neural System Design

Why Neural Networks and Company?
Still in active use
No chance to solve some problems in other way
Human ability vs. classical programs
Works as primitive humans brain
Artificial intelligence has power!
ANN + Fuzzy Logic + Expert Systems + Rough Sets + Ant Algorithms
= SoftComputing
The Story
1943 McCulloch & Pitts
model of artificial neuron
1949 Hebb
information stored by biological neural nets
1958 Rosenblatt
perceptron model
1960 Widrow & Hoff
first neurocomputer - Madaline
1969 Minsky & Papert
XOR problem single-layer perceptron limitations
1986 McCleland & Rumelhart
backpropagation algorithm
Where Softcomputing is in Use?
Letters, signs, characters, digits recognition

Recognition of ship types data from sonar

Electric power prediction

Different kinds of simulators and computer games

Engine diagnostic in planes, vehicles

Rock-type identification

Bomb searching devices
Neural Networks Realisation
Set of connected identical neurons

Artificial neuron based on a biological neuron

Hardware realisation digital device

Software realisation simulators

Artificial neural network idea, algorithm, mathematical formulas

Works in parallel

No programming learning process necessary


Learning
With a Teacher




Without a Teacher



Klasyfikator
Wektor cech
(dane nauki)
Wynik
klasyfikacji
Nauczyciel Teacher
Learning
vector
Parameters
Weights
Result of
learning
Klasyfikator
Wektor cech
(dane testowe)
Wynik
klasyfikacji
Learning
vector
Result of
learning
Parameters
Weights
Softcomputing vs. Classical Computer
Different limitations of softcomputing methods

No softcomputing:

operations based on symbols: editors, algebraic equations

calculations with a high level of precision

Softcomputing is very nice, but not as universal as computer



Anatomy Foundations (1)
Nervous System 2-ways, symmetrical
set of structures, divided into 4 parts:

Spinal Cord
receiving and transmission of data

Prolonged Cord
breathing, blood system, digestion

Cerebellum
movement control

Brain (ca. 1.3 kg) 2 hemispheres
feeling, thinking, movement



brain
brain stern
cerebellum
prolonged cord
spinal cord
nervous system
Anatomy Foundations (2)
Anatomy Foundations (3)
Cerebral cortex thickness: 2 mm, area: ca. 1.5 m
2

Cerebral cortex divided into 4 part lobes

Each lobe is corrugated

Each hemisphere is responsible for half part of body:
right for left part, left for right part

Hemispheres are identical in case of a structure, but
their functions are different



Anatomy Foundations (4)
Brain composed by fibres with large number of branches
Two types of cells in nervous tissue: neurons and gley cells
There are more gley cells:
no data transfer among neurons
catering functions
Ca. 20 milliard neurons in cerebral cortex
Ca. 100 milliard neurons in whole brain
Neuron: dendrites inputs, axon output, body of neuron
Neuron: thousands of synapses connections to other neurons

Anatomy Foundations (5)
Neurons in work:
chemical-electrical signal transferring
cell generates electrical signals
electric pulse is changed into a chemical signal at the end of axon
chemical info passed by neurotransmitters
50 different types of neurons
neurons driven by a frequency of hundreds of Hz
neurons are rather low devices!


Anatomy Foundations (6)
Biological and Artificial Neural Nets
Artificial neural networks are a good solution for:
testing already identified biological systems
pattern recognition
alternative configurations to find the basic features of them
Artificial neural networks are primitive brothers of biological nets
Biological nets have sophisticated internal features important for their normal work
Biological nets have sophisticated time dependences ignored in most artificial networks
Biological connections among neurons are different and complicated
Most architectures of artificial nets are unrealistic from the biology point of view
Most learning rules for artificial networks are unreal in biology point of view
Most biological nets we can compare to already learned artificial nets to realise function
described in a very detailed way


Linear ANN - ADALINE (ADAive Linear Neuron)
single neurons answer:
+ .
.
.
x
x
x
1
2
M
w
w
w
1
2
M
w
0
y
1

=
+ =
M
j
j j
w x w y
1
0
M number of input neurons
K number of output neurons

=
= =
M
j
j j
x w y
0
~
)
~
( x w x
T
) ,..., , (
~
1 0 M
x x x col = x
1
0
= x
) ,..., , (
1 0 M
w w w col = w
scalar description vector description
multi-output net:
Single-Layer Multi-Output Network
1 x x x
1 2 M
y y y
1 2 K
w w w
10 20 K0
w w w
11 12 1K
w w w
21 22 2K
w w w
M1 M2 MK
W
kj
Output
neuron
Input
neuron
k-neurons answer:

=
=
M
j
j kj K
x w y
0
) (x
column = x w y(x)
T
WX y(X) =
(
(
(
(

=
KM K K
M
M
w w w
w w w
w w w

1 0
2 21 20
1 11 10
W
Learning Procedure
experimental data: N - series
N
x x x ,..., ,
2 1
N
K K K
t t t ,..., ,
2 1
learning data
required answers
,
N
K
N
t x function implemented by net
error function mean-square error:
( ) ( )

= =
=
N
n
K
k
n
k k
t y W E
1 1
2
2
1
) ( w

= = =
|
|
.
|

\
|
=
N
n
K
k
M
j
n
k
n
j jk
t x w W E
1 1
2
0
2
1
) (
looking for a minimum of E(W) function:
0
) (
,
=
c
c
.
kj
j k
w
W E
Pseudoinverse Algorithm

= =
=
|
|
.
|

\
|
=
c
c
N
n
n
j
M
j
n
k
n
j kj
kj
x t x w
w
W E
1 0 '
'
0 2
2
1 ) (

= = =
= .
N
n
M
j
N
n
n
j
n
k
n
j
n
j kj
x t x x w
j k
1 0 ' 1
'
,
where:
(
(
(
(
(

=
N
M
N
M
M
x x
x x
x x

1
2 2
1
1 1
1
1
1
1
X
(
(
(
(
(

=
N
K
N N
K
K
t t t
t t t
t t t

2 1
2 2
2
2
1
1 1
2
1
1
T
(
(
(
(

=
KM K K
M
M
w w w
w w w
w w w

1 0
2 21 20
1 11 10
W
finally:
( ) T X W X X
T T T
= T XW
T
= T X X) (X W
T 1 T T
=
rse pseudoinve
=

T, X W
T
Gradient-Type Algorithms (1)
y
x
iterative approach:
t t t
q x
x
y
x x
c
c
=
+1
steps:
random weight vector
new weight vector following: -V
w
E
repeat process generating the sequence of weights vectors:
components of weight vectors calculated by
) (t
w
t
t t
q w
w
kj
kj kj
w
E
w w
c
c
=
+
) (
) 1 (
error function:

=
n
n
E E ) ( ) ( w w

= =
|
|
.
|

\
|
=
K
k
M
j
n
k
n
j kj
n
t x w E
1
2
0
2
1
) (w
Gradient-Type Algorithms (2)
sequential approach:
kj
n
kj kj
w
E
w w
c
c
=
+
q
t t ) 1 (
{ }
n
j
n
k
n
j
n
k
n
k
kj
n
x x t y
w
E
* * ) ( o = =
c
c
x
error delta rule: Widrow-Hoff rule:
n
k
n
k
n
k
t y = ) (x o
n
j
n
k jk kj
x w w qo
t t
=
+ ) 1 (
algorithm: 1. set start values by a random way for example
2. calculate a net answer for available x
n

3. calculate an error value o
k
n

4. calculate a new weight vector w
kj
(t+1)
according to the delta rule
5. repeat steps 2. 4. until E less than required value
Perceptron (1)
x
1
x
2
x
M
w
0
w
iM
w
i2
w
i1
y
i
1
the story:
Rosenblatt (1962)
classification task
Widrow & Hoff (1960) - ADALINE
answer:
) ( ) (
0
x w
T
g x w g X y
M
j
j j
=
|
|
.
|

\
|
=

=
w
0
threshold value
activation function:

>
<
=
0 1
0 1
) (
a for
a for
a g

<
>
=
0 0
0 1
) (
a for
a for
a g
bipolar unipolar

=
=
M
j
j j
x w a
0
Perceptron (2)
error function:
jk
w
E
c
c
does not exist, because g(a) is not differentiable
perceptron criterions:
compare the actual value of y
i
and the required output value d
i
and:
if y
i
= d
i
the weights values of W
ij
and w
0
are unchanged
if y
i
= 0 and the required value d
i
=1 update the weights as follow:


where: t previous cycle, t+1 actual cycle
if y
i
= 1 and d
i
= 0 update the weights according to:


where: b
i
polarity, d
i
required neurons output signal
, ) ( ) 1 (
j ij ij
x t W t W + = + , 1 ) ( ) 1 ( + = + t b t b
i i
, ) ( ) 1 (
j ij ij
x t W t W = + 1 ) ( ) 1 ( = + t b t b
i i
Perceptron (3)

=
=
p
k
k
i
k
i
d y E
1
2 ) ( ) (
, ) (
summarising:
1. look-up the input learning vectors
2. if classification is correct weights are not changed
3. if classification is wrong:
if t
n
= +1 add qx
n
to the weight values
else subtract qx
n
from the weight values
value of q is not important can be set to 1, it only scales w
i
Perceptron Problems (1)
linear separability: XOR problem Minsky & Papert (1969):
y(x)=0
C
2

X
1

X
2

C
1

In
1
In
2
Out
XOR
0 0 0
0 1 1
1 0 1
1 1 0
In In Out
1 2
non-linear separable problem
solution: multilayer net
Perceptron Problems (2)
multilayer network for
XOR problem solution:
w w
w w
-2w w
u =
1
w
u =
2
w
u=2w
o
1
o
2
s
1
s
2
S