TIME DELAY NEURAL NETWORKS

Introduction
Time delay neural networks (TDNN) have the ability to
represent relationships between events in time
The neuron is fed not only the current input, but previous
inputs also
In this way, a TDNN neuron has the ability to relate and
compare current input to the past history of events

TIME DELAY NEURAL NETWORKS

Architecture

TIME DELAY NEURAL NETWORKS

Architecture

TIME DELAY NEURAL NETWORKS

Architecture

TIME DELAY NEURAL NETWORKS

Architecture

TIME DELAY NEURAL NETWORKS

Architecture

TIME DELAY NEURAL NETWORKS

Training of Weights
Since the time
copies of the
neurons
are
duplicates,

shifted
TDNN
mere

and they are supposed to

look for the same event in
the input pattern;
hence their weights are
constrained to be the same

TIME DELAY NEURAL NETWORKS

Training of Weights
Back-propagation forward and backward pass are done
normally as if the time shifted copies are the same events
Rather than changing the weights on time-shifted
connections separately, they are averaged and updated on
each connection by the same value
In this way, the network is forced to discover useful features
in the input, regardless of when in time they actually
occurred

TIME DELAY NEURAL NETWORKS

Utilization
The network is presented with the complete timedependent pattern, and it calculates its output
TDNN are not shift-sensitive, i.e. misaligned input patterns
do not result in poor performance

TIME DELAY NEURAL NETWORKS

Utilization
They require patterns of fixed time lengths
If the length of the pattern is longer, we can truncate it
If the length is shorter, we can duplicate the last input vector
to complete the specified length
This is safe only if the pattern does not contain useful
information at its end

TIME DELAY NEURAL NETWORKS

Utilization
Recurrent nets are shown only the current input. Previous
inputs are implicitly present in the network
TDNN is shown the complete pattern explicitly

TIME DELAY NEURAL NETWORKS

Utilization
This weight sharing concept has been extended to spatial
dimensions, and has been used in optical character
recognition systems, where the location of an image in the
input space is not precisely known

TIME DELAY NEURAL NETWORKS

References
Read
Section 3.1.5, Engelbrecht
&
Page 642 (course folder)