Deep Learning

and
Convolutional
Neural Networks
Ronald XIE 8th May, 2014

Deep Learning and Convolutional Neural Networks | Page 1

Robot
Tensor voting system
localization and
Robot operating
Scene
system Labeling
Deep learning system

Robot operating system

Outline
Motivation
Deep Learning
Convolutional Neural Networks
Applications

Deep Learning and Convolutional Neural Networks | Page 2

Outline
Motivation
Deep Learning
Convolutional Neural Networks
Applications

Deep Learning and Convolutional Neural Networks | Page 3

Motivation feature representation

Deep Learning and Convolutional Neural Networks | Page 4

Motivation feature representation

Deep Learning and Convolutional Neural Networks | Page 5

Motivation feature representation

Deep Learning and Convolutional Neural Networks | Page 6

Motivation feature representation

Deep Learning and Convolutional Neural Networks | Page 7

Motivation feature representation

Deep Learning and Convolutional Neural Networks | Page 8

Motivation learning non-linear features

Given a dictionary of simple non-linear functions:

Proposal 1: linear combination

Proposal 2: composition

Deep Learning and Convolutional Neural Networks | Page 9

Motivation learning non-linear features

Given a dictionary of simple non-linear functions:

Proposal 1: linear combination

Proposal 2: composition

Deep Learning and Convolutional Neural Networks | Page 10

Motivation learning non-linear features

Linear Combination

BAD: It may require

an exponential
number of
templates!!!
Deep Learning and Convolutional Neural Networks | Page 11

Motivation learning non-linear features

Composition

GOOD:
Re-use of
intermediate parts
Distributed
representations is
more efficient
Deep Learning and Convolutional Neural Networks | Page 12

Motivation learning non-linear features

Deep Learning and Convolutional Neural Networks | Page 13

Motivation deep learning in practice

Google Brain
- Big success on image & speech recognition

Microsoft
- Simultaneous interpretation system

Baidu
- Institute of Deep Learning

Deep Learning and Convolutional Neural Networks | Page 14

Motivation deep learning in practice

Deep Learning and Convolutional Neural Networks | Page 15

Outline
Motivation
Deep Learning
Convolutional Neural Networks
Applications

Deep Learning and Convolutional Neural Networks | Page 16

Deep Learning definition

What is deep learning?
It's a Convolutional Net.
It's a Contrastive Divergence.
It's a Feature Learning.
It's a Unsupervised Learning.

It's just old Neural Nets.

It's a Deep Belief Net.
Deep Learning and Convolutional Neural Networks | Page 17

Deep Learning definition

What is deep learning?
A Deep Learning method is: a method which makes
predictions by using a sequence of non-linear processing
stages. The resulting intermediate representations can be
interpreted as feature hierarchies and the whole system is
jointly learned from data.
Some deep learning methods are supervised, others are
unsupervised.
It's a large family!
Deep Learning and Convolutional Neural Networks | Page 18

Deep Learning types

Three types of deep architectures:
Feed-Forward: Multilayer Neural Nets, Convolutional Nets

Feed-Back: Sparse Coding, Deconvolutional Net

Bi-Directional: Deep Boltzmann Machines, Auto-Encoders

Deep Learning and Convolutional Neural Networks | Page 19

Deep Learning types

Three types of training protocols:
Purely Supervised
- Initialize parameters randomly
- Train in supervised mode
- Used in most practical systems for speech and image recognition

Unsupervised layer-wise + supervised classifier on top

- Train each layer unsupervised, one after the other
- Train a supervised classifier on top, keeping the other layers fixed
- Good when very few labeled samples are available

Unsupervised layer-wise + global supervised fine-tuning

- Train each layer unsupervised, one after the other
- Add a classifier layer, and retrain the whole thing supervised
- Good when label set is poor
Deep Learning and Convolutional Neural Networks | Page 20

Outline
Motivation
Deep Learning
Convolutional Neural Networks
Applications

Deep Learning and Convolutional Neural Networks | Page 30

Convolution Basic concert

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 32

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 33

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 34

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 35

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 36

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 37

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 38

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 39

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 40

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 41

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 42

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 43

CNNs typical architecture

Deep Learning and Convolutional Neural Networks | Page 44

CNNs typical architecture

Deep Learning and Convolutional Neural Networks | Page 45

CNNs conclusion
Connect each hidden unit to a small
patch of the input.
Share the weight across hidden units.
Subsampling layers are useful to reduce
computational burden and increase
invariance.

Deep Learning and Convolutional Neural Networks | Page 46

Outline
Motivation
Deep Learning
Convolutional Neural Networks
Applications

Deep Learning and Convolutional Neural Networks | Page 47

Applications
Scene recognition:

Deep Learning and Convolutional Neural Networks | Page 52

Applications
Localization:

Deep Learning and Convolutional Neural Networks | Page 52