Tema06 Deep Learning PDF

Deep Learning
Francisco Herrera
Research
R h Group
G on Soft
S ft Computing
C ti and
d
Information Intelligent Systems
(SCI2S)
http://sci2s.ugr.es
Dept. of Computer Science and A.I.
University of Granada, Spain
Email: herrera@decsai.ugr.es
Deep Learning
Deep Learning
D L i : El aprendizaje
di j profundo
f d es un
conjunto de algoritmos que intenta modelar
abstracciones de alto nivel en los datos mediante el
uso de arquitecturas compuestas de transformación
no lineales múltiples.
p
Bibliografía:
L. Deng and D. Yu.
Deep Learning methods and applications.
Foundations and Trends in Signal Processing
Vol. 7, Issues 3-4, 2014.
Nota: Deep Learning introduce el uso
de estructuras de aprendizaje que
requieren de arquitecturas de
procesamiento eficiente y distribuido
(
(GPU, Spark,
k …)) y muestra resultados
l d
importantes en el procesamiento de
imágenes, habla, lenguaje natural, ...
Deep Learning
Deep Learning (deep structure learning):

machine learning algorithms based on learning
multiple levels of representation/abstraction.
representation/abstraction
Amazing
a g improvements
p o e e ts in e error
o rate
ate in object
recognition, object detection, speech recognition, and
more recently, in natural language
processing/understading.
processing/understading
Yoshua Bengio
http://www iro umontreal ca/ bengioy/talks/DL Tutorial NIPS2015 pdf
http://www.iro.umontreal.ca/~bengioy/talks/DL-Tutorial-NIPS2015.pdf
Deep Learning
Al
Algunas d
definiciones
fi i i (D
(Deng and
d Yu,
Y 2014)
Deep Learning
Al
Algunas d
definiciones
fi i i (D
(Deng and
d Yu,
Y 2014)
Deep Learning
Al
Algunas d
definiciones
fi i i (D
(Deng and
d Yu,
Y 2014)
Deep Learning
Al
Algunas d
definiciones
fi i i (D
(Deng and
d Yu,
Y 2014)
Deep Learning
Al
Algunas d
definiciones
fi i i (D
(Deng and
d Yu,
Y 2014)
Deep Learning
(also called Hierarchical Learning)
Hierarchical Learning
• Natural progression from

low level to high level
structure as seen in natural
complexity
• Easier to monitor what is
being learnt and to guide
the machine to better
subspaces
• Usually best when input
space is locally structured –
spatial or temporal: images,
language, etc. vs arbitrary
input features
Deep Learning
Human information processing mechanisms (e.g., vision

and audition) suggest the need of deep architectures for
extracting
i complex
l structure and
db building
ildi iinternall
representation from rich sensory inputs.
Historically, the concept of deep learning originated

from artificial neural network research. (Hence, one
may occasionally hear the discussion of “new-
“new
generation neural networks.”) Feed-forward neural
networks or MLPs with many y hidden layers,
y , which are
often referred to as deep neural networks (DNNs), are
good examples of the models with a deep architecture.
Deep Learning
Machine learning: Shallow-structured arquitectures
 Gaussian
G i mixture
i t models
d l (GMM
(GMMs),
)
 Linear or nonlinear dynamical systems,
 Conditional, random fields (CRFs)
 Maximum entropy (MaxEnt) models,
 Support vector machines (SVMs)
 Logistic regression/kernel regression
 Multilayer perceptrons (MLPs) with a single hidden layer
including extreme learning machines (ELMs).
(ELMs)
These architectures typically contain at most

one or two layers of nonlinear feature
transformations.
Deep Learning
Traditional recognition approaches
Features are not learning

Deep Learning
“Shallow” vs. “deep” architectures

Deep Learning
Backpropagation
Credits: The Evolution of Neural Learning Systems: A Novel Architecture Combining the Strengths of NTs, CNNs,
and ELMs. N Martinel, C Micheloni… - IEEE SMC Magazine, 2015 - ieeexplore.ieee.org
Deep Learning
Backpropagation
• Minimize error of
calculated
l l d output
k • Adjust weights
• Gradient Descent
wjk • Procedure
• Forward Phase
j • Backpropagation
of errors
• For each sample,
p ,
multiple epochs
vij
i
Deep Learning
P bl
Problems with
ith Backpropagation
B k ti
 Multiple hidden Layers
 Get stuck in local optima

 start weights from random positions
 Slow convergence to optimum

 large training set needed
 Only use labeled data

 most data is unlabeled
Deep Learning
Deep Architecture (Train networks with many layers)
 Multiple hidden layers

 Motivation (why go deep?)
 Approximate
pp o ate co
complex
p e dec
decision
s o bou
boundary
da y
• Fewer computational units for
same functional mapping
 Hierarchical Learning
• Increasingly complex features
 Work well in different domains
• Vision, Audio, …
Deep Learning
Yoshua Bengio
Credits: http://www.iro.umontreal.ca/~bengioy/talks/DL-Tutorial-NIPS2015.pdf
Deep Learning
(Hierarchical Learning)
Hierarchical Learning/deep structure learning:

Automating Feature Discovery
From simplest features to

p
complex one
Fron unsupervised
learning to supervised
a g
learning
Deep Learning
Deep Architecture (Train networks with many layers)
 S
Some Models:
M d l
 Deep networks for unsupervised or generative learning:
deep belief network (DBN),
(DBN) stack of restricted Boltzmann
machines (RBMs), autoencoder …
 Deep networks for supervised learning: Deep Neural

Networks (DNN), Convolutional neural network (CNN). …
 Hybrid deep networks: DBN-DNN (when DBN is used to

initialize the training of a DNN, the resulting network is
sometimes
ti called
ll d the
th DBN–DNN)
DBN DNN)
Deep Learning
 Autoencoder
An autoencoder neural
network is an unsupervised
learning algorithm that
applies backpropagation,
setting the target values to
be equal to the inputs.
The aim of an autoencoder

is to learn a representation
(
(encoding)
di ) ffor a set off
data, typically for the
purpose of dimensionality
reduction.
Deep Learning
 Autoencoder (autoencoder de h2o h2o, una sola capa interna

de 3 neuronas y 1000 "epochs". En todos los autoencoders
uso la tangente hiperbólica como función de activación.
WDBC (569 instancias
i t i con 30 atributos
t ib t de
d entrada)
t d )
Credito: D. Charte
Deep Learning
Ejemplo: Diseño de un Clasificador para Iris

 Problema simple muy conocido: clasificación de lirios.
 Tres clases de lirios: setosa, versicolor y virginica.
 Cuatro atributos: longitud y anchura de pétalo y sépalo,
respectivamente.
 150 ejemplos,
j l 50 de d cadad clase.
l
 Disponible en
http://www.ics.uci.edu/~mlearn/MLRepository.html
Deep Learning
setosa, versicolor (C) y virginica

Deep Learning
setosa, versicolor y virginica
IRIS: Conjunto entrenamiento original
setosa versicolor virginica
1
0,9
0,8
0,7
alo
Anchura Péta
0,6
0,5
0,4
0,3
0,2
,
0,1
0
0 0,2 0,4 0,6 0,8 1
Longitud Pétalo
Deep Learning
 Autoencoder ((autoencoder de h2o,, salida de la capa

p intermedia
capas internas de [8, 5, 3, 5, 8] neuronas y 100 "epochs" (el
tridimensional), y [8, 5, 2, 5, 8] neuronas con 1000 "epochs" (el
bidimensional).
setosa, versicolor y virginica
Credito: D. Charte
Deep Learning
 Hybrid deep networks: DBN

DBN-DNN
DNN
Credits: L. Deng and D. Yu. Deep Learning methods and applications.

Foundations and Trends in Signal Processing. Vol. 7, Issues 3-4, 2014, pag. 246
Deep Learning
Convolutional Neural Networks (Supervised)
Each module consists of a convolutional layer and a pooling

layer.
Typically tries to compress large data (images) into a

smaller set of robust features, based on local variations.
Basic convolution can still create many features.
CNNs have been found highly effective and been commonly

used in computer vision and image recognition.
Deep Learning
Convolutional Neural Networks
C layers are
convolutions,
convolutions
S layers
pool/sample
Deep Learning

Deep Learning
Deep Learning
Deep Learning
De la academia a la industria: DNNresearch Inc y
Google Deepmind
Google Brain is a deep learning

research project at Google
En 2013, Google adquirió la compañía DNNresearch Inc

creada por uno de los pioneros de Deep Learning (Geoffrey
Hinton).
En enero de 2014 se hizo con el control de la ‘startup’

p
Deepmind Technologies una pequeña empresa londinense en la
trabajaban que algunos de los mayores expertos en ‘deep
learning .
learning’.
Deep Mind: Start up-2011

Demis
D i Hassabis
H bi Sh
Shane L
Legg y Mustafa
M t f
Suleyman
NIPS2012, un caso de éxito de CNN para el challenge
NIPS2012
ILSVRC 2010
ImageNet
g Classification with Deep p Convolutional Neural Networks
Part of: Advances in Neural Information Processing Systems 25 (NIPS 2012)
ImageNet
g is a dataset of over 15
million labeled high-resolution
images belonging to roughly 22,000
categories. The images were
collected from the web and labeled
by human labelers using Amazon’s
Mechanical Turk crowd-sourcing
tool. Starting in 2010, as part of
the Pascal Visual Object Challenge,
an annual competition called the
ImageNet Large-Scale Visual
Recognition Challenge (ILSVRC) has
been held. ILSVRC uses a subset of
ImageNet with roughly 1000
images in each of 1000 categories.
In all, there are roughly 1.2 million
training images, 50,000 validation
images, and 150,000 testing
images.
Deep Learning
Retos en los Juegos “inteligentes”
http://arxiv.org/abs/1312.5602
Deep Learning
Juegos Arcade (Breakout)
http://elpais.com/elpais/2015/02/25/ciencia/1424860455_667336.html
http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html
Deep Learning
Schematic illustration of the convolutional neural network.
V Mnih et al.
al Nature 518,
518 529-533 (2015)
doi:10.1038/nature14236
Deep Learning
Deep Learning
http://arxiv.org/abs/1509.01549 https://chessprogramming.wikispaces.com/Giraffe
Deep Learning
https://www.technologyreview.com/s/541276/deep-learning-machine-
https://www.technologyreview.com/s/541276/deep learning machine
teaches-itself-chess-in-72-hours-plays-at-international-master/
Ref: arxiv.org/abs/1509.01549 :
Giraffe: Using Deep Reinforcement Learning to Play Chess
Algunos datos:
von Neumann introduced the minimax algorithm in 1928
363 features
The evaluator network converges in about 72 hours on a machine with 2x10-core Intel Xeon
E5-2660v2 CPU.
Giraffe is able to play at the level of an FIDE International Master
Deep Learning
http://www.nature.com/nature/journal/v529/n7587/full/
nature16961.html
Deep Learning
http://elpais.com/elpais/2016/01/26/ciencia/1453766578 683799.html
http://elpais.com/elpais/2016/01/26/ciencia/1453766578_683799.html
Deep Learning
Each simulation traverses the tree by

y selecting
g the
edge with maximum action value Q, plus a bonus u(P)
that depends on a stored prior probability P for that
edge. b, The leaf node may be expanded; the new node
is processed once by t…
t
nature16961.html
Deep Learning
Neural network training pipeline and architecture
D Silver et al. Nature 529, 484–489 (2016)

doi:10.1038/nature16961
nature16961.html
Deep Learning
How AlphaGo (black, to play) selected its move in an informal

game against Fan Hui
D Silver et al. Nature 529, 484–489 (2016)

doi:10.1038/nature16961
nature16961.html
Deep Learning
http://www.nature.com/news/google-ai-algorithm-
masters-ancient-game-of-go-1.19234
http://www.nature.com/news/the-
g
go-files-champion-preps-for-1-
p p p
million-machine-match-1.19541
https://actualidad.rt.com/ciencias/201602-inteligencia-
artificial-alphago-google-gana-leyenda-go
9/03/2016
Deep Learning
https://gogameguru.com/tag/deepmind-alphago-lee-sedol/
Deep Learning
https://gogameguru.com/alphago-defeats-lee-sedol-4-1/
IMAGENET (ILSRVC): Microsoft Wins ImageNet
Using Extremely Deep Neural Networks
Microsoft's
Mi ft' network
t k was really
ll deep
d
at 150 layers (extremely deep
neural network). To do this the
team had to overcome a
fundamental problem inherent in
training deep neural networks. As
the network gets deeper training
becomes more difficult so you
encounter a seemingly paradoxical
situation that adding layers makes
the performance worse.
The solution proposed is
called deep p residual learning.g
http://www.image-
net org/challenges/LSVRC/
net.org/challenges/LSVRC/
http://www.i-programmer.info/news/105-artificial-intelligence/9266-microsoft-wins-imagenet-
using-extremely-deep-neural-networks.html
IMAGENET (ILSRVC 2015): Microsoft Wins
ImageNet
g Using
g Extremely
y Deep
p Neural
Networks
Deep Learning
Retos en la “pintura”
Deep Learning
Credits: http://arxiv.org/abs/1508.06576
Deep Learning
http://www.deepart.io/
Deep Learning
Ejemplos del resultado de DeepART
van Goth
http://www.deepart.io/
Deep Learning
Modelo de CNN utilizado y descripción de la metodología
Deep Learning
Digit Recognizer Kaggle
Caso estudio: Digit Recognizer Kaggle (A. Herrera-Poyatos)
Andrés Herrera Poyatos

Repositorio en GitHub con el código:
https://github.com/andreshp/Kaggle
Deep Learning
Caso estudio: Digit Recognizer Kaggle
 Desarrollar un reconocedor de dígitos es uno de

los problemas clásicos de la ciencia de datos.
 Sirve de benchmark para probar los nuevos
algoritmos. ¡Ningún un humano acierta el 100%!
 Aplicación
li ió práctica:
á i d
detección
ó de
d matrículas,
í l
conversión de escritura a mano en texto …
9 6 6 6 4 0 7 4 0 1
3 1 3 4 7 2 7 1 2 1
1 7 4 2 3 5 1 2 4 4
Créditos: A. Herrera-Poyatos
Deep Learning
 Kaggle mantiene una competición pública:
http://www.kaggle.com/c/digit-recognizer
 Datos a analizar: MNIST DATA (60.000 instances)

http://yann.lecun.com/exdb/mnist/
p //y / / /
 Rodrigo Benenson has compiled an informative

summary y page
p g
http://rodrigob.github.io/are_we_there_yet/build/classificatio
n_datasets_results.html
Deep Learning
 Data Set:
 Training Set: 42.000 Imágenes
 Test Set: 28.000 Imágenes
á
 Imagen:
 10 clases: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
 28x28 píxeles
 Ej
Ejemplo:
l
 Puntuación para la clasificación general en

Kaggle: índice de acierto sobre un 25%
del Test Set.
Deep Learning
1 Primer paso
1.
1. Utilizar los algoritmos

g más conocidos para
p usarlos
como benchmark
• KNN con k = 10 0.96557 en Kaggle
• Random
d Forest con 1000 árboles
á b l 0.96829 en Kaggle
l
2
2. Optimizar los parámetros de un algoritmo sencillo
• Cross Validation sobre KNN para encontrar el mejor valor de k.
Solución: K=1 0.97114 en Kaggle
Deep Learning
2 Visualización
2.
 Media de todas las imágenes del training set por
clases:
 Observación: Incluso las medias no están

centradas (ver 6 y 7). Esto provoca problemas
para clasificarlas
l f l correctamente.
Solución: Preprocesamiento
Deep Learning
3 Preprocesamiento
3.
 Idea: Eliminar las filas y columnas de píxeles en
bl
blanco.
 Problema: Las nuevas imágenes tienen

dif
diferentes
t didimensiones.
i
Deep Learning
3 Preprocesamiento
3.
 Solución: Redimensionar las imágenes a 20x20
píxeles
p e es (t
(tras
as el
e proceso
p oceso anterior
a te o laa imagen
age más
ás
grande tiene esa dimensión)
 Media de las imágenes

á del training set preprocesadas:
 ¡Todas están centradas!

 KNN con kk=11 sobre
b los
l datos
d t preprocesados
d
0.97557 en Kaggle
Deep Learning
Lib
Librería
í que contiene
ti Deep
D L
Learning:
i H2O
http://0xdata.com/
Récord
é del mundo en el problema MNIST sin preprocesamiento
http://0xdata.com/blog/2015/02/deep-learning-
performance/
f /
 Soporte para R, Python, Hadoop y Spark

 Se puede instalar en cualquier máquina,
á
incluyendo un portatil, cluster de ordenadores, …
 Funcionamiento: Crea una máquina
virtual con Java en la que optimiza el
paralelismo de los algoritmos.
Deep Learning
http://cran.r project.org/web/packages/h2o/index.html
http://cran.r-project.org/web/packages/h2o/index.html
Deep Learning
http://www.h2o.ai/resources/
Deep Neural Network (DNN), includes:

-The
The default initialization scheme is the uniform
adaptive option, which is an optimized
initialization based on the size of the network.
-H2O’s Deep Learning framework supports
regularization
l i i techniques
h i to prevent overfitting
fi i
(among them, dropout (Hinton et al., 2012)).
- It uses the implemented adaptive learning rate
algorithm ADADELTA (Zeiler, 2012)
automatically combines the benefits of learning
rate annealing and momentum training to avoid
slow convergence.
- It utilizes
tili HOGWILD!,
HOGWILD! the
th recently
tl developed
d l d
lock-free parallelization scheme (Niu et al,
2011).
Deep Learning
http://www.h2o.ai/resources/
Machine Learning with Sparkling Water:

H2O + Spark
Sparkling
S kli Water
W t allows
ll users to
t
combine the fast, scalable machine
learning algorithms of H2O with the
capabilities of Spark.
Spark With
Sparkling Water, users can drive
computation from Scala/R/Python
and utilize the H2O Flow UI,
UI
providing an ideal machine learning
platform for application developers.
Deep Learning
4 Deep Learning sobre MNIST DATA preprocesados

4.

https://github.com/andreshp/Kaggle Créditos: A. Herrera-Poyatos
Deep Learning

4.
hidden=C(1024,1024,2048)

https://github.com/andreshp/Kaggle Créditos: A. Herrera-Poyatos
Deep Learning

4.
 Tiempo de Ejecución: 2.5 horas de cómputo con un
Procesador Intel i5 a 2.5
2 5 GHz.
GHz
 Resultados conseguidos:
 Deep p Learningg 0.98229 en Kaggle
gg
 Preprocesamiento + Deep Learning 0.98729 en
Kaggle
 ¡El primer
i resultado
lt d era 0.96557!
0 96557!
Deep Learning
Digit Recognizer and Convolutional NN
A complete description
http://neuralnetworksanddeeplearning.com/chap6.html
By Michael Nielsen /
Jan 2016
Deep Learning
C
Convolutional
l ti l neurall networks
t k use th
three b
basic
i ideas:
id l
local
l
receptive fields, shared weights, and pooling. Let's look at each of these
ideas in turn.
Local receptive fields: To be more precise, each neuron in the first hidden
layer will be connected to a small region of the input neurons, say, for
example, a 5×5 region, corresponding to 25 input pixels. So, for a
particular hidden neuron
neuron, we might have connections that look like this:
Deep Learning
L
Local
l receptive
ti fi ld
fields:
24×24 neurons
28×28 input image

Deep Learning
Sh
Shared
d weights
i ht and
d biases:
bi the same weights and bias
for each of the 24×24 hidden neurons (sigmoide function)
The map from the input layer to the hidden layer a feature map.
In the example shown, there are 3 feature maps.

If we have 20 feature maps that
that's
s a total of 20×26=520 parameters
defining the convolutional layer. By comparison, suppose we had a fully
connected first layer, with 784=28×28 input neurons, 30 hidden neurons,
23,550 parameters.
Deep Learning
The 20 images correspond to 20 different feature maps

Deep Learning
P li
Pooling l
layers: the pooling layers do is simplify the
information in the output from the convolutional layer, one
common procedure for pooling is known as max-pooling, in the
2x2 region input.
Deep Learning
R l experiment
Real i t for
f DIGIT
DIGIT:
Different results, and preprocessing are analyzed in the

chapter Expanding the training data,
chapter. data to displace each
training image by a single pixel, either up one pixel, down one
pixel, left one pixel, or right one pixel.
Deep Learning
Final experiment for DIGIT (ensemble

( bl with
ith different
diff t
configurations): 99.67 percent accuracy, 33 of the 10,000 test
images. The label in the top right is the correct classification,
according to the MNIST data,
data while in the bottom right is the label
output by our ensemble of nets:
Deep Learning
On the left, the raw input digits. On the right, graphical representations of the
l
learned
d features.
f t In
I essence, the
th network
t k learns
l to
t “see”
“ ” lines
li and
d loops.
l
Credits: https://www.datarobot.com/blog/a-primer-on-deep-learning/
Deep Learning
Deep Learning
Credits: L. Deng and D. Yu. Deep Learning methods and applications.

Foundations and Trends in Signal Processing. Vol. 7, Issues 3-4, 2014, pag. 325
Deep Learning
Digit Recognizer
http://yann lecun com/exdb/mnist/

Deep Learning
Digit Recognizer
http://yann lecun com/exdb/mnist/

Deep Learning
Digit Recognizer
http://rodrigob.github.io/are_we_there_yet/build/classification_
http://rodrigob.github.io/are we there yet/build/classification
datasets_results.html
Deep Learning
Digit Recognizer
Deep Learning
Digit Recognizer
http://rodrigob.github.io/are_we_there_yet/build/classification_
http://rodrigob.github.io/are we there yet/build/classification
datasets_results.html
Deep Learning
Librerías de Deep Learning
htt //
http://www.teglor.com/b/deep-learning-libraries-language-cm569/
t l /b/d l i lib i l 569/
Python
Matlab
C++
R
Ja a …
Java,
Deep Learning

http://www.teglor.com/b/deep learning libraries language cm569/
http://www.teglor.com/b/deep-learning-libraries-language-cm569/
Caffe is a deep learning framework made with expression, speed, and modularity in
mind. It is developed by the Berkeley Vision and Learning Center (BVLC) and by
community contributors. Google's DeepDream is based on Caffe Framework. This
framework is a BSD
BSD-licensed
licensed C++ library with Python Interface.
Interface
Lasagne is a lightweight library to build and train neural networks in Theano. It is
governed by simplicity, transparency, modularity, pragmatism , focus and
restraint principles.
nolearn contains a number of wrappers and abstractions around existing neural
network libraries, most notably Lasagne, along with a few machine learning utility
modules.
Deeplearning4j is the first commercial-grade, open-source, distributed deep-
learning library written for Java and Scala. It is designed to be used in business
environments, rather than as a research tool.
Deep Learning
http://caffe.berkeleyvision.org/
Caffe is a deep learning framework made with

expression, speed, and modularity in mind.
It is developed by the Berkeley Vision and Learning

Center (BVLC) and by community contributors.
Google's DeepDream is based on Caffe Framework.

This framework is a BSD-licensed C++ library with
P th
Python Interface.
I t f
SparkNet
https://github.com/amplab/SparkNet
Deep Learning
http://www.kdnuggets.com/2015/12/spark-deep-learning-training-with-
http://www.kdnuggets.com/2015/12/spark deep learning training with
sparknet.html By Matthew Mayo, KDnuggets.
https://github.com/amplab/SparkNet
Deep Learning
https://pypi.python.org/pypi/Theano
http://deeplearning.net/software/theano/
NN Lasagne http://lasagne.readthedocs.org/en/latest/index.html
nolearn - Web: https://pythonhosted.org/nolearn/

https://pythonhosted org/nolearn/
Including: DBN y CNN.
Deep Learning
Tensor Flow
https://www.tensorflow.org/
TensorFlow™
T Fl ™ iis an open source software
ft library
lib
for numerical computation using data flow
graphs.
The flexible architecture allows you to deploy

computation to one or more CPUs or GPUs in a
desktop server
desktop, server, or mobile device with a single API.
API
TensorFlow was originally developed by researchers

and engineers working on the Google Brain Team
within Google's Machine Intelligence research
organization for the purposes of conducting machine
learning and deep neural networks research,
research but the
system is general enough to be applicable in a wide
variety of other domains as well.
Deep Learning
Tensor Flow
https://www.tensorflow.org/
Scikit Flow: Easy

y Deep
p Learning
g
with TensorFlow and Scikit-learn
https://github.com/tensorflow/skflow Deep Neural Network

Convolutional NN
http://www.kdnuggets.com/2016/02/scikit-flow-easy-deep-learning-tensorflow-scikit-
learn.html
Deep Learning
http://deeplearning4j.org/
Deep Learning

Deep Learning
http://spark-packages.org/user/deeplearning4j
Deep Learning

Deep Learning
deepnet implements some deep learning architectures and neural network

algorithms, including BP,RBM,DBN,Deep autoencoder and so on.
Deep Learning

Deep Learning
In summary
darch: Package for Deep Architectures and Restricted

Boltzmann Machines
deepnet: deep learning toolkit in R
autoencoder: Sparse Autoencoder for Automatic Learning

of Representative Features from Unlabeled Data
Deep Learning
Relevant researchers
The Fathers of Deep Learning

Credits: https://www.datarobot.com/blog/a-primer-on-deep-learning/
Deep Learning
Credits: http://www.slideshare.net/david.kh/promises-of-deep-learning
Deep Learning
Páginas de los 3 investigadores de referencia:
https://www.cs.toronto.edu/~hinton/
(Geoffrey E. Hinton)
University of Toronto
Google Lab - Toronto
http://www.iro.umontreal.ca/~bengioy/yoshua_en/index.
html
(Yosua Bengio)
Université
é de
d Montréal
é l
http://yann.lecun.com/
(Yann LeCun)
Director of AI Research, Facebook
Deep Learning
Final Comments: Overview sobre las estructuras
de representación para aprendizaje y deep learning
En este artículo de review, Bengio y coautores hacen una revisión muy

interesante sobre la representación para el aprendizaje de características,
fundamental para entender deep learning, analizando el estado del arte y
las perspectivas futuras.
Deep Learning
Readings: Recent Overview
Deep Learning
Deep learning, Yann LeCun, Yoshua Bengio & Geoffrey Hinton
Abstract
Deepp learningg allows computational
p models that are composed
p of
multiple processing layers to learn representations of data with
multiple levels of abstraction. These methods have dramatically
improved the state-of-the-art in speech recognition, visual object
recognition, object detection and many other domains such as
drug discovery and genomics. Deep learning discovers intricate
structure in large data sets by using the backpropagation
algorithm to indicate how a machine should change its internal
parameters that are used to compute the representation in each
layer from the representation in the previous layer. Deep
convolutional
l ti l nets
t have
h brought
b ht about
b t breakthroughs
b kth h in
i
processing images, video, speech and audio, whereas recurrent
nets have shone light on sequential data such as text and speech.
Deep Learning
Deep learning, Yann LeCun, Yoshua Bengio & Geoffrey Hinton
Convolutional neural networks
Deep Learning
Final Comments: The future of of deep learning (by
LeCun, Bengio and Hinton)
Unsupervised learning91, 92, 93, 94, 95, 96, 97, 98 had a catalytic effect in reviving interest
in deep learning, but has since been overshadowed by the successes of purely
supervised learning. Although we have not focused on it in this Review, we expect
unsupervised learning to become far more important in the longer term. Human and
animal learning is largely unsupervised: we discover the structure of the world by
observing it, not by being told the name of every object.
Human vision is an active process that sequentially samples the optic array in an
intelligent, task-specific way using a small, high-resolution fovea with a large, low-
resolution surround. We expect much of the future progress in vision to come from
systems that are trained end-to-end and combine ConvNets with RNNs that use
reinforcement learning to decide where to look. Systems combining deep learning
and reinforcement learning are in their infancy, but they already outperform passive
vision systems
y 99 at classification tasks and p produce impressive
p results in learningg to
play many different video games . 100
Natural language understanding is another area in which deep learning is poised to

make a large impact over the next few years. We expect systems that use RNNs to
understand sentences or whole documents will become much better when they y
learn strategies for selectively attending to one part at a time76, 86.
Ultimately, major progress in artificial intelligence will come about through systems
that combine representation learning with complex reasoning. Although deep
learningg and simple
p reasoning g have been used for speech p and handwriting g
recognition for a long time, new paradigms are needed to replace rule-based
manipulation of symbolic expressions by operations on large vectors101.
Deep Learning
Final Comments
En el enlace a Deep Learning de la Wikipedia se hace un rápido recorrido

sobre Deep learning, los diferentes modelos de redes neuronales
asociados, así como algunos de los campos actuales de aplicación.
https://en wikipedia org/wiki/Deep learning
https://en.wikipedia.org/wiki/Deep_learning
Existe una gran variedad de arquitecturas Deep Neural Network

Deep Learning
Final Comments
En el enlace a Deep Learning de la Wikipedia se hace un rápido recorrido

sobre Deep learning, los diferentes modelos de redes neuronales
asociados, así como algunos de los campos actuales de aplicación.
https://en wikipedia org/wiki/Deep learning
https://en.wikipedia.org/wiki/Deep_learning

Tema06 Deep Learning PDF

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Tema06 Deep Learning PDF

Încărcat de

Drepturi de autor:

Formate disponibile

Deep Learning

Deep Learning (deep structure learning):

• Natural progression from

Human information processing mechanisms (e.g., vision

Historically, the concept of deep learning originated

Machine learning: Shallow-structured arquitectures

These architectures typically contain at most

Traditional recognition approaches

Features are not learning

“Shallow” vs. “deep” architectures

 Multiple hidden Layers

 Get stuck in local optima

 Slow convergence to optimum

 Only use labeled data

Deep Architecture (Train networks with many layers)

 Multiple hidden layers

Hierarchical Learning/deep structure learning:

From simplest features to

Deep Architecture (Train networks with many layers)

 Deep networks for supervised learning: Deep Neural

 Hybrid deep networks: DBN-DNN (when DBN is used to

The aim of an autoencoder

 Autoencoder (autoencoder de h2o h2o, una sola capa interna

Ejemplo: Diseño de un Clasificador para Iris

setosa, versicolor (C) y virginica

setosa, versicolor y virginica

IRIS: Conjunto entrenamiento original

setosa versicolor virginica

 Autoencoder ((autoencoder de h2o,, salida de la capa

setosa, versicolor y virginica

 Hybrid deep networks: DBN

Credits: L. Deng and D. Yu. Deep Learning methods and applications.

Convolutional Neural Networks (Supervised)

Each module consists of a convolutional layer and a pooling

Typically tries to compress large data (images) into a

Basic convolution can still create many features.

CNNs have been found highly effective and been commonly

Convolutional Neural Networks

Convolutional Neural Networks

Google Brain is a deep learning

En 2013, Google adquirió la compañía DNNresearch Inc

En enero de 2014 se hizo con el control de la ‘startup’

Deep Mind: Start up-2011

Juegos Arcade (Breakout)

Each simulation traverses the tree by

Neural network training pipeline and architecture

D Silver et al. Nature 529, 484–489 (2016)

How AlphaGo (black, to play) selected its move in an informal

D Silver et al. Nature 529, 484–489 (2016)

Modelo de CNN utilizado y descripción de la metodología

Caso estudio: Digit Recognizer Kaggle (A. Herrera-Poyatos)

Andrés Herrera Poyatos

Caso estudio: Digit Recognizer Kaggle

 Desarrollar un reconocedor de dígitos es uno de

Caso estudio: Digit Recognizer Kaggle

 Kaggle mantiene una competición pública:

 Datos a analizar: MNIST DATA (60.000 instances)

 Rodrigo Benenson has compiled an informative

Caso estudio: Digit Recognizer Kaggle

 Puntuación para la clasificación general en

Caso estudio: Digit Recognizer Kaggle

1. Utilizar los algoritmos

Caso estudio: Digit Recognizer Kaggle