introduction to deep learning -...
TRANSCRIPT
Big Data Machine Learning Deep Learning
Introduction to Deep Learning
Joana Frontera-PonsGrigorios Tsagkatakis
Dictionary Learning on Manifolds workshopNice, September 2017
1
Big Data Machine Learning Deep Learning
Big Data
The 5VsVolume
2
Big Data Machine Learning Deep Learning
Big Data
The 5VsVolume
Velocity
3
Big Data Machine Learning Deep Learning
Big Data
The 5VsVolume
Velocity
Variety
4
Big Data Machine Learning Deep Learning
Big Data
The 5VsVolume
Velocity
Variety
Veracity
5
Big Data Machine Learning Deep Learning
Big Data
The 5VsVolume
Velocity
Variety
Veracity
Value
6
Big Data Machine Learning Deep Learning
Big Data & Data Driven Science
Astronomy & Astrophysics
Sky Survey Project Volume Velocity Variety
Sloan Digital Sky Survey (SDSS) 50 TB 200 GB per day
Images, redshifts
Large Synoptic Survey Telescope (LSST )
~ 200 PB 10 TB per day Images, catalogs
Square Kilometer Array (SKA ) ~ 4.6 EB 150 TB per day Images, redshifts
Astrophysics and Big Data: Challenges, Methods, and Tools. Mauro Garofalo, Alessio Botta, and Giorgio Ventre.
7
Big Data Machine Learning Deep Learning
Handling Big Data
• Machine Learning + Big Data -> Data science
8
Big Data Machine Learning Deep Learning
Types of Machine Learning
Supervised learning: present example inputs and theirdesired outputs (labels) → learn a general rule that mapsinputs to outputs.
9
Big Data Machine Learning Deep Learning
Types of Machine Learning
Unsupervised learning: no labels are given → find structure ininput.
10
Big Data Machine Learning Deep Learning
Types of Machine Learning
Reinforcement learning: system interacts with environmentand must perform a certain goal without explicitly telling itwhether it has come close to its goal or not.
11
Big Data Machine Learning Deep Learning
Feature extraction in ML
Image Low-level
vision features
(edges, SIFT, HOG, etc.)
Object detection
/ classification
Input data (pixels)
LearningAlgorithm(e.g., SVM)
feature representation (hand-crafted)
Features are not learned
12
Big Data Machine Learning Deep Learning
Pixel 2
Pix
el 1
No CarCar
Pixel 1
Pixel 2
Learning Algorithm
Feature learning
13
Big Data Machine Learning Deep Learning
Pixel 2
Pix
el 1
No CarCar
Feature learning
14
Feature Representation
Feature 2Fe
atu
re 1
Pixel 1
Pixel 2
Learning Algorithm
Big Data Machine Learning Deep Learning
SIFT Spin image
Textons
Computer vision features
15
SURF, MSER, LBP, Color-SIFT, Color histogram, …..
HoG
Big Data Machine Learning Deep Learning
Beyond SotA
Limitations
Hand-crafting
Case-specific
Single source data
Deep Learning
Learning-based
Generic
Multi-source
16
Feature representation
1st layer “Edges”
2nd layer “Object parts”
3rd layer “Objects”
Pixels
Big Data Machine Learning Deep Learning
Biological Motivation
• 86Billion neurons
• 1014-1015 synapses
17
Big Data Machine Learning Deep Learning
Key components
Layers (input/hidden/output)
Weights
Activations
18
Big Data Machine Learning Deep Learning
Why today
• Big Data
• Processing recourses (GPU/Cloud)
• Advances in optimization
19
Big Data Machine Learning Deep Learning
Brief history of DL
20
Big Data Machine Learning Deep Learning
CIFAR 10 and CNN
21
Big Data Machine Learning Deep Learning
Since then…
22
Big Data Machine Learning Deep Learning
• T
• Support Vector Machine
• Logistic Regression
• Convolutional Neural Net
• Recurrent Neural Net
• Denoising Autoencoder
• Restricted Boltzmannmachines
• Sparse coding
• Stacked Denoising Autoencoder
• Deep Boltzmann machines
• Hierarchical Sparse Coding
DeepShallow
Supervised
Unsupervised
Taxonomy of learning methods
23
Big Data Machine Learning Deep Learning
Training & Testing (inference)
• Training: determine weights• Supervised: labeled training examples
• Unsupervised: no labels available
• Reinforcement: examples associated with rewards
• Inference: apply weights to new examples
24
Big Data Machine Learning Deep Learning
Training DNN
1. Get batch of data
2. Forward through the network -> estimate loss
3. Backpropagate error
4. Update weights based on gradient
Errors
25
Big Data Machine Learning Deep Learning
Gradient Descent
Minimize function J w.r.t. parameters θ
Stochastic Gradient Descent (SGD)
Extensions: momentum, Nesterov, Adagrad
𝜃 = 𝜃 − 𝑛𝛻𝜃J(θ)
𝜃 = 𝜃 − 𝑛𝛻𝜃J(θ; 𝑥𝑖 , 𝑦 𝑖 )
26
Learning rate
Big Data Machine Learning Deep Learning
Visualization
27
Big Data Machine Learning Deep Learning
Characteristics
28
Under-fitting
Over-fitting
Big Data Machine Learning Deep Learning 29
Sasen Cain (@spectralradius)