introduction to deep learning -...

Big Data Machine Learning Deep Learning

Introduction to Deep Learning

Joana Frontera-PonsGrigorios Tsagkatakis

Dictionary Learning on Manifolds workshopNice, September 2017

1

Big Data Machine Learning Deep Learning

Big Data

The 5VsVolume

2

Big Data Machine Learning Deep Learning

Big Data

The 5VsVolume

Velocity

3

Big Data Machine Learning Deep Learning

Big Data

The 5VsVolume

Velocity

Variety

4

Big Data Machine Learning Deep Learning

Big Data

The 5VsVolume

Velocity

Variety

Veracity

5

Big Data Machine Learning Deep Learning

Big Data

The 5VsVolume

Velocity

Variety

Veracity

Value

6

Big Data Machine Learning Deep Learning

Big Data & Data Driven Science

Astronomy & Astrophysics

Sky Survey Project Volume Velocity Variety

Sloan Digital Sky Survey (SDSS) 50 TB 200 GB per day

Images, redshifts

Large Synoptic Survey Telescope (LSST )

~ 200 PB 10 TB per day Images, catalogs

Square Kilometer Array (SKA ) ~ 4.6 EB 150 TB per day Images, redshifts

Astrophysics and Big Data: Challenges, Methods, and Tools. Mauro Garofalo, Alessio Botta, and Giorgio Ventre.

7

Big Data Machine Learning Deep Learning

Handling Big Data

• Machine Learning + Big Data -> Data science

8

Big Data Machine Learning Deep Learning

Types of Machine Learning

Supervised learning: present example inputs and theirdesired outputs (labels) → learn a general rule that mapsinputs to outputs.

9

Big Data Machine Learning Deep Learning

Types of Machine Learning

Unsupervised learning: no labels are given → find structure ininput.

10

Big Data Machine Learning Deep Learning

Types of Machine Learning

Reinforcement learning: system interacts with environmentand must perform a certain goal without explicitly telling itwhether it has come close to its goal or not.

11

Big Data Machine Learning Deep Learning

Feature extraction in ML

Image Low-level

vision features

(edges, SIFT, HOG, etc.)

Object detection

/ classification

Input data (pixels)

LearningAlgorithm(e.g., SVM)

feature representation (hand-crafted)

Features are not learned

12

Big Data Machine Learning Deep Learning

Pixel 2

Pix

el 1

No CarCar

Pixel 1

Pixel 2

Learning Algorithm

Feature learning

13

Big Data Machine Learning Deep Learning

Pixel 2

Pix

el 1

No CarCar

Feature learning

14

Feature Representation

Feature 2Fe

atu

re 1

Pixel 1

Pixel 2

Learning Algorithm

Big Data Machine Learning Deep Learning

SIFT Spin image

Textons

Computer vision features

15

SURF, MSER, LBP, Color-SIFT, Color histogram, …..

HoG

Big Data Machine Learning Deep Learning

Beyond SotA

Limitations

Hand-crafting

Case-specific

Single source data

Deep Learning

Learning-based

Generic

Multi-source

16

Feature representation

1st layer “Edges”

2nd layer “Object parts”

3rd layer “Objects”

Pixels

Big Data Machine Learning Deep Learning

Biological Motivation

• 86Billion neurons

• 1014-1015 synapses

17

Big Data Machine Learning Deep Learning

Key components

Layers (input/hidden/output)

Weights

Activations

18

Big Data Machine Learning Deep Learning

Why today

• Big Data

• Processing recourses (GPU/Cloud)

• Advances in optimization

19

Big Data Machine Learning Deep Learning

Brief history of DL

20

Big Data Machine Learning Deep Learning

CIFAR 10 and CNN

21

Big Data Machine Learning Deep Learning

Since then…

22

Big Data Machine Learning Deep Learning

• T

• Support Vector Machine

• Logistic Regression

• Convolutional Neural Net

• Recurrent Neural Net

• Denoising Autoencoder

• Restricted Boltzmannmachines

• Sparse coding

• Stacked Denoising Autoencoder

• Deep Boltzmann machines

• Hierarchical Sparse Coding

DeepShallow

Supervised

Unsupervised

Taxonomy of learning methods

23

Big Data Machine Learning Deep Learning

Training & Testing (inference)

• Training: determine weights• Supervised: labeled training examples

• Unsupervised: no labels available

• Reinforcement: examples associated with rewards

• Inference: apply weights to new examples

24

Big Data Machine Learning Deep Learning

Training DNN

1. Get batch of data

2. Forward through the network -> estimate loss

3. Backpropagate error

4. Update weights based on gradient

Errors

25

Big Data Machine Learning Deep Learning

Gradient Descent

Minimize function J w.r.t. parameters θ

Stochastic Gradient Descent (SGD)

Extensions: momentum, Nesterov, Adagrad

𝜃 = 𝜃 − 𝑛𝛻𝜃J(θ)

𝜃 = 𝜃 − 𝑛𝛻𝜃J(θ; 𝑥𝑖 , 𝑦 𝑖 )

26

Learning rate

Big Data Machine Learning Deep Learning

Visualization

27

Big Data Machine Learning Deep Learning

Characteristics

28

Under-fitting

Over-fitting

Big Data Machine Learning Deep Learning 29

Sasen Cain (@spectralradius)