introduction to deep learning
TRANSCRIPT
![Page 1: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/1.jpg)
Introduc)on to Deep Learning
Massimiliano Ruocco
![Page 2: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/2.jpg)
Outline
• Introduction and Motivation for DL • From NN to Deep Learning • Deep Learning Models • Deep Learning in the Real World • Conclusion
![Page 3: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/3.jpg)
Introduction and Motivation for DL
![Page 4: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/4.jpg)
Introduction Deep Learning - WHAT
Class of ML training algorithm
![Page 5: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/5.jpg)
Introduction Deep Learning - Motivations
• ML Algorithms: – Supervised – Unsupervised – Semi-supervised – Reinforcement Learning
![Page 6: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/6.jpg)
• ML Algorithms: unsupervised learning
Data Representa)on
Input Clustering Output
Example (Marketing/Customer segmentation): • Input : Customers of a specific product • Output: Customer subgroups
Introduction Deep Learning - Motivations
![Page 7: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/7.jpg)
• ML Algorithms: supervised learning
Data Representa)on
Input Classifica)on/ Regression
Output
Training Labeled DataSet
Data Representa)on
Example (spam detection): • Input : Email • Output: Spam/NotSpam • Training Set: Data set of mail labeled as Spam/Not Spam
Introduction Deep Learning in ML and AI
![Page 8: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/8.jpg)
• ML Algorithms: supervised learning
Data Representa)on
Input Classifica)on/ Regression
Output
Training Labeled DataSet
Data Representa)on
Example (spam detection): • Input : Email • Output: Spam/NotSpam • Training Set: Data set of mail labeled as Spam/Not Spam
Introduction Deep Learning – Representation Problem
![Page 9: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/9.jpg)
• Data Representation: – feature set selection – #features
• Main Issues: – Course of dimensionality – Overfitting – Handcrafted features
• How to tackle: Representation Learning
Introduction Deep Learning – Representation Problem
![Page 10: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/10.jpg)
• Deep learning methods: – Representations are expressed in terms of other, simpler representations
Introduction Deep Learning - WHAT
![Page 11: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/11.jpg)
• Deep Learning algorithm as application of Machine Learning to Artificial intelligence
Ar#ficial Intelligence (i.e. knowledge bases)
Machine Learning (i.e. Support Vector Machine)
Representa#on Learning (i.e. Autoencoders)
Deep Learning (i.e. Mul=layer Perceptron)
Introduction Deep Learning in ML and AI
![Page 12: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/12.jpg)
From Neural Network to Deep Learning
![Page 13: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/13.jpg)
• Neural Network: Basic – Different layers of neurons/perceptrons – Human brain analysis – Input, Hidden Layer, Output
• Neural Network: Applications – Classification (Spam Detection) – Pattern Recognition (Character recognition)
Introduction From Neural Network to Deep Learning
![Page 14: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/14.jpg)
• The core: Neuron
Introduction From Neural Network to Deep Learning
W1
W2
W3
x1
x2
xn Sigmoid func)on
1/(1+e-‐z)
Output hw(x)
x = [x0…xn]T w = [w0…wn]T z = wTx
![Page 15: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/15.jpg)
• Neural Network – Single layer
Introduction From Neural Network to Deep Learning
![Page 16: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/16.jpg)
• Forward Propagation: – process of computing the output
Introduction From Neural Network to Deep Learning
x1
x2
x3
a12
a22
W(1)
W(2)
a(2) z(2)
z(3)
X
z(2) = XW(1) a(2) = f(z(2)) z(3) = a(2)W(2) y = f(z(3))
![Page 17: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/17.jpg)
• Training a Neural Network: – Learning the parameters (weights)
• Supervised • Unsupervised • Reinforcement Learning
• Employing a Neural Network: – Selecting the Architecture – # Layers – # Units per layer – Kind of learning algorithm
Introduction From Neural Network to Deep Learning
![Page 18: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/18.jpg)
• Training a Neural Network: – Backward Propagation
• Gradient descent • Objective: Minimize the cost function J
Introduction From Neural Network to Deep Learning
x1
x2
x3
a12
a22
W(1)
W(2)
a(2) z(2)
z(3)
X
![Page 19: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/19.jpg)
• DNN à Typically artificial neural netwok with 3 or more levels of non-linear operations
Introduction From Neural Network to Deep Learning
![Page 20: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/20.jpg)
• Using Back propagation for Deep NN – Does not scale – Bad performance for random initialization – Local Optima – Vanishing gradient problem
Introduction Issues in Training DNN
![Page 21: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/21.jpg)
Introduction The Breakthrough
2006*+ Backward Propaga#on Greedy-‐layer wise training +
Supervised fine tuning
* Hinton et al. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527–1554, 2006 + Ranzato et al. Efficient learning of sparse representations with an energy-based model. Advances in Neural Information Processing Systems 19 (NIPS’06),
![Page 22: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/22.jpg)
• Deep learning methods: – Class of ML algorithm – Use cascade of many levels of non linear processing units for feature extraction
– Hierarchy of concepts – Multiple-layered model – NN with high number of hidden layers – NEW LEARNING ALGORITHM Overcoming previous training problems
Introduction Deep Learning - Summary
![Page 23: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/23.jpg)
Deep Learning Models
![Page 24: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/24.jpg)
Deep Learning Models Introduction
• Two main classes: – Generative
• Deep Network for supervised Learning
– Discriminative • Deep Network for unsupervised learning
– Hybrid
![Page 25: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/25.jpg)
Deep Learning Models Generative – Deep Belief Network
• Generative graphic model • Mix directed and undirected between vars • Learn to reconstruct the input
![Page 26: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/26.jpg)
Deep Learning Models Generative – Deep Belief Network
• Training algorithm – Iteratively apply RBM training to each pair of layers
![Page 27: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/27.jpg)
Deep Learning Models Discriminative – Convolutional NN
• CNN in Computer Vision: Image Recognition – Feed-forward multilayer network – Kind of back propagation for learning – Receptive fields – Learn suitable representation of the image
![Page 28: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/28.jpg)
Deep Learning Models Discriminative – Convolutional NN
• CNN in Computer Vision: Image Recognition – Key concepts:
• Max pooling • Sparse Connectivity • Convolution
![Page 29: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/29.jpg)
Deep Learning in the Real World
![Page 30: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/30.jpg)
• NLP • Image Classification/Computer Vision • Speech Recognition
Introduction Deep Learning – Application Field
![Page 31: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/31.jpg)
• [Google] 2013 acquired DNNresearch of professor Geoff Hinton to improve the state of the art in image recognition in photos
• [Facebook] 2013 hired deep learning expert Yann to head up the company’s new artificial intelligence lab specialized in deep learning for computer vision and image recognition
• [Pinterest] 2014 announced it has acquired Visual Graph
• [Google + Baidu]: 20G13 - Deep Learning Visual Search Engine
Deep Learning in the Real World Facts
![Page 32: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/32.jpg)
• [Baidu] 2013: Deep Learning Visual Search Engine
• [Google] 2013 Photo Search Engine
• [Microsoft] 2013 Search by voice on Xbox console
• [Google] 2014 word2vec for word tagging or text messaging suggestion
Deep Learning in the Real World Products
![Page 33: Introduction to Deep learning](https://reader035.vdocuments.us/reader035/viewer/2022082207/55a56fcc1a28ab2c518b45ab/html5/thumbnails/33.jpg)
Thanks for the aUen)on