(based on andres rodriguez’ course “deep learning 101 ... · 3 deep learning • a branch of...

Prepared for Intel Delta 7 series.

(based on Andres Rodriguez’ course “Deep Learning 101” https://software.intel.com/en-us/ai/academy )

• Deep learning overview and usages

• Intel optimized deep learning environment

• hardware, software, tools

• Training models

• Getting started resources

Content Outline

Deep Learning

• A branch of machine learning

• Data is passed through multiple non-linear transformations

• Objective: Learn the parameters of the transformation that minimize a cost function

Types of Deep Learning

• Supervised learning

• Data -> Labels

• Unsupervised learning

• No labels; Clustering; Reducing dimensionality

• Reinforcement learning

• Reward actions (e.g., robotics)

Types of Deep Learning

• Supervised learning

• Data -> Labels

• Unsupervised learning

• No labels; Clustering; Reducing dimensionality

• Reinforcement learning

• Reward actions (e.g., robotics)

Step 1: Training (Over Hours/Days/Weeks)

Supervised Learning

Person

90% person 8% traffic light

Input data

Output Classification

Create Deep network

Step 2: Inference (Real Time)

New input from camera and

sensors

Output Classification

Trained neural network model

97% person

Trained Model

Bigger Data Better Hardware Smarter Algorithms

Why Now?

Image: 1000 KB / picture

Audio: 5000 KB / song

Video: 5,000,000 KB / movie

Transistor density doubles every 18 months

Cost / GB in 1995: $1000.00

Cost / GB in 2015: $0.03

Advances in algorithm innovation, including neural networks, leading to better accuracy in training models

Why machine learning?

• The explosive growth of smart and connected devices. – agriculture, retail, industrial and automated driving (by 2020 all cars sold in USA will

have automated control regime )

• computer vision, using technology like Intel® RealSense™,

• Exponential growth of data generated: 1.5GB/day/human and upto 4 TB/day/device by 2020

• Cloud services running on customer data : Amazon, Google, Picasa etc etc.

8 Intel Confidential

Classification

Label the image

• Person

• Motorcyclist

• Bike https://people.eecs.berkeley.edu/~jhoffman/talks/lsda-baylearn2014.pdf

Detection

Detect and

label objects

https://people.eecs.berkeley.edu/~jhoffman/talks/lsda-baylearn2014.pdf

Semantic Segmentation

Label every pixel

https://people.eecs.berkeley.edu/~jhoffman/talks/lsda-baylearn2014.pdf

Natural Language Object Retrieval

http://arxiv.org/pdf/1511.04164v3.pdf

Visual and Textual Question Answering

https://arxiv.org/pdf/1603.01417v1.pdf

Visuomotor Control

https://arxiv.org/pdf/1504.00702v5.pdf

Speech Recognition

The same architecture is used for English and Mandarin Chinese speech recognition

http://svail.github.io/mandarin/

Q&A Natural Language Understanding

https://arxiv.org/pdf/1506.07285.pdf

Personal Assistant

Deep Learning Use Cases

Cloud Service Providers

Financial Services

Healthcare

Automotive

• Personal assistant

• Image & Video recognition/tagging

• Natural language processing

• Automatic Speech recognition

• Targeted Ads

• Fraud / face detection

• Gaming, check processing

• Computer server monitoring

• Financial forecasting and prediction

• Network intrusion detection

• Recommender Systems

Optimized Deep Learning Environment

Fuel the development of vertical solutions

Deliver best single node and multi-node performance

Accelerate design, training, and deployment

Drive optimizations across open source deep learning frameworks

Intel® Deep Learning SDK

Intel® Omni-Path Architecture (Intel® OPA)

Maximum performance on Intel architecture Intel® Math Kernel

Library (Intel® MKL)

Training Inference

Intel® MKL-DNN

Drive optimizations across open source machine learning frameworks

Training

Intel® MKL-DNN

Inference

• Knights Landing

• Up to >6 peak SP TFLOPs per socket

• Binary-compatible with Intel® Xeon® processors

• Up to 72 cores, 512-bit SIMD vectors with 2 VPU/core

• Integrated memory delivers superior bandwidth

• Integrated Intel® Omni-Path fabric (dual-port; 50 Gb/s )

• Distributes the training workload

Intel® Xeon Phi™ + Omni-Path Deep Learning Training Scalable System

48p Sw

4x KNL

48p Sw

“12-12"

48p Sw

4x KNL

0 1 11

48p Sw

“4-4-4-4-4-4"

4x KNL

14 15 16

Storage

server

“2-2"

2013 2016

1ST GENERATION

XEON PHI

2ND GENERATION

XEON PHI

Knights Mill

Optimized for Deep Learning

Optimized for scale-out

Flexible, high capacity memory

Enhanced variable precision

Improved efficiency

Knights Mill: Next Gen Intel® Xeon Phi™ COMING 2017

Training Inference

Intel® MKL-DNN

Diversity in Deep Networks

VVariety in Network Topology

Recurrent NNs common for NLP/ASR, DAG for GoogLeNet, Networks with memory…

BBut there are a few well defined building blocks

Convolutions common for image recognition tasks

GEMMs for recurrent network layers—could be sparse

ReLU, tanh, softmax

GoogLeNet

Recurrent NN

CNN - AlexNet

Intel® Math Kernel Library (Intel® MKL)

Intel® MKL-DNN

Intel® MKL-DNN – free open source DNN functions designed for max Intel HW performance and high-velocity integration with DL frameworks

• Open source DNN functions included in MKL 2017

• IA optimizations contributed by community

• Binary GEMM functions

• Apache* 2 license

Targeted release: Q4’ 2016 (APIs and preview Q3’ 2016)

Intel® Math Kernel Library (Intel® MKL)

• Optimized AVX-2 and AVX-512 instructions

• Intel® Xeon® and Intel® Xeon Phi™ processors

• Supports all common layers types

• Coming soon: Winograd-based convolutions

Naïve Convolution

https://en.wikipedia.org/wiki/Convolutional_neural_network

Cache Friendly Convolution

arxiv.org/pdf/1602.06709v1.pdf

Recent MKL SGEMM Improvements

• Up to 4× performance improvements for small sizes

• New APIs to eliminate data packing overheads if A or B matrices are re-used

• Pack once and use multiple times

• Up to 1.2× additional performance improvements over SGEMM

• Available in Intel MKL 2017 (released: September 6, 2016)

Intel® Data Analytics Acceleration Library

(Intel® DAAL)

Training Inference

Intel® MKL-DNN

Deep Learning Tools

Programming languages

Top Frameworks

INTEL OPTIMIZED Caffe

•All the goodness of BVLC Caffe* + Integrated with Intel® MKL 2017 Multi-node distributed training

Forrest Iandola, et al., “Scaling DNN Training on Intel Platforms.” 2016

Intel® Data Analytics Acceleration Library

(Intel® DAAL)

Training Inference

Intel® MKL-DNN

35 Real-time graphs to see how the accuracy rate is trending

High-Level Workflow

.prototxt

.caffemodel

Trained Model

Model Optimizer

FP Quantize

Model Compress

End-Point

SW Developer

Import

Inference Run-Time

OpenVX

Application Logic

Forward Result

Real-time Data Validation Data

Model Analysis

MKL-DNN

Deploy-ready model

INFERENCE ON THE END POINT

Training Inference

Intel® MKL-DNN

Training

• Gradient descent and variants

• Batch sizes

• Distributed training

• Challenges

Gradient Descent 𝐽 𝒘(0) = 𝑐𝑜𝑠𝑡(𝒘(0), 𝒙𝑖)

𝑖=1

𝒘 𝒘(0)

𝑖=1

𝒘 𝒘(0)

𝑑𝐽 𝒘(0)

𝑑𝒘

𝑖=1

𝒘 𝒘(0)

𝒘(1) = 𝒘(0) − 𝑑𝐽 𝒘(0)

𝑑𝒘

𝑖=1

𝒘 𝒘(0)

𝒘(1) = 𝒘(0) − 𝛼𝑑𝐽 𝒘(0)

𝑑𝒘

learning rate

𝑖=1

𝒘 𝒘(0)

𝒘(1) = 𝒘(0) − 𝛼𝑑𝐽 𝒘(0)

𝑑𝒘

𝒘(1)

too small

𝑖=1

𝒘 𝒘(0)

𝒘(1) = 𝒘(0) − 𝛼𝑑𝐽 𝒘(0)

𝑑𝒘

𝒘(1)

too large

𝑖=1

𝒘 𝒘(0)

𝒘(1) = 𝒘(0) − 𝛼𝑑𝐽 𝒘(0)

𝑑𝒘

𝒘(1)

good enough

𝑖=1

𝒘 𝒘(2)

𝒘(2) = 𝒘(1) − 𝛼𝑑𝐽 𝒘(1)

𝑑𝒘

𝒘(1)

𝑖=1

𝒘(3) = 𝒘(2) − 𝛼𝑑𝐽 𝒘(2)

𝑑𝒘

𝒘(2)

𝒘(3)

𝑖=1

𝒘(4) = 𝒘(3) − 𝛼𝑑𝐽 𝒘(3)

𝑑𝒘

𝒘(4)

𝒘(3)

Gradient Descent

Saddle Point

Gradient Descent

Saddle Point

Gradient Descent

Saddle Point

Gradient Descent

Saddle Point

Gradient Descent

Saddle Point

Gradient Descent

Saddle Point

Gradient Descent

Saddle Point

Gradient Descent

𝐽 𝒘 = 𝑐𝑜𝑠𝑡(𝒘, 𝒙𝑖)

𝑖=1

Gradient Descent

𝑖=1

Gradient Descent

𝑖=1

Gradient Descent

𝑖=1

Gradient Descent

𝑖=1

Gradient Descent

𝑖=1

Gradient Descent

𝑖=1

Stochastic Gradient Descent (SGD)

𝑖=1

𝐽𝑏1 𝒘 = 𝑐𝑜𝑠𝑡(𝒘, 𝒙𝑖)

𝑖∈𝑏1

𝐽𝑏𝑀 𝒘 = 𝑐𝑜𝑠𝑡(𝒘, 𝒙𝑖)

𝑖∈𝑏𝑀

𝑀 = 𝑁/(𝑏𝑎𝑡𝑐ℎ 𝑠𝑖𝑧𝑒)

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

Stochastic Gradient Descent (SGD) + Momentum

𝑖=1

𝑖∈𝑏1

𝑖∈𝑏𝑀

⋮ 𝑀 = 𝑁/(𝑏𝑎𝑡𝑐ℎ 𝑠𝑖𝑧𝑒)

𝒗(𝑡+1) = 𝜇𝒗(𝑡) − 𝛼𝑑𝐽𝑏𝑚 𝒘

𝑑𝒘

𝒘(𝑡+1) = 𝒘(𝑡) + 𝒗(𝑡+1)

𝒘(𝑡+1) = 𝒘(𝑡) − 𝛼𝑑𝐽𝑏𝑚 𝒘

(𝑡)

𝑑𝒘

SGD SGD + Momentum

Momentum

gradient

velocity 𝒘

𝑑𝒘

𝒘(𝑡+1) = 𝒘(𝑡) + 𝒗(𝑡+1)

Momentum

gradient

velocity 𝒘

𝑑𝒘

𝒘(𝑡+1) = 𝒘(𝑡) + 𝒗(𝑡+1)

Momentum

gradient

velocity 𝒘

𝑑𝒘

𝒘(𝑡+1) = 𝒘(𝑡) + 𝒗(𝑡+1)

Momentum

gradient

velocity 𝒘

𝑑𝒘

𝒘(𝑡+1) = 𝒘(𝑡) + 𝒗(𝑡+1)

Momentum

gradient

velocity 𝒘

𝑑𝒘

𝒘(𝑡+1) = 𝒘(𝑡) + 𝒗(𝑡+1)

Momentum

gradient

velocity 𝒘

𝑑𝒘

𝒘(𝑡+1) = 𝒘(𝑡) + 𝒗(𝑡+1)

Batch Size 𝐽𝑏1 𝒘 = 𝑐𝑜𝑠𝑡(𝒘, 𝒙𝑖)

𝑖∈𝑏1

𝑖∈𝑏𝑀

𝑇𝑖𝑚𝑒 𝑇𝑜 𝑇𝑟𝑎𝑖𝑛 (𝑇𝑇𝑇)

𝑏𝑎𝑡𝑐ℎ 𝑠𝑖𝑧𝑒

Batch Size 𝐽𝑏1 𝒘 = 𝑐𝑜𝑠𝑡(𝒘, 𝒙𝑖)

𝑖∈𝑏1

𝑖∈𝑏𝑀

sweet spot

Overfitting

• Larger networks have a lot of weights

• The network can memorize the weights and do excellent on the training data and very poor on the validation data

• The networks does not generalize

Overfitting

Overfitting -- Solutions

• More training data

• Stop training when validation performance gets worse

• Penalize large weights – weight_decay

• Randomly ignored some weights in fully connected layers – dropout_ratio

• Training with single precision floating points

Caffe solver.prototxt

net: "models/bvlc_googlenet/train_val.prototxt"

base_lr: 0.01

lr_policy: "step"

stepsize: 320000

gamma: 0.96

max_iter: 10000000

momentum: 0.9

weight_decay: 0.0002

https://github.com/BVLC/caffe/blob/master/models/bvlc_googlenet/solver.prototxt

batch_size: 32

𝑑𝒘

𝒘(𝑡+1) = 𝒘(𝑡) + 𝒗(𝑡+1)

base_lr: 0.01

lr_policy: "step"

stepsize: 320000

gamma: 0.96

max_iter: 10000000

momentum: 0.9

batch_size: 32

𝑑𝒘

𝒘(𝑡+1) = 𝒘(𝑡) + 𝒗(𝑡+1)

base_lr: 0.01

lr_policy: "step"

stepsize: 320000

gamma: 0.96

max_iter: 10000000

momentum: 0.9

batch_size: 32

𝑑𝒘

𝒘(𝑡+1) = 𝒘(𝑡) + 𝒗(𝑡+1)

base_lr: 0.01

lr_policy: "step"

stepsize: 320000

gamma: 0.96

max_iter: 10000000

momentum: 0.9

batch_size: 32

base_lr: 0.01

lr_policy: "step"

stepsize: 320000

gamma: 0.96

max_iter: 10000000

momentum: 0.9

batch_size: 32

1 epoch = 1 cycle through all training samples

base_lr: 0.01

lr_policy: "step"

stepsize: 320000

gamma: 0.96

max_iter: 10000000

momentum: 0.9

batch_size: 32

1 epoch = 1 cycle through all training samples

=10M iter ∗ 32 imgs/iter

1.28M imgs/epoch = 250 epochs

base_lr: 0.01

lr_policy: "step"

stepsize: 320000

gamma: 0.96

max_iter: 10000000

momentum: 0.9

batch_size: 32

𝑑𝒘

𝒘(𝑡+1) = 𝒘(𝑡) + 𝒗(𝑡+1)

base_lr: 0.01

lr_policy: "step"

stepsize: 320000

gamma: 0.96

max_iter: 10000000

momentum: 0.9

batch_size: 32

𝑑𝒘

𝒘(𝑡+1) = 𝒘(𝑡) + 𝒗(𝑡+1)

Caffe solver.prototxt cont’…

test_iter: 1000

test_interval: 4000

snapshot: 40000

snapshot_prefix: "models/bvlc_googlenet/bvlc_googlenet“

solver_mode: CPU

test_iter: 1000

test_interval: 4000

snapshot: 40000

solver_mode: CPU

test_iter: 1000

test_interval: 4000

snapshot: 40000

solver_mode: CPU

Choosing hyperparams

• Experience. Experience. Experience

• Look through examples and models and practice modifying them

Multi-node Distributed training

Model Parallelism

• Break the model into N nodes

• The same data is in all the nodes

Data Parallelism

• Break the dataset into N nodes

• The same model is in all the nodes

• Good for networks with few weights, e.g., GoogleNet

• Intel Optimized Caffe uses this. Model+Data Parallelism is work in progress.

Data Parallelism

Forrest Iandola, et al., “Scaling DNN Training on Intel Platforms.” 2016

Scaling Efficiency: Intel® Xeon Phi™ Processor Deep Learning Image Classification Training Performance - MULTI-NODE Scaling

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance . *Other names and brands may be property of others Configurations: • Intel® Xeon Phi™ Processor 7250 (68 Cores, 1.4 GHz, 16GB MCDRAM), 128 GB memory, Red Hat* Enterprise Linux 6.7, Intel® Optimized Framework

1 2 4 8 16 32 64 128

# OF INTEL® XEON PHI™ PROCESSOR 7250 (68-CORES, 1.4 GHZ, 16 GB) NODES

OverFeat AlexNet VGG-A GoogLeNet

Challenges • Very large batch sizes train too slow and may

not reach the same accuracy

• GoogleNet performance decreases for batch sizes > 1024 𝑇𝑖𝑚𝑒 𝑇𝑜 𝑇𝑟𝑎𝑖𝑛 (𝑇𝑇𝑇)

sweet spot

Challenges • Total batch size = 1024:

• 1024 nodes each w/node batch size 1 (too much communication)

• 256 nodes each w/node batch size 4

• 64 nodes each w/node batch size 16 (most communication is hidden in the computation)

sweet spot

Challenges • Total batch size = 1024:

• 1024 nodes each w/node batch size 1 (too much communication)

• 256 nodes each w/node batch size 4

• 64 nodes each w/node batch size 16 (most communication is hidden in the computation)

• Guideline: Batch size is correlated with learning rate (not always linearly but it’s a good place to start)

sweet spot

https://software.intel.com/en-us/articles/training-and-deploying-deep-learning-networks-with-caffe-optimized-for-intel-architecture

Resources • Getting started with Caffe tutorial

• https://software.intel.com/en-us/articles/training-and-deploying-deep-learning-networks-with-caffe-optimized-for-intel-architecture

• Deep learning SDK

• https://software.intel.com/en-us/machine-learning/deep-learning/sdk-signup

• Intel Optimized Frameworks

• https://github.com/intel/caffe

• https://github.com/intel/theano

• https://github.com/intel/torch

• other frameworks coming soon…

For questions contact Nadya Plotnikova, Sr SW Engineer, CRT/DCG Russia

Nadezhda.Plotnikova@intel.com

This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure.

Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance.

Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction.

Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K.

The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate.

Intel, the Intel logo, Pentium, Celeron, Atom, Core, Xeon and others are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.

Legal Notices & Disclaimers

(based on andres rodriguez’ course “deep learning 101 ... · 3 deep learning • a branch of...

Documents

introduction to deep learning - joyce...

deep learning for reinforcement learning in · pdf filedeep...

deep learning lab course 2017 (deep learning...

executing deep learning strategies masterclass preview -...

deep-water turbidites as holocene san andres fault

introducing deep learning with matlab - mathworks ·...

deep learning for depth learning cs 229 course project...

an introduction to deep learning · deep learning an...

lecture 3: deep learning optimizations - github...

andres rodriguez at ai frontiers: catalyzing deep learning's...

unsupervised deep learning -...

machine learning and optimization - andres munoz,...

introduction to deep learning @ startup.ml by andres...

deep learning in real world @deep learning tokyo

learning deep learning

binary deep learning -...

the deep learning revolution - nvidia...the deep learning...

introducing deep learning with matlab - systematics ·...

deep learning i - korea...

deep learning