Upload File

deep learning and gpus - hpc advisory council€¦ · accelerating deep learning: cudnn...

  • Home
  • Documents
  • Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates
34
Julie Bernauer Deep Learning and GPUs

Upload: others

Post on 20-May-2020

23 views

Category:

Documents


2 download

Report

TRANSCRIPT

Page 1: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

Julie Bernauer

Deep Learning and GPUs

Page 2: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

2

GPU Computing

Page 3: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

3

GPU Computing

x86

Page 4: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

4

CUDA

A simple sum of two vectors (arrays) in C

GPU friendly version in CUDA

Framework to Program NVIDIA GPUs

__global__ void vector_add(int n, const float *a, const float *b, float *c) { int idx = blockIdx.x*blockDim.x + threadIdx.x; if( idx < n ) c[idx] = a[idx] + b[idx]; }

void vector_add(int n, const float *a, const float *b, float *c) { for( int idx = 0 ; idx < n ; ++idx ) c[idx] = a[idx] + b[idx]; }

Page 5: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

5

GPU accelerated libraries “Drop-in” Acceleration for Your Applications

Linear Algebra FFT, BLAS,

SPARSE, Matrix, cuSolver

Numerical & Math RAND, Statistics

Data Struct. & AI Sort, Scan, Zero Sum

Visual Processing Image & Video

NVIDIA

cuFFT,

cuBLAS,

cuSPARSE

NVIDIA

Math Lib NVIDIA cuRAND

NVIDIA

NPP

NVIDIA

Video

Encode

GPU AI – Board Games GPU AI – Path Finding

Page 6: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

6

Deep Neural Networks

Page 7: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

7

Image Classification with DNNs

Training

Inference

cars buses trucks motorcycles

truck

Page 8: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

8

Image Classification with DNNs

Training

cars buses trucks motorcycles

Typical training run

Pick a DNN design

Input 100 million training images spanning 1,000 categories

One week of computation

Test accuracy

If bad: modify DNN, fix training set or update training parameters

Page 9: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

9

Deep Neural Networks and GPUs

Page 10: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

10

ACCELERATING INSIGHTS

GOOGLE DATACENTER

1,000 CPU Servers 2,000 CPUs • 16,000 cores

600 kWatts

$5,000,000

STANFORD AI LAB

3 GPU-Accelerated Servers 12 GPUs • 18,432 cores

4 kWatts

$33,000

Now You Can Build Google’s

$1M Artificial Brain on the Cheap

“ “

Deep learning with COTS HPC systems, A. Coates, B. Huval, T. Wang, D. Wu, A. Ng, B. Catanzaro ICML 2013

ACCELERATING INSIGHTS

Page 11: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

11

2012: GOOGLE BRAIN 2016: AlphaGO

Modern AI

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2009 2010 2011 2012 2013 2014 2015 2016

Traditional CV

Deep Learning

IMAGENET Accuracy Rate

Page 12: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

12

DEEP LEARNING EVERYWHERE

INTERNET & CLOUD

Image Classification Speech Recognition

Language Translation Language Processing Sentiment Analysis Recommendation

MEDIA & ENTERTAINMENT

Video Captioning

Video Search Real Time Translation

AUTONOMOUS MACHINES

Pedestrian Detection Lane Tracking

Recognize Traffic Sign

SECURITY & DEFENSE

Face Detection Video Surveillance Satellite Imagery

MEDICINE & BIOLOGY

Cancer Cell Detection Diabetic Grading Drug Discovery

Page 13: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

13

NVIDIA GPU: the engine of deep learning

NVIDIA CUDA ACCELERATED COMPUTING PLATFORM

WATSON CHAINER THEANO MATCONVNET

TENSORFLOW CNTK TORCH CAFFE

Page 14: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

14

Accelerating Deep Learning: cuDNN

GPU-accelerated Deep Learning

subroutines

High performance neural network

training

Accelerates Major Deep Learning

frameworks: Caffe, Theano, Torch

Up to 3.5x faster AlexNet training in

Caffe than baseline GPU

Perf

orm

ance

AlexNet training throughput based on 20 iterations, CPU: 1x E5-2680v3 12 Core 2.5GHz. 128GB System Memory, Ubuntu 14.04

Caffe Performance

K40

K40+cuDNN1

M40+cuDNN3

M40+cuDNN4

0

1

2

3

4

5

6

11/2013 9/2014 7/2015 12/2015

developer.nvidia.com/cudnn

http://developer.nvidia.com/digits
http://developer.nvidia.com/digits
Page 15: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

15

Multi-GPU communication: NCCL

• Research library of accelerated collectives that is easily

integrated and topology-aware so as to improve the scalability

of multi-GPU applications

• Pattern the library after MPI’s collectives

• Handle the intra-node communication in an optimal way

• Provide the necessary functionality for MPI to build on top to

handle inter-node

Collective library

github.com/NVIDIA/nccl

http://github.com/NVIDIA/nccl
Page 16: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

16

NCCL Example

#include <nccl.h>

ncclComm_t comm[4];

ncclCommInitAll(comm, 4, {0, 1, 2, 3});

foreach g in (GPUs) { // or foreach thread

cudaSetDevice(g);

double *d_send, *d_recv;

// allocate d_send, d_recv; fill d_send with data

ncclAllReduce(d_send, d_recv, N, ncclDouble, ncclSum, comm[g], stream[g]);

// consume d_recv

}

All-reduce

Page 17: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

17

Platform

Page 18: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

18

Data Scientist Embedded platform

An end-to-end solution

Deploy

Model Classification

Detection

Segmentation Train

Network

Solver

Dashboard

Page 19: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

19

CUDA for deep learning development

TITAN X GPU CLOUD DEVBOX

DEEP LEARNING SDK

cuSPARSE cuBLAS DIGITS NCCL cuDNN

Page 20: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

20

Tesla for hyperscale datacenters

270M Items sold/day 43% on mobile devices

TESLA M4 TESLA M40

POWERFUL: Fastest Deep Learning Performance LOW POWER: Highest Hyperscale Throughput

10M Users 40 years of video/day

HYPERSCALE SUITE

Deep Learning SDK

GPU REST Engine

GPU Accelerated FFmpeg

Image Compute Engine

GPU support in Mesos

Page 21: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

21

Jetson for Intelligent machines

10M Users 40 years of video/day

270M Items sold/day 43% on mobile devices

JETSON SDK JETSON TX1

GPU 1 TFLOP/s 256-core Maxwell

CPU 64-bit ARM A57 CPUs

Memory 4 GB LPDDR4 | 25.6 GB/s

Storage 16 GB eMMC

Size 50mm x 87mm

Power Under 10W

Page 22: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

22

Using the GPU for Deep Learning

Page 23: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

23

DIGITSTM

Quickly design the best deep neural network (DNN) for your data

Train on multi-GPU (automatic)

Visually monitor DNN training quality in real-time

Manage training of many DNNs in parallel on multi-GPU systems

Interactive Deep Learning GPU Training System

developer.nvidia.com/digits

http://developer.nvidia.com/digits
Page 24: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

24

For the developer: DIGITSTM devbox

Four TITAN X GPUs with 12GB of memory per GPU

1600W Power Supply Unit

Ubuntu 14.04

NVIDIA-qualified driver

NVIDIA® CUDA® Toolkit 7.0

NVIDIA® DIGITS™ SW

Caffe, Theano, Torch, BIDMach

Fastest Deskside Deep Learning Training Box

Page 25: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

25

For the datacenter: multi-GPU servers

For accelerated training

Facebook

Big Sur Opencompute server

Page 26: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

26

TESLA M40 World’s Fastest Accelerator

for Deep Learning 0 1 2 3 4 5 6 7 8 9

Tesla M40

CPU

8x Faster Caffe Performance

# of Days

Caffe Benchmark: AlexNet training throughput based on 20 iterations, CPU: E5-2697v2 @ 2.70GHz. 64GB System Memory, CentOS 6.2

CUDA Cores 3072

Peak SP 7 TFLOPS

GDDR5 Memory 12 GB

Bandwidth 288 GB/s

Power 250W

Reduce Training Time from 8 Days to 1 Day

Page 27: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

27

What else can one do with Deep Learning?

Page 28: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

28

Action Recognition

Page 29: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

29

Segmentation

Clement Farabet, Camille Couprie, Laurent Najman and Yann LeCun: Learning Hierarchical Features for Scene Labeling, IEEE Transactions on Pattern Analysis and Machine Intelligence,

August, 2013

https://www.youtube.com/watch?v=KkNhdlNs13U

Page 30: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

30

Image Captioning

“Automated Image Captioning with ConvNets and Recurrent Nets”

—Andrej Karpathy, Fei-Fei Li

Page 31: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

31

Playing Games

https://www.youtube.com/watch?v=V1eYniJ0Rnk

Google’s DeepMind

Page 32: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

32

Resources

Page 33: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

33

Want to try?

Deep Learning https://developer.nvidia.com/deep-learning

Hands-on labs https://nvidia.qwiklab.com/

Question? Email [email protected]

Links and resources

https://developer.nvidia.com/deep-learning
https://developer.nvidia.com/deep-learning
https://developer.nvidia.com/deep-learning
https://nvidia.qwiklab.com/
https://nvidia.qwiklab.com/
mailto:[email protected]
Page 34: Deep Learning and GPUs - HPC Advisory Council€¦ · Accelerating Deep Learning: cuDNN GPU-accelerated Deep Learning subroutines High performance neural network training Accelerates

Introduction to Deep Learning - Computer Graphics€¦ · Introduction to Deep Learning CS468 Spring 2017 Charles Qi. What is Deep Learning? Deep learning allows computational models

cuDNN Release Notes...cuDNN Release Notes RN-08667-001_v8.0.2 | ii Table of Contents Chapter 1. cuDNN Overview.....1 Chapter 2. cuDNN Release 8.x

CUDNN Library (1)

Deep Learning in real world @Deep Learning Tokyo

20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

Deep Learning I - Korea Universitydsba.korea.ac.kr/wp/wp-content/seminar/Deep learning/Deep Learning... · Deep Learning I Deep learning is research on learning models with multilayer

Chainer Documentation - Read the Docs · Chainer Documentation, Release 1.18.0 1.2.5Install Chainer with CUDA and cuDNN cuDNN is a library for Deep Neural Networks that NVIDIA provides

Learning Deep Learning

UNSUPERVISED DEEP LEARNING - TAUweb.eng.tau.ac.il/.../uploads/2016/12/Unsupervised-Deep-Learning.pdf · UNSUPERVISED DEEP LEARNING Erez Aharonov Noam Eilon Deep Learning Seminar School

NCCL 2 - NVIDIA · GPU Direct P2P . 4 NCCL Architecture NCCL CUDA Caffe CUBLAS Caffe2 Torch TF MXNET CNTK Deep Learning Frameworks NVIDIA GPUs CUDNN . 5 AGENDA NCCL History Design

Deep Learning for Reinforcement Learning in Pacman · Deep Learning for Reinforcement Learning in Pacman Deep Learning für Reinforcement Learning in Pacman Vorgelegte Bachelor-Thesis

What’s New in MATLAB and Simulink...Deploying Deep Learning Applications ARM Compute Library Intel MKL-DNN Library 1 2 3 Deep Learning Networks Coder Products NVIDIA TensorRT & cuDNN

Lecture 3: Deep Learning Optimizations - GitHub Pagesuvadlc.github.io/lectures/nov2019/lecture3.pdf · Lecture 3: Deep Learning Optimizations Deep Learning @ UvA. UVA DEEP LEARNING

Deep learning / representation learning

Lecture 1: Introduction to Deep Learning · uva deep learning course –efstratios gavves introduction to deep learning - 1

DU-06702-001 v07 | December 2017 CUDNN Developer Guidedocs.nvidia.com/deeplearning/sdk/pdf/cuDNN-Developer-Guide.pdf · 3.22. cudnnConvolutionFwdAlgo_t ... 4.113. cudnnDropoutGetStatesSize

fast.ai - Learning Deep Learning

REAL WORLD PROBLEM SIMPLIFICATION USING DEEP LEARNING … · GPU Servers: Dual Xeon E5-2690 [email protected] with 16GB PCIe GPUs configs as shown Ubuntu 14.04.5, CUDA 9.0.103, cuDNN 7.0.1.13;

cuDNN Developer Guide · 2020. 7. 28. · cuDNN Developer Guide DU-06702-001_v8.0.2 | 2 Chapter 2. Programming Model The cuDNN library exposes a Host API but assumes that for operations

THE CONVERGENCE OF HPC AND DEEP LEARNING · 2020. 1. 15. · DeepStream SDK NCCL cuBLAS cuSPARSE cuDNN TensorRT ECOSYSTEM TOOLS HPC +450 Applications COMPUTEWORKS CUDA C/C++ FORTRAN

DU-08670-001 v07 | April 2018 CUDNN Installation Guide cuDNN on Linux cuDNN DU-08670-001_v07 | 4 1. Navigate to your directory containing cuDNN Debian file. 2. Install

CNN INFERENCE WITH cuDNN - Nvidia...2 AGENDA Introduction to cuDNN cuDNN Best Practices: Memory Management Done Right Choosing the Right Convolution Algorithm & Tensor Layout Tensor

Deep Learning for Laser Based Odometry Estimation - …juxi.net/workshop/deep-learning-rss-2016/papers/Nicolai - Deep... · Deep Learning for Laser Based Odometry Estimation ... deep

Deep Learning: An Introduction from the NLP Perspective Deep Learning - An...Deep Learning Approach 1 Deep Learning Approach 2 Disclaimer I am not (yet) an expert in Deep Learning

Tutorial: Deep Reinforcement Learning - Machine Learning ...hunch.net/~beygel/deep_rl_tutorial.pdfTutorial: Deep Reinforcement Learning - Machine Learning

Deep Learning Lab Course 2017 (Deep Learning …ais.informatik.uni-freiburg.de/teaching/ws17/deep...Deep Learning Lab Course 2017 (Deep Learning Practical) Labs: (Computer Vision)

Lecture 1: Introduction to Deep Learning - uvadlc.github.io · Lecture 1: Introduction to Deep Learning Efstratios Gavves. UVA DEEP LEARNING COURSE UVA DEEP LEARNING COURSE –EFSTRATIOS

NVIDIA DGX-1 ARTIFIcIAL INTELLIGENcE SYSTEM · 2019. 1. 6. · DEEP LEARNING LIBRARIES NVIDIA cuDNN and NCCL DEEP LEARNING USER SOFTWARE NVIDIA® DIGITS™ ACCELERATED SOLUTIONS CONTAINERIZATION

CNN & cuDNN - Piazza

Pipelines and RAPIDS Learning with the Kubeflow S91030 - Hybrid … · 2019-03-29 · DASK/SPARK DEEP LEARNING FRAMEWORKS CUDNN RAPIDS CUDF CUML CUGRAPH. BENCHMARKS cuML — XGBoost

cuDNN API Reference · cuDNN API Reference DA-09702-001_v8.0.2 | 1 Chapter 1. Introduction NVIDIA® CUDA® Deep Neural Network library™ (cuDNN) offers a context-based API that allows

Deep Learning - South Dakota State UniversityDeep Learning Deep learning (also known as deep structured learning, hierarchical learning or deep machine learning) is a branch of machine

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

Deep Learning for Generic Object Detection: A Survey · deep learning, probabilistic models, autoencoders, manifold learning, and deep networks 17 Deep learning LeCun et al. (2015)

GPU ACCELERATED DEEP LEARNING WITH CUDNN · Deep learning with COTS HPC systems A. Coates, B. Huval, T. Wang, D. Wu, A. Ng, B. Catanzaro ICML 2013 GOOGLE DATACENTER 1,000 CPU Servers