distributed deep learning with keras and … modules •datavec •arbiter •nn •datasets •rl4j...
TRANSCRIPT
DISTRIBUTED DEEP LEARNING WITH KERAS AND TENSORFLOW ON APACHE
SPARKGUGLIELMO IOZZIA
MSD
MOSCOW, OCTOBER 10TH 2019
#guglielmoiozzia
ABOUT ME
Currently at
Previously at
Author
I got some awards lately I love cooking
Champion #guglielmoiozzia
MSD IRELAND
+ 50 years
Approx. 2,000 employees
$2.5 billion investment to date
Approx 50% MSD’s top 20 products manufactured here
Export to + 60 countries
€6.1 billion turnover in 2017
2017 + 300 jobs & €280m investment
MSD Biotech, Dublin, coming in 2021
CORE TOPICS
• What is it?Deep Learning
• 2 of the most popular frameworks for DLKeras and Tensorflow
• Why is it so difficult?Why Distributed Deep Learning on Spark?
• Why and How?DL in Python on the
JVM
DEEP LEARNING
It is a subset of Machine
Learning which is based on
Multilayer Neural Networks
DEEP LEARNING
http://www.asimovinstitute.org/wp-content/uploads/2019/04/NeuralNetworkZoo20042019.png
DL FRAMEWORKS POPULARITY
TENSORFLOW
It is an end-to-end open source platform for ML. It has a comprehensive, flexible ecosystem of tools, libraries and community resources for researchers and developers.
https://www.tensorflow.org/
KERAS
Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano.
It allows for easy prototyping and runs seamlessly on CPUs and GPUs.
https://keras.io/
KERAS & TENSORFLOW
Starting from TensorFlow r1.14
Speed
It achieves high performance for
both batch and streaming data,
using a state-of-the-art DAG
scheduler, a query optimizer,
and a physical execution engine.
Ease of Use
It offers over 80 high-level
operators that make it easy to
build parallel apps. And you can
use it interactively from the
Scala, Python, R, and SQL shells.
Generality
Combine SQL, streaming, and
complex analytics.
Runs Everywhere
It runs on Hadoop, Apache
Mesos, Kubernetes,
standalone, or in the cloud.
It can access diverse data
sources.
CHALLENGES OF TRAINING MNNS IN SPARK
• Different execution models between Spark and the DL frameworks
• GPU configuration and management
• Performance
• Accuracy
WHY DISTRIBUTED DL ON THE JVM?
DEEPLEARNING4J
It is an Open Source,
distributed, Deep Learning
framework written for JVM
languages.
It is integrated with
Hadoop and Apache
Spark.
It can be used on
distributed GPUs and
CPUs.
WHY DISTRIBUTED DL ON THE JVM?
TensorFlow
DL4J MODULES
• DataVec
• Arbiter
• NN
• Datasets
• RL4J
• DL4J-Spark
• Model Import
• ND4J
It is an Open Source linear algebra
and matrix manipulation library which
supports n-dimensional arrays and it
is integrated with Apache Hadoop
and Spark.
DL4J + APACHE SPARK
• DL4J provides high level API to design, configure train and evaluate
MNNs.
• Spark performances are excellent in particular for ETL/streaming, but
in terms of computation, in a MNN training context, some data
transformation/aggregation needs to be done using a low-level
language.
• DL4J uses ND4J, which is a C++ library that provides high level Scala
API to developers.
DATA PARALLELISM AND MODEL PARALLELSM
HOW TRAINING HAPPENS IN SPARK WITH DL4J
Parameter Averaging(DL4J 1.0.0-alpha)
Asynchronous SDG(DL4J 1.0.0-beta+)
HOW TRAINING HAPPENS IN SPARK WITH DL4J
The key classes users should be familiar with to get started with distributed training in DL4J are:
• TrainingMaster: It specifies how distributed training will be conducted in practice. Implementations include Gradient Sharing or Parameter Averaging .
• SparkDl4jMultiLayer and SparkComputationGraph: They are wrappers around the MultiLayerNetwork and ComputationGraph classes in DL4J that enable the functionality related to distributed training.
• RDD<DataSet> and RDD<MultiDataSet>: Spark RDDs with DL4J’s DataSet or MultiDataSet classes that define the source of the training or evaluation data.
MODEL IMPORT IN DL4J
Keras TensorFlow
Train the Model
Save it as .h5
Load Model and
Weights
Load New Data
Predict
Train the Model
Save it as .pb
Load Model and
Weights
Load New Data
Predict
KerasModelImport
TFGraphMapper
MODEL IMPORT IN DL4J
Keras TensorFlow
Train the Model
Save it as .h5
Load Model and
Weights
Load New Data
Predict
Train the Model
Save it as .pb
Load Model and
Weights
Load New Data
Predict
MODEL IMPORT IN DL4J: EXAMPLE
Keras
Train the Model
Save it as .h5
Load Model and
Weights
Load New Data
Predict
Import the VGG16
Model.
Test it.
MODEL IMPORT IN DL4J: EXAMPLE
Keras
Train the Model
Save it as .h5
Load Model and
Weights
Load New Data
Predict
MODEL IMPORT IN DL4J: EXAMPLE
Keras
Train the Model
Save it as .h5
Load Model and
Weights
Load New Data
Predict
MODEL IMPORT IN DL4J: EXAMPLE
Keras
Train the Model
Save it as .h5
Load Model and
Weights
Load New Data
Predict
DL4J MODEL IMPORT IN ACTION
KERAS MODEL IMPORT: SUPPORTED FEATURES
• Layers
• Losses
• Activations
• Initializers
• Regularizers
• Constraints
• Metrics
• Optimizers
DL4J VISUAL FACILITIES
MEMORY UTILIZATION: SOMETHING TO TAKE CARE OF
More on DL with DL4J on Spark in my book
http://tinyurl.3c1om/y9jkvtuy
Thanks!
Any questions?
You can find me at
@GuglielmoIozzia
https://ie.linkedin.com/in/giozzia
googlielmo.blogspot.com