tensorflow flexible, scalable, portable
TRANSCRIPT
TensorFlowFlexible, Scalable, Portable
Rajat MongaEngineering Director, TensorFlow
repositoryfor “machine learning”
category on GitHub
#1
Released in Nov. 2015
10,000+ commits since Nov, 2015
450+ contributors
1M+ binary downloads
#20 most popular repository on GitHub by stars
Used in ML classes at many universities:Toronto, Berkeley, Stanford, ...
Some Stats
TensorFlow powered Cucumber Sorter
From: http://workpiles.com/2016/02/tensorflow-cnn-cucumber/
Flexible
MatMul
Add Relu
biases
weights
examples
labels
Xent
Directed graph
Async SGD with Parameter Server
Model Workers
Parameter Server
p∆p
p’ = p - λ *∆p
Async SGD with TensorFlow
Model Workers
PS Workers
p∆p
p’ = p - λ *∆p
Sync SGD with TensorFlow
Model Workers
PS Workers
p∆p
delta = ∑∆pp’ = p - λ * delta
Alternate version of Parameter sharing
Workers
More ML Algorithms● Clustering e.g. k-means● Random Forests● Logistic Regression● Bayesian methods
Scalable
Deferred Execution
Operations build the dataflow graph; eval() fetches the result.
import tensorflow as tf
with tf.Session():
x = tf.constant([[5, 6], [7, 8]])
z = tf.matmul(x, x) + tf.matmul(x, [[1, 0], [0, 1]])
# Run graph to fetch z.
result = z.eval()
MatMul
const x const
MatMul
+
Parallelism in Op implementations
MatMul
...
...
...
...
...
...
...MatMul
MatMul
Task Parallelism in DataFlow graph
Data Parallelism
MatMul
Input
MatMulParam
...
...MatMul
MatMul
...
...
Model Parallelism
MatMul
Matrix
MatrixMatMul
Split Concat
GPU 0 CPU
Add Mul
biases
learning rate
−=......
Distribution across Devices
GPU 0 CPU
Add Mul
biases
learning rate
−=......
Distribution● TensorFlow inserts Send/Recv Ops to transport tensors across devices● Recv ops pull data from Send ops
Send Recv
GPU 0 CPU
Add Mul
biases
learning rate
−=......
Distribution● TensorFlow inserts Send/Recv Ops to transport tensors across devices● Recv ops pull data from Send ops
Send Recv
Send Recv
Send
Recv SendRecv
Scale across machines
Portable
Platforms
TensorFlow Core Execution Engine
CPU GPU Android iOS ...
Device Abstraction
Languages
TensorFlow Core Execution Engine
CPU GPU Android iOS ...
Language Abstraction
C++Python ...
Now what?
AutoTrash
AutoTrash
Rajat Monga@rajatmonga
Thank You!