deep learning with tensorflow: understanding tensors, computations graphs, images, and text
TRANSCRIPT
![Page 1: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/1.jpg)
Deep Learning with TensorFlowUnderstanding Tensors, Computation Graphs, Images, Text
Viswanath PuttaguntaTechnical Program Manager, Linaro
![Page 2: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/2.jpg)
Vish Puttagunta
Technical Program Manager, Linaro
Statistics/Signal Processing in Images, Audio, RF
Data Analysis and Machine Learning on ARM SoCs.
Leading collaboration in ARMTM Ecosystem
Enterprise, Mobile, Home, Networking, IoT
Built on Open Source
![Page 3: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/3.jpg)
Goals
Basics behind Neural Networks
Picturing Neural Networks. Understand the operations.
Representing Neural Networks using Tensors and Computation GraphsVision (MNIST), Text (Word2Vec)
TensorFlowGives you good tools to build a Neural Network Model.
Some basic models… but Google's secret sauce still inside Google :)
![Page 4: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/4.jpg)
The Basics
Linear Regression
Weight / Bias
Cost Function
Gradient Descent to optimize a cost function
Logistic Regression
Neuron Activation
![Page 5: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/5.jpg)
Linear Regression
Objective: Fit line/curve to minimize a cost function Line has "Weight / Slope" and "Bias / y-
intercept)
![Page 6: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/6.jpg)
Cost Function, Gradient Descent
Source: https://www.youtube.com/watch?v=SqA6TujbmWw&list=PLE6Wd9FR--Ecf_5nCbnSQMHqORpiChfJf&index=16https://youtu.be/WnqQrPNYz5Q?list=PLaXDtXvwY-oDvedS3f4HW0b4KxqpJ_imw&t=284
● Cost Function, J( )𝞡○ Weight / Slope: 𝞡1
○ Bias / y-intercept: 𝞡2
● Gradient Descent○ Go down to the lowest
point in the cost function in small steps
BIG IDEA● Starts with some Weights & Biases● Define a Cost Function● Optimize the Cost function by doing
some sort of Gradient Descent● Lots of Help from TensorFlow.
![Page 7: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/7.jpg)
Weight, Bias in 0,1 Dimensions
![Page 8: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/8.jpg)
Neuron and Activation Function
BIG IDEAS● Only one of the Neurons will fire at a time (value shoots greater than 0.5)● Which neuron depends on "x", Weights, Biases● Rest of neurons will not fire. (value much less than 0.5)● Outcomes are between 0 and 1 → Probabilities (all outputs add to one)
○ → Good for classification
![Page 9: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/9.jpg)
Neuron and Activation Function
BIG IDEAS● Only one of the Neurons will fire at a time (value shoots greater than 0.5)● Which neuron depends on "x", Weights, Biases● Rest of neurons will not fire. (value much less than 0.5)● Outcomes are between 0 and 1 → Probabilities (all outputs add to one)
○ → Good for classification
![Page 10: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/10.jpg)
Neuron and Activation Function
BIG IDEAS● Only one of the Neurons will fire at a time (value shoots greater than 0.5)● Which neuron depends on "x", Weights, Biases● Rest of neurons will not fire. (value much less than 0.5)● Outcomes are between 0 and 1 → Probabilities (all outputs add to one)
○ → Good for classification
![Page 11: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/11.jpg)
Same picture: Less mess, more classes
![Page 12: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/12.jpg)
Simple Neural Network for MNIST
Source: https://www.tensorflow.org/versions/r0.7/tutorials/mnist/beginners/index.html
![Page 13: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/13.jpg)
Simple Neural Network for MNIST
Weights actually Mean something!!
![Page 14: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/14.jpg)
Weights/Bias → Evidence → Probability
BIG IDEAS● After proper Gradient Descent, Weights Generalize well.● Correlation operation corresponds to "Evidence"● Then do softmax(evidence) and you will get Probability(x specific 𝞊
class)
Source: https://www.tensorflow.org/versions/r0.7/tutorials/mnist/beginners/index.html
![Page 15: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/15.jpg)
MNIST Simple: Operations / Dimensions
● input image(x)○ 28x28 matrix → 1x784○ each element type: 0 or 1
● Weights (W0, W1…. W9)○ 28x28 matrix → 1x784○ each element: float(-1, +1)
● Eg: W1 x (Correlation)Ⓧ○ Multiply Accumulate!
■ matmul(W, xT)○ single float (-1, +1)
● Biases (B0, B1…. B9)○ single floating pt (-1, +1)
● Evidence: Eg: W1 x + B1Ⓧ○ single floating pt (-
784.0,+784.0)● Probability
○ single float (0,1)● Real output (Y), Predicted
o/p(Y^)○ 1x10 matrix○ each is 0 or 1
![Page 16: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/16.jpg)
MNIST Simple: In TensorFlow
![Page 17: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/17.jpg)
MNIST Simple: In TensorFlow
![Page 18: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/18.jpg)
MNIST Simple: Back-Propagation
![Page 19: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/19.jpg)
MNIST using Convolution Neural Network
![Page 20: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/20.jpg)
Convolution: Signal/Image processing
Source
![Page 22: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/22.jpg)
Weight, Bias in 0,1 Dimensions
![Page 23: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/23.jpg)
MNIST: Using CNN
![Page 24: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/24.jpg)
MNIST: Using CNN
![Page 25: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/25.jpg)
MNIST: Using CNN
![Page 26: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/26.jpg)
MNIST: Using CNN
![Page 27: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/27.jpg)
Text Analytics
Word2VecThinking of words and as vectors
Sequence-to-Sequence modelsRecurrent Neural Networks (LSTM)
Understanding sequence of words
"Don't like" is closer to "hate"
![Page 28: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/28.jpg)
Words as vectors (Embeddings)
list → [human, cow] → int list [0, 1] → embeddings [(2,2), (4,0)]
![Page 29: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/29.jpg)
Sequence of words as vectors!
![Page 30: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/30.jpg)
Projections/Dimensionality Reduction (PCA/TSNE)
![Page 31: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/31.jpg)
Projections/Dimensionality Reduction (PCA/TSNE)
![Page 33: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/33.jpg)
word2vec: using Neural Network
![Page 34: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/34.jpg)
word2vec: using Neural Network
Note:Only trying to understand relationship between words and understand best way to vectorize words!
Not really keen on sequence of words! (Later RNN)
![Page 35: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/35.jpg)
word2vec: using NN (softmax classifier)
Realize wv could easily be w50000
words in vocabulary!
![Page 36: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/36.jpg)
word2vec: using NN (softmax classifier-pitfall)
Too many computations for every set of inputs (Eg: Classifying across 50,000 classes if you vocabulary size is 50,000)
![Page 37: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/37.jpg)
word2vec: using Noise Classifier
Big Idea: Instead of trying to move every word(vector) by little bit in each stepMove few random words (~16) at a timeIf you have lots of sentences, you should be good
![Page 38: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/38.jpg)
word2vec: Compute Graph
![Page 39: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/39.jpg)
word2vec Code Walk Throughhttps://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/word2vec/word2vec_basic.py
![Page 40: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/40.jpg)
Summary
TensorFlow was simple to install (Ubuntu 14.04)sudo apt-get install python-pip python-dev
sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.8.0rc0-cp27-none-linux_x86_64.whl
Tutorials are 'relatively' easy to followGood pointers to papers and blogs
Would like to hear Google's perspective onHow good of open source citizens they will be.
Compared to Theano, Caffe, Torch, AOSP..
![Page 41: Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Images, and Text](https://reader031.vdocuments.us/reader031/viewer/2022012919/587186221a28ab2c198b4ee9/html5/thumbnails/41.jpg)
ReferencesConvolution Neural Network: https://youtu.be/n6hpQwq7Inw
TensorFlow Tutorials: https://www.tensorflow.org/versions/r0.7/tutorials/index.html
Khan Academy (Linear algebra): https://www.khanacademy.org/
LSTM, Text Analytics: Alec Ratford, Indico
https://www.youtube.com/watch?v=VINCQghQRuM
Current trends in Deep Learning (Andrew Ng, Baidu)
https://www.youtube.com/watch?v=O0VN0pGgBZM