Large Scale Data Analysis Using Deep Learning (Prof. U Kang)
Introduction to TensorFlow
2017.04.17 Beunguk Ahn
1
What is TensorFlow?Consturction PhaseExecution PhaseExamplesGotchas and Tips
Contents
2
What is TensorFlow?Consturction PhaseExecution PhaseExamplesGotchas and Tips
Contents
3
You already know how to use You already installed
PrerequisitesPython
TensorFlow
4
TensorFlowOpen source so�ware library for numerical
computation using data flow graphs.
Simply, it is one of widely used frameworks for deeplearning programs.
5
TensorFlowOpen source so�ware library for numerical
computation using data flow graphs.
Simply it is one of widely used frameworks for deeplearning programs.
6
Role of TensorFlowMachine learning flow in general
7
Why use TensorFlow?Large community engagement(Google)Multiple GPU SupportData and Model ParallelismPython + NumpyTensorboard
However, you don't have to use TensorFlow.Other frameworks like Theano, Caffe, Torch, etc are also good options.
8
TensorTensor is simply multi dimension array
1-D Tensor : vector2-D Tensor : matrix3-D Tensor : tensor4-D Tensor : tensor ...
1Dtensor = [ 1, 2] # vector 2Dtensor = [[ 1, 2], # matrix [ 3 ,4]] 3Dtensor = [[[ 1, 2], # tensor [ 3, 4]], [[ 5, 6], [ 7, 8]], [[ 9,10], [11,12]]]
9
Shape of TensorShape of tensor is expressed as tuple
import tensorflow as tf
scalar = tf.constant(1) # () zero = tf.constant([]) # (0,) vector = tf.constant([1,2,3,4]) # (4,) matrix = tf.constant([[1,2],[3,4]]) # (2, 2) tensor = tf.constant([[[1],[ 3]], # (3, 2, 1) [[5],[ 7]], [[9],[11]]]) print '\n'.join( map(str, [scalar, zero, vector, matrix, tensor]))
>>> Tensor("Const:0", shape=(1,), dtype=int32) >>> Tensor("Const_1:0", shape=(0,), dtype=float32) >>> Tensor("Const_2:0", shape=(4,), dtype=int32) >>> Tensor("Const_3:0", shape=(2, 2), dtype=int32) >>> Tensor("Const_4:0", shape=(3, 2, 1), dtype=int32)
10
Data Type of TensorWhat kind of data contained in tensor
Also float(32,64), [u]int(8,16,32,64), bool, complex(64,128) are also available
import tensorflow as tf
int32 = tf.constant([[1,2],[3,4]]) # int32 float32 = tf.constant([[1.0, 2], [3, 4.0]]) # float32 float64 = tf.constant([[1.0, 2], [3, 4.0]], # float64 dtype=tf.float64) _string = tf.constant([["A", "B"], ["C", "D"]], # string dtype=tf.string) print '\n'.join( map(str, [int32, float32, float64, _string]))
>>> Tensor("Const:0", shape=(2, 2), dtype=int32) >>> Tensor("Const_1:0", shape=(2, 2), dtype=float32) >>> Tensor("Const_2:0", shape=(2, 2), dtype=float64) >>> Tensor("Const_3:0", shape=(2, 2), dtype=string)
11
TensorFlow program in two partsConstruction Phase
Build a graphDeclare Variables, Placeholders, Operations,Optimizers
Execution PhaseInitialize model and runStart session and iterate through the graph
12
What is TensorFlow?Consturction PhaseExecution PhaseExamplesGotchas and Tips
Contents
13
VariableStores a scalar or tensorUsed as an input for OperationsRequires initial value for future useAt constuction phase, variables are "just" defined.
Value will be assigned a�er SessioninitializedYou will get an idea through slides
14
Variableimport tensorflow as tf
# Define variable, yet initialized w = tf.Variable(tf.random_normal([1, 3]), name="weight")
with tf.Session() as sess:
w_value = sess.run(w) # get value for var 'w'
print '\n'.join(map(str, ([w, w_value])))
"tensorflow.python.framework.errors_impl. FailedPreconditionError: Attempting to use uninitialized value weight"
15
Variableimport tensorflow as tf
# Define variable, yet initialized w = tf.Variable(tf.random_normal([1, 3]), name="weight")# Define init operation init_op = tf.global_variables_initializer()
with tf.Session() as sess: sess.run(init_op) # initialize all declared variables w_value = sess.run(w) # get value for var 'w'
print '\n'.join(map(str, ([w, w_value])))
>>> Tensor("weight/read:0", shape=(2, 3), dtype=float32)>>> [[-0.50282216 -1.94712341 -0.47356459]]
16
Variableimport tensorflow as tf
# Define variable, yet initialized w = tf.Variable(tf.random_normal([1, 3]), name="weight")# Define init operation init_op = tf.global_variables_initializer()
with tf.Session() as sess: sess.run(init_op) # initialize all declared variables w_value = sess.run(w); x_value = sess.run(w) sess.run(init_op) # re-initialize y_value = sess.run(w)
print '\n'.join(map(str, ([w_value, x_value, y_value])))
>>> [[-0.50282216 -1.94712341 -0.47356459]] >>> [[-0.50282216 -1.94712341 -0.47356459]] # same >>> [[ 1.1028192 0.25910807 2.37004709]] # changed
17
PlaceholderPlaceholder for tensorMust be fed with data on execution for operationthat takes the placeholder as operand
18
Placeholderimport tensorflow as tf
x = [[1, 2]] y = [[1], [3]] pla = tf.placeholder(tf.int32, (1,2)) matmul = tf.matmul(pla, y)
with tf.Session() as sess: A = sess.run(matmul)
print '\n'.join(map(str, ([pla, A])))
>>> "You must feed a value for placeholder tensor 'Placeholder' with dtype int32 and shape [1,2]"
19
Placeholderimport tensorflow as tf
x = [[1, 2]] y = [[1], [3]] pla = tf.placeholder(tf.int32, (1,2)) matmul = tf.matmul(pla, y)
with tf.Session() as sess: A = sess.run(matmul, feed_dict={pla : x})
print '\n'.join(map(str, ([pla, A])))
>>> Tensor("Placeholder:0", shape=(2, 3), dtype=int32) >>> [[7]]
20
Shape of PlaceholderShape could be partially known
import tensorflow as tf
p1 = tf.placeholder(tf.float32, [1, None])p2 = tf.placeholder(tf.float32, [None, 2])p3 = tf.placeholder(tf.float32, [None, None]) ops = [tf.multiply(x, 10) for x in [p1, p2, p3]] mat1 = [[1, 2]] mat2 = [[3, 4], [5, 6]] mat3 = [[1], [2]]
with tf.Session() as sess: print '\n'.join(map(str, sess.run(ops, feed_dict={ p1:mat1, p2:mat2, p3:mat3})))
>>> [[ 10. 20.]] >>> [[ 30. 40.] [ 50. 60.]] >>> [[ 10.] [ 20.]]
21
Shape of PlaceholderUseful when you feed input/output to your model
without giving the size of batchimport tensorflow as tf
# Placeholder MNIST # image size 28x28 x = tf.placeholder(tf.float32, [None, 784]) # 10 classes [0..9] y_ = tf.placeholder(tf.float32, [None, 10])
# Softmax regression Wx = b W = tf.Variable(tf.zeros([784, 10])) b = tf.Variable(tf.zeros([10])) y = tf.nn.softmax(tf.matmul(x, W) + b)
...
22
OperationOperation is graph node that performs computationon tensors
Computation should be run onSession
It takes inputs and generates outputs(optional)Usually, it ends up with prediction, cost function, orupdating variables
We have already seen some operations through the slides
23
OperationLinear Regression (Mean square error)
xW + b = y,L = ( W −12m′ ∑
m′
i=1 xi yi)2
import tensorflow as tf, numpy as np # tf Graph Input X = tf.placeholder("float") Y = tf.placeholder("float")
# Set model weights W = tf.Variable(np.random.randn(), name="weight") b = tf.Variable(np.random.randn(), name="bias")
# Construct a linear model pred = tf.add(tf.multiply(X, W), b)
# Mean square error cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples)
24
OperationIt is better to read through documentation so that you
don't have to reinvent the wheel
Mathmatmul, reduce_sum, argmax, norm...
Assignassign, assign_add, assign_sub...
Tensor transformslice, split, stack, concat...
25
OptimizerProvides methods to compute gradients for a lossand apply gradients to variablesGetting cost function as input and
CalcualtegradientsProcess gradientsApply gradients
"minimize" method on Optmizers do all 3 stepsMany optimizers are alreafy implemented
Gradient Descent Optimizer, Adagrad Optimizer, RMSProp Optmizer...
26
Optimizerimport tensorflow as tf, numpy as np ... # Construct a linear model pred = tf.add(tf.multiply(X, W), b)
# Mean square error cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples) # Gradient descent optimizer = tf.train\ .GradientDescentOptimizer(learning_rate)\ .minimize(cost)
with tf.Session() as sess: for epoch in range(training_epochs): for (x, y) in zip(train_X, train_Y): # SGD m'=1 sess.run(optimizer, feed_dict={X: x, Y: y}) # update weights
27
What is TensorFlow?Consturction PhaseExecution PhaseExamplesGotchas and Tips
Contents
28
SessionSession launchs the graph, initalize them, and runoperations as you definedWe have seen sessions through the slide, and I hopeyou got the idea of what it is
29
SessionYour data will be lost when you close the sessionimport tensorflow as tf
var = tf.Variable(tf.random_normal((1, 3))) init = tf.global_variables_initializer()
with tf.Session() as sess: sess.run(init) print sess.run(var) # Fetch train data # Calcuate cost # Optimize # Do Test # Display or Save print var.eval()
"No default session is registered. Use `with sess.as_default()` or pass an explicit session to ..."
30
What is TensorFlow?Consturction PhaseExecution PhaseExamplesGotchas and Tips
Contents
31
MNISTFamous example for deep learning
Data: Hand-writing digit[0-9] image data(28*28)
Each image will be treated as 784 length ofvector
Task: Classify what digit in each image10classes
Model: So�max regression modelAlso use 3 hiddenlayers
32
MNISTx = set of images
y = set of labels
laye = x +r1 W1 b1
laye = laye +r2 W2 r1 b2
pred = laye +W3 r2 b3
cost = y ∗ log(softmax(pre − ))1m′ ∑
m′
i=1 di yi33
Hyper parameters
Reference:
# Parameters learning_rate = 0.001 training_epochs = 15 batch_size = 100 display_step = 1
# Network Parameters n_hidden_1 = 256 # 1st layer number of features n_hidden_2 = 256 # 2nd layer number of features n_input = 784 # MNIST data input (img shape: 28*28) n_classes = 10 # MNIST total classes (0-9 digits)
TensorFlow Exmaples
34
Placeholder & Variablesimport tensorflow as tf
# tf Graph input x = tf.placeholder("float", [None, n_input]) # input y = tf.placeholder("float", [None, n_classes]) # output
# Store layers weight & bias h1 = tf.Variable(tf.random_normal([n_input, n_hidden_1])) h2 = tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])) out = tf.Variable(tf.random_normal([n_hidden_2, n_classes]))
b1 = tf.Variable(tf.random_normal([n_hidden_1])) b2 = tf.Variable(tf.random_normal([n_hidden_2])) b3 = tf.Variable(tf.random_normal([n_classes]))
35
36
Build a model# Hidden layer 1 with RELU activation layer_1 = tf.add(tf.matmul(x, h1), b1) layer_1 = tf.nn.relu(layer_1)
# Hidden layer 2 with RELU activation layer_2 = tf.add(tf.matmul(layer_1, h2), b2) layer_2 = tf.nn.relu(layer_2)
# Output layer with linear activation pred = tf.matmul(layer_2, out) + b3
37
38
Optimizer# Define loss and optimizer y_hat_softmax = tf.nn.softmax(pred) cost = tf.reduce_mean( -tf.reduce_sum(y * tf.log(y_hat_softmax), [1])) optimizer = tf.train.AdamOptimizer( learning_rate=learning_rate)\ .minimize(cost)
# Initializing the variables init = tf.global_variables_initializer()
39
40
Run & Learn# Launch the graph with tf.Session() as sess: sess.run(init)
# Training cycle for epoch in range(training_epochs): avg_cost = 0. total_batch = int(mnist.train.num_examples/batch_size) # Loop over all batches for i in range(total_batch): batch_x, batch_y = mnist.train.next_batch(batch_size) # Run optimization op (backprop) and cost op (to get loss value) _, c = sess.run([optimizer, cost], feed_dict={x: batch_x, y: batch_y}) # Compute average loss avg_cost += c / total_batch # Display logs per epoch step if epoch % display_step == 0: print "Epoch:", '%04d' % (epoch+1), "cost=", \ "{:.9f}".format(avg_cost) print "Optimization Finished!"
# Test model correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1)) # Calculate accuracy accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float")) print "Accuracy:", accuracy.eval({x: mnist.test.images, y: mnist.test.labels})
41
Result>>> ... >>> Epoch: 0010 cost= 3.325720582 >>> Epoch: 0011 cost= 2.454958990 >>> Epoch: 0012 cost= 1.861004718 >>> Epoch: 0013 cost= 1.412745127 >>> Epoch: 0014 cost= 1.158697252 >>> Epoch: 0015 cost= 0.943136121 >>> Optimization Finished! >>> Accuracy: 0.9485
42
What is TensorFlow?Consturction PhaseExecution PhaseExamplesGotchas and Tips
Contents
43
Gotchas & TipsDon't forget to put value to feed_dict
import tensorflow as tf
pla = tf.placeholder(tf.int32, [2,2]) x = tf.Variable([[1,2],[3,4]]) y = tf.Variable([[3,4],[1,2]]) init = tf.global_variables_initializer() matmul = tf.matmul(pla, y)
with tf.Session() as sess: sess.run(init)
f_dict = { pla: x } # x is actually tensor "object" result = sess.run(matmul, feed_dict=f_dict) print result
"ValueError: setting an array element with a sequence."
44
Gotchas & TipsDon't forget to put value to feed_dict
import tensorflow as tf
pla = tf.placeholder(tf.int32, [2,2]) x = tf.Variable([[1,2],[3,4]]) y = tf.Variable([[3,4],[1,2]]) init = tf.global_variables_initializer() matmul = tf.matmul(pla, y)
with tf.Session() as sess: sess.run(init) x_v = sess.run(x) f_dict = { pla: x_v } # x_v is now "array(2,2)" result = sess.run(matmul, feed_dict=f_dict) print result
>>> [[ 5 8] [13 20]]
45
Gotchas & TipsBetter not to make Variable/Placeholder/Operation
inside epochimport tensorflow as tf
with tf.Session() as sess: for epoch in range(100): # It will make session overflow! v = tf.Variable([[1,2]]) op = tf.add(1,2) pla = tf.placeholder(tf.int32, [1,2])
print '\n'.join(map(str, [v, op, pla]))
>>> ... >>> Tensor("Variable_99/read:0", shape=(1, 2), dtype=int32) >>> Tensor("Add_99:0", shape=(), dtype=int32) >>> Tensor("Placeholder_99:0", shape=(1, 2), dtype=int32)
46
Useful ReferencesTensorFlow-ExamplesDCGAN-tensorflowStyle Transfer inTensorFlowWord2vec in TensorFlow
47
Q & A
48