Large Scale Data Analysis Using Deep Learning (Prof. U Kang)

Introduction to TensorFlow

2017.04.17 Beunguk Ahn

( )beunguk.ahn@gmail.com

1

What is TensorFlow?Consturction PhaseExecution PhaseExamplesGotchas and Tips

Contents

2

What is TensorFlow?Consturction PhaseExecution PhaseExamplesGotchas and Tips

Contents

3

You already know how to use You already installed

PrerequisitesPython

TensorFlow

4

TensorFlowOpen source so�ware library for numerical

computation using data flow graphs.

Simply, it is one of widely used frameworks for deeplearning programs.

5

TensorFlowOpen source so�ware library for numerical

computation using data flow graphs.

Simply it is one of widely used frameworks for deeplearning programs.

6

Role of TensorFlowMachine learning flow in general

7

Why use TensorFlow?Large community engagement(Google)Multiple GPU SupportData and Model ParallelismPython + NumpyTensorboard

However, you don't have to use TensorFlow.Other frameworks like Theano, Caffe, Torch, etc are also good options.

8

TensorTensor is simply multi dimension array

1-D Tensor : vector2-D Tensor : matrix3-D Tensor : tensor4-D Tensor : tensor ...

1Dtensor = [ 1, 2] # vector 2Dtensor = [[ 1, 2], # matrix [ 3 ,4]] 3Dtensor = [[[ 1, 2], # tensor [ 3, 4]], [[ 5, 6], [ 7, 8]], [[ 9,10], [11,12]]]

9

Shape of TensorShape of tensor is expressed as tuple

import tensorflow as tf

scalar = tf.constant(1) # () zero = tf.constant([]) # (0,) vector = tf.constant([1,2,3,4]) # (4,) matrix = tf.constant([[1,2],[3,4]]) # (2, 2) tensor = tf.constant([[[1],[ 3]], # (3, 2, 1) [[5],[ 7]], [[9],[11]]]) print '\n'.join( map(str, [scalar, zero, vector, matrix, tensor]))

>>> Tensor("Const:0", shape=(1,), dtype=int32) >>> Tensor("Const_1:0", shape=(0,), dtype=float32) >>> Tensor("Const_2:0", shape=(4,), dtype=int32) >>> Tensor("Const_3:0", shape=(2, 2), dtype=int32) >>> Tensor("Const_4:0", shape=(3, 2, 1), dtype=int32)

10

Data Type of TensorWhat kind of data contained in tensor

Also float(32,64), [u]int(8,16,32,64), bool, complex(64,128) are also available

import tensorflow as tf

int32 = tf.constant([[1,2],[3,4]]) # int32 float32 = tf.constant([[1.0, 2], [3, 4.0]]) # float32 float64 = tf.constant([[1.0, 2], [3, 4.0]], # float64 dtype=tf.float64) _string = tf.constant([["A", "B"], ["C", "D"]], # string dtype=tf.string) print '\n'.join( map(str, [int32, float32, float64, _string]))

>>> Tensor("Const:0", shape=(2, 2), dtype=int32) >>> Tensor("Const_1:0", shape=(2, 2), dtype=float32) >>> Tensor("Const_2:0", shape=(2, 2), dtype=float64) >>> Tensor("Const_3:0", shape=(2, 2), dtype=string)

11

TensorFlow program in two partsConstruction Phase

Build a graphDeclare Variables, Placeholders, Operations,Optimizers

Execution PhaseInitialize model and runStart session and iterate through the graph

12

What is TensorFlow?Consturction PhaseExecution PhaseExamplesGotchas and Tips

Contents

13

VariableStores a scalar or tensorUsed as an input for OperationsRequires initial value for future useAt constuction phase, variables are "just" defined.

Value will be assigned a�er SessioninitializedYou will get an idea through slides

14

Variableimport tensorflow as tf

# Define variable, yet initialized w = tf.Variable(tf.random_normal([1, 3]), name="weight")

with tf.Session() as sess:

w_value = sess.run(w) # get value for var 'w'

print '\n'.join(map(str, ([w, w_value])))

"tensorflow.python.framework.errors_impl. FailedPreconditionError: Attempting to use uninitialized value weight"

15

Variableimport tensorflow as tf

# Define variable, yet initialized w = tf.Variable(tf.random_normal([1, 3]), name="weight")# Define init operation init_op = tf.global_variables_initializer()

with tf.Session() as sess: sess.run(init_op) # initialize all declared variables w_value = sess.run(w) # get value for var 'w'

print '\n'.join(map(str, ([w, w_value])))

>>> Tensor("weight/read:0", shape=(2, 3), dtype=float32)>>> [[-0.50282216 -1.94712341 -0.47356459]]

16

Variableimport tensorflow as tf

# Define variable, yet initialized w = tf.Variable(tf.random_normal([1, 3]), name="weight")# Define init operation init_op = tf.global_variables_initializer()

with tf.Session() as sess: sess.run(init_op) # initialize all declared variables w_value = sess.run(w); x_value = sess.run(w) sess.run(init_op) # re-initialize y_value = sess.run(w)

print '\n'.join(map(str, ([w_value, x_value, y_value])))

>>> [[-0.50282216 -1.94712341 -0.47356459]] >>> [[-0.50282216 -1.94712341 -0.47356459]] # same >>> [[ 1.1028192 0.25910807 2.37004709]] # changed

17

PlaceholderPlaceholder for tensorMust be fed with data on execution for operationthat takes the placeholder as operand

18

Placeholderimport tensorflow as tf

x = [[1, 2]] y = [[1], [3]] pla = tf.placeholder(tf.int32, (1,2)) matmul = tf.matmul(pla, y)

with tf.Session() as sess: A = sess.run(matmul)

print '\n'.join(map(str, ([pla, A])))

>>> "You must feed a value for placeholder tensor 'Placeholder' with dtype int32 and shape [1,2]"

19

Placeholderimport tensorflow as tf

x = [[1, 2]] y = [[1], [3]] pla = tf.placeholder(tf.int32, (1,2)) matmul = tf.matmul(pla, y)

with tf.Session() as sess: A = sess.run(matmul, feed_dict={pla : x})

print '\n'.join(map(str, ([pla, A])))

>>> Tensor("Placeholder:0", shape=(2, 3), dtype=int32) >>> [[7]]

20

Shape of PlaceholderShape could be partially known

import tensorflow as tf

p1 = tf.placeholder(tf.float32, [1, None])p2 = tf.placeholder(tf.float32, [None, 2])p3 = tf.placeholder(tf.float32, [None, None]) ops = [tf.multiply(x, 10) for x in [p1, p2, p3]] mat1 = [[1, 2]] mat2 = [[3, 4], [5, 6]] mat3 = [[1], [2]]

with tf.Session() as sess: print '\n'.join(map(str, sess.run(ops, feed_dict={ p1:mat1, p2:mat2, p3:mat3})))

>>> [[ 10. 20.]] >>> [[ 30. 40.] [ 50. 60.]] >>> [[ 10.] [ 20.]]

21

Shape of PlaceholderUseful when you feed input/output to your model

without giving the size of batchimport tensorflow as tf

# Placeholder MNIST # image size 28x28 x = tf.placeholder(tf.float32, [None, 784]) # 10 classes [0..9] y_ = tf.placeholder(tf.float32, [None, 10])

# Softmax regression Wx = b W = tf.Variable(tf.zeros([784, 10])) b = tf.Variable(tf.zeros([10])) y = tf.nn.softmax(tf.matmul(x, W) + b)

...

22

OperationOperation is graph node that performs computationon tensors

Computation should be run onSession

It takes inputs and generates outputs(optional)Usually, it ends up with prediction, cost function, orupdating variables

We have already seen some operations through the slides

23

OperationLinear Regression (Mean square error)

xW + b = y,L = ( W −12m′ ∑

m′

i=1 xi yi)2

import tensorflow as tf, numpy as np # tf Graph Input X = tf.placeholder("float") Y = tf.placeholder("float")

# Set model weights W = tf.Variable(np.random.randn(), name="weight") b = tf.Variable(np.random.randn(), name="bias")

# Construct a linear model pred = tf.add(tf.multiply(X, W), b)

# Mean square error cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples)

24

OperationIt is better to read through documentation so that you

don't have to reinvent the wheel

Mathmatmul, reduce_sum, argmax, norm...

Assignassign, assign_add, assign_sub...

Tensor transformslice, split, stack, concat...

25

OptimizerProvides methods to compute gradients for a lossand apply gradients to variablesGetting cost function as input and

CalcualtegradientsProcess gradientsApply gradients

"minimize" method on Optmizers do all 3 stepsMany optimizers are alreafy implemented

Gradient Descent Optimizer, Adagrad Optimizer, RMSProp Optmizer...

26

Optimizerimport tensorflow as tf, numpy as np ... # Construct a linear model pred = tf.add(tf.multiply(X, W), b)

# Mean square error cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples) # Gradient descent optimizer = tf.train\ .GradientDescentOptimizer(learning_rate)\ .minimize(cost)

with tf.Session() as sess: for epoch in range(training_epochs): for (x, y) in zip(train_X, train_Y): # SGD m'=1 sess.run(optimizer, feed_dict={X: x, Y: y}) # update weights

27

What is TensorFlow?Consturction PhaseExecution PhaseExamplesGotchas and Tips

Contents

28

SessionSession launchs the graph, initalize them, and runoperations as you definedWe have seen sessions through the slide, and I hopeyou got the idea of what it is

29

SessionYour data will be lost when you close the sessionimport tensorflow as tf

var = tf.Variable(tf.random_normal((1, 3))) init = tf.global_variables_initializer()

with tf.Session() as sess: sess.run(init) print sess.run(var) # Fetch train data # Calcuate cost # Optimize # Do Test # Display or Save print var.eval()

"No default session is registered. Use `with sess.as_default()` or pass an explicit session to ..."

30

What is TensorFlow?Consturction PhaseExecution PhaseExamplesGotchas and Tips

Contents

31

MNISTFamous example for deep learning

Data: Hand-writing digit[0-9] image data(28*28)

Each image will be treated as 784 length ofvector

Task: Classify what digit in each image10classes

Model: So�max regression modelAlso use 3 hiddenlayers

32

MNISTx = set of images

y = set of labels

laye = x +r1 W1 b1

laye = laye +r2 W2 r1 b2

pred = laye +W3 r2 b3

cost = y ∗ log(softmax(pre − ))1m′ ∑

m′

i=1 di yi33

Hyper parameters

Reference:

# Parameters learning_rate = 0.001 training_epochs = 15 batch_size = 100 display_step = 1

# Network Parameters n_hidden_1 = 256 # 1st layer number of features n_hidden_2 = 256 # 2nd layer number of features n_input = 784 # MNIST data input (img shape: 28*28) n_classes = 10 # MNIST total classes (0-9 digits)

TensorFlow Exmaples

34

Placeholder & Variablesimport tensorflow as tf

# tf Graph input x = tf.placeholder("float", [None, n_input]) # input y = tf.placeholder("float", [None, n_classes]) # output

# Store layers weight & bias h1 = tf.Variable(tf.random_normal([n_input, n_hidden_1])) h2 = tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])) out = tf.Variable(tf.random_normal([n_hidden_2, n_classes]))

b1 = tf.Variable(tf.random_normal([n_hidden_1])) b2 = tf.Variable(tf.random_normal([n_hidden_2])) b3 = tf.Variable(tf.random_normal([n_classes]))

35

36

Build a model# Hidden layer 1 with RELU activation layer_1 = tf.add(tf.matmul(x, h1), b1) layer_1 = tf.nn.relu(layer_1)

# Hidden layer 2 with RELU activation layer_2 = tf.add(tf.matmul(layer_1, h2), b2) layer_2 = tf.nn.relu(layer_2)

# Output layer with linear activation pred = tf.matmul(layer_2, out) + b3

37

38

Optimizer# Define loss and optimizer y_hat_softmax = tf.nn.softmax(pred) cost = tf.reduce_mean( -tf.reduce_sum(y * tf.log(y_hat_softmax), [1])) optimizer = tf.train.AdamOptimizer( learning_rate=learning_rate)\ .minimize(cost)

# Initializing the variables init = tf.global_variables_initializer()

39

40

Run & Learn# Launch the graph with tf.Session() as sess: sess.run(init)

# Training cycle for epoch in range(training_epochs): avg_cost = 0. total_batch = int(mnist.train.num_examples/batch_size) # Loop over all batches for i in range(total_batch): batch_x, batch_y = mnist.train.next_batch(batch_size) # Run optimization op (backprop) and cost op (to get loss value) _, c = sess.run([optimizer, cost], feed_dict={x: batch_x, y: batch_y}) # Compute average loss avg_cost += c / total_batch # Display logs per epoch step if epoch % display_step == 0: print "Epoch:", '%04d' % (epoch+1), "cost=", \ "{:.9f}".format(avg_cost) print "Optimization Finished!"

# Test model correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1)) # Calculate accuracy accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float")) print "Accuracy:", accuracy.eval({x: mnist.test.images, y: mnist.test.labels})

41

Result>>> ... >>> Epoch: 0010 cost= 3.325720582 >>> Epoch: 0011 cost= 2.454958990 >>> Epoch: 0012 cost= 1.861004718 >>> Epoch: 0013 cost= 1.412745127 >>> Epoch: 0014 cost= 1.158697252 >>> Epoch: 0015 cost= 0.943136121 >>> Optimization Finished! >>> Accuracy: 0.9485

42

What is TensorFlow?Consturction PhaseExecution PhaseExamplesGotchas and Tips

Contents

43

Gotchas & TipsDon't forget to put value to feed_dict

import tensorflow as tf

pla = tf.placeholder(tf.int32, [2,2]) x = tf.Variable([[1,2],[3,4]]) y = tf.Variable([[3,4],[1,2]]) init = tf.global_variables_initializer() matmul = tf.matmul(pla, y)

with tf.Session() as sess: sess.run(init)

f_dict = { pla: x } # x is actually tensor "object" result = sess.run(matmul, feed_dict=f_dict) print result

"ValueError: setting an array element with a sequence."

44

Gotchas & TipsDon't forget to put value to feed_dict

import tensorflow as tf

pla = tf.placeholder(tf.int32, [2,2]) x = tf.Variable([[1,2],[3,4]]) y = tf.Variable([[3,4],[1,2]]) init = tf.global_variables_initializer() matmul = tf.matmul(pla, y)

with tf.Session() as sess: sess.run(init) x_v = sess.run(x) f_dict = { pla: x_v } # x_v is now "array(2,2)" result = sess.run(matmul, feed_dict=f_dict) print result

>>> [[ 5 8] [13 20]]

45

Gotchas & TipsBetter not to make Variable/Placeholder/Operation

inside epochimport tensorflow as tf

with tf.Session() as sess: for epoch in range(100): # It will make session overflow! v = tf.Variable([[1,2]]) op = tf.add(1,2) pla = tf.placeholder(tf.int32, [1,2])

print '\n'.join(map(str, [v, op, pla]))

>>> ... >>> Tensor("Variable_99/read:0", shape=(1, 2), dtype=int32) >>> Tensor("Add_99:0", shape=(), dtype=int32) >>> Tensor("Placeholder_99:0", shape=(1, 2), dtype=int32)

46

Useful ReferencesTensorFlow-ExamplesDCGAN-tensorflowStyle Transfer inTensorFlowWord2vec in TensorFlow

47

Q & A

48