[1]Deep Learning and TensorFlow – Episode 1

Deep Learningand TensorFlowEpisode 1Artificial Neural Networks

Università degli Studi di Pavia

[2]Deep Learning and TensorFlow – Episode 1

Main References for the This Course

[3]Deep Learning and TensorFlow – Episode 1

Google Cloud Platform Coupon

•

•

•

[4]Deep Learning and TensorFlow – Episode 1

Prologue:"Deep Learning"?

[5]Deep Learning and TensorFlow – Episode 1

Deep Learning in the Artificial Intelligence Cosmos

[6]Deep Learning and TensorFlow – Episode 1

AI strikes back?

[7]Deep Learning and TensorFlow – Episode 1

Artificial Perception

[8]Deep Learning and TensorFlow – Episode 1

Artificial Neural NetworksThe Origins

[9]Deep Learning and TensorFlow – Episode 1

Emulation or simulation? Connectionism

[10]Deep Learning and TensorFlow – Episode 1

Artificial Neural Network

[11]Deep Learning and TensorFlow – Episode 1

Function approximation:Linear Combination

[12]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

d

[13]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[14]Deep Learning and TensorFlow – Episode 1

•

•

•

Function Approximation: linear combination

[15]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[16]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[17]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[18]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[19]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[20]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[21]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[22]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[23]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[24]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[25]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[26]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[27]Deep Learning and TensorFlow – Episode 1

Function Approximation: linear combination

[28]Deep Learning and TensorFlow – Episode 1

Function approximation:Feed-Forward Neural Network

[29]Deep Learning and TensorFlow – Episode 1

Feed-Forward Neural Network

h d

[30]Deep Learning and TensorFlow – Episode 1

Feed-Forward Neural Network

[31]Deep Learning and TensorFlow – Episode 1

Feed-Forward Neural Network

[32]Deep Learning and TensorFlow – Episode 1

Feed-Forward Neural Network

[33]Deep Learning and TensorFlow – Episode 1

Feed-Forward Neural Network

[34]Deep Learning and TensorFlow – Episode 1

Universality of FF Neural Networks

[35]Deep Learning and TensorFlow – Episode 1

x W

Universality of FF Neural Networks

[36]Deep Learning and TensorFlow – Episode 1

i j

Universality of FF Neural Networks

[37]Deep Learning and TensorFlow – Episode 1

LearningFeed-Forward Neural Networks

[38]Deep Learning and TensorFlow – Episode 1

Learning with FF Neural Networks

[39]Deep Learning and TensorFlow – Episode 1

Learning with FF Neural Networks

[40]Deep Learning and TensorFlow – Episode 1

Gradient Descent (GD): intuition

D

[41]Deep Learning and TensorFlow – Episode 1

Gradient Descent (GD): convergence

[42]Deep Learning and TensorFlow – Episode 1

Gradient Descent (GD): practicalities

[43]Deep Learning and TensorFlow – Episode 1

Gradient Descent (GD): practicalities

[44]Deep Learning and TensorFlow – Episode 1

Gradient Descent for FF Neural Networks

[45]Deep Learning and TensorFlow – Episode 1

Gradient Descent for FF Neural Networks

[46]Deep Learning and TensorFlow – Episode 1

Computing Gradients

[47]Deep Learning and TensorFlow – Episode 1

Flow Graphs

[48]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[49]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[50]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[51]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[52]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[53]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[54]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[55]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[56]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[57]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[58]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[59]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[60]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[61]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[62]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[63]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[64]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[65]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[66]Deep Learning and TensorFlow – Episode 1

An aside: Flow Graph

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[67]Deep Learning and TensorFlow – Episode 1

Differentiation ofFlow Graphs

[68]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[69]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[70]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L

sub

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

[71]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

mul

2

sqr

[72]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

mul

2

sqr

[73]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

mul

2

sqr

[74]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

mul

2

sqr

[75]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

mul

2

mul

1

[76]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

wmul

0.5

sum

abs

sum

b dot

W x

mul

2

mul

1 mul

1

[77]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

w

dot

mul

0.5

sum

abs

sum

b dot

W x

mul

2

mul

1 mul

1

[78]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

w

dot

mul

0.5

sum

abs

sum

b dot

W x

mul

2

mul

1 mul

1

mul

0.5

[79]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

w

dot

mul

0.5

sum

abs

sum

b dot

W x

mul

2

mul

1 mul

1

mul

0.5 sum

mul

1

mul

1

[80]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

w

dot

mul

0.5

sum

abs

div

sum

b dot

W x

mul

2

mul

1 mul

1

mul

0.5 sum

mul

1

dot

mul

1

[81]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

w

dot

mul

0.5

sum

abs

div

sum

b dot

W x

mul

2

mul

1 mul

1

mul

0.5 sum

mul

1mul

dot

1

mul

1

[82]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

w

dot

mul

0.5

sum

abs

div

sum

b dot

W x

mul

2

mul

1 mul

1

mul

0.5 sum

mul

1mul

dot

1

mul

1

[83]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

w

dot

mul

0.5

sum

abs

div

sum

b dot

W x

mul

2

mul

1 mul

1

mul

0.5 sum

mul

1mul

dot

1

mul

1

[84]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

w

dot

mul

0.5

sum

abs

div

sum

b dot

W x

mul

2

mul

1 mul

1

mul

0.5 sum

mul

1mul

dot

1

mul

1

[85]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

w

dot

mul

0.5

sum

abs

div

sum

b dot

W x

mul

2

mul

1 mul

1

mul

0.5 sum

mul

1mul

dot

1

mul

1

[86]Deep Learning and TensorFlow – Episode 1

Computing Gradients

sqr

L grad L

sub

mul

y sum

b

dot

w

dot

mul

0.5

sum

abs

div

sum

b dot

W x

mul

2

mul

1 mul

1

mul

0.5 sum

mul

1mul

dot

1

mul

1

[87]Deep Learning and TensorFlow – Episode 1

LearningFeed-Forward Neural Networks

(contd.)

[88]Deep Learning and TensorFlow – Episode 1

Function Approximation: FF Neural Networks

[89]Deep Learning and TensorFlow – Episode 1

Function Approximation: FF Neural Networks

[90]Deep Learning and TensorFlow – Episode 1

Stochastic Gradient Descent for FF Neural Networks

[91]Deep Learning and TensorFlow – Episode 1

Stochastic Gradient Descent (SGD): convergence

[92]Deep Learning and TensorFlow – Episode 1

Speed of Convergence

[93]Deep Learning and TensorFlow – Episode 1

Mini-batch Gradient Descent for FF Neural Networks

[94]Deep Learning and TensorFlow – Episode 1

Qualitative comparison of GD methods

•

•

•

[95]Deep Learning and TensorFlow – Episode 1

(Mini) Batchesin Matrix Form

[96]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms

[97]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms

[98]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms

N

[99]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms

N

[100]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms

[101]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms

[102]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms

[103]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms

b

[104]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms

b

[105]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms

b

[106]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms

b

[107]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms

[108]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms

[109]Deep Learning and TensorFlow – Episode 1

More on Matrix Forms