Upload File

deep learning and tensorflow - paviadeep learning: a theoretical introduction –episode 2 [1] deep...

  • Home
  • Documents
  • Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli
60
[1] Deep Learning: a theoretical introduction – Episode 2 Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli Studi di Pavia

Upload: others

Post on 20-May-2020

24 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[1]Deep Learning: a theoretical introduction – Episode 2

Deep Learningand TensorFlowEpisode 2The Quest for Deeper Networks

Università degli Studi di Pavia

http://vision.unipv.it/AI/AIRG.html
Page 2: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[2]Deep Learning: a theoretical introduction – Episode 2

Feed-Forward Neural Network

Page 3: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[3]Deep Learning: a theoretical introduction – Episode 2

Feed-Forward Neural Network

Page 4: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[4]Deep Learning: a theoretical introduction – Episode 2

Training Feed-Forward Neural Networks

Page 5: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[5]Deep Learning: a theoretical introduction – Episode 2

The Quest forDeeper Networks

Page 6: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[6]Deep Learning: a theoretical introduction – Episode 2

Shallow vs. Deep Feed-Forward Neural Networks

Page 7: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[7]Deep Learning: a theoretical introduction – Episode 2

Shallow vs. Deep Feed-Forward Neural Networks

Page 8: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[8]Deep Learning: a theoretical introduction – Episode 2

Shallow vs. Deep Feed-Forward Neural Networks

Page 9: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[9]Deep Learning: a theoretical introduction – Episode 2

Shallow vs. Deep Feed-Forward Neural Networks

Page 10: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[10]Deep Learning: a theoretical introduction – Episode 2

Parity Circuits

Page 11: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[11]Deep Learning: a theoretical introduction – Episode 2

Parity Circuits

Page 12: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[12]Deep Learning: a theoretical introduction – Episode 2

Parity Circuits

Page 13: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[13]Deep Learning: a theoretical introduction – Episode 2

Parity Circuits

Page 14: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[14]Deep Learning: a theoretical introduction – Episode 2

Parity Circuits

Page 15: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[15]Deep Learning: a theoretical introduction – Episode 2

Depth and piecewise linear functions

Page 16: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[16]Deep Learning: a theoretical introduction – Episode 2

Depth and piecewise linear functions

Page 17: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[17]Deep Learning: a theoretical introduction – Episode 2

k > 2 h(2)

Depth and piecewise linear functions

Page 18: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[18]Deep Learning: a theoretical introduction – Episode 2

h

k h

pmax d k

Depth and piecewise linear functions

Page 19: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[19]Deep Learning: a theoretical introduction – Episode 2

About why they did not useDeep Networks

from the beginning

Page 20: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[20]Deep Learning: a theoretical introduction – Episode 2

Problem: vanishing or exploding Gradients

Page 21: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[21]Deep Learning: a theoretical introduction – Episode 2

Problem: vanishing or exploding Gradients

Page 22: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[22]Deep Learning: a theoretical introduction – Episode 2

• g

•

• W(i)

Problem: vanishing or exploding Gradients

k

Page 23: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[23]Deep Learning: a theoretical introduction – Episode 2

Problem: initial values of the parameters

Page 24: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[24]Deep Learning: a theoretical introduction – Episode 2

A bag of wonderful tricks

Page 25: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[25]Deep Learning: a theoretical introduction – Episode 2

Why ReLU is better (sometimes)

Page 26: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[26]Deep Learning: a theoretical introduction – Episode 2

•

Why ReLU is better (sometimes)

Page 27: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[27]Deep Learning: a theoretical introduction – Episode 2

•

•

Why ReLU is better (sometimes)

Page 28: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[28]Deep Learning: a theoretical introduction – Episode 2

Overfitting

Page 29: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[29]Deep Learning: a theoretical introduction – Episode 2

Dropout

Page 30: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[30]Deep Learning: a theoretical introduction – Episode 2

Dropout

Page 31: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[31]Deep Learning: a theoretical introduction – Episode 2

Dropout

Page 32: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[32]Deep Learning: a theoretical introduction – Episode 2

Dropout

Page 33: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[33]Deep Learning: a theoretical introduction – Episode 2

•

•

Contrasting Overfitting

Page 34: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[34]Deep Learning: a theoretical introduction – Episode 2

Improving on MBGD

Page 35: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[35]Deep Learning: a theoretical introduction – Episode 2

Improving on MBGD

Page 36: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[36]Deep Learning: a theoretical introduction – Episode 2

Improving on MBGD

Page 37: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[37]Deep Learning: a theoretical introduction – Episode 2

AdaGrad

Page 38: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[38]Deep Learning: a theoretical introduction – Episode 2

AdaGrad

B

Page 39: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[39]Deep Learning: a theoretical introduction – Episode 2

AdaGrad

a1 a2

Page 40: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[40]Deep Learning: a theoretical introduction – Episode 2

•

•

AdaGrad

d

Page 41: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[41]Deep Learning: a theoretical introduction – Episode 2

•

•

AdaGrad

d

Page 42: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[42]Deep Learning: a theoretical introduction – Episode 2

AdaGrad

Page 43: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[43]Deep Learning: a theoretical introduction – Episode 2

AdaDelta

Page 44: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[44]Deep Learning: a theoretical introduction – Episode 2

Improving on MBGD

Page 45: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[45]Deep Learning: a theoretical introduction – Episode 2

Improving on MBGD

Page 46: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[46]Deep Learning: a theoretical introduction – Episode 2

Improving on MBGD

Page 47: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[47]Deep Learning: a theoretical introduction – Episode 2

An aside:function approximation vs. classification

Page 48: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[48]Deep Learning: a theoretical introduction – Episode 2

Classification: Softmax

Page 49: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[49]Deep Learning: a theoretical introduction – Episode 2

Classification: Softmax

Page 50: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[50]Deep Learning: a theoretical introduction – Episode 2

Classification: Softmax

i

Page 51: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[51]Deep Learning: a theoretical introduction – Episode 2

Classification: Softmax

Page 52: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[52]Deep Learning: a theoretical introduction – Episode 2

Classification: Softmax

Page 53: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[53]Deep Learning: a theoretical introduction – Episode 2

Classification: Softmax

Page 54: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[54]Deep Learning: a theoretical introduction – Episode 2

Classification: Softmax

m wl

Page 55: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[55]Deep Learning: a theoretical introduction – Episode 2

Classification: Softmax

Page 56: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[56]Deep Learning: a theoretical introduction – Episode 2

Classification: Softmax

Page 57: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[57]Deep Learning: a theoretical introduction – Episode 2

Classification: Softmax

h

Page 58: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[58]Deep Learning: a theoretical introduction – Episode 2

Another aside:autoencoders

Page 59: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[59]Deep Learning: a theoretical introduction – Episode 2

Auto-encoders

Page 60: Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

[60]Deep Learning: a theoretical introduction – Episode 2

Auto-encoders


Introduction to tensorflow · Introduction to tensorflow. Why do you need a deep learning framework? Speed: ... - Deep-learning/Machine learning specific - Simpler to do common tasks

Deep learning with Tensorflow in R

TensorFlow A system for large-scale machine learning · 2018-10-31 · TensorFlow–A system for large-scale machine learning. Introduction to TensorFlow ... number of machine learning

On-device machine learning: TensorFlow on Android

TensorFlow - tutorialspoint.com · TensorFlow is an open source machine learning framework for all developers. It is used for implementing machine learning and deep learning applications

Deep Learning Lab11: TensorFlow 101€¦ · TensorFlow 4 Read and Preprocess Data tf.keras Premade Estimators TensorFlow Hub Distribution Strategy GPU CPU TPU SavedModel TensorFlow

Learning in TensorFlow Deep Reinforcement - DanijarLearning in TensorFlow Danijar Hafner · Stanford CS 20SI · 2017-03-10. Gu16. Barron16. ... Sketch out implementations in TensorFlow

TensorFlow: Large-Scale Machine Learning on Heterogeneous

TensorFlow and Keras - Statinfer...Contents •Deep Learning frameworks •What is TensorFlow •Key terms in TensorFlow •Working with TensorFlow •Regression Model building •MNIST

Introduzione Deep Learning & TensorFlow

Welcome to TensorFlow! - Stanford University...Welcome to TensorFlow! CS 20SI: TensorFlow for Deep Learning Research Lecture 1 1/13/2017 1 2 Agenda Welcome Overview of TensorFlow Graphs

Deep-Learning-TensorFlow Documentation - Read the · PDF fileDeep-Learning-TensorFlow Documentation, Release latest Thisprojectis a collection of various Deep Learning algorithms implemented

Intro to Machine Learning with TensorFlow Nanodegree

TensorFlow Lite for microcontrollers - tinyML · TensorFlow Lite for microcontrollers Deploy machine learning models on tiny devices 1 Daniel Situnayake @dansitu. TensorFlow Lite

Deep-Learning-TensorFlow Documentation - Read the · PDF fileDeep-Learning-TensorFlow Documentation, Release stable This repository is a collection of various Deep Learning algorithms

Università degli Studi di Pavia Deep Learning and TensorFlow · Deep Learning and TensorFlow –Episode 4 [1] Deep Learning and TensorFlow Episode 4 TensorFlow Basics Part 1 Università

Deep Learning for Computer Vision with Tensorflow · What is TensorFlow? •TensorFlow was originally developed by researchers and engineers working on the Google Brain Team. •TensorFlow

Large Scale Deep Learning with TensorFlow

Applying Transfer Learning in TensorFlow

Deep Learning with Keras and TensorFlow

Deep Learning and TensorFlow - Pavia€¦ · Deep Learning and TensorFlow–Episode 1 [1] ... Artificial Neural Networks Università degli Studi di Pavia. Deep Learning and TensorFlow–Episode

Machine learning and TensorFlow

Machine Learning in PandaRoot - jlab.org · • TensorFlow (Deep Learning) • Keras (on top of TensorFlow with GPUs) • MLlib Apache Spark • Sci-Kit Learn • PyTorch (Deep Learning)

TENSORFLOW - ajou.ac.krcvml.ajou.ac.kr/wiki/images/5/55/Ch4_1_TensorFlow.pdf · 2017-01-12 · What is TensorFlow? •TensorFlow is a deep learning library open-sourced by Google

Learning in TensorFlow Deep ReinforcementLearning in TensorFlow Danijar Hafner · Stanford CS 20SI · 2017-03-10. Gu16. Barron16. Hafner16. Repeat until end of episode: Most methods

Image Recognition with Deep Learning Techniques and TensorFlow

Chomsky Go with the TensorFlow · All Done: TensorFlow 2.0 4 TensorFlow 2.0 offers a good introduction to deep learning Hoang Tu Nguyen Explainable Machine Learning with Python and

Scalable TensorFlow Learning on Spark Clusters€¦ · Scalable TensorFlow Learning on Spark Clusters . What is TensorFlowOnSpark . Why TensorFlowOnSpark at Yahoo? • Major contributor

TensorFlow Input Pipeline - Stanford Universityweb.stanford.edu/class/cs20si/lectures/slides_09.pdf · TensorFlow Input Pipeline CS 20SI: TensorFlow for Deep Learning Research Lecture

Deep Learning and TensorFlow...Deep Learning: a theoretical introduction –Episode 3 [1]Deep Learning and TensorFlow Episode 3 Deep Convolutional Neural Networks Università degli

Introduction to TensorFlow, by Machine Learning at Berkeley

- Deep learning hardware - PyTorch and TensorFlow

Deep Learning, Keras, and TensorFlow

Introduction to Machine Learning with TensorFlow

Deep Learning with Tensorflow