iwann 2019 transfer learning tutorialiwann.uma.es/wp-content/uploads/transfer_learning2.pdf · deep...

Transfer learning tutorial

Dario Garcia Gasulla [email protected] Vilalta Arias [email protected]

IWANN 2019

Transfer Learning

“Don’t be a hero”Andrej Karpathy

3

Your first driving lesson

4


Imagine learning to drive a car without knowing absolutely nothing about anything


5

Randomly initialized

Deep Neural Network

6

Your first driving lesson Any previous learning can be useful

Knowing how to cook is better than knowing nothing at all

7

Your first driving lesson Any previous learning can be useful

Knowing how to cook is better than knowing nothing at all

We naturally reuse what we previously learnt to be able to solve a new task.

How transferlearning emerged

8

9

Image classification

• 1998 LeNet-5

Gradient-based learning applied to document recognition.

Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner

Image classification

10

• 2012 AlexNet

ImageNet Classification with Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton

1998 LeNet-5

2012 AlexNet

2014 VGG19

2014 GoogLeNet

2015 Inception-V3

2015 ResNet-56

11

ImageNet classification results

12From image-net.org

13

At what price?

• Data available– 1,000 images per class

• Computational cost– Specific hardware

– Energy cost

• Human effort– Highly skilled professionals

– Architecture design

– Hyper-parameter fine tuning

14

At what price?



– Energy cost




We can’t do that for every single problem!!

15

At what price?



– Energy cost





16

At what price?



– Energy cost





→ Transfer Learning to the rescue

What is transfer learning?

17

18

What is learning about?

19


Train set

20


Train set

Test set

21


Train set

Test setMODEL

22


Train set

Test setMODEL

23


Train set

Test setMODEL

24


Train set

Test set

25


Train set

Test set

DOG

26


Train set

Test set

DOG

27


Train set

Test set

DOG

28


Train set

Test set

29


Train set

Test set

30


Train set

Test set

GENERALIZATION

31

What is transferlearning about?

Train set

Test set

32


GENERALIZATIONTrain set Test set

33



Train a Machine Learning Model on a train set with the hope that what has been learnt will be useful to solve a different task.

34



Train a Machine Learning Model on a train set with the hope that what has been learnt will be useful to solve a different task.

Formalizing transferlearning

35

Pan, Sinno Jialin, and Qiang Yang.

"A survey on transfer learning."

IEEE Transactions on knowledge

and data engineering (2010)

36

Formalizing transfer learning

Domain:

Task:

37


Domain: 𝒟 = {𝒳, 𝑃 𝑋 }

• A feature space 𝒳

• A marginal probability distribution 𝑃 𝑋 ,𝑤ℎ𝑒𝑟𝑒 𝑋 = 𝑥1, … , 𝑥𝑛 ∈ 𝒳

Task:

≠

➔ ≠“The Elgar Concert

Hall at the University

of Birmingham for

our third conference”

➔ Bag of words

➔ Content vector

38



• A feature space 𝒳

• A marginal probability distribution 𝑃 𝑋 ,𝑤ℎ𝑒𝑟𝑒 𝑋 = 𝑥1, … , 𝑥𝑛 ∈ 𝒳

Task: 𝒯 = {𝓎, 𝑓(·)}

• A label space 𝓎

CAT, DOG ≠ LION, WOLF

• An objective predictive function 𝑓 · ⟺ 𝑃(𝑦|𝑥)

≠

➔ ≠“The Elgar Concert

Hall at the University

of Birmingham for

our third conference”

➔ Bag of words

➔ Content vector

39

Formalizing transfer learning Source Target


Task: 𝒯 = {𝓎, 𝑓(·)}

40



• A feature space 𝒳– The Same (different)

• A marginal probability distribution 𝑃 𝑋– Different

– Similar

Task: 𝒯 = {𝓎, 𝑓(·)}

➔ ➔

41





– Similar

Task: 𝒯 = {𝓎, 𝑓(·)}

• A label space 𝓎– Different {CAT, DOG} {LION, WOLF}

– The same {FELINE, CANINE} {FELINE, CANINE}

• An objective predictive function 𝒇𝑺 · 𝒇𝑻 ·– Different (but similar?)

➔ ➔

42


Train set

Test set

43

What is transferlearning about? Source

domain

Target domain

44

What is transferlearning about? Source

domain

Target domain

45





– Similar

Task: 𝒯 = {𝓎, 𝑓(·)}




➔ ➔

46





– Similar

Task: 𝒯 = {𝓎, 𝑓(·)}




➔ ➔

47

Formalizing transferlearning

Source Target

𝒇𝑺 · 𝒇𝑻 ·

𝒇𝑺 · = 𝒇𝑻 · =

• Are they similar?

• Can we just use 𝒇𝑺 · to approximate 𝒇𝑻 · ?

• Can we reuse part of it?

Representation learning

48

49

Deep Neural Networks are representationlearning techniques

• Support Vector Machine (SVM) is just a classifier (a very good one).

• SVM find the best boundary separating the data instances into different classes in a given feature space.

50


• Support Vector Machine (SVM) is just a classifier (a very good one).

• SVM find the best boundary separating the data instances into different classes in a given feature space.

51


• SVMs using the kernel trick can overcome the linear limitation through an implicit mapping to a higher dimensional feature space

52

Deep Neural Networks are representation learning techniques

OutputInput

53


Linear

Classifier

OutputInput

54


Linear

Classifier

Intermediate

representations

OutputInput

Reusing DNNs knowledge

55DNNs


56

Feature Extraction

Fine-tuning


57

Feature Extraction

Fine-tuning

58

Feature extraction

Linear

Classifier

Intermediate

representations

OutputInput

59

Feature extraction

Linear

Classifier

Intermediate

representations

OutputInput

Pre-trained CNN, frozen weights

60

Feature extraction

Linear

Classifier

Intermediate

representations

OutputInput

61

Feature extraction

Intermediate

representations

Input

62

Feature extraction

Intermediate

representations

Input

S

V

M

Target

Task

Labels

63

Simple solutions

• DNN last layer features + SVM(Feature extraction)

64

Simple solutions


We need: Similar task and domain


65

Feature Extraction

Fine-tuning

66

Fine tuning

Linear

Classifier

Intermediate

representations

OutputInput

67

Fine tuning

Linear

Classifier

Intermediate

representations

OutputInput

What if the tasks

are quite different?

68

Fine tuning

Linear

Classifier

Intermediate

representations

OutputInput

Source

Task

What if the tasks

are quite different?

69

Fine tuning

Intermediate

representations

Input

Features learned

for the Source

Task

70

Fine tuning

Intermediate

representations

Input

Features learned

for the Source

Task

Can we make

them better?

71

Fine tuning

Intermediate

representations

Input

D

N

N

Target

Task

Labels

D

N

N

72

Fine tuning

Intermediate

representations

Input

D

N

N

Target

Task

Labels

D

N

N

Error back-propagation

73

Fine tuning

Intermediate

representations

Input

D

N

N

Target

Task

Labels

D

N

N


74

Fine tuning

Intermediate

representations

Input

D

N

N

Target

Task

Labels

D

N

N


75

Fine tuning

Intermediate

representations

Input

D

N

N

Target

Task

Labels

D

N

N


76

Fine tuning

Intermediate

representations

Input

D

N

N

Target

Task

Labels

D

N

N


77

Fine tuning

Intermediate

representations

Input

D

N

N

Target

Task

Labels

D

N

N


• Difficult to control how much do we re-train.

→ Reduced learning rate (1/10)

→ Early stopping

→ Alternate source/target sampling

78

Simple solutions



• Add one or several NN layers +

Fine-tuning pre-trained layers

79

Simple solutions




Fine-tuning pre-trained layersWe need: Enough data

80

Beyond the last layer

Linear

Classifier

Intermediate

representations

OutputInput

81

Knowledge inside DNN

82


83


Convolutional

filter

84


Image width

Image

height

# Filters

Depth

85


Image width

Image

height

# Filters

Depth

Spatial

Average

Pooling

86


Image width

Image

height

# Filters

Depth

Spatial

Average

Poolingfi

fifi

f1

87

Knowledge inside DNNConv3_3 FC7

Spatial

Average

Pooling

88


Spatial

Average

Pooling

Conv3_3 FC7

89


90


Task

Labels

Input

Domain More

influenced by

domain

More

influenced by

task

91


Task

Labels

Input

Domain More

influenced by

domain

Simple

features

More

influenced by

task

Complex

features

Visualizations from: Yosinski, Jason, et al. "Understanding neural networks through deep visualization."

arXiv preprint arXiv:1506.06579 (2015).

92

Fine tuning inside DNN?

Fine-tuning

Conv3_3 FC7

Task

Labels

Input

Domain

93


Fine-tuning

Conv3_3 FC7

Task

Labels

Input

Domain

94


Fine-tuning

Conv3_3 FC7

Task

Labels

Input

Domain

95

Feature extraction inside DNN?Conv3_3 FC7

Task

Labels

Input

Domain

Feature extraction

96


Task

Labels

Input

Domain

Feature extraction

97


Task

Labels

Input

Domain

Feature extraction

98


Task

Labels

Input

Domain

Feature extraction

99


Task

Labels

Input

Domain

Feature extraction

VGG16 dim: 12,416

100


Task

Labels

Input

Domain

Feature extraction Different range

of values

VGG16 dim: 12,416

101

Feature behavior in Transfer LearningConv3_3 FC7

0 50 100 150 200 250 300 350

102

Standardization: features in same range

0 50 100 150 200 250 300 350

L2 n

orm

Typically, it is used

a L2-normalization

0,00 0,13 0,27 0,40 0,54 0,67 0,80 0,94

103


Featu

re

Sta

nd

ard

isati

on

Each feature independently

-5 -4 -3 -2 -1 0 1 2 3 4 50 50 100 150 200 250 300 350

L2 n

orm


a L2-normalization

0,00 0,13 0,27 0,40 0,54 0,67 0,80 0,94

104


Featu

re

Sta

nd

ard

isati

on

Each feature independently

-5 -4 -3 -2 -1 0 1 2 3 4 50 50 100 150 200 250 300 350

L2 n

orm


a L2-normalization

0,00 0,13 0,27 0,40 0,54 0,67 0,80 0,94

105

Standardization: features in contextConv3_3 FC7

• Distinct feature behavior for

a subset of data wrt rest

106

Stardardization: features in context

CUB-200 - birds

Garcia-Gasulla et al.

On the behavior of

convolutional nets for

feature extraction. 2017

107

Curse of dimensionalityConv3_3 FC7

Task

Labels

Input

Domain

VGG16 dim: 12,416

108


Task

Labels

Input

Domain

VGG16 dim: 12,416

PCA ?

109


Task

Labels

Input

Domain

VGG16 dim: 12,416

PCA

110


Task

Labels

Input

Domain

VGG16 dim: 12,416

PCA

• Unsupervised:

Only takes into account

the domain

111


Task

Labels

Input

Domain

VGG16 dim: 12,416

Supervised Feature Selection

?

112


Task

Labels

Input

Domain

VGG16 dim: 12,416

Supervised Feature Selection

?• High computational cost!

113


Task

Labels

Input

Domain

VGG16 dim: 12,416

114


Task

Labels

Input

Domain

VGG16 dim: 12,416

• Which is the real problem?

115


Task

Labels

Input

Domain

VGG16 dim: 12,416


• Too many features?

• Too few images?

116


Task

Labels

Input

Domain

VGG16 dim: 12,416



• Too few images? A requirement

117


Task

Labels

Input

Domain

VGG16 dim: 12,416



• Too much information!

• Too few images?

118

Meaningful discretizationConv3_3 FC7

Task

Labels

Input

Domain

FC7

Real

values

1 -1 0 1 1 0 0 0 1 -1 0 1 1 0 0 0 1 -1 0 1 1 0 0 0 1 -1 0 1 1 0 0 0 1 0 0 0 1 -1 0 1 1 0 0 0 1 -1 0 1 1

Quantitization to {-1,0,1} based on

feature values

Conv3_3

Discrete

values

119

Full-Network Embedding Recipe

1. Spatial Average Pooling

2. Standardisation

3. Discretization

Garcia-Gasulla, et al.

An out-of-the-box full-

network embedding for

convolutional neural

networks. 2018.

120

Full Network embedding

121

FNE - Results: Similar source task

Network pre-trained on Places2 for mit67 and on ImageNet for the rest.

122



123



+2.9

+4.2

124



-0.3 -0.4 -0.5

+2.9

+4.2

125



-0.3 -0.4 -0.5

+2.9

+4.2

126

FNE - Results: Dissimilar source task

Network pre-trained on ImageNet for mit67 and on Places2 for the rest.

127



128



+3.3 +11.9 +15.4 +17.5 +8.0 +9.3 +10.6

129

Simple solutions




Fine-tuning pre-trained layersWe need: Enough data

• Full Network Embedding– Robust to different task and domain

– Works with little data

Practicaltips

130

131

Practicaltips

Fine-tuning

• Whenever possible don’t start from scratch.

132

Practicaltips

Fine-tuning


• External data can help prevent overfitting (even from a different problem).

133

Practicaltips

Fine-tuning


• External data can help prevent overfitting (even from a different problem).

• Begin freezing as much as possible and proceed with

caution (particularly for large models)

134

Practicaltips

Feature

extraction

• Easy baseline for every problem.

135

Practicaltips

Feature

extraction


• ImageNet, a model to pre-train them all. (but not always)

136

Practicaltips

Feature

extraction



• Always normalize features.

137

Practicaltips

Feature

extraction




• If source and target task are closely related:

Last two layers are your best chance.

138

Practicaltips

Feature

extraction




• If source and target task are closely related:

Last two layers are your best chance.

• If source and target task are quite different:

Try everything

Use FNE

139

UsefulReferences • Zhe Xu, Shaoli Huang, Ya Zhang, and Dacheng Tao. Augmenting

strong supervision using webdata for fine-grained categorization.

2015.

• Steve Branson, Grant Van Horn, Serge Belongie, and Pietro

Perona. Bird species categorizationusing pose normalized deep

convolutional nets. 2014.

• Chang Liu, Yu Cao, Yan Luo, Guanling Chen, Vinod Vokkarane, and

Yunsheng Ma. Deep-food: Deep learning-based food image

recognition for computer-aided dietary assessment. 2016.

Don’t start training

from scratch

140

UsefulReferences • Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How

transferable are features in deep neural networks? 2014.

• Dario Garcia-Gasulla, Ferran Parés, Armand Vilalta, Jonatan

Moreno, Eduard Ayguadé, Jesús Labarta, Ulises Cortés, and

Toyotaro Suzumura. On the behavior of convolutional nets for

feature extraction. 2017Transfer learning 101

141

UsefulReferences

• Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, Atsuto

Maki, and Stefan Carlsson. Factors of transferability for a generic

convnet. 2016.

• Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and

Stefan Carlsson. CNN features off-the-shelf: an astounding baseline

for recognition. 2014.

• Yunchao Gong, Liwei Wang, Ruiqi Guo, and Svetlana Lazebnik.

Multi-scale orderless pooling of deep convolutional activation

features. 2014.

• Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning

Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional

activation feature for generic visual recognition. 2014.

• Arsalan Mousavian and Jana Kosecka. Deep convolutional features

for image based retrieval and scene categorization. 2015.

Basic feature

extraction

142

UsefulReferences • Garcia-Gasulla, Dario, Armand Vilalta, Ferran Parés, Eduard

Ayguadé, Jesus Labarta, Ulises Cortés, and Toyotaro Suzumura. An

out-of-the-box full-network embedding for convolutional neural

networks. 2018.

• Vilalta, Armand, Dario Garcia-Gasulla, Ferran Parés, Jonathan

Moreno, Eduard Ayguadé Jesús Labarta, Ulises Cortés, and

Toyotaro Suzumura. Full-Network Embedding in a Multimodal

Embedding Pipeline. 2017.

Multi-layer feature

extraction

143

UsefulReferences • Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy:

Deep transfer learning through selective joint fine-tuning. 2017.

• Marcel Simon and Erik Rodner. Neural activation constellations:

Unsupervised part model discovery with convolutional networks.

2015.Advanced fine tuning

to solve small tasks

CTE-Power9

144

Hands on session

16:00

2 GPUs per account

52 computing nodes. Each one:

• 2 x IBM Power9 8335-GTH @

2.4GHz(total 160 threads)

• 512GB of main memory

• 4 x GPU NVIDIA V100 (Volta)

with 16GB each

Available through PRACE and RES

[email protected]

[email protected]

145

iwann 2019 transfer learning tutorialiwann.uma.es/wp-content/uploads/transfer_learning2.pdf · deep...

Documents