Upload File

russell, 2009 - brown university...stochastic gradient descent try to speed up processing with...

  • Home
  • Documents
  • Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
87

Upload: others

Post on 27-Sep-2020

1 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 2: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Russell, 2009

Page 3: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Simultaneous contrast

Page 4: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 5: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 6: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Training Neural Networks

• Build network architecture

and define loss function

• Pick hyperparameters – learning rate, batch size

• Initialize weights + bias in each layer randomly

• While loss still decreasing• Shuffle training data

• For each data point i=1…n (maybe as mini-batch)• Gradient descent

• Check validation set loss

“Epoch“

Page 7: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Stochastic Gradient Descent

Try to speed up processing with random training subsets

Loss will not always decrease (locally) as training data point is random, but converges over time.

Wikipedia

Momentum

𝜽𝑡+1 = 𝜽𝑡 − 𝛾 𝛼𝜕𝐿

𝜕𝜽𝑡−1

+𝜕𝐿

𝜕𝜽𝑡

Gradient descent step size is weighted combination over time to dampen ping pong.

Page 8: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Regularization

• Penalize weights for simpler solution• Occam’s razor

• Dropout half of neurons foreach minibatch

• Forces robustness

𝜆

Page 9: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

But James…

…I thought we were going to treat machine learning like a black box? I like black boxes.

Deep learning is: - a black box - also a black art.

- Grad student gradient descent : (

http://www.isrtv.com/

Page 10: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Why do we initialize the weights randomly?

What if we set zero weights?

Setting zero weights makes all neurons equivalent as there is no difference in the gradient computed across neurons. Called “symmetric updates”.

Setting zero bias is OK.Typically, we standardize the data beforehand by subtracting the mean and dividing by std. dev.

Thus, a zero bias is a good initialization.

Where ҧ𝑥 is the mean of the input feature across the dataset (not spatially across the image)

Page 11: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Why do we initialize the weights randomly?

What if we set zero weights?

Wx = 0 for the layer output

-> produces uniform softmax output (each class equally probable; i.e., on MNIST, 0.1)

Gradient update rule and supervision label 𝑦𝑗 still provides the right signal

𝑝𝑗 vs 1 – 𝑝𝑗

…but all neurons not of class j receive same gradient update.

Page 12: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

So what is a good initialization of the weights?

In general, initialization is very important.

Good strategy: He et al. 2015:

For ReLU, draw random weights from Gaussian distribution with variance = 2 / # inputs to layer

[Image Andre Perunicic]

Page 13: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

What is activation function for?

To allow multiple layers; to avoid resulting composition of linear functions collapsing to a single layer.

Difference between CNN and convolution in feature extraction?

No difference! Same operation [correlation/convolution]

Why do we shave off pixels?

We only use the valid region of convolution; typically no padding.

Some recent works special case these edge convolutions.

Why multidimensional kernels?

We wish to convolve over the outputs of many other learned kernels -> ‘integrating’ information via weighted sum of previous layer outputs.

Page 14: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

How to know which kernels to use in 2nd+ convolution layers?

Gradient descent + back propagation learns them.

How to set weights on fully connected layers?

Gradient descent + back propagation learns them.

What even is back propagation again?

Computing the (change in) contribution of each neuron to the loss as the parameters vary.

Project 4 written has some good references.

Page 15: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

How do we decide the parameters for network architecture?For less complicated situations, we can use ‘trial and error’. Is there any other method?

‘Grid search’ -> trial and error

‘Bayesian optimization’ -> meta-learning; optimize the hyperparameters

General strategy:

Bottleneck -> extract information (kernel) and learn to squeeze representation into a smaller number of parameters.

But James – I happen to have a thousand GPUs:Neural Architecture Search!

Page 16: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

I’ve heard about many more terms of jargon!

Skip connections

Residual connections

Batch normalization

…we’ll get to these in a little while.

Page 17: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

When something is not working…

…how do I know what to do next?

Page 18: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

The Nuts and Bolts of Building Applications using Deep Learning• Andrew Ng - NIPS 2016

• https://youtu.be/F1ka6a13S9I

https://youtu.be/F1ka6a13S9I
Page 19: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Bias/variance trade-off

Scott Fortmann-Roe

Bias = accuracyVariance = precision

"It takes surprisingly long time to grok bias and variance deeply, but

people that understand bias and variance deeply are often able to

drive very rapid progress." --Andrew Ng

Page 20: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 21: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Go collect a dataset

• Most important thing:• Training data must represent target application!

• Take all your data• 60% training

• 40% testing• 20% testing

• 20% validation (or ‘development’)

Page 22: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Properties

• Human level error = 1%

• Training set error = 10%

• Validation error = 10.2%

• Test error = 10.4%

“Bias”

“Variance”

Overfitting to validation

Page 23: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

[Andrew Ng]

Val

Val

Val

Page 24: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

http://theorangeduck.com/page/neural-network-not-working

Daniel Holden

http://theorangeduck.com/page/neural-network-not-working
Page 25: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 26: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 27: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 28: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 29: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 30: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 31: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 32: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 33: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 34: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 35: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 36: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 37: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 38: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 39: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Object Detectors Emerge in Deep Scene CNNs

Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, AntonioTorralba

Massachusetts Institute of Technology

ICLR 2015

Page 40: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

How Objects are Represented in CNN?

CNN uses distributed code to represent objects.

Agrawal, et al. Analyzing the performance of multilayer neural networks for object recognition. ECCV, 2014

Szegedy, et al. Intriguing properties of neural networks.arXiv preprint arXiv:1312.6199, 2013.

Zeiler, M. et al. Visualizing and Understanding Convolutional Networks, ECCV 2014.

Conv2

Conv3

Conv4

Pool5

Conv1

Page 41: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Estimating the Receptive Fields

Estimated receptive fieldspool1

Actual size of RF is much smaller than the theoretic size

conv3 pool5

Segmentation using the RF of Units

More semantically meaningful

Page 42: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Annotating the Semantics of Units

Top ranked segmented images are cropped and sent to Amazon Turk for annotation.

Page 43: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Annotating the Semantics of Units

Pool5, unit 76; Label: ocean; Type: scene; Precision: 93%

Page 44: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Pool5, unit 13; Label: Lamps; Type: object; Precision: 84%

Annotating the Semantics of Units

Page 45: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Pool5, unit 77; Label:legs; Type: object part; Precision: 96%

Annotating the Semantics of Units

Page 46: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Pool5, unit 112; Label: pool table; Type: object; Precision: 70%

Annotating the Semantics of Units

Page 47: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Annotating the Semantics of Units

Pool5, unit 22; Label: dinner table; Type: scene; Precision: 60%

Page 48: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

ImageNet vs. PlacesNet

http://places2.csail.mit.edu/demo.html

ImageNet• ~1 mil object-level images

over 1000 classes

PlacesNet• ~1.8 million images from

365 scene categories (at most 5000 images per category).

http://places2.csail.mit.edu/demo.html
Page 49: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Distribution of Semantic Types at Each Layer

Page 50: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Distribution of Semantic Types at Each Layer

Object detectors emerge within CNN trained to classify scenes, without any object supervision!

Page 51: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

How can free will coexist with divine preordination?

Ah yes.

Related works:

- The Ontological Argument

- The Problem of Evil

- Ship of Theseus / Sorites Paradox

- What is Art

- 0.99 ሶ9 = 1

- Chinese Room AI

Page 52: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Short cuts to AI

With billions of images on the web, it’s often possible to find a close nearest neighbor.

We can shortcut hard problems by “looking up” the answer, stealing the labels from our nearest neighbor.

Page 53: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

So what is intelligence?

Weak AI: The simulation of a ‘mind’ is a model for the ‘mind’.

Strong AI: The simulation of a ‘mind’ is a ‘mind’.

Page 54: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Chinese Room experiment, John Searle (1980)

If a machine can convincingly simulate an

intelligent conversation, does it understand?

Page 55: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Chinese Room experiment, John Searle (1980)

Most of the discussion consists of

attempts to refute it.

"The overwhelming majority," notes BBS

editor Stevan Harnad,“ still think that the

Chinese Room Argument is dead wrong.“

The sheer volume of the literature that has

grown up around it inspired Pat Hayes to

quip that the field of cognitive science

ought to be redefined as "the ongoing

research program of showing Searle's

Chinese Room Argument to be false.”

If a machine can convincingly simulate an

intelligent conversation, does it understand?

Searle imagines himself in a room, acting

as a computer by manually executing a

program that convincingly simulates the

behavior of a native Chinese speaker.

Page 56: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point
Page 57: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Mechanical Turk

• von Kempelen, 1770.

• Robotic chess player.

• Clockwork routines.

• Magnetic induction (not vision)

• Toured the world; played Napoleon Bonaparte and Benjamin Franklin.

Page 58: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Mechanical Turk

• It was all a ruse!

• Ho ho ho.

Page 59: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

"Can machines fly?" Yes; aeroplanes exist.

"Can machines fly like a bird?" No, because aeroplanes don’t flap.

"Can machines perceive?”“Can machines understand?" Are these question like the first, or like the second?

[Adapted from Norvig]

Page 60: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Ornithopters

James Hays

Page 61: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Festo SmartBird [2011]

Page 62: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Interesting CNN properties…or other ways to measure reception

http://yosinski.com/deepvis

http://yosinski.com/deepvis
Page 63: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

What input to a neuron maximizes a class score?

Neuron of choice i

An image of random noise x.

Repeat:

1. Forward propagate: compute activation ai(x)

2. Back propagate: compute gradient at neuron ∂ai(x) / ∂x3. Add small amount of gradient back to noisy image.

Andrej Karpathy

To visualize the function of a specific unit in a neural network, we synthesize an input to that unit which causes high activation.

Page 64: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

What image maximizes a class score?

[Understanding Neural Networks Through Deep Visualization, Yosinski et al. , 2015]

http://yosinski.com/deepvis

Andrej Karpathy

http://yosinski.com/deepvis
Page 65: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

What image maximizes a class score?

Andrej Karpathy

Page 66: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Wow!

They just ‘fall out’!

Page 67: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Francois Chollet - https://blog.keras.io/the-limitations-of-deep-learning.html

https://twitter.com/fchollet
Page 68: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Breaking CNNs

Intriguing properties of neural networks [Szegedy ICLR 2014]Andrej Karpathy

http://arxiv.org/pdf/1312.6199v4.pdf
Page 69: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Breaking CNNs

Deep Neural Networks are Easily Fooled: High Confidence Predictionsfor

Unrecognizable Images [Nguyen et al. CVPR 2015]Jia-bin Huang

http://arxiv.org/pdf/1412.1897.pdf
Page 70: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Adversarial Patches

• https://www.theverge.com/2018/1/3/16844842/ai-computer-vision-trick-adversarial-patches-google

• https://arxiv.org/pdf/1712.09665.pdf

[Brown et al., Google, 2018]

https://www.theverge.com/2018/1/3/16844842/ai-computer-vision-trick-adversarial-patches-google
Page 71: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Francois Chollet - https://blog.keras.io/the-limitations-of-deep-learning.html

https://twitter.com/fchollet
Page 72: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Francois Chollet - https://blog.keras.io/the-limitations-of-deep-learning.html

https://twitter.com/fchollet
Page 73: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Curiosity: The success of obfuscating gradients• https://github.com/anishathalye/obfuscated-

gradients

https://github.com/anishathalye/obfuscated-gradients
Page 74: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Reconstructing images

Question: Given a CNN code, is it possible

to reconstruct the original image?

Andrej Karpathy

Page 75: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Reconstructing images

Find an image such that:

- Its code is like a given code

- It “looks natural”

- Neighboring pixels should look similar

Andrej Karpathy

Image

Page 76: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Reconstructing images

Understanding Deep Image Representations by Inverting Them

[Mahendran and Vedaldi, 2014]

original imageReconstructions

from the 1000

log probabilities

for ImageNet

(ILSVRC)

classes

Andrej Karpathy

Page 77: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Reconstructing images

Reconstructions from the representation after last last poolinglayer

(immediately before the first Fully Connected layer)

Andrej Karpathy

Page 78: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

DeepDream

DeepDream https://github.com/google/deepdream

Andrej Karpathy

https://github.com/google/deepdream
Page 79: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

DeepDream

DeepDream modifies the image in a way that “boosts” all activations, at any layer

This creates a feedback loop: e.g., any slightly detected dog face will be made

more and more dog-like over time.

Andrej Karpathy

Page 80: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

DeepDream

Andrej Karpathy

Deep Dream Grocery Trip

https://www.youtube.com/watch?v=DgPaCWJL7XI

Deep Dreaming Fear & Loathing in Las Vegas: the Great San Francisco AcidWave

https://www.youtube.com/watch?v=oyxSerkkP4o

https://www.youtube.com/watch?v=DgPaCWJL7XI
https://www.youtube.com/watch?v=oyxSerkkP4o
Page 81: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Style transfer

Page 82: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Neural Style

[ A NeuralAlgorithm of Artistic Style by Leon A. Gatys,

Alexander S. Ecker, and Matthias Bethge, 2015]

good implementation by Justin Johnson in Torch:

https://github.com/jcjohnson/neural-style

Andrej Karpathy

https://github.com/jcjohnson/neural-style
Page 83: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Neural Style

Step 1: Extract content targets (ConvNet activations ofall

layers for the given content image)

content activations

e.g.

at CONV5_1 layer we would have a [14x14x512] array of target activations

Andrej Karpathy

Page 84: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Neural StyleStep 2: Extract style targets (Gram matrices of

ConvNet activations of all layers for the given style

image)

style gram matrices

e.g.at CONV1 layer (with [224x224x64] activations) would give a [64x64] Gram

matrix of all pairwise activation covariances (summed across spatial locations)

Andrej Karpathy

Page 85: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Neural Style

Step 3: Optimize over image to have:- The content of the content image (activations match content)

- The style of the style image (Gram matrices of activations match style)

Adapted from Andrej Karpathy

matchcontent

matchstyle

Page 86: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Neural Style

make your own easily on deepart.io

Andrej Karpathy

Page 87: Russell, 2009 - Brown University...Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will not always decrease (locally) as training data point

Dataset Distillation

• https://arxiv.org/pdf/1811.10959.pdf


AVON SUPPORT UNIT SWIFTWATER RESCUE - SportsTG · AVON SUPPORT UNIT SWIFTWATER RESCUE Page 2 of 34 C:\Avon Descent\Training\Training Manual\Training Manual.doc Rescue swimmers will

Overview of Stochastic Gradient Descent Algorithmscs6320/cv_files/GradientDescentOverview.… · •Mini-batch gradient descent is typically the algorithm of choice when training

it. The realistic water survival training T · open-water parachute descent trainer (9F7) gives flight personnel the most realistic overwater training possible in descent procedures

Federal Aviation Optimized Profile Descent (OPD ... CDA- OPD... · Optimized Profile Descent (OPD) Implementation Federal Aviation ... ATC is responsible for ... Training and Notification

Domino: Extracting, Comparing, and Manipulating Subsets ...people.seas.harvard.edu/~alex/papers/2014_infovis_domino.pdf · Domino: Extracting, Comparing, and Manipulating Subsets

Descent and Essential Descent Spectrum of Linear Relations · Descent and Essential Descent Spectrum ... [email protected], ... we study the descent spectrum and the essential

DESCENT PREPARATION MANAGED DESCENT T/D · PDF filepf descent phase pnf descent preparation managed descent iaf t/d 80 nm ~10 minutes a320 -version 05a

Support Vector Machines: Training with Stochastic Gradient ... · Machine Learning Support Vector Machines: Training with Stochastic Gradient Descent 1. Support vector machines

Section 2.2 Subsets

Descent and Monadicity - Informatics Conferences …conferences.inf.ed.ac.uk/ct2019/slides/5.pdfDescent category Descent theory Monadicity via descent Descent and Monadicity Fernando

An Overview of Gradient Descent Optimization Algorithms ... · Outline 1 Introduction Basics 2 Gradient Descent Variants Basic Gradient Descent Algorithms Limitations 3 Gradient Descent

PERFECT SUBSETS OF GENERALIZED BAIRE SPACES AND LONG subsets of... · 2016. 4. 11. · PERFECT SUBSETS OF GENERALIZED BAIRE SPACES AND LONG GAMES PHILIPPSCHLICHT ... Väänänen [Vää91,

Rescue and descent, made simple and secure. · Always refer to user manual and seek proper training before attempting rescue and descent options. Specifications: Max. descent height

A distributed block coordinate descent method for training ...keerthis.com/jmlrfp2.pdf · Distributed block coordinate descent for l 1 regularized classifiers comparing our method

Instruction Subsets in Software Diversity

Stochastic Gradient Descent Training for L1-regularizaed Log-linear Models with Cumulative Penalty

Memristor-based multilayer neural networks with online gradient descent training

“Reduce Continuing CFIT Accidents by Training Constant Descent Final Approaches

Northam’s Avon Descent Association · 2019. 6. 24. · Avon Descent – Event Rules 2019 – Northam’s Avon Descent Association Author: Northam’s Avon Descent Association Subject:

Sorting It All Out Mathematical Topics Mathematical Topics Subsets of real numbers and the relationships between these subsets Subsets of real numbers

Lymphocyte subsets in human immunodeficiency virus ... · Lymphocyte subsets in human immunodeficiency virus-unexposed Brazilian individuals from ... subsets in human immunodeficiency

Log Subsets

Subsets. Subsets are sort of like nested Russian dolls: the subset “fits inside” the set

Estimating Training Data Influence by Tracing Gradient Descent

3.8 Subsets ( )

Learning to learn by gradient descent by gradient descentpapers.nips.cc/...to...descent-by-gradient-descent.pdf · Learning to learn by gradient descent by gradient descent Marcin

PILAE: A Non-gradient Descent Learning Scheme for Deep ...pguo/pdf/2018/PILAE A Non... · Most of the learning algorithms for training autoencoders are based on the gradient descent

Introduction - American Mathematical Society€¦ · DESCENT REPRESENTATIONS AND MULTIVARIATE STATISTICS 3055 For type B we have Theorem 1.5. For any pair of subsets S1 ⊆ [n −

Memory Subsets

Convex Independent Subsets

Chapter 21 Kinship and Descent. Chapter Outline What are descent groups? What functions do descent groups serve? How do descent groups evolve?

Budgeted Stochastic Gradient Descent for Large-Scale SVM Training

Russell, 2009cs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 3. 13. · Stochastic Gradient Descent Try to speed up processing with random training subsets Loss will

1 Lecture 10: descent methods Gradient descent (reminder)

The General Inefficiency of Batch Training for Gradient ...axon.cs.byu.edu/~martinez/classes/678/Papers/Wilson_batch orig.pdfKey words: Batch training, on-line training, gradient descent,