DEEP LEARNING: REVOLUTION OR HYPE?

Adriaan Schakel

Berlin Machine Learning Group

May 13, 2014

VOL. CLXII . . No. 55,965 © 2012 The New York Times NEW YORK, SATURDAY, NOVEMBER 24, 2012

Late EditionToday, partly sunny, windy, colder,high 46. Tonight, cloudy, chilly, low33. Tomorrow, mostly sunny,breezy, high 49. Monday, mostlysunny. Weather map, Page D6.

$2.50

By KAREEM FAHIM and DAVID D. KIRKPATRICK

CAIRO — Protests eruptedacross Egypt on Friday, as oppo-nents of President MohamedMorsi clashed with his support-ers over a presidential edict thatgave him unchecked authorityand polarized an already dividednation while raising a specter, thepresident’s critics charged, of areturn to autocracy.  

In an echo of the uprising 22months ago, thousands of pro-testers chanted for the downfallof Mr. Morsi’s government in Cai-ro, while others ransacked the of-fices of the president’s formerparty in Suez, Alexandria andother cities.

Mr. Morsi spoke to his support-ers in front of the presidentialpalace here, imploring the publicto trust his intentions as he casthimself as a protector of the revo-lution and a fledgling democracy.

In a speech that was by turnsdefensive and conciliatory, he ul-timately gave no ground to thecritics who now were describinghim as a pharaoh, in another echoof the insult once reserved for thedeposed president, Hosni Muba-rak.

“God’s will and elections mademe the captain of this ship,” Mr.Morsi said.

The battles that raged on Fri-day — over power, legitimacyand the mantle of the revolution— posed a sharp challenge not

only to Mr. Morsi but also to hisopponents, members of secular,leftist and liberal groups whosecrippling divisions have stifledtheir agenda and left them un-able to confront the more popularIslamist movement led by theMuslim Brotherhood.

The crisis over his power grabcame just days after the Islamistleader won international praisefor his pragmatism, includingfrom the United States, for bro-kering a cease-fire between Ha-mas and Israel.

On Friday, the State Depart-ment expressed muted concernover Mr. Morsi’s decision. “Oneof the aspirations of the revolu-

Violent Protests in Egypt

As Leader Expands Power

Morsi Declares ‘God’s Will and Elections

Made Me the Captain of This Ship’

MOHAMED ABD EL GHANY/REUTERS

Riot police officers chasedprotesters in Cairo on Friday.

Continued on Page A9

By JODI RUDOREN and ISABEL KERSHNER

KHAN YUNIS, Gaza Strip —In the 12 years that he has livedhere in the Abassam neighbor-hood adjacent to Gaza’s easternborder, Eyad Qudaih said, he hadnever ventured more than 20yards east of his white stuccohome because Israel said thearea was off limits.

But on Friday morning, em-boldened by the new cease-fire,he took his four young daughters300 yards east, to the small plot ofland where he dreams of growingwheat as his father once did.

“It was like someone who washungry and had a big meal,” Mr.Qudaih said, shortly after touch-ing the border fence for the firsttime. “Grilled sheep with nuts.”

But around 11 a.m., the mo-ment was interrupted by thesound of gunfire. A spokesmanfor the Israeli military said sol-diers had fired warning shots andthen at the feet of some Palestin-ians who tried to cross the border

fence into Israeli territory. Mr.Qudaih’s cousin Anwar Qudaih,20, was killed, and nine otherswere wounded, Health Ministryofficials here in Gaza said.

The episode, which happenedat the same spot where an anti-tank missile fired by Palestinianshit an Israeli jeep, wounding fourIsraeli soldiers two weeks ago,did not fracture the truce thatended the recent fighting be-tween Gaza and Israel. But it didshowcase the confusion that re-mains over the cease-fire deal an-nounced Wednesday in Cairo.While Hamas officials have beenboasting about the concessionsthey say they have exacted fromIsrael, Israeli officials say noth-ing has been agreed upon beyondthe immediate cessation of hostil-ities.

On Thursday, the Israeli de-fense minister, Ehud Barak, said

Tension and Confusion Linger

In Gaza Strip After Cease-Fire

Continued on Page A8

PHOTOGRAPHS BY MARCUS YAM FOR THE NEW YORK TIMES

Crowds filled Macy’s in NewYork and countless storesaround the nation as shop-pers, increasingly optimisticabout the nation’s economy,went out in search of bargainsin that annual spending ritualknown as Black Friday. Manyconsumers appeared ready toshop till they dropped — andsome even did, dozing off onfurniture displays in depart-ment stores offering predawndeals. Starting Thursdaynight, hundreds to thousandsof shoppers lined up outsideSears, Target, Old Navy andother retailers that were offer-ing time-sensitive deals onmerchandise. Page B1.

Optimistic ConsumersGo on Shopping Spree

By JOHN MARKOFF

Using an artificial intelligencetechnique inspired by theoriesabout how the brain recognizespatterns, technology companiesare reporting startling gains infields as diverse as computer vi-sion, speech recognition and theidentification of promising newmolecules for designing drugs.

The advances have led to wide-spread enthusiasm among re-searchers who design software toperform human activities likeseeing, listening and thinking.They offer the promise of ma-chines that converse with hu-mans and perform tasks likedriving cars and working in fac-tories, raising the specter of auto-mated robots that could replacehuman workers.

The technology, called deeplearning, has already been put touse in services like Apple’s Sirivirtual personal assistant, whichis based on Nuance Communica-tions’ speech recognition service,and in Google’s Street View,which uses machine vision toidentify specific addresses.

But what is new in recentmonths is the growing speed andaccuracy of deep-learning pro-grams, often called artificial neu-ral networks or just “neural nets”for their resemblance to the neu-ral connections in the brain.

“There has been a number ofstunning new results with deep-learning methods,” said YannLeCun, a computer scientist atNew York University who didpioneering research in handwrit-ing recognition at Bell Laborato-ries. “The kind of jump we are

Learning Curve:

No Longer Just

A Human Trait

Continued on Page A3

A forester working for NewYork City’s parks departmentmade a horrifying discovery lastweek, beside a huge pile of fallen

trees destined for thewood chipper.

A dead man. And with that dis-

covery on Nov. 15, addthis to the huge list oftroubles Hurricane

Sandy has brought to the neigh-borhoods hit hardest: wreckagefrom the storm seems to havecreated inviting spots for killersto dump bodies.

Hours after the discovery, inForest Park in Queens, a secondbody was found on storm-rav-aged Rockaway Beach. Workerscleaning up around O’DonohuePark heard a shriek from one oftheir own, standing over a dune

near the shoreline. There, aman’s elbow protruded from thecold sand.

There is no evidence the casesare related, but they appear to bethe first victims discarded in thechanging landscape that followedthe storm’s landfall — placeswhere people, especially the po-lice, might not think to look.

On the beach, it was unclearhow the man died. The medicalexaminer’s office said the casewas pending further investiga-tion. But the man had been tiedup and placed in a garbage bag,and there were signs of blunttrauma and bruises, the policesaid. The body carried no identifi-cation, and facial-recognitiontesting on the corpse did not

In Storm’s Debris, the Macabre

OZIER MUHAMMAD/THE NEW YORK TIMES

A worker cleaning up after Hurricane Sandy on Rockaway Beach in Queens, above, found the body of a man that showed signs of blunt trauma and was stuffed in a garbage bag. Continued on Page A16

MICHAELWILSON CRIMESCENE

By MICHAEL CIEPLY and BROOKS BARNES

WELLINGTON, New Zealand— Standing by his desk in NewZealand’s distinctive round Par-liament building, known locallyas the Beehive, Prime MinisterJohn Key proudly brandished anornately engraved sword. It wasused, he said, by Frodo Baggins,the protagonist of the “Lord ofthe Rings” trilogy, and in thefilms it possesses magical powersthat cause it to glow blue in thepresence of goblins.

“This was given to me by thepresident of the United States,”said Mr. Key, marveling thatPresident Obama’s official gift toNew Zealand was, after all, aNew Zealand product.

In Mr. Key’s spare blond-wood

office — with no goblins in sight— the sword looked decidedlyunmagical. But it served as a re-minder that in New Zealand, thebusiness of running a countrygoes hand in hand with the busi-ness of making movies.

For better or worse, Mr. Key’sgovernment has taken extrememeasures that have linked its for-tunes to some of Hollywood’s big-gest pictures, making this coun-try of 4.4 million people, slightlymore than the city of Los Ange-les, a grand experiment in the fu-sion of film and government.

That union has been on enthu-siastic display here in recentweeks as “The Hobbit: An Unex-pected Journey,” the first of threerelated movies by the directorPeter Jackson, approached itsworld premiere on Wednesday inWellington (and on Dec. 14 in the

United States). Anticipation inNew Zealand has been building,and there are signs everywhereof the film’s integration into Kiwilife — from the giant replica ofthe movie’s Gollum creature sus-pended over the waiting area atWellington Airport to the giftshops that are expanding to meetanticipated demand for Hobbitmerchandise (elf ears, $14).

But the path to this momenthas been filled with controversy.Two years ago, when a disputewith unions threatened to derailthe “Hobbit” movies — endan-gering several thousand jobs anda commitment of some $500 mil-lion by Warner Brothers — Mr.Key persuaded the Parliament torewrite its national labor laws.

It was a breathtaking solution,

New Zealand Wants a Hollywood Put on Its Map

Continued on Page A10

A meeting of the European Union col-lapsed amid bitter discord over a newbudget for the bloc and proposals totrim its administrative costs. PAGE A7

INTERNATIONAL A4-11

A New Setback for EuropeThirty years after the Chia Pets fad, chiais gaining a second life as a nutritionalitem, with seeds being added to fruitdrinks and a variety of foods. PAGE B1

BUSINESS DAY B1-8

A Ubiquitous Pet, Now Edible“Liz and Dick,” a tribute to the love af-fair between Elizabeth Taylor and Rich-ard Burton, starring Lindsay Lohan,will be broadcast on Sunday night. A re-view by Alessandra Stanley. PAGE C1

ARTS C1-8

Lindsay Channels Liz (Sort Of)Pat Schilleris a lineback-er and anN.F.L. long-shot who isclosing in onhis dream.Also: Histo-ry accordingto OliverStone. MAGAZINE

THIS WEEKEND

Goal to Go

Charles M. Blow PAGE A21

EDITORIAL, OP-ED A20-21

“Zero Dark Thirty,” a movie about thehunt for Osama bin Laden that is repletewith scenes of torture, is not for the faintof heart or for those expecting typicalHollywood fare. PAGE C1

Capturing Bin Laden on FilmSo as not to split the liberal vote, theman seen as a top contender in the pres-idential race bows out. PAGE A8

South Korean Candidate Quits

Upset with President Obama’s re-elec-tion, some Texans are calling for theirstate to secede. PAGE A12

NATIONAL A12-14

An Outbreak of Secessionitis

A company’s decision to exceed NewYork City elevation rules could be a har-binger of building mandates. PAGE A15

NEW YORK A15-19

Storm Protections Paid Off

The precinct where Donald Trump liveswas one of only two in Manhattan thatPresident Obama did not win. PAGE A15

Trump Tower Glows Red

U(D54G1D)y+&!{!.!=!$It was not a happy reunion for theKnicks when they faced their formerteammate Jeremy Lin in Houston. Linhelped the Rockets register a lopsided131-103 victory. PAGE D1

SPORTS SATURDAY D1-6

Linsanity Turns Against Knicks

C M Y K Nxxx,2012-11-24,A,001,Bs-4C,E3

One-line version NYT Nov 24 2012:

"advances in an artificial intelligence technology

that can recognize patterns offer the possibility of

machines that perform human activities like seeing,

listening and thinking"

Three years ago, researchers at thesecretive Google X lab in MountainView, California, extracted some10 million still images from YouTubevideos and fed them into Google Brain— a network of 1,000 computers pro-

grammed to soak up the world much as ahuman toddler does. After three days lookingfor recurring patterns, Google Brain decided,all on its own, that there were certain repeat-ing categories it could identify: human faces,human bodies and … cats1.

Google Brain’s discovery that the Inter-net is full of cat videos provoked a flurry ofjokes from journalists. But it was also a land-mark in the resurgence of deep learning: athree-decade-old technique in which mas-sive amounts of data and processing power

help computers to crack messy problems thathumans solve almost intuitively, from recog-nizing faces to understanding language.

Deep learning itself is a revival of an evenolder idea for computing: neural networks.These systems, loosely inspired by the denselyinterconnected neurons of the brain, mimichuman learning by changing the strength ofsimulated neural connections on the basis ofexperience. Google Brain, with about 1 mil-lion simulated neurons and 1 billion simu-lated connections, was ten times larger thanany deep neural network before it. Projectfounder Andrew Ng, now director of theArtificial Intelligence Laboratory at StanfordUniversity in California, has gone on to makedeep-learning systems ten times larger again.

Such advances make for exciting times in

THE LE ARNING MACHINESUsing massive amounts of data to recognize photos and speech,

deep-learning computers are taking a big step towards true artificial intelligence.

B Y N I C O L A J O N E S

BR

UC

E R

OLFF/S

HU

TTER

STO

CK

1 4 6 | N A T U R E | V O L 5 0 5 | 9 J A N U A R Y 2 0 1 4

FEATURENEWS

© 2014 Macmillan Publishers Limited. All rights reserved

Google Brain

-1M neuron-like processing units

-1G synaptic connections

-10M images of cats (200×200 pixels)

-16K cores during one week

arXiv:1112.6209

redorbit.com

extremetech.com

Wikipedia: 75 employees

Products:

www.miamibasshistory.com

Hype?

LeCun interview on KDnuggets.com February 20, 2014:

"Around 2003, Geoff Hinton, Yoshua Bengio and myself

initiated a kind of ‘conspiracy’ to revive the

interest of the machine learning community in the

problem of learning representations. It took until

2006-2007 to get some traction, primarily through new

results on unsupervised training (or unsupervised

pre-training, followed by supervised fine-tuning),

with work by Geoff Hinton, Yoshua Bengio, Andrew Ng

and myself."

George Dahl in Nature:

"We wanted to show the world that these deep neural

networks were really useful and could really help"

Perceptron (1957)

Frank Rosenblatt (psychologist/cognitive scientist)

Neuron computes weighted sum of its input signals xiand outputs:

y = θ( n∑

i=1

wixi − u)

Generalizations:

AK Jain, J Mao, KM Mohiuddin - IEEE computer, 1996

Perceptron

New York Times July 7, 1958:

-first machine that could learn new skills by

trial and error

-interest faded towards the end of the 1960s

Minsky & Papert (1969)

Perceptron learning rule: linearly separable data only

Conjectured:

1) limitations cannot be lifted by multilayer architecture

2) perceptron learning rule cannot be generalized

to multilayer networks

commons.wikimedia.org

Paul Werbos in: Talking Nets Anderson, Rosenfeld (2000)

Backpropagation

-supervised learning

-error function

E = 12(t − y)2

-weights adjusted by means of steepest descent:

∆ wji = −α ∂ E

∂ wji

Backpropagation

1) initialize the weights to small random values

2) choose an input pattern

3) propagate the signal forward through the network

4) determine the error and propagate it backwards through

the network to assign credit/blame to each unit

5) update weights

-fell out of grace by the late 1990s

ape.iict.ch

Hopfield (1982)

-notion of an energy function

-symmetric synaptic connections

-binary units

-energy:

E = −1

2

∑i ,j

wji si sj +∑i

ui si

-minimize energy through "binary threshold rule":

si = θ(∆ Ei

)where

∆ Ei = E (si = 0) − E (si = 1) =∑j

wji sj − ui

Boltzmann Machine

Ackley, Hinton, Sejnowski (1985):

-stochastic version of Hopfield network where

P(si = 1) =1

1 + exp(−∆ Ei/T )

-use unsupervised learning algorithm

-units get partitioned into visible and hidden ones:

-operated in two modes: clamped and free-running

Monte Carlo

Stan Ulam at Los Alamos

AIP Emilio Segre Visual Archives

commons.wikimedia.org

Boltzmann Learning Rule

Maximize log-likehood of training data:

L = log∏

v t ∈ TP(v t), P(v) =

∑h

P(v , h)

and (T = 1)

P(v , h) =1

Ze−E(v ,h), Z =

∑v ,h

e−E(v ,h)

Steepest ascent:

∆ wji = + α∂ L

∂ wji

Working this out gives weight updates:

∆ wji = α(〈 vi hj 〉clamped − 〈 vi hj 〉free

)

Restricted Boltzmann Machine

Paul Smolensky (1986):

-hidden units can be updated in parallel:

P(h|v) =∏j

P(hj |v)

w/ P(hj |v) the logistic function

Hinton (1999): contrastive divergencewww.coursera.org/course/neuralnets

Deep Learning

Stack RBMs to form a deep-layered network:

ccrma.stanford.edu

Deep Learning

Auto-Encoders:

-training objective (unsupervised):

maximize log-likehood of reconstruction

L =∑i

[ xi log x̂i + (1− xi ) log(1− x̂i )]

w/ x̂i = fi [c(x)], c: encoding, f : decoding

Denoising Auto-Encoders (Vincent et al., 2008):

-stochastically corrupted input

kiyukuta.github.io

Word2vec

Mikolov et al., "Distributed Representations of Words":

-represent words in a way that encodes semantics

rep(France) - rep(Paris) ∼ rep(Germany) - rep(Berlin)

-Code (in C) at code.google.com/p/word2vec/

arXiv:1310.4546

Atari

Volodymyr Mnih et al. (2013), playing Atari 2600 games:

Network:

-raw video frames as input

-convolution network for preprocessing

-training through Q-learning (Chris Watkins, 1989)

-apart from preprocessing, only one hidden layer

-better than expert human player on 3-4 out of 7 games

arXiv:1312.5602