8/17/2014 Information Processing: Neural Networks and Deep Learning

http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 1/12

More Next Blog» Create Blog Sign In

Information ProcessingPessimism of the Intellect, Optimism of the Will Archive Favorite posts Twitter: @steve_hsu

Stephen Hsu

Google+ Profile

Saturday, August 16, 2014

Neural Networks and Deep Learning

One of the SCI FOO sessions I enjoyed the most this year was adiscussion of deep learning by AI researcher JuergenSchmidhuber. For an overview of recent progress, see this recentpaper. Also of interest: Michael Nielsen's pedagogical bookproject.

An application which especially caught my attention is describedby Schmidhuber here:

Many traditional methods of Evolutionary Computation[15-19] can evolve problem solvers with hundreds ofparameters, but not millions. Ours can [1,2], by greatlyreducing the search space through evolving compact,compressed descriptions [3-8] of huge solvers. Forexample, a Recurrent Neural Network [34-36] withover a million synapses or weights learned (without a

8/17/2014 Information Processing: Neural Networks and Deep Learning

http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 2/12

teacher) to drive a simulated car based on a high-dimensional video-like visual input stream.

More details here. They trained a deep neural net to drive a carusing visual input (pixels from the driver's perspective, generatedby a video game); output consists of steering orientation andaccelerator/brake activation. There was no hard coded structurecorresponding to physics -- the neural net optimized a utilityfunction primarily defined by time between crashes. It learnedhow to drive the car around the track after less than 10k trainingsessions.

For some earlier discussion of deep neural nets and theirapplication to language translation, see here. Schmidhuber hasalso worked on Solomonoff universal induction.

These TED videos give you some flavor of Schmidhuber's senseof humor :-) Apparently his younger brother (mentioned in thefirst video) has transitioned from theoretical physics to algorithmicfinance. Schmidhuber on China.

8/17/2014 Information Processing: Neural Networks and Deep Learning

http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 3/12

Older Post

Posted by Stephen Hsu at 7:28 PM Labels: ai, brainpower, computing, machine learning, scifoo,singularity

0 Comments information processing Login

Sort by Best Share

Start the discussion…

Be the first to comment.

Subscribe Add Disqus to your site

Favorite

Links to this post

Create a Link

Home

Subscribe

Posts

8/17/2014 Information Processing: Neural Networks and Deep Learning

http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 4/12

Blog Archive▼ 2014 (116)

▼ 08 (7)

Neural Networks and Deep Learning

Y Combinator: "fund for the pivot"

Designer babies: selection vs editing

SCI FOO 2014: photos

Next Super Collider in China?

@ SCI FOO 2014

It's all in the gene: cows

► 07 (15)

► 06 (12)

► 05 (19)

► 04 (12)

► 03 (18)

► 02 (16)

► 01 (17)

► 2013 (211)

► 2012 (222)

► 2011 (267)

► 2010 (234)

► 2009 (204)

► 2008 (255)

► 2007 (196)

► 2006 (154)

► 2005 (253)

► 2004 (100)

Labels

physics (285)finance (245)globalization (220)genetics (205)brainpower (193)photos (130)technology (120)China (119)science (118)economics (116)american society (115)credit crisis (115)psychometrics (110)

Comments

8/17/2014 Information Processing: Neural Networks and Deep Learning

http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 5/12

travel (99)innovation (86)psychology (81)universities (81)human capital (80)credit crunch (78)iq (78)higher education (77)startups (75)gilded age (66)biology (62)careers (62)income inequality (60)elitism (58)autobiographical (55)cognitive science (55)ai (52)books (51)evolution (49)quantum mechanics (49)genius (46)bgi (45)cdo (45)caltech (43)kids (43)social science (43)derivatives (41)genetic engineering (41)statistics (38)talks (38)education (36)mma (36)mortgages (36)bubbles (35)history of science (35)behavioral economics (34)genomics (32)hedge funds (31)mathematics (31)bounded rationality (30)intellectual history (29)silicon valley (29)efficient markets (27)podcasts (27)quants (27)literature (26)many worlds (26)black holes (25)geopolitics (25)sports (25)subprime (25)MSU (24)entrepreneurs (24)expert prediction (24)

8/17/2014 Information Processing: Neural Networks and Deep Learning

http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 6/12

housing (24)political correctness (24)taiwan (24)academia (23)foo camp (23)obama (22)physical training (22)ufc (22)machine learning (21)sci fi (21)ultimate fighting (21)athletics (20)film (20)google (20)jiujitsu (20)wall street (20)berkeley (19)cds (19)economic history (19)feynman (19)history (19)computing (18)goldman sachs (18)movies (18)affirmative action (17)bjj (17)harvard (17)security (17)treasury bailout (17)freeman dyson (16)race relations (16)von Neumann (16)internet (15)japan (15)singularity (15)nuclear weapons (14)university of oregon (14)blogging (13)neuroscience (13)oppenheimer (13)personality (13)quantum field theory (13)realpolitik (13)biotech (12)gender (12)happiness (12)india (12)politics (12)probability (12)venture capital (12)algorithms (11)conferences (11)hedonic treadmill (11)malcolm gladwell (11)

8/17/2014 Information Processing: Neural Networks and Deep Learning

http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 7/12

meritocracy (11)music (11)scifoo (11)autism (10)cosmology (10)fitness (10)mutants (10)net worth (10)olympics (10)social networks (10)string theory (10)aspergers (9)chess (9)entropy (9)geeks (9)neanderthals (9)new yorker (9)wwii (9)ability (8)cryptography (8)flynn effect (8)italy (8)keynes (8)nobel prize (8)robot genius (8)usain bolt (8)aig (7)alpha (7)ashkenazim (7)basketball (7)christmas (7)complexity (7)crossfit (7)dating (7)eugene (7)football (7)harvard society of fellows (7)hugh everett (7)manhattan (7)nerds (7)real estate (7)television (7)Einstein (6)alan turing (6)anthropic principle (6)art (6)blade runner (6)data mining (6)determinism (6)dna (6)fx (6)games (6)les grandes ecoles (6)nassim taleb (6)

8/17/2014 Information Processing: Neural Networks and Deep Learning

http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 8/12

qcd (6)success (6)teaching (6)turing test (6)volatility (6)Fermi problems (5)academia sinica (5)bayes (5)bobby fischer (5)climate change (5)energy (5)environmentalism (5)france (5)free will (5)game theory (5)height (5)james salter (5)luck (5)noam chomsky (5)nsa (5)philip k. dick (5)philosophy of mind (5)poker (5)privacy (5)prostitution (5)renaissance technologies (5)software development (5)tail risk (5)war (5)warren buffet (5)100m (4)Go (4)Iran (4)Poincare (4)borges (4)cambridge uk (4)charles darwin (4)class (4)creativity (4)econtalk (4)fake alpha (4)global warming (4)godel (4)government (4)hormones (4)humor (4)inequality (4)iraq war (4)kerviel (4)markets (4)microsoft (4)monsters (4)paris (4)perimeter institute (4)

8/17/2014 Information Processing: Neural Networks and Deep Learning

http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 9/12

pseudoscience (4)research (4)russia (4)soros (4)trento (4)vietnam war (4)200m (3)babies (3)bill gates (3)brain drain (3)censorship (3)charlie munger (3)chet baker (3)cold war (3)correlation (3)demographics (3)ecosystems (3)encryption (3)equity risk premium (3)facebook (3)fannie (3)feminism (3)fst (3)information theory (3)intellectual property (3)intellectual ventures (3)judo (3)kasparov (3)lewontin fallacy (3)lhc (3)michael lewis (3)mixed martial arts (3)moore's law (3)nathan myhrvold (3)neal stephenson (3)new york times (3)nonlinearity (3)patents (3)path integrals (3)quantum computers (3)risk preference (3)sad but true (3)search (3)sec (3)sivs (3)society generale (3)solar energy (3)alibaba (2)assortative mating (2)bear stearns (2)bruce springsteen (2)charles babbage (2)cloning (2)david mamet (2)

8/17/2014 Information Processing: Neural Networks and Deep Learning

http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 10/12

democracy (2)digital books (2)donald mackenzie (2)eliot spitzer (2)empire (2)exchange rates (2)freddie (2)gaussian copula (2)industrial revolution (2)james watson (2)jim simons (2)language (2)ltcm (2)magic (2)mba (2)mccain (2)national character (2)nicholas metropolis (2)no holds barred (2)offices (2)oligarchs (2)olympiads (2)palin (2)pca (2)pop culture (2)population structure (2)prisoner's dilemma (2)rationality (2)skidelsky (2)socgen (2)sprints (2)thailand (2)variance (2)abx (1)anathem (1)andrew lo (1)antikythera mechanism (1)athens (1)atlas shrugged (1)ayn rand (1)bay area (1)beats (1)book search (1)bunnie huang (1)car dealers (1)carlos slim (1)catastrophe bonds (1)cdos (1)ces 2008 (1)chance (1)cheng ting hsu (1)children (1)cochran-harpending (1)cpi (1)

8/17/2014 Information Processing: Neural Networks and Deep Learning

http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 11/12

david x. li (1)dick cavett (1)dolomites (1)drugs (1)eharmony (1)epidemics (1)escorts (1)faces (1)fads (1)favorite posts (1)fiber optic cable (1)francis crick (1)gary brecher (1)gizmos (1)greece (1)greenspan (1)heinlein (1)hypocrisy (1)igon value (1)iit (1)inflation (1)information asymmetry (1)iphone (1)jack kerouac (1)jaynes (1)jfk (1)john dolan (1)john kerry (1)john paulson (1)john searle (1)john tierney (1)jonathan littell (1)las vegas (1)lawyers (1)lee kwan yew (1)lehman auction (1)les bienveillantes (1)lowell wood (1)lse (1)mating (1)mcgeorge bundy (1)mexico (1)michael jackson (1)mickey rourke (1)migration (1)mit (1)money:tech (1)monkeys (1)myron scholes (1)netwon institute (1)networks (1)newton institute (1)nfl (1)oliver stone (1)

8/17/2014 Information Processing: Neural Networks and Deep Learning

http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 12/12

phil gramm (1)philanthropy (1)philip greenspun (1)portfolio theory (1)power laws (1)randomness (1)recession (1)sales (1)simulation (1)singapore (1)skype (1)smpy (1)standard deviation (1)star wars (1)starship troopers (1)students today (1)supercomputers (1)systemic risk (1)teleportation (1)tierney lab blog (1)tomonaga (1)twitter (1)tyler cowen (1)ussr (1)venice (1)violence (1)virtual meetings (1)virtual reality (1)war nerd (1)wealth effect (1)