![Page 1: 11 deep learning limitations - seas.upenn.edu](https://reader033.vdocuments.us/reader033/viewer/2022042311/625bc6e5224fab1c7e713bbb/html5/thumbnails/1.jpg)
Deep Learning Limitations and Extensions
Lyle UngarUniversity of Pennsylvania
Learning objectivesWhere deep learning failsFuture directions
![Page 2: 11 deep learning limitations - seas.upenn.edu](https://reader033.vdocuments.us/reader033/viewer/2022042311/625bc6e5224fab1c7e713bbb/html5/thumbnails/2.jpg)
Deep learning thus far l is data hungryl is shallow and has limited capacity for transferl has no natural way to deal with hierarchical structurel has struggled with open-ended inferencel is not sufficiently transparentl has not been well integrated with prior knowledgel cannot inherently distinguish causation from correlationl presumes a largely stable worldl works well as an approximation, but often cannot be trustedl is difficult to engineer Gary Marcus
![Page 3: 11 deep learning limitations - seas.upenn.edu](https://reader033.vdocuments.us/reader033/viewer/2022042311/625bc6e5224fab1c7e713bbb/html5/thumbnails/3.jpg)
Hierarchical structureu Sentences have structure
l The teenager who previously crossed the Atlantic set a record for flying
around the world.u Dialog has structure at many time scales.u As do images, movies, animals, people, companies,…
![Page 4: 11 deep learning limitations - seas.upenn.edu](https://reader033.vdocuments.us/reader033/viewer/2022042311/625bc6e5224fab1c7e713bbb/html5/thumbnails/4.jpg)
Common senseu Who is taller, Prince William or his baby son Prince George? u Can you make a salad out of a polyester shirt?u If you stick a pin into a carrot, does it make a hole in the
carrot or in the pin?
Gary Marcus
![Page 5: 11 deep learning limitations - seas.upenn.edu](https://reader033.vdocuments.us/reader033/viewer/2022042311/625bc6e5224fab1c7e713bbb/html5/thumbnails/5.jpg)
Instability: who quarterbacked Superbowl 33Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls. He is also the oldest quarterback ever to play in a Super Bowl at age 39. The past record was held by John Elway, who led the Broncos to victory in Super Bowl XXXIII at age 38 and is currently Denver’s Executive Vice President of Football Operations and General Manager. Quarterback Jeff Dean had jersey number 37 in Champ Bowl XXXIV.
![Page 6: 11 deep learning limitations - seas.upenn.edu](https://reader033.vdocuments.us/reader033/viewer/2022042311/625bc6e5224fab1c7e713bbb/html5/thumbnails/6.jpg)
Google translate is very cleveru the bat 球棒
u the bat ate 蝙蝠吃了
![Page 7: 11 deep learning limitations - seas.upenn.edu](https://reader033.vdocuments.us/reader033/viewer/2022042311/625bc6e5224fab1c7e713bbb/html5/thumbnails/7.jpg)
but still unreliableu the bat el murciélago
u the bat ate el bate comió
![Page 8: 11 deep learning limitations - seas.upenn.edu](https://reader033.vdocuments.us/reader033/viewer/2022042311/625bc6e5224fab1c7e713bbb/html5/thumbnails/8.jpg)
What’s hot in ML?u Solving problems
l Cancer diagnosis; Game playing with deep-RLu GANS (Generative Adversarial Networks)u Why gradient descent does so well (theory!)u AutoMLu Deep Q-learning Networks(DQN)u GPT-3 (Transformers)
![Page 9: 11 deep learning limitations - seas.upenn.edu](https://reader033.vdocuments.us/reader033/viewer/2022042311/625bc6e5224fab1c7e713bbb/html5/thumbnails/9.jpg)
Why does gradient descent regularize?
u http://www.offconvex.org/2019/07/10/trajectories-linear-nets/
![Page 10: 11 deep learning limitations - seas.upenn.edu](https://reader033.vdocuments.us/reader033/viewer/2022042311/625bc6e5224fab1c7e713bbb/html5/thumbnails/10.jpg)
AutoML learns network structure
Human built Learned by RLhttps://research.googleblog.com/2017/05/using-machine-learning-to-explore.html
![Page 11: 11 deep learning limitations - seas.upenn.edu](https://reader033.vdocuments.us/reader033/viewer/2022042311/625bc6e5224fab1c7e713bbb/html5/thumbnails/11.jpg)
Big open directions (an opinion)u Multitask learning / domain transferu One shot learning
l “See one do one”u Integrating deep learning with external data
l e.g. results of database queries or searchu Learning generalizable “deep structure”
l Whatever that means?