![Page 1: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/1.jpg)
DeepMDPLearning Latent Space Continuous Models for
Representation Learning
Carles Gelada, Saurabh Kumar, Jacob Buckman,
Ofir Nachum, Marc G. Bellemare
![Page 2: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/2.jpg)
212
Simple Representations for RL
![Page 3: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/3.jpg)
Neural networks
DeepMDPLatent Space Model:
MDP:
& trained via the following two losses:
![Page 4: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/4.jpg)
Reward Loss
![Page 5: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/5.jpg)
Transition Loss
![Page 6: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/6.jpg)
Tractable Losses
![Page 7: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/7.jpg)
Deep Policies
![Page 8: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/8.jpg)
Representation Quality
![Page 9: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/9.jpg)
Only Discards:
Ferns, N., Panangaden, P., and Precup, D. Metrics for Finite Markov Decision
Processes. In Proceedings of the 20th Conference on Uncertainty in Artificial
Intelligence, UAI ’04, pp. 162–169, 2004.
![Page 10: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/10.jpg)
Phi as a Representation
![Page 11: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/11.jpg)
Donut World
![Page 12: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/12.jpg)
DeepMDP on Donut World
2D latent space
+
DeepMDP losses
![Page 13: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/13.jpg)
DeepMDP on Donut World
Visualization of latent distance
![Page 14: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/14.jpg)
DeepMDPAuxiliary Task
Base C51 agent
+
DeepMDP losses
![Page 15: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/15.jpg)
DeepMDPAuxiliary Task
Base C51 agent
+
DeepMDP losses
![Page 16: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/16.jpg)
● DeepMDPs as Models of the Environment
● Norm-MMD Metrics and their Associated Smoothness
![Page 17: Learning Latent Space Continuous Models for ... - ICML11-14-00)-11-15-05-5163... · BattleZone Gravitar Carnival CrazyClimber BankHeist Centipede Hero Qbert RoadRunner TimePilot Tutankham](https://reader034.vdocuments.us/reader034/viewer/2022042208/5eab377dd9f5df562f639a9a/html5/thumbnails/17.jpg)
Thanks For Listening
Poster #108