reinforcement learning - redwood center for theoretical …€¦ · reinforcement learning ? ?...

Reinforcement Learning VS265 - Neural Computation, 2018

Upload: others

Post on 19-May-2020

4 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Reinforcement Learning

VS265 - Neural Computation, 2018

What we have covered

Passive Learning Today:Active Learning (RL)

Page 3: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

What is Reinforcement Learning?

Page 4: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

How is it different than other models?

Passive Learning Active Learning (RL)

? ?

Page 5: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Why is this hard?

● Actions affect future data● Rewards are sparse● Feedback is delayed

Reinforcement Learning

? ?

Outline● Markov Decision Processes (MDPs)● How to maximize reward (Q-Learning)● Connection to neurons in the Ventral Tegmental Area

(VTA)● How to learn in large, unstructured**, environments● Open Questions

Markov Decision Process

Markov Decision Process (MDP)An MDP fully describes an Environment:

○ S: State Space○ A: Action Space○ P: Transition Kernel - ○ R: Reward Function -

Markov Decision Process (MDP)● Markov

○● Decision

○ Decide on an action at each time point○

● Process○ States evolve over time

Page 10: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Markov Decision Process (MDP)

A B

Page 11: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning - Algorithm

Page 12: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning - Algorithm● Find a good policy, , that maximizes the

expected sum of rewards over time:

Page 13: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning - Algorithm● Q(s,a) is the total expected reward starting from state s,

taking action a, and then following optimal policy

Page 14: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning - Update Rule

Page 15: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning - Update Rule

Page 16: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning - Update RuleState-Value Function:

Page 17: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning - Update RuleAction-Value Function:

Page 18: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning - Update Rule

Page 19: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning - Update Rule

Temporal Difference

Critic (New Belief)

Belief

Iterate:

Page 20: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning - Exercise

Page 21: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning (Exercise)

Temporal Difference

A B

Page 22: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Connection to VTA

Page 23: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Connection to VTA

Theoretical Neuroscience, ch.9 (Dayan & Abbot)(Adapted from Mirenowicz & Schultz, ‘94 & Schultz ‘98)

Page 24: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning in large environments

Page 25: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning in large environments

A B

Page 26: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning in large environments● Deep Q-Networks (DQN): Estimate Q using a

neural network

Page 27: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning in large environments● Objective Function: Use the temporal difference

signal

Page 28: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Deep-Q-Network

● Use a Convolutional Neural Network (CNN) as the function approximator

● Experience Replay - Store experiences in a data-set and randomly sample them during learning

Mnih, Volodymyr, et al. "Playing atari with deep reinforcement learning." arXiv preprint arXiv:1312.5602 (2013).

http://www.youtube.com/watch?v=LJ4oCb6u7kk

Page 29: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Open Questions

Page 30: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Open Questions● Credit assignment in worlds with sparse rewards● Exploration vs. Exploitation● Generalization to the real world● Continual Learning

Page 31: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning in even more complex worlds

A B

Page 32: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Resources● David Silver’s Lectures

○ http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html

● CS294 - Deep Reinforcement Learning ○ http://rll.berkeley.edu/deeprlcourse/

http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html

http://rll.berkeley.edu/deeprlcourse/

Page 33: Reinforcement Learning - Redwood Center for Theoretical …€¦ · Reinforcement Learning ? ? Outline Markov Decision Processes (MDPs) How to maximize reward (Q-Learning) Connection

Q-Learning

Temporal Difference

A B

Learning in structured MDPs with convex cost functions

Reinforcement Learning in MDPs by Lease-Square Policy Iteration

10703 Deep Reinforcement Learning and Controlrsalakhu/10703/Lectures/Lecture3_exactmethods… · Solving MDPs! • Prediction: Given an MDP and a policy ﬁnd the state and action

Tsianikas S., Yousefi N., Zhou J., Coit D.W....specific microgrid in a predetermined time horizon. Introduction Nomenclature MDPs and Reinforcement Learning The problem is solved using

Reinforcement Learning When All Actions are Not Always ... · motivate the question we aim to address: how can we develop ... studying MDPs with stochastic action sets by deﬁning

Inverse Reinforcement Learning CS885 Reinforcement

Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning Applications Summary

MDPs (cont) & Reinforcement Learning Tamara Berg CS 560 Artificial Intelligence Many slides throughout the course adapted from Svetlana Lazebnik, Dan Klein,

Contextual MDPs for PAC Reinforceme t Learning With Rich Observations

Reinforcement Learning - uni-freiburg.degki.informatik.uni-freiburg.de/.../recordings/reinforcement.pdf · Reinforcement Learning 3 What is Reinforcement Learning? Learning from interaction

Bayesian Reinforcement Learning - mlg.eng.cam.ac.ukmlg.eng.cam.ac.uk/rowan/files/BayesianReinforcementLearning.pdf · Introduction Bayesian Reinforcement Learning Bayesian Reinforcement

Reinforcement Learning in Finite MDPs: PAC Analysis · Reinforcement Learning in Finite MDPs: PAC Analysis Alexander L. Strehl∗ [email protected] Facebook 1601 S California Ave

Deep Learning for Reinforcement Learning in · PDF fileDeep Learning for Reinforcement Learning in ... Deep Learning for Reinforcement Learning in Pacman Deep Learning für ... Während

arXiv · E cient Reinforcement Learning in Factored MDPs with Application to Constrained RL Xiaoyu Chen [email protected] Key Laboratory of Machine Perception, MOE, School of EECS,

10703 Deep Reinforcement Learning€¦ · 10/09/2018 · 10703 Deep Reinforcement Learning! Tom Mitchell September 10, 2018 Solving known MDPs Many slides borrowed from ! Katerina

Markov Decision Processes (MDPs) (cont.)guestrin/Class/15781/slides/mdps-rl...1 Markov Decision Processes (MDPs) (cont.) Machine Learning – 10701/15781 Carlos Guestrin Carnegie Mellon

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

Reinforcement Learning: Learning algorithms

1 ECE-517 Reinforcement Learning in Artificial Intelligence Lecture 7: Finite Horizon MDPs, Dynamic Programming Dr. Itamar Arel College of Engineering

Inverse Reinforcement Learning - Peoplecbfinn/_files/bootcamp_inverserl.pdf · Apprenticeship Learning via Inverse Reinforcement Learning. Good introduction to inverse reinforcement

Generalization in Reinforcement Learning: Successful ...papers.nips.cc/paper/1109-generalization-in-reinforcement-learning... · Generalization in Reinforcement Learning: Successful

Near-optimal Regret Bounds for Reinforcement Learningjmlr.org/papers/volume11/jaksch10a/jaksch10a.pdfReinforcement learning of MDPs is a standard model for learning with delayed feedback

Optimal Learning & Bayes -Adaptive MDPs

Reinforcement Learning in Factored MDPs: Oracle-Efﬁcient

Reinforcement Learning Lecture Inverse Reinforcement Learningipvs.informatik.uni-stuttgart.de/mlr/wp-content/uploads/2017/07/09... · Reinforcement Learning Inverse Reinforcement

Cooperative Inverse Reinforcement Learning...Cooperative Inverse Reinforcement Learning Dylan Hadfield-Menell CS237: Reinforcement Learning May 31, 2017

MDPs and RL Outline Reinforcement Learningcs188/sp11/slides/SP11 cs188 lectur… · Reinforcement Learning ! Reinforcement learning: ! Still assume an MDP: ! A set of states s ∈

Between MDPs and Semi-MDPs: A Framework for Temporal ...dprecup/publications/SPS-aij.pdfBetween MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning Richard

Reinforcement Learning in Large or Unknown MDPs

1 ECE-517: Reinforcement Learning in Artificial Intelligence Lecture 6: Optimality Criterion in MDPs Dr. Itamar Arel College of Engineering Department

Multi-Objective Reinforcement Learning using Sets of Pareto … · 2020. 10. 19. · learning and multi-objective reinforcement learning. 2.1 Reinforcement Learning A reinforcement

Reinforcement Learning Chapter 13 What is Reinforcement Learning? Q-Learning Examples 1

Deep reinforcement learning for controlling frontal person ...poseidon.csd.auth.gr/papers/PUBLISHED/JOURNAL/pdf/... · Deep reinforcement learning Markov Decision Processes (MDPs)

Reinforcement Learning - Multi-Agent Reinforcement

Eick: Reinforcement Learning. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning