decision making -- an overviehecke/archiv/decmak.pdf · phenomenological models of rationality...
TRANSCRIPT
Decision Making – an Overview
Decision Making – an Overview
Hecke [email protected]
CNS SeminarNikolaus 2007
göttingen
Decision Making – an Overview
What we will learn today göttingen
Decision Making – an Overview
Decision Making Sessions
today, 6.12.
Overview on decision making (Hecke)
Emotions in decision making (Wei)
next week, 13.12.
POMDPs (Hecke)
Neural Correlates of decision making (Dima)
göttingen
Decision Making – an Overview
Outline
1 Overviewing the Overview
2 Learning vs. Decision Making
3 Neuroeconomics
4 Utility Theory
5 Social Decision Making
6 Mentalizing and Theory of Mind
7 Reinforcement Learning
8 Actor-Critic Model
9 Sutton-Barto-model
göttingen
Decision Making – an Overview
Overviewing the Overview
Overviewing the Overview
Decision Making before Neuroscience
the basis of human decisions is interesting for a wide fieldof research
very important for money making
mathematical formalization in game theory
phenomenological models of rationality
Decision Making in the Neurosciences
learning theories from AI research
finding neural correlates of everything
structurally realistic models of decision making pathways inthe brain
göttingen
Decision Making – an Overview
Learning vs. Decision Making
Learning vs. Decision Making
Asking Psychologists or Pedagogues
Learning is a relatively permanent change in behaviour.
Behaviour...
... now is how an individual decides to react on its environment.
Therefore we have a great overlap between decisionmaking and learning.
at least concerning computational models.
göttingen
Decision Making – an Overview
Learning vs. Decision Making
What is Decision Making göttingen
Decision Making – an Overview
Neuroeconomics
Neuroeconomics
Ferdinand Porsche"Wir wollen Autos bauen, die keiner braucht
aber jeder haben will."See my old talk at www.nld.ds.mpg.de/hecke/archiv/eco.pdf
göttingen
Decision Making – an Overview
Neuroeconomics
Rational Choice Theory
dominant theoretical paradigm in microeconomics
central to modern political science
and other disciplines such as sociology
individuals choose the best action according to stablepreference functions and constraints facing them
göttingen
Decision Making – an Overview
Neuroeconomics
Neuroeconomics
homo economicus
is widely assumed to be a rational and self-interested decisionmaker.
in reality
routines to approximate optimal decision making:
emotionally driven
experience based
taking into account decisions of others
estimate outcome of alternative actions
Review: D. Lee: Neural basis of quasi-rational decision making. Current Opinion in
Neurobiology 2006, 16: 191–198
göttingen
Decision Making – an Overview
Neuroeconomics
;-) göttingen
Decision Making – an Overview
Utility Theory
Utility Theory
Background
choice between different alternatives
choose a preferred alternative over a nonpreferred
preference might be characterized in terms of severalfactors
Mathematical Formalization by Fishburn (1970)
based on binary relations
preference as a weak order
utility as an order preserving function from alternatives toreal numbers
göttingen
Decision Making – an Overview
Utility Theory
Utility Theory
decision under certainty
assign a number (utility) to each alternative such that
an alternative is preferred over another if and only if it’sutility is hgher
the utility of an alternative is the sum of the utilities of itsfactors
question of strength of preference
decision under uncertainty
preference based on utilities of consequences
and probabilities of theses consequences
probilities are highly subjective, confidence a person has inthe truth of a proposition
göttingen
Decision Making – an Overview
Utility Theory
What the World Thinks of Dec Mak göttingen
Decision Making – an Overview
Social Decision Making
Social Decision Making
game theory
characterizes the problem of social interaction
by a payoff matrix for multiple players and individual actions
Classical game theory (rational decisions!) predicts thereaching of a Nash equilibrium.
Nash equilibrium
a set of strategies for all players from which noplayer can increase their payoffs by altering theirstrategies individually.
göttingen
Decision Making – an Overview
Social Decision Making
Social Decision Making
Ultimatum Game Trust Game
göttingen
Decision Making – an Overview
Social Decision Making
göttingen
Decision Making – an Overview
Mentalizing and Theory of Mind
Mentalizing and Theory of Mind
Mentalizing
repeated playing with the same other players
accumulation of information about the behaviors
theory of mind is
the ability to representmental states of others:
beliefs
desires
knowledge
theory of mind
might play a key role inoptimizing decision makingstrategies during socialinteractions
mostly unique to humans
göttingen
Decision Making – an Overview
Mentalizing and Theory of Mind
Prisonners Dilemma Game
Two players choose toCooperate or to Defect.
They learn about the other’schoice afterwards.
They get reward according tothe matrix on the left.
göttingen
Decision Making – an Overview
Mentalizing and Theory of Mind
göttingen
Decision Making – an Overview
Reinforcement Learning
Reinforcement Learning
Conditioning
Classical Conditioning
reward or punishment are delivered independently of anyactions
Instrumental Conditioning
reward or punishment are based on the animals behaviour
Reinforcement Learning
learning about stimuli or actions solely on the basis ofreward or punishment
minimally supervised
göttingen
Decision Making – an Overview
Reinforcement Learning
Reinforcement Learning
Predict reward: value function
V (s) = E [r(t) + γr(t + 1) + γ2r(t + 2)|s(t) = s]
Q(s, a) = E [r(t) + γr(t + 1) + γ2r(t + 2)|s(t) = s, a(t) = a]
Select action:
greedy: a = argmaxQ(s, a)
Boltzmann: P(a|s) ∝ exp[βQ(s, a)]
Update prediction: TD error
δ(t) = r(t) + γV (s(t + 1)) − V (s(t))
∆V (s(t)) = αδ(t)
∆Q(s(t), a(t)) = αδ(t)
göttingen
Decision Making – an Overview
Reinforcement Learning
RL in Basal Ganglia göttingen
Decision Making – an Overview
Actor-Critic Model
Actor-Critic Model
Critic reward prediction
Actor action selection
Learning external reward rinternal reward δ
göttingen
Decision Making – an Overview
Sutton-Barto-model
Temporal Difference Model
a model of classical conditioning
”reinforcement in classical conditioning is the timederivative of a composite association combining innate(US) and acquired (CS) associations”
learning the rewards to base decisions on
∆V = (level of US processing) × (level of CS processing)
∆Vi = βY × αi Xi
göttingen
Decision Making – an Overview
Sutton-Barto-model
Sutton-Barto-model göttingen
Decision Making – an Overview
Sutton-Barto-model
Summarizing the Overview
Decision Making before Neuroscience
the basis of human decisions is interesting for a wide fieldof research
very important for money making
mathematical formalization in game theory
phenomenological models of rationality
Decision Making in the Neurosciences
learning theories from AI research
finding neural correlates of everything
structurally realistic models of decision making pathways inthe brain
who noticed the 10 differences to the introduction?
göttingen
Decision Making – an Overview
Sutton-Barto-model
göttingen