Games in Humans and Non-human Primates: Scanners to Single Units

The prospects for game theoretical approach to neuroscience of decision makingneuroscience of decision making

Prepared for Neuroeconomics book study

Chapter 6. by Sanfey and Dorris2009.04.07, 07p.m.

Kyongsik Yun, Ph.D. CandidateKAIST

yunks@kaist.edu

From matrix to real worldThe gap between the experiments and real world in decision making studies

• Traditional decision making studies were done in the confined experimental settings, isolating a specific

• Traditional decision making studies were done in the confined experimental settings, isolating a specific aspect of complex decision making (e.g. risky, uncertain, ambiguous).

• However, real life decisions are more complex and made in the context of a social interaction.made in the context of a social interaction.

The combination of game theoretical tasks and modern neuroscientific methods is the solution.

• This chapter examines– Game theory– Game theory– Invasive electrophysiological techniques in

monkeys during game-theoretic tasks– Non-invasive imaging techniques in humans

during game-theoretic tasks

Game theory attempts to mathematically capture behavior in strategic situations, in which an individual's success in

making choices depends on the choices of others. – von Neumann and Morgenstern (1947) Utility Theory

Utility theory is unable to explain why people are often simultaneously attracted

• Prospect theory – (Kahneman & Tversky 1979)

Utility theory is unable to explain why people are often simultaneously attracted to both insurance and gambling.

Real world decision making is less selfish and more willing to consider reciprocity and equity (Camerer 2003).

Ultimatum GamePlayer 1 can offer a fair (F) or unfair (U) proposal; player 2 can accept (A) or reject (R).

Guth et al. 1982Guth et al. 1982

Nash Equilibrium is at odds: Low offers of less than 20% of the total amount are rejected about half of the time.Neuroscience has begun to offer clues as to the mechanisms underlying these decisions.

Trust Game

48 = 12 MU own endowment + 36 MU tripled transfer

Baumgartner et al. Neuron 2008

48 = 12 MU own endowment + 36 MU tripled transferA rational and selfish trustee will never honor the trust given by the investor. In real situation, a majority of investors send some amount of their money to the trustee.

Prisoner’s dilemma game-simultaneous trust game

• The Nash Equilibrium is mutual defection.• Players exhibit 50% mutual cooperation. • Players exhibit 50% mutual cooperation.

Cooperate

Cooperate

defect

Cooperate

defect

Public goods game: a generalized form of PDG

• The self-interested solution is to free-ride and hope that everyone else contributes.and hope that everyone else contributes.

• Players on average contribute about half of their endowment to the public good.

Mixed strategy game (e.g. matching pennies and the inspection game)• A pure strategy provides a complete

definition of how a player will play a game.definition of how a player will play a game.• A mixed strategy is an assignment of a

probability to each pure strategy.

• The Nash Equilibrium is to select randomly with equal probabilities. with equal probabilities.

• Players attempt to infer the strategy of our opponent, using theory-of-mind processes.

It is unclear whether the decisions emerge from strategic or altruistic motivations.

• Ultimatum Game proposer behavior– fairness? Fear to be rejected?– fairness? Fear to be rejected?

• Examining these games in a neural context can begin to offer clues as to the motivations behind the decisions. motivations behind the decisions.

The benefits of combined approach between experimental economics and neuroscience methods

• More precise characterizations of behavior (how decisions are actually behavior (how decisions are actually made)

• Neuroscience can provide important biological constraints on the processes involved.involved.

The animal model (Rhesus Macaque)

• To investigate “black box” during social interactionsinteractions

• For studying higher-order decision processes

• General organization of their nervous system is similar to that of humans

• Monkeys and humans display • Monkeys and humans display comparable strategies in mixed-strategy games

Advantages and disadvantages of a systems neurophysiology (invasive animal model) approach• Advantage:

– direct access to the neural substrate– Exquisite temporal (<1ms) and spatial (ind. neurons) resolution– Exquisite temporal (<1ms) and spatial (ind. neurons) resolution– Artificial manipulation of neuronal activity can provide causal evidence, complementing the correlational evidence provided by neuronal recordings.

• Disadvantages– Limitations exist in using non-human primates to infer the neural processes underlying human social interactions– Not suitable for more sophisticated games (UG, PDG)– Not suitable for more sophisticated games (UG, PDG)– No verbal instruction and using only operant conditioning techniques

Saccadic ( jumping-fixating) eye movements

• A saccade is a fast movement of an eye, head or other part of an animal's body head or other part of an animal's body or device. It can also be a fast shift in frequency of an emitted signal or other quick change.

Benefits of visuosaccadic system investigation for decision making

• Efficiently extract visual information in • Efficiently extract visual information in perceptual decision making

• The neural circuitry underlying visual processing and saccadic control is well understood. understood.

Adapting games for non-human primatesLIP: estimating the desirability of sensory stimuliSC: selecting/preparing actions, (control eye movement)PFC: evaluating the consequences of actionsPFC: evaluating the consequences of actions

LIP – gateway for the emergence of the brain

Newsome and Shadlen 2001 Nature

emergence of the brain

Encoding the desirability of choice stimuli in LIP

• At the end of visual processing streamprocessing stream

• Encode the saliency of visual targets

Platt and Glimcher 1999 Nature

Sugrue, Corrado, Newsome 2005 NRN

Visuosaccadic version of mixed-strategy inspection game

Nash Eq.

Dorris and Glimcher 2004 Neuron

Inspection cost0.5 0.9 0.3

LIP firing rates varies with desirability, not probability of outcome

Dorris and Glimcher 2004 Neuron

Correlation between LIP Firing Rates and an Estimate of the Relative Desirability of Choices

Dorris and Glimcher 2004 Neuron

Evolving response selection in midbrain superior colliculus

• Intimately involved in saccade generation• See Figure 6.4 (Dorris unpublished)• See Figure 6.4 (Dorris unpublished)– B) Preferred (black)/ unpreferred (red)– C) Stimulus induced saccades deviate

slightly towards the target.– D) angular deviation of saccades increases – D) angular deviation of saccades increases

as the time of target presentation approaches. (일찍 자극, 적게 변화)

Evaluating the consequences of actions in frontal cortex

• dlPFC is involved in integrating choice (i.e. left vs right) and reward (i.e. (i.e. left vs right) and reward (i.e. rewarded or unrewarded) information.

• dACC encoded critical information about the temporal delay of previous rewards within a sequence of responses (Seo and within a sequence of responses (Seo and Lee 2007)

General significance of decision making studies in non-human primates

• Recording single neurons allow for moment-to-moment correlations moment-to-moment correlations between neuronal activity and behavioral responses

• Providing unprecedented insight into the neuronal mechanisms underlying neuronal mechanisms underlying stochastic choice.

Games in humans• Imaging studies of social interaction– fMRI, PET– fMRI, PET

– Stimuli of other human faces (static, 2D) (Winston et al. 2002)

–Mentalizing (Gallagher et al. 2000)–Mentalizing (Gallagher et al. 2000)–Moral reasoning (Greene et al. 2001)

Pioneering works of its kind

Automatic and intentional brain responses during evaluation of trustworthiness of faces

Explicit judgments: whether an individual was trustworthyImplicit judgments: an unrelated age assessment

STS

Insula

l. Amyg r. Amyg.

The findings extend a proposed model of social cognition by highlighting a functional dissociation between automatic engagement of amygdala versus intentional engagement of STS in social judgment.

Winston, Strange, O’Doherty, & Dolan NN 2002

Reading the mind in cartoons and stories

medial prefrontal cortex (paracingulate cortex)Gallagher et al. 2000

Problems in the previous studies• Does the pattern of brain activation in

response to the picture of a static, 2D face response to the picture of a static, 2D face accurately reflect the brain’s response to the dynamic, embodied faces that we encounter in everyday life?

• Is the pattern of brain activation in response to reasoning regarding hypothetical, fictitious scenarios the same response to reasoning regarding hypothetical, fictitious scenarios the same as when grappling with significant real life social problems?

Solutions to improve ecological validity of experiments in neuroeconomics

• To interact with other people in real social exchanges from outside the social exchanges from outside the scanner (McCabe et al. 2001)

• Hyperscanning technology (Montague et al. 2002)– Data collection efficiency– Data collection efficiency– Open new vistas in social cognitive

neuroscience

Another solution is to manipulate specific neurotransmitter systems

• Tryptophan depletion can be used to decrease brain serotonin levels, reducing cooperative behavior serotonin levels, reducing cooperative behavior (Wood et al. 2006)

• Oxytocin elevation increases trust (Kosfeld et al. 2005)

• TMS to temporarily activate or deactivate a brain region and then examine its effects on decision making (van’t Wout et al. 2005)making (van’t Wout et al. 2005)

• Lesion study (Koenigs and Tranel 2007)

Trust Game behavior

Oxytocin increased trust.Oxytocin increased trust.

Baumgartner et al. Neuron 2008

Oxytocin: filled barsPlacebo: open bars

Fool me once, shame on you. Fool me twice, shame on oxytocin.

Baumgartner et al. Neuron 2008

What is the role of PFC in emotion vs. cognition conflict decision making?

Economic rationality:• economic value maximization

Social rationality: • Social value maximization• economic value maximization

• Absolute valuation• vmPFC lesion -> R↑

• Social value maximization• Fairness monitoring• Relative valuation• dlPFC disruption -> A↑

Knoch et al. Science 2006Koenigs & Tranel, J Neurosci. 2007

Current research direction:social motivation reward

• Striatum: scale reward magnitude (Cromwell and Schultz 2003), specifically the magnitude of monetary reward or punishment (O’Doherty 2004, Knutson and Cooper 2005)punishment (O’Doherty 2004, Knutson and Cooper 2005)• Striatum may register social prediction errors to guide decisions about reciprocity (Rilling et al. 2002)• Caudate: related to how much reciprocity the investor had shown on previous trials, “intention to trust” (King-Casas et al. 2005)• caudate: altruistically punish (de Quervain et al. 2004)• Social altruism: the striatum was engaged both by receiving money and by donations to charity (Moll et al. 2006), and • Social altruism: the striatum was engaged both by receiving money and by donations to charity (Moll et al. 2006), and the activation was enhanced when the donation was voluntary as opposed to forced (Harbauch et al. 2007). -> important implications for informing public policy

Caution should be used when attempting to “reverse engineer” from patterns of brain activity to cognitive and social processes

• Correlation analysis, TMS modulation, or patient work is crucial to buttress the causal relationship • Correlation analysis, TMS modulation, or patient work is crucial to buttress the causal relationship between behavior and brain activity. • These results appear to demonstrate that complex social processes recruit more basic mechanisms within the human brain, providing support for the notion that the brain uses a common reward metric. This also furthers the connection between the disparate branches of neuroeconomics from primary disparate branches of neuroeconomics from primary and secondary rewards (food, money) to more abstract social rewards (reciprocity, fairness).

Competition, Cooperation and Coordination

• Classical models of decision making, • Classical models of decision making, both utility theory and game theory, have largely ignored the influence of emotions on how decisions are made, but recent research has begun to but recent research has begun to demonstrate their powerful effect.

The emotional brain The Papez circuit theory (1937) of the functional neuroanatomy of emotion

Emotional experiences or feelings occur when the cingulate cortex integrates these signals from the hypothalamus with

stream of thinking

stream of feeling

these signals from the hypothalamus with information from the sensory cortex.

Output from the cingulate cortex to the

Dalgleish 2004 NRN

Output from the cingulate cortex to thehippocampus (3) and then to the hypothalamus (4) allows top–down cortical control of emotional responses.

MacLean’s limbic system theory of the functional neuroanatomy of emotion

the hippocampus received sensory inputs from the outside world as well asinformation from the internal bodily information from the internal bodily environment (viscera and body wall).

Dalgleish 2004 NRN

Emotional experience was afunction of integrating these internal and external information streams.

Map of brain areas commonly activated in social decision-making studies.

Sanfey 2007 Science

Theory of Mind: STS, MPFC, OFC

Neuroscientific studies of emotions• Negative emotional states have been observed behaviorally as a result of both inequity and non-reciprocity, such as unfair offers in a UG.unfair offers in a UG.• These emotional reactions have evolved precisely to foster mutual reciprocity, to make reputation important, and to punish those seeking to take advantage of others (Nawaket al. 2000)• Neuroscienctific studies of this nature offer the potential to go beyond speculation and to examine the causal relationship between an emotional reaction and subsequent relationship between an emotional reaction and subsequent social decision, as well as investigating whether areas specialized for the processing of basic emotions may be coopted for more complex affective reactions.

Behavior of social interactionin the Ultimatum Game

40Sanfey et al. Science, 2003

Participants had a stronger emotional reaction to unfair offers from humans than to those from computers.

Neuroimaging results of the Ultimatum Game

Sanfey et al. Science, 2003

Anterior insula: pain, distress, hunger, thirst (Denton et al. 1999), autonomic arousal (Critchley et al. 2000)arousal (Critchley et al. 2000)Right anterior insula: aversive conditioning (Seymour et al. 2005)Defection rate in PDG is correlated with right anterior insula activation (Rillinget al. 2008) Anterior insula may play a role in marking a social interaction as aversive, and thus discouraging trust.

dlPFC is linked to cognitive processes • Frontal, “top-down” processes in reward studies suppress striatal activation. studies suppress striatal activation.

• TMS increased acceptance rate of unfair offers, providing strong evidence for a causal relationship between activation in this area and social decision making (Knoch et al. 2006)(Knoch et al. 2006)–Why social? Disruption of dlPFC using TMS

might reduce cognitive processes to accept the offer, decreasing acceptance rates.

Modulating affective system in the Ultimatum Game

• People in a sad mood (watching a 5-minute video rated as “sad”) reject more 'unfair' offers (Harle and Sanfey 2007).video rated as “sad”) reject more 'unfair' offers (Harle and Sanfey 2007).• Examining decision making performance in participants with disregulated emotional processing, such as patients with depression or schizophrenia, may be a useful future avenue of research. useful future avenue of research. • Contradictory study: vmPFC lesion patients also reject unfair offers more frequently than do controls (Koenigs and Tranel 2007)

Significance of neuroscience approach to decision making

• Decision making appears to involve the interaction among multiple subsystems governed by different parameters and interaction among multiple subsystems governed by different parameters and possibly even different principles. • Neuroscience approach to decision making has the potential to inform economic theories of interactive decision making (inequity aversion, social utility functions)(inequity aversion, social utility functions)• Economic models based on the underlying neural patterns may provide useful constraint.