The partial reinforcement extinction effect (PREE)

Frode SvartdalUniversity of Tromsø

Oct. 2013

Extinction: Basics Extinction is defined in terms of a

reinforcement process Extinction contingencies

The stimulus (SR or US) is discontinued The learning contingency is discontinued

Extinction process The conditioned response is reduced

(strength, frequency, etc.) Relearning, … not forgetting

Catania, 1984)

Extinction: BasicsOperant conditioning

Extinction: BasicsClassical conditioning

Factors affecting the extinction rate In general: Fast acquisition / high rate of

responding fast extinction Amount of reward

High fast extinction Variability

Stimulus Response Reinforcement

Some forms of learning do not extinguish (easily)

Evaluative conditioning (e.g., Diaz, Ruiz, & Beyens, 2005)

= high ext. persistence

Factors affecting the extinction rate

Partial Reinforcement Extinction Effect

Partial (Intermittent) Reinforcement (PRF) increased extinction response

Continuous Reinforcement (CRF) reduced extinction persistence

First demonstrations

Operant conditioning;free operant; rats;Skinner (1938)

Classical conditioning;blink response; students;Humphreys (1939)

100%

50%

Ferster & Culbertson, 1975

Free operant

PRF

CRF

Free operant

Compared to CRF:

PRF • higher asymptotes• more persistent responding under extinction

EXTINCTION

Rats, maze running speed under extinction (Weinstock, 1954)

CRF

PRF (30%)

Classical conditioning (rats): PREE

25%

50%

100%

Extinction

PRF response rateLOWER than CRF response rate

15%

Classical conditioning; eyelid; human subjects(Svartdal & Flaten, in prep.)

Operant conditioning; humans;Svartdal, 2003, Exp. 4

Conclusions (… preliminary)

PREE is a very robust outcome Measures & species

Bar pressing, rats Maze running, rats Pecking, pigeons Blink reflex, humans, rabbits …

Contingency Operant/instrumental

Discrete trial Free operant

Classical

But…

How general is the PREE? Reversed PREE observed under some

conditions Generalized PREE observed under some

conditions Alternative methods of analysis

Nevin (1988): ”PREE is an artefact because of wrong method of analzing extinction performance”

Response unit issue PREE or not dependig on how the response

is defined (Mowrer & Jones, 1945!

Reversed PREE

What happens if the subject is exposed to a mixture of PRF and CRF contingencies?

Reversed PREE

Pavlik & Carlton, 1965: Rats; bar pressing, free operant

Gr. 1: Single contingency; CRF Gr. 2: Single contingency; PRF Gr. 3: Two signalled schedules

alternated for the same subjects; CRF + PRF

Reversed PREEConventionalPREE

Reversed PREE

ReversedPREE

Reversed PREE

Pavlik & Carlton (1965): Single reinforcement schedules (CRF vs. PRF) in

between-groups experiments PREE Two schedules (CRF vs. PRF) for the same subjects

Reversed PREE

Other research Reversed PREE observed Generalized PREE (overall increased persistence,

but no difference between conditions) Conventional PREE rarely if ever observed in within-

subjects manipulations of CRF - PRF

PREE as a generalization: Ecological validity

If applied to a situation with a very specific schecule for a specific behavior PREE

Example:

Single mother – child is begging for toys only from mom

If applied to various situations with mixed contingencies Reversed PREEGeneralized PREE

Example:

Mother and father – child begs for toys from both

Response unit issue

Free operant responding: What is the response unit?

Mowrer & Jones,1945:

What should be counted as the response unit - single responses or the unit of responses required for reinforcement?

Free-operant Intermittent reinforcemet, e.g., FR4

Response unit

FR4

Reinforced responses

PREE Total responses

Total responses / reinforcement ratio

Reversed PREE

Nevin: PREE is an artefact

PREE: Alternative analyses

Nevin, 1988: Behavioral momentum• ”RPREE” is the rule – the response is

stronger following CRF• in free-operant responding (but not in discrete-

trial experiments) • following extended training

• Extinction performance• Traditional measure: Number of responses • Nevin: Slope of the extinction curve

PREE

RPREE

SHORT LONG

Absolute numberof responses

Relative to initial ext response level

Nevin, 1988

PREE vs. RPREE – important variables

Dependent measure No. of responses vs. relative change

Type of situations Free operant vs. discrete trial

Complexity of situation One vs. more schedules (e.g., multiple

schedule) Design

Between groups vs. within subjects

PREE typically observed

Measure Number of responses

Situation Discrete trial

Schedule Single

Design Between-groups manipulation of reinforcer rate

Other CRF schedule must be 100%

PREE: My interests

Interaction PREE & Reversed PREE Cognition (verbalization) related to

behavioral PREE

The experimental situation

”Computer responses”presented

Left, right

Subject responsesrecorded

Left, right

The experimental situation

Task Complete a four-response chain of responses started by

the computer E.g.: Computer: L R Subject: R L

Instructed task: Identify and apply the functional rule(s) ”Obtain as many correct answers as you can.”

Rules (depending on experiment) ”Repeat computer sequence” ”Reverse computer sequence”

Feedback (visual, autitory) for correct answer; nothing happens if answer is incorrect

The experimental situation

Manipulations (between groups and/or within groups)

RuleReverse (typically used)Repeat

ContingencyCRF (100%)PRF (20-60%)

The experimental situation

Reward rate manipulated Between groups Within subjects (multiple schedule)

Discrete trial situation; fixed number of trials 180 acquisition trials 40 extinction trials

Conventinal PREE; operant responding; students; Svartdal, 2003, Exp. 4

Reversed & conventional PREE; operant responding; students; Svartdal, 2000

Reversed PREE Purpose: Explore the relationship

between PREE and RPREE PREE vs. RPREE: Contradiction or

compatible effects? Method

Independent groups: PRF and CRF Within: CRF and PRF

Svartdal, 2000 ctd.

Multiple schedule, alternating Group 40/40

Half trials (signalled): 40% Half trials (signalled): 40%

Group 80/80 Half trials (signalled): 80% Half trials (signalled): 80%

Group 80/40 Half trials (signalled): 80% Half trials (signalled): 40%

PRF

”CRF”

”CRF” + PRF

PREE80%40%

* No. of responses: RPREE* Relative change: No difference

Svartdal, 2000 ctd.

Relationship between schedule components

Simplest assumption: Modulation between component schedules: 60% + context = 60% reference 60% + context = 100% reduced

persistence 60% + context = 20% increaced

persistence

Performance of a 60% schedule depending onother schedule = 100%, 60%, or 20%Svartdal, 2000

Svartdal, F. (2000). Persistence during extinction: Conventional and Reversed PREE under multiple schedules. Learning and Motivation, 31, 21-40.

Cognition in PREE

• Currently: Strong cognitive arguments to interpret conditioning in terms of cognition

• Classical conditioning: Lovibond & Shanks, 2002• Operant conditioning: Shanks & St John, 1994• Implicit learning doubted: Shanks, 2005• Extinction: Lovibond, 2004

• Basic argument:

CONTINGENCY CONSCIOUS APPREHENSION BEHAVIORAL CHANGE

CONTINGENCY CONSCIOUS APPREHENSION NO BEHAVIORAL CHANGE

• Large number of studies supporting this assumption

Cognition in PREE

So, since the behvioral PREE is very robust, a ”cognitive PREE” must be easy to measure

Basic prosedure: Behavioral acquisition under

100% vs. 60% reinforcer rate Measurement of verbalized PREE

Cognition in PREE

Prediction of persistence:”How likely is it that you will continueresponding if reward no longer appears?”

Several experiments havedemonstrated no sensitivityto learning history inpredictions

T rl1 T rl3 T rl5 T rl7 T rl9 T rl11 T rl13 T rl15 T rl17 T Ext0 ,0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1,0

Re

spo

nse

s (pro

po

rtion

of p

ossib

le)

100/100 60/60

3 extinction trials;immediate behavioralsensitivity

No differencein predictions

Svartdal & Silvera, in prep.

Cognition in PREE

Retrospective judgments:”How many responses did you emit afterreward no longer appeared?”

Subjects are very accurate in descrbing their own behavior, including their own extinction persistence

Cognition in PREE

Svartdal, F. (2003). Extinction after partial reinforcement: Predicted vs. judged persistence. Scandinavian Journal of Psychology, 44, 55-64.

Meta-cognitive PREE?

We all have long experience with various contingencies

Maybe a ”meta-cognition” evolves: Uncertain outcomes Persist Certain outcomes Quit

Meta-cognitive PREE?

Scenarioes presented to subjects, manipulation Reliable outcome vs. Unreliable outcome

Persistence judgments of behavior

Meta-cognitive PREE?

Naive students: No effect ofoutcome manipulation

Meta-cognitive PREE?

Psychology students(have read about PREE)

Naive students

Meta-cognitive PREE?

Svartdal, F. (2000). Persistence during extinction: Are judgments of persistence affected by contingency information? Scandinavian Journal of Psychology, 41, 315-328.

PREE: Theory

Mowrer & Jones: Diskriminasjonshypo- tesen PRF:

Læringbetingelsene ekstinksjonsbetingelsene Generalisering til ekstinksjon

CRF: Læringbetingelsene # ekstinksjonsbetingelsene Liten generalisering til ekstinksjon

PREE: Theory

Amsel: Frustrasjonshypotesen PRF:

Forventning om belønning frustrasjon når belønning uteblir

Frustrasjons-cues assosieres med læringssituasjonen

Under ekstinksjon: Frustrasjon pga uteblitt belønning

Læringssituasjonen ekstinksjonssituasjonen

CRF: Frustrasjon oppstår ikke under læring Læringssituasjonen # ekstinksjonssituasjonen

PREE: Theory

Capaldi: Sequential hypothesis PRF:

Ikke-belønnede trials blir signal på at belønning snart vil følge: … N N N R N N N R …

Dvs.: Det opparbeides en forventning om belønning når belønning uteblir

Under ekstinksjon: Mange responser pga forventning om belønning

CRF: Ingen erfaring med uteblitt belønning under

læring Under ekstinksjon: Få responser

PREE: Theory

Status: Diskriminasjonshypotesen står svakt Amsels hypotese står rimelig sterkt Capaldis hypotese står ganske sterkt Nevins modell: Ingen hypotese i vanlig

forstand Discrete-trial-situasjonen

Capaldi og Amsel dominerende Fri-operant-situasjonen

Svak teoretisk forståelse