Replicable Effects of Primes on Human Behavior

B. Keith PayneUniversity of North Carolina at Chapel Hill

Jazmin L. Brown-IannuzziUniversity of Kentucky

Chris LoerschUniversity of Colorado

The effect of primes (i.e., incidental cues) on human behavior has become controversial. Earlystudies reported counterintuitive findings, suggesting that primes can shape a wide range of humanbehaviors. Recently, several studies failed to replicate some earlier priming results, raising doubtsabout the reliability of those effects. We present a within-subjects procedure for priming behavior,in which participants decide whether to bet or pass on each trial of a gambling game. We report 6replications (N � 988) showing that primes consistently affected gambling decisions when thedecision was uncertain. Decisions were influenced by primes presented visibly, with a warning toignore the primes (Experiments 1 through 3) and with subliminally presented masked primes(Experiment 4). Using a process dissociation procedure, we found evidence that primes influencedresponses through both automatic and controlled processes (Experiments 5 and 6). Results provideevidence that primes can reliably affect behavior, under at least some conditions, without intention.The findings suggest that the psychological question of whether behavior priming effects are realshould be separated from methodological issues affecting how easily particular experimental designswill replicate.

Keywords: priming, behavior priming, automatic cognitive processes, decision making

The effects of primes on human behavior are attracting atten-tion, again. The first time was in the 1990s when researchersreported some of the earliest behavior priming effects. Exposingresearch participants to words related to rudeness, for example,made them more likely to interrupt the experimenter; exposingthem to words related to the elderly slowed their walking speed;and exposing them to photos of African Americans increasedaggressive behavior (Bargh, Chen, & Burrows, 1996). In relatedresearch, participants correctly answered more general knowledgequestions after thinking about the qualities of professors than afterthinking about the qualities of soccer hooligans (Dijksterhuis &van Knippenberg, 1998). Subsequently, hundreds of studies re-ported evidence that incidental exposure to words, pictures, orother cues could influence a wide range of behaviors, includingpro-social behavior (Macrae & Johnston, 1998; Shariff & Noren-zayan, 2007), consumer choices (e.g., Berger & Fitzsimons, 2008),and voting decisions (Berger, Meredith, & Wheeler, 2008). Infollow-up questionnaires, subjects in these studies generallyshowed no awareness that their behaviors had been influenced bythe primes.

Earlier research had established that presenting words as primescould facilitate later memory and perceptual processing of thosesame objects (e.g., Jacoby & Dallas, 1981; Tulving & Schacter,1990) and semantically related stimuli (e.g., Meyer & Schva-neveldt, 1971; Neely, 1977). The effects of primes on higher levelbehaviors like social interactions and decision making, however,suggested that primes might have effects more powerful thanpreviously acknowledged. Behavior priming effects are importantfor psychological theory because they provide evidence about theinfluence of automatic or unconscious processes on behavior.Many dual-process theories have drawn on behavior priming ef-fects to explain how automatic and controlled processes relate toeach other and how they interact to drive behavior (e.g., Chaiken& Trope, 1999; Kahneman, 2011; Sherman, Gawronski, & Trope,2014). Recently, however, behavior priming is attracting attentionfor a different reason. Several articles have reported failures toreplicate influential priming studies. For example, Harris, Coburn,Rohrer, and Pashler (2013) reported two experiments that found noeffects of achievement primes on test performance, failing toreplicate the results of Bargh, Gollwitzer, Lee-Chai, Barndollar,and Trötschel (2001). In another article, Shanks and colleagues(2013) reported nine experiments that failed to replicate the find-ings of Dijksterhuis and van Knippenberg (1998), which indicatedthat priming subjects with professor versus soccer hooligan af-fected performance in answering general knowledge questions.

When replications find different results than an original study,there are many possible reasons. In some cases, the replicationstudy may not follow the exact protocol of the original study. Evenwhen the procedure is replicated exactly, two different samplesmay produce different estimates of effect size due to sampling

B. Keith Payne, Department of Psychology and Neuroscience, Univer-sity of North Carolina at Chapel Hill; Jazmin L. Brown-Iannuzzi, Depart-ment of Psychology, University of Kentucky; Chris Loersch, Departmentof Psychology and Neuroscience, University of Colorado.

We thank Larry Jacoby for his advice on implementing the processdissociation procedure within the gambling paradigm used here.

Correspondence concerning this article should be addressed to B. KeithPayne, University of North Carolina at Chapel Hill. E-mail: payne@unc.edu

THIS ARTICLE HAS BEEN CORRECTED. SEE LAST PAGE

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Journal of Experimental Psychology: General © 2016 American Psychological Association2016, Vol. 145, No. 10, 1269–1279 0096-3445/16/$12.00 http://dx.doi.org/10.1037/xge0000201

1269

variability, even if they are drawn from a population with the sametrue effect (Stanley & Spence, 2014). Another reason for differ-ences across studies is that unidentified moderating variables maydiffer between studies. In such cases, establishing the role of themoderating variable requires specifying a priori theoretical predic-tions. Several relatively new theoretical frameworks have recentlymade progress toward this goal (Cesario, Plaks, Hagiwara, Na-varrete, & Higgins, 2010; Loersch & Payne, 2011; Schröder &Thagard, 2013; Wheeler, Demarree, & Petty, 2007; see Molden,2014a for a review). Finally, low statistical power is a majorproblem for many original and replication studies. Underpoweredstudies result in effect size estimates with large confidence inter-vals, raising the risk of both false positives and false negatives (foran in-depth discussion of replication issues, see the November2012 special issue of Perspectives on Psychological Science).

Despite these complications, it seems that the most commoninterpretation of a null replication is that the effects in question arenot real. For example, after considering and rejecting severalprocedural and statistical explanations for why their results mayhave differed from the original study, Harris and colleagues,(2013) concluded, “Another obvious possibility is that the high-performance-goal prime results of Bargh et al. are invalid, and maybe due, for example, to Type 1 errors” (p. 6). Similarly, Pashler,Coburn, and Harris (2012) concluded, “Another possibility, how-ever, is that the Williams and Bargh results are simply not valid,representing, for example Type 1 errors” (p. 6). And Shanks andcolleagues (2013) concluded, “A fifth and final possibility is thatsome or all of the published results on intelligence priming werefalse positives” (p. 7).

Although scholarly articles like the aforementioned have madeefforts to consider alternative methodological and statistical expla-nations, the message understood by the popular press is often lessnuanced. A Chronicle of Higher Education blog, for example,described a replication project by saying,

The project is part of Open Science Framework, a group interested inscientific values, and its stated mission is to ‘estimate the reproduc-ibility of a sample of studies from the scientific literature.’ This is amore polite way of saying ‘We want to see how much of what getspublished turns out to be bunk.’ (Bartlett, 2012)

Despite the complexities of debates over replications, manyreaders see nonsignificant replication attempts and read, “bunk.”

A meta-analysis of priming studies that included words asprimes and behavioral dependent variables (Weingarten et al.,2016) found a small but significant average effect (d � .35; 95%CI [.29, .41]). Using several methods for estimating publicationbias (e.g., trim-and-fill analysis, funnel plot, and p curve), themeta-analysis not only found evidence for the existence of publi-cation bias, but also concluded that publication bias alone cannotexplain the average effects. These analyses are important, but theyare unlikely to change the minds of critics because it is alwayspossible that the methods insufficiently corrected for publicationbias. We believe that to better understand the effects of primes onhigher level human behavior, it is important to be clear about thephenomena in question, and to better separate psychological ques-tions regarding how primes influence behavior from the methodsused to study these questions.

There Is No Such Thing as “Social Priming”

Many readers of this literature have interpreted failures to rep-licate experimental effects within specific paradigms as evidenceagainst a broad category of phenomena that have collectively beenlabeled social priming. But, what does social priming mean? Thereis not, in principle, any meaningful distinction between semanticactivation of concepts that have to do with other people andconcepts that do not. The original use of the term appears to betraceable to an open letter by Daniel Kahneman, sent to a numberof social psychology researchers. The letter expressed concernover some then-recent studies that had failed to replicate previ-ously published priming effects (described in the preceding text).It was addressed to “a collection of people who were described tome . . . as students of social priming” (Kahneman, 2012). The termwas not defined in the letter, however it seems to be used to referto the researchers who conduct studies (i.e., social psychologists)rather than to a phenomenon itself. The term social priming wasthen repeated in popular press coverage of the exchange (e.g.,Yong, 2012) and in social media discussions (e.g., Zwaan, 2013;Neuroskeptic, 2015). This conceptual confusion has led to doubtsabout entire areas of research based on a handful of specific studiesthat failed to replicate particular experimental paradigms.

A term based on the disciplinary affiliations of researchersrather than a description of the phenomenon in question is clearlynot useful for advancing scholarly understanding of the topic. Theterm does not refer to any meaningful class of phenomena that canbe grouped together in a coherent way. We argue that there is nosuch thing as social priming—not because that term refers to anunreliable phenomenon, but because it is conceptually incoherent.The conceptual confusion inherent in that term has led manycommentators to interpret a handful of specific studies with nullresults as evidence against a broad area of research comprisingmultiple phenomena and multiple methods (Kahneman, 2012;Yong, 2012).

Several authors have recently attempted to clarify the empiricalphenomena of interest in this debate, with a relatively high degreeof consensus. Molden (2014b) suggested that the criteria are (1)stimulation of a mental representation that influences subsequentevaluations, judgments, or actions; and (2) this effect occurs eitherwithout awareness of the prime’s influence or without intention toutilize the prime stimulus (although not necessarily without aware-ness of the stimulus presentation). Similarly, Doyen, Klein, Si-mons, and Cleeremans (2014) suggested that the features commonacross procedures include (1) experimenters presenting a primestimulus, (2) the prime activating an internal representation, (3) theactivated representation influencing other representations (this dis-tinguishes semantic or associative priming from repetition prim-ing), and (4) activated representations leading to changes in be-havior. Finally, Ferguson and Mann (2014) summarized the issuesuccinctly:

We suspect that what some people mean when they talk about socialpriming is behavioral or goal priming. Behavioral priming occurswhen incidental processing of a cue changes behavior . . . , whereasgoal priming occurs when exposure to some cue shifts one’s goalpursuit (which is usually measured via behavior . . .). (p. 37)

Consistent with these descriptions, we are concerned with un-intended effects of primes on behavior in the present article.

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

1270 PAYNE, BROWN-IANNUZZI, AND LOERSCH

Critics of behavior priming have often contrasted it with therobustness of semantic priming (e.g., Pashler, Coburn, & Harris,2012; Shanks et al., 2013). In semantic priming, presentation of aword (e.g., DOCTOR) facilitates faster responding to a relatedword (e.g., NURSE) in lexical decision, naming, or perceptualidentification tasks (e.g., Meyer & Schvaneveldt, 1971; Schva-neveldt & McDonald, 1981). Semantic priming refers to the effectsof incidental primes on the processing of subsequent information.The present article, in contrast, concerns whether incidentallyprocessed stimuli can reliably affect not only the processing ofsubsequent information (as in semantic priming) but also a per-son’s actions and decisions (as in behavior priming).

Studies of semantic priming and behavior priming typicallydiffer in many ways. One key difference, on which we focus here,is that behavior priming studies typically use a between-subjectsdesign in which one group of participants receives one kind ofprime and another group receives a different kind of prime. Se-mantic priming, in contrast, uses within-subjects designs in whicheach subject responds to all prime�target combinations over doz-ens or hundreds of trials.

Many within-subjects designs have demonstrated robust prim-ing effects in evaluation or judgment of stimuli. Briefly presentedprime stimuli reliably affect the speed with which target stimuli areevaluated, as in evaluative priming tasks (Fazio, Jackson, Dunton,& Williams, 1995; Fazio, Sanbonmatsu, Powell, & Kardes, 1986);the likelihood of positively evaluating an ambiguous target image,as in the affect misattribution procedure (Payne, Cheng, Govorun,& Stewart, 2005); the likelihood of misidentifying harmless itemsas weapons (Correll, Park, Judd, & Wittenbrink, 2002; Payne,2001); and the likelihood of perceiving trait attributes in ambigu-ous faces (Krieglmeyer & Sherman, 2012). Meta-analytic evidencesuggests these effects are highly replicable (Cameron, Brown-Iannuzzi, & Payne, 2012). These effects have played little role inrecent debates over the replicability of “social priming” or “be-havioral priming” effects. This may be because, like semanticpriming, these paradigms focus on processing of subsequent stim-uli rather than on decisions or actions that are traditionally con-sidered to be “behaviors” (see Ferguson & Mann, 2014).

In our view, the emphasis on the replicability of early primingeffects conflates the important theoretical question with method-ological aspects of specific studies. The theoretical question iswhether, and how, primes affect behavior. Methodological issuesinclude the use of between-subjects manipulations, and the use ofonly one or a few observations per participant for the dependentvariable. Both between-subjects designs and “single shot” mea-surements lead to low statistical power. Studies using such designscan be expected to generate data with large amounts of errorvariance and, as a result, are likely to be difficult to replicate.Critically, these methodological weaknesses would reduce repli-cability regardless of the psychological phenomenon in question.So, we must ask, do reported failures to replicate behavior primingstudies tell us something about the effects of primes on humanbehavior? Or do they tell us about the effects of particular exper-imental design factors on error variance?

In the reported experiments, we aimed to take a step towardseparating the psychological question from the methodologicalquestions. To do so, we tested whether a high-powered within-subjects design, similar to those used to study semantic priming,would produce a robust demonstration of behavior priming. If

primes reliably affect behavior when high-powered experimentaldesigns are used, then it shifts the debate in important ways. Itwould demonstrate, most simply, that primes can affect subsequentbehavior, not just the ability to process subsequent stimuli. Suchfindings, if replicated, would challenge one of the major possibleconclusions in this debate, namely, that priming effects on behav-ior are illusory. Research efforts may then turn productively to-ward systematically identifying the conditions under which primesdo (and do not) influence behavior. We consider in the generaldiscussion some other ways that behavior priming and semanticpriming studies may differ, and suggest additional steps for futureresearch to systematically understand the similarities, differences,and boundary conditions.

Theoretical Rationale for the Present Approach

Rather than replicating previous studies, we start from thecurrent state of knowledge about how and why primes are likely toaffect behavior. We designed a simple gambling task to capture theessential phenomenon of interest, in which a prime stimulus mayunintentionally influence a subsequent behavior. We based ourpredictions on the Situated Inference Model (Loersch & Payne,2011, 2014). The model claims that primes affect behavior whenthe primes make associated thoughts or feelings more accessible,and that accessible information is misattributed to one’s ownresponse to the behavioral options afforded by the situation. Forexample, if a prime makes a subject think about betting and shemistakenly thinks that mental content is self-generated, then she ismore likely to bet. This misattribution or source confusion has animportant relationship to the lack of awareness that is of interest inbehavior priming effects. In behavior priming studies, it is aware-ness of the influence of the primes on behavior that is of interest,rather than awareness of the prime stimuli themselves. Researcherssometimes use masked presentation of primes because subjectscannot be aware of a prime’s influence if they cannot consciouslyidentify the prime. But in such cases, masked presentation ismerely a means for ensuring that subjects are unaware of theprimes’ influence on their behavior (Molden, 2014b). If subjectsmisattribute a primed thought to their own self-generated thoughts,then this misattribution also represents a kind of unawareness,because the person mistakenly experiences influences of theprimes for self-generated influences.

Critically, because the Situated Inference Model argues thatprimes influence behavior when associations activated by theprimes are misattributed to one’s own responses, we predict thatthe primes will influence behavior when the player’s hand isneither very strong nor very weak. Only when the hand is moderatedoes the potential exist to confuse primed thoughts for one’s ownthoughts. In such cases, participants might think about betting, forexample, either because they generated the thought to bet, orbecause the prime activated a thought to bet. If the prime activatedthe thought to bet when the player’s hand is very weak, it wouldbe more obvious that the thought must have originated with theprime. We therefore hypothesized that primes would affect behav-ior when the player’s hand was moderately strong but not when thehand was very strong or very weak. We examined this possibilityin a series of six replications, using both laboratory and Internetsamples.

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

1271REPLICABLE PRIMING

Method for Experiments 1, 2, and 3

The first three experiments are close replications and so aredescribed together. In each experiment, participants played a sim-ple card game (inspired by Black Jack) against the computer. Oneach trial, the participant and the computer were each dealt twoplaying cards (see Figure 1). Participants were asked to look attheir two cards and, based on their additive value, choose whetherto bet or pass. Hand value was determined by the numeric value ofthe cards, with face cards worth 10 and Aces worth 11. If theparticipants’ hand total was higher than the computer’s (not shownto the participant), they won 5 points. If the participants’ hand totalwas less than the computer’s, they lost 5 points. No points wereawarded or lost if the participant passed. Participants were in-structed to try to win as many points as possible.

Before participants were able to indicate their choice, a primeword was flashed on the screen. Participants were told that theword was a “distractor” that was not informative, and that theyshould “try not to let this word influence your decisions.” Previousresearch has found that when primes and targets are presented inquick succession, participants have difficulty disentangling theirsemantic (Gawronski, Deutsch, & Seidel, 2005; Imhoff, Schmidt,Bernhardt, Dierksmeier, & Banse, 2011; Sava et al., 2012) andaffective (Payne et al., 2005; Gawronski & Ye, 2014) responses tothe two items even when the primes are plainly visible.

The prime word was randomly chosen from a list of wordsrelated to betting or passing (specific stimuli used are described inthe procedure sections of each experiment). On each trial, theparticipant’s hand was shown for 600 ms, followed by a 150-msblank screen. Next, a randomly selected prime was presented for300 ms, followed by a 100-ms blank screen. Participants were thenasked to indicate whether they wanted to bet or pass by clickingone of two options with a computer mouse. This screen wasdisplayed until a decision was made. Finally, the participants werepresented with feedback: whether they won 5 points, lost 5 points,or did not lose or gain any points. Hand totals ranged from 4 to 21.The hand value and prime type were randomly determined on eachtrial. At the end of all trials, participants’ total point amount wasdisplayed.

Because we did not expect any variability for extremely highand low hands, we decided a priori to designate the lowest (Hands4–7) and the highest (Hands 18–21) as filler trials in which theoutcome was always win (for the high hands) or lose (for the lowhands). We considered the intermediate hands (Hands 8–17) to bethe critical trials. On critical trials the outcome was randomlydetermined such that if participants bet, then they would be equallylikely to win or lose points. This allowed the reward experiencedby participants to be independent of the primes and hands, pre-venting contingency learning from influencing results. The designof each study was 2 (prime: bet or pass) � 10 (hand value) withboth factors within-subjects.

Note that the individual data points within each of these exper-iments includes only three observations per subject, so effects atspecific hand levels should be interpreted with caution. Thus, theoverall pattern of responses is more informative than specificwithin-hand comparisons, which vary across studies due to thesmall number of observations. Following the presentation of thesix individual studies, we present the aggregated results which areless vulnerable to such variance.

Procedure

Experiment 1. Participants were 153 college students (42male, 111 female; Mage � 18.69 years, SDage � .89). The gameincluded 80 trials (60 critical and 20 noncritical), with hand valueand prime type randomly determined on a trial-by-trial basis. Thebehavioral primes were randomly selected from a set of six words,three related to betting (bet, gamble, or wager) and three related topassing (pass, fold, or stay). After completing the priming task,participants were asked how often they played cards, gambledwhile playing cards, and played Black Jack. Then participantscompleted a series of demographic questions (age, gender, andrace/ethnicity). These variables did not moderate our results andwill not be discussed further. No participants were excluded fromanalyses.

Experiment 2. Participants were 219 Amazon MechanicalTurk users (118 male, 94 female, 6 nonresponses; Mage � 34.45years, SDage � 12.07). One participant pressed only a singleresponse key throughout the entire priming task and was excludedfrom the analyses. In this version of the game we dropped theprimes “bet” and “pass” because these words were used to labelthe response options. We wanted to ensure that the effect repli-cated without priming the response labels themselves. The gameincluded 90 total trials (80 critical and 10 noncritical). All otheraspects of the procedure were identical to Experiment 1.

Experiment 3. Participants were 115 Amazon MechanicalTurk users (70 male, 45 female; Mage � 33.94 years, SDage �11.55). Two participants pressed one response throughout theentire priming task and were excluded from the analyses. Theprocedure was identical to that used in Experiment 2.

Results

We hypothesized that primes would influence betting behaviorwhen the hand value was moderate but not when it was high orlow. To test this hypothesis, we analyzed the proportion of “bet”decisions for each hand and prime condition using a repeated-measures analysis of variance (ANOVA). Figure 2 displays theFigure 1. Schematic depiction of one trial in the betting game.

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

1272 PAYNE, BROWN-IANNUZZI, AND LOERSCH

proportion of “bet” decisions as a function of hand value and primetype. In Experiment 1, there was a main effect of prime such thatsubjects were significantly more likely to bet when they wereprimed with bet-related words (M � .48, SE � .02) than when theywere primed with pass-related words (M � .46, SE � .01), F(1,152) � 3.96, p � .048, �p

2 � .03. There was also a main effect ofhand value such that as hand value increased participants weremore likely to bet, F(9, 1368) � 357.31, p � .001, �p

2 � .70.Importantly, these main effects were qualified by the predictedinteraction between prime and hand value, F(9, 1368) � 4.60, p �.001, �p

2 � .03. Consistent with our hypothesis, the pattern sug-gests that participants were more likely to respond in prime-consistent ways when hand values were ambiguous.

We obtained the same general pattern of results for Experiment2 (see Figure 3). Again, there was a main effect of prime, such thatwhen people were primed with bet-related words they were sig-nificantly more likely to bet (M � .49, SE � .015) than when theywere primed with pass related words (M � .45, SE � .013), F(1,217) � 17.67, p � .001, �p

2 � .08. There was also a main effect ofhand value, such that participants were more likely to bet as their

hand value increased, F(9, 1953) � 247.33, p � .001, �p2 � .53. As

before, these main effects were qualified by the predicted interac-tion between prime and hand value, F(9, 1953) � 2.74, p � .003,�p

2 � .01.These results were replicated in Experiment 3. There was a main

effect of prime, such that subjects were significantly more likely tobet when primed with bet-related words (M � .53, SE � .018) thanwhen primed with pass-related words (M � .51, SE � .018), F(1,112) � 6.11, p � .02, �p

2 � .05. There was also a main effect ofhand value, such that as hand value increased participants weremore likely to bet, F(9, 1008) � 140.39, p � .001, �p

2 � .56.Again, these main effects were qualified by the predicted interac-tion between prime and hand value, F(9, 1008) � 2.40, p � .01,�p

2 � .02 (see Figure 4).

Experiment 4: Replication With MaskedPrime Stimuli

Experiments 1 through 3 instructed participants to ignore theprimes and found effects of the primes despite those instruc-tions, thereby suggesting that the effect was unintended. Thisapproach is similar to the logic of the Stroop color naming task,in which participants are instructed to read the font color ofwords and to ignore their meaning. Nonetheless, performancetypically shows that word meaning causes interference, sug-gesting involuntary reading. Another approach to demonstrat-ing that participants are not deliberately using primes to informbehavior is to present the primes briefly and with maskingstimuli to limit subjects’ conscious perception. If subjects can-not perceive the prime stimulus, then they cannot strategicallyuse it to respond. Experiment 4 used the same procedure as inprior experiments, but briefly presented the primes with for-ward and backward masks stimuli to degrade participants’conscious perception of the stimuli.

Method

Participants were 116 undergraduate students (53 male, 63 fe-male; Mage � 18.89, SDage � 1.08) who participated for partial

Figure 4. The proportion of “bet” decisions as a function of prime andhand value in Experiment 3. Asterisks indicate where the contrast betweenbet and pass primes is significant, p � .05.

Figure 2. The proportion of “bet” decisions as a function of prime andhand value in Experiment 1. Asterisks indicate where the contrast betweenbet and pass primes is significant, p � .05.

Figure 3. The proportion of “bet” decisions as a function of prime andhand value in Experiment 2. Asterisks indicate where the contrast betweenbet and pass primes is significant, p � .05.

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

1273REPLICABLE PRIMING

course credit. The data were collected using laboratory PC’s sep-arated into cubicles for privacy. On each trial, the hand of cardswas shown for 600 ms, followed by a blank screen for 150 ms.Next, a premask word displayed for 100 ms, followed by a primedisplayed for 12 ms, followed by a postmask word displayed for100 ms. Finally, a blank screen was presented for 288 ms, and thenparticipants were then asked to indicate whether they wanted to betor pass by clicking one of two options with a computer mouse. Theprimes were gamble, wager, fold, and stay. Premask and postmaskstimuli were six-letter words each drawn randomly from a list of90 neutral words unrelated to gambling.

Because masked priming generally has smaller effects thanvisibly presented priming (Kouider & Dehaene, 2007), we wantedto increase the statistical power in this study. On the basis of theresults of the previous studies, we limited the critical trials to hands10–15, and designated the lowest (Hands 4 through 9) and thehighest (Hands 16 through 21) as filler trials in which the outcomewas always win (for the high hands) or lose (for the low hands). Oncritical trials the outcome was randomly determined so that thereward experienced by participants was independent of the primesand hands. We increased the number of critical trials, resulting in96 critical trials and 24 noncritical trials. The design of the studywas 2 (prime: bet or pass) � 6 (hand value) with both factorswithin-subjects.

Subjects completed an awareness test following the gamblingtask. Subjects were told that three words would be presentedsequentially on each trial and that they should write them down.Each trial of the awareness check was identical to the gamblingtask, except that instead of choosing whether to bet, subjects werepresented with three dialog boxes labeled first word, second word,and third word in which to type the words. They were instructedto type 99 if they did not see a word. Participants completed 16trials, each of the four prime words was presented twice, and fourneutral words (gander, wagon, fond, and stag) were each presentedtwice. These stimuli were matched in length and the first twoletters to a prime word so that the words could not be guessed byseeing only the first letters. We scored the awareness check bycounting the number of correctly identified prime words, irrespec-tive of the order. Although the prime item was always the secondword, we scored responses as correct if the presented prime wascorrectly identified even if it was listed as the first or third word.

Results

Awareness check. One participant correctly identified a sin-gle prime word. This subject was removed from the analyses tofollow, although including this participants’ data does not changeany results. This low rate of detection suggests that the maskingprocedure was effective at disrupting conscious perception of theprimes.

Primary results. Data were analyzed using the same repeatedmeasures ANOVA as in previous studies. Figure 5 displays theproportion of bet decisions as a function of hand value and primetype. There was no main effect of Prime, F(1, 114) � .08, p � .78,�p

2 � .001. There was a main effect of Hand value, such that ashand value increased participants were more likely to bet, F(5,570) � 292.44, p � .001, �p

2 � .72. Replicating Experiments 1through 3, the predicted interaction between prime and hand valuewas significant, F(5, 570) � 5.36, p � .001, �p

2 � .05.

The pattern was the same as that observed in the previousexperiments with visibly presented primes, although the maineffect of the prime was smaller. If participants could not report theidentity of the prime, it is unlikely that they strategically used it toinform their decisions. Althoughs this evidence increases our con-fidence that participants did not consciously use the primes, it is,of course, difficult to conclusively prove that this is the case. Toprovide converging evidence that our results are indeed driven byunintended influences of the primes in two final studies, we useda process dissociation approach to separate automatic and inten-tionally controlled use of the primes.

Method for Experiments 5 and 6: SeparatingAutomatic and Controlled Influences

The first four experiments provided consistent evidence forreplicable effects of primes on behavior. Although the data suggestthat this effect was unintended, convergent results from a differentmethod would increase confidence in that inference. To gainadditional evidence about participants’ ability to use primed infor-mation intentionally and unintentionally, we applied a processdissociation procedure in our final two replications (Jacoby, 1991).This procedure was originally developed to separate conscious andunconscious influences of memory (Jacoby, 1991; 1998). It hassince been used in many studies to separate intentionally con-trolled and automatic (i.e., unintentional) influences in a variety ofexperimental paradigms, including the Stroop task (Lindsay &Jacoby, 1994), priming tasks measuring implicit race bias (Payne,2001; Payne & Bishara, 2009; Payne, Lambert, & Jacoby, 2002),and decision-making tasks (Ferreira, Garcia-Marques, Sherman, &Sherman, 2006).

The parameters of the procedure were the same as in Experi-ments 1 through 3. We modified the game to employ a two blockdesign that allowed us to set intentional and unintentional influ-ences in opposition, and in concert. One block (the prime-exclusion condition), used the same instructions as in Experiments1 through 3. Participants were told that the word presented beforetheir behavioral decision was merely a distractor. The instructionsstated the word is “not informative, so try not to let this word

Figure 5. The proportion of “bet” decisions as a function of prime andhand value in Experiment 4. Asterisks indicate where the contrast betweenbet and pass primes is significant, p � .05.

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

1274 PAYNE, BROWN-IANNUZZI, AND LOERSCH

influence your decision.” In the other block (the prime-inclusioncondition), participants were instead told that this word was aninformative hint. The instructions stated the word is “a hint and isinformative on more trials than not, so you may want to considerthis word in your decision.”

Process dissociation procedure analyses. In our task, partic-ipants could be influenced by the prime either intentionally (e.g.,by using the prime as a hint) or automatically (e.g., makingprime-consistent responses despite trying to ignore the prime).Note that the model does not take the hand value into accountwhen estimating automatic and controlled influences. Both esti-mates refer to using the primes—either automatically or intention-ally. In the inclusion blocks, participants are encouraged to use theprime to guide responses. Therefore, prime-consistent responsesmight arise either through intentionally controlled use of the prime(C), or through automatic influences (A) even if controlled strat-egies fail (1-C). Equation 1 describes the expected probability ofprime-consistent responses as a function of A and C processes inthe inclusion blocks.

P (Prime-consistent response�Inclusion) � C � A* (1�C)

(1)

Alternatively, in the exclusion blocks, participants are instructedto ignore the primes. In this case, primes are expected to automat-ically influence responses (A) only when intentional control fails(1-C). Equation 2 describes the expected probability of prime-consistent responses as a function of A and C processes in theexclusion blocks.

P (Prime-consistent response�Exclusion) � A* (1�C) (2)

Given these equations and the probabilities of prime-consistentresponses in each block type from the data, we can derive A andC estimates using Equations 3 and 4.

C � P (Prime-consistent response�Inclusion)

� P (Prime-consistent response�Exclusion) (3)

A � P (Prime-consistent response�Exclusion) ⁄ (1�C) (4)

Block order was randomly determined for each participant. Eachblock consisted of 40 critical trials (i.e., hand totals ranging from8 to 17) and five filler trials. At the end of each block, participantslearned how many points they had earned during that game. Thepoint total was reset between the first and second block. Experi-ments 5 and 6 used identical 2 (prime: bet, pass) � 10 (handvalue) � 2 (block type: inclusion, exclusion) designs, with allfactors manipulated within-subjects.

Procedure. In Experiment 5, participants were 209 AmazonMechanical Turk users (121 male, 88 female; Mage � 33.33 years,SDage � 11.79). In Experiment 6, participants were 184 AmazonMechanical Turk users (72 male, 112 female; Mage � 36.86 years,SDage � 13.16). Five participants pressed one button throughout atleast one block of trails and were excluded from the analyses.

Results

Analysis of betting responses. In Experiment 5, the Analysisof Variance revealed a significant three-way interaction betweenPrime, Hand value, and Block type (inclusion vs. exclusion), F(9,1872) � 3.95, p � .001, partial �2 � .02. To probe this interaction,we examined the Prime � Hand value interaction within eachblock type. For the prime-exclusion block, there was a main effectof prime such that participants were significantly more likely to betwhen they were primed with bet (M � .532, SE � .014) than whenthey were primed with pass (M � .504, SE � .014), F(1, 208) �8.32, p � .004, �p

2 � .04. There was also a main effect of handvalue, such that as hand value increased, participants were morelikely to bet, F(9, 1872) � 210.97, p � .001, �p

2 � .50. Replicatingthe results of Experiments 1 through 4, these main effects werequalified by the predicted interaction between prime and handvalue, F(9, 1872) � 5.78, p � .001, �p

2 � .03. As seen in Figure6, primes affected responses primarily when hands were of amoderate value.

Figure 6. The proportion of “bet” decisions in the inclusion and exclusion block as a function of primeand hand value in Experiment 5. Asterisks indicate where the contrast between bet and pass primes issignificant, p � .05.

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

1275REPLICABLE PRIMING

For the prime-inclusion block, there was a large main effect ofprime such that when subjects were primed with bet related wordsthey were significantly more likely to bet (M � .65, SE � .016)than when they were primed with pass related words (M � .44,SE � .015), F(1, 208) � 123.63, p � .001, �p

2 � .37. There wasalso a main effect of hand value such that as hand value increasedparticipants were more likely to bet, F(9, 1872) � 149.81, p �.001, �p

2 � .42. These main effects were qualified by an unex-pected interaction between Prime and Hand value, F(9, 1872) �4.77, p � .001, �p

2 � .02, likely reflecting a ceiling effect in theinclusion block when the hand value was high and a betting relatedprime was presented. Critically, unlike the results in the exclusioncondition, participants were significantly more likely to bet whenthey were primed with bet related words than pass related words atevery hand value.

This pattern of results was replicated in Experiment 6. Again,there was a significant three-way interaction between Prime, Handvalue, and Block type (inclusion vs. exclusion), F(9, 1602) � 2.32,p � .014, �p

2 � .01. For the prime-exclusion block, there was amain effect of Prime such that when participants were primed withbet related words they were significantly more likely to bet (M �.49, SE � .015) than when they were primed with pass relatedwords (M � .45, SE � .014), F(1, 178) � 12.59, p � .001, �p

2 �.07. There was also a main effect of Hand value such that as handvalue increased participants were more likely to bet, F(9, 1602) �187.62, p � .001, �p

2 � .51. Again, these main effects werequalified by the predicted interaction between Prime and Handvalue, F(9, 1602) � 3.89, p � .001, �p

2 � .02. As shown in Figure7, prime-consistent responses were more common when handswere of a moderate value.

For the prime-inclusion block, there was a main effect of primesuch that when people were primed with bet related words theywere significantly more likely to bet (M � .596, SED � .018) thanwhen they were primed with pass related words (M � .427, SED �.016), F(1, 178) � 73.201, p � .001, �p

2 � .29. There was also amain effect of hand value such that as hand value increased

participants were more likely to bet, F(9, 1602) � 133.97, p �.001, �p

2 � .43. Again, these main effects were qualified by aninteraction between prime and hand value, F(9, 1602) � 2.79, p �.003, �p

2 � .02, likely due to a ceiling effect at the highest values(see Figure 8). As in Experiment 4, primes significantly affectedresponses at every hand value.

To compute A and C estimates, we aggregated the hand valuesinto two sets: the ambiguous hands (Hands 11 through 14) andunambiguous hands (Hands 8 through 10 and 15 through 17). Weestimated A and C parameters separately for each set using Equa-tions 1 through 4.

In Experiment 5, as predicted, A estimates were significantlyhigher on ambiguous hands, M � .61 (95% CI [.59, .64]) thanunambiguous hands, M � .58 (95% CI [.56, .60]), F(1, 208) �19.59, p � .001, �p

2 � .09. An estimate of no automatic effectswould be reflected by an A estimate of .50. As can be seen by theconfidence intervals, the A estimates in both conditions weresignificantly greater than .50. Thus, the primes had significantautomatic effects over all, and these effects were stronger onambiguous than unambiguous hands.

Estimates of controlled responding reflect how much more theprimes affected responses when subjects tried to use the primesthan when they tried not to use the primes. C estimates were notdifferent on ambiguous hands, M � .09 (95% CI [.06, .12]) versusunambiguous hands, M � .10 (95% CI [.08, .12]), F(1, 208) � .55,p � .46, �p

2 � .003. Estimates of controlled responding weresignificantly greater than the chance value of zero (no control), butcontrolled use of the primes did not depend on hand ambiguity.

The process dissociation results replicated in Experiment 6.Again, A estimates were greater than .50 in both conditions, butthe effects were larger in ambiguous hands, M � .59 (95% CI [.56,.61]) than in the unambiguous hands, M � .57 (95% CI [.55, .59]),F(1, 178) � 4.84, p � .029, �p

2 � .03. Control estimates weregreater than zero in both conditions, but no different in the am-biguous hands, M � .06 (95% CI [.04, .09]) versus unambiguous

Figure 7. The proportion of “bet” decisions in the inclusion and exclusion block as a function of hand valueand prime type in Experiment 6. Asterisks indicate where the contrast between bet and pass primes is significant,p � .05.

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

1276 PAYNE, BROWN-IANNUZZI, AND LOERSCH

hands, M � .07, (95% CI [.05, .09]), F(1, 178) � .019, p � .89,�p

2 � .000.The process dissociation results converge with the results of the

previously reported studies. Participants were partially able tostrategically use the primes when they believed they were helpfulhints, and resist using the primes when they were uninformative.However, when control failed, the primes exerted an automaticinfluence, especially when the hand was more ambiguous. To-gether with the results of Experiments 1 through 4, this providesfurther evidence that the primes can induce an unintended influ-ence on behavior.

Combined analysis. The overall pattern of priming effectswas consistent across the six replications reported above. How-ever, the significance levels at particular hand values varied some-what across studies. This is unsurprising because there are fewobservations at each hand value. By aggregating results across thestudies we can more precisely estimate the priming effect acrossthe range of hand values. In the aggregate analysis we includeddata from Hand Values 10 through 15 because those values werepresent across all experiments. For Experiments 5 and 6 we usedonly data from the exclusion blocks. In this aggregate analysis itwas not necessary to use meta-analytic summary statistics becauseall experiments used the same basic procedure and utilized iden-tical dependent variables. In sum, this analysis includes data from988 participants.

Across experiments, there was a significant main effect ofprime, F(1, 987) � 58.25, p � .001, �p

2 � .06, a significant maineffect of hand value, F(5, 4935) � 756.23, p � .001, �p

2 � .43, anda significant Prime � Hand value interaction, F(5, 4935) � 17.36,p � .001, �p

2 � .02. As can be seen in Figure 8, the data conformsto our predicted pattern of results, in which primes influencedecisions more when hand values are moderate than when they areextreme. As expected, this resulted in a significant quadratic effectfor the interaction term, F(1, 987) � 21.63, p � .001, �p

2 � .02.

General Discussion

In 2012, Daniel Kahneman sent an open letter to a number ofscientists, arguing that behavior priming was “the poster child for

doubts about the integrity of psychological research” (Kahneman,2012). Since that letter, the replication failures cited above haveled a number of individuals to conclude that reported primingeffects were created by “cooked” experiments that were “guaran-teed to find an effect” (Zwaan, 2013), and only exist because of“confirmation bias, the use of questionable research practices, andselective reporting” (Wagenmakers, 2014).

Here we provided evidence that primes can produce replicableeffects on human behavior. Our procedure captured the definingelements of previous priming studies, in that ostensibly irrelevantprimes influenced behavioral responses, and appeared to do sounintentionally. These studies used a within-subjects experimentaldesign with high statistical power, which was created based on apriori theoretical predictions. We believe that these results provideimportant evidence for debates about the reality and replicabilityof priming phenomena. Whatever the reasons for particular repli-cation failures, we find evidence that primes can reliably affectbehavior when high powered methods are used.

This comparison between within-subjects and between-subjectsdesigns raises an obvious question: In studies where failed nullreplications have been reported, is it the psychological effect ofprimes on behavior that is in question, or is it mundane method-ological factors like statistical power? Our results suggest thatmethodological factors play an important role. If so, we may reviseour beliefs about effect sizes and variability of previously reportedeffects, but not about the basic psychological effect in question.

Our results have implications for the fundamental theoreticalperspectives that have been informed by priming experiments.Most broadly, the unintentional effects of primes on behaviorsupport dual process theories that posit a role for automatic pro-cesses as well as consciously controlled processes in guidinghuman behavior (for reviews, see Gawronski & Payne, 2011;Sherman et al., 2014). Even when individuals know that a partic-ular stimulus is irrelevant for the task at hand, exposure to thatstimulus can set processes in motion that bias our behavior inunintended ways.

This contrast between within-subjects and between-subject’sdesigns has a historical parallel in research on implicit biases.Early influential studies used between-subjects designs to demon-strate that subtle exposure to racial stereotypes could influencesocial judgments (Devine, 1989; Lepore & Brown, 1997). Thesestudies provided “existence proofs” that stereotypes could haveunintentional effects on judgments, but testing moderator variablesin these designs required large factorial between-subject’s designs.Research on implicit bias accelerated dramatically when laterresearch developed within-subjects’ designs that could providerobust measures of stereotype-priming on an individual level (e.g.,Fazio et al., 1995; Greenwald, McGhee, & Schwartz, 1998). Asresult, theoretical models of implicit attitudes became much moredetailed and empirically better supported (e.g., Fazio, 1990;Gawronski & Bodenhausen, 2006; Petty & Briñol, 2006; see Petty,Fazio, & Briñol, 2012).

Many questions remain. Our experiments differed from pastpriming studies in some ways beyond the within-subjects design.For example, the time between prime and response was only a fewseconds in our studies. Past studies have often examined behav-ioral outcomes after several minutes, and the effects of primes maydiffer across time scales (Wentura & Rothermund, 2014). Onerelevant study presented dollar signs as primes in a gambling task,

Figure 8. The proportion of “bet” decisions as a function of prime andhand value aggregated across all studies (N � 988). Asterisks indicatewhere the contrast between bet and pass primes is significant, p � .05.

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

1277REPLICABLE PRIMING

and found that the primes increased betting when the decision wasmade immediately, but not after a 5-min delay (Gibson & Zielas-kowski, 2013). However, like most behavior priming studies, thatstudy used a between-subjects priming manipulation and a singleobservation as the dependent variable. Results may be differentwhen more powerful experimental designs are used. Future re-search should systematically vary potential moderating variables(including time) to better understand the conditions under whichprimes are more or less likely to affect behavior.

Because the paradigm introduced here appears to produces arobust effect and can be easily implemented in both laboratory andonline settings, it may provide a useful tool for researchers inter-ested in investigating the moderators and mechanisms of priming.Our materials are available online at http://bkpayne.web.unc.edu/,and we encourage others to replicate our findings in their ownlaboratories. It is our hope that the paradigm will serve as a toolthat helps us move beyond arguments about whether behaviorpriming is real, and return attention to important questions abouthow primes have their effects on human behavior.

References

Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of socialbehavior: Direct effects of trait construct and stereotype-activation onaction. Journal of Personality and Social Psychology, 71, 230–244.http://dx.doi.org/10.1037/0022-3514.71.2.230

Bargh, J. A., Gollwitzer, P. M., Lee-Chai, A., Barndollar, K., & Trötschel,R. (2001). The automated will: Nonconscious activation and pursuit ofbehavioral goals. Journal of Personality and Social Psychology, 81,1014–1027. http://dx.doi.org/10.1037/0022-3514.81.6.1014

Bartlett, T. (2012). Is psychology about to come undone? [Blog post].Retrieved from http://chronicle.com/blogs/percolator/is-psychology-about-to-come-undone/29045

Berger, J., & Fitzsimons, G. (2008). Dogs on the street, pumas on your feet:How cues in the environment influence product evaluation and choice.JMR, Journal of Marketing Research, 45, 1–14. http://dx.doi.org/10.1509/jmkr.45.1.1

Berger, J., Meredith, M., & Wheeler, S. C. (2008). Contextual priming:Where people vote affects how they vote. Proceedings of the NationalAcademy of Sciences of the United States of America, 105, 8846–8849.http://dx.doi.org/10.1073/pnas.0711988105

Cameron, C. D., Brown-Iannuzzi, J. L., & Payne, B. K. (2012). Se-quential priming measures of implicit social cognition: A meta-analysis of associations with behavior and explicit attitudes. Person-ality and Social Psychology Review, 16, 330 –350. http://dx.doi.org/10.1177/1088868312440047

Cesario, J., Plaks, J. E., Hagiwara, N., Navarrete, C. D., & Higgins, E. T.(2010). The ecology of automaticity. How situational contingenciesshape action semantics and social behavior. Psychological Science, 21,1311–1317. http://dx.doi.org/10.1177/0956797610378685

Chaiken, S., & Trope, Y. (Eds.). (1999). Dual-process theories in socialpsychology. New York, NY: Guilford Press.

Correll, J., Park, B., Judd, C. M., & Wittenbrink, B. (2002). The policeofficer’s dilemma: Using ethnicity to disambiguate potentially threaten-ing individuals. Journal of Personality and Social Psychology, 83,1314–1329. http://dx.doi.org/10.1037/0022-3514.83.6.1314

Devine, P. G. (1989). Stereotypes and prejudice: Their automatic andcontrolled components. Journal of Personality and Social Psychology,56, 5–18. http://dx.doi.org/10.1037/0022-3514.56.1.5

Dijksterhuis, A., & van Knippenberg, A. (1998). The relation betweenperception and behavior, or how to win a game of trivial pursuit. Journalof Personality and Social Psychology, 74, 865–877. http://dx.doi.org/10.1037/0022-3514.74.4.865

Doyen, S., Klein, O., Simons, D. J., & Cleeremans, A. (2014). On the otherside of the mirror: Priming in cognitive and social psychology. SocialCognition, 32(Suppl.), 12–32. http://dx.doi.org/10.1521/soco.2014.32.supp.12

Fazio, R. H. (1990). In M. P. Zanna (Ed.), Multiple processes by whichattitudes guide behavior: The MODE model as an integrative framework(pp. 75–109). New York, NY: Academic Press.

Fazio, R. H., Jackson, J. R., Dunton, B. C., & Williams, C. J. (1995).Variability in automatic activation as an unobtrusive measure of racialattitudes: A bona fide pipeline? Journal of Personality and SocialPsychology, 69, 1013–1027. http://dx.doi.org/10.1037/0022-3514.69.6.1013

Fazio, R. H., Sanbonmatsu, D. M., Powell, M. C., & Kardes, F. R. (1986).On the automatic activation of attitudes. Journal of Personality andSocial Psychology, 50, 229–238. http://dx.doi.org/10.1037/0022-3514.50.2.229

Ferguson, M. J., & Mann, T. C. (2014). Effects of evaluation: An exampleof robust “social” priming. Social Cognition, 32(Suppl.), 33–46. http://dx.doi.org/10.1521/soco.2014.32.supp.33

Ferreira, M. B., Garcia-Marques, L., Sherman, S. J., & Sherman, J. W.(2006). Automatic and controlled components of judgment and decisionmaking. Journal of Personality and Social Psychology, 91, 797–813.http://dx.doi.org/10.1037/0022-3514.91.5.797

Gawronski, B., & Bodenhausen, G. V. (2006). Associative and proposi-tional processes in evaluation: An integrative review of implicit andexplicit attitude change. Psychological Bulletin, 132, 692–731. http://dx.doi.org/10.1037/0033-2909.132.5.692

Gawronski, B., Deutsch, R., & Seidel, O. (2005). Contextual influences onimplicit evaluation: A test of additive versus contrastive effects ofevaluative context stimuli in affective priming. Personality and SocialPsychology Bulletin, 31, 1226 –1236. http://dx.doi.org/10.1177/0146167205274689

Gawronski, B., & Payne, B. K. (Eds.). (2011). Handbook of implicit socialcognition: Measurement, theory, and applications. New York, NY:Guilford Press.

Gawronski, B., & Ye, Y. (2014). What drives priming effects in the affectmisattribution procedure? Personality and Social Psychology Bulletin,40, 3–15. http://dx.doi.org/10.1177/0146167213502548

Gibson, B., & Zielaskowski, K. (2013). Subliminal priming of winningimages prompts increased betting in slot machine play. Journal ofApplied Social Psychology, 43, 106–115. http://dx.doi.org/10.1111/j.1559-1816.2012.00985.x

Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. (1998). Measuringindividual differences in implicit cognition: The implicit association test.Journal of Personality and Social Psychology, 74, 1464–1480. http://dx.doi.org/10.1037/0022-3514.74.6.1464

Harris, C. R., Coburn, N., Rohrer, D., & Pashler, H. (2013). Two failuresto replicate high-performance-goal priming effects. PLoS ONE, 8(8),e72467. http://dx.doi.org/10.1371/journal.pone.0072467

Imhoff, R., Schmidt, A. F., Bernhardt, J., Dierksmeier, A., & Banse, R.(2011). An inkblot for sexual preference: A semantic variant of the affectmisattribution procedure. Cognition and Emotion, 25, 676–690. http://dx.doi.org/10.1080/02699931.2010.508260

Jacoby, L. L. (1991). A process dissociation framework: Separating auto-matic from intentional uses of memory. Journal of Memory and Lan-guage, 30, 513–541. http://dx.doi.org/10.1016/0749-596X(91)90025-F

Jacoby, L. L. (1998). Invariance in automatic influences of memory:toward a user’s guide for the process-dissociation procedure. Journal ofExperimental Psychology: Learning, Memory, and Cognition, 24, 3–26.

Jacoby, L. L., & Dallas, M. (1981). On the relationship between autobio-graphical memory and perceptual learning. Journal of ExperimentalPsychology: General, 110, 306–340. http://dx.doi.org/10.1037/0096-3445.110.3.306

Kahneman, D. (2011). Thinking, fast and slow. New York, NY: Macmillan.

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

1278 PAYNE, BROWN-IANNUZZI, AND LOERSCH

Kahneman, D. (2012). Kahneman’s open letter from 26 September. Re-trieved from http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf

Kouider, S., & Dehaene, S. (2007). Levels of processing during non-conscious perception: A critical review of visual masking. PhilosophicalTransactions of the Royal Society of London Series B, Biological Sci-ences, 362, 857–875. http://dx.doi.org/10.1098/rstb.2007.2093

Krieglmeyer, R., & Sherman, J. W. (2012). Disentangling stereotypeactivation and stereotype application in the stereotype misperceptiontask. Journal of Personality and Social Psychology, 103, 205–224.http://dx.doi.org/10.1037/a0028764

Lepore, L., & Brown, R. (1997). Category and stereotype activation: Isprejudice inevitable? Journal of Personality and Social Psychology, 72,275–287. http://dx.doi.org/10.1037/0022-3514.72.2.275

Lindsay, D. S., & Jacoby, L. L. (1994). Stroop process dissociations: Therelationship between facilitation and interference. Journal of Experimen-tal Psychology: Human Perception and Performance, 20, 219–234.http://dx.doi.org/10.1037/0096-1523.20.2.219

Loersch, C., & Payne, B. K. (2011). The situated inference model: Anintegrative account of the effects of primes on perception, behavior, andmotivation. Perspectives on Psychological Science, 6, 234–252. http://dx.doi.org/10.1177/1745691611406921

Loersch, C., & Payne, B. K. (2014). Situated inferences and the what, who,and where of priming. Social Cognition, 32(Suppl.), 137–151. http://dx.doi.org/10.1521/soco.2014.32.supp.137

Macrae, C. N., & Johnston, L. (1998). Help, I need somebody: Automaticaction and inaction. Social Cognition, 16, 400–417. http://dx.doi.org/10.1521/soco.1998.16.4.400

Meyer, D. E., & Schvaneveldt, R. W. (1971). Facilitation in recognizingpairs of words: Evidence of a dependence between retrieval operations.Journal of Experimental Psychology, 90, 227–234. http://dx.doi.org/10.1037/h0031564

Molden, D. C. (Ed.). (2014a). Understanding priming effects in socialpsychology. New York, NY: Guilford Press.

Molden, D. C. (2014b). Understanding priming effects in social psychol-ogy: What is “social priming” and how does it occur? Social Cognition,32(Suppl.), 1–11. http://dx.doi.org/10.1521/soco.2014.32.supp.1

Neely, J. H. (1977). Semantic priming and retrieval from lexical memory:Roles of inhibitionless spreading activation and limited-capacity atten-tion. Journal of Experimental Psychology: General, 106, 226–254.http://dx.doi.org/10.1037/0096-3445.106.3.226

Neuroskeptic. (2015). Social priming: Money for nothing? [Blog post].Retrieved from http://blogs.discovermagazine.com/neuroskeptic/2015/07/23/social-priming-money-for-nothing/#.Vwg4xvkrJ9A

Pashler, H., Coburn, N., & Harris, C. R. (2012). Priming of social distance?Failure to replicate effects on social and food judgments. PLoS ONE, 7(8), e42510. http://dx.doi.org/10.1371/journal.pone.0042510

Pashler, H., Rohrer, D., & Harris, C. R. (2013). Can the goal of honesty beprimed? Journal of Experimental Social Psychology, 49, 959–964.http://dx.doi.org/10.1016/j.jesp.2013.05.011

Payne, B. K. (2001). Prejudice and perception: The role of automatic andcontrolled processes in misperceiving a weapon. Journal of Personalityand Social Psychology, 81, 181–192. http://dx.doi.org/10.1037/0022-3514.81.2.181

Payne, B. K., & Bishara, A. J. (2009). An integrative review of processdissociation and related models in social cognition. European Reviewof Social Psychology, 20, 272–314. http://dx.doi.org/10.1080/10463280903162177

Payne, B. K., Cheng, C. M., Govorun, O., & Stewart, B. D. (2005). Aninkblot for attitudes: Affect misattribution as implicit measurement.

Journal of Personality and Social Psychology, 89, 277–293. http://dx.doi.org/10.1037/0022-3514.89.3.277

Payne, B. K., Lambert, A. J., & Jacoby, L. L. (2002). Best laid plans:Effects of goals on accessibility bias and cognitive control in race-basedmisperceptions of weapons. Journal of Experimental Social Psychology,38, 384–396. http://dx.doi.org/10.1016/S0022-1031(02)00006-9

Petty, R. E., & Briñol, P. (2006). Understanding social judgment: Multiplesystems and processes. Psychological Inquiry, 17, 217–223.

Petty, R. E., Fazio, R. H., & Briñol, P. (Eds.). (2012). Attitudes: Insightsfrom the new implicit measures. New York, NY: Psychology Press.

Sava, F. A., MaricuToiu, L. P., Rusu, S., Macsinga, I., Vîrga, D., Cheng,C. M., & Payne, B. K. (2012). An inkblot for the implicit assessment ofpersonality: The semantic misattribution procedure. European Journalof Personality, 26, 613–628. http://dx.doi.org/10.1002/per.1861

Schröder, T., & Thagard, P. (2013). The affective meanings of automaticsocial behaviors: Three mechanisms that explain priming. PsychologicalReview, 120, 255–280. http://dx.doi.org/10.1037/a0030972

Schvaneveldt, R. W., & McDonald, J. E. (1981). Semantic context and theencoding of words: Evidence for two modes of stimulus analysis. Jour-nal of Experimental Psychology: Human Perception and Performance,7, 673–687. http://dx.doi.org/10.1037/0096-1523.7.3.673

Shanks, D. R., Newell, B. R., Lee, E. H., Balakrishnan, D., Ekelund, L.,Cenac, Z., . . . Moore, C. (2013). Priming intelligent behavior: Anelusive phenomenon. PLoS ONE, 8(4), e56515. http://dx.doi.org/10.1371/journal.pone.0056515

Shariff, A. F., & Norenzayan, A. (2007). God is watching you: PrimingGod concepts increases prosocial behavior in an anonymous economicgame. Psychological Science, 18, 803–809. http://dx.doi.org/10.1111/j.1467-9280.2007.01983.x

Sherman, J. W., Gawronski, B., & Trope, Y. (Eds.). (2014). Dual-processtheories of the social mind. New York, NY: Guilford Press.

Stanley, D. J., & Spence, J. R. (2014). Expectations for replications: Areyours realistic? Perspectives on Psychological Science, 9, 305–318.http://dx.doi.org/10.1177/1745691614528518

Tulving, E., & Schacter, D. L. (1990). Priming and human memorysystems. Science, 247, 301–306. http://dx.doi.org/10.1126/science.2296719

Wagenmakers, E. J. (2014). Behavioral priming: Time to nut up or shut up[Blog post]. Retrieved from http://osc.centerforopenscience.org/2014/03/26/behavioral-priming/

Weingarten, E., Chen, Q., McAdams, M., Yi, J., Hepler, J., & Albarracín,D. (2016). From primed concepts to action: A meta-analysis of thebehavioral effects of incidentally presented words. Psychological Bul-letin, 142, 472–497. http://dx.doi.org/10.1037/bul0000030

Wentura, D., & Rothermund, K. (2014). Priming is not priming is notpriming. Social Cognition, 32(Suppl.), 47–67. http://dx.doi.org/10.1521/soco.2014.32.supp.47

Wheeler, S. C., Demarree, K. G., & Petty, R. E. (2007). Understanding therole of the self in prime-to-behavior effects: The active-self account.Personality and Social Psychology Review, 11, 234–261. http://dx.doi.org/10.1177/1088868307302223

Yong, E. (2012, October 3). Nobel laureate challenges psychologists toclean up their act. Nature. Advance online publication. http://dx.doi.org/10.1038/nature.2012.11535

Zwaan, R. (2013). How to cook up your own social priming article [Blogpost]. Retrieved from http://rolfzwaan.blogspot.nl/2013/09/how-to-cook-up-your-own-social-priming.html

Received August 12, 2015Revision received April 22, 2016

Accepted May 24, 2016 �

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

1279REPLICABLE PRIMING

Correction to Payne et al. 2016

In the article “Replicable Effects of Primes on Human Behavior” by B. Keith Payne, Jazmin L.Brown-Iannuzzi, and Chris Loersch (Journal of Experimental Psychology: General, Vol. 145. No.10, pp. 1269–1279. http://dx.doi.org/10.1037/xge0000201), the graph in Figure 5 did not containthe asterisk mentioned in the figure caption, which was intended to indicate a statistically significantdifference between bet and pass prime. The online version of this article has been corrected.

http://dx.doi.org/10.1037/xge0000253

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.