magnitude and accuracy differences between judgements of ...webpages.acs.ttu.edu/mserra/serra...

Magnitude and accuracy differences between judgementsof remembering and forgetting

Michael J. Serra and Benjamin D. England

Department of Psychology, Texas Tech University, Lubbock, TX, USA

Metacognition researchers have recently begun to examine the effects of framing judgements of learning(JOLs) in terms of forgetting (rather than remembering) on the judgements’ magnitude and accuracy.Although a promising new direction for the study of metamemory, initial studies have yielded incon-sistent results. To help resolve these inconsistencies, in four experiments we had college students (N=434) study paired associates and make JOLs framed in terms of either remembering or forgetting overtwo study–test trials. Our goals were to further document the effects of framing on the magnitude andaccuracy of JOLs and to consider explanations for why specific patterns tend to emerge. The presentexperiments provide evidence that (a) judgements of forgetting are psychologically anchored at the mid-point of the JOL scale, whereas judgements of remembering are anchored at a lower point, (b) differ-ences in absolute accuracy (calibration) by frame are largely artefactual and stem from differences inanchoring, (c) differences in JOL magnitude and absolute accuracy by frame do not obtain whenmemory cues are salient to participants, and (d) a forget frame impairs the relative accuracy (resolution)of JOLs across trials by reducing participants’ reliance on cues such as memory for past test performance.

Keywords: Metacognition; Metamemory; Judgements of learning; Accuracy; Framing.

Metacognitive judgements such as judgements oflearning (JOLs) reflect people’s monitoring andevaluations of their own cognitive states (for arecent review, see Dunlosky & Metcalfe, 2009).Importantly, people often consult these judgementswhen controlling or adjusting their ongoing cogni-tive processes (e.g., Metcalfe & Finn, 2008;Thiede, Anderson, & Therriault, 2003). As such,not only can metacognitive experiences inform jud-gement and decision making (Schwarz, 2004), butmetacognitive monitoring and control is also itself

a form of judgement and decision making. It istherefore somewhat surprising that so little researchhas examined the effects of framing—a major topicin the field of judgement and decision making—on metacognitive monitoring and control. Towardsthis end, the purpose of the present experimentswas to further examine the effects of framing JOLsin terms of either remembering or forgetting onthe magnitude and accuracy of these judgements.

In most laboratory studies of JOLs, participantsprovide judgements of their memory for individual

Correspondence should be addressed to Michael J. Serra, Department of Psychology, MS 2051, Psychology Building, Texas Tech

University, Lubbock, TX 79409-2051, USA. E-mail: [email protected]

We presented some of these studies at the 55th Annual Meeting of the Southwestern Psychological Association and the 50th

Annual Meeting of the Psychonomic Society. We thank John Dunlosky, Sarah K. Tauber, and several anonymous reviewers for

their helpful suggestions and feedback on this research.

# 2012 The Experimental Psychology Society 2231http://www.psypress.com/qjep http://dx.doi.org/10.1080/17470218.2012.685081

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY

2012, 65 (11), 2231–2257

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

memory items in terms of the likelihood of remem-bering (e.g., “How likely do you feel you will be toremember this item on the test?”). Although theaccuracy of such judgements (i.e., how well the jud-gements predict future memory performance) isoften above chance, a wealth of metacognitionresearch has demonstrated that such judgementstypically demonstrate only modest accuracy inmost situations (Dunlosky & Metcalfe, 2009) and—perhaps worse—are prone to errors and illusions(Serra & Metcalfe, 2009). Recently, researchershave begun to examine whether changing theframing of JOLs from being about the likelihoodof remembering to being about the likelihood offorgetting improves the accuracy of these judge-ments (e.g., Finn, 2008; Koriat, Bjork, Sheffer, &Bar, 2004; Kornell & Bjork, 2009; Rhodes &Castel, 2008; Tauber & Rhodes, in press-a; seealso Halamish, McGillivray, & Castel, 2011, whoonly examined judgements of forgetting). Inaddition to increasing our understanding of howpeople make metacognitive judgements ingeneral, there is also an applied promise of thisresearch. Affecting the manner in which studentsmake metacognitive self-evaluations of theirongoing learning by simply changing the phrasingof the judgement prompt might be a simple andeffective way to improve the accuracy of students’judgements and the efficacy of their study choices(Finn, 2008; see also Koriat et al., 2004). As wedescribe next, however, research into this questionhas thus far yielded inconsistent results, so it isnot yet possible to recommend this method as away to improve monitoring.

Past research on the framing of JOLs

Within the domain of judgement and decisionmaking, framing often seems to change howpeople make judgements and—more importantly—decisions. For example, prospect theory(Kahneman & Tversky, 1979; for a review of find-ings see Kahneman & Tversky, 1984) suggests thatpeople view potential gains and losses differently,with losses having larger psychological effectsthan gains of the same magnitude (e.g., losing 20dollars affects someone psychologically more than

does gaining 20 dollars). Because of this difference,people typically work to avoid losing resources (i.e.,loss aversion). Furthermore, people will favour lessrisky bets when framed in terms of gain but willfavour riskier bets when framed in terms of loss.For example, when given a choice to either forcertain save one third of a population from adisease or the riskier option that could either savethe entire population or none of it, people willopt for the certain option (for certain save onethird versus possibly saving everyone) whenframed in terms of saving lives, but will opt forthe riskier option (possibly losing no one versusfor certain losing two-thirds) when framed interms of loss of lives (see Kahneman & Tversky,1984). Although the choices are exactly the samemathematically regardless of how they are framed,the different frames focus people on two differentaspects of the scenario (saving lives versus the lossof life), and this leads them to make differentdecisions under the two frames.

Although framing is a well-established topic ofstudy within the decision-making literature, exam-ining the effects of framing on metacognitive jud-gements such as JOLs is a fairly new endeavour.As we describe in the following sections, somestudies to date have found that different frames (i.e., remembering versus forgetting) affect JOLsand restudy decisions, whereas other studies havefound no differences by frame. To further compli-cate matters, within the set of studies demonstrat-ing differences by frame, the effects have not beenconsistent across studies.

Within the metamemory literature, Koriat et al.(2004) first demonstrated that framing memorypredictions in terms of forgetting instead ofremembering (e.g., “How likely do you feel youwill be to forget this item?”) could affect the judge-ments. Participants in their experiments were askedto imagine that other students had studied 60paired associates and to indicate how many itemsthey thought that person would either rememberor forget over a given time span (e.g., a week; ayear). Surprisingly, participants did not factorretention intervals into their remember-framedpredictions (Figure 1, Panel A) when retentionintervals were varied between subjects (e.g., they

2232 THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2012, 65 (11)

SERRA AND ENGLAND

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

judged that items would be equally likely to beremembered 10 minutes later as a year later).When the same retention intervals were askedabout in terms of the likelihood of forgetting(again between subjects), however, participants pre-dicted that items would be less likely to be forgotten10 minutes later than one week later, and less likelyto be forgotten one week later than one year later(Figure 1, Panel A). The authors concluded thatframing the prediction in terms of forgettingcaused participants to consider different—and pre-sumably more predictive—memory cues whenmaking their JOLs than when they were framedin terms of remembering. They dubbed this ideathe forgetting-notion hypothesis.

Although Koriat et al.’s (2004) studies did notdirectly demonstrate that the frame of the judge-ment changed people’s cue utilization whilemaking their predictions, this idea is supportedindirectly by other work that demonstrates thatasking participants to better consider the bases fortheir judgements can improve the accuracy ofthose judgements. For example, Koriat,Lichtenstein, and Fischhoff (1980) asked partici-pants to consider reasons why their answers togeneral-information questions might be correct orincorrect. When participants considered why theymight be wrong, the overconfidence of their judge-ments was reduced, and the judgements thereforebecame more accurate. According to the

forgetting-notion hypothesis, the forget framemight work in a similar way: The forget frame pre-sumably causes participants to think about whythey might forget an item rather than why theymight remember it, and the former is presumablyless prone to biases (e.g., confirmation bias) inmonitoring than is the latter. This idea is alsoindirectly supported by the main tenets of prospecttheory (Kahneman & Tversky, 1979) in that theremember (gain) and forget (loss) frames presum-ably change what information about memorypeople consider when making metamemory judge-ments and predictions.

Finn (2008) recently expanded upon Koriatet al.’s (2004) findings and examined the effect offraming on item-by-item JOLs using a more stan-dard paired-associates learning method. In Finn’sexperiments, judgements of forgetting (afterreverse scoring) demonstrated better absolute accu-racy (the difference of the mean JOLs and meanrecall; sometimes referred to as calibration) thandid judgements of remembering on a singlestudy–test trial (Figure 2, Panel A; but see Finn,2008, Experiment 3, in which a difference in thismeasure did not obtain even though judgementmagnitude differed by frame). In Finn’sExperiments 1A and 2, immediate judgements offorgetting were typically less overconfident (i.e.,more accurate) than were immediate judgementsof remembering. This apparently occurred

Figure 1. Panel A: Participants’ mean predicted recall performance (% of 60 items retained) for immediate, 10-minute, 1-week, and 1-year

retention intervals in Koriat et al. (2004), Experiments 4B (10-minute remember-framed predictions), 4C (all other remember-framed

predictions), and 7 (forget-framed predictions). Panel B: Just the predictions for a 10-minute retention interval. Forget-framed predictions

were reverse scored.

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2012, 65 (11) 2233

JUDGEMENTS OF FORGETTING

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

because, in all experiments, judgements of forget-ting (after reverse scoring) were significantly lowerthan were judgements of remembering, and actualmemory performance never differed by frame.Furthermore, in Finn’s Experiments 2 and 3, par-ticipants who made judgements of forgettingchose to restudy more items than did those whomade judgements of remembering. This outcomeis most in line with the ideas of prospect theory,as it demonstrates that the frame of the JOLs notonly changed people’s judgements about theirmemories, but also changed the decisions theymade about their memories. Presumably, the“loss” aspect of the forget frame made peoplemore likely to restudy items than did the rememberframe (gain) because people are naturally averse to

losses such as forgetting or, more generally, nega-tive outcomes such as doing poorly on a test(Kahneman & Tversky, 1979).

Although Finn (2008) did not directly demon-strate that the frame manipulation affected cue util-ization, the data nevertheless led to the conclusionthat the forget frame improves the absolute accu-racy of immediate JOLs by focusing participantson memory cues that better predict future learningthan do the cues participants attend to whenmaking judgements of remembering (cf. Koriatet al., 2004; see also Koriat, 1997, 2008). Takentogether, these results suggest that framing JOLsin terms of forgetting rather than rememberingmight be a simple way of improving the absoluteaccuracy of JOLs and increasing the efficacy of

Figure 2. Participants’ mean recall performance and mean JOLs (judgements of learning) from previously published studies. Panel A:

immediate-JOL data from Finn (2008), Experiment 1A. Panel B: delayed-JOL data from Finn (2008), Experiment 1B. Panel C:

immediate-JOL data from the first trials of Kornell and Bjork (2009), Experiments 6 (remember frame) and 7 (forget frame). Panel D:

immediate-JOL data for items of different font sizes from Rhodes and Castel (2008), Experiments 1 (remember frame) and 5 (forget

frame). Forget-framed JOLs were reverse scored.


SERRA AND ENGLAND

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

study. That said, the relative accuracy (i.e., how wellthe values of the judgements discriminated learnedfrom unlearned items regardless of the actual mag-nitude of the judgement; sometimes referred to asresolution) of the JOLs in Finn’s (2008) studiesnever differed significantly by frame. A differencewould have been expected if participants consultedmore predictive cues to make one type of judge-ment than they consulted to make the other. Inaddition, relative-accuracy correlations were mostoften numerically lower for forget-framed JOLsthan for remember-framed JOLs, which is theopposite of what would have been expected ifpeople used more predictive cues to make judge-ments of forgetting than judgements ofremembering.

Although Finn (2008) is the only set of pub-lished studies designed specifically to study theeffects of framing on item-by-item JOLs, at leasttwo other papers (Kornell & Bjork, 2009; Rhodes& Castel, 2008) contain comparisons of memoryjudgements framed in terms of either rememberingor forgetting for other purposes (cf. Koriat et al.,2004). For example, Kornell and Bjork (2009) con-ducted 12 studies that demonstrate that people donot predict additional learning to occur withrepeated study of the same materials. Althoughthese studies were typical of other multitrial learn-ing (and metamemory) studies in that they involvedparticipants studying and testing over the sameitems over multiple trials, Kornell and Bjork’sstudies differed somewhat from most otherwise-similar metamemory studies in that the authorsasked their participants to make learning predic-tions for each item for all four trials after theystudied each item on the first study trial. Whenasked to predict future learning in this way, partici-pants predicted that their learning would stay con-stant across the study trials, a pattern that Kornelland Bjork referred to as “the stability bias inhuman memory”. Notably, Kornell and Bjorkdemonstrated that this occurred even when theyasked participants to make predictions of theirfuture forgetting rather than remembering (i.e.,their Experiment 7), a pattern that indirectly con-tradicts the general findings of Koriat et al. (2004,also Experiment 7) and the basic idea behind the

forgetting-notion hypothesis (i.e., Finn, 2008;Koriat et al., 2004). To aid in the comparison ofresults across studies, we have included some ofKornell and Bjork’s Trial 1 results in our compari-son figure (Figure 2, Panel C), as they are the mostrelevant to the design of the present experiments.

Examining a different question, Rhodes andCastel (2008) conducted six experiments examin-ing the influence of perceptual factors on JOLs.Most notably, they considered how presentingmemory items on a computer screen in either a rela-tively larger (48 pt) or a smaller (18 pt) font sizewould affect participants’ JOLs. Although fontsize did not affect memory for the items, partici-pants judged items shown in the larger font sizeto be more memorable than items shown in thesmaller font size (Figure 2, Panel D).Importantly, this outcome obtained regardless ofwhether they asked participants to make JOLs interms of remembering (their Experiment 1) or for-getting (their Experiment 5). This presumablyoccurred because the font size cue was salient tothe participants. Although font size did not actuallyaffect the participants’ memory for the items, theexperience of studying the items in the larger fontsize (which was easier to read than the smallerfont size) led them to judge that they were morelikely to remember the larger items than the smallitems. In much the same way, Finn (2008) demon-strated that the magnitude and absolute accuracy ofdelayed JOLs (i.e., JOLs made at a delay from thestudy of each item, see Rhodes & Tauber, 2011, fora recent review) were not affected by framing, pre-sumably because participants used the same cue—covert retrieval—to inform their JOLs (cf.Nelson, Narens, & Dunlosky, 2004) regardless ofthe frame (Figure 2, Panel B).

More importantly for the present purposes, theeffect of framing on JOL magnitude—and there-fore, indirectly, absolute accuracy—has so far beeninconsistent across studies. For example, judge-ments of remembering were significantly higherafter reverse scoring than were judgements offorgetting in Finn (2008, Experiment 1A), but theopposite pattern obtained in Kornell and Bjork(2009, Experiments 6 and 7; see Figure 2, PanelC). Similarly, themean predicted recall performance



Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

(% of 60 items expected to be retained) for a 10-minute retention interval in Koriat et al. (2004,Experiments 4B and 7) demonstrated a patternalmost identical to that of Kornell and Bjork(2009): Forget-framed judgements were higher inmagnitude (after reverse scoring) than were judge-ments of remembering (Figure 1, Panel B).Furthermore, people’s memory predictions in theKoriat et al. (2004) and Kornell and Bjork (2009)studies are quite similar: Participants expected toremember about 50% of the items when researchersframed the question in terms of forgetting (andreverse-scored the predictions), but only expectedto remember about 30% of the items when research-ers framed the question in terms of remembering. Amean of 30% is in accord with earlier proposals thatparticipants tend to anchor immediate, remember-framed JOLs at or around the 30% point on theJOL scale, which causes the mean of such JOLs totypically fall around 30% (England & Serra, inpress; Scheck, Meeter, & Nelson, 2004; Scheck &Nelson, 2005). As previously noted, JOLs inRhodes and Castel (2008; see Figure 2, Panel D)and delayed JOLs in Finn (2008; see Figure 2,Panel B) did not differ in magnitude or absoluteaccuracy by frame. Further, in Finn’s (2008)Experiment 3, participants studied Spanish–English translations of varying difficulties, anddifferences in absolute accuracy by frame also didnot obtain in this experiment, presumably becausethe item difficulty (i.e., memorability) was salientto participants. As such, addressing these inconsis-tencies across studies was one purpose of thepresent experiments.

The present experiments

Given the existing data and the authors’ interpret-ations of them, there are two possibilities for howframing affects the absolute accuracy of JOLs.First, the forgetting-notion hypothesis (Koriatet al., 2004; see also Finn, 2008) suggests that aforget frame causes participants to rely on better(i.e., more predictive) cues to make their JOLsthan does a remember frame, which causes judge-ments of forgetting to exhibit better absolute accu-racy than do judgements of remembering.

Although there are some data that indirectlysupport this idea (Koriat et al., 2004; immediateJOLs in Finn, 2008), to date no studies havedemonstrated direct evidence for this possibility(e.g., evidence indicating that participants’ utiliz-ation of any specific cues differed by frame).More problematically, there are also other datathat clearly do not support the forgetting-notionhypothesis, either because no differences in absol-ute accuracy obtained by frame (i.e., delayed JOLsin Finn, 2008, Experiment 1b; immediate JOLsin Finn, 2008, Experiment 3; immediate JOLs inRhodes & Castel, 2008) or because judgements ofremembering demonstrated better absolute accu-racy than did judgements of forgetting (i.e.,Kornell & Bjork, 2009). Further, differences incue usage by frame should also produce differencesin relative accuracy, an outcome that to our knowl-edge has only been examined in one set of studies(Finn, 2008) and did not obtain.

In contrast, a second possibility is that framingdoes not actually affect the absolute accuracy ofJOLs. Rather, a more parsimonious explanationmight be that the absolute accuracy of JOLs islargely artefactual in nature and that apparentdifferences in absolute accuracy by frame stemmostly from participants’ basing their JOLsaround different psychological “anchor points” onthe JOL scale (cf. England & Serra, in press;Meeter & Nelson, 2003; Scheck et al., 2004;Scheck & Nelson, 2005). When people makeJOLs, they presumably do so by adjusting theJOL for a given item up or down from a presetpsychological anchor based on whether they haveaccumulated evidence that they will or will notremember that item. Given that adjustment fromthe anchor is often insufficient (Scheck & Nelson,2005), JOLs will often exhibit poor absolute accu-racy (unless memory performance happens to fallnear the anchor). Anchoring and framing couldpotentially interact to affect the absolute accuracyof JOLs if (a) the different frames cause people toadopt different anchors on the JOL scale or (b)the different frames cause people to factor differentinformation into their adjustment from an anchoror to make larger/smaller adjustments from ananchor. Either way, when framing produces a


SERRA AND ENGLAND

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

difference in JOL magnitude (but memory is notaffected), whichever type of JOL happens to havea mean judgement magnitude closer to the meanmemory-performance level will appear to be themore accurate frame. Although this pattern by defi-nition indicates greater absolute accuracy, thispossibility nevertheless suggests that any apparentdifferences in absolute accuracy by frame shouldbe carefully considered before a strong conclusionis made about the effect of framing on the “accu-racy” of JOLs (see also Scheck & Nelson, 2005).If a forget frame actually improves the absoluteaccuracy of JOLs compared to a remember frame,then the accuracy advantage for forget-framedJOLs over remember-framed JOLs should persistacross changes in—or manipulations of—actualmemory performance.

Towards these ends, the present experimentshad three main purposes. First, given the inconsist-ent effect of framing on JOL magnitude and absol-ute accuracy in previous studies, we sought tofurther document the effect of this manipulationon these measures. Second, we sought to comparethe forgetting-notion hypothesis (Koriat et al.,2004; see also Finn, 2008) with the possibilitythat any effect of framing on the absolute accuracyof JOLs is largely artefactual (i.e., that the differ-ences are largely the result of differences in anchor-ing—rather than accuracy—by frame). Third, wewished to further examine the idea that the framechanges participants’ cue utilization by comparingthe relative accuracy of judgements of rememberingand forgetting over multiple study–test trials. Ourgeneral approach for examining these issues wasto have participants study paired-associates itemsover two study–test trials, which should producean increase in memory-test performance acrossthe trials. Based on the forgetting-notion hypoth-esis, if a forget frame truly improves absolute accu-racy compared to a remember frame, then thisaccuracy advantage should persist across trialseven as the level of memory performance changes.If, however, absolute accuracy differences byframe are largely artefactual and stem from differ-ences in the anchoring (or adjustment) of JOLs,then any obtained pattern should switch acrosstrials as performance increases (i.e., any group

with higher magnitude JOLs should be highlyoverconfident on Trial 1 and only slightly under-confident on Trial 2, whereas any group withlower magnitude JOLs should be only slightly over-confident on Trial 1 and highly underconfident onTrial 2). We predict this pattern because memoryjudgements such as JOLs tend to exhibit a stabilitybias in that they are fairly “flat” across trials eventhough memory typically improves with repeatedstudy (e.g., Kornell & Bjork, 2009).

Using two study–test trials will also allow us tofurther consider the basic tenet of the forgetting-notion hypothesis, which is that the forget frameleads people to consult more predictive cues tomake their JOLs than does the remember frame.Relative accuracy typically improves across study–test trials (in particular, from Trial 1 to Trial 2)because people change the type of cue they use toinform their JOLs (cf. Koriat 1997, 2008). Onespecific cue that becomes available on a secondtrial—and that people tend to use naturally—ismemory for past test (MPT) performance (Ariel& Dunlosky, 2011; England & Serra, in press;Finn & Metcalfe, 2007, 2008; King,Zechmeister, & Shaughnessy, 1980). Given thatmemory performance for the items on Trial 1 ishighly predictive of memory performance on Trial2, when people base their Trial 2 JOL for an itemon whether or not they remembered that item onthe Trial 1 test, they typically achieve very goodrelative accuracy on Trial 2. If people base forget-framed JOLs on better-predictive cues than theybase remember-framed JOLs, the relative accuracyof forget-framed JOLs should exceed—or at leastmatch—that of remember-framed JOLs acrosstrials.

To preview, in Experiment 1 we examined theeffect of framing on the magnitude, absolute accu-racy, and relative accuracy of both immediate anddelayed JOLs across two study–test trials. InExperiments 2 and 3, we examined whether par-ticipants use different anchors on the JOL scaleto make immediate JOLs framed in terms ofeither remembering or forgetting and consideredhow this might lead to differences in absolute accu-racy by frame. In Experiment 4, we examinedwhether making a cue for JOLs salient to the



Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

participants eliminated differences in the magni-tude, absolute accuracy, and relative accuracy ofimmediate JOLs due to framing.

EXPERIMENT 1

The purpose of Experiment 1 was (a) to collectadditional data using immediate and delayedJOLs framed in terms of either remembering orforgetting and (b) to consider the magnitude andaccuracy of JOLs framed in terms of either remem-bering or forgetting over two study–test trials. Todo so, we had participants in Experiment 1 study64 paired associates over two study–test trials.Groups made their JOLs in terms of either the like-lihood of remembering or the likelihood of forget-ting and either immediately after studying eachitem (immediate-JOL groups) or at an eight-itemdelay (delayed-JOL groups). Given previousresults with delayed JOLs (i.e., Finn, 2008, seealso Rhodes & Tauber, 2011), we did not expecteither the absolute or relative accuracy of delayedJOLs to differ by frame, nor did we expect frameto interact with trial. The reason for both of thesepredictions is that we did not expect the cues thatparticipants typically use to make delayed JOLs(i.e., covert retrieval during the delayed JOL) tochange by frame or across trials. In contrast, theresults for immediate JOLs are more difficult topredict because of the inconsistent effects offraming on the magnitude and accuracy of immedi-ate JOLs in previously published studies. Ingeneral, however, if it is true that people makeforget-framed immediate JOLs using more predic-tive cues than they use to make remember-framedimmediate JOLs (i.e., Finn, 2008; Koriat et al.,2004), then we expected absolute and relative accu-racy to be better for forget-framed JOLs than forremember-framed JOLs on both trials. In contrast,if the apparent effect of framing on the absoluteaccuracy of immediate JOLs is largely artefactual,

then we expected absolute accuracy to change asmemory performance improved across trials (i.e.,regardless of frame, JOLs will exhibit overconfi-dence when memory is low and underconfidencewhen it is high regardless of the frame).

Method

Participants, materials, and designThe participants were 192 undergraduates enrolledin General Psychology at Texas Tech University.They took part in this experiment for coursecredit. The study materials were the 66 pairedassociates from the Appendix of Serra, Dunlosky,and Hertzog (2008). The design was a 2(framing: remember vs. forget)× 2 (type of JOL:immediate vs. delayed) between-participants fac-torial design.1

ProcedureThe researchers randomly assigned each participantto one of the four groups when they entered thelaboratory, with the restriction that an equalnumber of participants were eventually assignedto the four groups. The participants firstread detailed instructions describing the task andthe judgement that they would be making foreach item. The instructions for each participantwere framed in the same way as the JOLswould later be framed for that participant(variations by framing condition are in squarebrackets):

After you study each pair of words, you will be asked to predict

how likely you think you will be to [remember/forget] the item

on the upcoming test. You will make this judgement on a scale

from 0 to 100. A “0” would indicate that you feel there is a 0%

chance you will [remember/forget] the item (i.e., you feel you

definitely WILL NOT [remember/forget] it) and a “100”

would indicate that you feel there is a 100% chance you will

[remember/forget] the item (i.e., you feel you definitely WILL

[remember/forget] it).

1 We did not have a reason to directly compare the two types of JOL (immediate vs. delayed) to each other, so we conducted all

analyses separately for immediate and delayed JOLs rather than analysing the type of JOL as an independent variable. Interested

readers, however, can safely make comparisons across the type of JOL if they choose to. Towards this end, Rhodes and Tauber

(2011) included data from the remember-framed immediate- and delayed-JOL groups in Experiment 1 in their meta-analysis of

delayed JOLs.


SERRA AND ENGLAND

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

The instructions did not mention the potentialdelay of the JOLs but, to foreshadow, typicaldelayed-JOL effects obtained (i.e., good absoluteand relative accuracy), so we have no reason tobelieve that omitting this aspect of the procedurefrom the instructions affected the JOLs.

Personal computers running a custom computerprogram then presented the materials to the partici-pants and collected participants’ responses. Thecomputer program first randomly selected 64 ofthe 66 pairs to serve as each participant’s studylist on both trials. The program then randomizedthe order of the 64 items for each participant andpresented each item individually for a fixed dur-ation of 4 s/item. In the immediate-JOL groups,participants made a JOL for each item immediatelyafter studying it until they studied and judged all 64items in that way. No time was inserted betweenthe offset of one item’s JOL and the onset of thenext item’s presentation. In the delayed-JOLgroups, participants studied 8 items and thenmade the JOLs for those 8 items in turn. Giventhat the study and judgement making of otheritems occurred between the study and judgementmaking for any given item, an average of 31.5 spassed between the study and judgement of anyindividual item. The participants then studiedand judged additional sets of 8 items in this wayuntil they studied and judged all 64 items.Regardless of the type of JOL, we manipulatedthe framing of the JOL prompt in terms of remem-bering or forgetting between participants as in Finn(2008). To obtain JOLs, the computer programdisplayed the stimulus word (the first word ineach pair) and asked the participant, “How likelydo you think you will be to [remember/forget] thesecond word of this pair approximately 5 minutesfrom now?” Participants made JOLs on a scalefrom 0 (0% chance of remembering or forgetting)to 100 (100% chance of remembering or forget-ting). The computer program accepted all whole-number responses between 0 and 100. Themaking of the JOLs was participant paced.

After participants had studied and judged all theitems, the program again randomized the order ofthe items and presented each stimulus word indivi-dually for paired-associate recall. The instructions

encouraged participants to type the response corre-sponding to each stimulus word, but allowed themto omit responses if they chose. The entire pro-cedure then repeated for a second trial duringwhich participants studied the same 64 items in anewly randomized order (again for 4 s/item),judged them in the same way as on Trial 1, andtested over them as before.

Results

RecallWe scored the response words recalled by the par-ticipants as either correct or incorrect; we did notaward partial credit because the majority of errorswere omission errors. Figure 3 shows the meanrecall for the immediate-JOL groups and Figure 4shows the mean recall for the delayed-JOLgroups. Recall increased from Trial 1 to Trial 2for the immediate-JOL groups, F(1, 94)= 472.2,MSE= 94.6, p, .001, ηp

2= .8, and for thedelayed-JOL groups, F(1, 94)= 430.9, MSE=88.1, p, .001, ηp

2= .8. Framing did not affectrecall for either the immediate- or the delayed-JOL groups, nor did frame ever interact with trial.

JOL magnitudeWe reverse-scored forget-framed JOLs (i.e., 100 –

JOL) as in Finn (2008) and Koriat et al. (2004) sowe could directly compare them to remember-framed JOLs on the same scale. We first calculatedthe mean JOL for each participant on both trialsand then calculated the overall means for each ofthe two groups. Figure 3 shows the mean JOLfor the immediate-JOL groups, and Figure 4shows the mean JOL for the delayed-JOL groups.JOLs increased from Trial 1 to Trial 2 for theimmediate-JOL groups, F(1, 94)= 9.4, MSE=118.2, p= .003, ηp

2= .1, and for the delayed-JOLgroups, F(1, 94)= 107.3, MSE= 108.3,p, .001, ηp

2= .5. More importantly, forget-framed immediate JOLs were higher than wereremember-framed JOLs, F(1, 94)= 10.7,MSE= 633.2, p= .001, ηp

2= .1, but delayedJOLs did not differ by frame. Framing did notinteract with trial for either the immediate- or thedelayed-JOL groups.



Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

Absolute accuracyWe operationalized absolute accuracy as the meandifference score between participants’ JOLs andrecall. For the immediate-JOL groups, absoluteaccuracy changed across trials, F(1, 94)= 237.4,MSE= 133.4, p, .001, ηp

2= .7, and framingaffected absolute accuracy, F(1, 94)= 7.7,MSE= 778.8, p= .007, ηp

2= .08. Immediateremember-framed JOLs were somewhat

overconfident on Trial 1, F(1, 47)= 5.6, MSE=205.2, p= .022, ηp

2= .1, and highly underconfi-dent on Trial 2, F(1, 47)= 23.3, MSE= 325.9,p, .001, ηp

2= .3, whereas immediate forget-framed JOLs were highly overconfident on Trial1, F(1, 47)= 46.6, MSE= 186.9, p, .001,ηp2= .5, and only somewhat underconfident onTrial 2, F(1, 47)= 7.1, MSE= 194.2, p= .01,ηp2= .1 (Figure 3). The two factors, however, did

Figure 4. Mean JOL (judgement of learning) and recall performance across trials for the delayed-JOL groups in Experiment 1. Forget-framed

JOLs were reverse scored. Error bars are the standard error of the mean.

Figure 3. Mean JOL (judgement of learning) and recall performance across trials for the immediate-JOL groups in Experiment 1. Forget-

framed JOLs were reverse scored. Error bars are the standard error of the mean.


SERRA AND ENGLAND

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

not interact significantly. For the delayed-JOLgroups, absolute accuracy changed across trials, F(1, 94)= 73.3, MSE= 103.4, p, .001, ηp

2= .4,as JOLs in both groups demonstrated overconfi-dence on Trial 1. That said, although the overcon-fidence was highly significant for the forget-framedgroup, F(1, 47)= 34.2, MSE= 150.5, p, .001,ηp2= .4, the difference for the remember-framedgroup was marginal, F(1, 47)= 3.5, MSE=266.6, p= .07, ηp

2= .07. Both groups were accurateon Trial 2 (Figure 4). Neither the main effect offraming nor the interaction of framing with trialwas significant for delayed JOLs.

We also considered the groups’ overall absoluteaccuracy by calculating a related measure, partici-pants’ shift to underconfidence (Meeter & Nelson,2003). This measure is basically a total bias scoreand can be calculated by subtracting participants’absolute accuracy scores on Trial 2 from their absol-ute accuracy scores on Trial 1. These total scoresdid not differ by frame for either the immediate-or the delayed-JOL groups in Experiment 1,further indicating that there were no actualdifferences in absolute accuracy by frame ineither case.

SlopesAs in Scheck et al. (2004), we considered partici-pants’ anchoring by calculating the changes (i.e.,slopes) in participants’ JOLs and recall acrosstrials (Table 1). Although all values were positive,slopes were greater for changes in recall than forchanges in JOLs for both immediate JOLs, F(1,94)= 237.4, MSE= 133.4, p, .001, ηp

2= .7,and delayed JOLs, F(1, 94)= 73.3, MSE=103.4, p, .001, ηp

2= .4. Slopes were not affectedby the frame of the JOL. No interactions were sig-nificant. These results suggest that participants’reliance on anchors to make their JOLs impairedtheir ability to fully capture increases in recallacross trials (cf. Scheck et al., 2004). We explorethis possibility further in Experiments 2 and 3.

Relative accuracyWe operationalized relative accuracy as a gammacorrelation between participants’ JOLs andmemory performance on a given trial (Table 2).2

Overall, the relative accuracy for the immediateJOLs improved from Trial 1 to Trial 2, F(1,90)= 10.4, MSE= 0.2, p= .002, ηp

2= .1. Moreimportantly, relative accuracy for immediate JOLs

Table 1. Slopes of JOLs and recall across trials

ExperimentRemember frame Forget frame

JOLs Recall JOLs Recall

M SD M SD M SD M SD

Experiment 1

Immediate JOLs +6.1 15.6 +30.9 12.1 +3.5 15.2 +30.1 15.2

Delayed JOLs +16.7 13.0 +27.6 13.1 +14.4 16.3 +28.6 13.4

Experiment 2

2-s/item rate +4.4 13.8 +28.5 13.5 +4.5 19.0 +28.9 12.3

6-s/item rate +8.5 16.3 +35.6 10.0 +10.3 18.0 +31.5 14.9

Experiment 3 +9.5 16.7 +30.9 14.2 +6.1 13.9 +30.6 15.4

Experiment 4 +12.3 14.5 +29.1 9.3 +10.4 15.1 +28.6 9.2

Note: JOL= judgement of learning. Slopes are the mean difference between the Trial 2 value and the Trial 1 value (i.e., Trial 2 JOL –

Trial 1 JOL and Trial 2 Recall – Trial 1 Recall).

2 One potential criticism of forget-frame JOLs is that people might be confused about how to make these judgements and use the

scale “backwards”. These positive relative-accuracy correlations (especially those for delayed JOLs), however, suggest that people can

make forget-frame JOLs correctly, and can do with considerable accuracy in some situations.



Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

was higher for remember-framed JOLs than forforget-framed JOLs, F(1, 90)= 4.4, MSE= 0.2,p= .04, ηp

2= .05. The interaction betweenframing and trial was marginally significant, F(1,90)= 3.9, MSE= 0.2, p= .05, ηp

2= .04. Follow-up t tests confirmed that framing did not affectrelative accuracy on Trial 1, t(93)= 0.2, p= .9,d= 0.03, but that relative accuracy was higher forremember-framed JOLs than for forget-framedJOLs on Trial 2, t(91)= 2.6, p= .01, d= 0.55.Relative accuracy for all delayed JOLs was quitegood (i.e., means around +.7) and did not differby frame or by trial. That said, the interactionbetween these two factors was marginally signifi-cant for delayed JOLs, F(1, 87)= 3.6, MSE=0.09, p= .06, ηp

2= .04.It is well established that participants’ use of

memory for past test (MPT) information to makeJOLs (after an initial study–test trial) plays amajor role in improving the relative accuracy ofimmediate JOLs across trials (e.g., Ariel &Dunsloky, 2011; England & Serra, in press; Finn& Metcalfe, 2007, 2008). As such, we consideredwhether the forget frame reduced participants’ useof MPT information on Trial 2, which mighthelp explain why relative accuracy was impaired

on Trial 2 for forget-framed immediate JOLs. Asin Finn and Metcalfe (2007, 2008), we examinedparticipants’ use of the MPT heuristic to maketheir Trial 2 JOLs by calculating within-participantgamma correlations between Trial 2 JOLs andTrial 1 memory performance for both immediate-JOL groups (Table 2). As expected, the meanMPT correlation was lower for the forget-framedgroup than for the remember-framed group, F(1,91)= 6.9, MSE= 0.3, p= .01, ηp

2= .07.Although less relevant to the relative accuracy ofdelayed JOLs, for completeness we also calculatedMPT for the delayed groups (Table 2). These cor-relations did not differ by frame for delayed JOLs,F(1, 90)= 0.1, MSE= 0.1, p= .7, ηp

2= .001.

Discussion

The purpose of Experiment 1 was to examine theeffect of framing on the magnitude and accuracyof immediate and delayed JOLs over two study–test trials. As we expected based on previousresearch examining the effects of framing onsingle-trial delayed JOLs (Finn, 2008,Experiment 1B), framing did not affect the magni-tude, absolute accuracy, or relative accuracy of

Table 2. Relative accuracy and MPT utilization

Remember frame Forget frame

Trial 1 Trial 2 Trial 1 Trial 2

Rel. acc. Rel. acc. MPT-U Rel. acc. Rel. acc. MPT-U

Experiment M SD M SD M SD M SD M SD M SD

Experiment 1

Immediate JOLs +.23 .31 +.54 .32 +.82 .34 +.22 .42 +.29 .56 +.52 .70

Delayed JOLs +.74 .33 +.68 .46 +.78 .35 +.65 .48 +.75 .35 +.80 .32

Experiment 2

2-s/item rate +.39 .30 +.56 .41 +.77 .44 +.28 .40 +.32 .61 +.59 .69

6-s/item rate +.31 .18 +.58 .19 +.87 .13 +.29 .24 +.40 .47 +.64 .64

Experiment 3 +.37 .23 +.55 .30 +.81 .33 +.32 .36 +.46 .35 +.69 .44

Experiment 4 +.27 .22 +.57 .23 +.85 .20 +.17 .38 +.33 .54 +.64 .58

Note: JOL= judgement of learning. MPT=memory for past test. Relative accuracy (“Rel. acc.”) is the mean gamma correlation

between JOLs and recall on a given trial. MPT-utilization (“MPT-U”) is the mean gamma correlation between JOLs on Trial 2

and recall on Trial 1.


SERRA AND ENGLAND

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

delayed JOLs in our Experiment 1. It seems likelythat making delayed JOLs is so likely to elicit par-ticipants’ covert retrieval of responses (e.g., Nelsonet al., 2004) that the retrieval outcome might be themain cue that people use to make delayed JOLs nomatter how we frame the judgements (see alsoFinn, 2008). Further, using the outcome of suchcovert retrieval attempts to inform JOLs producesextremely polarized JOLs (i.e., very high and verylow JOL values). As such, it seems logical thatthe magnitude, absolute accuracy, and relative accu-racy of the delayed JOLs did not differ by frame oracross trials.

In contrast, framing clearly affected the magni-tude and accuracy of immediate JOLs in ourExperiment 1. First, although JOLs had equivalentrelative accuracy across frames on Trial 1 (cf. Finn,2008), forget-framed JOLs had significantly lowerrelative accuracy than did remember-framed JOLson Trial 2. This latter outcome seemed to obtainbecause participants in the forget-framed groupmade less use of MPT information to informtheir Trial 2 JOLs (a highly predictive cue, seeAriel & Dunlosky, 2011; England & Serra, inpress; Finn & Metcalfe, 2007, 2008) than didthose in the remember-framed group. These rela-tive-accuracy results are certainly not in line withthe forgetting-notion hypothesis, which suggeststhat a forget frame leads participants to consultmore predictive cues to make forget-framed JOLsthan to make remember-famed JOLs.

Second, unlike in Finn (2008), in ourExperiment 1 our participants’ remember-framedJOLs were consistently lower than were our partici-pants’ forget-framed JOLs across trials (but seeKoriat et al., 2004; Kornell & Bjork, 2009). As aresult, forget-framed JOLs were highly overconfi-dent on Trial 1 when recall was low, and remem-ber-framed JOLs were highly underconfident onTrial 2 when recall was high. In contrast, remem-ber-framed JOLs were fairly well calibrated onTrial 1 when recall was low, and forget-framedJOLs were fairly well calibrated on Trial 2 whenrecall was high. Recall performance did not differby frame (in this or any of our other experiments),so the different patterns of absolute accuracy byframe are not attributable to differential recall.

As such, it does not seem parsimonious (nor is itin line with the forgetting-notion hypothesis) topropose that remember-framed JOLs start outnearly accurate on Trial 1 and then become inac-curate on Trial 2, whereas forget-framed JOLsstart out inaccurate on Trial 1 and then becomenearly accurate on Trial 2. Rather, a simpler expla-nation might be that the absolute accuracy of JOLsis largely artefactual in nature and that these appar-ent differences in absolute accuracy by frame stemmostly from participants’ basing their JOLsaround different “anchor points” on the JOL scale(cf. England & Serra, in press; Meeter & Nelson,2003; Scheck et al., 2004; Scheck & Nelson,2005) and not adjusting them much across trials(cf. Kornell & Bjork, 2009). It seems possiblethat our participants based their immediate remem-ber-framed JOLs around a low point on the JOLscale (i.e., around 35%; see also England & Serra,in press; Scheck & Nelson, 2005) and theirimmediate forget-framed JOLs around the mid-point on the JOL scale (i.e., around 50%). Weexplore this possibility more directly (and ingreater detail) in Experiments 2 and 3.

EXPERIMENT 2

In Experiments 2 and 3, we examined the possi-bility that the differences in immediate-JOL mag-nitude and absolute accuracy by frame inExperiment 1 stemmed from participants’ usingdifferent anchors on the JOL scale for the twodifferent frames. Inspection of Figure 3 (as well asdata in Koriat et al., 2004, and Kornell & Bjork,2009) suggests that remember-framed JOLsmight be anchored around the 35% point on theJOL scale whereas forget-framed JOLs might beanchored around the 50% point on the JOL scale.To examine this possibility more directly, inExperiment 2 we manipulated the per-item studytime (either 2 s/item or 6 s/item) to create groupswith significantly different levels of recall perform-ance. Again, participants made immediate JOLs interms of either the likelihood of remembering orthe likelihood of forgetting. Regardless of theframing of the JOLs, we expected the absolute



Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

accuracy of the JOLs to be dependent on the par-ticipants’ level of recall. If anchoring can largelyaccount for the different magnitude and absoluteaccuracy of JOLs by frame, then manipulating thelevel of recall performance up or down should notaffect JOL magnitude, but should affect absoluteaccuracy. If, however, judgements of forgettingtruly demonstrate better absolute accuracy than dojudgements of remembering, then judgements offorgetting should be more sensitive to the manipu-lated level of memory performance than are judge-ments of remembering and should thereforedemonstrate better absolute accuracy betweengroups and across study–test trials as performancevaries.

Method

Participants, materials, and designThe participants were 112 undergraduates enrolledin General Psychology at Texas Tech Universitywho took part in this experiment for coursecredit. None had participated in the earlier exper-iment. We used the same list of paired associatesfor the study materials as that used in earlier exper-iments. The design was a 2 (framing: remember vs.forget)× 2 (study duration: 2 s/item vs. 6 s/item)between-participants factorial design.

ProcedureThe procedure of Experiment 2 was identical tothat of the two immediate-JOL groups inExperiment 1 except that in Experiment 2 wemanipulated the per-item study duration betweengroups to produce different levels of memory per-formance. Whereas participants in Experiment 1studied all items for 4 s each on both study–testtrials, participants in Experiment 2 studied allitems for either 2 s or 6 s each on both study–testtrials.

Results

RecallFigure 5 shows the mean recall for the four groups.Recall increased across the trials, F(1, 108)=665.1, MSE= 81.6, p, .001, ηp

2= .9. Moreimportantly, the study-time manipulation was suc-cessful at producing a difference in recall: Recallwas higher for the groups that studied the itemsfor 6 s than for the groups that studied the itemsfor 2 s, F(1, 108)= 10.1, MSE= 807.0, p= .002,ηp2= .09. Trial interacted with study time, as theincrease in recall across trials was greater forthe groups that studied the items for 6 s thanfor the groups that studied the items for 2 s,

Figure 5. Mean JOL (judgement of learning) and recall performance across trials in Experiment 2. Forget-framed JOLs were reverse scored.

Error bars are the standard error of the mean.


SERRA AND ENGLAND

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

F(1, 108)= 4.1, MSE= 81.6, p= .046, ηp2= .04.

No other interactions were significant.

JOL magnitudeWe again reverse-scored forget-framed JOLs (i.e.,100 – JOL) so we could directly compare them toremember-framed JOLs on the same scale. Figure5 shows the mean JOL for the four groups. JOLsincreased across the trials, F(1, 108)= 18.8,MSE= 142.5, p, .001, ηp

2= .2. Forget-framedJOLs were higher than were remember-framedJOLs, F(1, 108)= 15.1, MSE= 679.7, p, .001,ηp2= .12. As expected, the study-time manipulationdid not affect JOL magnitude even though itaffected the level of recall. No interactions weresignificant.

Absolute accuracyAbsolute accuracy changed across trials, F(1,108)= 192.4, MSE= 170.6, p, .001, ηp

2= .6,and framing affected absolute accuracy, F(1,108)= 4.9, MSE= 874.1, p= .03, ηp

2= .04.Remember-framed JOLs were somewhat overcon-fident on Trial 1 for the 2-s/item group, F(1, 27)=12.4, MSE= 225.1, p= .002, ηp

2= .3, but wereonly marginally overconfident for the 6-s/itemgroup, F(1, 27)= 3.6, MSE= 202.6, p= .07,ηp2= .1. Remember-framed JOLs were highlyunderconfident on Trial 2 for both the 2-s/itemgroup, F(1, 27)= 5.1, MSE= 272.3, p= .03,ηp2= .2, and the 6-s/item group, F(1, 27)= 32.8,MSE= 169.2, p, .001, ηp

2= .5. In contrast,forget-framed JOLs were highly overconfident onTrial 1 for both the 2-s/item group, F(1, 27)=20.4, MSE= 306.7, p, .001, ηp

2= .4, and the 6-s/item group, F(1, 27)= 10.2, MSE= 301.9,p= .004, ηp

2= .3, but neither group was signifi-cantly underconfident on Trial 2 (Figure 5).Although the effect of study time on absolute accu-racy was only marginally significant, F(1, 108)=2.8,MSE= 874.1, p= .1, ηp

2= .03, the differenceswere nevertheless in the predicted direction (i.e.,greater overconfidence when performance waslower and greater underconfidence when perform-ance was higher) in that the study-time manipu-lation affected recall but not the magnitude of theJOLs. No interactions were significant.

We also considered participants’ shift-to-under-confidence scores as another measure of absoluteaccuracy as in Experiment 1. These scores werealso not affected by either framing or the studytime manipulation, further indicating that therewere no actual differences in absolute accuracy byframe.

SlopesAlthough all values were positive (Table 1), slopeswere greater for changes in recall than for changesin JOLs, F(1, 108)= 192.4, MSE= 170.6,p, .001, ηp

2= .6. Slopes were also greater for thegroups that studied the items for 6 s than for thegroups that studied the items for 2 s, F(1, 108)=4.9, MSE= 277.5, p= .03, ηp

2= .04. No othermain effects or interactions were significant.

Relative accuracyOverall, the relative accuracy of the JOLs (Table 2)improved across trials, F(1, 100)= 14.4, MSE=0.1, p, .001, ηp

2= .1. More importantly, relativeaccuracy was higher for remember-framed JOLsthan for forget-framed JOLs, F(1, 100)= 6.5,MSE= 0.2, p= .01, ηp

2= .06. The study timemanipulation did not affect relative accuracy.Follow-up t tests confirmed that relative accuracywas higher for remember-framed JOLs than forforget-framed JOLs on Trial 2, t(106)= 2.5,p= .01, d= 0.5, but did not differ on Trial 1. Nointeractions were significant.

The mean MPT correlation (Table 2) was lowerfor the forget-framed groups than for the remem-ber-framed groups, F(1, 100)= 4.5, MSE= 0.2,p= .04, ηp

2= .04. The study-time manipulationdid not affect MPT utilization, nor did studytime interact with frame.

Discussion

Our study-time manipulation was successful at pro-ducing differences in recall performance but—aswould be expected based on the anchoring expla-nation for JOLs—differential performance didnot affect the magnitude of the JOLs, which wasconsistently higher for the forget-framed groupsthan for the remember-framed groups. Although



Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

the main effect of this study time manipulation onabsolute accuracy was only marginally significant,the mean scores nevertheless reflect exactly whatwe predicted: JOLs were more overconfidentwhen recall was low than when recall was highand were more underconfident when recall washigh than when recall was low. These outcomesprovide further evidence that JOL magnitudes arelargely dependent upon psychological anchorpoints on the JOL scale (England & Serra, inpress; Scheck et al., 2004; Scheck & Nelson,2005) and that remember- and forget-framedJOLs are anchored around different points on theJOL scale. We further examined these two issuesin Experiment 3.

EXPERIMENT 3

The purpose of Experiment 3 was (a) to once againreplicate the immediate-JOL findings from theearlier experiments and (b) to provide further evi-dence that JOLs framed in terms of either remem-bering or forgetting utilize different anchor pointson the JOL scale. To do so, we again used a vari-ation of the methodology of the immediate-JOLgroups in Experiment 1. As in Experiment 1, wehad participants in Experiment 3 study 64 pairedassociates over two study–test trials while makingimmediate JOLs in terms of either the likelihoodof remembering or the likelihood of forgetting. InExperiment 3, however, participants also madesecond-order judgements—SOJs—to indicatetheir confidence in the accuracy of their JOLs.We have previously used this methodology todemonstrate that participants are more confidentin the accuracy of delayed than immediate JOLs(Dunlosky, Serra, Matvey, & Rawson, 2005), thatparticipants are more confident in JOLs for setsof items with higher than with lower relative accu-racy (also Dunlosky et al., 2005), and that younger

adults are more confident in the accuracy of theirimmediate JOLs than are older adults (Serraet al., 2008). More recently, Miller and Geraci(2011) used a similar SOJ procedure to demon-strate that low-performing students in a classroomsetting are less confident in their predictions oftheir grades on upcoming exams than are high-per-forming students.3

By considering the SOJ–JOL relationship forJOLs framed in terms of either remembering orforgetting, we can better identify potentialanchors for the two types of JOL. As previouslydescribed, as a participant acquires more infor-mation that they will or will not remember agiven item, they presumably adjust their JOLaway from the anchor as they feel is appropriate(England & Serra, in press; Scheck et al., 2004;Scheck & Nelson, 2005). Therefore, participantsshould be more confident—and assign higherSOJs—to items for which they acquired moreinformation (see Dunlosky et al., 2005) andshould be less confident—and assign lower SOJs—to items for which they acquired little infor-mation (i.e., for items at or near the anchor). Assuch, plotting the SOJ–JOL relationship for JOLsframed in terms of either remembering or forget-ting should reveal the potential anchor point(s) ofthese judgements (i.e., SOJ magnitude should bethe lowest near the anchor point). Based on theresults of the preceding experiments, we expectthis point to be around 35% for remember-framed JOLs and around 50% for forget-framedJOLs.

Method

Participants, materials, and designThe participants were 56 undergraduates enrolledin General Psychology at Texas Tech Universitywho took part in this experiment for coursecredit. None had participated in the earlier

3 Although such “metacognition about metacognition” has not received much empirical study in the cognitive-psychology litera-

ture, it has been considered in greater depth in the social-psychology literature (albeit in a less direct way, e.g., Briñol, Petty, &

Tormala, 2006; McCaslin, Petty, & Wegener, 2010). Furthermore, prominent cognitive conceptualizations of metacognition (i.e.,

Nelson & Narens, 1994) concede the likelihood of “meta-metacognition” and stress that such thinking is in fact no different from

any other metacognition, as it is simply another form of “thought about thought”.


SERRA AND ENGLAND

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

experiments. We used the same list of pairedassociates for the study materials that we used inearlier experiments. The type of framing (remem-ber vs. forget) was the only between-participantsindependent variable.

ProcedureThe procedure of Experiment 3 was identical tothat of the two immediate-JOL groups inExperiment 1 except that in Experiment 3 partici-pants also made an SOJ for each of their JOLs onboth study–test trials. After participants made aJOL for an item, the computer program askedthem, “You judged that you feel you have a[value]% chance of [remembering/forgetting] thecurrent item on the test. How accurate do youthink the judgement you just made is? (0= ‘defi-nitely NOT accurate’; 100= ‘definitely accurate’)”.The [value] was replaced with the JOL the partici-pant assigned to the just-judged item. The[remember/forget] option matched the framingcondition of the JOLs. Participants did not seeany aspect of the actual item (i.e., the entire pairor the stimulus word alone) referred to at thispoint, to focus their judgement on the JOL, noton the item. As noted, the prompt instructed par-ticipants to make their SOJs on a 0 (the JOL is defi-nitely not accurate) to 100 (the JOL definitely is

accurate) scale and hence did not need to bereverse-scored for the forget group.

Results

RecallFigure 6 shows the mean recall for the two groups.Recall increased across the trials, F(1, 54)= 242.5,MSE= 109.2, p, .001, ηp

2= .8. Framing did notaffect recall, nor did frame interact with trial.

JOL magnitudeWe again reverse-scored forget-framed JOLs (i.e.,100 – JOL) so we could directly compare them toremember-framed JOLs on the same scale.Figure 6 shows the mean JOL for the twogroups. JOLs increased across the trials, F(1,54)= 14.6, MSE= 117.6, p, .001, ηp

2= .2, andforget-framed JOLs were higher than were remem-ber-framed JOLs, F(1, 54)= 5.1, MSE= 445.3,p= .03, ηp

2= .09. Framing did not interact withtrial.

Absolute accuracyAbsolute accuracy of the JOLs changed acrosstrials, F(1, 54)= 65.2, MSE= 225.5, p, .001,ηp2= .6, and framing affected absolute accuracy,F(1, 54)= 8.9, MSE= 782.1, p= .004, ηp

2= .1

Figure 6. Mean JOL (judgement of learning), SOJ (second-order judgement), and recall performance across trials in Experiment 3. Forget-

framed JOLs were reverse scored. Error bars are the standard error of the mean.



Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

(Figure 6). Remember-framed JOLs were overcon-fident on Trial 1, F(1, 27)= 5.2, MSE= 267.3,p= .03, ηp

2= .2, and underconfident on Trial 2, F(1, 27)= 6.1, MSE= 299.3, p= .02, ηp

2= .2,whereas forget-framed JOLs were highly overconfi-dent on Trial 1, F(1, 27)= 84.3, MSE= 123.7,p, .001, ηp

2= .8, but accurate on Trial2. Framing did not interact with trial. We also con-sidered participants’ shift-to-underconfidencescores as in Experiments 1 and 2. These scoreswere not affected by framing.


2= .6. No other main effects or inter-actions were significant.

SOJ magnitude and distributionFigure 6 shows the mean SOJ for the two groups.SOJs did not differ by frame or by trial, and thesetwo factors did not interact. More importantly,we also considered the relationship between JOLand SOJ magnitudes on the two trials (Figure 7).Although the computer program (in all exper-iments) permitted participants to use any value

from 0 to 100 for their JOLs, participants mostoften responded with values that were multiplesof 10 (i.e., 30; 70). They also used multiples of 5(i.e., 25; 65) frequently, but rarely used othervalues (e.g., 22; 73). As such, we grouped JOLsinto 11 categories to more easily consider theSOJ–JOL relationship (i.e., 0, 10, 20, 30, 40, 50,60, 70, 80, 90, 100) by rounding all JOLs to thenearest multiple of 10 (e.g., a JOL of 38 becamea 40). On both trials, the SOJ–JOL relationshipfor the remember-framed group demonstrated a“checkmark” pattern as in earlier studies(Dunlosky et al., 2005; Serra et al., 2008). Thissuggests that remember-framed JOLs are psycho-logically anchored even lower than expected, ataround the 25% point on the JOL scale.Presumably, participants adjusted their JOL valuefor each item away from that point as they accumu-lated evidence that they would or would notremember each item on the test (cf. Dunloskyet al., 2005; Scheck & Nelson, 2005). In contrast,on both trials the SOJ–JOL relationship for theforget-framed group demonstrated a more sym-metric “u-shape” pattern. This suggests thatforget-framed JOLs are psychologically anchoredaround the 65% point on the JOL scale, and par-ticipants adjusted their JOL value for each item

Figure 7. Mean SOJ (second-order judgement) for each JOL (judgement of learning) block for both trials in Experiment 3. Forget-framed

JOLs were reverse scored. Error bars are the standard error of the mean.


SERRA AND ENGLAND

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

away from that point as they accumulated evidencethat they would or would not forget each item onthe test.4


2= .3. Relative accuracy washigher for remember-framed JOLs than forforget-framed JOLs, but the difference was not sig-nificant in this experiment (likely because the com-parison was greatly underpowered: observed powerwas .1). Although replicating the significant effectof framing on relative accuracy was not the focusof this experiment, the pattern for the mean relativeaccuracy correlations for Experiment 3 is neverthe-less completely in line with those in the otherexperiments (Table 2). Framing did not interactwith trial.

The mean MPT correlation was lower for theforget-framed group than for the remember-framed group (Table 2), but the difference wasnot significant in this experiment (again likelybecause the comparison was greatly underpowered:observed power was .2). Again, replicating the sig-nificant effect of framing on MPT utilization wasnot the focus of this experiment.

Discussion

Framing did not affect participants’ overall confi-dence in their JOLs (i.e., mean SOJs). Althoughthis suggests that participants might use the sameprocesses to make judgements of both remember-ing and forgetting, further consideration of thedata tells a more nuanced story. Differences inthe relationship between JOL and SOJ magnitudeson the two trials (Figure 7) suggest that the anchorpoint for forget-framed JOLs is at the midpoint ofthe JOL scale (around 50–70%) whereas the anchorpoint for remember-framed JOLs is at the lowerend of the JOL scale, around 20–40% (see alsoDunlosky et al., 2005; England & Serra, in press;Scheck & Nelson, 2005).

Although participants seem to make both typesof judgement using anchoring-and-adjustment pro-cesses, they probably start out at two different pointson the scale and might factor evidence into theirJOLs somewhat differently to adjust from there.The SOJ–JOL relationship for forget-framedJOLs seems to indicate that participants weighinformation relevant to either remembering or for-getting quite evenly (i.e., the symmetrical u-shaped shape), whereas the SOJ–JOL relationshipfor remember-framed JOLs seems to indicate thatparticipants mostly consider information relevantto remembering (i.e., the asymmetrical checkmarkshape). The latter relationship is similar to theoriesof how people make delayed JOLs that suggestthat participants first make a quick yes/no recalldecision and only then adjust their JOLs higher asthey accumulate evidence they will remember theitem later (e.g., Son & Metcalfe, 2005). Such pro-cesses would probably result in the checkmarkpattern seen in Figure 7 for immediate JOLs if par-ticipants first quickly made a yes/no decision: Therewould be little reason to consider the “no” itemsfurther, and hence they might confidently assignthem low JOLs such as a 0% or 10% (with relativelyhigh SOJs). For items that participants feel theywillrecall, however, more deliberate processes wouldthen take over to better consider the likelihood ofremembering each item, with higher JOLs receivinghigher SOJs. This would result in a positive linearrelationship between JOLs and SOJs as seen forremember-framed JOLs above 20%.

Framing aside, the present results further high-light the utility of SOJs for identifying aspects ofJOL data not immediately apparent from theJOLs alone (cf. Dunlosky et al., 2005). Forexample, if we had assumed that higher judgementsare naturally associated with higher confidence thenare lower judgements, we would have been forcedto conclude that participants in the forget-framedgroup were more confident in the accuracy oftheir JOLs than were participants in the remem-ber-framed group (at least based on the meanJOLs in Figure 6). The mean SOJs, however,

4 Note that although the reverse scoring of the forget-framed JOLs might have produced a “flipped” SOJ–JOL pattern for these

JOLs, the two halves of Figure 7 do not seem to be mirror images of each other.



Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

demonstrate that the two groups were equally con-fident in the accuracy of their judgements despitethe obvious differences in the magnitudes of theirJOLs across the two frames. More important forthe present purposes, if we had estimated theanchor of the remember-framed JOLs basedsolely on inspection of the data from Experiments1 and 2, we would probably have continued to over-estimate the anchor at around 35% (cf. Scheck &Nelson, 2005). Instead, the SOJ data suggest thatthe anchor might be even lower for these judge-ments (and that the anchor for forget-framedJOLs might be higher than 50%), a fact that isprobably normally masked by simply using themean JOL to estimate the anchor.

EXPERIMENT 4

So far, three data sets have demonstrated no differ-ences in JOL magnitude or absolute accuracy byframe (although immediate JOLs in Finn’s, 2008,Experiment 3 differed in magnitude by frame,absolute accuracy did not differ in that case): thedelayed JOLs in Finn (2008, Experiment 1B),the delayed JOLs in the present Experiment 1,and the immediate JOLs for large and small fontsizes in Rhodes and Castel (2008, Experiments 1and 5). So why didn’t these JOLs show magnitudeor absolute-accuracy differences across the twoframes? Both Finn and Rhodes and Castelsuggested that differences by frame did not obtainin their respective experiments because participantshad salient cues to use to make their JOLs regardlessof the framing. Finn (2008) suggested that “whendiagnostic information about whether the itemcan be retrieved from memory is available, as isthe case when one is making a delayed JOL,other cues do not play as large a role in makingthe judgement” (p. 819). It is widely acceptedthat people make delayed JOLs by attempting toretrieve the target word when presented with thecue word alone during a delayed JOL (e.g.,Nelson et al., 2004). This cue is presumably quitesalient to participants regardless of the framing ofthe JOL, which helps to explain why delayedJOLs do not demonstrate differences in magnitude

or absolute accuracy by frame (see also the delayedJOLs in the present Experiment 1 as well).Similarly, Rhodes and Castel’s (2008) font-sizemanipulation was presumably so salient that par-ticipants used it as a cue for their JOLs regardlessof the framing of the JOLs and regardless of thefact that font size was not predictive of later recall.

Given these past results and suppositions, inExperiment 4 we attempted to extend the lack of adifference by frame in terms of the magnitude andaccuracy of immediate JOLs when a memory cueis salient (cf. Rhodes & Castel, 2008) to a differentcue: image generation. JOLs are known to be nega-tively correlated with the latency to generate aninteractive image during the encoding of memoryitems (e.g., paired associates as in the present exper-iments; see Hertzog, Dunlosky, Robinson, &Kidder, 2003), so we predicted that participantswould naturally utilize this cue regardless of framewithout us explicitly prompting them to do so.

To examine this possibility, in Experiment 4 weadapted the basic procedure of the immediate-JOLgroups of Experiment 1 and added image-gener-ation instructions. Specifically, we instructed partici-pants to try to generate an interactive image for eachpaired associate they studied. They clicked icons onthe computer screen to indicate when they generatedan image (or if they did not). They then made animmediate JOL framed in terms of either remem-bering or forgetting. Although we did not instructparticipants to use the outcome or latency of theirimage generation as a cue for their JOLs, weassumed this procedure would make these cuessalient and that participants would factor the infor-mation into their JOLs as in previous studies (e.g.,Hertzog et al., 2003). As such, we predicted thatthe magnitude, absolute accuracy, and relative accu-racy of the JOLs would be equal for the two groupsbecause they would be using the same cue(s) for theirJOLs regardless of the frame.

Method

Participants, materials, and designThe participants were 74 undergraduates enrolled inGeneral Psychology at Texas Tech University whotook part in this experiment for course credit.


SERRA AND ENGLAND

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

None had participated in the earlier experiments.We used the same list of paired associates for thestudy materials as that used in earlier experiments.The type of framing (remember vs. forget) was theonly between-participants independent variable.

ProcedureThe procedure of Experiment 4 was identical to thatof the two immediate-JOL groups in Experiment 1except that in Experiment 4 we included image-generation instructions. Specifically, the computerprogram instructed participants to “try to think ofan image to relate the words of each pair”. If theycould come up with an image for a pair, participantswere to click an “Image Formed” icon on the screenas soon as they did; if they could not, participantswere to click an “Image Not Formed” icon on thescreen. The computer program recorded thelatencies of either selection. As such, study time inExperiment 4 was essentially participant paced.Participants engaged in image generation duringthe study phase of both trials in Experiment 4.

Results

RecallFigure 8 shows the mean recall for the two groups.Recall increased across the trials, F(1, 72)= 721.5,

MSE= 42.6, p, .001, ηp2= .9. Framing did not

affect recall, nor did frame interact with trial.

JOL magnitudeWe again reverse scored forget-framed JOLs (i.e.,100 – JOL) so we could directly compare them toremember-framed JOLs on the same scale. Figure8 shows the mean JOL for the two groups. JOLsincreased across the trials, F(1, 72)= 43.5,MSE= 109.5, p, .001, ηp

2= .4. As expected, themagnitude of forget-framed JOLs was not differentfrom that of remember-framed JOLs in this exper-iment. Framing did not interact with trial.

Absolute accuracyAbsolute accuracy changed across trials, F(1, 72)=73.2,MSE= 154.5, p, .001, ηp

2= .5, but framingdid not affect absolute accuracy. Both groups’ jud-gements were accurate on Trial 1, but both theremember-framed, F(1, 36)= 28.8, MSE=174.1, p, .001, ηp

2= .4, and the forget-framed,F(1, 36)= 18.7, MSE= 192.1, p, .001, ηp

2= .3,groups’ JOLs were highly underconfident on Trial2 (Figure 8). Framing did not interact with trial.We also considered participants’ shift-to-under-confidence scores as in the previous experiments.These scores were not affected by framing.

Figure 8. Mean JOL (judgement of learning) and recall performance across trials in Experiment 4. Forget-framed JOLs were reverse scored.

Error bars are the standard error of the mean.



Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015


2= .5. No other main effects or inter-actions were significant.

Image-generation dataWe examined whether the ability to generate animage for a pair of words or not was related toJOLs by calculating gamma correlations betweenimage-generation outcome and JOL magnitude(Table 3); these did not differ significantly byframe. For items for which participants were ableto successfully generate an image, we calculatedgamma correlations between the latency to generatean image and JOL magnitude (Table 3). Again,there were no significant differences by frame.


2= .3. Surprisingly—and despitethe fact that participants’ utilization of the image-generation information did not differ by frame—relative accuracy was higher for remember-framedJOLs than for forget-framed JOLs, F(1, 70)=5.5, MSE= 0.2, p= .02, ηp

2= .07. Follow-up ttests confirmed that relative accuracy was higher

for remember-framed JOLs than for forget-framed JOLs on Trial 2, t(70)= 2.5, p= .01, d=0.6, but did not differ on Trial 1. Framing didnot interact with trial. The mean MPT correlation(Table 2) was lower for the forget-framed groupthan for the remember-framed group, F(1, 71)=4.2, MSE= 0.2, p= .05, ηp

2= .06.

Discussion

As expected, including a salient cue for immediateJOLs in Experiment 4 eliminated differences in themagnitude and absolute accuracy of the JOLs byframe (see also Finn, 2008; Rhodes & Castel,2008). It seems that the magnitude and absoluteaccuracy of the immediate JOLs were equivalentacross frames in this experiment because the twogroups incorporated information about image gen-eration (success or latency) into their JOLs in thesame way regardless of the framing of the JOLs(Table 3). These outcomes mimic earlier studiesutilizing salient memory cues such as font size(Rhodes & Castel, 2008) and delayed JOLs(Finn, 2008).

Surprisingly, though, relative accuracy andMPT utilization still differed by frame inExperiment 4 (Table 2) even though other cueusage did not differ by frame (see Table 3; butnote that most of these other correlations were

Table 3. Image-generation data in Experiment 4

Remember frame Forget frame

Trial 1 Trial 2 Trial 1 Trial 2

Experiment Measure M SD M SD M SD M SD

Percentage of items for which images were successfully generated 77.9 17.6 84.0 18.4 83.7 13.7 84.6 21.6

Correlation of JOLs to whether images were successfully

generated

+.80 .36 +.82 .22 +.67 .56 +.60 .70

Mean JOL for items with successful image generation 49.7 18.2 59.2 17.2 48.7 15.8 61.0 24.1

Mean JOL for items without successful image generation 24.1 16.5 25.1 18.0 28.5 28.1 34.4 27.8

Image-generation latency

(in seconds)

5.7 2.4 3.0 0.9 5.7 2.7 3.2 1.4

Correlation of JOLs to image-generation latency –.05 .21 –.41 .23 –.05 .24 –.31 .38

Note: JOL= judgement of learning.


SERRA AND ENGLAND

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

numerically lower for the forget-framed group thanfor the remember-framed group as well). As in theprevious experiments, these results suggest that aforget frame can impair the relative accuracy ofJOLs and that a forget frame not only does nothelp participants to attune to predictive cuesabout their memory, but might actually causethem to discount or even neglect such cuesaltogether (as well as some nonpredictive cues).These results provide further evidence against theforgetting-notion hypothesis for the effects of aforget frame on the accuracy of JOLs.

GENERAL DISCUSSION

The purpose of the present experiments was tofurther examine the effects of framing JOLs interms of either remembering or forgetting on themagnitude, absolute accuracy, relative accuracy,and cue utilization of people’s JOLs. We soughtto determine whether a forget frame consistentlyproduces greater or lesser confidence in memorythan does a remember frame (cf. Finn, 2008;Koriat et al., 2004; Kornell & Bjork, 2009) andto compare the forgetting-notion hypothesis forthe magnitude and absolute accuracy of JOLs(Koriat et al., 2004; see also Finn, 2008) withour own hypothesis that the effect of framing onthe magnitude and absolute accuracy of JOLs islargely the result of anchoring differences byframe (cf. Scheck & Nelson, 2005). We also exam-ined whether the presence of salient memory cues(i.e., image generation or covert retrieval fordelayed JOLs) eliminates differences in the magni-tude and accuracy of JOLs by frame (cf. Finn,2008; Rhodes & Castel, 2008) and examinedwhether framing affects the relative accuracy andcue utilization of JOLs over multiple study–testtrials.

First, the present Experiments 1 through 3suggest that a forget frame consistently producesgreater confidence in memory than does a remem-ber frame. Although this outcome was in contrastto the results of Finn (2008), the same pattern ofresults was obtained by Kornell and Bjork (2009,Trial 1 predictions) and Koriat et al. (2004, 10-

minute retention predictions). Kornell and Bjork(2009) themselves recognized the discrepancy inJOL magnitudes between their studies and that ofFinn (2008), noting that:

[f]raming the predictions in terms of forgetting, in Experiment

7, appears to have globally increased participants’

predictions. . . . It is unclear why making predictions in terms

of forgetting rather than learning increased [predictions of learn-

ing]; based on previous research, one might expect a question

framed in terms of forgetting to make people more conservative,

not more confident. (Kornell & Bjork, 2009, p. 458).

One possible interpretation of the discrepancyacross studies is that the effect of framing on themagnitude of JOLs is simply inconsistent andunpredictable. Or, fine differences in methodologybetween the past and present studies that we havenot yet identified might have produced the differ-ences noted. Another possibility is that personalitydifferences existed between the samples used byFinn and those used in the studies exhibiting theopposite pattern (cf. Stankov & Lee, 2008).Regardless, the present data provide further docu-mentation that a forget frame most often producesgreater confidence in memory than does a remem-ber frame (Koriat et al., 2004; Kornell & Bjork,2009) and suggests that the pattern obtained byFinn (2008) might be the exception rather thanthe rule. Why exactly this pattern obtains isbeyond the scope of the present research, butsome other recent data from our lab might helpto inform this issue. England and Serra (in press)recently demonstrated that remember-framedJOLs seem to be anchored around the 35% pointon the JOL scale because participants expect thememory task to be difficult; when told the taskwill be easy, they anchor these JOLs around50%. As such, one possibility is that the forgetframe leads participants to believe that thememory task will be easier than does the rememberframe. Given that this possibility seems counterin-tuitive (cf. Kornell & Bjork, 2009), it certainlydeserves further empirical investigation.

Second—and most important for the presentpurposes—we considered the main tenet of the for-getting-notion hypothesis (Finn, 2008; Koriatet al., 2004), which is that forget-framed JOLsdemonstrate better absolute accuracy than do



Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

remember-framed JOLs because the former utilizemore predictive cues for memory than do the latter.Instead, we proposed that any difference in absoluteaccuracy by frame is actually an artefactual aspect ofdifferential anchoring by frame (cf. Scheck &Nelson, 2005), the stability bias in memory(Kornell & Bjork, 2009), and limits of the absol-ute-accuracy measure itself (cf. Scheck & Nelson,2005). In support of our hypothesis, we demon-strated that people’s judgements of rememberingand forgetting were both equally insensitive tonatural improvements in memory performanceacross study–test trials (Experiments 1 through 3)and when we purposefully manipulated memoryperformance between groups (Experiment 2).Increases in participants’ JOLs across trials didnot keep pace with increases in their memory per-formance in either frame (Table 1), and forget-framed JOLs were biased by a nonpredictive cueas equally as were remember-framed JOLs(Experiment 4). As such, we currently cannotendorse the forgetting-notion hypothesis that aforget frame yields JOLs that demonstrate betterabsolute accuracy than does a remember frame.

We suggest that future research that claims tosupport the improved-absolute-accuracy aspect ofthe forgetting-notion hypothesis must demonstratethat forget-framed JOLs are more sensitive tofactors that affect memory than are remember-framed JOLs across different levels of memory per-formance. Based on the present results, it seemsthat simply calculating absolute accuracy for onelevel of memory performance (e.g., on a singlestudy–test trial) is not a reliable way to demonstratethat one frame results in better absolute accuracythan does another frame. Although the anchoringexplanation for JOL magnitude turns absoluteaccuracy into a largely artefactual measure, absoluteaccuracy nevertheless seems to factor into partici-pants’ study-time allocation and restudy decisions(e.g., Finn, 2008; Metcalfe & Finn, 2008) andhence should not be completely abandoned as atopic of study. Towards this end, further examiningthe effects of framing on study choices might be amore fruitful avenue for research than is examiningthe effects of framing on the absolute accuracy ofJOLs. For example, if restudy choices are largely

determined by absolute accuracy, then given thepattern of absolute accuracy in the present exper-iments, had we asked our participants to makerestudy choices, we probably would have obtainedthe opposite outcome to Finn (2008), with partici-pants selecting more items for restudy in theremember-framed groups than in the forget-framed groups. If, however, restudy choices arelargely determined by a gain/loss orientation thatcan be affected by the frame of the JOLs (cf.Kahneman & Tversky, 1979), then we mighthave obtained the same outcomes as Finn (2008).Again, although this question is beyond the scopeof the present experiments, we do feel that it is aworthwhile question for future research.

Also related to the issues of JOL magnitude andabsolute accuracy, we demonstrated that peopleanchor immediate JOLs at different points on theJOL scale depending on the frame of the judge-ments. The results of Experiments 1 through 3suggest that people tend to anchor judgements offorgetting around the midpoint of the scale (i.e.,around 50%, but perhaps as high as 70%),whereas they tend to anchor judgements of remem-bering around a lower point on the scale (i.e.,around 30%, with a range between 20% to 40%;see also England & Serra, in press; Scheck et al.,2004; Scheck & Nelson, 2005). Furthermore, theSOJ results of Experiment 3 suggest that peoplemight make judgements of remembering and for-getting somewhat differently. When people makejudgements of forgetting, they seem to start at apsychological anchor point in the middle of thescale and adjust their JOLs up or down as theyaccumulate evidence that they will remember orforget the current item. This results in a U-shaped SOJ–JOL relationship. In contrast, whenpeople make judgements of remembering, theyseem to start at a psychological anchor pointlower on the scale and first decide whether theywill or will not remember the item. A decisionthat they will not remember the item results in alow JOL (i.e., 0% or 10%) with moderate confi-dence; a decision that they will remember theitem leads them to further adjust their JOLshigher—and with greater confidence—as theyaccumulate evidence that they will remember the


SERRA AND ENGLAND

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

item. This results in a checkmark-shaped SOJ–JOL relationship (see also Dunlosky et al., 2005;Serra et al., 2008).

In contrast to immediate JOLs, which were themain focus of the present experiments, it is worthnoting that delayed JOLs do not seem to demon-strate framing effects (the present Experiment 1;Finn, 2008) and do not seem to be influenced byanchoring. Most likely, both of these outcomesobtain because participants use the outcome ofsuch covert retrieval attempts to inform theirJOLs, and this produces extremely polarizedJOLs (i.e., very high and very low JOL values).As such, the magnitude and accuracy of theseJOLs are not affected by framing, and accuracy—both absolute and relative—is quite good (cf.Nelson et al., 2004).

Third, we examined whether the presence ofsalient memory cues eliminated differences in themagnitude and accuracy of JOLs by frame (cf.Finn, 2008; Rhodes & Castel, 2008; see alsoTauber & Rhodes, in press-a). In the presentexperiments, both delayed JOLs (Experiment 1)and immediate JOLs made after participants gener-ated an interactive image for each paired associate(Experiment 4) demonstrated equal magnitudeand absolute accuracy regardless of how weframed the JOLs. Although delayed JOLs exhib-ited equivalent relative accuracy by frame, immedi-ate remember-framed JOLs continued todemonstrate superior relative accuracy to forget-framed JOLs even when other apparently salientcues were available that were either predictive ofmemory performance (i.e., MPT information) ornot (i.e., cues related to image generation). Assuch, although the present results confirm theidea that some cues are so salient to participantsthat they will utilize them regardless of how theJOLs are framed, these data further disconfirmthat the forget frame attunes participants to focuson and utilize more predictive cues than does theremember frame.

Finally, relative accuracy was consistently betterfor immediate JOLs framed in terms of remember-ing than for those framed in terms of forgetting.Although this difference was never significantwithin any single experiment on the first study–test

trial (Table 2; cf. Finn, 2008), combining the Trial1 relative-accuracy correlations for immediate JOLsacross the four experiments yielded a small—andonly marginally significant—advantage for remem-ber-framed JOLs over forget-framed JOLs, F(1,320)= 3.0, MSE= 0.1, p= .09, ηp

2= .009(Cohen’s d= 0.16). More notably, relative accuracyon Trial 2 was consistently better for remember-framed JOLs than for forget-framed JOLs, bothwithin each experiment (Table 2), and when datafrom all four experiments were combined, F(1,320)= 20.0, MSE= 0.2, p, .001, ηp

2= .06(Cohen’s d= 0.49). The Trial 2 advantage ofremember-framed JOLs over forget-framed JOLsseemed to stem from participants’ greater use ofpast test information (MPT; see Table 2) toinform the former type of judgement (cf. Ariel &Dunlosky, 2011; England & Serra, in press; Finn& Metcalfe, 2007, 2008; King et al., 1980; Tauber& Rhodes, in press-b). MPT use was greater forthe remember-framed groups than for the forget-framed groups both within each experiment (Table2) and when data from all four experiments werecombined, F(1, 320)= 16.9, MSE= 0.2,p, .001, ηp

2= .05 (Cohen’s d= 0.46). Most impor-tant for the present purposes, this outcome contra-dicts the main tenet of the forgetting-notionhypothesis in that not only does a forget frame notseem to direct participants to use more predictivecues for memory than does the remember frame,but it might actually cause them to completely dis-count this highly predictive cue when makingJOLs. Of course, MPT information is not the onlycue that people consult to make JOLs across trials(Ariel & Dunlosky, 2011; England & Serra, inpress; Finn & Metcalfe, 2008; Tauber & Rhodes,in press-b) so future research should examinewhether people’s use of other information such asnew learning or new forgetting (cf. Ariel &Dunlosky, 2011) changes based on frame as well.

These findings not only suggest that forget-framed JOLs might be a poor choice to use inapplied learning situations where maximizing therelative accuracy of students’ JOLs is of tantamountimportance (cf. Serra &Metcalfe, 2009), but are alsoimportant for helping us to better understand howpeople make JOLs in general (e.g., Koriat, 1997).



Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

For example, current theories of how people makeJOLs (e.g., Koriat, 2008) assume a bottom-up selec-tion of cues for memory based on the utility of thosecues (e.g., how predictive those cues have been formemory in the past). If people select and utilizecues for memory via bottom-up processes,however, then framing should never affect JOLs,because the cues (and their previously experiencedutility) would always be the same. Instead, thepresent results suggest that cue selection is a moretop-down process such that the framing of theprompt for the judgement can change which cuespeople select and how people make their judge-ments, regardless of the utility of the cues.

In summary, the present results add to the small—but growing—literature examining the effect offraming on metacognitive judgements. This is apromising research area that can greatly inform the-ories of how people make metacognitive judge-ments, even if the effects of different frames onsuch judgements do not actually yield conclusionsor outcomes (i.e., changes in monitoring accuracy)that are directly applicable in the classroom. Thatsaid, future research should continue to examinethe effects of framing on restudy decisions (cf.Finn, 2008), which might have important impli-cations for applied situations.

Original manuscript received 16 June 2011

Accepted revision received 27 March 2012

First published online 30 May 2012

REFERENCES

Ariel, R., & Dunlosky, J. (2011). The sensitivity of judg-ment-of-learning resolution to past test performance,new learning, and forgetting. Memory & Cognition,39, 171–184. doi:10.3758/s13421-010-0002-y

Briñol, P., Petty, R. E., & Tormala, Z. L. (2006). Themeaning of subjective ease and its malleability.Psychological Science, 17, 200–206.

Dunlosky, J., & Metcalfe, J. (2009). Metacognition.Thousand Oaks, CA: Sage Publications.

Dunlosky, J., Serra, M. J., Matvey, G., & Rawson, K. A.(2005). Second-order judgements about judgementsof learning [Special issue]. Journal of General

Psychology, 132, 335–346.

England, B. D., & Serra, M. J. (in press). The contri-bution of anchoring and past-test performance tothe underconfidence-with-practice effect. Psycho-

nomic Bulletin & Review. doi: 10.3758/s13423-012-0237-7

Finn, B. (2008). Framing effects on metacognitive moni-toring and control. Memory & Cognition, 36,

813–821. doi:10.3758/MC.36.4.813Finn, B., & Metcalfe, J. (2007). The role of memory for

past test in the underconfidence with practice effect.Journal of Experimental Psychology: Learning,

Memory, and Cognition, 33, 238–244. doi: 10.1037/0278-7393.33.1.238

Finn, B., & Metcalfe, J. (2008). Judgments of learn-ing are influenced by memory for past test. Journalof Memory and Language, 58, 19–34. doi: 10.1016/j.jml.2007.03.006

Halamish, V., McGillivray, S., & Castel, A. D. (2011).Monitoring one’s own forgetting in younger andolder adults. Psychology and Aging, 26, 631–635.doi:10.1037/a0022852

Hertzog, C., Dunlosky, J., Robinson, A. E., & Kidder,D. P. (2003). Encoding fluency is a cue used for jud-gements about learning. Journal of Experimental

Psychology: Learning, Memory & Cognition, 29,

22–34. doi:10.1037/0278-7393.29.1.22Kahneman, D., & Tversky, A. (1979). Choices, values,

and frames. American Psychologist, 39, 341–350.Kahneman, D., & Tversky, A. (1984). Prospect theory:

An analysis of decision under risk. Econometrica, 47,263–291.

King, J. F., Zechmeister, E. B., & Shaughnessy, J. J.(1980). Judgements of knowing: The influence ofretrieval practice. American Journal of Psychology, 93,329–343.

Koriat, A. (1997). Monitoring one’s own knowledgeduring study: A cue-utilisation approach to judge-ments of learning. Journal of Experimental

Psychology: General, 126, 349–370.Koriat, A. (2008). Easy comes, easy goes? The link

between learning and remembering and its exploita-tion in metacognition. Memory & Cognition, 36,

416–428. doi:10.3758/MC.36.2.416Koriat, A., Bjork, R. A., Sheffer, L., & Bar, S. K. (2004).

Predicting one’s own forgetting: The role of experi-ence-based and theory-based processes. Journal of

Experimental Psychology: General, 133, 643–656.doi:10.1037/0096-3445.133.4.643

Koriat,A., Lichtenstein, S.,&Fischhoff,B. (1980).Reasonsfor confidence. Journal of Experimental Psychology:

Human Learning and Memory, 6, 107–118.


SERRA AND ENGLAND

Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

Kornell,N.,&Bjork,R.A. (2009).A stability bias in humanmemory: Overestimating remembering and underesti-mating learning. Journal of Experimental Psychology:

General, 138, 449–468. doi:10.1037/a0017350McCaslin, M. J., Petty, R. E., &Wegener, D. T. (2010).

Self-enhancement and theory-based correction pro-cesses. Journal of Experimental Social Psychology, 46,830–835.

Meeter, M., & Nelson, T. O. (2003). Multiple studytrials and judgements of learning. Acta Psychologica,113, 123–132.

Metcalfe, J., & Finn, B. (2008). Evidence that judge-ments of learning are directly related to studychoice. Psychonomic Bulletin Review, 15, 174–179.doi:10.3758/PBR.15.1.174

Miller, T. M., & Geraci, L. (2011). Unskilled but aware:Reinterpreting overconfidence in low-performing stu-dents. Journal of Experimental Psychology: Learning,

Memory & Cognition, 37, 502–506. doi:10.1037/a0021802

Nelson, T. O., & Narens, L. (1994). Why investigatemetacognition? In J. Metcalfe & A. Shimamura(Eds.), Metacognition: Knowing about knowing

(pp. 1–25). Cambridge, MA: The MIT Press.Nelson, T. O., Narens, L., & Dunlosky, J. (2004). A

revised methodology for research on metamemory:Pre-judgement recall and monitoring (PRAM).Psychological Methods, 9, 53–69. doi:10.1037/1082-989X.9.1.53

Rhodes, M. G., & Castel, A. D. (2008). Memory predic-tions are influenced by perceptual information:Evidence for metacognitive illusions. Journal of

Experimental Psychology: General, 137, 615–625.doi:10.1037/a0013684

Rhodes, M. G., & Tauber, S. K. (2011). The influence ofdelaying judgements of learning on metacognitiveaccuracy: A meta-analytic review. Psychological

Bulletin, 137, 131–148.Scheck, P., Meeter, M., & Nelson, T. O. (2004).

Anchoring effects in the absolute accuracy of

immediate versus delayed judgements of learning.Journal of Memory and Language, 51, 71–79.doi:10.1016/j.jml.2004.03.004

Scheck, P., & Nelson, T. O. (2005). Lack of pervasive-ness of the underconfidence-with-practice effect:Boundary conditions and an explanation via anchor-ing. Journal of Experimental Psychology: General, 134,124–128. doi:10.1037/0096-3445.134.1.124

Schwarz, N. (2004). Metacognitive experiences in consu-mer judgement and decision making. Journal

of Consumer Psychology, 14, 332–348.Serra, M. J., Dunlosky, J., & Hertzog, C. (2008). Do

older adults show less confidence in their monitoringof learning? Experimental Aging Research, 34,

379–391. doi:10.1080/03610730802271898Serra, M. J., & Metcalfe, J. (2009). Effective

implementation of metacognition. In D. J. Hacker,J. Dunlosky, & A. C. Graesser (Eds.), Handbook of

metacognition in education (pp. 278–298). New York,NY: Routledge.

Son, L. K., & Metcalfe, J. (2005). Judgements of learn-ing: Evidence for a two-stage process. Memory &

Cognition, 33, 1116–1129.Stankov, L., & Lee, J. (2008). Confidence and cognitive

test performance. Journal of Educational Psychology,100, 961–976. doi:10.1037/a0012546

Tauber, S. K., & Rhodes, M. G. (in press-a). Measuringmemory monitoring with judgements of retentioninterval (JOR). Quarterly Journal of Experimental

Psychology, Advance online publication. doi:10.1080/17470218.2012.656665

Tauber, S. K., & Rhodes, M. G. (in press-b). Multiplebases for young and older adults’ judgements-of-learning (JOLs) in multitrial learning. Psychology andAging, Advance online publication. doi:10.1037/a0025246

Thiede, K. W., Anderson, M. C. M., & Therriault, D.(2003). Accuracy of metacognitive monitoring affectslearning of texts. Journal of Educational Psychology,95, 66–73. doi:10.1037/0022-0663.95.1.66



Dow

nloa

ded

by [

Tex

as T

ech

Uni

vers

ity L

ibra

ries

] at

12:

12 1

6 O

ctob

er 2

015

magnitude and accuracy differences between judgements of ...webpages.acs.ttu.edu/mserra/serra...

Documents