accuracy of feeling-of-knowing judgments for predicting...

Journal of Experimental Psychology: General Copyright 1984 by the1984, Vol. 113, No. 2, 282-300 American Psychological Association, Inc.

Accuracy of Feeling-of-Knowing Judgments for PredictingPerceptual Identification and Relearning

Thomas O. Nelson and Daniel GerlerUniversity of Washington

Louis NarensUniversity of California, Irvine

SUMMARY

The feeling of knowing refers to predictions about subsequent memory performanceon previously nonrecalled items. The most frequently investigated type of subsequentperformance has been recognition. The present research explored predictive accuracywith two new feeling-of-knowing criterion tests (in addition to recognition): relearningand perceptual identification. In two experiments, people attempted to recall the answersto general-information questions such as, "What is the capital of Australia?", then madefeeling-of-knowing predictions for all nonrecalled answers, and finally had a criterion testto assess the accuracy of the feeling-of-knowing predictions. Experiment 1 demonstratedthat perceptual identification can be employed successfully as a criterion test for thefeeling of knowing. This opens a new way for metamemory research via perception.Moreover, the feeling-of-knowing accuracy for predicting perceptual identification wasnot significantly correlated with the feeling-of-knowing accuracy for predicting recognition,in accord with the idea that these two tests assess memory differently. Experiment 2demonstrated that relearning performance can also be predicted by feeling-of-knowingjudgments.

Both experiments showed that there is a positive relationship between the feeling ofknowing and the amount of time elapsing before a memory search is terminated duringrecall. Further analyses showed that this relationship is substantial for nonrecalled itemsfor which the person did not guess an answer (omission errors), but the relationship isnull or negative for nonrecalled items that the person guessed incorrectly (commissionerrors).

Several theoretical mechanisms that may underlie the feeling of knowing are proposed.

The first empirical research on the feeling tern?" the alternatives were Pluto, Venus,of knowing was the recall-judgment-recog- Earth, and Jupiter). A recognition test is usefulnition study by Hart (1965a). First, the subject as a criterion test because it generally is moreattempted to recall the answers to general-in- sensitive than recall for assessing memory (information questions (e.g., "Which planet is the sense of differentiating between items thatthe largest in our solar system?"). This served the recall test treats homogeneously in termsto provide a subset of questions for which the of all being nonrecalled), and thus it may besubject did not recall the correct answers. Next, correlated with the feeling of knowing, whichthe subject made feeling-of-knowing judg- presumably is also more sensitive than recallments on these questions by predicting whether (Hart, 1967a).he or she would be able to recognize the correct This paradigm, or modifications of it, hasanswers (hereinafter, "he or she" is referred to been used in dozens of feeling-of-knowing ex-more briefly as "he"). Finally, to assess the periments. Various subject populations-haveaccuracy of the feeling-of-knowing judgments, been examined, including psychology under-Hart gave a multiple-choice recognition test graduates (Hart, 1965a), children (Wellman,on each question (e.g., for the question, 1977), the mentally retarded (Brown & Law-"Which planet is the largest in our solar sys- ton, 1977), geography undergraduates (Gru-

282

FEELING OF KNOWING 283

neberg & Monks, 1974), and older adults(Lachman, Lachman, & Thronesbery, 1979).Various kinds of items have been examined,including general-information questions (Hart,1965a), laboratory paired associates (Hart,1967a), nonsense syllables (Blake, 1973), labelsfor pictured objects (Wellman, 1977), namesof famous people (Gruneberg & Sykes, 1978),definitions of words (Gardiner, Craik, &Bleasdale, 1973), names of entertainers (Read& Bruce, 1982), capitals of countries (Gru-neberg & Monks, 1974), and names of personalacquaintances (Gruneberg, Smith, & Winfrow,1973).

The standard finding is that the accuracyof the feeling of knowing for predicting sub-sequent recognition on nonrecalled items isintermediate, being reliably above chance butfar from perfect (Blake, 1973). One reason toexpect imperfect accuracy is that the criteriontest of recognition is "noisy" in at least twoways:

1. Although the difficulty of a recognitiontest increases with the degree of target-dis-tractor similarity (Bernbach, 1967), functional(as opposed to nominal) target-distractor sim-ilarity is probably a multidimensional con-struct for which some of the potential dimen-sions may be unknown. Also, the perceived"similarity" is likely to be idiosyncratic andto vary across subjects. Thus, it is possible thatsometimes the distractors will be good (in thesense of being similar to the target) for an itemjudged high in the feeling of knowing and badfor an item judged low in the feeling of know-ing and sometimes vice versa. Moreover, cor-rect recognition can occur either because thesubject recognizes the target or because thesubject eliminates the distractors. The feeling

Portions of these results were presented at the annualmeeting of the Psychonomic Society in Minneapolis, Min-nesota, 1982. This research was supported by NationalInstitute of Mental Health Grant MH-3220S.

Thanks go to A. Shimamura for his computer pro-gramming and other contributions to early phases of theproject; to R. Landwehr and A. Singh for their computerprogramming; to M, McCloskey and M. Watkins for theircomments on the manuscript; and to J. Leonesio, J. Raley,A. Weisman, and D. Malcolm for their help in the datacollection and data analysis.

Requests for reprints should be sent to Thomas O. Nel-son, Psychology Department (NI-25), University of Wash-ington, Seattle, Washington 98195.

of knowing is conceptually related only to theformer; the latter is a source of noise.

2. The practical limit on the number ofdistractors per multiple-choice item necessarilyproduces still more noise due to guessing. Forinstance, even if an item judged low in thefeeling of knowing is not known at all, thesubject may fortuitously choose the correctanswer. This yields an even greater reductionin the observed accuracy of the feeling ofknowing.

The present research explored the relationbetween the feeling of knowing and criteriontests that do not require distractors. Also, be-cause a major goal of our research programis to determine the overall relation betweenthe feeling of knowing (as an instance of meta-memory; see Flavell & Wellman, 1977) andthe target-memory system, a natural researchstrategy is to substitute criterion-memory teststhat reflect processes different from those re-flected by recognition tests or other previouslyexamined criterion tests.

CRITERION TESTS FORFEELING-OF-KNOWING RESEARCH

Three criterion tests have been used in pre-vious studies: (a) Most studies used recognitiontests (e.g., Hart, 1965a); (b) a few used cued-recall tests in which the first letter of the answeris provided by the experimenter (Freedman &Landauer, 1966; Gruneberg & Monks, 1974;Gruneberg, Monks, & Sykes, 1977); and (c) afew have examined reminiscence during ad-ditional uncued recall tests (Gardiner et al.,1973; Gruneberg et al., 1973; Gruneberg &Sykes, 1978; Hart, 1967b; Read & Bruce,1982). As Schacter (1983) recently pointed out:It would be desirable to explore the relation of the feelingof knowing to modes of expressing knowledge other thanrecall and recognition. . . , Future studies that attemptto elucidate the nature of and relations between the variousmodes of expressing knowledge are likely to sharpen ourinsight into many facets of human memory, (p. S3)

To select new criterion tests for the feelingof knowing, it is helpful to consider how dif-ferent tests might assess memory:

A forced-choice recognition memory test isessentially a test ofdiscriminatingbetween thetarget and the distractors (Bernbach, 1967).

A cued-recall test in which the experimenterprovides some letters of the target is a test of

284 T. NELSON, D. GERLER, AND L. NARENS

redintegration of the whole by one or more ofits parts (Horowitz & Prytulak, 1969). It canbe conceptualized in terms of successivelymore specific cues for recall because the ex-perimenter initially provides no letters of thetarget, then provides the first letter, and perhapsthen provides the first two letters (Gruneberg& Monks, 1974).

Reminiscence across a second uncued-recalltest is less well understood but may be a si-multaneous test of an item's variability com-bined with its initial closeness to the thresholdof recall. Still more speculatively, perhaps thisvariability is related to the distribution of"memory strength" for the item, where mem-ory strength is conceptualized as a hypotheticalunidimensional entity (cf. Eich, 1982; Hayes-Roth, 1977; Hull, 1943; Ratcliff, 1978; Wick-elgren, 1977) or as one dimension of an un-derlying multidimensional entity.

The two new criterion tests for the feelingof knowing explored here—perceptual iden-tification and relearning—can be consideredconcatenation-like tests in which the target it-self is presented and so can add to whateveris already in memory about the target (Bahrick,1967; Nelson, 1978).' The perceptual-identi-fication test consists of successively longer ta-chistoscopic exposures that are assumed to addsmall increments to whatever is already inmemory about the target; this test employs animmediate test of identification that occurs assoon as each exposure terminates. The re-learning test differs from the perceptual-iden-tification test in at least three ways: Relativeto the perceptual-identification test, the re-learning test (a) employs a longer retentioninterval between the presentation and test ofa given item, (b) is thought to add larger in-crements to whatever is already in memoryabout the target (because of the greater pre-sentation time relative to the perceptual-iden-tification test), and (c) presents all of the itemsfor study before any item is tested. Aside fromthese differences, the perceptual-identificationtest and the relearning test assess memory inthe same basic way, via a concatenation-likemechanism that is fundamentally differentfrom the mechanisms of the previous feeling-of:knowing criterion tests.

What hypotheses should be proposed aboutthe relations between the feeling of knowingand relearning arid between the feeling of

knowing and perceptual identification? Be-cause relearning is more sensitive than rec-ognition (Nelson, 1978; replicated in Gron-inger & Groninger, 1980) and because rec-ognition is correlated with the feeling ofknowing, the feeling of knowing should becorrelated with relearning. For the relation be-tween the feeling of knowing and perceptualidentification, an a priori hypothesis is notproposed because (a) the relative sensitivitiesof perceptual identification and recognition areunclear from the previous literature—conse-quently, perceptual identification is not nec-essarily expected to be successful as a criteriontest for the feeling of knowing—and (b) somecritical aspects of the interpretation of per-ceptual identification are highly controversial(discussed next).

EXPERIMENT 1:PERCEPTUAL IDENTIFICATION

We had two reasons for exploring perceptualidentification as a criterion testifor the feelingof knowing. First, its concatenation-like prop-erties were of interest. Second, perceptualidentification has been suggested by previousinvestigators (Hart, 1965b, p. 73; Schacter,1983, p. 53) as a potentially useful criteriontest for the feeling of knowing.

Our paradigm built on previous findingsthat prior exposure to words enhances sub-sequent perceptual identification of thosewords (Jacoby & Dallas, 198-1; Murrell &Morton, 1974; Neisser, 1954). In our experi-ment, however, any prior exposure to the tar-gets came from the subject's preexperimentalhistory. Whether those prior exposures willenhance perceptual identification depends on

1 Concatenation refers to linking one thing together withanother, such as when a 2-g weight is added to a 3-g weightto produce a 5-g weight (Coombs, Dawes, & Tversky, 1970,p. 8; Krantz, Luce, Suppes, & Tversky, 4971). We use theterm concatenation-like'.because the process cannot stricth/be said to be concatenation because of nonadditivity onthe observed performance scale (e.g., Haber & Hershenson,1965). However, concatenation may be; occurring on theunderlying strength dimension and not be evident in per-formance because the relation between strength and per-formance is nonlinear. Until more is known about strengthand how it relates to performance, the term concatenation-like seems more appropriate for the supposition of thetarget's presentation being linked together with whateveris already in memory about the target.


the type of information to which perceptualidentification is sensitive, and this currently iscontroversial (e.g., see Jacoby & Dallas, 1981,p. 306, for one view; for another view, seeAllport, 1977; Fowler, Wolford, Slade, & Tas-sinary, 1981, p. 345; Marcel, 1983). Moreover,even if perceptual identification uncovers someprior information in memory about nonre-called items, the feeling of knowing may nottap that particular information (see Fowler etal, 1981, p. 345; but also see Merikle, 1982).

MethodDesign and Subjects

A within-subjects design was employed. The criteriontest for each of nine nonrecalled items was a perceptual-identification test for which the correct answer was re-peatedly presented tachistoscopically; for a given item, theexposure time was increased until the subject identifiedthe correct answer (cf. "method of ascending limits" inWinnick & Daniel, 1970). For nine other items, the cri-terion test was a four-alternative-forced-choice (four-AFC)recognition test. The subjects were 32 University of Wash-ington undergraduates who participated for course credit.They were fluent in English and had normal or corrected-to-normal/vision.

ProcedureThe procedure consisted of four discrete phases. First,

the increment time and start time were established for theeventual perceptual-identification test (preliminary phase).Second, a recall test was given for general-informationitems until the subject incorrectly answered 21 questions.Third, the subject made relative feeling-of-knowing judg-ments on those 21 incorrect items. Fourth, the two criteriontests assessed the accuracy of the subject's feeling-of-knowing judgments.

Preliminary PhaseThe high variability in subjects' thresholds (Marcel,

1983) necessitated that the start and increment times bedetermined individually for each subject. The algorithmfor this is reported elsewhere (Nelson, Gerler, & Narens,in press). It yields a start time and increment time suchthat (a) the start time is not so slow that the subject tendsto identify the answer on the first exposure, and (b) thestart time is not so fast and the increment time is not sosmall that-the subject tends not to identify the answereven after 10 incremented exposures. Thus, the start andincrement times were determined for each subject so asto have sufficient fineness to yield perceptual identificationon an intermediate number of exposures (i.e., neither aceiling effect nor a floor effect). These start and incrementtimes were used later during the perceptual-identificationphase.

Recall PhaseThe items were 145 general-information questions from

the 240 that are available in the FACTRETRIEVAL computerprogram (Shimamura, Landwehr, & Nelson, 1981) andthat originally came from the Nelson and Narens norms(1980b). These questions span a wide variety of topicsthat include history, sports, art, geography, science, lit-erature, and entertainment. The 145 selected items eachhad an answer that was five to seven letters in length (e.g.,"What is the name of Socrates' most famous student?"Answer = Plato).

The questions were presented in a different randomorder to each subject. Each question was presented indi-vidually on a video monitor, and the subject typed his orher response on the keyboard of an Apple II computer.The subject was asked to search "memory hard in anattempt to find the answer." If the subject had no guessat all concerning the answer to a particular question thenhe typed "next." Response latencies were recorded for allresponses. To determine correctness, the computer ex-amined only the first three letters of the subject's response.This yielded a purer pool of nonrecalled items by notinsisting on perfect spelling, After the subject missed theanswer to 21 questions (9 of which eventually were usedin the recognition test, 9 of which eventually were usedin the perceptual-identification test, and 3 of which werecatch-trial items as discussed later), the computer generateda printout so that slides of the correct answers could beselected for the eventual perceptual-identification phase.

Feeling-of-Knowing PhaseSubjects made feeling-of-knowing judgments on the 21

incorrectly answered items. Three questions at a time weredisplayed on the video monitor. Subjects were instructedto indicate their relative feeling of knowing by selectingwhichever of those 3 questions they believed they weremost likely to recognize the answer to. Then that questiondisappeared from the screen, and subjects made a relativefeeling-of-knowing judgment for the remaining 2 questionsby selecting whichever question they believed they weremore likely to recognize the answer to. These feeling-of-knowing judgments occurred/or every triad of items. Theparticular triads presented to each subject consisted of arandom choice of all possible triads with the restrictionthat every possible unordered pair from the 21 items ap-peared exactly once. For N items, this yields N(N - l)/6triads, or a total of 70 triads for the 21 items (Burton &Nerlove, 1976). This procedure yields a feeling-of-knowingrank order (with the possibility of tied ranks) in terms ofhow many times a given item is chosen over all other items(Nelson & Narens, 1980a), where the item chosen mostfrequently is assigned the highest feeling-of-knowing rank.To minimize an inappropriately high feeling of knowingfor items that the subject had guessed incorrectly duringthe recall phase, the subject was informed that all of theitems in the feeling-of-knowing judgments either had aresponse of "next" during the recall phase or had beenanswered incorrectly (Krinsky & Nelson, in press).

Criterion-Test PhaseThis consisted of the perceptual-identification test and

the four-AFC recognition test. Excluding the top, bottom,


Table 1Mean and 95% Confidence Interval for Each Measure in Experiment 1

Measure

^(recall)/"(recognition)Perceptual-identification start timePerceptual-identification increment timeMedian number of exposures/"(correct on perceptual identification)Gamma: Feeling of knowing and recognitionGamma: Feeling of knowing and number of exposuresGamma: £(Incorrect recall) and recognitionGamma: L(Incorrect recall) and number of exposuresGamma: ZXIncorrect recall) and feeling of knowing

M

.41

.4927.4 ms4.1 ms5.8.80.29*

-.16*.10

-.07.17*

95% Confidenceinterval

.35 <-> .46

.43 *•» .5523.8 «-» 30.9

3.6 <-* 4.65.1 «-» 6.6.75 <-» .86.15 «-» .43

-.04 «-» -.27-.06 <-» .25-.19 <-» .04

.07 «-> .26

Note. Each measure is based on the scores from 32 subjects.* p < .05, correlatipn is reliably different from zero.

and middle items in the feeling-of-knowing rank order,the remaining 18 items were divided into two sets: theitems with even-numbered feeling-of-knowing rank po-sitions (i.e., 2,4,6, etc.) and the items with odd-numberedfeeling-of-knowing rank positions. The perceptual-iden-tification test occurred on one set of items, whereas therecognition test occurred on the other set. Whether theperceptual-identification test occurred on the even-num-bered versus odd-numbered items was counterbalancedacross subjects. Order of the perceptual-identification testand the recognition test was also counterbalanced acrosssubjects. The procedure for these criterion tests was asfollows.

Perceptual identification. Two display screens wereused. One screen was a video monitor for the nontachisto-scopic aspects of the experiment. Adjacent to the videomonitor was the back-projection screen for the displayfrom a tachistoscopic slide projector that was controlledby the computer.

The question portion of each item appeared individuallyon the video monitor and remained there until that itemwas no longer tested. The subject was instructed to lookat the tachistoscopic screen and to press the space barwhen ready to view the answer portion of the item. Im-mediately after the exposure, a second slide projector dis-played for 256 ms a visual-noise mask consisting of ran-domly overlapping letters. Then the subject made a bestguess at the correct answer. He was told not to be concernedwith any guesses during previous exposures but insteadto focus on the correct answer for the current exposure;this was done to discourage a strategy of repeating thesame answer. The subject was also informed that somequestions might have to be answered correctly more thanonce; this discouraged a strategy of guessing a differentanswer after each exposure. One practice item was pre-sented at the start time and increment time establishedduring the preliminary phase. The answer to this itemwas presented either until the subject identified it correctlytwice or until 10 exposures had elapsed (whichever camefirst). The computer then advanced to the test items forwhich each answer was presented until it was identifiedcorrectly once or until 10 exposures had elapsed, where-

upon the computer advanced to the next item. Additionaldetails of the perceptual-identification test are reported inNelson et al. (in press).

The test items also contained three other items for catchtrials. These consisted of questions for which the correctanswer was never presented. These were the top, middle,and bottom items in the feeling-of-knowing rank order.They were utilized to assess the possibility that some itemsmight be identified correctly even without the correct an-swer being presented, due either to reminiscence or to astrategy of switching guesses across the trials on a givenitem. The slides that replaced the correct answers for thecatch-trial items contained seven nonletter characters fromthe top of the typewriter keyboard (e.g., ampersand, percentsign, etc.). These catch-trial items occupied the 4th, 7th,and 11th positions in the perceptual-identification pre-sentation order. The order of the items in these 3 catch-trial positions was counterbalanced across subjects (theremaining 9 perceptual-identification presentation posi-tions were randomly assigned to the nine items that hadthe correct answers presented). The exposure duration foreach of the three catch trials was 5 ms on the first exposure,with a 1-ms increment on each subsequent exposure. Thusthe longest possible exposure for a catch trial was 14 ms.This short exposure duration was used so the subject wouldnot realize that no correct answer was being presented forthe catch-trial question.

Recognition. The recognition test was self-paced, andthe order of the nine items was random. For each item,the video monitor displayed the question and a randomordering of the four alternatives. The distractors were fromthe same category as the correct answer (e.g., for the ques-tion, "What is the last name of the inventor of the wirelessradio?", the alternatives were Marconi, Morse, Bell, andFaraday) and came from the recognition phase of FACT-RETRIEVAL (Shimamura et al., 1981).

Results and Discussion

The primary results are summarized in Ta-ble 1. The first column shows the 11 measures


.9

''

.5

0

A PRESENTEDo NONPRESENTED

ALLPRESENTED/*

2 3 4 5 6 7 8 9 1 0 1 2 3 4 5 6 7 8 9 1 0PERCEPTUAL-IDENTIFICATION TRIAL

Figure 1. Mean cumulative /"(correct) on each perceptual-identification trial for presented items and non-presented items. (Left panel shows performance on all items. Right panel shows performance on highestand lowest feeling-of-knowing presented items and on highest, middlemost, and lowest feeling-of-knowingnonpresented items.)

obtained from each subject, the second columnshows the mean value of the 32 individual-subject values, and the third column showsthe 95% confidence interval for the mean.Some of these measures are primarily descrip-tive and will be mentioned only briefly. Thediscussion of the remaining findings is orga-nized around focal theoretical issues. (The al-pha level for reliability was set at p < .05,unless noted otherwise.)

Recall and Recognition

During the recall phase, the subjects cor-rectly recalled the answers to approximately41% of the questions. During the recognitionphase, performance was at an intermediatelevel (mean proportion correct = .49), whichwas desirable to minimize floor and ceilingeffects.

Perceptual Identification

During the preliminary phase, the meanvalues of the parameters for the eventual per-ceptual-identification phase were 27.4 ms forthe start time and 4.1 ms for the incrementtime. These values successfully produced non-

floor and nonceiling performance during thesubsequent perceptual-identification phase:The mean across subjects of each subject'smedian number of exposures was 5.8, whichis intermediate between the floor value of 1and the ceiling value of 10. The predeterminedupper limit of 10 exposures per item was suf-ficient to allow most of the items (i.e., 80%)to be identified correctly.

The left panel of Figure 1 shows a moredetailed examination of perceptual identifi-cation in terms of the rate of increase in correctperformance as a function of the number ofexposures. The data for presented answersshow that the mean cumulative P(correct) in-creased fairly steadily across perceptual-iden-tification trials, which suggests that the incre-ment size established in the preliminary phaseadded approximately 10% correct for each ad-ditional exposure beyond the first exposure.Substantially more perceptual identificationoccurred for presented answers (sixth row ofTable 1) than for catch-trial items [whoseoverall mean P(correct) = .18, with a 95%binomial confidence interval of .10 <-* .25].This demonstrates that perceptual identifica-tion on presented answers is due to something


PRESENTEDo NONPRESENTED

HGHESTMIDDLEMOST FOK

FOK /

2 3 4 5 6 7 8 9 10 2 3 4 5 6 7 8 9 10PERCEPTUAL-IDENTIFICATION TRIAL

Figure 2. Mean cumulative /'(correct) on each perceptual-identification trial for presented and nonpresenteditems that were not identified on the first perceptual-identification trial.

other than mere guessing or reminiscenceacross repeated test trials. The data for catch-trial items, which show no reliable increaseacross perceptual-identification trials, aremarkedly different from the data for presentedanswers.

Reminiscence

The finding of reliably greater-than-chanceperformance on catch-trial items is in accordwith conclusions from previous research onthe feeling of knowing and reminiscence (e.g.,Gruneberg et al., 1973; Hart, 1967b; Read &Bruce, 1982). Reminiscence, operationalizedas correct recall of catch-trial items during theperceptual-identification phase after nonrecallduring the recall phase, varied reliably withthe feeling of knowing (right panel of Figure1). The item with the highest feeling ofknowing had reliably greater reminiscence[P(correct) = .38] than did either the itemwith the middlemost feeling of knowing[P(correct) = .13] or the item with the lowestfeeling of knowing [P(correct) = .03], by asign test for each comparison. Reminiscencedid not differ reliably for the item with themiddlemost feeling of knowing versus the item

with the lowest feeling of knowing, but theamount of reminiscence is near the floor forthese items.

Reminiscence may be due either to (a)spontaneous recovery of the answer during theinterval between the recall phase and the per-ceptual-identification phase (cf. Read & Bruce,1982) or/and to (b) the person initially retriev-ing two answers to a question, giving one (theincorrect answer) during recall and giving theother during the first trial of perceptual iden-tification (cf. Hart, 1966). The latter predictsmore reminiscence on commission-error items(incorrect guess during recall) than on omis-sion-error items (no guess during recall).This prediction was confirmed. The meanP(correct) during perceptual-identificationcatch trials was reliably greater for commis-sion-error items than for omission-error items,t(\9) = 3.04, mean P(correct) = .29, and meanP(correct) = .05, respectively.

Reminiscence occurred primarily on thefirst perceptual-identification trial. This is ob-vious both in Figure 1 and in Figure 2, whichshows performance on each perceptual-iden-tification trial for only those items not correctlyidentified on the first perceptual-identificationtrial.


Feeling of Knowing andCriterion Performance

For each subject, the Goodman-Kruskalgamma correlation was computed between thefeeling-of-knowing rank and criterion perfor-mance. (The rationale for using the gammacorrelation is given by Nelson, 1984.) Thefeeling of knowing is reliably correlated withsubsequent recognition (see Table 1).

The new finding here is that the feeling ofknowing is also reliably correlated with sub-sequent perceptual identification. One ex-ample of this is shown in the right panel ofFigure 1 where the presented item with thehighest feeling of knowing has a higher cu-mulative P(correct) than the presented itemwith the lowest feeling of knowing. The dif-ference is most striking midway through theperceptual-identification trials (e.g., on Trial5) and decreases as performance approachesasymptote.

The differential perceptual identificationshown for the presented items in Figure 1 maybe partly due to reminiscence, especially onthe first perceptual-identification trial. How-ever, the right panel of Figure 2 shows thatperceptual identification was greater for thehighest feeling-of-knowing presented item thanfor the lowest feeling-of-knowing presenteditem, even when reminiscence was minimal.This manifests itself in two ways. First, a signtest showed that the cumulative P(correct) isreliably higher for the highest feeling-of-know-ing presented item than for the lowest feeling-of-knowing presented item on Trials 4 and 5and is marginally higher (.05 < p < .10) onTrial 3. We have no explanation for why theeffect disappears so abruptly by Trial 7. Sec-ond, the number of perceptual-identificationtrials required to obtain a given level of per-formance can be compared for the highestfeeling-of-knowing item versus the lowest feel-ing-of-knowing item; for example, for a cu-mulative P(correct) of .4 in Figure 2, the high-est feeling-of-knowing item required approx-imately four perceptual-identification trialswhereas the lowest feeling-of-knowing itemrequired six.

Table 1 shows that the correlation betweenthe feeling of knowing and the number of ex-posures required for perceptual identificationis significant and negative, indicating that items

with a higher feeling of knowing require fewerexposures to be identified. However, the ab-solute value of this correlation was not largeand was not higher than the absolute value ofthe correlation between the feeling of knowingand recognition; the difference between thesetwo correlations was not significant, paired

1.36.2

If there is a stable unidimensional feeling-of-knowing ability that underlies accurate pre-dictions of criterion performance, then thereshould be a positive relation between the ab-solute value of the correlation for the feelingof knowing and recognition versus the absolutevalue of the correlation for the feeling ofknowing and perceptual identification. TheSpearman correlation computed across sub-jects on this pair of gamma correlations fromeach subject yielded p = -.06, which is notreliably different from zero. This lack of re-lation, especially given the ranges of -.53 to+ 1.00 for the feeling-of-knowing/recognitioncorrelations and -.65 to +.50 for the feeling-of-knowing/perceptual-identification correla-tions, indicates either that the feeling of know-

2 Because the design of this experiment yielded recog-nition that was dichotomous ("correct" or "wrong")whereas perceptual identification was more fine-grained(1, 2, 3,4, 5, 6, 7, 8, 9, or 10 exposures) and because theabsolute magnitude of the gamma correlation is knownto decrease as the fineness of the criterion scores increases(Blalock, 1974), we recomputed each subject's feeling-of-knowing/perceptual-identification gamma correlation bydichotomizing that subject's perceptual-identification per-formance. This dichotomization was accomplished by de-termining a cut point in the distribution of each subject'snumber of exposures, so as to minimize the differencebetween the number of items above the cut point versusthe number of items below the cut point. This procedureis similar to a median split on the number of exposures;the two procedures would be identical if no two items hadthe same number of exposures. However, because ties inthe number of exposures were common and because ourgoal was to dichotomize the number of items rather thanthe number of exposures, we employed the reported pro-cedure rather than a median split. Hence there becameonly two possible values for the number of exposures oneach item, namely "above" versus "below" that subject'scut-point number of exposures. This reanalysis raised theabsolute value of the gamma correlation between the feelingof knowing and perceptual identification: the mean of therecomputed individual-subject correlations was -.21, witha 95% confidence interval of -.05 <-» -.37. However, thisrecomputed correlation also was not significantly differentfrom the correlation between the feeling of knowing andrecognition, /(31) = 0.73.


ing does not map unidimensionally into dif-ferent criterion tests such as recognition versusperceptual identification or/and that the widerange in individual-subject correlations is dueto some kind of noise. The former possibilityis in accord with previous findings that theeffects of a given variable sometimes differ forrecognition versus perceptual identification(Jacoby & Dallas, 1981). Before any firm con-clusions are drawn, however, the latter pos-sibility should be assessed (e.g., by determiningthe test-retest stability of individual-subjectfeeling-of-knowing correlations).

Search Termination

Previous studies have reported that peopleterminate a memory search fastest when norelevant information is found (Glucksberg &McCloskey, 1981; Kolers & Palef, 1976). Theamount of time that a person searches memoryfor a nonrecalled answer could, in principle,be determined either by whether the personknows the answer or by whether the personfeels that he knows the answer (or both). Anideal search-termination person would con-tinue searching for items that he knows andwould discontinue searching for items that hedoesn't know, regardless of what he feels thathe knows.

For each subject, the gamma correlation wascomputed between the latency of incorrect re-call and various other measures. The latencyof incorrect recall (see Table 1) is not reliablycorrelated with either recognition or percep-tual identification (two operational definitionsof what the subject knows), but is reliably cor-related with the feeling of knowing (i.e., withwhat the subject feels that he knows). Thispattern has been reported elsewhere for thecriterion test of recognition (Nelson, Leonesio,Shimamura, Landwehr, & Narens, 1982) andsuggests that people are far from being idealsearch-termination devices.

The individual-subject correlations betweenthe latency of incorrect recall and the feelingof knowing ranged from —.31 to +.59, whichis a wide enough range to suggest the possibilityof individual differences. Because half of eachsubject's items were tested via recognition andhalf were tested via perceptual identification,separate gamma correlations between the la-tency of incorrect recall and the feeling of

knowing were computed on each of these twosubsets of items. A Spearman rho correlationwas then computed across subjects on the pairof gamma correlations from each subject, withthe expectation being a rho of zero if thereare no reliable individual differences in therelation between search termination and thefeeling of knowing (i.e., if the aforementionedrange is due only to noise). The obtained resultwas p = +.41, which indicated that the subjectsare reliably different in the degree to whichtheir latency of incorrect recall and their feelingof knowing are related.

EXPERIMENT 2: RELEARNING

Experiment 2 was designed to explore theaccuracy of the feeling of knowing for pre-dicting relearning performance. Hart (1967a)has speculated the following:

The threshold for activation of a FK [feeling of knowing]signal from the MEMO [memory-monitoring] process isthought to lie between the recall and savings [during re-learning] thresholds. The model implies that Ss would beunable to predict those unrecalled items for which theywould show a savings score. In other words, memory tracescan exist that are too weak to be monitored. This impli-cation has not been tested, (p. 690)

However, the feeling of knowing might haveabove-chance accuracy at predicting relearn-ing. Like perceptual identification, relearninghas concatenation-like properties (Bahrick,1967; Nelson, 1978; Nelson, Fehling, &Moore-Glascock, 1979). Moreover, previousresearch found that relearning detects extrainformation that recognition does not detect(Nelson, 1978). If feeling-of-knowing judg-ments tap this extra information, they mayeven be more accurate at predicting relearningthan at predicting recognition.

Method

Design and Subjects

A between-subjects design was employed. For one groupof subjects (N = 43), the criterion test was recognition,and FACTRETRIEVAL (Shimamura et al., 1981) was usedwithout modification. For the second group (N - 44), thecriterion test was relearning. The subjects were from thesame population as in Experiment 1.

Stimuli and Apparatus

The items consisted of the 240 general-informationquestions from FACTRETRIEVAL. These included all of the


items from Experiment 1 plus additional items whosecorrect answers were longer or shorter than five to sevenletters in length. Items were presented on a video monitorconnected to an Apple II computer, and the subject typedhis responses on the computer keyboard. Instructions tosubjects, data collection, and statistical analysis were com-puter-controlled (Shimamura et al., 1981).

Procedure

The procedure consisted of three discrete phases. Thefirst was a recall test on general-information items, whichlasted until the subject incorrectly answered 12 questions.Second, the subject made relative feeling-of-knowing judg-ments on the 12 incorrectly answered items. This wasaccomplished via paired comparisons (Nelson & Narens,1980a). Third, the criterion test assessed the accuracy ofthe feeling-of-knowing judgments.

Recall Phase

This was identical to the recall phase in Experiment 1except for terminating after 12 incorrectly answered ques-tions rather than after 21.

Feeling-of-Knowing PhaseFollowing the recall phase, pairs of questions from the

set of 12 incorrectly answered questions were displayedon the video monitor. For each pair, the subject selectedwhichever question he felt he was more likely to recognizethe correct answer to. All possible N(N — 1 )/2 = 66 pairedcomparisons of the 12 questions occurred, and the pre-sentation sequence of the pairs was random. The 2 questionsin each pair were displayed 1 above the other, and thedisplay order of the 2 questions was random.

After the feeling-of-knowing judgments, 12 of the 66pairs were presented again for a second feeling-of-knowingjudgment to allow for an estimate of retest reliability (Nel-son & Narens, 1980a). The choice of which 12 pairs wouldbe retested was random except that none of the retestedpairs came from the last 16 of the original 66 pairs; Sixof the retested pairs were presented with the display orderreversed such that, for each pair, the question previouslypresented on top was now presented on the bottom. The2 questions within each of the remaining 6 retested pairswere presented in their original display order.

The feeling-of-knowing rank order was derived by tal-lying the number of times that a given item was chosenover all others in the paired comparisons, not countingthe retest phase. This yields (a) a relative feeling-of-knowingrank order of the items, (b) an index of retest reliability,and (c) the number of intransitivities in the subject's feeling-of-knowing judgments (i.e., if Item A is chosen over ItemB, and Item B is chosen over Item C, then an intransitivityoccurs if Item C is chosen over Item A).

Criterion-Test PhaseHalf of the subjects had a relearning study-test trial,

and half had a four-AFC recognition test.Relearning. The subject viewed one of the 12 incor-

rectly answered questions for 5 s. Then the question dis-

appeared and was followed by a 1-s "READY" signal. Thecorrect answer was then displayed for 1 s and was im-mediately followed by a three-digit number. When thenumber appeared, the subject counted aloud backward bythrees for 30 s; this procedure was employed to minimizeshort-term memory effects and rehearsal between successiveitems. Then the next question was presented, and so onuntil each of the 12 questions and answers had been pre-sented. Finally the subject had a relearning recall test forthe answers to the 12 questions, using the same procedureas in the recall phase. Both the study order and test orderof the 12 questions were random except for the restrictionthat none of the last 3 questions from the relearning studyphase was among the first 3 relearning test questions.

Pilot research had shown that a relatively short displayduration for the relearning study trial was necessary toavoid a ceiling effect on relearning performance. To fa-miliarize the subject with the timing and procedure, arelearning practice trial occurred prior to relearning thetest items. The practice item was not one of the FACTRE-TRIEVAL questions.

Recognition. This was identical to the recognition phasein Experiment 1 except that there were 12 recognitionitems rather than 9.

Results and Discussion

The primary results are summarized in Ta-ble 2. Measures pertaining to recall and thefeeling of knowing are based on N = 87 be-cause all subjects were treated identically dur-ing those phases. Measures involving recog-nition or relearning are based on N = 43 orN = 44, respectively. As in Experiment 1, thefindings are organized around focal theoreticalissues.

Overall Performance

Recall (first row of Table 2) was similar tothat in Experiment 1 (first row of Table 1).

The retest reliability during the feeling-pf-knowing judgments was substantially betterthan chance [the chance /'(mismatch) = .50],although not perfect at P(mismatch) = 0. Also,the P(mismatch) was reliably greater [t(86) =2.62] when the two items in a retested pairedcomparison were presented in reversed order(M = .21) than in the same order as the originalpaired comparison (M = .13). This appearsto be attributable to some subjects who hada position bias (e.g., tending to choose the up-permost item on the video screen). The mainramification is methodological: To avoid un-derestimating or overestimating the magnitudeof /'(mismatch), retests should use either arandom order of the items within each pairedcomparison or a counterbalanced order with


Table 2Mean and 95% Confidence Interval for Each Measure in Experiment 2

N

87878743444344434487

Measure

^(recall)P(mismatch)/ftntransitivity)^(recognition)^(relearning)Gamma: Feeling of knowing and recognitionGamma: Feeling of knowing and relearningGamma: £(incorrect recall) and recognitionGamma: £(incorrect recall) and relearningGamma; L(incorrect recall) and feeling of knowing

M

.40

.17

.08

.44

.66

.28*

.34*

.02

.02

.12*

95% Confidenceinterval

.35

.14

.06

.39

.59

.14

.19-.09-.10

.06

<-+.44«-» .20<-».io<->.49~.72<->.42«.48«-> .14<-».13<-> .18

Note. N = number of subjects on which each measure is based.* p < .05, correlation is reliably different from zero.

equal numbers of same and reversed presen-tations.

The .P(intransitivity) was substantially belowchance [the chance P(intransitivity) = .25 ifjudgments are made randomly], although notat-P(intransitivity) = 0. Across subjects, the mag-nitudes of P(intransitivity) and /"(mismatch)were reliably related, Spearman p = +.35. Thiscorrelation is not surprising because onesource of intransitivities is unreliability(Krantz, Luce, Suppes, & Tversky, 1971), suchthat those subjects who are more unreliablewould be expected to have more intransitivities(although not necessarily vice versa becauseunreliability is not the only source of intran-sitivities; see Krantz et al., 1971).

Both recognition and relearning were at anintermediate level. This was desirable to avoidfloor and ceiling effects.

Feeling of Knowing andCriterion Performance

(The feeling of knowing is reliably correlated

with subsequent recognition, similar to thecorresponding finding in Experiment 1 (Table1). The new finding here is that the feeling ofknowing is also a reliable predictor of relearn-ing (see seventh row of Table 2). Although thesample mean is higher for the correlation be-tween the feeling of knowing and relearningthan for the correlation between the feeling ofknowing and recognition, the difference be-tween these two correlations is not statisticallysignificant, f(85) = .61.3

Search Termination

Similar to Experiment 1, the present find-ings suggest that the amount of time that peo-ple search for nonrecalled answers is deter-mined not by what they know but rather bywhat they feel that they know. Table 2 showsthat the latency of incorrect recall is not re-liably correlated with either recognition or re-learning. However, the correlation between thelatency of incorrect recall and the feeling ofknowing is reliably positive.

Although statistically significant, this cor-relation and the analogous one in Experiment1 (last row of Table 1) are small. More detailedanalyses were conducted by partitioning thedata to examine this relation separately fordifferent types of recall failures. The resultsare striking and are summarized in Figure 3.When the recall failure was an omission error,the correlation was reliably positive and sub-stantial for all three groups of subjects. Thatis, for omission-error items whose labels havenot been retrieved, people search longer whenthe feeling of knowing is high rather than low.

3 The direction of this difference may be maintainedand become statistically significant with a larger sampleof subjects, but the substantial size of the present confidenceintervals suggests that the number of subjects needed toattain conventional levels of statistical significance wouldpreclude such an experiment. Given the same standarddeviations and difference between the means found here,the number of subjects that would be needed for signifi-cance at the .05 level is 455 subjects per group (Hays,1973, Equations 10.13.1 and 10.15.2),


+.50

+40

+30

'I +20

0

-.\0

-.20

KH;: Omission lllla Commission

.3?.. .36

EXP. 1 Recognifion Relearning

EXRF

.?. Mean (and standard error) of the individual-subject gamma correlations between the latency of in-correct recall and the feeling of knowing. (Entry insideeach bar shows the number of subjects on which the meanis based; indeterminant correlations are excluded. Barsshow data for two kinds of recall errors, omission versuscommission, in three groups of subjects: Experiment 1subjects, recognition group in Experiment 2, and relearninggroup in Experiment 2.)

In sharp contrast, when the recall failure is acommission error, the correlation is not reli-ably different from zero for any of the threegroups. That is, when people retrieve the wronganswer but believe they are correct, their la-tency will be no different (or perhaps may even

be shorter) than when they are less sure thatthey are correct. Thus, the correlation betweenthe latency of incorrect recall and the feelingof knowing is qualitatively and reliably dif-ferent for omission-error recall failures thanfor comission-error recall failures, t(21) - 2.94for Experiment 1, t(32) = 4.09 for the rec-ognition group in Experiment 2, and *(35) =4.33 for the relearning group. Implications ofthis finding are described later.

Given this difference between omission er-rors and commission errors concerning the re-lation between the latency of incorrect recalland the feeling of knowing, one may expectthat these two kinds of recall failure will alsodiffer in latency of incorrect recall. In partic-ular, the latency of incorrect recall may belonger for omission errors than for commissionerrors, which in turn may be similar to thelatency of correct recall because the personmay process commission errors and correctrecalls in a similar way (cf. Krinsky & Nelson,in press).

Contrary to this expectation, the latenciesfor commission errors were more like the la-tencies for omission errors than like the la-tencies for correct recall. For the same threegroups of subjects as in Figure, 3, Table 3 showsthe mean (and 95% confidence interval) of theindividual-subject median latencies for eachtype of recall outcome. Although the latenciestended to be shorter in Experiment 1 than inExperiment 2, every group of subjects dis-played the same pattern of latencies for omis-sion errors versus commission errors; thepaired t values from subjects contributing both.kinds of latencies were /(29) = 0.06 for Ex-

Table 3Mean (and 95% Confidence Interval) for Individual-Subject Median Latencies of Correct Recall,Incorrect Commission-Error Recall, and Incorrect Omission-Error Recall

Subjects

Latency

Experiment 2

Experiment 1 Recognition group Relearning, group

Correct recall

Commission error

Omission error

9.8(8.8 *-» 10.7)

20.4(17.5 «-» 23.3)

20.5(15.0 <-» 26.0)

13.4(11.9<-» 14.9)

25.9(23.2 <-» 28.6)

26.0(20.4 «-» 31.7)

15.2(13.3 «-» 17.1)

26.4(22.8 <-> 30.0)

-25.1(19.3 «-» 30.9)


periment 1, f(35) = 0.14 for the recognitiongroup, and t(35) = 0.36 for the relearninggroup. In contrast, the paired t values for thecomparison of correct-recall latencies withcommission-error latencies were r(31) = 8.93for Experiment 1, t(4l) = 9.19 for the rec-ognition group, and f(40) = 6.31 for the re-learning group; the paired t values for thecomparison of ciorrect-recall latencies withomission-error latencies were f(29) = 3.94 forExperiment 1, f(34) = 4.18 for the recognitiongroup, and f(32) = 3.13 for the relearninggroup (all six ts indicate reliable differences).Moreover, as Table 3 shows for every groupof subjects, the 95% confidence interval forlatency of correct recall did not even overlapwith that of commission errors or omissionerrors.

Thus, in both experiments the relation be-tween search termination and the feeling ofknowing was different for commission errorsversus omission errors. At the same time,however, people searched for approximatelythe same amount of time whether their in-correct recall was a commission error or omis-sion error, and they searched longer when theywere incorrect than when they were correct.

GENERAL DISCUSSION

The feeling of knowing may determine howlong a memory search is continued, especiallywhen the recall failure is an omission error(Experiments 1 and 2). Because search ter-mination is related to the feeling of knowingrather than to subsequent criterion perfor-mance (Experiments 1 and 2), if the accuracyof the feeling of knowing can be improved,then a corresponding improvement may occurin the efficiency of memory search. That is,the person would continue searching for itemsthat are in memory, thereby avoiding unnec-essary memory failures due to prematuresearch termination (a problem in Korsakoffpatients; see Hirst, 1982, p. 454), and wouldstop searching for items not in memory. Animprovement in feeling-of-knowing accuracymay be particularly relevant for individualswhose correlation between the feeling ofknowing and search termination is high (re-liable individual differences in the magnitudeof these correlations occurred in Experi-ment 1).

The magnitude of feeling-of-knowing ac-curacy did not differ reliably for perceptualidentification versus recognition (Experiment1) or for relearning versus recognition (Ex-periment 2). Thus, the information in memorythat is assessed during feeling-of-knowingjudgments does not seem to be assessed anybetter (or worse) by a recognition test than byconcatenation-like tests such as relearning andperceptual identification. Caution should beexercised, however, when generalizing thisconclusion to other versions of these criteriontests. For instance, feeling-of-knowing accu-racy probably would be greater for eight-AFCrecognition than for four-AFC recognitionbecause the former is less noisy than the latter(Nelson, 1984). Different versions of relearningalso may yield different feeling-of-knowing ac-curacy; for example, Bahrick (1982) reportedunusually high feeling-of-knowing accuracywhen the delay between a relearning studyphase and a relearning test phase was 1 month.Feeling-of-knowing accuracy may also varyacross different versions of perceptual iden-tification, such as repeated exposures at a con-stant exposure duration (Haber & Hershenson,1965).

The feeling-of-knowing accuracy here wasabove chance, but far from perfect. However,a low correlation between the feeling of know-ing and criterion performance does not nec-essarily reflect an inherently poor metacog-nitive ability. For instance, although feeling-of-knowing accuracy was somewhat low forpredicting perceptual identification on allitems, it was substantial for predicting per-ceptual identification on the highest and lowestfeeling-of-knowing items (Figure 1). The na-ture of to-be-judged items affects feeling-of-knowing accuracy, such that differentiatingamong items is more difficult when the un-derlying information in memory is similar forthe items. Acquisition parameters also can af-fect the degree of feeling-of-knowing accuracy.For instance, Nelson et al. (1982) found thatthe correlation between the feeling of knowingand recognition was .00 for items originallylearned to a criterion of one correct acquisitionresponse but +.40 for items originally learnedto a criterion of four correct acquisition re-sponses.

No reliable relation occurred between thefeeling-of-knowing accuracy for predicting


perceptual identification and the feeling-of-knowing accuracy for predicting recognition(Experiment 1). This could be due to noise (apossibility that can be explored by examiningthe reliability of individual subjects' feeling-of-knowing accuracy) and/or to the possibilitythat recognition and perceptual identificationassess memory in fundamentally differentways. Other research also has found a non-significant correlation between criterion testssimilar to the ones examined here (see Jacoby& Dallas, 1981). Predictions about the relationbetween a given criterion test and the feelingof knowing should improve as more is knownabout (a) how various criterion tests assess in-formation in memory and (b) how informationin memory influences the feeling of knowing.

Theoretical Mechanisms Underlyingthe Feeling of Knowing

On what information do people base theirfeeling-of-knowing judgments? The data hereand in the literature do not yet provide enoughconstraints to determine the mechanisms thatunderlie the feeling of knowing. Nevertheless,a discussion of the theoretical mechanisms thatcurrently seem reasonable as potential basesfor the feeling of knowing may be timely andmay help to guide future research. The presentdiscussion is not a critical review of the evi-dence in the literature, but rather is a briefdescription of some plausible mechanisms thatare summarized in Table 4. They frequentlyoverlap—one mechanism can involve aspectsof another—but their emphases are different.

Trace-Access Mechanisms

The trace-access mechanisms share thecharacteristic that the person is presumed tohave access to the nonrecalled item duringfeeling-of-knowing judgments. Differentmechanisms presumably monitor different as-pects of the nonrecalled item.

Subthreshold Strength

Here the focus is on the amount of asso-ciative strength between the cue and target(Hart, 1967a; Read & Bruce, 1982, p. 296),for example, between the cue "What is thecapital of Australia?" and the target "Can-berra." Supposedly there is a threshold amount

Table 4Some Plausible Theoretical MechanismsUnderlying the Feeling of Knowing

Trace-access mechanismsSubthreshold strengthForward-backward associationsCorrect semantic referent but no labelPartial recall of labelWrong referentMultidimensional target

Inferential mechanismsRelated episodic informationClaimed related episodic informationRecognition of cueExpertise on topicsActuarial informationSocial desirability

of strength, R, for recall and another thresholdamount of strength, F, for the feeling of know-ing (with F < R). When the amount of strengthin the soughtafter item is greater than R, theperson correctly recalls the target. When theamount of strength is less than R but greaterthan F, incorrect recall occurs, but the personfeels that he knows the target. When theamount of strength is less than F± incorrectrecall occurs, and the person feels that he doesnot know the target (see Hart 1967a, Figure1). More than two gradations of the feeling ofknowing could be postulated by assuming ad-ditional thresholds, each of which would cor-respond to a different feeling-of-knowing rating(Hart, 1967a, Experiment 2; Read & Bruce,1982). The primary problem with thesubthreshold-strength mechanism is that theamount of strength is hypothetical and notdirectly measurable, making tests of themechanism difficult. Nevertheless, this mech-anism is popular, as Gruneberg (1983) men-tioned: "Most accounts of the nature of feelingof knowing assume that the individual in someway makes a preliminary estimate of the like-lihood of an item being in store by assessingeither the strength or. . . ." (p. 20)

Forward-Backward Associations

This is an extension of the subthreshold-strength mechanism. Instead of one associa-tion between the cue and target, this mecha-nism proposes two associations: a forward as-sociation from the cue to the target and abackward association from the target to the

296 T. NELSON, D. GERLER, AND L. KARENS

cue (Hart, 1967b, p. 196). For instance, theremay be an association from "What is the cap-ital of Australia?" to "Canberra" and anotherassociation in the opposite direction. In oneversion of this mechanism, the feeling ofknowing would presumably be monitoring thestrength of the forward association, whereasrecognition would be based on both the for-ward association and the backward association(cf. Wolford, 1971). Consequently, an itemwith associations that are asymmetric instrength could, for instance, produce a feelingof not knowing (due to a weak forward as-sociation), but a high likelihood of recognition(due to a strong backward association).

An assumed model of criterion performancein terms of forward-backward associationsmay be required to assess the validity of thismechanism. For example, if one assumes that(a) the feeling of knowing monitors only theforward association, (b) releaming is based ononly the forward association, and (c) recog-nition is based on both the forward and back-ward associations, then one may expect thefeeling of knowing to be a more accurate pre-dictor of relearning than of recognition, par-ticularly when the items have asymmetric as-sociations. Experiment 2 did not yield suchresults, which (given the independent supportof Assumption C by Wolford, 1971) suggestseither that Assumption A or B is incorrect orthat the associations were not asymmetricenough to produce a detectable difference. Amore direct test of Assumption A has not beenconducted, but would be desirable. Perhapsthe methodology of Nelson et al. (1982) couldbe employed to manipulate overlearning inthe forward versus backward direction ofpaired-associate acquisition prior to assessingthe feeling of knowing and criterion perfor-mance.

Correct Semantic Referent but No Label

This mechanism assumes that the target ofan item consists of two portions: the semanticreferent and a label for the referent (Eysenck,1979; Gardiner et al., 1973). The cue elicitsthe correct semantic referent (or what has beenreferred to as "contextual information" byRead & Bruce, 1982), but not the label forthat referent. For instance, the cue "What isthe capital of Australia?" may produce the se-

mantic referent of the target (which could in-clude images of Canberra from photographs—cf. May & Clayton, 1973—or descriptions frompersonal experience), but not the label "Can-berra." Such an occurrence would yield anomission error, perhaps one associated with ahigh feeling of knowing (see Search Termi-nation section in Experiment 2).

Partial Recall of Label

Here some (but not all) attributes of thelabel are recalled. In their study of the tip-of-the-tongue phenomenon, Brown and McNeill(1966) called this "generic recall" and wrotethat it involves "letters (or phonemes), affixes,syllables, and stress location" (p. 335). Thefeeling of knowing appears to be greater foritems whose labels are partially recalled (Blake,1973; Koriat & Lieblich, 1974; Wellman,1977).

Wrong Referent

, Here the cue elicits the wrong semantic ref-erent (Koriat & Lieblich, 1974> 1977; also seeEysenck, 1979, r3.245; Yarmey, 1973, p. 288).For instance, the cue "What is the capital ofAustralia?" may elicit the semantic referent ofSydney. In turn, the corresponding label of"Sydney" may be elicited (whfch would yielda commission error) or may pot be elicited(which would yield an omission error). Whenthe cue elicits a semantic referent that the per-son believes is correct (even though the ex-perimenter considers it to be wrong), the per-son tends to report a high feeling of knowing(Koriat & Lieblich, 1974, 1977; Krinsky &Nelson, in press). Investigators who want toexamine this mechanism could either developword-definition items with th$ properties de-scribed by Koriat and Lieblich < 1977, p. 156f)or use general-information iljems that haveboth a low probability of recall and a highfeeling of knowing. For instance, the cue"What is the last name of the Union generalwho defeated the Confederate army at the CivilWar battle of Gettysburg?" (see Nelson & Na-rens, 1980b) tends to elicit a high feeling ofknowing and the wrong answer (e.g.; Grant,Sherman, or Sheridan), rather than the correctanswer (Meade).


Multidimensional TargetThis mechanism is more abstract than the

aforementioned mechanisms (although itcould include aspects of them) and emphasizesthe multidimensionality of a target (Koriat &Lieblich, 1977; Underwood, 1969; also seeBower, 1967). A person may not retrieve in-formation from enough dimensions for recallto be correct, but the information that is re-trieved could serve as the basis for a high feel-ing of knowing (e.g., information from di-mensions of the semantic differential; Eysenck,1979). Such a mechanism is consistent Withthe finding of no relationship between the feel-ing-of-knowing accuracy for predicting per-ceptual identification and the feeling-of-knowing accuracy for predicting recognition(Experiment 1). These different criterion testsmay tap different dimensions of the underlyingmemory structure. Moreover, the feeling ofknowing may tap a richer combination of di-mensions than does any one criterion test. Thatis, each criterion test may tap a different subsetof whatever information in memory is tappedby the feeling of knowing; hence, even thoughthe feeling of knowing may be completely validas a monitor of nonrecalled information inmemory, no criterion test would be correlatedperfectly with the feeling of knowing. Consis-tent with this possibility, Seamon, Brody, andKauff (1983) recently reported evidence of amultidimensional memory structure that rec-ognition taps only partially.

Inferential MechanismsFor these mechanisms,the feeling of know-

ing does not monitor the nonrecalled targetitem. Instead, other information in memoryis monitored and serves as the basis for aninference about the likelihood of correct per-formance on the criterion test.

Related Episodic InformationThe concept of "related information" is

sometimes vague and misleading. For example,suppose that someone does not recall the an-swer to the question, "What is the name ofthe man who assassinated Abraham Lincoln?"and has to make a feeling-of-knowing judg-ment about whether he will recognize the cor-rect answer. If the person recalled that the as-

sassination occurred in Ford's theatre duringApril of 1865 and that the assassin broke aleg leaping to the stage, we might expect thisrelated information about the assassinationepisode to a trigger a high feeling of knowing.Accordingly, the conclusion drawn from thisexample might be that related information isan important basis for the feeling of knowing.However, suppose instead that the questionhad been, "What is the shoe size of the manwho assassinated Abraham Lincoln?" and thatthe person did not recall the answer, but re-called the same related information about theassassination episode as mentioned above.Now we would not expect the person to havea high feeling of knowing, even though the"related information about the assassinationepisode" is identical to that from the first sit-uation!

Perhaps the related episodic informationthat is critical pertains less to the episode ad-dressed in the question than to prior episodesin the answerer's life during which the sought-after item was encountered. If the person hadno prior encounters with the item (e.g., "Whatis President Reagan's telephone number?"),then he probably would have a feeling of notknowing (Kolers & Palef, 1976). By contrast,if information about prior encounters with thesoughtafter item is present in memory, a highfeeling of knowing is more likely (cf. avail-ability in Tversky & Kahneman, 1974). Forinstance, a person who is unable to recall theanswer to the question "What is the capitalof Australia?" might have been in a geographyclass where everyone learned the capitals ofall major countries, and this might serve asthe basis for a high feeling of knowing. Con-sistent with this, Nelson et al. (1982) foundthat the feeling of knowing for nonrecalleditems was a direct function of the prior amountof over-learning. This mechanism also mightinvolve other aspects of prior encounters, suchas how difficult the item was to learn.

Claimed Related Episodic InformationThis mechanism is similar to the previous

one, except here the emphasis is on what theperson recalls about prior encounters with theitem rather than on the prior encounters perse (cf. Centner & Collins, 1981). For instance,suppose that there are two nonrecalled items,


A and B, and that there had been more over-learning on Item A than on Item B, but theperson claims to have had more overlearningon Item B than on Item A. Which item willreceive the higher feeling of knowing? By hav-ing people make various judgments about theirprior encounters with nonrecalled items, fu-ture investigators may be able to separate theeffect of prior encounters per se from the effectof what the person claims those prior en-counters were.

Recognition of Cue

Here the feeling of knowing for the non-recalled target is based on the person's degreeof recognition for the cue (cf. Hoffding, 1891).Even kindergartners, when making a feeling-of-knowing judgment about a nonrecalled tar-get, utilize information about whether theyhave seen the cue previously (Wellman, 1977).Moreover, the feeling of knowing for a non-recalled target can be increased by merelyadding redundancy to the cue (e.g., by re-peating it or by adding alternative wording;Koriat & Lieblich, 1977). Thus, memory fac-tors concerning the cue produce the inferenceabout whether the nonrecalled target is known.

Expertise on Topics

Here the person draws an inference aboutthe correctness of his criterion performancebased on his expertise with the general topicof the item (Bradley, 1981; also see Gentner& Collins, 1981). Koriat and Lieblich (1974,1977) initiated research on the relation be-tween the feeling of knowing and informationcommon to a category of items versus infor-mation specific to a particular item. The for-mer information, even in the absence of thelatter information, may trigger a high feelingof knowing. There has not, however, been re-search on how a person's rated expertise fora category is related to his feeling of knowingfor nonrecalled items in that category. For in-stance, relative to other people, a person whohas (or at least claims to have) expertise aboutsports may have an unrealistically high feelingof knowing for an extremely difficult (i.e., lownormative probability) sports item such as thename of the player who had the most hits inthe 1937 World Series. Put differently, the per-

son may underutilize base-rate informationabout actuarial difficulty because it may notseem representative of his expertise (cf. rep-resentativeness in Tversky & Kahneman,1974). However, given that correct recognitionis sometimes mediated by recognition of gen-eral-category information instead of specific-item information (MacLeod & Nelson, 1976),a high feeling of knowing based on expertiseabout the category sometimes may be appro-priate.

Actuarial Information

In contrast to the individual basing his feel-ing of knowing on idiosyncratic information(e.g., such as his own expertise on the topic),he may utilize actuarial information about thenormative difficulty of the item. Nelson, Leo-nesio, Landwehr, and Narens (1983) foundcorrelations of approximately +.25 betweenthe individual's feeling-of-knowing ranking ofitems and the ranking obtained from the.P(recall) in the Nelson and Narens norms(1980b). This mechanism would tend to in-crease the accuracy of feeling-of-knowingjudgments because an individual's criterionperformance on nonrecalled items tends to becorrelated with the group's initial performance(Brown, 1923;Grunebergetal., 1977, p. 369).Furthermore, Nelson et al. (1983) found thatactuarial predictions derived from the /'(recall)in the Nelson and Narens norms (1980b) weremore accurate for predicting an individual'scriterion performance than were the individ-ual's own feeling-of-knowing judgments. Be-cause people are accurate at estimating ac-tuarial memory performance, (Arbuckle &Cuddy, 1969; Seamon & Virostek, 1978; Un-derwood, 1966), perhaps they should give ac-tuarial information more weight during feel-ing-of-knowing judgments.

Social Desirability

Here the person reports a high feeling ofknowing based on what he thinks he ought toknow "because of a desire not to be thoughtstupid" (Gruneberg et al., 1977, p. 370). Al-though such demand characteristics may serveas a basis for absolute feeling-of-knowingjudgments from the Hart methodology (e.g.,by lowering the person's threshold for reporting


a high feeling of knowing), they may be lessinfluential during the relative feeling-of-knowing judgments from the Nelson-Narensmethodology.

The aforementioned mechanisms are pri-marily static insofar as they do not emphasizethe possibility of changes in the feeling ofknowing. Research on the dynamic aspects ofthe feeling of knowing (e.g., effects of feedback;Krinsky & Nelson, in press) is just beginning.One potentially useful direction for future re-search may be to explore dynamic processessuch as changes in the feeling of knowing dur-ing the course of an unsuccessful memorysearch.

References

Allport, A. (1977). On knowing the meaning of words weare unable to report: The effects of visual masking. In5. Domic (Ed.), Attention and performance VI. Hillsdale,NJ: Erlbaum.

Arbuckle, T. Y, & Cuddy, L. L. (1969). Discriminationof item strength at time of presentation. Journal of Ex-perimental Psychology, 81, 126-131.

Bahrick, H. P. (1967). Relearning and the measurementof retention. Journal of Verbal Learning and VerbalBehavior, 6, 89-94.

Bahrick, H. P. (1982, August). Maintenance of knowledge.Invited address at the meeting of the American Psy-chological Association, Washington, DC.

Bernbach, H. A. (1967). Decision processes in memory.Psychological Review. 74, 462-480.

Blake, M. (1973). Prediction of recognition when recallfails: Exploring the feeling-of-knowing phenomenon.Journal of Verbal Learning and Verbal Behavior, 12,311-319.

Blalock, H. M., Jr. (1974). Beyond ordinal measurement:Weak tests of stronger theories. In H. M. Blalock, Jr.(Ed.), Measurement in the social sciences: Theories andstrategies (pp. 424-455). Chicago: Aldine.

Bower, G. H. (1967). A multicomponent theory of thememory trace. In K. W. Spence & J. T. Spence (Eds.),The psychology of learning and motivation (Vol. 1). NewYork: Academic Press.

Bradley, J. V. (1981). Over-confidence in ignorant experts.Bulletin of the Psychonomic Society, 17, 82-84.

Brown, A. L., & Lawton, S. C. (1977). The feeling ofknowing experience in educable retarded children. De-velopmental Psychology, 13, 364-370.

Brown, R., & McNeill, D. (1966). The "tip of the tongue"phenomenon. Journal of Verbal Learning and VerbalBehavior, 5, 325-337.

Brown, W. (1923). To what extent is memory measuredby a single recall? Journal of Experimental Psychology,6, 377-382.

Burton, M. L., & Nerlove, S. B. (1976). Balanced designsfor triads tests: Two examples from English. Social Sci-ence Research, S, 247-267.

Coombs, C. H., Dawes, R. M., & Tversky, A. (1970).

Mathematical psychology: An elementary introduction.Englewood Cliffs, NJ: Prentice-Hall.

Eich, J. M. (1982). A composite holographic associativerecall model. Psychological Review, 89, 627-661.

Eysenck, M. W. (1979). The feeling of knowing a word'smeaning. British Journal of Psychology, 70, 243-251.

Flavell, J. H., & Wellman, H. M. (1977). Metamemory.In R. V. Kail, Jr., & J. W. Hagen (Eds.), Perspectiveson the development of memory and cognition. Hillsdale,NJ: Erlbaum.

Fowler, C. A., Wolford, G., Slade, R., & Tassinary, L.(1981). Lexical access with and without awareness.Journal of Experimental Psychology: General, 110, 341-362.

Freedman, J. L., & Landauer, T, K. (1966). Retrieval oflong-term memory: "Tip-of-the-tongue" phenomenon.Psychonomic Science, 4, 309-310.

Gardiner, J. M., Craik, F. I. M., & Bleasdale, F. A. (1973).Retrieval difficulty and subsequent recall. Memory &Cognition, 1, 213-216.

Centner, D., & Collins, A. (1981). Studies of inferencefrom lack of knowledge. Memory & Cognition, 9,434-443.

Glucksberg, S., & McCloskey, M. (1981). Decisions aboutignorance: Knowing that you don't know. Journal ofExperimental Psychology: Human Learning and Mem-ory, 7, 311-325.

Groninger, L. K., & Groninger, L. P. (1980). A comparisonof recognition and sayings as retrieval measures: A reex-amination. Bulletin of the Psychonomic Society, 15,263-266.

Gruneberg, M. M. (1983). Strategies in human memory.In A. Mayes (Ed.), Memory in animals and man. London:Van Nostrand.

Gruneberg, M. M., & Monks, J. (1974). Feeling of knowingand cued recall. Ada Psychologica, 38, 257-265.

Gruneberg, M. M., Monks, J., & Sykes, R. N. (1977).Some methodological problems with feeling of knowingstudies. Acta Psychologica, 41, 365-371.

Gruneberg, M. M., Smith, R. L., & Winfrow, P. (1973).An investigation into response blockaging. Acta Psy-chologica, 37, 187-196.

Gruneberg, M. M., & Sykes, R. N. (1978). Knowledgeand retention: The feeling of knowing and reminiscence.In M. M. Gruneberg, P. E. Morris, & R. N. Sykes (Eds.),Practical aspects of memory (pp. 189-196). New \brk:Academic Press.

Haber, R. N., & Hershenson, M. (1965). Effects of repeatedbrief exposures on the growth of a percept. Journal ofExperimental Psychology, 69, 40-46.

Hart, J. T. (1965a). Memory and the feeling-of-knowingexperience. Journal of Educational Psychology, 56, 208-216.

Hart, J. T. (1965b). Recall, recognition, and the memorymonitoring process. Doctoral dissertation, StanfordUniversity. (University Microfilms No. 66-2565).

Hart, J. T. (1966). Methodological note on feeling-of-knowing experiments. Journal of Educational Psychol-ogy, 57, 347-349.

Hart, J. T. (1967a). Memory and the memory-monitoringprocess. Journal of Verbal Learning and Verbal Behavior,6, 685-691.

Hart, J. T. (1967b). Second-try recall, recognition, and thememory-monitoring process. Journal of EducationalPsychology, 58, 193-197.


Hayes-Roth, B. (1977). Evolution of cognitive structuresand processes. Psychological Review, 84, 260-278.

Hays, W. L. (1973). Statistics for the social sciences (2nded.). New York: Holt, Rinehart, & Winston. ,

Hirst, W. (1982). The amnesic syndrome: Descriptionsand explanations. Psychological Bulletin, 91, 435-460.

Hoffding, H. (1891). Outlines of psychology (M. E.Lowndes, Trans.). London: Macmillan.

Horowitz, L. M., & Prytulak, L. S. (1969). Redintegrativememory. Psychological Review, 76, 519-531.

Hull, C. L. (1943). Principles of behavior. New York: Ap-pleton-Century-Crofts.

Jacoby, L. L., & Dallas, M. (1981). On the relationshipbetween autobiographical memory and perceptuallearning. Journal of Experimental Psychology: General,110, 306-340.

Kolers, P. A., & Palef, F, R. (1976). Knowing not. Memory& Cognition, 4, 553-558.

Kpriat, A., & Lieblich, I. (1974). What does a person ina "TOT" state know that a person in a "don't know"state doesn't know. Memory. & Cognition, 2, 647-655..

Koriat, A., & Lieblich, I. (1977). A study of memorypointers. Acta Psychologica, 41, 151-164.

Krantz, D. H., Luce, R. D., Suppes, P., & Tversky, A.(1971). Foundations of measurement (Vol. 1). New York:Academic Press.

Krinsky, R., & Nelson, T. O. (in press). The feeling ofknowing for different types of retrieval failures. ActaPsychologica,

Lachman, J. L., Lachman, R., & Thronesbery, C. (1979).Metamemory through the adult lifespan. DevelopmentalPsychology, 15, 543-551.

MacLeod, C. M., & Nelson, T. Q (1976). A nonmonotoniclag function for false alarms to associates. AmericanJournal of Psychology, 89, 127-135.

Marcel, A. (1983). Conscious and unconscious perception:An approach to the relations between phenomenal ex-perience and perceptual processes. Cognitive Psychology,15, 238-300.

May, J. E., & Clayton, K. N. (1973). Imaginal processesduring the attempt to recall names. Journal of VerbalLearning and Verbal Behavior, 12, 683-688.

Merikle, P. M. (1982). Unconscious perception revisited.Perception & Psychophysics, 31, 298-301.

Murrell, G. A., & Morton, J. (1974). Word recognitionand morphemic structure. Journal of ExperimentalPsychology, 102, 963-968.

Neisser, U. (1954). An experimental distinction betweenperceptual'process and verbal response. Journal of Ex-perimental Psychology, 47, 399-402.

Nelson, T. O. (1978). Detecting small amounts of infor-mation in memory: Savings for rionrecognized items.Journal of Experimental Psychology: Human Learningand Memory, 4, 453-468.

Nelson, T. O. (1984). A comparison of current measuresof the accuracy of feeling-of-knowing predictions. Psy-chological Bulletin, 95, 109-133.

Nelson, T. O, Fehling, M. R., & Moore-Glascock, J. (1979).The nature of semantic savings for items forgotten fromlong-term memory. Journal of Experimental Psychology:General, 108, 225-258.

Nelson, T. O., Gerler, D., & Narens, L. (in press). A tech-nique for determining perceptual-identification param-eters for use in feeling-of-knowing research. Psycholog-ical Documents.

Nelson, T. O, Leonesio, R. J., Landwehr, R. E, & Narens,L. (1983). Accuracy of feeling-of-knowing predictions vs.predictions from normative recall. Unpublished manu-script, University of Washington, Seattle.

Nelson, T. O., Leonesio, R. J., Shimamura, A, P., Landwehr,R. E, & Narens, L. (1982). Overlearning and the feelingof knowing. Journal of Experimental Psychology:Learning, Memory, and Cognition, 8, 279-288.

Nelson, X O., & Narens, L. (1980a). A new technique forinvestigating the feeling of knowing. Acta Psychologica,46, 69-80.

Nelson, T. Q, & Narens, L. (1980b). Norms of 300 general-. information questions: Accuracy of recall, latency ofrecall, and feeling-of-knowing ratings. Journal of VerbalLearning and Verbal Behavior, 19,, 338-368.

Ratcliff, R. (1978). A theory of memory retrieval. Psy-chological Review, 85, 59-108-.

Read, J. D., & Bruce, D. (1982). Longitudinal trackingof difficult memory retrievals. Cognitive Psychology, 14,280-300.

Schacter, D. L. (1983). Feeling of knowing in episodicmemory. Journal of Experimental Psychology: Learning,Memory, and Cognition, 9, 39-54., '

Seamon, J. G., Brody, N., & Kauff, D. M. (1983). Affectivediscrimination of stimuli that are not recognized: Effectsof shadowing, masking, and cerebral laterality. Journalof Experimental Psychology: Learning, Memory, &Cognition, 9, 544-555.

Seamon, J. G., & Virostek, S. (1978). Memory performanceand subject-defined depth of processing. Memory &Cognition, 6, 283-287.

Shimamura, A. P., Landwehr, R. E, & Nelson, T. O. (1981).FACTRETRIEVAL: A program for assessing someone's re-call of general-information facts,: feeling-of-knowingjudgments for nonrecalled facts, iand recognition ofnonrecalled facts. Behavior Research Methods & In-strumentation, 13, 691-692.

Tversky, A., & Kahneman, D. (1974). Judgment underuncertainty: Heuristics and biases.'Science, 185, 1-124-1131. ;

Underwood, B. J. (1966). Individual and group predictionsof item difficulty for free learning. Journal of Experi-mental Psychology, 71, 673-679.

Underwood, B. J. (1969). Attributes of memory. Psycho-logical Review, 76, 559-573.

Wellman, H. M. (1977). Tip of the tongue and feeling ofknowing experiences: A developmental study of memorymonitoring. Child Development, 48, 13-21.

Wickelgren, W. A. (1977). Speed-accuracy tradeoff andinformation processing dynamics, Acta Psychologica,41, 67-85.

Winnick, W. A., & Daniel, S. A. (J970). Two kinds ofresponse priming in tachistoscopic recognition. Journalof Experimental Psychology, 84, 74-81.

Wolford, G. (1971). Function of distinct associations forpaired-associate performance. Psychological Review, 78,303-313.

Yarmey, A. D. (1973). I recognize your face but I can'tremember your name: Further evidence on the tip-of-the-tongue phenomenon. Memory & Cognition, 1,287-290.

Received September 6, 1983Revision received November 1, 1983