tomal roc 2011

8/12/2019 Tomal Roc 2011

1/12

Differential Sensitivity of TOMAL Subtests and Index Scoresto Pediatric Traumatic Brain Injury

Nicholas S. Thaler and Sally J. Barney

Department of Psychology, University of Nevada-Las Vegas, Las Vegas, Nevada

Cecil R. Reynolds

Department of Educational Psychology, Texas A&M University, College Station, Texas

Joan Mayfield

Our Childrens House at Baylor University, Dallas, Texas

Daniel N. Allen

Department of Psychology, University of Nevada-Las Vegas, Las Vegas, Nevada

The objective of the present study was to examine and compare the subtest, index, andfactor scores of the Test of Memory and Learning (TOMAL), using receiver-operatingcharacteristic curves, to investigate their sensitivity and specificity to traumatic braininjury (TBI) in children and adolescents. One hundred and fifty participants who hadsustained TBI were compared to 150 controls matched on age and gender from theTOMALs standardization sample. Results indicated that the greatest area under thecurve (AUC) was for the Object Recall (OR) subtest score, the Composite MemoryIndex (CMI), and the attention factor score. The optimal CMI cutoff score for a TBIdiagnosis was 83. When factor scores were compared, the attention factor and two ver-bal factors had significantly larger AUCs than the three nonverbal factors. These find-ings suggest that the OR subtest and CMI are most sensitive to TBI, and that whencomponents were broken into factors with no overlapping subtests, attention and verbalmemory were optimal for classifying TBI.

Key words: memory, pediatric, ROC, TBI, TOMAL

In the last decade, there has been a substantial increasein research on neuropsychological assessment for youthwho have sustained a traumatic brain injury (TBI). Out-

comes for this population have been well documented,and TBI has been found to affect childrens cognitive,behavioral, emotional, and academic functioning nega-tively (Hooper, Alexander, & Moore, 2004; Lowther &

Mayfield, 2004; Max et al., 1998; Roberts & Furuseth,1997; Wassenberg, Max, & Lindgren, 2004). Impairedattention and memory have been reported routinely in

pediatric TBI (Dennis, Guger, Roncadin, Barnes, &Schachar, 2001; Hooper et al., 2004; Max et al., 1998;Wassenberg et al., 2004), with severity of these deficitsserving as an important predictor of outcome(Catroppa & Anderson, 2002, 2007). A number of neu-ropsychological memory batteries including theWide-Range Assessment of Memory and Learning-Second Edition (WRAML-2; Sheslow & Adams,2003), the California Verbal Learning Test for Children

Address correspondence to Daniel N. Allen, Ph.D., Neuropsychol-

ogy Research Program, Department of Psychology, University ofNevada-Las Vegas, 4505 Maryland Parkway, Las Vegas, NV 89154.E-mail: [email protected]

APPLIED NEUROPSYCHOLOGY,18: 168178, 2011

Copyright # Taylor & Francis Group, LLCISSN: 0908-4282 print=1532-4826 online

DOI: 10.1080/09084282.2011.595443

8/12/2019 Tomal Roc 2011

2/12

(CVLT-C; Delis, Kramer, Kaplan, & Ober, 1994), theChildrens Memory Scale (Wechsler, 1995), and the Testof Memory and Learning (TOMAL; Reynolds & Bigler,1994) have been used to assess these deficits in this popu-lation with reliable and comparable findings. However,relatively few studies have examined the ability of scoresfrom these batteries to distinguish populations and spe-

cific deficits in pediatric TBI.Researchers have previously examined the ability of

scores from intelligence batteries to distinguish deficitsin both child and adult TBI populations, and findingsgenerally converge in identifying selectively impairedperformance on working memory and processing speedsubtests (Allen, Thaler, Donohue, & Mayfield, 2010;Calhoun & Mayes, 2005; Donders & Janke, 2008;Donders, Tulsky, & Zhu, 2001; Mayes & Calhoun,2004). Although useful within the clinical setting, intelli-gence batteries provided limited information on specificcognitive processes such as memory. With regard tomemory batteries, researchers typically select subtests

within the battery for evaluation of specific constructssuch as working memory. However, a lack of criterionvalidity studies on memory batteries may hinder theirapplication to the very clinical populations they weredesigned to evaluate (Allen, Knatz, & Mayfield, 2006).

Existing criterion validity studies do confirm somereliable findings across batteries. For example, a studyof WRAML scores found that children with severeinjuries performed approximately one standard devi-ation below the mean on verbal, visual, and generalindexes, while children with mild and moderate injuriesperformed significantly better than the severe group onthe visual memory index (Farmer et al., 1999). However,

none of the groups performed significantly worse on theVisual Learning Delayed subtests compared with thecontrols. Additional validity studies have found thatadults with moderate-to-severe TBI perform signifi-cantly poorer on the Continuous Visual Memory Testcompared with controls (Strong & Donders, 2008) andscored approximately one third to three quarters of astandard deviations below the standardization sample(SS) on the California Verbal Learning Test-II(Jacobs & Donders, 2007). Studies on the CVLT-C havefound that both severity of injury and processing speedappear to mediate the impact of performance in childrenwith TBI (Donders & Minnema, 2004; Donders &Nesbit-Greene, 2004). These findings provide evidenceof general memory impairment in TBI cases and providesupport for the use of memory batteries when applied tochildren with TBI, although additional research on thesebatteries is necessary.

The TOMAL may be particularly salient for assessingneuropsychological function in youth with TBI, as it is ameasure of broad-band verbal and nonverbal memoryas well as narrow-band memory skills, attention, and

learning (Schmitt & Decker, 2009). The TOMAL hasbeen used to evaluate neurocognitive deficits in clinicalpopulations including attention-deficit hyperactivity dis-order (ADHD; Thaler, Allen, McMurray, & Mayfield,2010), reading disabilities (Howes, Bigler, Lawson, &Burlingame, 1999), genetic disorders (Lajiness-ONeillet al., 2005), and TBI (Lowther & Mayfield, 2004),

and its large SS and high reliability ratings have estab-lished the TOMAL as a psychometrically sound mem-ory battery (Dumont, Whelley, Comtois, & Levine,1994). However, only a few studies have examined theTOMALs psychometric properties in children withTBI. Lowther and Mayfield (2004) compared 70 chil-dren who had sustained a moderate-to-severe TBI with70 controls on their performance on the TOMAL. Theauthors found that with the exception of two NonverbalDelayed Recall subtests, children with TBI performedsignificantly lower on indexes and subtests comparedwith controls. Comparisons between children with mod-erate and severe injuries indicated trends in the expected

direction, with the severe group generally performingworse on subtests and indexes, although thesedifferences were not significant.

A recent study by Allen, Leany, and colleagues (2010)focused on neurocognitive heterogeneity in childrenwith TBI and identified six comparable TOMAL factorsin 150 children who had sustained TBI as well as 150controls matched for gender and age from theTOMALs SS. These factors were identified using prin-cipal components analysis for both TBI and controlsamples, and their structure differed from prior factorsolutions for the TOMAL (cf., Alexander & Mayfield,2005; Reynolds & Bigler, 1994, 1996; Thaler et al.,

2010), possibly due to differences in the types of analysesused, TOMAL subtests included in the analyses, thepopulations under investigation, or some combinationof these factors. In any case, the factor structure ident-ified by Allen, Leany, et al. (2010) approached simplestructure, was interpretable, and was stable across TBIand control groups. As found in Lowther and Mayfield(2004), significant differences were present between con-trols and TBI groups on the TOMAL subtest and indexscores, and as expected, similar differences were presentfor factor score comparisons. Thus, the study by Allen,Leany, et al. (2010) confirmed and extended the resultsreported by Lowther and Mayfield, primarily by includ-ing a larger sample and by examining factor score differ-ences. In addition, while the extant findings support theTOMALs sensitivity to childhood TBI, further investi-gation of its subtest, composite, and factor scoressensitivities to TBI is warranted.

Signal detection theory has found wide applicationin psychology through receiver-operating characteristic(ROC) analyses (Swets, 1996). More specifically, ROCanalyses assess the discriminative ability of test scores

DIFFERENTIAL SENSITIVITY OF TOMAL 169

8/12/2019 Tomal Roc 2011

3/12

in classification studies using a variety of instrumentsand clinical populations (Basic et al., 2009; Horwitz,Lynch, McCaffrey, & Fisher, 2008; Sagen et al., 2009;Thaler et al., 2010). Therefore, the purpose of thepresent study was to investigate scores at multiple levelson the TOMAL in children who sustained TBI by calcu-lating and comparing sensitivity and specificity rates of

scores across subtests, indexes, and factor scores usingROC analyses. While prior research suggests that mostif not all subtests will differentiate those with TBI fromnormal controls, differing sensitivities and specificitiesamong the scores may also be hypothesized. Forexample, a meta-analysis of 28 publications on pediatricTBI (Babikian & Asarnow, 2009) suggests that attentionand verbal memory appear to be the most impacted byTBI, while visual memory and visual perception arethe most resilient. In consideration of this meta-analysisand other criterion validity studies on memory batteries(Farmer et al, 1999; Jacobs & Donders, 2007; Lowther &Mayfield, 2004; Strong & Donders, 2008), we hypothe-

sized that given that our sample sustained moderate-to-severe injuries, the TBI group would perform betweenone and two standard deviations below the controlgroup. We also hypothesized that the Verbal MemoryIndex (VMI) and Attention=Concentration Index(ACI) would have greater classification rates thanthe TOMALs Nonverbal Memory Index (NMI).Finally, we compared the six factors identified by Allen,Leany, and colleagues (2010) and anticipated thatan attention factor would be most sensitive to a diag-nosis of TBI, followed by verbal factors, and then thenonverbal factors.

METHOD

Participants

Participants included 150 children who had sustainedTBI (TBI group) and 150 children with no history ofTBI (control group) who were studied in the Allen,Leany, et al. (2010) study. Children in the TBI groupwere selected from a consecutive series of cases referredfor neuropsychological evaluation. Structural braindamage resulting from TBI was confirmed using appro-priate neuroimaging, laboratory, and examination find-ings. On average, injuries were severe in nature asindicated by Glasgow Coma Scale (GCS) scores thatwere available for 101 of our participants with TBI(median 7). Children were selected for the TBI groupif they had a diagnosis of TBI and no other neurologicalor neurodevelopmental disorders (including stroke,ADHD, and anoxic brain injury) and if they had beenadministered the TOMAL as part of their evaluation.Participants in the control group were selected from

the TOMAL SS to match the TBI group on age andgender. When more than one case matched a TBI par-ticipant on age and sex, one case from the viable caseswas included via random selection. Using these criteria,133 participants were matched on age and sex, with theremaining cases randomly selected from the SS. Demo-graphic and clinical information for both groups is

available in Table 1. Comparisons between the groupsfound no significant differences due to age, F(1,299) 1.68, p .20, or gender, v2 .66,p .42.

Measures

The TOMAL (Reynolds & Bigler, 1994) is a broad mea-sure of memory, learning, and attention for childrenaged 5 to 19 years and is composed of 10 core and 4 sup-plementary subtests. Core subtests include Memory forStories (MFS), Facial Memory (FM), Word SelectiveReminding (WSR), Visual Selective Reminding (VSR),Object Recall (OR), Abstract Visual Memory (AVM),Digits Forward (DF), Visual Sequential Memory(VSM), Paired Recall (PR), and Memory for Location(MFL). Four of these subtests have a delayed compo-nent including Memory for Stories Delayed (MFSD),Facial Memory Delayed (FMD), Word SelectiveReminding Delayed (WSRD), and Visual SelectiveReminding Delayed (VSRD). Supplemental subtestsinclude Letters Forward (LF), Digits Backward, LettersBackward, and Manual Imitation (MI). Subtests have astandard score mean of 10 and a standard deviation of 3

TABLE 1

Demographic and Clinical Information

Variables

TBI group

(n 150)Control Group

(n150)

Age M11.7 (SD 3.7) M11.5 (SD3.1)Months Since Injury M7.0 (SD3.1)

% Male 57.3 52.7% Ethnicity

Caucasian 54.5 74.7

African American 22.8 11.0Hispanic=Latino 14.0 8.9

Asian American 0.7 3.4Other 0.7 2.1

% Closed Head Injury 93.3

% Mechanism of InjuryMotor VehicleAccident

56.7

Struck by Motor

Vehicle

20.0

Gunshot 4.0Fall 2.7

4-Wheeler Accident 5.3

Bike Accident 2.2Skiing 3.3Other 6.0

170 THALER ET AL.

8/12/2019 Tomal Roc 2011

4/12

and combine to produce several indexes including theVMI, the NMI, a Composite Memory Index (CMI), aDelayed Recall Index (DRI), and the ACI. All indexeshave a mean of 100 and a standard deviation of 15. A

more recent version (TOMAL-2; Reynolds & Voress,2007) contains the same subtests and the same organiza-tion of subtests into scales and indexes, and has morethan 90% item overlap with the TOMAL.

Data Analysis

Comparisons initially were made between the SS andTBI groups on demographic variables to determine thesuccess of group matching based on the case selectionmethod. To establish the sensitivity of the TOMAL toTBI, SS and TBI between-group comparisons weremade using multivariate analysis of variance(MANOVA) for the TOMAL subtest and index scores,with post-hoc univariate analyses used to examine spe-cific subtest and composite score differences betweenthe groups when overall MANOVAs were significant.

TOMAL indexes were then compared between theTBI group and control group using ROC curves toestablish the sensitivity, specificity, and optimal cutoffscores of TOMAL subtests and indexes for TBI. Inaddition, ROC analyses were used to compare the two

groups using the factor scores derived from thesix-factor structure first identified by Allen, Leany, andcolleagues (2010). In that study, the first factor (atten-tion) was composed of the digit and letter span tasks

and primarily assessed attention=concentration. Thesecond factor (verbal memory) consisted of the WSR,WSRD, PR, and OR subtests. The third factor (memoryfor stories) also assessed verbal memory and was com-posed of MFS and MFSD. The fourth factor (nonverballearning) consisted of nonverbal learning subtestsincluding VSR and VSRD, while the fifth factorassessed facial memory with the FM and FMD subtests.Finally, the sixth factor (abstract nonverbal) was com-posed of abstract nonverbal subtests including AVMand MFL. ROC analyses also were conducted for thesesix factors because, unlike the TOMAL SupplementaryIndex scores, the factors, like the primary TOMALindexes, share no subtests in common and so may bemore specific and sensitive than either the Supplemen-tary Index or subtest scores. As in Allen, Leany, et al.(2010), factor scores were calculated by summing thestandard scores of each factors subtests and then calcu-lating the average of these subtests. Prior studies (e.g.,Allen, Strauss, Kemtes, & Goldstein, 2007) have demon-strated that such methods produce factor scores compa-rable to regression-based scores calculated through

TABLE 2

TOMAL Data for the Standardization Sample (SS) and Traumatic Brain Injury (TBI) Groups

SS (N150) TBI (N150)F

Variable Mean SD Mean SD (df1,298) d

TOMAL Subtest ScoresMemory for Stories 10.6 2.5 8.1 3.0 39.2 0.91

Word Selective Reminding 10.1 2.8 7.8 3.4 27.9

0.74Object Recall 9.8 3.0 5.8 3.3 29.7 1.27

Digits Forward 9.8 3.6 6.7 2.6 6.0 0.99Paired Recall 10.0 3.0 7.4 3.8 25.0 0.76Letters Forward 9.9 3.2 6.7 2.7 7.3 1.08

Digits Backward 10.9 3.6 8.2 2.3 16.5 0.89Letters Backward 10.4 3.7 7.9 2.6 13.7 0.78

Facial Memory 10.4 2.8 7.6 3.0 13.3 0.97Visual Selective Reminding 10.2 3.5 6.5 3.2 16.7 1.10

Abstract Visual Memory 10.1 3.0 7.3 3.5 36.1 0.86Visual Selective Reminding 10.5 2.9 8.3 2.7 12.6 0.79Memory for Location 9.8 4.0 7.7 4.3 31.0 0.51

Memory for Stories Delayed 9.9 2.8 6.5 3.3 40.2 1.11Facial Memory Delayed 9.6 2.5 8.9 2.4 5.7 0.29Word Selective Reminding Delayed 9.5 2.4 7.8 2.9 30.0 0.64

Visual Selective Reminding Delayed 9.7 1.7 8.4 2.3 19.6 0.64

TOMAL Index ScoresVerbal Memory Index 100.4 13.4 80.4 16.1 73.3 1.37Nonverbal Memory Index 101.0 13.2 82.6 15.4 90.8 1.31

Composite Memory Index 101.0 12.1 80.8 14.8 130.7 1.49Delayed Recall Index 97.8 8.8 86.0 12.9 48.9 1.06

Attention=Concentration Index 101.0 18.9 82.6 13.2 16.9 1.13

Bonferroni correction p < .0023 for significance at the .05 level.


8/12/2019 Tomal Roc 2011

5/12

principal component or maximum likelihood factoranalyses but have the advantage of retaining similarcharacteristics to the original TOMAL standard scores.This also is a reasonable approximation to unit weight-ing in the calculation of a factor score and is more likelyto generalize outside of the sample.

Procedure

The TOMAL was administered as part of a clinicalneuropsychology evaluation at a pediatric restorativecare facility by either a pediatric neuropsychologistor trained doctoral-level graduate students undersupervision of the neuropsychologist.

RESULTS

Subtest and Index Differences

The SS groups performance on the TOMAL subtestand index scores approximated the SS mean, and theTBI group performed approximately 1.3 standard devia-tions below the SS group on most TOMAL scores.Although introducing some redundancy into the analy-ses, we tested group differences for both subtest andindex scores because of a relative lack of informationon TOMAL performance in children with TBI. MANO-VAs indicated significant differences between the groupson the subtest scores, F(68, 528) 5.59, p< .001,g

2p .419, and index scores, F(68, 528) 12.4, p< .001,

g2p .31. Following Bonferroni correction for multiple

comparisons, significant differences were present

between the TBI and SS groups on all of the TOMALscores (see Table 2).

ROC analyses were then conducted to determine thesensitivity and specificity of TOMAL subtest and indexscores to brain damage. The area under the ROC curve(AUC) serves as an indicator of a test scores ability todistinguish groups, with an AUC of 1.00 indicating per-fect classification and an AUC of 0.50 indicating a rateno greater than chance. Therefore, the more an AUCapproaches 1.00, the better overall classification rate ofthe test, and AUCs between 0.80 and 0.90 are regardedas good classification accuracy (Hosmer & Leme-show, 2000). By comparing AUCs for different tests,in this case the TOMAL scores, significant improvementin classification can be identified. The method describedby Hanley and McNeil (1983) was used to make pair-wise comparisons and detect significant differencesbetween AUCs of the various subtest, index, and factorscores.

Table 3 contains the AUCs, standard error of thecurves, and 95% confidence intervals for 17 TOMALsubtests (the MI subtest was not included in the analysis

because it was infrequently administered due to thelevels of motor impairment in much of the TBI sample)as well as pairwise contrasts for the six most sensitivesubtests. Inspection of the curves indicates that theOR subtest had the highest AUC at 0.82, a good classi-fication rate, followed by the MFSD, VSR, LF, DF, andFM subtests. See Figure 1 for subtest ROC curves.

Next, classification accuracy was calculated for thetop six subtests scores using discriminant functionanalysis (DFA). DFA indicated that the OR subtest cor-rectly classified 76.0% of the cases, the MFSD subtest72.0% of the cases, the VSR subtest 71.0% of the cases,the DF subtest 71.0% of the cases, the LF subtest 68.7%of the cases, and the FM subtest 68.0% of the cases.Pairwise comparisons indicated that the ORs AUCwas not significantly greater than the MFSD, VSR,LF, and DF subtests but was significantly greater than

TABLE 3

ROC Analyses for the TOMAL Subtest Scores (Subtests Ordered

from Greatest to Least AUC)

Subtest AUC SE 95% CI of Area

Object Recall 0.82 .024 0.770.87

Memory for Stories Delayed 0.78 .026 0.730.83Visual Selective Reminding 0.78 .026 0.730.83

Letters Forward 0.77 .027 0.720.82Digits Forward 0.76 .028 0.700.81Facial Memory 0.75 .028 0.700.81Memory for Stories 0.74 .029 0.680.79

Digits Backward 0.72 .029 0.660.78Abstract Visual Memory 0.71 .029 0.660.77

Visual Sequential Memory 0.71 .030 0.650.77Letters Backward 0.71 .030 0.650.77

Word Selective Reminding 0.70 .030 0.640.76Paired Recall 0.69 .030 0.630.75Word Selective Reminding Delayed 0.68 .031 0.620.74

Visual Selective Reminding Delayed 0.68 .031 0.620.74Memory for Location 0.64 .032 0.580.71

Facial Memory Delayed 0.57 . 033 0.510.64

Contrasts z score SE p

OR vs. MFSD 1.16 .031 .25

OR vs. VSR 1.30 .030 .19OR vs. LF 1.47 .031 .14

OR vs. DF 1.83 .032 .07OR vs. FM 2.12 .031 .03MFSD vs. VSR 0.11 .035 .91

MFSD vs. LF 0.28 .035 .78MFSD vs. DF 0.67 .035 .50

MFSD vs. FM 0.90 .034 .37VSR vs. LF 0.19 .032 .85

VSR vs. DF 0.58 .035 .57VSR vs. FM 0.74 .036 .46LF vs. DF 0.56 .025 .58

LF vs. FM 0.58 .035 .57

DF vs. FM 0.17 .039 .86

ORObject Recall; MFSDMemory for Stories Delayed;

VSRVisual Selective Reminding; LFLetters Forward; DFDigits Forward; FMFacial Memory.

172 THALER ET AL.

8/12/2019 Tomal Roc 2011

6/12

the FM subtest. Therefore, it can be inferred that thefirst five subtests have comparable classification accura-cies with each other while the FM is significantly lessaccurate than the OR subtest. When the first five sub-tests were analyzed together, they correctly classified82.0% of the cases.

Next, ROC curves were examined for TOMAL indexscores. Table 4 displays the AUC curves, standarderrors, and confidence intervals for the VMI, NMI,

CMI, DRI, and ACI scores.As seen in the table, the CMI had the highest AUC at

0.857. Pairwise comparisons indicated that the CMIsAUC was significantly greater than the VMIs AUC.However, the VMIs AUC was not significantly greaterthan the NMIs AUC nor was it significantly greaterthan the ACIs AUC. The VMIs AUC was significantlygreater than the DRIs AUC. However, the NMI, ACI,and DRI did not significantly differ from each other.Figure 2 displays the five index scores ROC curvesand their comparisons.

DFA next calculated overall classification rates forthe indexes. The CMI correctly classified 77.3% of thecases, the VMI 75.3% of the cases, the NMI 72.0% ofthe cases, the ACI 73.5% of the cases, and theDRI 71.3% of the cases. When all five indexes wereselected as predictors, they correctly classified 76.9% ofthe cases.

Table 5 contains sensitivity and specificity estimates,and true positives (TP), true negatives (TN), false posi-tives (FP), and false negatives (FN) for the CMI indexscore curve presented in Figure 2. The CMI was selected

TABLE 4

ROC Analyses for the TOMAL Index Scores (Indexes Ordered from

Greatest to Least AUC)

Index AUC SE 95% CI of Area

Composite Memory 0.86 .021 0.820.90

Verbal Memory 0.83 .024 0.780.88Nonverbal Memory 0.82 .024 0.770.87

Attention=Concentration 0.80 .026 0.750.86Delayed Recall 0.76 .028 0.710.82


VMI vs. NMI 0.66 .024 .50

VMI vs. CMI 2.16 .011 .03VMI vs. DRI 2.52 .024 .01

VMI vs. ACI 1.02 .025 .31NMI vs. CMI 2.90 .014

8/12/2019 Tomal Roc 2011

7/12

for display, as it had the best AUC of all indexes. Thereis a tradeoff between sensitivity and specificity in that asone increases, the other typically decreases. Therefore,the optimal cutoff score is best determined by addingsensitivity and specificity percentages of each score andseeing which pair yields the highest sum. TP, TN, FP,and FN represent the actual number of cases in eachcategory.

As seen in the table, the optimal cutoff score for theCMI is 83, and this had the highest TP rate for TBI,with the smallest tradeoff in FP rate. Additional cutoff

rates were examined for the VMI, NMI, ACI, andDRI. The VMI detected the most TBI cases at a cutoffscore of 89, the NMI at a score of 89, the ACI at a scoreof 92, and the DRI at a score between 90 and 92.

To determine the likelihood that a child with such aprofile would have TBI, children who had TOMALindex scores below all established cutoff points werefurther investigated. Of the original 150 TBI children,57 (38%) had profiles below the cutoff scores of the fiveindexes. Thirty-six of these children had GCS with amean of 5.9 (SD 2.4), capturing children in our samplewith severe injuries. Of the original 150 controls, only 4(7%) performed below the cutoff scores.

Factor Score Differences

The six factors first identified by Allen, Leany, andcolleagues (2010) were next analyzed using ROC curves.Table 6 displays the AUCs, standard errors, confidenceerrors, and pairwise contrasts for the six factors.

As seen in the table, the attention factor had thegreatest AUC, followed by the verbal factor, which

TABLE 5

ROC Curve Details for the TOMAL Composite Memory Index

CMI Score Sn (%) Sp (%) TP TN FP FN

46 1.00 0.01 150 1 149 0

51 1.00 0.01 150 2 148 0

52 1.00 0.02 150 3 147 054 1.00 0.03 150 5 145 0

55 1.00 0.04 150 6 144 056 1.00 0.05 150 8 142 057 1.00 0.07 150 10 140 058 1.00 0.07 150 11 139 0

59 1.00 0.10 150 15 135 061 1.00 0.11 150 17 133 0

62 1.00 0.12 150 18 132 063 0.99 0.13 148 19 131 2

64 0.99 0.14 148 21 129 265 0.99 0.15 148 22 128 266 0.99 0.17 148 25 125 2

67 0.99 0.19 148 29 121 268 0.99 0.21 148 32 118 2

69 0.99 0.24 148 36 114 270 0.99 0.26 148 39 111 271 0.98 0.31 147 47 103 3

72 0.98 0.35 147 52 98 3

73 0.98 0.35 147 53 97 374 0.98 0.36 147 54 96 375 0.98 0.37 147 55 95 3

76 0.97 0.41 146 62 88 477 0.97 0.42 146 63 87 479 0.97 0.50 145 75 75 5

80 0.96 0.51 144 76 74 681 0.96 0.52 144 78 72 6

82 0.95 0.55 143 82 68 783 0.95 0.55 142 83 67 8

84 0.92 0.57 138 86 64 1285 0.91 0.59 136 88 62 1486 0.88 0.60 132 90 60 18

87 0.85 0.62 128 93 57 22

88 0.85 0.63 128 95 55 2289 0.84 0.71 126 106 44 24

90 0.83 0.72 124 108 42 2691 0.80 0.76 120 114 36 30

92 0.76 0.76 114 114 36 3693 0.72 0.78 108 117 33 42

94 0.69 0.81 103 122 28 4795 0.67 0.83 101 124 26 49

96 0.64 0.85 96 128 22 5497 0.63 0.87 94 130 20 5698 0.62 0.89 93 133 17 57

99 0.57 0.89 85 134 16 65100 0.55 0.91 83 136 14 67101 0.52 0.93 78 139 11 72

102 0.49 0.93 74 140 10 76

103 0.44 0.95 66 142 8 84104 0.39 0.97 58 146 4 92105 0.34 0.97 51 146 4 99

106 0.28 0.98 42 147 3 108107 0.27 0.98 41 147 3 109108 0.27 0.99 40 148 2 110

109 0.25 0.99 38 148 2 112110 0.23 0.99 34 148 2 116

111 0.19 0.99 28 148 2 122

(Continued)

TABLE 5

Continued

CMI Score Sn (%) Sp (%) TP TN FP FN

112 0.16 0.99 24 148 2 126113 0.14 0.99 21 148 2 129114 0.11 0.99 16 148 2 134

115 0.10 0.99 15 148 2 135

116 0.08 0.99 12 148 2 138117 0.07 0.99 10 148 2 140118 0.06 0.99 9 148 2 141

120 0.05 0.99 8 149 1 142121 0.04 1.00 6 150 0 144124 0.03 1.00 5 150 0 145

126 0.01 1.00 2 150 0 148128 0.00 1.00 0 150 0 150

CMIComposite Memory Index; SnSensitivity; SpSpecificity; TPTrue Positives; TNTrue Negatives; FPFalse

Positives; FN False Negatives.Specificity and sensitivity are reported in decimal form, while TP,

TN, FP, and FN are the number of individuals in the TBI group

(n 150) or control group (n160) who are correctly or incorrectlyclassified.

174 THALER ET AL.

8/12/2019 Tomal Roc 2011

8/12

did not have a significantly smaller AUC. The attentionand verbal factors had significantly larger AUCs com-pared with facial memory and abstract nonverbal

factors, but otherwise, factors did not differ fromeach other. See Figure 3 for the ROC curves of the sixfactors.

DFA indicated that the attention factor correctlyclassified 76.0% of the cases, the verbal factor 71.0%of the cases, the memory for stories factor 70.7% ofthe cases, the nonverbal learning factor 67.3% of thecases, the abstract nonverbal factor 65.0% of the cases,and the facial memory factor 63.3% of the cases. Allsix factors correctly predicted 77.3% of the cases.

DISCUSSION

This study provides additional evidence concerning thepsychometric properties of the TOMAL when used toevaluate children with TBI by identifying those subtests,indexes, and factor scores that are most sensitive tobrain damage via ROC analyses. Similar to previousreports (e.g., Lowther & Mayfield, 2004), the currentTBI sample obtained universally lower scores than thecontrols on the TOMAL subtests and indexes. The

DRI had the smallest difference between the two groups,although the TBI group still scored approximately twothirds of a standard deviation below the controls on thisindex.

ROC analyses of the TOMAL subtests, indexes, and

factors indicated that OR scores were the most sensitiveto TBI, with the greatest AUC and DFA classificationaccuracy, although the AUC for OR was not signifi-cantly greater than the four subtests with the next high-est AUCs. The OR subtests high ROC curve may berelated to its content; when administering this subtest,examiners show a series of pictures and verbally namethe pictures for the participants and subsequentlyprompt the participants to verbally recall the stimuli ata later time. Therefore, this subtest relies on both visualand verbal memory processing to complete successfully.Left and right hippocampal damage is significantly asso-ciated with verbal and nonverbal memory deficits,respectively (Ariza et al., 2006), and furthermore, diffuseaxonal injury may result in poor interhemispheric trans-fer of information in moderate and severe TBI cases(Fork et al., 2005; Salorio et al., 2005). As our samplewas predominantly classified with moderate andsevere injuries, it is likely that both intrahemisphericand interhemispheric damage occurred in a majority ofthe children, who consequently had memory-processingdifficulties in both verbal and nonverbal modalities. The

FIGURE 3 ROC curves of the six factors.

TABLE 6

ROC Analyses for the TOMAL Factor Scores (Factors Ordered from

Greatest to Least AUC)

Index AUC SE 95% CI of Area

AF 0.80 .026 0.750.85

VF 0.78 .027 0.720.83MFSF 0.77 .027 0.720.82

NLF 0.77 .027 0.710.82FMF 0.70 .030 0.640.75ANF 0.71 .030 0.650.77


AF vs. VF 0.99 .029 .32AF vs. MFSF 1.06 .033 .29

AF vs. NLF 1.27 .031 .20AF vs. FMF 3.05 .036

8/12/2019 Tomal Roc 2011

9/12

concept that tasks that rely on both verbal and visualprocessing are the most sensitive to TBI is also sup-ported by the CMI, which includes both verbal and non-verbal subtests in a composite score and is the indexwith the greatest AUC in our study.

Other sensitive subtests include the MFSD, VSR,LF, and DF. With the exception of the LF and DF

subtests, these subtests share little in common in con-tent, method of administration, and type of memoryassessed. The MFSDs high AUC is of interest, as otherdelayed subtests of stimuli (WSRD, VSRD, and FMD)were relatively insensitive to TBI. Lowther and May-field (2004) found that the VSRD and FMD subtestswere the only ones for which scores did not significantlydiffer from the controls, and we have found similar find-ings in our sample. Perhaps most striking is the separ-ation between the VSR and VSRD, and the FM andFMD subtests, as the immediate subtests had relativelyhigh AUCs (0.78 and 0.75), while the delayed subtestsdetected brain injury at a rate only slightly better than

chance. Other studies have found that adults with TBIperformed significantly lower than controls on a num-ber of visual learning tasks but not on tasks of delayedretention as measured by the Shum Visual LearningTest (Shum, Harris, & OGorman, 2000), while childrenwith moderate or severe TBI did not significantly differfrom controls on the WRAML Visual LearningDelayed subtest yet differed on numerous other subtestsincluding the Visual Learning Immediate and VerbalLearning Delayed subtests (Farmer et al., 1999).Whether or not these findings are truly indicative ofan immediate=delayed nonverbal difference in TBIpopulations requires further and more rigorous

evaluation.TOMAL index scores also exhibit differential sensi-

tivity to TBI. As previously stated, the CMI standsout as the most sensitive index and highlights the globalimpairments of brain damage on both verbal and non-verbal memory tasks. The VMI, NMI, and ACI all haveAUCs that did not significantly differ from each other,indicating that when verbal, nonverbal, and attention=concentration tasks are considered separately, they havecomparable sensitivity to brain damage. This is contraryto our hypothesis that the VMI and ACI would be moresensitive than the NMI to brain damage. It may be thatthe CMI has the largest AUC specifically because it iscomposed of subtests from these three indexes, therebyaccounting for the heterogeneous cognitive deficienciesthat may be present in brain injury. The DRI has anAUC that was significantly lower than the CMI andVMIs AUC, implicating it as the index that correctlyclassified the fewest TBI cases. This is a reflection ofthe relatively insensitive delayed recall subtests, as onlythe MFSD subtest had an AUC approaching a goodclassification status.

Optimal cutoff index scores were estimated by sum-ming the sensitivity and specificity percentiles of eachTOMAL index for each available score. From this, wefind that the CMI has the best classification accuracyat a score of 83, which is approximately one standarddeviation below the mean. At this score, 94.7% of theTBI cases were correctly identified, while 55.3% of the

control cases were correctly excluded. The high TP yetcomparatively weak TN rate provides insight on the nat-ure of neurocognitive profiles in pediatric TBI, though italso establishes that the TOMAL by itself is not suf-ficient to accurately rule out controls. When all parti-cipants with scores below the five indexes cutoffscores were examined, nearly all had a diagnosis ofTBI, though the majority of the TBI cases did not meetthese criteria. This low-sensitivity=high-specificity ratehas been found in other screening tools for brain dam-age (e.g., Horwitz et al., 2008) and suggests that veryfew unimpaired youth would score below the indexescutoff scores, although several brain-injured youth

might score above the cutoffs on at least one of theindexes.

The TOMAL factors have the advantage of repre-senting purer constructs than the indexes, and indeed,the rank order of AUC curves agree with the existingliterature on domains most impaired by TBI. Numerousstudies have evidenced that attention is adversely affec-ted in brain injury (Anderson, Catroppa, Morse,Haritou, & Rosenfeld, 2005; Catroppa, Anderson,Morse, Haritou, & Rosenfeld, 2008; Dennis et al.,2001; Thaler, Allen, Park, McMurray, & Mayfield,2010; Willmott, Ponsford, Hocking, & Schonberger,2009), while the meta-analysis of outcomes in pediatric

TBI has suggested that nonverbal memory is more pre-served in TBI than verbal memory is (Babikian &Asarnow, 2009). In our study, the attention factor hadthe largest AUC, followed by the two verbal factorsand then the three nonverbal factors. Though the atten-tion factor and verbal factors did not significantly differin AUC curves, the general trends fit with our hypoth-esis on memory performance. That the index scoresand the factors differed on AUC rank order may bebecause the factors represent independent and quantitat-ively derived constructs directly from the data set whilethe indexes were based on theoretical knowledge ofmemory and have some overlap with each other; forexample, the Digit Span subtest is shared by both theVMI and the ACI. For this reason, the ROC analyseson the index scores may be best viewed from a clinicalperspective in that the CMI may have the most sensi-tivity to TBI, while the analyses on the factor scoresprovide further insight on the cognitive domains mostaffected by brain damage.

Although these findings are promising and provideinsight regarding the TOMALs sensitivity to TBI, some

176 THALER ET AL.

8/12/2019 Tomal Roc 2011

10/12

limitations should be addressed. First,most of oursamplesustained moderate-to-severe brain damage, and so subt-est and index score classification accuracies for mild TBIcases remain unknown. This is an issue, as recent litera-ture suggests that mild pediatric cases also exhibit cogni-tive impairment (Catroppa & Anderson, 2007; Hessen,Nestvold, & Anderson, 2007). Further, all children were

assessed between 3 and 21 months after injury, andfurther longer-term outcomes for our sample were una-vailable. Finally, the factor structure obtained fromAllen, Leany, and colleagues (2010) may not be generaliz-able to other samples, although this maynot be as much ofan issue because the factors chiefly served to identify dis-tinct components of attention and verbal and nonverbalmemory abilities. Regardless, the examination of the fac-tors indicated differential impairment in the constructsthey assessed in directions that agree with previous find-ings (e.g., Babikian & Asarnow, 2009). Finally, the cur-rent sample was one of convenience, and participantswere selected for inclusion in the current study based on

whether or not they had been referred for clinical neurop-sychological evaluation following TBI.

The current study used ROC analyses to establish theTOMAL subtests, indexes, and factors that are mostsensitive to brain damage. The overall findings suggestthat the OR subtest and CMI are the most sensitive, whiledelayed tasks of visual memory are the least affected.When factors were considered independently of indexcomposition, the attention factor emerged as most affec-ted by brain injury, followed by verbal memory factorsand finally nonverbal memory factors. A further investi-gation of the TOMAL-2s (Reynolds & Voress, 2007) sen-sitivity to brain damage when used for adults is warranted,

given that the TOMAL-2 was standardized to include anadult sample. Neither the TOMAL nor the TOMAL-2provide clinical norms for TBI populations, and so thefindings from this and other studies (Allen, Leany, et al.,2010; Lowther & Mayfield, 2004) provide the greatestinsight on TOMAL profile performance at this time.

ACKNOWLEDGEMENTS

Normal comparison data were from the TOMAL stan-dardization sample. Copyright 1994 by PRO-ED,Austin, TX. Used with permission. All rights reserved.

We thank the TOMAL publisher for allowing accessto the standardization data.

REFERENCES

Alexander, A. I., & Mayfield, J. (2005). Latent factor structure of the

Test of Memory and Learning in a pediatric traumatic brain-injuredsample: Support for a general memory construct. Archives ofClinical Neuropsychology, 20, 587598.

Allen, D. N., Knatz, D., & Mayfield, J. (2006). Validity of the chil-drens category test-level 1 in a clinical sample with heterogeneous

forms of brain dysfunction. Archives of Clinical Neuropsychology,21(7), 711720.

Allen, D. N., Leany, B. D., Thaler, N. S., Cross, C., Sutton, G. P., &

Mayfield, J. (2010). Memory and attention profiles in pediatric trau-matic brain injury. Archives of Clinical Neuropsychology,25, 618633.

Allen, D. N., Strauss, G. P., Kemtes, K., & Goldstein, G. (2007).Hemispheric contributions to nonverbal abstract reasoning and

problem solving. Neuropsychology, 21, 713720.Allen, D. N., Thaler, N. S., Donohue, B., & Mayfield, J. (2010).

WISC-IV profiles in children with traumatic brain injury: Similari-

ties to and differences from the WISC-III.Psychological Assessment,22(1), 5764.

Anderson, V., Catroppa, C., Morse, S., Haritou, F., & Rosenfeld, J.

(2005). Attentional and processing skills following traumatic braininjury in early childhood. Brain Injury, 19, 699710.

Ariza, M., Serra-Grabulosa, J., Junque, C., Ramrez, B., Mataro, M.,Poca, A.,. . .Sahuquillo, J. (2006). Hippocampal head atrophy after

traumatic brain injury. Neuropsychologia, 44(10), 19561961.Babikian, T., & Asarnow, R. (2009). Neurocognitive outcomes and

recovery after pediatric TBI: Meta-analytic review of the literature.Neuropsychology, 23(3), 283296.

Basic, D., Khoo, A., Conforti, D., Rowland, J., Vrantsidis, F.,

LoGiudice, D.,. . .

Prowse, R. (2009). Rowland Universal DementiaAssessment Scale, Mini-Mental State Examination, and general

practitioner assessment of cognition in a multicultural cohort ofcommunity-dwelling older persons with early dementia. AustralianPsychologist, 44(1), 4053.

Calhoun, S. L., & Mayes, S. D. (2005). Processing speed in childrenwith clinical disorders. Psychology in the Schools, 42(4), 333343.

Catroppa, C., & Anderson, V. (2002). Recovery in memory function inthe first year following TBI in children. Brain Injury,16(5), 167175.

Catroppa, C., & Anderson, V. (2007). Recovery in memory function,

and its relationship to academic success, at 24 months followingpediatric TBI. Child Neuropsychology, 13, 240261.

Catroppa, C., Anderson, V., Morse, S., Haritou, F., & Rosenfeld, J.(2008). Outcome and predictors of functional recovery 5 years fol-

lowing pediatric traumatic brain injury. Journal of Pediatric

Psychology, 33(7), 707708.Delis, D., Kramer, J., Kaplan, E., & Ober, B. (1994).California Verbal

Learning TestChildrens version. San Antonio, TX: Psychological

Corporation.

Dennis, M., Guger, S., Roncadin, C., Barnes, M., & Schachar, R.(2001). Attention-inhibitory control and social behavioral regu-

lation after childhood closed head injury: Do biological, develop-mental, and recovery variables predict outcome? Journal of theInternational Neuropsychological Society, 7, 683692.

Donders, J., & Janke, K. (2008). Criterion validity of the WechslerIntelligence Scale for Children-Fourth Edition after pediatric trau-

matic brain injury. Journal of the International NeuropsychologicalSociety, 14, 651655.

Donders, J., & Minnema, M. (2004). Performance discrepancies on the

California Verbal Learning Test-Childrens Version in children withtraumatic brain injury.Journal of the International Neuropsychologi-cal Society, 10(4), 482488.

Donders, J., & Nesbit-Greene, K. (2004). Predictors of neuropsycholo-

gical test performance after pediatric traumatic brain injury.Assessment, 11(4), 275284.

Donders, J., Tulsky, D. S., & Zhu, J. (2001). Criterion validity of new

WAIS-III subtest scores after traumatic brain injury. Journal of theInternational Neuropsychology Society, 7(7), 892898.

Dumont, R., Whelley, P., Comtois, R., & Levine, B. (1994). Test of

Memory and Learning (TOMAL): Test review. Journal of Psychoe-ducational Assessment, 12, 414423.


8/12/2019 Tomal Roc 2011

11/12

Farmer, J., Haut, J., Williams, J., Kapila, C., Johnstone, B., & Kirk,K. (1999). Comprehensive assessment of memory functioning

following traumatic brain injury in children. DevelopmentalNeuropsychology, 15(2), 269289.

Fork, M., Bartels, C., Ebert, A., Grubich, C., Synowitz, H., &

Wallesch, C. (2005). Neuropsychological sequelae of diffusetraumatic brain injury. Brain Injury, 19(2), 101108.

Hanley, J. A., & McNeil, B. J. (1983). A method of comparing theareas under ROC curves derived from same cases. Radiology, 148,

839843.

Hessen, E., Nestvold, K., & Anderson, V. (2007). Neuropsychologicalfunction 23 years after mild traumatic brain injury: A comparison of

outcome after pediatric and adult head injuries. Brain Injury,21(9),963979.

Hooper, S. R., Alexander, J., & Moore, D. (2004). Caregiver reports of

common symptoms in children following a traumatic brain injury.NeuroRehabilitation, 19, 175189.

Horwitz, J., Lynch, J., McCaffrey, R., & Fisher, J. (2008). Screeningfor neuropsychological impairment using Reitan and Wolfsons

preliminary neuropsychological test battery. Archives of Clinical

Neuropsychology, 23(4), 393398.

Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regression(2nd ed.). New York, NY: John Wiley & Sons.

Howes, N. L., Bigler, E. D., Lawson, J. S., & Burlingame, G. M.

(1999). Reading disability subtypes and Test of Memory andLearning.Archives of Clinical Neuropsychology, 14(3), 317339.

Jacobs, M. L., & Donders, J. (2007). Performance discrepancies onthe California Verbal Learning Test-Second Edition after trau-

matic brain injury. Archives of Clinical Neuropsychology, 23(1),113118.

Lajiness-ONeill, R., Beaulieu, I., Titus, J. B., Asamoah, A., Bigler, E.

D., Bawle, E. V., & Pollack, R. (2005). Memory and learning in chil-dren with 22q11.2 deletion syndrome: Evidence for ventral and dor-sal stream disruption? Child Neuropsychology, 11, 5571.

Lowther, J., & Mayfield, J. (2004). Memory functioning in childrenwith traumatic brain 575 injuries: A TOMAL validity study.Archives of Clinical Neuropsychology, 19(1), 89104.

Max, J., Koele, S.,Smith,W., Sato, Y.,Lindgren,S., Robin,D., & Arndt,

S. (1998). Psychiatric disorders in children and adolescents after severe

traumatic brain injury: A controlled study. Journal of the AmericanAcademy of Child & Adolescent Psychiatry,37(8), 832840.

Mayes, S., & Calhoun, S. (2004). Similarities and differences inWechsler Intelligence Scale for Children-Third Edition profiles:

Support for subtest analysis in clinical referrals. The Clinical Neu-ropsychologist , 18, 559572.

Reynolds, C. R., & Bigler, E. D. (1994).Test of Memory and Learning:

Examiners manual. Austin, TX: Pro-Ed.

Reynolds, C. R., & Bigler, E. D. (1996). Factor structure, factorindexes, and other useful statistics for interpretation of the Test of

Memory and Learning. Archives of Clinical Neuropsychology, 11,2943. doi:10.1016=0887-6177(95)0027-5

Reynolds, C. R., & Voress, J. K. (2007).Test of Memory and Learning(2nd ed.). Austin, TX: Pro-Ed.

Roberts, M. A., & Furuseth, A. (1997). Eliciting parental report fol-

lowing pediatric traumatic brain injury: Preliminary findings onthe Pediatric Inventory of Neurobehavioral Symptoms. Archivesof Clinical Neuropsychology, 12, 449457.

Sagen, U., Vik, T., Moum, T., Mrland, T., Finset, A., & Dammen, T.

(2009). Screening for anxiety and depression after stroke: Compari-son of the Hospital Anxiety and Depression Scale and theMontgomery and Asberg Depression Rating Scale. Journal of

Psychosomatic Research, 67(4), 325332.

Salorio, C., Slomine, B., Grados, M., Vasa, R., Christensen, J., &Gerring, J. (2005). Neuroanatomic correlates of CVLT-C perform-

ance following pediatric traumatic brain injury.Journal of the Inter-national Neuropsychological Society, 11(6), 686696.

Schmitt, A., & Decker, S. (2009). Review of Test of Memory andLearning: Second Edition. Journal of Psychoeducational Assess-ment, 27(2), 157166.

Sheslow, D., & Adams, W. (2003). Wide-range assessment of

memory and learning(2nd ed.). Lutz, FL: Psychological AssessmentResources, Inc.

Shum, D., Harris, D., & OGorman, J. (2000). Effects of severe trau-

matic brain injury on visual memory.Journal of Clinical and Experi-mental Neuropsychology, 22(1), 2539.

Strong, C., & Donders, J. (2008). Validity of the continuous visualmemory test after traumatic brain injury. Journal of Clinical andExperimental Neuropsychology, 30(8), 885891.

Swets, J. A. (1996). Signal detection theory and ROC analysis in psy-chology and diagnostics: Collected papers (scientific psychology ser-

ies). Mahwah, NJ: Psychology Press.

Thaler, N. S., Allen, D. N., McMurray, J. C., & Mayfield, J. (2010).Sensitivity of the Test of Memory and Learning to attention and

memory deficits in children with ADHD.The Clinical Neuropsychol-ogist, 24(2), 246264.

Thaler, N. S., Allen, D. N., Park, B. S., McMurray, J. C., & Mayfield,J. (2010). Attention-processing abnormalities in children with TBI

and ADHD: Differential impairment of component processes.Jour-

nal of Clinical and Experimental Neuropsychology, 32, 929936.Wassenberg, R., Max, J. E., & Lindgren, S. D. (2004). Sustained atten-

tion in children and adolescents after traumatic brain injury:Relation to severity of injury, adaptive functioning, ADHD, and

social background.Brain Injury, 18, 751764.Wechsler, D. (1995). Childrens Memory Scale. San Antonio, TX: The

Psychological Corporation.

Willmott, C., Ponsford, J., Hocking, C., & Schonberger, M. (2009).

Factors contributing to attentional impairments after traumaticbrain injury. Neuropsychology, 23(4), 424432.

178 THALER ET AL.

8/12/2019 Tomal Roc 2011

12/12

Copyright of Applied Neuropsychology is the property of Taylor & Francis Ltd and its content may not be

copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written

permission. However, users may print, download, or email articles for individual use.

tomal roc 2011

Documents