This article was downloaded by: [University of Notre Dame Australia]On: 27 April 2013, At: 11:24Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: MortimerHouse, 37-41 Mortimer Street, London W1T 3JH, UK

Journal of Clinical and ExperimentalNeuropsychologyPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/ncen20

The ecological validity of the Rey–OsterriethComplex Figure: Predicting everyday problems inchildren with neuropsychological disordersSimon R. Davies a b , Alan R. J. Field b , Thorvald Andersen b & Carmela Pestell ca Neurosciences Unit, Graylands Selby-Lemnos & Special Care Health Services,North Metropolitan Area Mental Health Service, Claremont, WA, Australiab Department of Psychology, Murdoch University, Murdoch, WA, Australiac Complex Attention & Hyperactivity Disorders Service (CAHDS), Joondalup, WA,AustraliaVersion of record first published: 07 Jul 2011.

To cite this article: Simon R. Davies , Alan R. J. Field , Thorvald Andersen & Carmela Pestell (2011): The ecologicalvalidity of the Rey–Osterrieth Complex Figure: Predicting everyday problems in children with neuropsychologicaldisorders, Journal of Clinical and Experimental Neuropsychology, 33:7, 820-831

To link to this article: http://dx.doi.org/10.1080/13803395.2011.574608

PLEASE SCROLL DOWN FOR ARTICLE

Full terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditions

This article may be used for research, teaching, and private study purposes. Any substantial orsystematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distributionin any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that thecontents will be complete or accurate or up to date. The accuracy of any instructions, formulae, anddrug doses should be independently verified with primary sources. The publisher shall not be liable forany loss, actions, claims, proceedings, demand, or costs or damages whatsoever or howsoever causedarising directly or indirectly in connection with or arising out of the use of this material.

JOURNAL OF CLINICAL AND EXPERIMENTAL NEUROPSYCHOLOGY2011, 33 (7), 820–831

The ecological validity of the Rey–Osterrieth ComplexFigure: Predicting everyday problems in children with

neuropsychological disorders

Simon R. Davies1,2, Alan R. J. Field2, Thorvald Andersen2, and Carmela Pestell3

1Neurosciences Unit, Graylands Selby-Lemnos & Special Care Health Services, North MetropolitanArea Mental Health Service, Claremont, WA, Australia2Department of Psychology, Murdoch University, Murdoch, WA, Australia3Complex Attention & Hyperactivity Disorders Service (CAHDS), Joondalup, WA, Australia

Despite its extensive use, the validity of the Rey Complex Figure in predicting everyday behavioral executive dys-function displayed by children with various neuropsychological disorders is unknown. The copied figures of 263children aged 6 to 17 years were rescored using an accuracy approach that measured visuospatial ability and threeprocess approaches developed to measure executive functioning. Controlling for age and IQ, partial correlationsbetween scores derived by all scoring methods and the parent ratings on an executive function inventory were zero.In contrast, all four scoring approaches were associated with parent ratings on questionnaires that indexed chil-dren’s academic achievement, developmental status, and adaptive ability. The findings suggest that the ecologicalvalidity of the Rey Complex Figure for children with various central nervous system disorders is more associatedwith visual-motor integration skills than executive functioning.

Keywords: Rey–Osterrieth Complex Figure; Ecological validity; Child neuropsychological disorders; Parent ratingquestionnaires; Behavioral executive dysfunction.

INTRODUCTION

Overview

The Rey–Osterrieth Complex Figure (RCF; Osterrieth,1946; Rey, 1941) is an instrument used extensively by clin-icians. Boone (2000) identified this over a decade ago. Henoted that it had been referenced in over 200 publica-tions at the time and that it was ranked the 21st mostused test in forensic neuropsychology. Rabin, Barr, andBurton’s (2005) recent survey of the assessment prac-tices of 747 clinical neuropsychologists suggests that itsuse remains high. The RCF was ranked 8th of the top40 neuropsychological instruments used by the clinicianssurveyed and 2nd of the top 40 executive functioninginstruments. Proponents of the RCF suggest that thepragmatic aspects of the RCF, such as its availability andease of administration, in addition to its measurementproperties, such as its dependence on visuoperceptual,

Address correspondence to Simon R. Davies, Neurosciences Unit, Graylands Selby-Lemnos & Special Care Health Services,North Metropolitan Area Mental Health Service, Private Bag No. 1, Claremont, Western Australia 6190, Australia. (E-mail:simon.davies@health.wa.gov.au).

visuoconstructional, graphomotor, and visual memoryfunctions and the executive functions of planning andorganization, are the main reasons for why it is highlyregarded (e.g., Baron, 2004; Lezak, Howieson, & Loring,2004; Strauss, Sherman, & Spreen, 2006).

To differentiate the specific cognitive processes thatcontribute to the successful or impaired performance ofboth children and adults, many different scoring methodshave been developed. These methods can be assigned totwo broad categories. Accuracy methods produce scoresthat reflect the degree to which figures drawn in the copyand recall conditions match the original design presentedto the child or adult. Accuracy is determined by assigningpoints to how well the various structural units of the RCFare drawn and placed in the overall design. These scoresgive some indication of someone’s visuospatial ability.Taylor’s (1959) and Waber & Holmes’s (1985) scoring sys-tems are perhaps the most well-known methods used toevaluate the accuracy of a child’s copy of the RCF.

© 2011 Psychology Press, an imprint of the Taylor & Francis Group, an Informa business

http://www.psypress.com/jcen DOI: 10.1080/13803395.2011.574608

Dow

nloa

ded

by [

Uni

vers

ity o

f N

otre

Dam

e A

ustr

alia

] at

11:

24 2

7 A

pril

2013

ECOLOGICAL VALIDITY OF THE REY COMPLEX FIGURE 821

Process methods produce scores on the ways in whichcopied and recalled figures are constructed in terms of thequality of order and direction and sequence with whichthe elements of the figures are drawn and placed. Theaim is to get an idea of whether the RCF was drawn inan organized and planned manner or in a fragmentedor confused way. A well-known scoring system used toassess the performance of children is the DevelopmentalScoring System (DSS; Bernstein & Waber, 1996). TheDSS has four variables that quantify performance—Organization, Style, Accuracy, Errors. The scoring prin-ciple for the Organization score is based on how wella child draws the major structural features of the RCF.The features include alignments (e.g., two segments ofa line are accurately aligned with another) and inter-sections (e.g., lines accurately intersect one another). Achild’s “appreciation for the organizational goodness ofcomplex, visually represented materials” (Bernstein &Waber, 1996, p. 2) is reflected in the Organization score.The RCF Organizational Strategy Score (RCF-OSS; P.Anderson, Anderson, & Garth, 2001) is another devel-oped comprehensive system also specifically designed forpediatric populations. It differs from the DSS in thatthe sequence in which structural elements of the RCFare drawn are assessed. Furthermore, the quantifica-tion of sequential organization is based on integratingthe scores assigned to the strategies used by a child tostart to draw the figure with the scores assigned to thestrategies that are used during the progression of thecopying, into a single variable. Other process approachesused to assess children’s executive functioning include theBoston Qualitative Scoring System (BQSS; Stern et al.,1999) and the Savage–Deckersbach adult scoring system(Deckersbach et al., 2000; Savage et al., 1999).

Despite the RCF’s continued widespread use and ongo-ing research into its properties, the RCF has limitations.There are plenty of normative data available for the var-ious scoring methods, and, for most of these methods,normal developmental trajectories have been plotted,with both accuracy and process-derived RCF scores con-sistently showing developmental trends indicating age-related improvements in visual–spatial ability and exec-utive functioning (e.g., Akshoomoff & Stiles, 1995; P.Anderson et al., 2001; V. Anderson, Anderson, Northam,Jacobs, & Catroppa, 2001; Bernstein & Waber, 1996;Frisk, Jakobson, Knight, & Robertson, 2005; Kirk, 1985;Mahone, Hagelthorn, et al., 2002; Nakano et al., 2006;Osterrieth, 1946; Waber & Holmes, 1985). However, crit-ics of the RCF point out that validity studies are rare. Inparticular, the extent to which scores derived by differentscoring methods predict the everyday real-world behav-ioral difficulties in children with central nervous systemproblems, otherwise known as the ecological validityof a test (e.g., Chaytor & Schmitter-Edgecombe, 2003;Chaytor, Schmitter-Edgecombe, & Burr, 2006; Sbordone,1996), has not been researched at all.

Validity of RCF scoring approaches

To evaluate the validity of accuracy scoring approaches,proponents of the various scoring methods have cor-related scores on the RCF with scores obtained from

performances on other neuropsychological tests of visu-ospatial function. Poulton and Moffitt (1995) testeda group of unselected nonclinical 13-year-olds andfound that their RCF scores, derived by the Waberand Holmes (1985) accuracy scoring method, correlatedhighly with the Performance Intelligence Quotient (PIQ)of the Wechsler Intelligence Scale for Children–Revised(WISC–R) and, in particular, with the Block Designand Object Assembly subtests. More recently, Demsky,Carone, Burns, and Sellers (2000) found that the size ofthe correlation between the RCF copy accuracy scoresderived using Taylor’s (1959) approach and scores onthe Developmental Test of Visual–Motor Integration–Third revision (Beery & Buktenica, 1989) obtained in alarge sample of 6- to 11-year-olds supported the RCF’sconcurrent and content validity.

The findings from studies that have investigated rela-tionships between RCF process scores and neuropsy-chological tasks of executive functions are mixed. P.Anderson et al. (2001) tested a large group of normallydeveloping children using the RCF-OSS and found thatscores correlated moderately with well-known executivefunction tests of planning and organization (i.e., Towerof London), verbal fluency (i.e., FAS), and set shifting(i.e., Trail Making Test; TMT). However, Watanabe et al.(2005) found in their group of children, comprising 5- to14-year-olds with a range of neurological disorders (e.g.,epilepsy, attention deficit/hyperactivity disorder, perva-sive developmental disorders, learning disorders), the cor-relations between the planning and organization scores ofthe BQSS and the number of perseverative errors madeon the Wisconsin Card Sorting Test (WCST) and thescores on the Mazes and Block Design subtests of theWechsler Intelligence Scale for Children–Third Edition(WISC–III) were not significant. Similarly, Beebe, Ris,Brown, and Dietrich (2004) did not find any relationshipbetween DSS scores of the RCF and various WCST andContinuous Performance Test indices.

In other approaches to establish the RCFs valid-ity, scores derived from various accuracy and processapproaches have been correlated with each other. Forexample, Harris, Bernstein, Waber, and Stern (1999)showed that there were moderate relationships betweenthe process scores of the BQSS and DSS as well asbetween some accuracy scores.

Evidence for validity has also been supported by stud-ies that have shown accuracy and process scores to be sen-sitive to perturbations of central nervous system (CNS)dysfunction in children. In many clinical studies, below-age-expected RCF scores have been produced by childrenwith reading disorders (Klicpera, 1983), learning difficul-ties (e.g., Kirkwood, Weiler, Bernstein, Forbes, & Waber,2001; Waber & Bernstein, 1995), acute lymphoblas-tic leukemia (e.g., Waber, Bernstein, Tarbell, & Sallan,1992; Waber et al., 1994), attention-deficit/hyperactivitydisorder (ADHD; e.g., Cahn et al., 1996; Mahone,Hagelthorn, et al., 2002; Seidman et al., 1995; Seidman,Biederman, Faraone, Weber, & Ouellette, 1997; Watanabeet al., 2005), bacterial meningitis (Grimwood et al., 1995),traumatic brain injury (e.g., V. Anderson & Catroppa,2005; Garth, Anderson, & Wrenall, 1997), and otherconditions such as epilepsy (e.g., Watanabe et al., 2005).

Dow

nloa

ded

by [

Uni

vers

ity o

f N

otre

Dam

e A

ustr

alia

] at

11:

24 2

7 A

pril

2013

822 DAVIES ET AL.

The relationship between RCF performance and gen-eral cognitive ability has also garnered some attention,which, in the context of this study, bears on divergentvalidity. Smith and Zahka (2006) found that the organi-zational RCF scores of the children and adolescents withvarious learning, behavioral, and attention deficit disor-ders, which were derived using the Savage–Deckersbachprocess scoring system, did not correlate at all with any ofthe ability scales or subtests of the Differential AbilitiesScale (Elliot, 1990). In contrast, an earlier study foundthat the RCF copy scores derived by the DSS in childrenwith ADHD have been found to be related to WISC–R IQs even after controlling for factors such as age,sex, and the presence of a learning disability (Mahone,Hagelthorn, et al., 2002).

Ecological validity of the RCF

Studies that have specifically investigated how well RCFscores predict how well children perform in their everydayreal-world activities, such as achieving success in schoolor functioning independently in the home or commu-nity, simply do not exist. Yet, researchers such as Mayes,Calhoun, Bixler, and Zimmerman (2009) and Orit, Dov,and Yair (2007) have shown that a child’s visual–motorintegration ability, as measured by their skill in being ableto draw and/or construct two- and three-dimensionalgeometric designs, is associated with academic achieve-ment and school-readiness. Furthermore, researchershave shown that poor performances on traditional stan-dardized tests of executive function (e.g., WCST, verbalfluency, TMT, etc.) displayed by children with traumaticbrain injuries and some developmental disorders, suchas ADHD, relate to everyday problems (e.g., Mangeot,Armstrong, Colvin, Yeates, & Taylor, 2002). Just howwell a child’s visuospatial skills and executive functions,as measured by how well he or she can accurately repro-duce or plan and organize a copy of the RCF, informs theclinician or researcher about the child’s ability to handlethe myriad of tasks that confronts him or her every day isnot yet known.

Mangeot et al. (2002) produced some data on thelatter issue, though the finding was unexpected. Theresearchers were interested in the long-term deficits inexecutive functions following childhood traumatic braininjury. As part of the executive function test battery,the researchers included the RCF and the BehaviorRating Inventory of Executive Function (BRIEF; Gioia,Isquith, Guy, & Kenworthy, 2000), which is question-naire that elicits the ratings of parents about their child’severyday executive function behavior problems. Mangeotet al. (2002) found that, contrary to expectations, scoresderived by the DSS (Bernstein & Waber, 1996) weresignificantly positively correlated with the BRIEF. Thefindings suggest that those children whose parents ratedthem the most problematic in their day-to-day behav-ioral executive functions were the children who producedthe better planned and organized RCFs. The data werereported and described but the unusual finding was notexplained.

Aims and hypotheses

The aim of this study is to examine whether RCF scoresderived by some accuracy and process approaches relateto parent descriptions of the day-to-day behavioral exec-utive function problems exhibited by children affected byvarious neuropsychological disorders. For a clinician tobe able to predict how well a child is able to function inhis or her everyday home and school environment fromthe administration and scoring of an inexpensive, short(in terms of duration of activity), and easily accessibletest would be a welcomed finding. Practically, such anoutcome could reduce testing time and the assessmentburdens placed on the child and the assessment agency.More importantly, such data could immediately informtreatments of choice by identifying those children strug-gling in their day-to-day activities who would benefitfrom compensatory strategies to address their executivedysfunction or by accommodating or remediating percep-tual or visual–spatial weaknesses (Akshoomoff & Stiles,1995).

On the basis that the process-approach-derived RCFscores reflect the executive functions of planning andorganization, it is expected that the children whose par-ents rate them the most behaviorally dysregulated wouldproduce piecemeal, randomly constructed (i.e., poorlyplanned and organized) copies of the RCF.

Hypothesis 1

Scores derived from the process methods used in thisstudy will negatively correlate with the several key indiceson the Behavior Rating Inventory of Executive Function(BRIEF; Gioia et al., 2000).

On the basis that the accuracy-approach-derived scoresreflect integrity of visual–spatial functions, it is expectedthat the children whose parents rate them as devel-opmentally delayed, uncoordinated, or having inad-equate abilities would produce inaccurate copies ofthe RCF.

Hypothesis 2

Scores derived from the accuracy method used inthis study will negatively correlate with the several keyadaptive behavior indices of the Personality Inventoryfor Children–Second Edition (PIC–2; Lachar & Gruber,2001), but not correlate at all with any indices of theBRIEF.

METHOD

Participants and materials

Ethics approval for this study was granted by the sepa-rate and independent human research ethics committeesof Murdoch University and the North MetropolitanArea Mental Health Service, Department of Health,Government of Western Australia. Archival data

Dow

nloa

ded

by [

Uni

vers

ity o

f N

otre

Dam

e A

ustr

alia

] at

11:

24 2

7 A

pril

2013

ECOLOGICAL VALIDITY OF THE REY COMPLEX FIGURE 823

were obtained from the records of 263 children andadolescents assessed at a Western Australian state-wideneuropsychological service between July 1, 2006, andJune 30, 2009. The Neurosciences Unit provides special-ized diagnostic assessment and ongoing managementfor children with known or suspected brain impairmentresulting from trauma, disease, or genetic conditions. Inaddition to parent and child interviews, psychologicaltesting comprises administration of a core batteryof tests and parent-rating questionnaires that can besupplemented with or added to other instruments forthe purposes of hypothesis testing. Each assessmenttakes between three and five hours to complete andis performed by a registered and experienced clinicalneuropsychologist or clinical psychologist.

One questionnaire of especial relevance to the studyis the Behavior Rating Inventory of Executive Function(BRIEF; Gioia et al., 2000). The BRIEF is an 86-itemparent questionnaire developed to assess the behavioralfeatures of executive function in children from 5 to 18years of age. The child’s behavior is rated on a 3-pointLikert scale (“Never,” “Sometimes,” “Often”) with higherratings indicative of greater perceived impairment. Eightinterrelated scales representing subdomains of executivebehavior are obtained (Inhibit, Shift, Emotional Control,Initiate, Working Memory, Plan/Organize, Organizationof Materials, and Monitor). Two composite scoresare derived from the eight scales with the Inhibit,Shift, and Emotional Control forming the BehavioralRegulation Index (BRI) and the Initiate, WorkingMemory, Plan/Organize, Organization of Materials, andMonitor forming the Metacognition Index (MI). TheBRI and MI are then combined to form an overall GlobalExecutive Composite (GEC). The scales show high inter-nal consistency ranging from .80 to .98 and test–retestreliability after 2 weeks of .84 to .88 for the index scores(Gioia & Isquith, 2004). In addition, the scales have goodcontent validity and demonstrate convergent and dis-criminant validity (Gioia & Isquith, 2004). Significantcorrelations between various scales and indices of theBRIEF and performances on traditional tests of exec-utive function in clinical and nonclinical samples havebeen published (e.g., V. Anderson, Anderson, Northam,Jacobs, & Mikiewicz, 2002; Mangeot et al., 2002; Parrishet al., 2007). The BRI, MI, and GEC were the variablesused in this study.

Among the questionnaires routinely completed by par-ents that also appeared to have relevance to this study wasthe Personality Inventory for Children–Second Edition(PIC–2; Lachar & Gruber, 2001). The PIC–2 is a 275-item(true/false) parent questionnaire that assesses narrowand broad dimensions of cognitive, behavioral, emo-tional, and interpersonal adjustment of children andadolescents. Nine adjustment scales (composed of 21adjustment subscales) are generated, including cogni-tive impairment, impulsivity and distractibility, delin-quency, family dysfunction, reality distortion, somaticconcern, psychological discomfort, social withdrawal,and social-skills deficits. Overall, the PIC–2 demon-strates sound criterion, construct, and discriminant valid-ity, and the nine adjustment scales demonstrate high

test–retest reliability (all rs > .88; Lachar, 2004; Lachar& Gruber, 2001). For this study, the three subscales ofthe Cognitive Impairment scale were considered mostrelevant and were therefore used in the analyses. The sub-scales were Inadequate Abilities (Cog1: Inadequate adap-tive competence is likely), Poor Achievement (Cog2: Pooradjustment to the classroom), and Developmental Delay(Cog3: Suggests adaptive and developmental deficits).Selection of these scales was based on the substantialcorrelations between them and a variety of individu-ally administered measures of adaptive behavior (e.g.,Vineland Adaptive Behavior Composite, r = –.54), intel-ligence (e.g., Wechsler Full Scale IQ, r = –.54), and aca-demic achievement (e.g., Wide Range Achievement Test,WRAT, Reading and Spelling, r = –.59, and WRATArithmetic, r = –.60).

Files included in the study were on the basis of com-plete administration of the RCF, the BRIEF, and thePIC–2. A final set of 263 files met criteria. This was83% of the 319 files available for scrutiny (i.e., the num-ber of children who were assessed at the Unit duringthe period of the study). Maximum diversity and rep-resentativeness of the clinical sample were maintainedby including all children irrespective of their diagno-sis, though acceptance of referrals to the unit for neu-ropsychological assessment requires the child to have anorganic, including symptomatic, mental disorder reflect-ing the presence of cerebral disease, brain injury, orother insult leading to cerebral dysfunction (e.g., epilepsy,diseases of the circulatory system, disorders related toshort gestation and low birth weight, congenital malfor-mations, deformations and chromosomal abnormalities,injury, poisoning, and certain other consequences ofexternal causes). This method of participant inclusionis common in child neuropsychological research (e.g.,Bodnar, Prahme, Cutting, Denckla, & Mahone, 2007;Green & Flaro, 2003) and was recently used by Smith andZahka (2006) who investigated the relationship betweengeneral cognitive ability and RCF accuracy scores andscores derived from an organizational process approachin a sample of 6- to 17-year-olds with acquired braininjuries.

Participant demographics and characteristics andquestionnaire scores were obtained and entered onto aSPSS database. These data are displayed in Table 1 andFigures 1 and 2. The sample comprised 178 males and 87females ranging from first-year primary school throughto the final year of secondary high-school education,though 67% of the sample were the 8- to 12-year-oldsattending primary school. Measures of central tendencyfor the IQ suggest the sample was quite impaired. Overhalf the sample (i.e., 56%) had an IQ that was, accord-ing to Wechsler (2003) classification criteria, in either the“borderline” or “low average” ranges.

The range of T scores on both the BRIEF andPIC–2 were wide and extended through the “normal”(i.e., T < 65) and “clinical” (i.e., T ≥ 65) ranges. Therewere no violations of normality with all the skewnessstatistics being less than 2 standard errors of skewness.These data suggest there were no restrictions in range ofparent-rating scores on either instrument.

Dow

nloa

ded

by [

Uni

vers

ity o

f N

otre

Dam

e A

ustr

alia

] at

11:

24 2

7 A

pril

2013

824 DAVIES ET AL.

TABLE 1Participant characteristics and questionnaire data

Variable M (SD) Median Range

Age (years) 10.9 (2.4) 10.8 5.8 – 17.7IQ 85 (16) 84 52 – 132

BRIEF BRI 68 (14) 66.5 39 – 94MI 63 (19) 71 39 – 86GEC 69 (10) 70 38 – 88

PIC–2 Cog1 72 (11) 72.5 44 – 90Cog2 67 (9) 67 41 – 85Cog3 66 (14) 67 43 – 90

Note. T-scores have been used for the BRIEF (Behavior Rating Inventory of Executive Function)and PIC–2 (Personality Inventory for Children–Second Edition). BRI = Behavior RegulationIndex; MI = Metacognitive Index; GEC = Global Executive Composite; Cog1 = InadequateAbilities; Cog2 = Poor Achievement; Cog3 = Developmental Delay.

50

40

30

20Fre

quency

10

04 6 8 10 12

Age group

14 16 18

Figure 1. Frequency distribution of age.

60

40

Fre

quency

20

045 55 65 75 85

IQ group

95 105 115 125 135

Figure 2. Frequency distribution of IQs grouped according toWechsler categories.

According to the International Statistical Classi-fication of Diseases and Related Health Problems (ICD-10; World Health Organisation, 2007), most children (i.e.,n = 96 or 45%) were classified with an organic, includingsymptomatic, mental disorder reflecting the presence ofcerebraldisease,braininjury,orotherinsult leadingtocere-braldysfunction(i.e.,F06andF07).Afurther30%(n = 78)were considered to display a range of undercontrolledand disruptive behaviors typically considered to reflectattention deficits and hyperactivity (i.e., F90). Childrenexhibiting specific disorders of psychological development(i.e., F80–F89) comprised another large group (n = 56,21%). The remaining 33 children (13%) were assignedwith a range of other disorders (e.g., tic disorders, F95;adjustment disorders F43.2; anxiety disorders, F41).

The ICD–10 etiological or medical conditions/disorders causing the presenting mental health prob-lems varied and included diabetes mellitus (E10), epilepsy(G40), diseases of the circulatory system (e.g., I21, I61),disorders related to short gestation and low birth weight(e.g., P07), congenital malformations, deformations andchromosomal abnormalities (e.g., Q85), injury, poison-ing and certain other consequences of external causes(e.g., S02), as well as a range of suspected but as yetunidentified conditions.

Each copy reproduction of the RCF was rescored usingthe Taylor (1959), Smith and Zahka (2006), Bernsteinand Waber (1996), and P. Anderson et al. (2001) scoringapproaches.

Scoring approaches

Taylor’s (1959) scoring system

Osterrieth’s (1946) original method that was adaptedby Taylor (1959) is the traditional and most commonmethod for scoring the accuracy of copies of RCFs(Knight, Kaplan, & Ireland, 2003). The method involvesscoring each of 18 elements of the RCF between 0.0and 2.0 points. Each element is evaluated for placement,degree of correctness, and level of distortion. A totalaccuracy score out of 36 points is obtained.

Dow

nloa

ded

by [

Uni

vers

ity o

f N

otre

Dam

e A

ustr

alia

] at

11:

24 2

7 A

pril

2013

ECOLOGICAL VALIDITY OF THE REY COMPLEX FIGURE 825

Smith and Zahka’s (2006) organizational scoringsystem

The scoring system was originally developed by Savageet al. (1999), modified by Deckersbach et al. (2000), witha final version developed by Smith and Zahka (2006).The five main organizational elements of the RCF (i.e.,the large rectangle, diagonals, horizontal and verticalmidlines, large triangle) are scored in a simple “all-or-nothing” fashion, such as either present or not present.Providing each element of the figure is drawn properlyand is not unfragmented, a RCF can be scored 0 (noelements present) to 5 (all elements present). It is a verysimple and extremely quick method of scoring a RCF. It isan attractive proposition for clinicians and researchers ifits usefulness in pediatric populations can be established,hence its inclusion in this study. And since only one studyto date has evaluated this approach in a pediatric sam-ple (Smith & Zahka, 2006), it would also be useful to seewhether previous findings can be replicated.

Bernstein and Waber’s (1996) DevelopmentalScoring System (DSS)

The Copy Organization Condition of the DSS wasused. This process scoring method quantifies, with a scorethat ranges from 1 (poorly organized) to 13 (highly orga-nized), the extent to which the figure is approached as ameaningful gestalt. This is done by evaluating the pres-ence of 24 criterial features (e.g., upper left corner ofthe base rectangle, diagonal lines of the left centre boxintersect) in terms of alignment (i.e., two segments ofa line should be accurately aligned with one another)or intersection (i.e., two or more lines must intersectproperly).

P Anderson, Anderson, and Garth’s (2001)RCF–Organization and Strategy Score(RCF-OSS)

In a review of the applicability the RCF had for assess-ing children, P. Anderson et al. (2001) concluded therewas (a) little evidence to show that copying the RCFwas integrally related to executive functioning in chil-dren, (b) a lack of consensus about which scoring systemshould be used, and (c) variability in the psychomet-ric properties underpinning the different scoring systems.Thus motivated to develop a reliable and valid and sen-sitive and specific scoring system with normative datathat could become a globally accepted procedure, theresearchers developed the RCF-OSS. This system wasdeveloped from a previous scoring system (Akshoomoff& Stiles, 1995) and is specifically designed for pediatricpopulations to assess whether the child has employeda logical and well-planned approach or a more ran-dom piecemeal approach. This process approach assessesstrategy and organization by analyzing the sequence ofdrawn elements with particular importance placed onthe base rectangle and midlines due to their role in thealignment of the internal and external attachments. Thescoring method produces ordinal values. Each drawing

is rated on a 7-point scale labeled according to the levelof strategy and organization (i.e., 1 = “unrecognizable,”2 = “poor,” 3 = “random,” 4 = “piecemeal/fragmented,”5 = “part-configural,” 6 = “conceptual,” 7 = “excellent”).We are not aware of any other published study that hasused this psychometrically sound developmental scoringsystem, so the furthering of research on the method,particularly with a clinical emphasis, seems apt.

RESULTS

Interrater reliability

Interrater reliability (IRR) estimates were obtained forall four scoring systems using Pearson’s product–momentcorrelation coefficients. For the Taylor (1959) approach,the accuracy scores of 50 randomly selected figuresmade by the primary investigator of this project (S.D.)were compared to the scores made by the assess-ing clinician (clinical psychologist or neuropsycholo-gist) who administered the test at the time of thechild’s assessment. The IRR was very high, r = .96,p < .01. In addition, the scores of another 18 ran-domly selected figures made by the investigator of thisproject were compared to the scores made by twoother clinicians (co-investigators, T.A. and C.C.) whodid not assess the child and were “blind” to the scoresof the investigator. These IRRs were also very high,r = .97, p < .01, and r = .93, p < .01, respectively. HighIRR estimates were also derived in the same man-ner from the 18 RCFs for the three other approaches:Smith and Zahka (2006), r = .89, p < .01, and r = .94,p < .01; V. Anderson, Anderson, et al. (2001); r = .83. p< .01, and r = .89, p < .01; Bernstein and Waber (1996),r = .85, p < .01, and r = .87. p < .01.

RCF performance

Concerns that hypothesized relationships between RCFscores and parent-questionnaire ratings could be affectedby restricted ranges in performance and violationsof normality, as well as age spread and reducedIQ—characteristics typical of such clinical samples ofconvenience—two sets of preliminary analyses were con-ducted on the RCF scores. Measures and plots of cen-tral tendency and skewness (see Table 2 and Figure 3)revealed that the distribution of scores produced bythe Taylor (1959) approach was significantly negativelyskewed, with higher scores more prevalent than lowerones. However, given that it was not extreme and thatcorrelation analysis is robust to such minor violations, notransformation of data was made.

Table 3 displays the correlations between the four RCFscoring approaches with age and IQ. Assumptions oflinearity and homoscedasticity and independence weresatisfied on inspection of scatterplots (between the depen-dent variables and between the standardized predictedand obtained residuals) and normal probability plots.IQ positively correlated with all scoring approaches,indicating that accuracy of RCF reproduction, as well as

Dow

nloa

ded

by [

Uni

vers

ity o

f N

otre

Dam

e A

ustr

alia

] at

11:

24 2

7 A

pril

2013

826 DAVIES ET AL.

TABLE 2Measures of central tendency and skewness for the four RCF scoring approaches

Variable M (SD) Median Mode Range Skew (SE)

Taylor (1959) 21.6 (9.6) 23.7 30 1 – 36 −.54 (.15)a

Smith & Zahka (2006) 2.2 (1.5) 2 2 0 – 5 .28 (.15)Bernstein & Waber (1996) 12.5 (5.8) 12 20 0 – 24 .07 (.15)P. Anderson et al. (2001) 3.9 (1.2) 4 5 1 – 7 −.16 (.15)

Note. RCF = Rey–Osterrieth Complex Figure.aSignificant skew. Violations of normality were determined when the value of a skewness statisticwas two times the standard error of skewness.

40 70

60

50

40

30

20

10

0

100

80

60

40

20

0

0 2 4 6

30

20Count

Count

Count

Count

10

0

20

10

0

0 5 10 15 20

Taylor (1959) Smith & Zahka (2006)

0 2 4 6 8

Anderson et al. (2001)

25 30 35

0 5 10 15 20

Bernstein & Waber (1996)

25

Figure 3. Frequency distributions of RCF (Rey–Osterrieth Complex Figure) scores for the Taylor (1959), Smith and Zahka (2006),Bernstein and Waber (1996), and P. Anderson et al. (2001) scoring approaches.

TABLE 3Correlation coefficients between RCF scores and age and IQ

Variable Taylor (1959) Smith & Zahka(2006)

Bernstein &Waber (1996)

P. Anderson et al.(2001)

IQ .22∗∗ .19∗Age .41∗∗ .33∗∗ .31∗∗ .25∗∗

Note. RCF = Rey–Osterrieth Complex Figure.∗p < .05, two-tailed. ∗∗p < .01, two-tailed.

Dow

nloa

ded

by [

Uni

vers

ity o

f N

otre

Dam

e A

ustr

alia

] at

11:

24 2

7 A

pril

2013

ECOLOGICAL VALIDITY OF THE REY COMPLEX FIGURE 827

some organizational abilities, is related to the intellectualability of these children with neuropsychological disor-ders. Age also positively correlated with all approaches,which is consistent with the plethora of studies that haveconsistently documented age-related increases in a varietyof different RCF scoring approaches in normal popula-tions of children (e.g., P. Anderson et al., 2001; Nakanoet al., 2006), though less is known about age-relatedchanges in RCF performances in children with CNSdysfunction.

Relationships between RCF scores andquestionnaire ratings

Given the significant relationships between RCF perfor-mances and age and IQ, these latter two variables werecontrolled for in further analyses. Partial correlationswere computed between the four scoring approaches andparent ratings on several subscales of the PIC–2 andBRIEF. These are displayed in Table 4.

RCF scores derived by all methods had high and sig-nificantly positive relationships with each other, withscores obtained by P. Anderson et al.’s (2001) processapproach strongly related to both Taylor’s (1959) accu-racy and Smith and Zahka’s (2006) process approaches.This pattern of findings suggests shared or commonstrategies between the different approaches that are notjust explained by age or IQ. This is consistent with previ-ous research on convergent validity of RCF scoring meth-ods (e.g., Harris et al., 1999). Interestingly, the Bernsteinand Waber (1996) and P. Anderson et al. (2001) scor-ing systems that were developed specifically for assessingchildren had the lowest positive relationship.

As expected, RCF scores derived by the Taylor (1959)method did not relate with parent ratings of chil-dren’s behavioral executive dysfunction, suggesting thatthe visuospatial skills needed for accuracy in copy-ing are rather unrelated to behavioral and metacog-nitive dysregulation in the everyday world. However,contrary to all expectations, all partial correlationsbetween the three RCF process scoring approaches andthe two major indices of the BRIEF were of zero ornear-zero magnitudes. Such findings suggest that the

executive function skills required to produce organizedand planned copies of the RCF bear no relationship atall with parent ratings of children’s behavioral executivedysfunction as measured by the BRIEF.

In contrast, 9 of the 12 partial correlations betweenthe various RCF scoring systems and the PIC–2 sub-scales were significant. Scores obtained by the Taylor(1959) and P. Anderson et al. (2001) approaches nega-tively related with parent ratings about their children’slevel of ability, adaptive functioning, and developmentstatus. The Smith and Zahka (2006) and Bernstein andWaber (1996) RCF derived scores also negatively relatedwith development status. These data suggest that thedegree of impairment in being able to reproduce accurateas well as planned and organized RCFs is associated withparent perceptions about their child’s abilities, achieve-ment, and developmental status as evaluated by thePIC–2.

DISCUSSION

Overview of study

The two main questions posed in this study were whethera child’s ability to produce planned and organized andaccurate copies of the RCF would correspond with hisor her everyday behaviors at home or at school, suchas keeping the bedroom clean, completing large assign-ments, or accommodating changes in routine. To addressthese questions, the scores of children’s copies of theRCF were derived from four different approaches—Taylor’s (1959) longstanding and traditional psychome-tric approach for determining accuracy; and the strat-egy and organizational process approaches developed bySmith and Zahka (2006), Bernstein and Waber (1996),and P. Anderson et al. (2001)—and were correlated withparent-ratings of their child’s behavior, using the BRIEFand PIC–2.

Hypotheses

Contrary to the predictions of Hypothesis 1, the RCFscores derived using the three process approaches did not

TABLE 4Partial correlation coefficients between the RCF scores and parent ratings on the BRIEF and PIC–2, controlling for age and IQ

Variable Taylor (1959) Smith & Zahka (2006) Bernstein & Waber (1996) P. Anderson et al. (2001)

Smith & Zahka (2006) .44∗∗ — — —Bernstein & Waber (1996) .44∗∗ .39∗∗ — —P. Anderson et al. (2001) .70∗∗ .64∗∗ .16∗ —BRI .01 −.05 .03 −.01MCI .05 .01 .02 −.01Cog1 −.18∗ −.11 −.01 −.21∗∗Cog2 −.13∗ −.15∗ .03 −.27∗∗Cog3 −.42∗∗ −.20∗∗ −.15∗ −.19∗

Note. RCF = Rey–Osterrieth Complex Figure. BRIEF = Behavior Rating Inventory of Executive Function. PIC–2 = PersonalityInventory for Children–Second Edition. BRI = Behavioral Regulation Index; MCI = Metacognitive Index; Cog1 = InadequateAbilities; Cog2 = Poor Achievement; Cog3 = Developmental Delay.∗p < .05, two-tailed. ∗∗p < .01, two-tailed.

Dow

nloa

ded

by [

Uni

vers

ity o

f N

otre

Dam

e A

ustr

alia

] at

11:

24 2

7 A

pril

2013

828 DAVIES ET AL.

correlate at all with any indices of the BRIEF. The threeRCF organization and strategy scoring approaches cor-related strongly with each other, even when controllingfor IQ and age, but correlations with the BRIEF wereessentially zero. The findings suggest that the executivefunctions of organization and planning used by childrento construct their copies of the RCF, as measured bythe process approaches investigated in this study, are notthe same as the executive functions of behavioral regula-tion and metacognition, as measured by the BRIEF (V.Anderson et al., 2002; Gioia et al., 2000). In other words,the ways in which these children constructed their RCFsdo not appear to provide any information about how theyfunction in their everyday world, at least according to theperceptions of parents.

With respect to Hypothesis 2, the expectation wasmet that RCF scores derived from Taylor’s (1959) accu-racy approach correlated with indices of the PIC–2 thatasked for parent descriptions of their children in termsof their cognitive functioning and academic achievementby endorsing questionnaire items that ask about studyhabits (e.g., doing homework on time), academic suc-cess (e.g., how easy is school for my child), and cognitivecompetence (e.g., presence of learning problems, educa-tional support unit placement). That these relationshipsremained when controlling for IQ and age, which havelong been known to contribute to performance on theRCF, suggests there is something unique about the RCFas a measure of visual–spatial ability in its ability toinform parents about a child’s developmental status, levelof ability, and adaptive functioning. As to the secondpart of Hypothesis 2, the finding that the BRIEF indicesand Taylor (1959) derived RCF scores were uncorrelatedwas not unexpected. We are unaware of any theoreticalliterature to suggest that there is a link between the psy-chological constructs of visual–spatial function and theexecutive functions of everyday behavior.

We are confident that the absence of associations is dueto the absence of a phenomenon and not due to method-ological problems. Being a clinical sample of conveniencethat most likely represents the experience of most clinicalsettings, there was a possibility that despite the range ofpresenting problems and medical/neurological etiologies,RCF performances would all be poor, and parent ques-tionnaire ratings would be clinically high. This was notthe case. The range and distribution of RCF scores andBRIEF and PIC–2 ratings indicate that the sample wereessentially normal and not restricted in range.

Perhaps the findings should not have been unexpected.It is often assumed that impaired brain processes thatlead to poor performance on neuropsychological testswill also lead to poor performance in situations outsidethe testing environment (Chaytor et al., 2006). Moreover,the executive functions of the frontal system seem tocommand particular attention because of the belief orperceived logical link that everyday real-world activi-ties, such as attending school or successfully engagingin rehabilitation, draw heavily on the executive functionsof the frontal system (Long, 1996; Silver, 2000; Tupper& Cicerone, 1991). But in the pediatric literature, theempirical foundation for these assumptions is small. For

example, the findings of the studies that have directlyexamined whether scores obtained on tests of executivefunction correlate with everyday problem behaviors inchildren with disorders uniquely characterized by exec-utive dysfunction, such as ADHD (Lawrence et al., 2004;Mahone, Cirino, et al., 2002; Molho & Silver, 1997), trau-matic brain injury (TBI; Mangeot et al., 2002; Vriezen& Pigott, 2002), phenylketonuria (V. Anderson et al.,2002), and epilepsy (Parrish et al., 2007), are mixed.Associations have been found in some studies but not inothers.

In this study, it is conceivable that lack of the rela-tionship is due to the structured, quiet, distraction-freeenvironment of the test-room, which does not allow forinfluences and constraints that arise from interactionsbetween a child and his or her social, educational, andphysical aspects of the natural environment (Heinrichs,1990; Sbordone, 1996). This problem is compounded fur-ther given that one-on-one laboratory-type testing cap-tures only a relatively small sample of behavior (Chaytoret al., 2006). Furthermore, the cognitive demands of theexternal environment will differ across settings as wellas interact with the extent and nature of the executivedeficit (Chaytor et al., 2006). In other words, the executivefunction demands of performing a discrete neuropsycho-logical test pale into insignificance when considering thatpurposeful goal-directed behavior in daily life involvesthe planning, implementation, and monitoring of manyand far more complex series of particular goals amongstcompeting goals at successive levels of abstraction in thehierarchical structure of action. In a couple of studies, nosignificant relationships were found between tests of exec-utive function and the BRIEF scales (Mahone, Cirino,et al., 2002; Vriezen & Pigott, 2002). The likelihood thata correlation could be demonstrated among a heteroge-neous group of neurologically impaired children may beeven smaller.

It could also have been the case that the selected samplewas not all that impaired in executive functions. Despiteover half of the children having parent-rated BRIEFT-scores above the clinical cutoff, it cannot be ruledout that some parents overreported his or her child’simpaired behaviors. The mean BRIEF T-scores in thisstudy were considerably higher than those in other studies(V. Anderson et al., 2002; Mangeot et al., 2002; Parrishet al., 2007; Vriezen & Pigott, 2002). The emotionalinvolvement of the respondent may have created a “nega-tive halo effect” (Denckla, 2002, p. 305). Cross-validatingparent reports by administering the teacher report ofthe BRIEF and/or including scores from the adminis-tration of other traditional executive function tests (e.g.,WCST, Tower tests) may have clarified this limitation ofthe study. The need to use multiple tests of executive func-tion in order to detect executive function problems inclinical populations is well recognized (e.g., Bennett, Ong,& Ponsford, 2005).

In other studies of ecological validity, where the corre-lations between the BRIEF and tests of executive func-tion have ranged from nonsignificant to modest at best,interpretation has favored the BRIEF, in that the exec-utive function tests have been seen as the source of the

Dow

nloa

ded

by [

Uni

vers

ity o

f N

otre

Dam

e A

ustr

alia

] at

11:

24 2

7 A

pril

2013

ECOLOGICAL VALIDITY OF THE REY COMPLEX FIGURE 829

problem. It will be interesting to see whether research thatadopts criterion measures instead of the BRIEF in questsfor investigating the ecological validity of existing execu-tive function tests produces similar or different patternsof findings.

Whereas there appears no predictive validity of RCFscores when using the BRIEF as an index of real-worldeveryday behavior, there were findings to suggest thatRCF scores said something about how parents rated chil-dren’s real-world functioning as measured by the PIC–2.All four scoring approaches were associated in vary-ing degrees with how parents described their childrenin terms of their cognitive functioning and academicachievement even when controlling for IQ and age. It isnot entirely clear what it is about the PIC–2 that makesit predictable in the way the BRIEF was unexpectedlynot. One criterion validity study of the PIC–2 (Lachar& Gruber, 2001) indicated that elevations on the cogni-tive subscales used in this study occurred most frequently,among 11 diagnosis-based groups, for the children diag-nosed with a pervasive developmental disorder (PDD).However, there were no children in the sample in thisstudy that had such a diagnosis. Such children do notmeet referral criteria for assessment.

An alternative explanation may be that the process-approach-derived RCF scores are actually proxy visual–spatial scores and may not be good indicators of executivefunctioning. First, studies that show lack of relation-ships between some RCF process approach measures(e.g., BQSS and DSS) and psychometric tests of executivefunction in these clinical populations are not new (e.g.,Beebe et al., 2004; Watanabe et al., 2005). Second, the lit-erature that has established relationships, some of thempredictive, between academic achievement and schoolreadiness with visual–motor integration (VMI) is longstanding (e.g., Duffey, Ritter, & Fedner, 1976) and recent(e.g., Mayes et al., 2009; Orit et al., 2007). Visual–motorintegration can be defined as the extent to which indi-viduals can integrate their visual and motor systems andis typically assessed by requiring them to copy a seriesof geometric designs (e.g., Beery & Buktenica, 1989).The plethora of literature suggests that good VMI abilityserves as a protective factor to the many normative schooland classroom-based challenges that a child faces. It hasbeen suggested this is because the majority of schooltime, up to 60%, involves manual activities such as draw-ing, coloring, writing, and cutting (McHale & Cermak,1992). It therefore appears from the findings of this studythat children’s copies of the RCF, irrespective of how thecopy drawings are scored, measure certain aspects aboutwhether they can coordinate motor movement with visualstimulus well more than the process or way in whichthey do this. This may not be the case for older childrenattending high school where the manual activities wouldbe very different to those used in primary school, but inthis study, 81% of the sample attended primary school.Support for this claim would be to find that measures ofVMI similarly bear no relationship with the BRIEF butsignificant associations with the cognitive subscales of thePIC–2.

On the basis of this claim, two other observations areworth mentioning. First, an alternative but consistentinterpretation of the zero correlations between the RCFscores and the BRIEF can be offered. That is, the processapproach scores did not associate with the BRIEF for thesame reason that the Taylor (1959) accuracy scores didnot. Second, the two zero correlations and only one lowcorrelation that the Bernstein and Waber (1996) organi-zational scoring approach had with the PIC–2 suggeststhat it may have the better executive function constructvalidity of the three approaches used in this study.

CONCLUSION

This study is the first that we are aware of where thesole focus was on examining the veridicality of RCFcopy scores, defined as the extent to which an exist-ing and established test originally designed for identify-ing underlying pathology is associated with or predic-tive of real-world behaviors (Franzen & Wilhelm, 1996;Gioia & Isquith, 2004). Moreover, the study was con-ducted on a large sample across a wide age range andin a clinical setting, thereby increasing the generaliz-ability and clinical utility of the main finding, that theecological validity of the RCF may be more a func-tion of visual–motor integration skill of the child beingassessed than his or her executive function ability. On theother hand, the heterogeneous, low-cognitive-functioningclinical sample that comprised young children throughto older adolescents are also limitations. Furthermore,being a retrospective study, the data that were availablefor collation and analysis were constrained by the typeof psychological instrument administered in the clini-cal setting and how it was used. The possibility that(a) other RCF accuracy and process approach scoringmethods, such as the BQSS, (b) different measures ofeveryday behavioral executive functioning, such as theChildren’s Executive Function Scale (Goulden, Silver,Harwood, & Levin, 1997) or Behavioral Assessment ofthe Dysexecutive Syndrome–Children’s Version (Emslie,Wilson, Burden, Nimmo-Smith, & Wilson, 2003), and (c)teacher questionnaire ratings of children show expectedpatterns of associations and dissociations awaits futureresearch. Establishing the predictive validity of simpleand quick RCF scoring methods would be a boon toclinicians.

Original manuscript received 31 December 2009Revised manuscript accepted 8 March 2011

First published online 7 July 2011

REFERENCES

Akshoomoff, N. A., & Stiles, J. (1995). Developmental trends invisuospatial analysis and planning: I. Copying the complexfigure. Neuropsychology, 9, 364–377.

Anderson, P., Anderson, V., & Garth, J. (2001). Assessmentand development of organisational ability: The Rey ComplexFigure Organisational Strategy Score (ROCF-OSS). TheClinical Neuropsychologist, 15, 81–94.

Dow

nloa

ded

by [

Uni

vers

ity o

f N

otre

Dam

e A

ustr

alia

] at

11:

24 2

7 A

pril

2013

830 DAVIES ET AL.

Anderson, V., Anderson, P., Northam, E., Jacobs, R., &Catroppa, C. (2001). Development of executive func-tions through late childhood and early adolescence in anAustralian sample. Developmental Neuropsychology, 20(1),385–406.

Anderson, V., & Catroppa, C. (2005). Recovery of executiveskills following paediatric traumatic brain injury (TBI): A 2year follow up. Brain Injury, 19(6), 459–470.

Anderson, V., Anderson, P., Northam, E., Jacobs, R., &Mikiewicz, O. (2002). Relationships between cognitive andbehavioural measures of executive function in children withbrain disease. Child Neuropsychology, 8, 231–240.

Baron, I. S. (2004). Neuropsychological evaluation of the child.New York, NY: Oxford University Press.

Beebe, D. W., Ris, M. D., Brown, T. M., & Dietrich, K. (2004).Executive functioning and memory for the Rey–Osterriethcomplex figure task among community adolescents. AppliedNeuropsychology, 11(2), 91–98.

Beery, K. E., & Buktenica, N. A. (1989). The developmentaltest of visual-motor integration. Odessa, FL: PsychologicalAssessment Resources.

Bennett, P. C., Ong, B., & Ponsford, J. (2005). Assessmentof executive dysfunction following traumatic brain injury:Comparison of the BADS with other clinical neu-ropsychological measures. Journal of the InternationalNeuropsychological Society, 11, 606–613.

Bernstein, J. H., & Waber, D. P. (1996). Developmental scoringsystem for the Rey–Osterrieth Complex Figure: Professionalmanual. Odessa, FL: Psychological Assessment Resources.

Bodnar, L. E., Prahme, M. C., Cutting, L. E., Denckla, M. B., &Mahone, E. M. (2007). Construct validity of parent ratingsof inhibitory control. Child Neuropsychology, 13, 345–362.

Boone, K. B. (2000). Test review. The Boston QualitativeScoring System for the Rey–Osterrieth Complex Figure.Journal of Clinical and Experimental Neuropsychology, 22(3),430–432.

Cahn, D. A., Marcotte, A. C., Stern, R. A., Arruda, J. E.,Akshoomoff, N. A., & Leshko, I. C. (1996). The BostonQualitative Scoring System for the Rey–Osterrieth ComplexFigure: A study of children with attention deficit hyperactiv-ity disorder. Clinical Neuropsychology, 10, 397–406.

Chaytor, N., & Schmitter-Edgecombe, M. (2003). The ecologicalvalidity of neuropsychological tests: A review of the liter-ature on everyday cognitive skills. Neuropsychology Review,13, 181–197.

Chaytor, N., Schmitter-Edgecombe, M., & Burr, R. (2006).Improving the ecological validity of executive functioningassessment. Archives of Clinical Neuropsychology, 21, 217–227.

Deckersbach, T., Savage, C. R., Henin, A., Mataix-Cols, D.,Otto, M. W., Wilhelm, S., et al. (2000). Reliability andvalidity of a scoring system for measuring organizationalapproach in the complex figure test. Journal of Clinical andExperimental Neuropsychology, 22, 640–648.

Demsky, Y., Carone, D. A., Jr., Burns, W. J., & Sellers, A. (2000).Assessment of visual–motor coordination in 6-to 11-yr-olds.Perceptual and Motor Skills, 91(1), 311–321.

Denckla, M. B. (2002). The behaviour rating inventory ofexecutive function: Commentary. Child Neuropsychology, 8,304–306.

Duffey, J. B., Ritter, D. R., & Fedner, M. (1976). Developmentaltest of visual-motor integration and the Goodenough draw-a-man test as predictors of academic success. Perceptual andMotor Skills, 43(2), 543–546.

Elliot, C. D. (1990). Differential ability scales. San Antonio, TX:The Psychological Corporation.

Emslie, H., Wilson, F. C., Burden, V., Nimmo-Smith, I.,& Wilson, B. A. (2003). Behavioural Assessment of theDysexecutive Syndrome for Children (BADS-C). ChildNeuropsychology, 13(6), 539–542.

Franzen, M. D., & Wilhelm, K. L. (1996). Conceptual founda-tions of ecological validity in neuropsychological assessment.In R. J. Sbordone & C. J. Long (Eds.), Ecological validity ofneuropsychological testing (pp. 91–112). Boca Raton, FL: GRPress.

Frisk, V., Jakobson, L. S., Knight, R. M., & Robertson, B.(2005). Copy and recall performance of 6–8-year-old chil-dren after standard vs. step-by-step administration of theRey–Osterrieth Complex Figure. Child Neuropsychology,11(2), 135–152.

Garth, J., Anderson, V., & Wrenall, J. (1997). Executive functionsfollowing moderate-to-severe frontal lobe injury: Impactof injury and age at injury. Paediatric Rehabilitation, 1,99–108.

Gioia, G. A., & Isquith, P. K. (2004). Ecological assessment ofexecutive function in traumatic brain injury. DevelopmentalNeuropsychology, 25, 135–158.

Gioia, G. A., Isquith, P. K., Guy, S. C., & Kenworthy, L.(2000). Test review: Behavior Rating Inventory of ExecutiveFunction. Child Neuropsychology, 6, 235–238.

Goulden, L. G., Silver, C. H., Harwood, H. S., & Levin, H. S.(1997). Utility of the children’s executive function scale inchildhood brain injury. Archives of Clinical Neuropsychology,12, 326.

Green, P., & Flaro, L. (2003). Word memory test performance inchildren. Child Neuropsychology, 9, 189–207.

Grimwood, K., Anderson, V., Bond, L., Catroppa, C., Hore,R., Keir, E., et al. (1995). Adverse outcomes of bac-terial meningitis in school-aged survivors. Pediatrics, 95,646–656.

Harris, N. S., Bernstein, J. H., Waber, D. P., & Stern, R. A.(1999). Comparison of two qualitative scoring systems forthe Rey–Osterrieth Complex Figure: The DSS and BQSS.Journal of the International Neuropsychological Society, 5,119.

Heinrichs, R. W. (1990). Current and emergent applicationsof neuropsychological assessment: Problems of validity andutility. Professional Psychology: Research and Practice, 21,171–176.

Kirk, U. (1985). Hemispheric contributions to the develop-ment of graphic skill. In C. Best (Ed.), Hemispheric functionand collaboration in the child (pp. 193–228). Orlando, FL:Academic Press.

Kirkwood, M. W., Weiler, M. D., Bernstein, J. H., Forbes, P. W.,& Waber, D. P. (2001). Sources of poor performance on theRey–Osterrieth Complex Figure Test among children withlearning difficulties: A dynamic assessment approach. TheClinical Neuropsychologist, 15(3), 345–356.

Klicpera, C. (1983). Poor planning as a characteristic ofproblem-solving behavior in dyslexic children: A studywith the Rey–Osterrieth complex figure test. ActaPaedopsychiatrica: International Journal of Child &Adolescent Psychiatry, 49(1–2), 73–82.

Knight, J. A., Kaplan, E., & Ireland, L. D. (2003). Surveyfindings of Rey–Osterrieth Complex Figure usage. In J.A. Knight (Ed.), The handbook of Rey–Osterrieth ComplexFigure usage: Clinical and research application. Lutz, FL:PAR.

Lachar, D. (2004). The Personality Inventory for Children,Second Edition (PIC–2), Personality Inventory for Youth(PIY), and Student Behavior Survey (SBS). In M. J.Hilsenroth & D. L. Segal (Eds.), Comprehensive handbook ofpsychological assessment: Vol.2. Personality assessment (pp.192–212). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Lachar, D., & Gruber, C. P. (2001). Personality Inventoryfor Children Second Edition (PIC–2): Standard form andbehavioral summary manual. Los Angeles, CA: WesternPsychological Services.

Lawrence, V., Houghton, S., Douglas, G., Durkin, K., Whiting,K., & Tannock, R. (2004). Executive function and ADHD: Acomparison of children’s performance during neuropsycho-logical testing and real-world activities. Journal of AttentionDisorders, 7, 137–149.

Lezak, M. D., Howieson, D. B., & Loring, D. W. (2004).Neuropsychological assessment. New York, NY: OxfordUniversity Press.

Long, C. J. (1996). Neuropsychological tests: A look at our pastand the impact that ecological issues may have on our future.In R. J. Sbordone & C. J. Long (Eds.), Ecological validity of

Dow

nloa

ded

by [

Uni

vers

ity o

f N

otre

Dam

e A

ustr

alia

] at

11:

24 2

7 A

pril

2013

ECOLOGICAL VALIDITY OF THE REY COMPLEX FIGURE 831

neuropsychological testing (pp. 1–14). Boca Raton, FL: GRPress.

Mahone, M. E., Cirino, P. T., Cutting, L. E., Cerrone, P. M.,Hagelthorn, K. M., Hiemenz, J. R., et al. (2002). Validityof the Behavior Rating Inventory of Executive Function inchildren with ADHD and/or Tourette syndrome. Archives ofClinical Neuropsychology, 17, 643–662.

Mahone, M. E., Hagelthorn, K. M., Cutting, L. E.,Scheuroholz, L. J., Pelletier, S. F., Rawlins, C., et al. (2002).Effects of IQ on executive function measures in children withADHD. Child Neuropsychology, 8, 52–65.

Mangeot, S., Armstrong, K., Colvin, A. N., Yeates, K. O., &Taylor, H. G. (2002). Long-term executive function deficitsin children with traumatic brain injuries: Assessment usingthe Behavioural Rating Inventory of Executive Functioning(BRIEF). Child Neuropsychology, 8, 271–284.

Mayes, S. D., Calhoun, S. L., Bixler, E. O., & Zimmerman,D. N. (2009). IQ and neuropsychological predictors ofacademic achievement. Learning and Individual Differences,19(2), 239–241.

McHale, K., & Cermak, S. A. (1992). Fine motor activitiesin elementary school: Preliminary findings and provisionalimplications for children with fine motor problems. AmericanJournal of Occupational Therapy, 46(10), 898–903

Molho, C. E., & Silver, C. H. (1997). An investigation of thevalidity of the children’s executive functions scale. Archivesof Clinical Neuropsychology, 12, 370–371.

Nakano, K., Ogino, T., Watanabe, K., Hattori, J., Ito, M., Oka,M., et al. (2006). A developmental study of scores of theBoston Qualitative Scoring System. Brain and Development,28, 641–648.

Orit, B., Dov, H., & Yair, B.-H. (2007). Predicting school adjust-ment from motor abilities in kindergarten. Infant and ChildDevelopment, 16, 597–615.

Osterrieth, P. A. (1946). Le test de copie d’une figure complexe[The Complex Figure Copy Test]. Archives de Psychologie,31, 206–356.

Parrish, J., Geary, E., Jones, J., Seth, R., Hermann, B.,& Seidenberg, M. (2007). Executive functioning in child-hood epilepsy: Parent-report and cognitive assessment.Developmental Medicine & Child Neurology, 49, 412–416.

Poulton, R. G., & Moffitt, T. E. (1995). The Rey–OsterriethComplex Figure Test: Norms for young adolescentsand an examination of validity. Archives of ClinicalNeuropsychology, 10(1), 47–56.

Rabin, L. A., Barr, W. B., & Burton, L. A. (2005). Assessmentpractices of clinical neuropsychologists in the United Statesand Canada: A survey of INS, NAN, and APA divi-sion 40 members. Archives of Clinical Neuropsychology, 20,33–65.

Rey, A. (1941). L’examen psychologique dans le casd’encephalopathie traumatique [Psychological examina-tion of traumatic encephalopathy]. Archives de Psychologie,28, 286–340.

Savage, C. R., Baer, L., Keuthen, N. J., Brown, H. D., Rauch, S.L., & Jenike, M. A. (1999). Organizational strategies medi-ate nonverbal memory impairment in obsessive–compulsivedisorder. Biological Psychiatry, 45, 905–916.

Sbordone, R. J. (1996). Ecological validity: Some critical issuesfor the neuropsychologist. In R. J. Sbordone & C. J. Long(Eds.), Ecological validity of neuropsychological testing (pp.15–41). Boca Raton, FL: GR Press.

Seidman, L. J., Benedict, K. B., Biederman, J., Bernstein, J. H.,Seiverd, K., Milberger, S., et al. (1995). Performance of chil-dren with ADHD on the Rey–Osterrieth Complex Figure: A

pilot neuropsychological study. Journal of Child Psychologyand Psychiatry, 36(8), 1459–1473.

Seidman, L. J., Biederman. J., Faraone, S. V., Weber, W., &Ouellette, C. (1997). Performance of children and adoles-cents from a large clinically referred sample. Journal ofConsulting and Clinical Psychology, 65(1), 150–160.

Silver, C. H. (2000). Ecological validity of neuropsychologicalassessment in childhood traumatic brain injury. The Journalof Head Trauma Rehabilitation, 15, 973–988.

Smith, S. R., & Zahka, N. E. (2006). Relationship betweenaccuracy and organizational approach on the Rey–OsterriethComplex Figure and the differential ability scales: A pilotinvestigation. Child Neuropsychology, 12, 383–390.

Stern, R. A., Javorsky, D. J., Singer, E. A., Singer, N.A., Somerville, J. A., Duke, L. M., et al. (1999). TheBoston Qualitative Scoring System for the Rey–OsterriethComplex Figure: Professional manual. utz, FL: PsychologicalAssessment Resources.

Strauss, E, Sherman, E. M. S., & Spreen, O. (2006). A com-pendium of neuropsychological tests administration, norms,and commentary. Oxford, UK: Oxford University Press.

Taylor, E. (1959). Psychological appraisal of children with cere-bral defects. Cambridge, MA: Harvard University Press.

Tupper, D. E., & Cicerone, K. D. (1991). Introduction:Developmental and rehabilitative issues in the neuropsychol-ogy of everyday life. In D. E. Tupper & K. D. Cicerone(Eds.), The neuropsychology of everyday life: Issues in devel-opment and rehabilitation (pp. 1–14). Norwell, MA: KluwerAcademic Publishers.

Vriezen, E. R., & Pigott, S. E. (2002). The relationship betweenparental report on the BRIEF and performance-based mea-sures of executive function in children with moderate tosevere traumatic brain injury. Child Neuropsychology, 8,296–303.

Waber, D. P., & Bernstein, J. H. (1995). Performance oflearning-disabled and non-learning-disabled children on theRey–Osterrieth Complex Figure: Validation of the develop-mental scoring system. Developmental Neuropsychology, 11,237–252.

Waber, D. P., Bernstein, J. H., Tarbell, N. J., & Sallan, S.E. (1992). Neuropsychological diagnostic profiles of chil-dren who received CNS treatment for acute lymphoblas-tic leukemia: The systematic approach to assessment.Developmental Neuropsychology, 8, 1–28.

Waber, D. P., & Holmes, J. M. (1985). Assessing children’s copyproductions of the Rey–Osterrieth Complex Figure. Journalof Clinical and Experimental Neuropsychology, 7, 264–280.

Waber, D. P., Isquith, P. K., Kahn, C. M., Romero, I., Sallan, S.E., & Tarbell, N. J. (1994). Metacognitive factors in the visu-ospatial skills of long-term survivors of acute lymphoblasticleukemia: An experimental approach to the Rey–OsterriethComplex Figure Test. Developmental Neuropsychology, 10,349–367.

Watanabe, K., Ogino, T., Nakano, K., Hattori, J., Kado, Y.,Sanada, S., et al. (2005). The Rey–Osterrieth Complex Figureas a measure of executive function in childhood. Brain andDevelopment, 27, 564–569.

Wechsler, D. (2003). Wechsler Intelligence Scale for Children–Fourth Edition. San Antonio, TX: PsychologicalCorporation.

World Health Organization. (2007). International sta-tistical classification of diseases and related healthproblems, 10th revision. Retrieved August 5, 2007,from the World Health Organization Web site:http://www.who.int/classifications/apps/icd/icd10online/

Dow

nloa

ded

by [

Uni

vers

ity o

f N

otre

Dam

e A

ustr

alia

] at

11:

24 2

7 A

pril

2013