on valid inferences: comments on weiss

ON VALID INFERENCES: COMMENTS ON WEISS

Gina Green*

New England Center for Children and E. K. Shriver Center for Mental Retardation

``Extraordinary claims require extraordinary evidence.'' This maxim mightwell serve as a touchstone for those working in early intensive behavioralintervention (EIBI) for autism. Clearly, many mainstream autism professionalsview claims about the e�cacy of EIBI as extraordinary, especially reports thatsome children with autism have achieved normal functioning (e.g., Lovaas, 1987;Maurice, 1993; McEachin, Smith, & Lovaas, 1993; Perry, Cohen, & DeCarlo,1995; for critiques see Gresham & MacMillan, 1997, 1998; Rogers, 1998;Schopler, Short, & Mesibov, 1989; Schopler, 1998). This is not entirely surpris-ing, since such reports contradict the widespread belief that autism is a severe,lifelong disability (see, e.g., Cohen & Volkmar, 1997; Mesibov, 1998; Rapin,1997). Further, as parental demand for EIBI has increased following publicationof Let me hear your voice (Maurice, 1993), so has criticism of behavioral inter-vention by many who espouse other approaches to autism. Some of thosecriticisms re¯ect longstanding misconceptions about behavioral interventionthat have never been substantiated by objective evidence, incomplete apprecia-tion of the substantial body of behavior analytic research that underpins EIBI,or invalid generalizations from one EIBI model (Lovaas) to all others (see, e.g.,Gresham & MacMillan, 1997, 1998; Mesibov, 1998; Wetherby, Schuler, &Prizant, 1997; Schopler, 1998). Some critics garner additional ammunition fromthe fact that competently delivered EIBI is relatively costly in the short run (e.g.,Schopler, 1998), while neglecting to mention that it is not nearly as costly in thelong run as ine�ective interventions (Jacobson, Mulick, & Green, 1998).These and other aspects of the current milieu of autism treatment create a

context in which every report about EIBI is likely to be scrutinized very closely.For the foreseeable future, studies purporting to demonstrate the e�cacy ofbehavioral intervention will likely be held to high standards, perhaps even higherthan those applied to other interventions for autism. That is, extraordinaryevidence will be demanded. At the very least, research on EIBI will be expectedto conform to the basic principles and methods of behavioral science (e.g., see

CCC 1072±0847/99/010023±05$17.50Copyright # 1999 John Wiley & Sons, Ltd.

Behavioral InterventionsBehav. Intervent. 14: 23±27 (1999)

* Correspondence to: Gina Green, New England Center for Children, 33 Turnpike Road, Southborough,MA 01772-2108.

Baer, 1993; Foxx, 1993; Green, 1996; Kazdin, 1993; Mesibov, 1993; Mundy,1993; Mundy & Crowson, 1997). That is as it should be; it is a challenge thatbehavior analysts and other behavioral scientists should welcome. Indeed, ifsound scienti®c research on EIBI is generated in response to this challenge, thenresearchers, clinicians, and consumers alike stand to bene®t. Publication of theforegoing report by Weiss provides a good opportunity to revisit the character-istics of sound scienti®c research and science-based practice in light of some ofthe criticisms of previous studies of EIBI. The comments that follow are o�erednot only to point out the limitations of the Weiss report, but also to suggesttactics that others might pursue to ensure that they have credible evidence tosupport claims about their particular brand of autism intervention.What constitutes credible evidence about the e�ectiveness of behavioral treat-

ment? Put another way, what are the necessary conditions for drawing validinferences that behavior change actually occurred, and that the change was dueto the treatment of interest and not to some other events? Kazdin (1982) pro-vided an elegant framework for considering these questions in his book onsingle-case research methods. The ®rst requirement, of course, is objective,accurate, and reliable measurement of the behavior(s) that may be a�ected by thetreatment. When human observers are used to measure and record behavior,certain practices are essential to counter the well-known tendency for suchobservations to be unreliable and biased. They include, at a minimum, the use ofprecise operational de®nitions and scoring criteria, direct observation andrecording of behavior as it occurs, and multiple independent observers, prefer-ably kept blind to the investigator's hypothesis, whose recordings are frequentlychecked against one another and against the de®nitions and criteria. Otherfactors that determine the accuracy and believability of behavioral data includethe appropriateness of the measurement system to the dimensions of the beha-vior of interest (i.e., frequency, rate, duration, intensity, percent of opportunities,topography, etc), the number of measured observations, and the adequacy ofsampling procedures when continuous observation and recording are notpossible (see, e.g., Barlow & Hersen, 1984; Hersen & Bellack, 1981; Kazdin,1982; Ollendick & Hersen, 1984). Another essential for inferring that observedbehavior change resulted from a particular treatment is the use of trueexperimental designs. Pre- or quasi-experimental designs, such as individual orgroup pretest±posttest arrangements or A±B (baseline±treatment) arrange-ments, do not support such inferences because the in¯uence of extraneousfactors cannot be ruled out with any con®dence (Campbell & Stanley, 1963;Kazdin, 1982). Documenting that the behavior was stable for some time beforetreatment and was not likely to improve without treatment can enhance thevalidity of quasi-experimental studies to some extent, as can including multiple

24 Commentary

Copyright # 1999 John Wiley & Sons, Ltd. Behav. Intervent. 14: 23±27 (1999)

cases, using multiple measures, and demonstrating that relatively immediate andsubstantial behavior change followed introduction of the treatment (Kazdin,1982).Which of the necessary conditions for valid inferences were met in the Weiss

study? Not many, unfortunately. The study purports to show a positive correla-tion between two dependent measures: mastery of early learning skills, andgeneralized outcomes after two years of EIBI. Skill mastery was inferred fromthe author's retrospective review of information recorded by a large number ofdi�erent instructors who reportedly provided treatment to the 20 childreninvolved. The ``measure'' was the number of days required for the children tomaster each skill, as reported by these instructors. This is a questionable metric,because it does not re¯ect how many learning opportunities or trials each childreceived each day, which would be a more meaningful and appropriate measureof acquisition. Indeed, it is not at all clear that any trial-by-trial data wereactually collected according to objective de®nitions and criteria, with periodicconcurrent recording and reliability checks by independent observers.In short, the report provides no evidence that standard direct observational

assessment practices were followed in collecting the skill mastery data. It seemslikely that in many instances the individuals who recorded the information onskill mastery did so, not at the moment they observed the child's response oneach trial, but some time after they had run a number of trials of various types.This has become an all-too-common practice in EIBI, and it is fraught with allthe well-documented problems of subjective, retrospective reports. An informaltest we conducted a while ago at the New England Center underscores theseproblems. We compared estimates of task performance and rates of aberrantbehavior recorded by teachers after they had been working with a student for anhour with data recorded by independent observers as the behaviors occurred.The teachers' after-the-fact recollections both under- and over-estimated thedirectly recorded levels of student behavior by no less than 25%, often more than50%. The implications of these error rates for both clinical and researchdecision-making should be obvious. Weiss reports no data on observer training,interobserver agreement, procedural integrity, or controls for informant bias, sothe probability of serious measurement error in the skill mastery data cannot beruled out. Since this is the measure on which the bulk of her analysis rests, it isdi�cult to place much con®dence in the inference that initial skill acquisitionrates were correlated with or predictive of the outcomes reported for the childrenin her sample.Additional problems are inherent in the outcome measures used here, the

Childhood Autism Rating Scale (CARS) and the Vineland. The author ack-nowledges some of their limitations in the Discussion, but does not provide

Commentary 25


readers with information as to how well those measures correlated (or not) withdirect measures of the children's everyday functioning. Aside from the pre-treatment CARS and Vineland scores, no baseline data were provided againstwhich to compare the e�ects of treatment, or to con®rm the stability of theproblem prior to treatment. Nor were speci®c, objective data on the children'sinitial diagnoses provided; the author herself apparently con®rmed that all of thechildren met DSM-IV criteria for autism or PDD-NOS. This raises a questionabout her choice of the CARS as the primary measure of autistic characteristics,since that instrument is not based on the DSM-IV criteria. Finally, of course,this was not a controlled study, so no inference can be made that the treatmentwas responsible for the reported outcomes.Applying the criteria outlined by Kazdin (1982), the Weiss report can be

characterized as a collection of multiple uncontrolled case studies with pre- andpost-assessment of uncertain objectivity, validity, and reliability. Therefore,inferences based on what is reported here must be considered tenuous. Theintriguing and important question the study raisesÐwhether initial skill acqui-sition rates predict long-term outcomes for children with autism who participatein EIBIÐremains unanswered. Hopefully the Weiss article and accompanyingcommentaries will prompt others working in this area to produce evidence that isreasonably unambiguous, if not extraordinary.

REFERENCES

Baer, D. M. (1993). Quasi-random assignment can be as convincing as random assignment.

American Journal on Mental Retardation, 97, 373±375.

Barlow, D. H., & Hersen, M. (1984). Single case experimental designs. New York: Pergamon.

Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for

research. Chicago: Rand±McNally.

Cohen, D. J., & Volkmar, F. R. (1997). Handbook of autism and pervasive developmental disorders

(2nd edn). New York: Wiley.

Foxx, R. M. (1993). Sapid e�ects awaiting independent replication. American Journal on Mental

Retardation, 97, 375±376.

Green, G. (1996). Early behavioral intervention for autism: what does research tell us? In

C. Maurice, G. Green, & S. Luce (Eds), Behavioral intervention for young children with autism

(pp. 29±44). Austin, TX: PRO-ED.

Gresham, F. M., & MacMillan, D. L. (1997). Autistic recovery? An analysis and critique of the

empirical evidence on the early intervention project. Behavioral Disorders, 22, 185±201.

Gresham, F. M., & MacMillan, D. L. (1998). Early intervention project: Can its claims be

substantiated and its e�ects replicated? Journal of Autism and Developmental Disorders, 28,

5±13.

26 Commentary


Hersen, M., & Bellack, A. S. (1981). Behavioral assessment. New York: Pergamon.

Jacobson, J. W., Mulick, J. A., & Green, G. (1998). Cost±bene®t estimates for early intensive

behavioral intervention for young children with autism: General model and single state case.

Behavioural Interventions, 13, 201±226.

Kazdin, A. E. (1982). Single-case research designs. New York: Oxford University Press.

Kazdin, A. E. (1993). Replication and extension of behavioral treatment of autistic disorder.

American Journal on Mental Retardation, 97, 377±379.

Lovaas, O. I. (1987). Behavioral treatment and normal educational and intellectual functioning in

young autistic children. Journal of Consulting and Clinical Psychology, 55, 3±9.

Maurice, C. (1993). Let me hear your voice: A family's triumph over autism. New York: Knopf.

McEachin, J. J., Smith, T., & Lovaas, O. I. (1993). Long-term outcome for children with autism

who received early intensive behavioral treatment. American Journal on Mental Retardation, 4,

359±372.

Mesibov, G. B. (1993). Treatment outcome is encouraging. American Journal on Mental

Retardation, 97, 379±380.

Mesibov, G. B. (1998). Preschool issues in autism: Introduction. Journal of Autism and Develop-

mental Disorders, 27, 637±640.

Mundy, P. (1993). Normal versus high-functioning status in children with autism. American

Journal on Mental Retardation, 97, 381±384.

Mundy, P., & Crowson, M. (1997). Joint attention and early social communication: Implications

for research on intervention with autism. Journal of Autism and Developmental Disorders, 6,

653±676.

Ollendick, T. H., & Hersen, M. (1984). Child behavioral assessment. New York: Pergamon.

Perry, R., Cohen, I., & DeCarlo, R. (1995). Case study: Deterioration, autism, and recovery in

two siblings. Journal of the American Academy of Child and Adolescent Psychiatry, 34, 232±237.

Rapin, I. (1997). Autism. The New England Journal of Medicine, 337, 97±104.

Rogers, S. J. (1998). Empirically supported comprehensive treatments for young children with

autism. Journal of Clinical Child Psychology, 27, 168±179.

Schopler, E. (1998). Ask the Editor. Journal of Autism and Developmental Disorders, 28, 91±92.

Schopler, E., Short, A., & Mesibov, G. (1989). Relation of behavioral treatment to ``normal

functioning''. Comment on Lovaas. Journal of Consulting and Clinical Psychology, 57, 162±164.

Wetherby, A., Schuler, A. L., & Prizant, B. M. (1977). Enhancing language and communication

development: Theoretical foundations. In D. J. Cohen, & F. R. Volkmar (Eds), Handbook of

autism and pervasive developmental disorders (pp. 513±538). New York: Wiley.

Commentary 27


on valid inferences: comments on weiss

Documents