You Can Go Anywhere from Where You Are:A Response to the Rejoinder

John B. Mordock, PhDConsulting Psychologist

The river of truth is always splitting up into arms which reunite. Is-landed between them, the inhabitants argue for a lifetime as to which isthe mainstream—Cyril Connolly

Bickman and his colleagues made several statements in the re-joinder that suggest I corresponded with them prior to publishing mycritique of the FBEP. I would like to clarify that the only feedback Ihave ever received from this group was a rejoinder to an article1 sub-mitted for publication after, not before, my critique. I had used utili-zation data from the Fort Bragg Demonstration Project (FBDP),where services were provided free, to raise the question aboutwhether clinicians, both in the FBDP and elsewhere, tend to over-utilize treatment as a solution to children's problems. While I had anopportunity to reply to their rejoinder on the service utilization arti-cle,2 their discussion provided a balance view of the topic and made aresponse from me unnecessary.

The authors also believe that I challenged the Fort Bragg findingsbecause I am among those psychologists who feel that since a contin-uum exists it must be effective. Who are these psychologists? Howmany psychologists deliver mental health services to children in thepublic sector where a service continuum can be utilized? Less than 1percent? Most communities have almost no children's mental healthservices, and those that have a continuum are just beginning to learnwhat can be expected from each service.

No one I know would be reluctant to embrace the notion that spe-cific treatments along the service continuum may not help many chil-dren? Every day those of us who do run intensive treatment pro-grams, where disturbed children are surrounded by other disturbed

Received June 25, 1997; Accepted August 15, 1997.Address correspondence to John B. Mordock, Ph.D., 52 Old Farms Road, Poughkeep-

sie, New York 12603.

Child Psychiatry and Human Development, Vol. 29(1), Fall 1998© 1998 Human Sciences Press, Inc. 93

children, wonder if there isn't a better way to help them. Almost ev-eryone knows that grouping disturbed children together for servicescan have negative as well as positive effects, and at least one studyhas empirically demonstrated the negative effects.3

I will restate the reasons I challenged the FBEP findings. First, thestudy has received, and continues to receive, considerably "morepress" than other, well-designed outcome studies of children's treat-ment; second, children in the FBEP got better regardless of theamount or the type of treatment received; and third, because I believethat the type of pre- post-methodology used in the FBEP was inap-propriate for a study of continuum effectiveness. If Bickman and hiscolleagues want to call me "unscholarly" for these beliefs then so be it;disturbed children in crisis have called me far worse.

Outcome Expectations

Typically, children are progressively referred along the mentalhealth continuum following failure to respond to less intensive treat-ments. Before the effects of a continuum can be assessed, however,one must know what effects services on the continuum are expectedto accomplish. The cart must not be put before the horse. And expec-tations change. The current expectation for hospitalization, and forhospital diversion services, is crisis stabilization. The initial expecta-tions for intensive case management (ICM) services were for substan-tial changes in parents and children. Outcome research reveals thatthese expectations were unrealistic. Neither children nor parentsshowed significant changes on standardized measures of psycho-pathology and parental functioning nor did parents achieve statedchild management objectives.4 A more realistic expectation is thatICM services will link children into community resources and preventthe psychiatric hospitalization of many children (but a significant per-centage will need periodic out-of-home placements).

The Same Outcome Measures for Different Services?

Should the same measures be used to assess outcomes for each ser-vice on the continuum when each service has a different objective? Ithink not. The same changes are not expected from hospitalization orhospital diversion as are expected from outpatient services. How

94 Child Psychiatry and Human Development

about day treatment (DT) or residential treatment (RT)? One expecta-tion for DT is that a child will make academic and social gains be-cause the milieu is designed to enable the child to learn these skills inspite of the psychopathology he or she displays. A major goal for RT isthat the family learn to cope with the child more effectively. The childreferred to RT often displays the same level of psychopathology as thechild in DT, but the family of the RT child cannot manage his or herpathology.

Typically, a researcher's expectation for outpatient services is thatthe child's score on a standardized outcome measure will move closerto or within the normal range. But what is the expectation if thechild's score is already in the normal range? And what is the expecta-tion if the child fails to respond to outpatient treatment, gets worse,and is referred to a more intensive treatment setting? Is the moreintensive program expected to have a more potent affect than out-patient treatment and enable the child's score on the out-patient out-come measure to move closer to the normal range? Or is the child'sscore expected to return to the level displayed when first referred tooutpatient treatment? Or is the expectation different from either ofthese objectives?

Which treatment is more potent—16 outpatient visits spread over afive month period or 16 days of hospitalization? This question ismeaningless—the two treatments are used for different purposes.The criteria for admission to and discharge from a hospital are totallydifferent than the criteria for admission or continued stay in an out-patient clinic.

But if an investigator does employ the same outcome measures foreach service on the continuum, than a concurrent research methodol-ogy will produce more useful information.

Concurrent Measurement

Brill and Lish remark "The major question left after a pre-poststudy with a positive outcome is: positive in relation to what?"5 Theconcurrent approach to outcome measurement incorporates measure-ment prior to and after treatment as well as measurement at multipleintermediate points. The intermediate points in a study of continuumeffectiveness would be those upon referral and discharge from eachservice on the continuum. Concurrent measurement can provide iden-tical information about the ultimate outcome of treatment and also

John B. Mordock 95

provide details about the intermediate time course of the change ef-fected by treatment.

Qualifying Statements

Bickman's group place too much emphasis on my statement aboutrandom assignment and too little emphasis on the other statementsin the same paragraph and other points made in the paper. WhileCambell and Stanley felt that findings from quasi-experimental de-signs could be useful when making practical decisions about educa-tional instruction, they also were aware that the findings were notdefinitive. Policy as opposed to practical decisions should be based ondefinitive findings.

The press given the findings from the FBEP include few of thequalifying statements usually found in other published studies, andeven the statements in this rejoinder communicate that Bickman andhis colleagues believe their findings to be true. One does not "prove"the null hypothesis. When psychotherapy outcome research was in itsinfancy, many studies failed to produced statistically significant dif-ferences between treated and untreated groups. Many of these studieswent unpublished because editorial policy at the time put a premiumon statistical significance. Nevertheless, investigators became moreknowledgeable about measurement methodology and nuisance vari-able control and statistically significant results began to appear.

Because some statistically significant differences would be expectedby chance, and because the effect sizes were so small, one renownedpsychologist loudly acclaimed that psychotherapy was of no practicalvalue. Nevertheless, continued research produced many studies withacceptable effect sizes and the efficiency of psychotherapy has beengenerally accepted.

At the same time, because of dissatisfaction with the use of stan-dardized measures to assess change in clients with individual treat-ment goals, the federal government provided considerable financialsupport to develop alternative outcome measures, such as Goal At-tainment Scaling,6,7 popular in the late 1960s and early 1970s andnow fallen into relative disuse with the more recent emphasis on em-ploying symptom targeted outcome measures with homogeneous treat-ment groups having similar if not identical goals.

Bickman and his colleagues discuss none of these issues. For exam-ple, they report change, or gain, scores of the children in the FBEP on

96 Child Psychiatry and Human Development

standardized outcome measures. Nowhere do they qualify their con-clusions with a discussion of measurement methodology problems.(The group have published several articles on measurement in chil-dren's treatment.) When they began their study, the Child BehaviorChecklist was the only instrument selected that had wide acceptanceelsewhere. For example, to my knowledge, the Child and AdolescentFunctional Assessment Scale had not been used in any other outcomestudy and its reliability across different samples had not been estab-lished.8

There are serious statistical problems in the use of change or gainscores. Change scores tend to be unreliable and usually correlate withinitial level, to some degree spuriously because of measurement error,a component that increases in importance as reliability declines. Ceil-ing effects distort interpretation of change at high levels and regres-sion towards the mean also increases in importance as reliability de-creases.9,10 None of these problems were discussed by Bickman and hiscolleagues.

Generalizing the Findings

One of my chief concerns was generalizing the results even if theywere valid. Issues about voluntary as opposed to mandatory treat-ment are secondary to the issue of the disturbance level in the treatedpopulation. I referred to the table on indirect costs because it gavecosts related to other services the children received, e.g. special classplacements, juvenile justice services, etc. This table suggested thatthe costs of these related services were not substantial. Readers areencouraged to read the tables to which Bickman refers (Section IV,Description of the Evaluation Sample) and judge for themselveswhether the premorbid history of the children in the ES is given insufficient detail.

Bickman goes on to say that the population that was studied ismost similar to the estimated 68% of the children in the UnitedStates who are covered by private health insurance and argues thatthe use of a mainstream sample makes the FBEP results more impor-tant than they would be if confined to a special interest group. Unfor-tunately, it is such groups that, outside of the FBDP area, make themost use of intermediate care facilities. I suggest that readers exam-ine Wagner's data on risk factors in children labeled SED by the pub-lic schools and compare it to the data on the FBEP sample.11

John B. Mordock 97

Service Utilization

My seemingly disparate views on service utilization are easily ex-plained. In the article referred to earlier, I expressed the concern thatFBDP clinicians, and by inference clinicians elsewhere, may placemany children in group homes or community residences to solvetreatment crises rather than struggle with families to modify treat-ment plans. Nevertheless, if a SED child is appropriately placed in aday treatment setting, 60.3 days, or two months, is an "inconsequen-tial amount of service," particularly when spread over several differ-ent placement episodes, as is 115.9 days, or just under four months, ofdifferent episodes of residential treatment (an average of 4.7 episodesper placed child, with an average length of stay of 20 days per epi-sode). Such brief stays will not result in major changes in psycho-pathology or development advancements in most seriously disturbedchildren and, therefore, these placements could be considered at bestas stabilization efforts.

A Concluding Remark

The older I get, the less certain I am about most things. But the onething I am certain about is that good research is hard to do. Meaning-ful findings are not necessarily the result of sample size, moneyspent, time devoted, or energy expended. I wish Bickman and his col-leagues well in their future research endeavors. But I would advisethem to approach the task with more humility.

References

1. Mordock, JB: Utilization rates in children's mental health systems: more evidencefor clinician's illusion?" Clin Child Psychol Psychiat 2:579-590, 1997.

2. Summerfelt WT, Salzer M, Bickman L: Interpreting differential rates of serviceuse: avoiding myopia. Clin Child Psychol Psychiat 2:591-596, 1997.

3. Dishion TJ, Andrews DW: Preventing escalation in problem behaviors with high-risk young adolescents: Immediate and 1-year outcomes. J Consult Clin Psychol63:538-548, 1995.

4. Huz S, Evans, ME, Rahn, DS, McNulty, TL: Evaluation of intensive case manage-ment for children and youth: Third year final report. Bureau of Evaluation andService Research, New York State Office of Mental Health, Albany, New York,1993.

5. Brill PL, Lish DL: Timing is everything: Pre-post versus concurrent measurement.Behavioral Healthcare Tomorrow 4:76-77, 1995, p. 76.

98 Child Psychiatry and Human Development

6. Seaberg JR, Gillespie DF: Goal attainment scaling: A critique. Soc Wrk Res & Abs13:4-7, 1973.

7. Kiresuk TJ, Sherman RE: A reply to the critique of goal attainment scaling. SocWrk Res & Abs 13:9-11, 1973.

8. Hodges K: The child and adolescent functional assessment scale (CAFAS). Un-published manuscript, Ann Arbor, MI, 1990.

9. Cronbach LJ, Furby L: How we should measure "change"—Or should we? PsychBull 74:68-80, 1970.

10. Manning WH, DuBois PH: Correlational methods in research on human learning.Percep Mot Skills 15:288-321, 1962.

11. Wagner MH: Outcomes for youths and serious emotional disturbance in secondaryschool and early adulthood. Future of Child 5:90-112, 1995.

99John B. Mordock