1988;82;415-424 pediatrics burns and david brent elizabeth j

1988;82;415-424 PediatricsBurns and David Brent

Elizabeth J. Costello, Craig Edelbrock, Anthony J. Costello, Mina K. Dulcan, Barbara J. Psychopathology in Pediatric Primary Care: The New Hidden Morbidity

http://www.pediatrics.orgthe World Wide Web at:

The online version of this article, along with updated information and services, is located on

Online ISSN: 1098-4275. Copyright © 1988 by the American Academy of Pediatrics. All rights reserved. Print ISSN: 0031-4005. American Academy of Pediatrics, 141 Northwest Point Boulevard, Elk Grove Village, Illinois, 60007.has been published continuously since 1948. PEDIATRICS is owned, published, and trademarked by the PEDIATRICS is the official journal of the American Academy of Pediatrics. A monthly publication, it

at Duke University on September 26, 2007 www.pediatrics.orgDownloaded from

http://www.pediatrics.org

http://pediatrics.aappublications.org

PEDIATRICS Vol. 82 No. 3 Pt. 2 September 1988 415

Psychopathology in Pediatric Primary Care:The New Hidden Morbidity

Elizabeth J. Costello, PhD, Craig Edelbrock, PhD,Anthony J. Costello, MD, Mina K. Dulcan, MD,Barbara J. Burns, PhD, and David Brent, MD

From the Departments of Psychiatry, the University of Pittsburgh; the University ofMassachusetts, Worcester; Emory University, Atlanta; and the University of Maryland,Baltimore

ABSTRACT. In a study ofemotional and behavioral prob-lems seen in children attending pediatric primary careclinics in a health maintenance organization, parents of789 children 7 to 11 years of age completed a behavior

screening questionnaire, the Child Behavior Checklist.Of the 195 (24.7%) children identified by the checklist asdisturbed, 126 were given a detailed psychiatric assess-ment using the Diagnostic Interview Schedule for Chil-then, a structured psychiatric interview of known validityand reliability. A randomly selected group of 174 nondis-

turbed children was also assessed. The pediatricians’judgment about the presence of emotional and behavioralproblems, made at the index clinic visit, was comparedwith diagnoses made from the computer-scored inter-views. Pediatricians diagnosed one or more such problems

in 5.6% of the children (weighted estimate: 95% confi-dence limits 3.8% to 7.6%), compared with 11.8% (95%confidence interval 9.3% to 13.5%) based on the interviewwith the parent. Pediatricians were highly specific, ie,84% of children assessed as nondisturbed had no psychi-atric disorder, but they showed low sensitivity, ie, theyonly identified 17% of the children with behavioral oremotional problems, giving a “hidden morbidity rate” of83% (ie, 83% of cases were not identified). The role ofprimary care pediatricians in the identification, preven-tion and treatment of what has been called “the newmorbidity” is discussed. We suggest that, on the basis of

these findings, emotional and behavioral problems inchildren have to be seen as “the new hidden morbidity.”Pediatrics 1988;82(pt 2):415-424; primary care, psycho-pathology, health maintenance organization.

ABBREVIATIONS. CBCL, Child Behavior Checklist; DISC, Di-agnostic Interview Schedule for Children; P, parent; C, child;

DSM-III, Diagnostic and Statistical Manual ofMental Disorders,ad 3.

Received for publication Feb 3, 1987; accepted Dec 1, 1987.Reprint requests to (E.J.C.) Division of Child and AdolescentPsychiatry, Department of Psychiatry, Box 3271, Duke Univer-sity Medical Center, Durham, NC 27710.

PEDIATRICS (ISSN 0031 4005). Copyright © 1988 by theAmerican Academy of Pediatrics.

In 1975 Haggerty et al described “the new mor-

bidity” in pediatric primary care’�316�:

The major health problems ofchildren today are different

from those that prevailed when pediatrics came intoexistence a century ago. . . . Learning difficulties andschool problems, behavioral disturbances, allergies,speech difficulties, visual problems, and the problems of

adolescents in coping and adjusting are today the most

common concerns about children.

In grappling with the problems posed by the new

morbidity, the pediatric profession finds itself fac-ing a large and complex task.2 As the AmericanAcademy of Pediatrics’ Committee on PsychosocialAspects of Child and Family Health pointed

out3�’26�:

The pediatrician is increasingly expected to be concernedwith the prevention, early detection, and management ofpsychosocial problems pertinent to optimal child andfamily health and development. . . . The extensive scien-tific and experiential knowledge base applicable to pedi-atric practice includes normal growth and development;adaptive and pathologic psychosocial development . . . thenature and significance of psychometric tests to assessstatus, aptitude, and achievements; . . . preventive inter-vention in primary care and counseling at-risk families;and principles of the psychotherapeutic role of the podia-

trician.

The process of incorporating the new morbidityand its treatment into standard pediatric care hasmajor implications for training, for professionalrole definition, and for the way in which the nec-

essary services are delivered to children. For ex-ample, to what extent should primary care pedia-

tricians take on the entire task of prevention, earlydetection, and management3 of psychosocial prob-lems, rather than acting in this area as primary

diagnosticians, referring children to other profes-



416 PSYCHOPATHOLOGY

sionals for further evaluation and care? These andrelated issues have been the focus of discussion and

debate at a time when pediatric residency trainingprograms have greatly expanded their teaching in

behavioral pediatrics as part of the continuing rev-olution in the definition of pediatrics as a specialty.2

One of the issues faced by the pediatric commu-nity in coming to terms with the new morbidity is

lack of information about the scale of the problem.How many children seen in pediatric primary care

have significant emotional and behavioral prob-

lems? Do primary care pediatricians identify these

children correctly? According to currently available

studies, the proportion of children identified as

disturbed by primary care pediatricians ranges from

0.6% to 16%, with a median rate of 4% to 7%#{149}4

However, these estimates are based solely on diag-

noses made by primary care pediatricians. To date,

no studies have been carried out in the UnitedStates to compare pediatricians’ diagnoses with

those based on standard psychiatric assessments of

the same children. Thus, it is not possible to besure whether the level of ascertainment observed

in current pediatric practice accurately reflects the

size of the real problem facing pediatricians. In the

adult population, there is considerable evidence of

“hidden � ie, that a large proportion ofmental illness is not detected by primary care phy-sicians.57 Is the same true in pediatrics?

The study described here was carried out to ad-dress this issue. Children attending primary care

pediatric settings for routine health care were eva!-

uated using instruments of known reliability andvalidity as measures of child psychopathology. By

comparing the findings with the pediatricians’ judg-ments about the presence and nature of the chil-

dren’s emotional and behavioral problems, we es-

timated the extent to which the new morbidity is

still a hidden morbidity.

METHODS

Design

In this study two approaches to assessing emo-tional and behavioral problems seen in a pediatric

setting were compared. The first approach was that

of primary care pediatricians going about their reg-

ular tasks, for whom assessment of children’s men-

tal health is one of many responsibilities. The study

was carried out with the full knowledge and coop-

eration of the pediatricians concerned, who were

asked to complete a brief checklist regarding eachstudy child they saw as part of their regular clinical

duties. The rate of diagnosed disorder identified by

this approach was, therefore, that detected by pri-

mary care pediatricians performing their regular“psychotherapeutic 3 on a day-to-day basis.

The second approach to assessment taken in the

study was modeled on the detailed assessments

carried out in a specialist mental health facility.Experienced psychiatric social workers interviewedthe children and their parents and collected addi-

tional information about family functioning and

school performance. The methods used were specif-

ically designed for community studies of child men-

tal health. The sampling strategy was designed to

provide a good estimate of the amount of psycho-pathology that would be found if every child 7 to

11 years of age visiting the pediatrician’s offices

were to be given a detailed psychiatric assessment

taking several hours of professional resources. Noa priori assumption is made that one estimate is

“correct” and the other “incorrect.” Referral formental health services depends on factors other

than diagnosis. However, diagnosis is a necessarystarting point, and an aim of the study is to de-

scribe, for the first time in a large-scale Americanstudy, the relationship between the two approaches.

A two-stage study design was used. All 7- to 11-

year-old children visiting the pediatricians’ officeswere screened, using a brief behavioral question-naire completed by a parent. On the basis of scoresfrom the questionnaire, a subsample was selected

for more detailed interviews and follow-up.

Setting

The study was carried out in a large health main-

tenance organization (HMO). Mental health serv-ices are provided almost entirely by the HMO’s own

staff of psychiatrists, psychologists, counselors, and

social workers. (Although provisions exist for psy-

chiatric hospitalizations outside the HMO, ifneeded, none of the children in this study used

outside services during the year of observation.)

Except for a restriction regarding the number ofdays of hospitalization the plan will pay for, cost is

not a barrier to referral or treatment for psychiatricproblems. All children must, however, be assessed

by a pediatrician before being referred for psychi-

atric evaluation. During the year preceding this

study, 3% of children studied had received some

form of mental health evaluation or treatmentthrough the HMO.

Pediatric clinic visits were scheduled every 20minutes by the HMO, although the pediatricians

had some latitude in this; for example, they wouldschedule patients they expected to require longer

as the last appointment of the day.

Two of the six medical centers run by the HMOwere used for the study. The “urban” site was close



ARTICLES 417

both to two large universities and to an area of thecity inhabited mainly by a poor, black populationand drew its subscribers from both settings. The

“suburban” site drew its subscribers from a wide

geographical area; most of them were white andworked mainly in light manufacturing and serviceindustries. Each site had approximately 1,400 chil-

dren 7 to 1 1 years of age enrolled. Because the

HMO was growing rapidly during the year of thestudy, this figure could only be approximate.

Measures

The Child Behavior Checklist (CBCL) is a ques-

tionnaire composed of behavior problems and social

competence items, completed by parents about thechild.8 It has been standardized on 1,300 “normal”

4- to 16-year-old children and 1,300 children re-

ferred for mental health care. The 90th percentilewas the cutoff point for nonreferred children and

the CBCL was found to classify correctly 82.6% of

the referred and nonreferred children.9 Based on

the 3% referral rate for mental health care in the

HMO found in this age group, about 12% of theHMO sample could be expected to have total be-

havior problem scores above the 90th percentile fornonreferred children.9”#{176} However, some recent

studies in which the CBCL was used as a screen for

some recent psychiatric disorders in children have

found that the cutoff point recommended by Ach-

enbach and Edelbrock9 on the basis of their nor-mative samples is not specific, ie, it identifies as

disturbed children who are not given a clinical

diagnosis after a more detailed evaluation. For ex-

ample, Bird et a!” found that the CBCL used inthe manner described for the present study identi-

fled 26% of the nondisturbed children as disturbed,and Verhulst et al’#{176}found that 22% of nondisturbed8-year-old children and 18% of nondisturbed 11-year-old children had scores in the clinical range ofthe CBCL. Thus, we anticipated that the CBCL

would identify more disturbed children than eitherthe pediatricians or the detailed psychiatric assess-

ment. This was acceptable for the purpose of this

study, which used the CBCL as a means to ensure

that, given limited resources for detailed interview-

ing, an adequate number of disturbed childrenwould be included in the interview stage of the

study. This was done to improve the stability of the

prevalence estimates derived from it. Because briefscreening questionnaires are unlikely to be com-

pletely accurate, a number of false-positive findings

from the interviewed sample is a necessary corollaryof a reasonable level of sensitivity to the presence

of clinically significant psychopathology. The use

of a screening questionnaire by pediatricians them-

selves as an aid to diagnosis is a different issue and

is discussed elsewhere.’2

The Diagnostic Interview Schedule for Children

(DISC)’3 was given to the subsample of 300 parents

and children by social workers with experience inchild psychiatry. The DISC consists of two parallel

interviews about the child’s emotional and behav-

ioral problems: the DISC-C for children and the

DISC-P for parents or caretakers. The NationalInstitute of Mental Health commissioned the de-

velopment of the DISC as a structured psychiatricinterview for epidemiologic studies that require

consistent, detailed information about a wide range

of current psychiatric symptoms. Responses to theDISC questions are coded as 0 (no), 1 (sometimes

or maybe), or 2 (yes). Diagnoses may be made by aclinician who has administered the interview, orthe responses can be scored by a computer to yieldboth symptom scores and Diagnostic and Statistical

Manual of Mental Disorders, ed 3 (DSM-III), diag-noses for a wide range of psychiatric disorders.’4

Computer diagnoses are based on the criteria pro

vided by DSM-III but are scored at two levels. Level1 is a precise operationalization of the DSM-III

criteria. Previous studies,’5”6 however, suggest thatat this level the DISC may be overinclusive by

clinical standards, and a more stringent level is alsoscored, with criteria derived from clinical practice.The two levels are labeled “possible” and “probable”

diagnoses. Disorders requiring additional informa-

tion, eg, a medical examination, are scored as “pos-

sible” diagnoses only. Only diagnoses meeting the

more stringent criterion are reported in this paper.

Each of the DISC interviews takes one to 1#{189}hours

to complete and is too lengthy for routine use in aprimary care setting. It is, however, a rich sourceof information about psychiatric symptoms and

syndromes and is well adapted for epidemiologicresearch.

In the absence of pathognomic tests for psychi-atric disorders, epidemiologic as well as clinical

studies are largely dependent for their validity on

the quality of the diagnostic information obtainedfrom interviews. In recent years several more or

less structured interviews have been developed in

an attempt to improve on the poor test-retest reli-ability of clinical assessments.’7 Although undoubt-

edly improving the reliability of data collection, the

use of these interviews raises new questions about

the validity of the diagnoses that are made using

them. A review of the available psychometric

information’6 shows that, as would be expected ofa highly structured interview, the DISC has highinterrater and test-retest reliability compared with

other interviews. If clinician’s judgment is the cri-

tenon, the DISC shows reasonable to good validity

for the major diagnostic categories.’�’9 More fine-



418 PSYCHOPATHOLOGY

grained distinctions, such as those among the var-

ious types of conduct disorder, generally show poor

concordance and are not reported in this paper.A characteristic of structured interviews not gen-

erally found in normal clinical practice is that chil-

dren often receive multiple diagnoses. This occurswhether clinical interviewers or computers are usedto aggregate the information and make the diag-

noses. It is probably the result of the more system-atic and widespread review of symptoms that struc-

tured interviews encourage. Thus, we can predict

that the children in this study are more likely toreceive multiple diagnoses from the DISC inter-

views than from the pediatricians. For this reason

the data were analyzed so as to compare childrenwith “one or more” diagnoses from either source,

as well as by specific diagnoses. There exists, how-

ever, the possibility that the computer algorithms

might have overdiagnosed specific disorders, thusproducing a low level of concordance between pe-

diatricians’ and DISC interviews. To test this, weexamined the agreement between psychiatric cli-nicians’ and computer diagnoses for each disorder,

for cases with and without a pediatrician’s diagno-sis.

A Health Practitioner Report, designed for thisstudy, was completed by the pediatrician for eachchild. Included in the report were: how many times

the pediatrician has seen the child before; the pre-

senting complaint; the primary medical diagnosticformulation, and a secondary one if appropriate;whether the pediatrician knows of any family his-

tory of emotional, behavioral, or learning problemsor major family stressors; whether the child is men-tally retarded or developmentally delayed; and any

current psychiatric diagnoses using the Intenna-

tional Classification of Diseases, rev 9, system(American version). Emotional and behavioral

problems were defined for these analyses in threeways: (1) the pediatricians’ diagnoses of current

emotional and behavioral problems, made at the

index visit; (2) the presence of one or more DSM-III diagnoses based on the psychiatric interview

with the parent (DISC-P); (3) a total behaviorproblem score at the index visit above the “clinical”level on the CBCL.

Procedure

Morning, afternoon, and evening clinical sessionswere sampled in proportion to the number of hoursof each type of session during a given month. Ap-proximately 5,700 hours of clinic time were heldduring the period of the study, from November 1984

through October 1985. During that period, research

assistants were present at the two sites for 50.7%of the clinic hours. Parents of all children eligible

for the study were recruited when they brought thechild into the clinic. All 7- to 11-year-old children

visiting the clinic for any reason were eligible, with

the following exceptions: only one age-appropriatechild was screened from any family; children were

only screened once and were only eligible for screen-ing three times; if they visited the clinic more often,but for some reason had not previously beenscreened, they were excluded from the samplingpool as a precaution against oversampling children

with multiple visits. There were only three of thesecases. When an eligible child came to one of the

clinics, a research assistant explained the study to

the accompanying adult (the mother in 88% ofcases) and asked her to complete the CBCL about

the index child. The Health Practitioner Form wasattached to the child’s medical record, to be com-

pleted by the pediatrician who saw the child at thatvisit. At the requests of the pediatricians, CBCLswere completed after the child had been seen.

A subsample of screened children was recruited

for intensive parent and child DISC interviews.This group consisted of all children scoring above

the 90th percentile on the screening questionnaire

and a random sample of children scoring in thenormal range. A total of 789 completed screeningswere obtained, and a subsample of 300 families wasinterviewed: 126 containing a high-scoring index

child and 174 containing a low-scoring child. Re-

fusal rates were 6% for the screening phase and26% for the interview phase. Families of children

scoring high and low on the CBCL were equally

likely to refuse to be interviewed, and the distribu-

tion of refusers by race, sex, age, social class, andstudy site was similar to that of the study popula-

tion as a whole. Interviewers, who were experiencedpsychiatric social workers specially trained to ad-

minister the DISC and to make DSM-III diagnoses,were unaware of the screening status of the child.

RESULTS

Rates of Diagnosis and Referral

Of the 789 children for whom CBCLs were com-

pleted, 195 (24.7%) scored above the cutoff pointdefining the 90th percentile of Achenbach and

Edelbrock’s nonreferred sample. Item by item com-parison of this group with the age-matched children

from their normative sample was carried out to testthe possibility that more “somatic” items, such asnausea, dizziness, and tiredness, were endorsed forthe pediatric sample. This was not the case; parentsof the pediatric sample endorsed every item in thescale more frequently than did parents of the nor-



ARTICLES 419

mative sample. Although this high rate of scores in

the clinical range is puzzling, it should be notedthat other recent community studies have found

the same thing.bofl19a It is possible that our findingis part of a general increase in parent reports of

behavior problems.The numbers of children with one or more clini-

cally significant psychiatric disorder, based on thepediatricians’ diagnosis and the psychiatric inter-

view with the parent are given in Table 1. The rates

found in the screen-positive and screen-negative

children interviewed were weighted to produce a

prevalence rate for the original screened sample;that is, the rate that would be expected had all ofthe 789 screened children been interviewed. Pedia-tricians diagnosed emotional or behavioral prob-lems in 5.6% of the sample (Table 2); the psychi-

atric interview with the parents identified 11.8% of

the sample as suffering from one or more clinically

significant psychiatric disorders. The 95% confi-dence intervals around the two estimates do notoverlap. It is clear that the standardized psychiatric

TABLE 1. Children Given S pecific Diagnoses*

Specific Diagnosis Pediatrician

(ICD-9A)Interview

WithParent

(DSM-

III)

Attention deficit disorder

(DSM-III)Hyperactivity (ICD-9A)

4 12

Conduct disorder and/or op-positional disorder (DSM-III)

13 37

Disorder of conduct (lCD-9A)

Anxiety disorders, phobias(DSM-III)

11 52

Anxiety disorder, fears, andphobias (ICD-9A)

Depression, dysthymia

(DSM-III)1 5

Major depressive disorder(ICD-9A)

Enuresis 8 27

Encopresis 9 7

* Results are numbers of children. Abbreviations: lCD-

9A, International Classification ofDiseases, rev 9A; DSM-III, Diagnostic and Statistical Manual of Mental Disor-dens, ed 3. Specific diagnoses include only those on whicha comparison between ICD-9A and DSM-III axis I is

possible and exclude developmental delay, mental retar-dation, learning disorders, psychologic factors in a phys-ical condition, and “other diagnoses” (ICD-9A).

assessment identified more than twice as manychildren as the pediatricians.

The rate of disorder identified by the pediatri-cians in this study was close to the median for

studies of primary care pediatric settings.4 The ratefound by the psychiatric interview with the parent

was at the median for the best available current

estimates of child psychopathology in the generalpopulation.20’2’ Taken together, the results of thescreening and the psychiatric interviews demon-

strate that the low rate of psychiatric disorderidentified by the pediatricians was not the result of

a low rate of psychopathology in the sample and

does not reflect rates found in studies of generalpopulation samples.

The numbers of diagnoses made in specific di-agnostic categories by pediatricians and computer

are shown in Table 1. In all categories except en-copresis, the computer made more diagnoses than

the pediatricians. The rank order of the primarydiagnoses made for the 789 screened children, di-

vided into the major International Classification of

Diseases, rev 9, categories, is shown in Table 3. Hadpsychopathology been diagnosed at the rate mdi-

cated by the psychiatric assessment described here,it would have been the third most frequent diag-nosis made. Children with a current diagnosis inthe category of “signs, symptoms, and ill-definedconditions” were significantly (P < .05) more likely

to be seen as disturbed than other children. There

were no significant associations with any other typeof diagnosis or with visits for prophylactic or well-

child care.

Agreement Among Diagnostic Measures

The concordance between pediatricians’ diag-noses and those made using the psychiatric inter-view with the parent is shown in Table 4. Because

our concern here is with agreement rather thanprevalence rates, the data are presented for the 300interviewed cases rather than as estimates for thetotal sample of 789. The sensitivity of the pediatri-

cian diagnoses relative to the psychiatric interview,ie, the proportion of patients correctly identified ashaving a psychiatric problem is also shown. The

TABLE 2. Psychiatric Disorders Diagnosed by Pedia-tricians and Psychiatric Interview With Parents*

Source of Diagnosis

Pediatrician Interview

With

Parent

Percentage 5.6 11.8

95% Confidence limits 3.8-7.6 9.3-13.5

* Weighted estimate based on 126 screen-positive and

174 screen-negative patients.



RankOrder of

Frequency

1

2

Diagnosis % ofSample

(n = 778)

21.3

17.1

0.60.55.8

20.6

11.8PositiveNegative(6 cases missing)

* Sensitivity, 17%; specificity, 84%;

83%. Six cases were missing.

hidden morbidity,

420 PSYCHOPATHOLOGY

specificity, ie, the proportion of children without

problems correctly identified as healthy, and thefalse-negative or hidden morbidity rate,5’7 the pro-

portion of psychiatric cases that was not identifiedby the pediatricians, are shown, too. It can be seen

that the pediatricians showed a high degree of spec-

ificity; that is, they correctly identified 84% of the

healthy children. However, they only identified

17% of the children with psychiatric problems and,

conversely, failed to identify 83% of disturbed chil-

dren.Taken together, Tables 1 and 4 demonstrate that,

in this sample of 7- to 11-year-old children attend-ing their primary care pediatrician, in which almost

TABLE 3. Primary Medical Diagnoses Made by Pro-viders*

Respiratory system disorders

Nervous system and senseorgan disorders (includesmiddle ear infections)

3 Infective and parasitic disor-

ders4 Accidents, poisoning, vio-

lence5 Disorders of the skin and

cellular tissue6 Signs, symptoms, and ill-de-

fined conditions7 Mental disorders8 Disorders of the musculo-

skeletal system9 Disorders of the digestive

system

10 Endocrine, nutritional, and

metabolic disorders11 Disorders of the genitouri-

nary system12 All other medical disorders

13 Congenital anomolies

No disorderProphylactic and well-child

proceduresEmotional and behavioral

disorders (Diagnostic In-terview Schedule for Chil-then-Parent)

12% had one or more clinically significant emo-

tional or behavioral problems, the pediatricians

failed to identify 83% of the disturbed children.

Further analyses by type of disorder showed thatpediatricians had the highest level of sensitivity to

functional enuresis and encopresis (27%) and the

lowest level of sensitivity to emotional problems

such as anxiety and depression (6%). Sensitivities

to behavioral problems and learning problems wereintermediate (13%). The sex, age, race, and socio-

economic status of the child did not significantlyinfluence these results. Of the children identified

by the pediatricians, 15 (65%) had been referred to

the HMO’s mental health services, or had receivedtreatment there, during the previous year. This was

equivalent to a referral rate of 3.6% for the sampleas a whole. Seven children of the referred group

were identified using the structured interview, andeight were identified by the clinical interviewers;

six were identified by all three measures. Thus,there was a “core group” of severely disturbed chil-

dren, equivalent to 1.2% ± 0.34% of the total sam-

ple, on whom everyone agreed, and who had been8.2 referred for treatment.

8 1 The question of whether the low level of concord-. ance between pediatrician and DISC-P diagnosis

5.3 can be explained in part by a low level of concord-ance between the computer-generated diagnoses

3.3 and the diagnoses that practicing psychiatric clini-

3 3 cians would make is addressed in Tables 5 and 6. If

18 the clinicians did not agree with the computer, wecould hardly expect the pediatricians to do so. Each

1.5 2 by 2 cross-tabulation shows the concordance be-

tween clinicians and computer. Emotional and be-1.3 havioral disorders were analyzed separately. It is

1.2

* Diagnoses taken from International Classification of

Diseases, rev 9A.

TABLE 4. Associations Between Pediatricians’noses of Psychiatric Disorder and Those Based onchiatric Interview With the Parent (n = 300)*

Diag-Psy-

�1 Diagnosis �1 Diagnosis Basedby on Parent Interview

Pediatrician . .

Positive Negative

9 14

43 228

TABLEPediatri

5. Relationship Between Clinician and Computer Diagnoses in Cases With a

cian Diagnosis*

Clinician

Diagnosis

Computer Diagnosis

. . .

Behavioral Disorders Emotional Disorders

Present Absent Present Absent

Present 3 1 1 1

Absent 0 4 2 3

* Results are numbers of diagnoses and exclude children with only enuresis, encopresis,

or a learning disorder.



ARTICLES 421

TABLEa Pediat

6. Relationshiprician Diagnosis*

Between Clinician and Computer Diagnoses in Cases Without

Clinician

Diagnosis

Computer Diagnosis

. . .

Behavioral Disorders Emotional Disorders

Present Absent Present Absent

Present 25 81 23 59- Absent 2 178 6 205* Results are numbers of diagnoses and exclude children with only enuresis, encopresis,

or a learning disorder.

clear that, although the clinicians identified morecases than the computer, the proportion of patients

for whom the DISC-P identified disorders in theabsence of a clinician diagnosis is low for bothbehavioral and emotional problems. Furthermore,

there were only two patients for whom the clini-cians and pediatricians agreed on the presence of a

disorder in the absence of a DISC-P diagnosis: oneof attention deficit disorder with hyperactivity and

one of anxiety disorder. In both cases the child metcriteria for a “possible,” but not a “probable,” DISC

diagnosis. By contrast, pediatricians identified cur-

rent behavioral problems in four children and anx-iety problems in five children not identified by theclinicians or the computer (Table 5). Thus, there is

no evidence from this study that the differences

between DISC-P and pediatricians’ diagnoses canbe ascribed to pediatricians agreeing with the cli-nicians rather than with the computer diagnoses.

Issues of Reliability and Validity

Highly structured psychiatric interviews are notthe normal method of assessing children’s mental

health in primary care settings. Even in specialistmental health clinics, where their use is spreading,

the interviewer’s clinical judgment is usually calledon, rather than a computer program, to make the

final diagnosis. The question thus arises of whetherthe results found in this study have to do withcharacteristics of the DISC rather than of the pe-

diatricians. It is possible to argue that, rather than

the pediatricians underdiagnosing child psycho-

pathology, the DISC overdiagnosed it, in this sam-pie.

There are two issues that must be considered inassessing the performance of a diagnostic test,

whether it be a laboratory test or a human usingclinical judgment: reliability and validity. Reliabil-ity, in this context, refers to one of the desirable

characteristics of any diagnostic test in medicine:

that it should give the same results in the hands of

anyone competent to administer it; ie, it should

measure characteristics of the patient, not of the

clinician. Highly structured interviews such as the

DISC are among the most reliable ways of assessing

child psychiatric symptoms.’7 However, the useful-ness of a diagnostic procedure lies not only in its

reliability but also in its ability accurately to distin-

guish people with the disease in question from thosewithout it. The diagnostic accuracy of a test (some-

times called criterion validit?2) depends on severalfactors in addition to reliability, above all that thecriterion should be clearly defined so that the same

pattern and severity of symptoms will always leadto the same diagnosis. It is also highly desirable

that the diagnostician should be able to learn

whether the diagnosis was right or wrong, becausewithout this feedback, improvement in accuracy is

difficult and uncertain. Psychiatry is still strugglingwith the problem of criterion validity. Some prob-

lems are empirically resolvable, at least in principle;for example, the DSM-III diagnosis of functional

encopresis’4�”�2� (“repeated . . . passage of feces ...

into places not appropriate for that purpose . . . at

least once a month after the age of four . . . not dueto a physical disorder”) provides a clearly definedset of criteria that a good diagnostic measure should

be able to identify with accuracy, whose course andoutcome can be reliably assessed, and whose clinicalimportance, in terms of suffering and impaired

functioning, is widely acknowledged. It is signifi-cant that encopresis and enuresis were the diag-

noses that showed the highest concordance between

pediatricians and the DISC in this study. In otherareas, however, it is much more difficult to decide

on the criteria for the presence of a disorder. Ex-perts disagree on the critical symptoms and sever-ity, duration, and level of impairment that define a

clinically significant case. Feedback, in the form of

systematic information about the course and out-come of various syndromes, is slow and uncertain.

Given this uncertainty about what constitutescriterion validity for many psychiatric disorders,

how should we evaluate the findings of this study?A structured interview, one can argue, may well behighly reliable for identifying symptoms and group-ing them into syndromes, but the pediatricians are

concerned with identifying children who really needexpert help, and they do this by clinical judgmentbetter than a rigid interview, however reliable, canpossibly do.



422 PSYCHOPATHOLOGY

The best test of this argument will be to follow

these children, both the “cases” and the “noncases,”

and observe which ones show clinically significantpatterns of illness and functional impairment

throughout time. This approach to accuracy isknown as the predictive validity�2 of a test. Mean-while, we have to make do with other approaches.

One of these, known as concurrent validity,22 com-

pares the performance of a given test against that

of other measures. In this study we can look at theperformance of the DISC and the pediatricians in

the light of two other assessments of the children:the CBCL and the psychiatric social workers’ di-

agnoses, made after the interviews. Even without a

clearly defined criterion against which to assess allthese measures, certain observations can be made.

Prevalence Estimates. The CBCL and the clinicalinterviews identified more children as disturbedthan either the DISC-P or the pediatricians. Thecriterion of “caseness” for the CBCL is the level of

reported symptoms found in most children referred

for mental health services.9 Although clearly far

from ideal as a measure, it has the virtue of being

based on observed behavior: in a representativesample of 2,600 children, 77% of those referred for

mental health services scored above the cutoffpoint, whereas 90% of nonreferred children scoredbelow it.9 Although later studies have suggested

that this level of symptomatology may be found

nowadays in a larger percentage of nonreferred

children,bo,hl,19a this does not alter the fact thatalmost one quarter of the children in this study hada level of reported symptoms similar to that seenin three of four psychiatric referrals.

The clinical interviewers in the study were ac-customed to evaluating children seen in a mental

health setting, where the a priori assumption is oneof “caseness.” In this study, where the interviewers

were unaware of the children’s CBCL scores andsaw them in their own homes, it might be expected

that by comparison with their clinical caseloadstheir a priori assumption would be one of normality.In the event, they identified one or more disordersin 44.2% of the children (95% confidence interval

36.7% to 48.7%). Thus, both clinical judgment and

an empirically derived symptom-based criteria of

“caseness” found more rather than fewer cases.

Concordance. Both clinical judgment and theCBCL showed much higher concordance with theDISC than with the pediatricians, ie, they agreedon the children identified as disturbed and nondis-turbed to a greater extent than the pediatriciansdid with any of the other measures. The DISC-P

“cases” were in general a subset of both the CBCLcases and the clinical interviewers’ cases. This is

demonstrated in Table 7 for the pediatricians com-pared with the CBCL, the interviewers and theDISC, using the statistic K as a measure of chance-corrected agreement. Thus, the pediatricians not

only identified fewer children as having clinicallysignificant cases of disorder, they also identified

different children.Impaired Functioning. Another measure of con-

current validity is how well a diagnostic test iden-

tifies children who are showing significant impair-

ment in important areas of social functioning, athome or at school. As an indicator of current socialfunctioning, data were available from the parents’responses to the social competence section of the

CBCL. This asks parents to rate children’s school

performance, involvement in hobbies and sports,and how they get on with friends and family, com-

pared with other children of the same age. Themean score and standard deviation for the samplewere very close to those for the normative sample.9By computing the odds ratio, or relative proportion

of children among “cases” and “noncases,” we could

test the extent to which the pediatricians, and theother diagnostic measures used, identified impaired

children. Impairment was defined for this purposeas a social competence score below the mean for

the 1,300 children in the normative sample whohad been referred for mental health services during

the previous year.9 The pediatricians identified only10% of these 116 children as disturbed compared

with 68% for the interviewers, 50% for the CBCL

and 29% for the DISC-P. However the odds ratioswere not different: 3.9:1 (pediatricians), 3.2:1 (in-

terviewers), and 3.8:1 (CBCL), and 4.0:1 (DISC-P).Thus, the probability of a child’s showing consid-

erable social impairment was three to four timeshigher in the presence of a psychiatric diagnosis

than in its absence. This measure of concurrentvalidity did not, however, distinguish among the

TABLE 7. Chance-Corrected Agreement on Diagnosis Between Two Measures of Psy-chopathology*

Interviewer Child Behavior

ChecklistDiagnostic Interview

Schedule forChildren-Parent

Pediatrician .06 .13” .15a

Clinical interviewer .40c .29cChild Behavior Checklist 37C

* Results are ic values. Significa nce: a p < .05; bp < .01; C JJ < .001.



ARTICLES 423

diagnostic methods, because the number of “false-

positive findings” identified by each measure in-

creased in parallel with the number of “true-posi-

tive findings.”In summary, three ways of comparing the con-

current validity of the structured psychiatric inter-view with the pediatricians’ clinical judgment have

been reviewed, in an effort to test the hypothesisthat the differences found could be ascribed to

overdiagnosis by the DISC rather than to underdi-

agnosis by the pediatricians. The results show that

the prevalence estimates of the other measures used

(clinical interviewer judgment and parent question-naire) were closer to the DISC-P than to the pedia-

tricians, that concordance (agreement on the pres-ence or absence of a disorder) was higher with theDISC than with the pediatricians, and that thepediatricians identified a disorder in 10% of thechildren with marked impairment in social func-tioning, compared with 29% in the case of theDISC. However, the DISC also diagnosed as dis-

turbed more children with a higher level of socialfunctioning, so that the odds ratios were similar.

None of these is a conclusive argument: but as

discussed earlier, more accurate methods, includingoutcome studies and the identification of valid bi-

ologic markers, are not yet available. It is for thereader to decide; in doing so, it might be useful to

bear in mind that previous community-based sur-veys have generally found rates of child psycho-

pathology in the 12% to 20% range, whereas studiesof pediatric practice show a median diagnostic rateof 4% to 7%#{149}4Thus, the weight of the evidence to

date is in the direction of underdiagnosis by pedia-tricians rather than overdiagnosis by the DISC.

DISCUSSION

The findings presented in this paper supportHaggerty et a!’ in their observation that mentalhealth problems are present in a significant number

of children visiting their primary care pediatrician.The results also suggest that the actual rate ofdiagnosis of psychiatric disorders is low, justifying

the use of the Skuse and Williams’ term “hiddenmorbidity” � to describe the rate of unidentifiedrelative to identified cases.

Before discussing the implications of these find-ings, it is important to emphasize aspects of the

study that limit its generalizability. First, the chil-

dren in the sample were all 7 to 11 years of age.There is evidence that the rate of pediatrician di-agnosis of psychiatric disorders increases in older

children and adolescents.23 Further studies withdifferent age groups are needed to clarify this issue.Second, the study was carried out in an HMO. Thiswas a deliberate decision, made to avoid the addi-

tional problems of calculating the effect of cost ofpsychiatric services on the diagnostic process for

each case individually. It is possible that pediatri-

cians working within an HMO use different criteria

for making a diagnosis from those working in fee-

for-service settings. It should, however, be noted

that previous studies show no difference in the

diagnostic rates of pediatricians in the two types of

setting.4 Third, the criterion for identifying child

psychopathology in this study was a diagnosis based

on information from the parent (DISC-P) ratherthan from the child, teacher, or other possible

sources, or from a combination of sources. In apsychiatric setting, it is usual for diagnoses to bebased on information from multiple informants;

therefore, it could be argued that this should be thecriterion used. We have chosen to use parental data

only in these analyses because parents are the main

source of information for pediatricians-it is theparent who decides to bring the child to the clinicand who decides whether to mention that the child

is having problems at home or at school. It should,

however, be noted that combining the responses ofboth children (DISC-C) and parents (DISC-P), and

using as criterion one or more diagnoses of an

emotional or behavioral disorder based on eitherinterview, results in a prevalence rate of 22.0%

(95% confidence interval 18.6% to 25.4%). This isalmost four times the rate identified by the pedia-

tricians (Table 1). Specificity using both interviews

as criterion increased to 95%, but sensitivity de-

creased to 13%. Thus, the pediatricians missed 87%of the diagnoses made on the basis of both inter-views, compared with 83% when only one interview

was used.Current psychiatric and epidemiologic research

is making rapid advances in the development of

methods for identifying children who are at high

risk for emotional and behavioral problems24 and

these are being adapted to fit the special needs ofthe primary care pediatrician.25 Among these meth-

ods are checklists and questionnaires designed tohelp both parent and pediatrician to identify prob-lems that need further evaluation.25 The CBCL has

been used experimentally in primary care settings26but probably provides more information than canbe rapidly assimilated in a brief office visit. Its usein conjunction with well-child examinations, per-

haps once every 2 years, would, however, exploit itsvalue as a normative, age-standardized measure andmerits consideration. At the same time, a widerange of specific treatment strategies for child psy-

chiatric disorders is being developed, some of which

could be used in the mental health departments ofHMOs or by appropriately trained pediatricians.’2

How much do childhood mental health problemsreally matter? Further research into the long-term



424 PSYCHOPATHOLOGY

implications of childhood psychopathology is ur-

gently needed, but it is clear that, although not all

disturbed children become disturbed adults, many

disturbed adults have shown significant signs of

mental illness as children.27 It is not yet known

how effectively early intervention can break thispattern. Primary care pediatricians, because they

see a large percentage of all children, and see them

repeatedly, are ideally placed to explore the full

value and limitations of early intervention. Theresults of the study reported here suggest, however,

that, despite the rapid growth in behavioral pedi-

atrics as a significant component of training, thereis still some way to go before the magnitude of the

new morbidity is fully recognized.

ACKNOWLEDGMENT

This research was supported by contract 278-83-

0006(DB) and Clinical Research Center grant MH30915from the National Institute of Mental Health.

REFERENCES

1. Haggerty RI, Roghmann KJ, Pless lB (eds): Child Healthand the Community. New York, John Wiley & Sons, 1975

2. Green M: Coming of age in general pediatrics. Pediatrics1983;72:275-282

3. American Academy of Pediatrics, Committee on Psychoso-

cial Aspects of Child and Family Health: Pediatrics and the

psychosocial aspects of child and family health. Pediatrics1982;70:126-127

4. Costello EJ: Primary care pediatrics and child psychopa-

thology: A review of diagnostic, treatment, and referral

practices. Pediatrics 1986;78:1044-1051

5. Skuse D, Williams P: Screening for psychiatric disorder in

general practice. Psycho! Med 1984;14:365-377

6. Regier DA, Goldberg ID, Taube CA: The de facto mental

health services system. Arch Gen Psychiatry 1978;25:685-692

7. Goldberg DP, Huxley P: Mental Illness in the Community.London, Tavistock, 1980

8. Achenbach TM, Edelbrock CS: Manual for the Child Be/iou-ion Checklist and Reuised Child Behauior Profile. Burlington,

VT, Department of Psychiatry, University of Vermont, 19839. Achenbach TM, Edelbrock CS: Behavioral problems and

competencies reported by parents of normal and disturbedchildren aged 4 through 16. Monogr Soc Res Child Deu1981;46:1

10. Verhulst FC, Berden GFMG, Sanders-Woudatra JAR: Men-

tal health in Dutch children: II. The prevalence of psychi-

atric disorder and relationship between measures. Acta Psy-chiatr Scand 1985;324(suppl 324):72

11. Bird HR, Canino G, Gould MS, et al: Use of the Child

Behavior Checklist as a screening instrument for epidemi-ological research in child psychiatry. J Am Aced ChildAdolesc Psychiatry 1987;26:207-213

12. Costello EJ, Pantino T: The new morbidity: Who should

treat it? J Deu Behau Pediatn 1987;8:288-291

13. Costello AJ, Edelbrock C, Kalas R, et al: Diagnostic Inter-uiew Schedule for Children. Bethesda, MD, National Insti-tute of Mental Health, 1982

14. Diagnostic and Statistical Manual of Mental Disorders, ed 3.Washington, DC, American Psychiatric Association, 1980

15. Costello AJ, Edelbrock CS, Dulcan MK, et al: Testing of theNIMH Diagnostic Interuiew Schedule for Children (DISC)in a Clinical Population: Final Report, National Institute ofMental Health contract No. RFP-DB-81-0027. Rockville,MD: Center for Epidemiological Studies, 1984

16. Costello EJ, Edelbrock CS, Costello AJ: Validity of theNIMH Diagnostic Interview Schedule for Children: A com-parison between psychiatric and pediatric referrals. J Ab-norm Child Psychol 1985;13:579-595

17. Costello AJ: Structured interviewing for the assessment of

child psychopathology, in Noshpitz JD (ed): Basic Handbookof Child Psychiatry. New York, Basic Books, 1987, vol 5, pp143-152

18. Anderson JC, Williams 5, McGee R, et al: DSM-III disorders

in preadolescent children. Arch Gen Psychiatry 1987;44:69-80

19. Edelbrock CS, Costello AJ, Dulcan MK, et al: Age differ-ences in the reliability of the psychiatric interview of thechild. Child Deu 1985;56:265-275

19a. Offord DR, Boyle MH, Szatmari P, et al: Ontario ChildHealth Study: II. Six-month prevalence ofdisorder and ratesof service utilization. Arch Gen Psychiatry 1987;44:832-836

20. Gould MS, Wunsch-Hitzig R, Dohrenwend BP: Formulationof hypotheses about prevalence, treatment and prognosticsignificance of psychiatric disorders in children in theUnited States, in Dohrenwend BP, Gould MS, Link B, et al(eds): Mental Illness in the United States: EpidemiologicalEstimates. New York, Praeger, 1980, pp 9-44

21. Links PS: Community surveys of the prevalence of child-

hood psychiatric disorders: A review. Child Dev 1983;54:531-548

22. Anastasi A: Evolving concepts of test validation. Ann ReuPsychol 1986;37:1-15

23. Goldberg ID, Roghmann KJ, Mclnerny TK, et al: Mental

health problems among children seen in pediatric practice:Prevalence and management. Pediatrics 1984;73:278-293

24. Achenbach TM: Assessment and Taxonomy of Child andAdolescent Psychopathology. Beverly Hills, CA, Sage Publi-cations, 1985

25. Jellineck MS, Murphy JM, Burns BJ: Brief psychosocial

screening in outpatient pediatric practice. J Pediatr1986;109:371-378

26. Hankin J, Goodman A, Starfield B: The impact of sharingthe results of a psychological screening on the psychosocialoutcome of the child. Presented at the NIMH ResearchConference on Mental Disorders in General Health CareSettings, June 1987, Seattle, Washington

27. Rutter M: Psychopathology and development, in Rutter M,

Hersov L (eds): Child and Adolescent Psychiatry, ed 2.Oxford, England, Blackwell Scientific Publications, 1985,pp 720-739



1988;82;415-424 PediatricsBurns and David Brent

Elizabeth J. Costello, Craig Edelbrock, Anthony J. Costello, Mina K. Dulcan, Barbara J. Psychopathology in Pediatric Primary Care: The New Hidden Morbidity

& ServicesUpdated Information

http://www.pediatrics.orgincluding high-resolution figures, can be found at:

Citations http://www.pediatrics.org#otherarticles

This article has been cited by 30 HighWire-hosted articles:

Permissions & Licensing

http://www.pediatrics.org/misc/Permissions.shtmlits entirety can be found online at: Information about reproducing this article in parts (figures, tables) or in

Reprints http://www.pediatrics.org/misc/reprints.shtml

Information about ordering reprints can be found online:


http://www.pediatrics.org

http://www.pediatrics.org#otherarticles

http://www.pediatrics.org/misc/Permissions.shtml

http://www.pediatrics.org/misc/reprints.shtml


1988;82;415-424 pediatrics burns and david brent elizabeth j

Documents