CLINICIANS' GUIDE TO RESEARCH METHODS AND STATISTICS

Deputy Editor: Robert J Harmon, M.D.

Sampling and External Validity

GEORGE A. MORGAN, PH.D., AND ROBERT]. HARMON, M.D.I

Sampling is the process of selectingpart of a larger group withthe intent of generalizing from the smaller group, called thesample, to the population, the larger group. If we are to makevalid inferences about the population, we must select the sam­ple so that it is representative of the total population.

With a few notable exceptions, modern survey techniqueshave proven to be quite accurate in predicting or reportinginformation about the attitudes of the American public fromsamples of about 1,000 participants. Historically, however,there have been a number of examples of major miscalcula­tions that can be traced to inadequate sampling techniques.An example is the grossly erroneous prediction, by a LiteraryDigest poll of a very large sample, that Franklin Rooseveltwould lose the 1936 presidential election when, in fact, hewon by a landslide. One of the problems was that the samplewas selected from automobile registrations, telephone direc­tories, and related sources. This led to oversampling ofaffluent individuals who were not representative of the votingpublic, especially during the Great Depression. In addition,only about 20% of the selected sample actually returned theirquestionnaires.

Steps in Selecting a Sample and Generalizing Results

There are many ways to select a sample from a population.The goal is to have an actual sample in which each participantrepresents a known fraction of the theoretical population sothat characteristics of the population can be recreated fromthe sample. Obtaining a representative sample is not easybecause things can go wrong at 3 stages of the research pro­cess. Figure 1 shows the key sampling concepts and the 3 steps(shown with arrows).

Accepted February 26, 1999.Dr. Morgan is Proftssor of Education and Human Development, ColoraM

State University, Fort Collim. Dr. Harmon is Proftssor ofPsychiatry and Pedi­atrics and Head, Division ofChild Psychiatry, University of ColoraM School ofMedicine, Denver.

The authors thank Helena Kraemerfor helpftlftedback and Nancy Plummerfor manuscript preparation. Parts of the column are adapted, with permission.from the publisher and the authors, .from Gliner fA and Morgan GA (in press),Research Design and Analysis in Applied Settings: An Integrared Approach.Mahwah, NI Erlbaum. Permission to reprint or adapt any part ofthis columnmust be obtainedfrom Erlbaum.

Reprint requests to Dr. Harmon, CPH Room 2K04, UCHSC Box C268-52,4200 East Ninth Avenue, Denver, CO 80262; e-mail: robert.harmon@Uchsc.edu.

0890-8567/99/3808-I051©1999 by the American Academy of Childand Adolescent Psychiarry.

]. AM. ACAD. CHILD ADOLESC. PSYCHIATRY, 38:8, AUGUST 1999

The first step is from the theoretical population to the acces­sible population. It may be that the accessible population isnot representative of the theoretical population. This is a com­mon problem because researchers often do not have access tothe geographical or socioeconomic range of participants to

which they would like to make generalizations. Unfortunately,the theoretical population usually is not specified in publishedresearch articles. One has to infer it from the context and thestated inclusion and exclusion criteria.

The second step in the sampling process is called the samplingdesign or selection of participants. This step, between the acces­sible population and the selected sample, is described in themethod section of an article and is the step over which theresearcher has the most control. In some cases, the accessiblepopulation is small enough that everyone is asked to participate.

The third step takes place between the selected sample andthe actual sample. The problem is that participants may notconsent to participate (i.e., there is a low response rate), so theactual sample may be considerably smaller than the selectedsample and may be quite unrepresentative of the selectedsample. This is often a problem with mailed surveys, especiallyif the survey is sent to busy people such as clinicians.

Types of Sampling

There are 2 major types of sampling designs that are used inobtaining the selected sample: probability and nonprobability.

Probability Sampling. In probability sampling, every partic­ipant has a known, nonzero chance of being selected. The par­ticipants or elements of the population are usually people, butcould be groups, animals, or events. With probability samples,researchers are able to make an estimate of the extent to whichresults based on the sample are likely to differ from whatwould have been found by studying the entire population.There are 4 main types of probability sampling. The mostbasic is the simple random sample, which occurs when all par­ticipants have an equal and independent chance of beingincluded in the sample. This technique can be implementedusing a random number table.

A similar frequently used technique is systematic samplingwith a random start. For example, we might randomly selectthe 4th person on a list as the first participant. If we wanted tosample 10% of the accessible population, we would then sys­tematically select every 10th participant, starting with the 4th.With simple random and systematic sampling, the population

1051

MORGAN AND HARMON

SamplingDesign

Target or Theoretical Population ill(e.g., all ADHD children in the USA) Selection

11st StepJ, N =100

~Selected Sample

N =500 1-(e.g. ADHD children

selected to be2nd Step in the study)

Accessible Population or SamplingFrame (e.g., ADHD children seen at 3rd Steone clinic)

N =75Actual Sample

(e.g., ADHD childrensampled who

agreed to participate)

p

Fig. 1 Schematic diagram of the sampling process. ADHD = attention-deficit/hyperactivity disordet.

must be finite and there must be a complete list of potentialparticipants. If the sample was large, the list complete, and itassured equal probability of selection, these 2 techniques willproduce a representative sample.

If some important characteristics of the accessible populationsuch as gender or race are known ahead of time, we can reducethe sampling variation and increase the likelihood that the sam­ple will be representative of the population by using stratifiedrandom sampling. When participants are geographically spread,it is common to stratify on the basis of geography so that appro­priate proportions come from the different regions.

Cluster sampling is a 2-stage sampling procedure that isespecially useful when the population is spread out geograph­ically and the researcher needs to collect data on site. Theusual strategy is to first select a number of clusters/sites (e.g.,clinics) randomly and then select all potential participantsfrom these selected clusters.

With a probability sample, the descriptive statistics fromthe sample also describe the population. However, with strati­fied or cluster sampling, one would need to weight the obser­vations appropriately to describe the population.

Nonprobability Sampling. Nonprobability samples are onesin which the probability of being selected is unknown. Timeand cost constraints lead researchers to use nonprobabilitysamples. There are several types of nonprobability samples,including quota, purposive, and convenience.

1052

Because the theoretical population is often infinite andprobability sampling is costly, convenience sampling is probablythe most common of all sampling methods. One has a con­venience sample both when one selected the accessible popu­lation by convenience, even if assessing all the persons in thatpopulation, and also if one selects some participants from theaccessible population based on convenience. Researchers latermay examine the demographic characteristics of their conven­ience sample and conclude that the participants are similar tothose in the larger population. This does not mean that thesample is, in fact, representative, but it does indicate anartempt by the researcher to check on representativeness. Anextended discussion of the types of sampling and the advan­tages and disadvantages of each can be found in Fowler(1993).

Why Are Nonprobability Samples Used So Frequently?

In addition to cost and time advantages, researchers,especially those using experimental designs, are not primarilyinterested in making inferences about the population fromthe descriptive data. These researchers are more interested inwhether the treatment has an effect on the dependent vari­able, and they assume that if the treatment is powerful, theeffect will show up in many kinds of participants. Manyresearchers seem to imply that external population validity isless important than internal validity.

]. AM. ACAD. CHILD ADOLESC. PSYCHIATRY, 38:8, AUGUST 1999

How Many Participants?

The question, "How many participants do I need for thisstudy?" is asked often. One part of the answer depends onwhom you ask and what their discipline is (Kraemer andThiemann, 1987). National opinion surveys almost alwayshave 1,000 or more participants, whereas sociological andepidemiological studies usually have at least several hundredparticipants. On the other hand, clinical trials with 10 to 20participants per group are not uncommon, and in some areas,single-subject designs are used. To some extent these dramaticdifferences in minimum sample sizes depend on differences intypes of designs, measures, and statistical analyses, but theyalso seem to be based in good part on custom.

The size of the sample should be large enough so one doesnot fail to detect important findings, but a large sample willnot necessarily help one distinguish between the merely statis­tically significant and the societally important findings.Statistical power analysis, discussed in a later article, can helpone can compute the sample size needed to find a statisticallysignificant result given certain assumptions (e.g., see Kraemerand Thiemann, 1987).

RESEARCH METHODS AND STATISTICS

Sampling and the Internal and External Validity of a Study

In an earlier article, we discussed internal and externalvalidity. We noted that external validity is influenced by therepresentativeness of the sample and that random selection ofthe sample is desirable. However, the internal validity of astudy is not directly affected by the type of sampling. Forexample, a randomized experiment may have a small conven­ience sample and still have high internal validity, in part,because of random assignment of participants to groups.Thus, although random selection and random assignment havedifferent effects on internal and external validity, both areimportant in evaluating the quality of a research study.

REFERENCES

Fowler FJ Jr (1993), Survey Research Methods, 2nd ed. Newbury Park, CA:Sage

Kraemer HC, Thiemann S (1987), How Many Subjects? Statistical PowerAnalysis in Research. Newbury Park, CA: Sage

The next article in this series appears in the October 1999 issue:

Measurement and Descriptive StatisticsGeorge A. Morgan, Jeffrey A. Gliner, RobertJ Harmon

Acute Psychosocial Impact of Pediatric Orthopedic Trauma With and Without Accompanying Brain Injuries. Terry Stancin,PhD, H. Gerry Taylor, PhD, George H. Thompson, MD, Shari Wade, PhD, Dennis Drotar, PhD, Keith Owen Yeates, PhD

Background: The acute psychosocial effects of orthopedic injuries on children and their families are poorly understood. Previousstudies have relied on retrospective reports or failed to take into account accompanying brain injuries. The purpose of the presentstudy was to examine prospectively the psychosocial impact of pediatric orthopedic traumatic fractures with and without accompa­nying brain injuries. Methods: Participants were 108 children 6 to 12 years old with orthopedic injuries requiring hospitalization:group 1 (n =80) had fractures only, group 2 (n =28) also had moderate or severe brain injuries. Using standardized measures andparent interviews, we obtained preinjury estimates of family functioning and child behavior problems and postinjury measures ofparental distress, family stresses, and child behavior. Results: Parents reported significant clinical distress (35% in group 1, 57% ingroup 2), family burdens (group 2 > group 1), and child behavioral changes (41 % in group 1,89% in group 2). Multiple regressionanalyses indicated that preinjury family status and brain injuries predicted postinjury parental and family distress. Conclusion:Pediatric orthopedic injuries have greater social effects on children with accompanying brain injuries and poorer preinjury familyfunctioning. J Trauma 45: 1031-1038.

J. AM. ACAD. CHILD ADOLESC. PSYCHIATRY, 38:8, AUGUST 1999 1053