Vot 71, No.2, pp. 149-164.@2005 Councilfor Exceptional Children.

Quality Indicators for GroupExperimental andQuasi-Experimental Researchin Special Education

RUSSELL GERSTEN

Instructional Research Group

LYNN S. FUCHS

DONALD COMPTON

Vanderbilt University

MICHAEL COYNE

University of Connecticut

CHARLES GREENWOOD

University of Kansas

MARK S. INNOCENTI

Utah State University

ABSTRACT: This article presents quality indicators for experimental and quasi-experimental studies

for special education. These indicators are intended not only to evaluate the merits of a completedresearch report or article but also to serve as an organizer of critical issues for consideration in re-search. We believe these indicators can be used widely, from assisting in the development of research

plans to evaluating proposals. In this article, the framework and rationale is explained by provid-ing brief descriptions of each indicator. Finally, we suggest a standard for determining whether apractice may be considered evidence-based. It is our intent that this standard for evidenced-basedpractice and the indicators be reviewed, revised as needed, and adopted by thefield of special edu-cation.T his article presents a set of quality such a set of indicators, given the current focus

indicators for experimental and on the need for an increase in rigorous, scientific

quasi-experimental studies for research in education. Recently, the National Re-

the field of special education. We search Council (NRC, 2002), in a report on sci-

believe there is a critical need for entific research in education, noted that they saw

Exceptional Chihdrnn 149

no reason why education could not be subject tothe same scientific methods as other disciplinessuch as chemistry or physics. They further arguedthat there is a place for all research methodologiesin educational research: survey research, qualita-tive research, and correlational research. As Feuer,Towne, and Shavelson (2002) noted, "If a re-search conjecture or hypothesis can withstandscrutiny by multiple methods, its credibility is en-hanced greatly. Overzealous adherence to the useof any given research design flies in the face ofthis fundamental principle" (p. 8).

Yet the NRC report is unequivocal in stat-ing numerous times that experiments using ran-domized trials are currently underutilized ineducational research, despite being "the singlebest methodological route to ferreting out system-atic relations between actions and outcome"(Feuer et al., 2002, p. 8). In that sense, they reit-erate a point made over a generation ago byCampbell and Stanley (1966), who highlightedthat controlled experimental research conductedin field settings was

the.. .only means for settling disputes regardingeducational practice, as the only way of verifyingeducational improvements, and as the only wayof establishing a cumulative tradition in whichimprovement can be introduced without thedanger of a faddish discard of old wisdom infavor of inferior novelties. (p. 2)

Until recently, there was no specific set ofstandards or quality indicators for evaluating thequality of either proposed or completed experi-mental or quasi-experimental intervention re-search. Few of the numerous educational researchtextbooks provide a clear list of criteria that fit therealities of contemporary field research, althoughthey present excellent ideas (e.g., Boruch, 1997;Gall, Borg, & Gall, 2002; Shadish, Cook, &Campbell, 2002). Fortuitously, slightly before webegan this endeavor, the What Works Clearing-house of the U.S. Department of Education re-leased the Study Design and ImplementationAssessment Device (DIAD; Valentine & Cooper,2003). Its major goal is to evaluate whether a re-search article or research report can be consideredsufficiendy valid and reliable to be entered into aresearch synthesis on the effectiveness of an inter-vention or approach.

Our goal is a bit broader. We intend thesequality indicators to be used not only to evaluatethe merits of a completed research report or arti-cle, but also to evaluate research proposals, disser-tation proposals, and grant applicationssubmitted to funding agencies. We also intendthis to be useful to researchers as they thinkthrough the design of a study, to serve as a check-list or organizer of critical issues for consideration.

We intentionally chose to look at qualityindicators, rather than a set of "yes/no" standards,because evaluating a research design always entailsweighing the relative strengths and weaknesses ina set of domains. Our goal is not to provide abasic primer on how to design a high-quality fieldresearch study. There are numerous introductory(e.g., Gall et al., 2002) and advanced texts (e.g.,Shadish et al., 2002) widely available to explainthe many concepts underlying the design of ex-perimental studies. Our hope is that our set of in-dicators will be field-tested and refined, and thenconsidered useful by journal editors and reviewersof federal grant proposals. We also envision thisset of indicators assisting researchers as they de-sign studies and guiding practitioners as they con-sider alternative educational practices foradoption in their classrooms and schools.

We intend these quality indicators to beused not only to evaluate the merits ofacompleted research report or article, butalso to evaluate research proposals, disser-tation proposals, and grant applicationssubmitted to funding agencies.

Quality indicators for group experimentaland quasi-experimental research proposals in spe-cial education are presented in Table 1. Table 2presents a set of quality indicators for evaluatingthe quality of completed experimental or quasi-experimental studies. These tables address similarissues, but serve slightly different purposes. Bothtables make a distinction between indicators thatare considered essential for quality versus indica-tors that are considered to be desirable to have in astudy.

Winter 2005150

TABLE I

Essential and Desirable Quality Indicators for Group Experimental and Quasi-Experimental Research Proposals

Essential Quality IndicatorsConceptualization Underlying the Study

1. Is a compelling case for the importance of the research made? Is the conceptualization based on well-designedstudies and does it reflect the scope of extant knowledge?

2. If an innovative approach is proposed, is it based on a sound conceptualization formed from sound research?3. Are the research questions appropriate and stated clearly for the purposes of this study? Are valid arguments

supporting the nature of intervention in the comparison group(s) presented?

Participants/Sampling

1. Will appropriate procedures be used to ensure that participants are comparable across intervention conditionson relevant characteristics? If random assignment is used, will information about participants prior to the inter-vention be made available to ensure that samples are comparable on salient characteristics?

2. Will sufficient information be provided to determine (or confirm) whether the participants demonstrated thedisability(ies) or learning/social learning difficulties presented?

3. Will appropriate procedures be used to increase the probability that teachers or interventionists will be compa-rable across conditions?

Implementation of the Intervention and the Nature of Comparison Condition(s)

1. Is the intervention clearly described?2. Are procedures for ensuring and assessing fidelity of implementation described?3. Is the nature of instruction/services provided in comparison conditions described?Outcome Measures

1. Will multiple measures be used to provide an appropriate balance between measures closely aligned with theintervention and measures of generalized performance?

2. Is evidence of reliability for the outcome measures provided? If not, will it be calculated?Quality Indicators for Data Analysis

1. Are the data analysis techniques appropriate and linked to key research questions and hypotheses?2. Is the variability within each sample accounted for either by sampling techniques (e.g., restricting range) or sta-

tistical techniques (blocking, analysis of covariance, growth curve analysis)?3. Is a power analysis provided to describe the adequacy of the minimum cell size?

Desirable Quality Indicators

1. Are data collectors and/or scorers blind to study conditions and equally (un)familiar to examinees across studyconditions?

2. Does the study provide not only internal consistency reliability but also test-retest reliability and interrater reli-ability (when appropriate) for outcome measures?

3. Are outcomes for capturing the intervention's effect measured beyond an immediate posttest?4. Is evidence of the validity of the measures provided? If not, will it be estimated based on data from the pro-

posed study or with data collected from other samples?5. "Will the research team assess more than surface features of fidelity implementation (e.g., number of minutes al-

located to the intervention or teacher/interventionist following procedures specified)? Additionally, will the re-search team examine the quality of implementation?

6. Will the research include actual audio or videotape excerpts that capture the nature of the intervention?7. Does the researcher conduct power analyses proper for varying levels of statistical analysis? For example, if data

will be analyzed at a classroom or day care center level, are analyses at that level sensitive enough to detect ef-fects?

Exceptional Children I151

TABLE 2

Essential and Desirable Quality Indicators for Group Experimental and Quasi-Experimental Research Articlesand Reports

Essential Quality Indicators

Quality Indicators for Describing Participants

1. Was sufficient information provided to determine/confirm whether the participants demonstrated the disabil-ity(ies) or difficulties presented?

2. Were appropriate procedures used to increase the likelihood that relevant characteristics of participants in thesample were comparable across condirions?

3. Was sufficient information given characterizing the interventionists or teachers provided? Did it indicatewhether they were comparable across cbnditions?

Quality Indicators for Implementation of the Intervention and Description of Comparison Conditions

1. Was the intervention clearly described and specified?

2. Was the fidelity of implementation described and assessed?

3. Was the nature of services provided in comparison conditions described?

Quality Indicators fir Outcome Measures

1. Were multiple measures used to provide an appropriate balance between measures closely aligned with the in-terventiona and measures of generalized performance?

2. Were outcomes for capturing the intervention's effect measured at the appropriate times?

Quality Indicators for Data Analysis

1. Were the data analysis techniques appropriately linked to key research questions and hypotheses? Were they ap-propriately linked to the unit of analysis in the study?

2. Did the research report include not only inferential statistics but also effectsize calculations?

Desirable Quality Indicators

1. Was data available on attrition rates among intervention samples? Was severe overall attrition documented? Ifso, is attrition comparable across samples? Is overail attrition less than 30%?

2. Did the study provide not only internal consistency reiiability but also test-retest reliability and interrater relia-bility (when appropriate) for outcome measures? Were data collectors and/or scorers blind to study conditionsand equally (un)familiar to examinees across study conditions?

3. Were outcomes for capturing the intervention's effect measured beyond an immediate posttest?

4. Was evidence of the criterion-related validity and construct validity of the measures provided?

5. Did the research team assess not only surface features of fidelity implementation (e.g., number of minutes allo-cated to the intervention or reacher/interventionist following procedures specified), but also examine quality ofimplementation?

6. Was any documentation of the nature of instruction or series provided in comparison conditions?

7. Did the research report include actual audio or videotape excerpts that capture the nature of the intervention?

8. Were results presented in a clear, coherent fashion?

aA study would be acceptable if it included only measures of generalized performance. It would not be acceptable if it only in-cluded measures that are tightly aligned.

We suggest that these indicators be used to Indicators and demonstrate at least one of the

define "acceptable" and "high" quality research quality indicators listed as Desirable as shown in

proposals and studies. To be considered accept- Tables 1 and 2. To be considered high quality, a

able quality, a research proposal or study would proposal or study would need to meet all but one

need to meet all but one of the Essential Quality of the Essential Quality Indicators and demon-

Winter 2005152

strate at least four of the quality indicators listedas Desirable. These definitions of acceptable andhigh quality are tentative and should be field-tested by universities, agencies that review grantapplications, and research organizations.

In the following section, we walk the readerthrough the framework by providing brief de-scriptions of the indicators. Finally, we suggest astandard for determining whether a specific prac-tice may be considered evidence based.

QUALITY INDICATORS FOR

GROUP EXPERIMENTAL AND

QUASI-EXPERIMENTAL

RESEARCH

CONCEPTUALIZATION OF THE

RESEARCH STUDY

Because a research !-udy without context willhave no impact, it is critical that the researcherclearly present the conceptualization and designof a study. In this section, we describe four indica-tors for providing context for a research study.

The conceptualization of the research study isbased on the findings of rigorously designedstudies that reflect the current scope of extantknowledge, including the findings of seminalstudies. OR If an innovative approach is pro-posed, it is based on sound conceptualizationand is rooted in sound research.

The literature review is a key section for theconceptual design of a study. Typically, it de-scribes prior research that leads to current concep-tualizations and study design. It is important thatthe review presents the existing information andmakes the case for the proposed research. The re-view of literature should reflect both recent andseminal research studies in the area, making thereader aware of how these studies relate to theproposed research study. If there is no recent re-search, the researcher should state this clearly.

Researchers should ensure that the reviewof literature not only is adequate in the breadth ofstudies covered, but also is focused on the smallset of issues that are critical for this particularstudy. The end result is that the researcher shouldpresent a concise but complete summary of thescientific knowledge base for the area in which

the problem exists, should illuminate areas ofconsensus and areas that require further investiga-tion, and create a theoretical and conceptual un-derstanding for why the study addresses animportant topic that has not been fully addressedin the past. Whether the researcher proposes aninnovative approach for which little existing em-pirical evidence exists, or interventions that aresubtle variants on evidence-based practices, he orshe should focus the review of literature on pro-viding a compelling argument for the approach,and setting the stage for the guiding researchquestions.

A compelling case for the importance of the re-search is made.

Feuer and colleagues (2002) make a strongcase for an increased emphasis on presenting ar-guments regarding the importance of any researchstudy when writing or discussing research. Part oftheir argument focuses on the increased salienceof educational research in the current political cli-mate. The question of how researchers choose thetopics on which they focus their research is im-portant, and the rationale for these choices mustbe conveyed to consumers or funders of researchprojects.

Fabes, Matrin, Hanish, and Updegraff(2000) recently discussed the issue of evaluatingthe significance of developmental research for the21st century. These authors identified four newtypes of "validity" which, with some adaptation,can be applied to educational research. Thesetypes of validity are incidence validity, impact va-lidity, sympathetic validity, and salience validity.

Incidence validity refers to the degree towhich a particular piece of research addresses atopic that significantly affects large numbers ofpeople. Impact validity is the degree to which a re-search topic is perceived to have serious and en-during consequences. Sympathetic validity reflectsthe tendency to judge the significance of the re-search based on the degree to which it generatesfeelings of sympathy for individuals affected bythe problem under study. Salience validity reflectsthe degree to which people, generally referring tothe public, are aware of the problem or topic.Fabes and colleagues (2000) note that it is diffi-cult for one study to incorporate all four types ofvalidity, but that it is important for researchers to

Exceptional Childre-1 153

be aware of these types of validity and to look atthe pattern of validity addressed by the study inquestion. These seem helpful concepts to use inconsidering this quality indicator.

Because a research study without contextwill have no impact, it is critical that theresearcher clearly present the conceptual-

iization and design of a study.

Valid arguments supporting the proposed in-tervention as well as the nature of the compar-ison group are presented.

It is important to characterize the interven-tion and place it within a context that makessense to the reader. The rationale for the proposedintervention should be sound and based, in part,on facts and concepts presented in the review ofthe literature. If the intervention is applied to newparticipants, settings, or context, there should beclear links based on argument and/or research forthis new use.

In addition to describing the intervention,it is important to describe the nature of the expe-rience or services to be received by the compari-son group (Feuer et al., 2002; Gersten, Baker, &Lloyd, 2000). Potentially, a variety of reasons mayexist for establishing comparison groups. Themost common is describing an innovative ap-proach to a traditional one. In this case, readersneed to know the nature of experiences of partici-pants in the more traditional comparison group.At times, researchers contrast theoretically op-posed or divergent interventions. This, too, is le-gitimate, especially when the contrast is focusedand of high interest. There are occasions when aresearcher is broaching a new topic and simplywants to know if it leads to change. These wouldbe the only instances where a no-treatment com-parison group would be appropriate. Designs thatare more elegant include multiple groups, someinvolving subtle contrasts (e.g., taught by an in-terventionist or teacher vs. a paraprofessional;identical intervention with or without progressmonitoring).

The research questions are appropriate for thepurpose of the study and are stated clearly.

In most research proposals, a succinct state-ment of the key questions addressed is critical.These questions should be linked in an integralfashion to the purpose statement. The criteria forthe research questions are simple and serve to pullthe conceptualization and design of the study to-gether. Hypothesized results should be specified,but the authors of a proposal should be candidabout issues for which they have no specific pre-diction and reviewers need to be aware that it isequally appropriate to frame clear research ques-tions without a specified hypothesis.

DESCRIBING PARTICIPANTS

Sufficient information to determine/confirmwhether the participants demonstrated the dis-ability(ies) or difficulties addressed is pre-sented.

A fundamental issue for generalizing studyresults is whether the participants actually experi-enced the disability(ies) or the difficulties thestudy was intended to address. To resolve thisissue, researchers must move beyond school dis-trict-provided labels. Informing readers that stu-dents met the state criteria for identifyingstudents as developmentally delayed provides lit-tle information on the type or extent of disabilitya student may have. Researchers need to provide adefinition of the relevant disability(ies) or difficul-ties and then include assessment results docu-menting that the individuals included in thestudy met the requirements of the definition. Forexample, if the study addressed students withmath difficulty, the researchers would (a) definemath difficulty (e.g., performing below the 16thpercentile on a test of mathematics computationand performing below the 16th percentile on atest of mathematics concepts) and (b) documentthat each participant included in the study metthese criteria. We recommend that researchers at-tempt to link their definitions to those in the cur-rent literature, which can be done by eitherreplicating the definition used in prior publishedresearch or by explaining the reason for extendingor redefining the operational definition, and pro-vide a new label for this alternative definition (to

Winter 2005154

increase specificity of terms in research).Researchers should provide enough infor-

mation about participants so that readers canidentify the population of participants to whichresults may be generalized. This additional infor-mation may include, but is not limited to, comor-bidity disability status (e.g., what percentage ofstudents with reading disabilities also had mathdisability and/or attention deficit/hyperactivitydisorder?); demographics (e.g., age, race, sex, sub-sidized lunch status; English language learner sta-tus, special education status); scores-relatedacademic assessments (with effect sizes also desir-able); and percentage of students receiving subsi-dized lunch for participating schools.

As part of describing participants, groupdifference on salient variables must also be pre-sented. Given the issues previously discussed onparticipant selection, comparability of groups onkey demographic variables must be examined,both to describe the participants as well as to usein later analyses. It is also the researchers' respon-sibility to document sample comparability atpretest on at least one outcome measure (or keypredictor of outcome). Demonstrating such com-parability eliminates the possibility that study ef-fects accrued because of preexisting differencesbetween the study groups on key performance ordemographic variables.

Appropriate procedures are used to increase theprobability that participants were comparableacross conditions.

The optimal method for assigning partici-pants to study conditions is through random as-signment, although in some situations, this isimpossible. It is then the researchers' responsibil-ity to describe how participants were assigned tostudy conditions (convenience, similar class-rooms, preschool programs in comparable dis-tricts, etc.). Researchers are urged, however, toattempt random assignment. In our experience,some type of random assignment can often be ne-gotiated with districts or school or clinic person-nel. The quality of the research design isinvariably higher with random assignment ofboth student participants and interventionproviders.

However, there will be situations whererandom assignment of students is not feasible. In

these cases, random assignment of teachers or in-terventionists can be an excellent alternative, ifused appropriately. Random assignment of class-rooms, or even schools to treatment conditions isalso a viable option. These are all legitimate de-sign strategies for randomized trials if appropriatestatistical analyses are used and researchers areaware of issues that can affect statistical power.For example, statistical analyses need to accountfor the nesting of students in classrooms. There-fore, power analyses may need to be conducted atboth the student and the classroom level. One ad-vantage of this type of design is that it is often notnecessary to assess each student in a class. A ran-domly selected subsample is often appropriate.Another advantage is that, in some cases, power isincreased because the standard deviation betweenclasses is often much lower than the standard de-viation between all the students. We recommendthat researchers work closely with a skilled statisti-cian on these issues.

It is important to note that random assign-ment of participants to study conditions does notguarantee equivalent study groups on key charac-teristics of the participants. In fact, given the het-erogeneity of many disability subgroups, it can bedifficult to achieve initial comparability on keyvariables, with or without random assignment.For this reason, matching participants on a salientvariable(s) and randomly assigning one memberof each pair to a treatment condition, or a strati-fied assignment procedure to study conditions isoften preferred.

Researchers need to provide a definitionof the relevant disability(ies) and then in-clude assessment results documenting thatthe individuals included in the study metthe requirements of the definitions.

Diferential attrition among intervention groups orsevere overall attrition is documented.

Researchers should document overall attri-tion of participants and ensure that attrition ratesbetween the intervention and comparison groupswere not substantially different. The reason for

Exception Children l55

this is to document that study groups remaincomparable at the conclusion of the study.

Sufficient information describing importantcharacteristics of the intervention providers issupplied, and appropriate procedures to in-crease the probability that interventionproviders were comparable across conditionsare used.

Researchers should supply enough informa-tion about the intervention providers (i.e., teach-ers or other individuals responsible forimplementation) so that readers understand thetype of individual who may be capable of admin-istering the intervention when it is used outsidethe context of that study (i.e., the interventionproviders to which results may be generalized).Relevant characteristics may include age, sex, race,educational background, prior experience with re-lated interventions, professional experience, andnumber of children with and without disabilitiesin the family (for parents). It may also be usefulto present information on assessments involvingknowledge of the topic to be taught, knowledgeof appropriate pedagogical methods, efficacy mea-sures, and/or attitudinal measures.

The researchers must describe how inter-vention providers were assigned to the variousstudy conditions. Random assignment is the pre-ferred method, but other approaches often arenecessary because of the circumstances in whichthe study is conducted. For example, researchersshould also consider situations where interventionproviders are counterbalanced across conditions.This can be done by having each interventionistteach one group with one method and anothergroup with another method or by switching inter-ventionists across conditions at the midpoint. Theburden is on the researchers to describe assign-ment methods and, when random assignment orcounterbalancing is not employed, to provide arationale for how those methods support the in-tegrity of the study.

Comparability of interventionists acrossconditions should be documented. Regardless ofwhether random assignment was employed, re-searchers must quantify relevant background vari-ables for the intervention providers associatedwith each of the study conditions. This is espe-cially true if the interventionists are not counter-

balanced across conditions. With such informa-tion, readers can be assured that the effects of theintervention are not due to preexisting differencesbetween the intervention providers (i.e., the fa-vorable effects for an intervention are in fact at-tributable to the intervention).

IMPLEMENTATION OF THE INTERVENTION

AND DESCRIPTION OF NATURE OF

SERVICES IN COMPARISON CONDITIONS

The intervention is clearly described and spec-ified.

Researchers should provide a precise de-scription of the independent variable to allow forsystematic replication as well as to facilitate cod-ing in research syntheses such as meta-analysis(Gersten, Baker, & Lloyd, 2000). Unfortunately,the instructional labels that are often assigned tointerventions are vague or misleading and mayvary considerably from study to study. For exam-ple, in a meta-analysis on the impact of variousinstructional approaches on students with learn-ing disabilities, Swanson and Hoskyn (1998)found that there was significant overlap in theway constructs with similar labels were opera-tionalized. They concluded that classifying typesof teaching in broad terms such as direct instruc-tion and strategy instruction was problematic andthat more fine-grained terminology and descrip-tions needed to be used.

Interventions should be clearly describedon a number of salient dimensions, includingconceptual underpinnings; detailed instructionalprocedures; teacher actions and language (e.g.,modeling, corrective feedback); use of instruc-tional materials (e.g., task difficulty, example se-lection); and student behaviors (e.g., whatstudents are required to do and say). Precision inoperationalizing the independent variable has be-come increasingly important as replication studiesbecome more refined and synthesis proceduressuch as meta-analysis become more common. Infact, the purpose of research syntheses is to "dis-cover the consistencies and account for the vari-ability in similar-appearing studies" (Cooper &Hedges, 1994, p. 4). When analyzing similaritiesand differences among instructional interventionsacross multiple studies, or when trying to deter-mine the effect that subtle changes in instruction

Winer 2005156

might have on learning outcomes, precise descrip-tions of instruction are critical.

Fidelity of implementation is described andassessed in terms of surface (the expected inter-vention is implemented) and quality (howwell the intervention is implemented)features.

Fidelity of implementation (also known astreatment fidelity or treatment integrity) refers tothe extent to which an intervention is imple-mented as intended (Gresham, MacMillan,Beebe-Frankenberger, & Bocian, 2000). Informa-tion about treatment fidelity is essential in under-standing the relationship between an intervention(i.e., independent variable) and outcome mea-sures (i.e., dependent variables). The goal of ex-perimental research in special education is todemonstrate that any changes in a dependentvariable are the direct result of implementing aspecified intervention. Without evidence aboutwhether the intervention was actually imple-mented as planned, however, it is impossible toestablish this relationship unequivocally.

This indicator is concerned with bothwhether treatment fidelity was measured and howit was measured. Although necessary, it is not suf-ficient to only note the number of days/sessionsthat intervention was conducted. At the least, re-searchers should observe the intervention using achecklist of treatment components and recordwhether the most central aspects of the interven-tion occurred. A fidelity score can then be derivedby calculating the percentage of treatment com-ponents that were implemented. Observationsshould take place over the entire course of thestudy on a regular basis. The research team shouldinclude some measure of interobserver reliability.Key features to look for in assessing this criterionare: (a) inclusion of key features of the practice(e.g., ensuring students use all five steps in thestrategy, ensuring students are asked to draw a vi-sual representation of a mathematical problem,ensuring teachers provide the specified number ofmodels of a cognitive strategy); (b) adequate timeallocation per day or week for the intervention;and (c) coverage of specified amount of materialin the curriculum or teacher guide (when appro-priate). Although we label this surfacefidelity, it isan essential part of any study and, thus, extremelyimportant.

Some interventions, such as home visitinginterventions, may not be as "visible," but still re-quire fidelity assessments. Videotapes of visitsmay need to be obtained and scored. Occasionallyinterventions will theorize a causal chain that re-quires assessment of intermediate links in thechain. For example, a home visiting interventionmay theorize that the home visitors working withparents in a certain way will change the way par-ents work with their child, which in turn will in-fluence the child's behavior. In this type ofsituation, assessment of the causal link, the waythe parents work with their child, is important tobe sure that the intervention worked as theorized.

More sophisticated measures of fidelity re-quire an observer to document not only the oc-currence of prescribed steps in the process, butalso that the interventionist employed the tech-niques with quality. For example, an observer canevaluate whether teachers model skills and strate-gies using clear language and interesting exam-ples, or whether they scaffold or providecorrective feedback consistently. Investigatingtreatment fidelity at this level provides a deeperunderstanding of implementation issues and canlead to important insights about interventioncomponents and teacher behaviors that are moredirectly related to desired outcomes. It also can bea lens into how well interventionists understandthe principles behind the concept. This level al-ways requires some level of inference.

Use of observational field notes or audio-tapes of select sessions can be used to gain an un-derstanding of quality. Often, selected use oftranscripts can enrich a journal article or researchreport on the study by giving the reader a sense ofhow the intervention plays out with students andof actual curricula materials (e.g., Fuchs & Fuchs,1998; Palincsar & Brown, 1984). Finally, it is im-portant to assess treatment fidelity in a way thatallows researchers to ascertain whether the level ofimplementation was constant or whether fidelityvaried or fluctuated within and across interven-tionists. Researchers should carry out this type ofanalysis both for individual interventionists overtime to determine if implementation was consis-tent across the duration of the intervention aswell as between interventionists to determine ifsome teachers delivered the treatment with greaterintegrity than did others.

Exceptional Childrcn 157

These data can serve numerous other pur-poses. If researchers find a reasonable amount ofvariability, as will often be the case in studies thatuse quality of implementation measures, they cancorrelate implementation scores with outcomes oruse contrasted groups analysis to see if quality orintensity of implementation relates to outcomemeasures. They also can use the profiles to get asense of typical implementation problems. Thesedata can be used to develop or refine professionaldevelopment activities.

The nature of services provided in comparisonconditions are described and documented.

One of the least glamorous and most ne-glected aspects of research is describing and assess-ing the nature of the instruction in thecomparison group (Gersten, Baker, et al., 2000).Yet, to understand what an obtained effect means,one must understand what happened in the com-parison classrooms. This is why researchers shouldalso describe, assess, and document implementa-tion in comparison groups.

At a minimum, researchers should examinecomparison groups to determine what instruc-tional events are occurring, what texts are beingused, and what professional development andsupport is provided to teachers. Other factors toassess include possible access to thecurriculum/content associated with the experi-mental group's intervention, time allocated for in-struction, and type of grouping used duringinstruction (Elbaum, Vaughn, Hughes, & Moody,1999). In some studies, assessment of the com-parison condition may be similar to what occursfor treatment fidelity. Every study will vary basedon specific research issues.

OUTCOME MEASURES

Multiple measures are used to provide an ap-propriate balance between measures closelyaligned with the intervention and measures ofgeneralized performance.

Far too often, the weakest part of an inter-vention study is the quality of the measures usedto evaluate the impact of the intervention. Inter-vention researchers often spend more time on as-pects of the intervention related to instructional

procedures than on dependent measures. How-ever, using tests of unknown validity invariablyweakens the power of a study. Thus, a good dealof the researcher's effort should be devoted to se-lection and development of dependent measures.In essence, the conclusion of a study depends notonly on the quality of the intervention and thenature of the comparison groups, but also on thequality of the measures selected or developed toevaluate intervention effects.

One of the foremost challenges in crafting astudy is selection of measures that are well alignedwith the substance and the intervention and yetare sufficiently broad and robust to (a) avoid criti-cism for "teaching to the test" through the specificintervention, and (b) demonstrate that generaliz-able skills have been successfully taught. An inter-vention may have adverse effects or additionalbenefits that a researcher should attempt to iden-tify by using measures of generalized perfor-mance. Multiple measures should be used in astudy, as no measure is perfect, and no measurecan assess all or even most of the important as-pects of performance that an intervention mightaffect. If a study addresses more than one con-struct (e.g., fluency with arithmetic facts andproblem-solving ability), it can be valuable to in-clude multiple tools to measure each facet of per-formance.

Finally, in special education research thatincludes a wide range of different types of stu-dents, it is critical that measures are able to ade-quately assess performance of students withdifferent disabilities. This may sometimes involveuse of accommodations or use of alternate for-mats such as structured interviews on the contentrather than written essays (see for example Bottge,Heinrichs, Mehta, & Ya-Hui, 2002; Ferretti,MacArthur, & Okolo, 2001; Gersten, Baker,Smith-Johnson, Peterson, & Dimino, in press).

Evidence of reliability and validity for theoutcome measures is provided.

Estimation of internal consistency reliabil-ity (often referred to by technical names such ascoefficient alpha or Cronbach's alpha) is a criticalaspect of a research study; however, this is oftenneglected. This omission is difficult to understandbecause coefficient alpha is easy to compute withcommon statistical packages, and the information

5Winter 200515H8

is very important. Internal consistency reliabilityhelps us to understand how well a cluster of itemson a test fit" together, and how well performanceon one item predicts performance on another.These analyses help researchers locate items thatdo not fit well, or items with a weak item-to-totalcorrelation. Revising these items or droppingthem from the measure used increases the reliabil-ity and thus increases the power of a study.

The bottom line for reliability are coeffi-cient alpha reliabilities of at least .6. This mayseem low; however, measurement and design ex-perts such as Nunnally and Bernstein (1994) andShadish et al. (2002), indicate that this level of re-liability is acceptable for a newly developed mea-sure or a measure in a new field for two reasons.First, in experimental research, as opposed to in-dividual assessment, we are concerned with theerror associated with the mean score. The stan-dard error of measurement of the mean is appre-ciably less for a sample mean than an individualtest score. Second, internal consistency of .6 orhigher indicates that a coherent measurementconstruct is being measured.

An ideal mix of outcome measures in astudy would also include psychometrically soundmeasures with a long record of accomplishment,such as the Woodcock Johnson or Walker-Mc-Connell, with Cronbach alpha reliabilities over .8and newer measures with coefficient alpha relia-bility of .6 or so. Although we strongly encourageinclusion of data on test-retest reliability, an in-tervention study is typically considered acceptablewithout such data, at the current point in time.

At the same time, some indication of con-current validity is also essential. For newly devel-oped measures, using the data generated in thestudy would be acceptable, as would data from astudy of an independent sample. Yet, to be highlyacceptable, the researcher should independentlyconduct some type of concurrent validity. Con-current validity becomes even more critical whenusing measures for groups other than those forwhich the test was designed (e.g., using a measuretranslated into Spanish for use with Spanish-speaking bilingual students). For studies to rankin the highly acceptable category, empirical dataon predictive validity of measures used and anyinformation on construct validity should be re-ported.

Outcomes for capturing the intervention's ef-fect are measured at the appropriate times.

The goal of measurement activities in ex-perimental research is to detect changes in a de-pendent variable, which are the result ofimplementing an independent variable. Depend-ing on the nature of the independent variable anda study's research questions, there may be criticalperiods of time in which intervention effects canbe captured through data collection activities.

Many times, the effects of an interventionare best detected immediately. For example, if re-searchers are interested in evaluating the initial ef-fectiveness of a 4-week intervention designed tointroduce a strategy for deriving word meaningsfrom context, data should be collected within afew days of the end of the intervention. If datacollection is delayed, initial intervention effectsmay fade or subsequent instruction or other un-controlled events may contaminate results.

Sometimes, however, important interven-tion effects cannot be detected immediately. In-formation about the delayed or long-term effectsof interventions is extremely important but oftennot addressed or measured in research studies. Forexample, researchers may be interested in deter-mining whether students are still able to apply theword-deriving strategy 6 weeks after the interven-tion ended. In cases such as this, researchersshould time data collection activities to align withresearch questions.

A good deal of the researcher's effortshould be devoted to selection and-devel-opment of dependent measures.

For some studies, it may be appropriate tocollect data at only pre- and posttest. In manycases, however, researchers should consider col-lecting data at multiple points across the course ofthe study, including follow-up measures. Multipledata points allow researchers to apply more com-plex statistical procedures such as HierarchicalLinear Modeling or Growth Curve Analyses.These analyses allow for an examination of indi-

Exceptiona Children1 159

vidual student trajectories as well as overall groupeffects.

These data can provide researchers with amore nuanced, complex picture of student growthas well as information about immediate and long-term intervention effects. Thus, we would con-sider these indicators of exemplary researchdesigns, provided the array of measures makessense developmentally.

Data collectors and/or scorers are blind tostudy conditions and equally (un)familiar toexaminees across study condition.

Data collection activities may inadvertentlyaffect a studys findings. One way this can happenis if data collectors and/or scorers are aware ofwhich students took part in the treatment andcontrol conditions. Experimenter bias may resultfrom this situation if researchers' expectations un-intentionally influence assessment or scoring pro-cedures. Study findings can also be affected ifsome participants are familiar with data collectorswhile other participants are not. In this case, par-ticipants may feel more comfortable with individ-uals that they know and therefore perform betterduring testing. Researchers should design andcarry out data collection activities in a way tominimize these threats to the study's internal va-lidity.

Having data collectors/scorers blind tostudy conditions and examinees is optimal for anystudy, although there will be times when it is im-possible to implement. In all cases, however, datacollectors should be trained and interrater/ob-server/tester reliabilities conducted.

Adequate interscorer agreement is docu-mented.

Although researchers must select outcomemeasures that align with the purpose of the studyand that possess acceptable levels of reliability andvalidity, the quality of the data from these mea-sures is only as good as the corresponding datacollection and scoring procedures. To ensure theinterpretability and integrity of study data, re-searchers should also consider carefully how datais collected and scored.

Researchers should ensure that test admin-istration and scoring is consistent and reliableacross all data collectors and scorers. Documenta-

tion of agreement is especially important whenusing newly developed measures or when scoringinvolves subjective decisions. Interscorer agree-ment can be evaluated by having multiple datacollectors administer assessments to a randomsample of study participants or by observing se-lected testing sessions. Similar procedures can beused to check agreement in data scoring. Ideally,reliability in data collection and scoring should beabove .90.

DATA ANALYSIS

We highlight several key themes in this set of in-dicators. The linkage of data analysis and unit ofanalysis to key research questions is important. Inaddition, we emphasize that the researcher use ap-propriate statistical techniques to adjust for anypretest differences on salient predictor variables.

A key issue is ensuring that data analysesand research questions are aligned with the appro-priate unit of analysis for a given research ques-tion. Researchers should actually define whichunit was used in the statistical analysis of inter-vention effects. For example, in one type of analy-sis, individual students may be the unit ofanalysis; in another case, where students aretaught in pairs, the pair may well be the appropri-ate unit of analysis.

The determination of the appropriate unitof analysis relates directly to the research hypothe-ses/questions, the sample, and assignment of sam-ple to experimental conditions. When studentsare the unit, the n equals the number of studentsin the analysis and each student contributeshis/her score. When classrooms are the units, then equals the number of classrooms, and individ-ual students' scores are averaged to reflect a classvalue. When a full class is a unit, it may well beadvisable to assess all of the special education stu-dents but assess only a stratified random sampleof the nondisabled students.

It is also possible that within the samestudy, for example, the teacher may be the appro-priate unit of analysis for some measures (e.g.,teacher knowledge of inclusive practices), and theindividual student may be the best unit of analy-sis for others (e.g., degree of engagement in aca-demic activities with peers). When appropriate,we urge the use of multilevel analyses because

Winter 2005160

they are uniquely designed to consider multipleunits within a single analysis.

The quality indicators related to data-ana-lytic strategies and study design are essential.However, it would require, at a minimum, a fullissue of this journal to begin to do justice to thetopic. Choice of appropriate data-analytic strategyhas always been partly art and partly science. Usu-ally, numerous strategies are appropriate, somemore elegant and sophisticated than others. In thepast decade, there have been enormous advancesin this area, with increased understanding of andwide use of structural equation modeling andmultilevel modeling. Thus, the array of options islarge and determining a strategy or set of strate-gies that is sensitive/powerful enough to detecttreatment effects, while also sophisticated enoughto look at important secondary patterns or themesin the data, is a complex task.

In this brief section, we highlight a few ofthe key themes that both those designing studiesand those evaluating proposals or research reportsneed to keep in mind when considering dataanalyses.

The data analysis techniques chosen are ap-propriate and linked in an integralfashion tokey research questions and hypotheses.

Statistical analyses are accompanied withpresentation of effect sizes. Merely cataloguingevery possible analysis that can be done is unac-ceptable. A brief rationale for major analyses andfor selected secondary analyses is critical.

The variability within each sample is ac-counted for either by sampling or statisticaltechniques such as analysis of covariance.

Postintervention procedures are used to ad-just for differences among groups of more than0.25 of a standard deviation on salient pretestmeasures. Planned comparisons are used whenappropriate.

The researcher should clearly link the unit ofanalysis chosen to the key statistical analyses.

Support for the unit(s) of analysis relativeto the conceptual and practical foci of the treat-ment is made. Multilevel analysis is used whenappropriate. Although data can always be ana-

Exceptional Children

lyzed at multiple levels, the reader of a proposal orreport should clearly know the levels for whichthe analysis will be most powerful and which unitof analysis is most appropriate for a specific re-search question and design. Appropriate sec-ondary analyses are conducted (e.g., testing forthe effectiveness of the treatment within impor-tant subgroups of participants or settings).

A power analysis is provided to describe theadequacy of the minimum cell size. A poweranalysis is conductedfor each unit of analysisto be examined (e.g., school and class as wellas student).

If an intervention is effective for the popu-lation targeted, statistical power denotes the prob-ability that a statistical test will accurately denotestatistical significance. Factors traditionally identi-fied as determining statistical power include, sam-ple size, anticipated effect size, and type ofstatistical test selected. Given the increasing use ofmultilevel models, we would like to add the num-ber of assessment waves as an additional factor af-fecting statistical power.

Under conditions of study economy (as isoften the case in special education interventionresearch), the appropriate number of units neededin the design is determined by the effect size oneexpects to detect (small vs. medium vs. large).Cohen's (1988) traditional break out of effect sizeranges is: .2 = small, .5 = moderate, and .8 andgreater = large. Effect sizes in the .40 or largerrange are often considered minimum levels for ed-ucational or clinical significance (Forness, Kavale,Blum, & Lloyd, 1997; Walberg, 1986). Experi-enced special education investigators determinethe number of participant units needed in a studyusing effect sizes determined from pilot data thathave used the intervention and relevant outcomevariables. Alternatively, one may obtain expectedeffect sizes from related published research for thepurpose of conducting a power analysis and plan-ning a new study. Power analyses should alwaysbe conducted to help make decisions concerningthe minimal sample size necessary and the num-ber of comparison conditions that are truly viable.In early stages of research on a topic, it makessense to design a study conservatively, with a largeenough sample to increase the chance of detectingsmall effects. Conducting a study in two or even

161

three waves or cycles often makes sense in theseinstances.

In some cases, the costs of designing a studycapable of detecting small effects may simply notbe interesting or worth it. Thus, smaller partici-pant units can be used. The point is that specialeducation researchers can conduct studies that arecorrectly powered relative to available resources,student population sizes, and goals of the researchwhen hypothesizing moderate-to-large effects andkeeping the number of experimental conditionssmall (i.e., two rather than three group design).Replicating a series of studies with smaller sam-ples powered to detect moderate-to-large effectsappears to offer a powerful approach to the dis-covery of causal relations in small populations/samples.

When proposing a study, researchers shouldprovide a power analysis for the most importantanalyses and either explicitly or implicitly indicatehow they address the major factors that influencethe power of a research design. In other words,they need to indicate whether they anticipate asmall, medium, or large effect and how they haveaddressed issues of sample heterogeneity/variabil-ity in their design and analysis strategies. Thepower analysis should indicate that the design hasadequate sample size to detect hypothesized ef-fects. The researchers should indicate the impactof waves of assessments on the power of theiranalyses.

DETERMINING WHEN A

PRACTICE IS EVIDENCE-BASED:

A MODEST PROPOSAL

Currently, there is a great deal of deliberation anddiscussion on what it means to call an educationalpractice or special education intervention evidencebased. Particularly controversial issues are the rela-tive weightings of randomized trials versus quasi-experiments and to what extent we can generalizethe findings across various subgroups of studentswith disabilities. Another key issue is how tomake the determination that an evidence-basedpractice is implemented with such low qualitythat we could no longer assert that it is likely toenhance learner outcomes.

The authors of this article were not in com-plete agreement on any of these issues. However,

we decided, with some trepidation, to present amodest proposal, a criterion for concluding apractice is evidence based. In reaching this deter-mination, we were heavily influenced by the re-search and scholarship on research synthesis andmeta-analysis over the past 20 years (e.g., Cooper& Hedges, 1994), and the research syntheses con-ducted in special education (e.g., Gersten,Schiller, & Vaughn, 2000; Swanson & Hoskyn.1998).

* There are at least four acceptable quality stud-ies, or two high quality studies that supportthe practice; and

* The weighted effect size is significantly greaterthan zero.

Again, to be considered acceptable, a studywould need to meet all but one of the EssentialQuality Indicators specified in Table 2 anddemonstrate at least one of the quality indicatorslisted as Desirable. To be considered high quality,a study would need to meet all but one of the Es-sential Quality Indicators specified in Table 2 anddemonstrate at least four of the quality indicatorslisted as Desirable.

The reason we stressed the weighted effectsize is that this statistic takes into account (a) thenumber of studies conducted on the specific in-tervention, (b) the number of participants in thestudies, and (c) the magnitude and consistency ofeffects. To us, these three components seem to bemost critical for making determinations as towhether a practice is evidence based.

We propose the following criteria for con-sidering a practice as promising:

* There are at least four acceptable quality stud-ies, or two high quality studies that supportthe practice; and

* There is a 20% confidence interval for theweighted effect size that is greater than zero.

To reach these recommendations, weweighed the quality of the research design (e.g.,Feuer et al., 2002; Shadish et al., 2002) heavily.There are several reasons for doing this. Recently,Simmerman and Swanson (2001) documentedthe impact of specific flaws in research design ona study's effect size. They argued, "Among thefactors that determine a study's credibility, it is

162 Win ter 2005

often the internal and external validity variablesthat establish research quality and have signifi-cant effects on treatment outcomes" (p. 211). Totest this hypothesis, Simmerman and Swansonexamined the effects of a large set of internal andexternal validity variables on treatment outcomesfor students with learning disabilities (LD) usingstudies identified in the Swanson and Hoskyn(1998) meta-analysis on treatment outcomes andLD.

Results indicate the following factors leadto lower effect sizes: (a) controlling for teacher ef-fects, (b) using standardized rather than just ex-perimenter-developed measures, (c) using theappropriate unit in data analyses, (d) reportingthe sample's ethnic composition, (e) providingpsychometric information, and (f) using multiplecriteria defining the sample. Simmerman andSwanson's (2001) results indicate that better con-trolled studies appear to be less biased in favor ofthe intervention. It is interesting to note that useof random assignment (versus use of quasi-experi-mental designs) approached, but did not reach,statistical significance (p = .0698).

Thus, issues addressed by these quality in-dicators appear to influence the inferences wemake about effective approaches for special edu-cation. Better controlled studies (in terms ofusing measures of documented reliability, control-ling for effects of teachers or interventionists, andusing the appropriate unit of analysis in statisticalcalculations) tend to be less biased. Use of stan-dardized measures of broad performance lead toweaker estimates of effectiveness of an interven-tion. The key lesson we can learn from the Sim-merman and Swanson (2001) analysis is thatresearch quality does matter and does have educa-tional implications.

We feel it is important to note that conduct-ing quality research is more expensive than con-ducting experiments that are compromised atvarious levels. We see signs of awareness of thisfact in both the National Research Council (2002)report on educational research and in some cur-rent federal initiatives. Routinely conducting qual-ity research in public schools will also require ashift in the culture of schools, much as it requireda shift in the culture of medical clinics and hospi-tals 50 years ago, and of public welfare programs 2decades ago. Active support by the U.S. Depart-

Exceptional Children

ment of Education will, in our view, be required.It would seem to go hand-in-hand with the cur-rent emphasis on scientifically based research.

This set of criteria is merely a first step. Weenvision the following as critical next steps:

We feel it is important to note that con-ducting quality research is more expensivethan conducting experiments that arecompromised at various levels.

* Field-testing of this system of indicators bycompetent individuals as they review grant ap-plications and other research proposals andmanuscripts submitted to journals for publica-tion.

* Refinements based on field-testing in both theareas of proposal review and of research syn-thesis.

* Consideration for adoption by journals in ourfield and/or funding agencies such as Instituteof Educational Sciences and Office of SpecialEducation Programs.

* Serious field-testing of the quality indicators'impact on evidence-based practice needs to beconducted. The issue of integrating findingsfrom different types of research (e.g., correla-tional, single-subject design, qualitative) needsto be considered as part of this field-testing ef-fort.

This would seem to be a reasonable effort forCEC's Division for Research to undertake in thenear future in conjunction with relevant federalagencies, other divisions of CEC, and, potentially,agencies that review research grant applicationsrelating to special education.

REFERENCES

Boruch, R. F (1997). Randomized experiments for plan-ning and evaluation: A practical guide. Thousand Oaks,CA: Sage.

Bottge, B. A., Heinrichs, M., Mehta, Z. D., & Ya-Hui,H. (2002). Weighing the benefits of anchored math in-

163

struction for students with disabilities in general educa-tion classes. Journal of Special Education, 35, 186 -200.

Campbell, D. T, & Stanley, J. C. (1966). Experimentaland quasi-experimental designs for research. Boston, MA:Houghton Mifflin.

Cohen, J. (1988). Statistical power analysis for the behav-ioral sciences (2nd ed.). Hillsdale, NJ: Lawrence EribaumAssociates.

Cooper, H., & Hedges, L. V. (Eds.). (1994). Handbook ofresearch synthesis. New York: Russell Sage Foundation.

Elbaum, B., Vaughn, S., Hughes, M., & Moody, S. W(1999). Grouping practices and reading outcomes forstudents with disabilities. Exceptional Children, 65,399-415.

Fabes, R. A., Matrin, C. L., Hanish, L. D., & Updegraff,K. A. (2000). Criteria for evaluating the significance ofdevelopmental research in the twenty-first century: Forceand counterforce. Child Development, 71, 212-221.

Ferretti, R. P., MacArthur, C. D., & Okolo, C. M.(2001). Teaching for historical understanding in inclu-sive classrooms. Learning Disability Quarterly, 24, 59-71.

Feuer, M. J., Towne, L., & Shavelson, R. J. (2002). Sci-entific culture and educational research. Educational Re-searcher, 31, 4-14.

Forness, S., Kavale, K. A., Blum, 1. M., & Lloyd, J. W(1997). Mega-analysis of meta-analyses: What works inspecial education and related services. TEACHING Ex-ceptional Children, 29, 4-9.

Fuchs, D., & Fuchs, L. S. (1998). Researchers and teach-ers working together to adapt instruction for diverse learn-ers. Learning Disabilities Research & Practice, 13, 126-137.

Gall, M. D., Borg, W R., & Gall, J. P. (2002). Educa-tional research: An introduction (7th ed.). White Plains,NY: Pearson/Allyn & Bacon.

Gersten, R, Baker, S., & Lloyd, J. W (2000). Designinghigh quality research in special education: Group experi-mental design. Journal ofSpecial Education, 34, 2-18.

Gersten, R., Baker, S., Smith-Johnson, J., Peterson, A.,& Dimino, J. (in press). Eyes on the prize: Teaching his-tory to students with learning disabilities in inclusive set-tings. Exceptional Children.

Gersten, R., Schiller, E. P., & Vaughn, S. (Eds.). (2000).Contemporary special education research: Syntheses of theknowledge base on critical instructional issues. Hillsdale,NJ: Lawrence Erlbaum Associates.

Gresham, F. M., MacMillan, D. L., Beebe-Franken-berger, M. E., & Bocian, K. M. (2000). Treatment in-tegrity in learning disabilities intervention research: Dowe really know how treatments are implemented? Learn-ing Disabilities Research &Practice, 15, 198-205.

National Research Council. (2002). Scientific research ineducation. (R J. Shavelson & L. Towne, Eds.), Commit-

tee on Scientific Principles for Educational Research.Washington, DC: National Academy Press.

Nunnally, J. C., & Bernstein, I. H. (1994). Psycholmetric

theory (3rd ed.). New York: McGraw-Hill.

Palincsar, A. S., & Brown, A. L. (1984). Reciprocal teach-ing of comprehension-fostering and comprehension-mon-itoring activities. Cognition and Instruction, 1, 117-175.

Shadish, W R., Cook, T. D., & Campbell, D. T. (2002).Experimental and quasi-experimental designs for generalcausal inference. Boston: Houghton Mifflin.

Simmerman, S., & Swanson, H. L. (2001). Treatmentoutcomes for students with learning disabilities: Howimportant are internal and external validity? Journal ofLearning Disabilities, 34, 221-236.

Swanson, H. L., & Hoskyn, M. (1998). Experimentalintervention research on students with learning disabili-ties: A meta-analysis of treatment outcomes. Review ofEducational Research, 68, 277-321.

Valentine, J. C., & Cooper, H. (2003). What WorksClearinghouse Study Design and Implementation Assess-ment Device (Version 1.0). Washington, DC: U.S. De-partment of Education.

Walberg, H. J. (1986). Synthesis of research on teaching.In M. C. Wittrock (Ed.), Handbook of research on teach-ing (3rd ed., pp. 214-229). New York: Macmillan.

ABOUT THE AUTHORS

RUSSELL GERSTEN (CEC #108), Director, In-structional Research Group, Signal Hill, Califor-nia. LYNN S. FUCHS (CEC #185), Professor,and DONALD COMPTON (CEC TN Federation),Assistant Professor, Vanderbilt University,Nashville, Tennessee. MICHAEL COYNE (CECCT Federation), Assistant Professor, University ofConnecticut, Storrs. CHARLES GREENWOOD(CEC #436), Director, Juniper Gardens ChildrensProject, University of Kansas, Lawrence. MARKS. INNOCENTI (CEC #512), Research AssociateProfessor, Utah State University, Logan.

Address all correspondence to Russell Gersten, In-structional Research Group, 2525 Cherry Ave.,Suite #300, Signal Hill, CA 90755. E-mail: rger-sten@inresg.org

We wish to acknowledge the excellent editorialfeedback provided by Susan Marks, Madhavi Jayan-thi, Jonathan R Flojo, and Michelle Spearman.

Manuscript received August 2003; accepted May2004.

164 W%inter 2005

COPYRIGHT INFORMATION

TITLE: Quality Indicators for Group Experimental andQuasi-Experimental Research in Special Education

SOURCE: Except Child 71 no2 Wint 2005WN: 0534903306003

The magazine publisher is the copyright holder of this article and itis reproduced with permission. Further reproduction of this article inviolation of the copyright is prohibited. To contact the publisher:http://www.cec.sped.org/

Copyright 1982-2005 The H.W. Wilson Company. All rights reserved.