1163620

7/29/2019 1163620

1/21

Toward a Conceptual Framework for Mixed-Method Evaluation DesignsAuthor(s): Jennifer C. Greene, Valerie J. Caracelli and Wendy F. GrahamSource: Educational Evaluation and Policy Analysis, Vol. 11, No. 3 (Autumn, 1989), pp. 255-274Published by: American Educational Research AssociationStable URL: http://www.jstor.org/stable/1163620 .

Accessed: 17/09/2013 18:20

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of

content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms

of scholarship. For more information about JSTOR, please contact [email protected].

.

American Educational Research Association is collaborating with JSTOR to digitize, preserve and extend

access toEducational Evaluation and Policy Analysis.

http://www.jstor.org

This content downloaded from 197.1.162.232 on Tue, 17 Sep 2013 18:20:52 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/action/showPublisher?publisherCode=aerahttp://www.jstor.org/stable/1163620?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/1163620?origin=JSTOR-pdfhttp://www.jstor.org/action/showPublisher?publisherCode=aera

7/29/2019 1163620

2/21

EducationalEvaluationandPolicyAnalysisFall 1989, Vol.11, No. 3, pp. 255-274

Towarda ConceptualFramework or Mixed-MethodEvaluationDesignsJenniferC. Greene,ValerieJ. Caracelli,andWendyF. GrahamCornellUniversity

In recentyearsevaluatorsof educationaland socialprogramshaveexpanded heirmethod-ological repertoirewith designs that includethe use of both qualitativeand quantitativemethods. Suchpractice,however,needs to be grounded n a theorythat can meaningfullyguide the designand implementation f mixed-method valuations. n this study,a mixed-methodconceptualrameworkwasdevelopedrom thetheoreticaliterature nd thenrefinedthroughan analysis of 57 empiricalmixed-method valuations.Five purposes or mixed-methodevaluationsare identified n this conceptualramework: riangulation, omplemen-tarity,development,nitiation,andexpansion.Foreachof the ive purposes,a recommendeddesign is also presentedin terms of seven relevantdesign characteristics.These designelementsencompassssuesaboutmethods, hephenomenaundernvestigation, aradigmaticframework,and criteriaor implementation.n theempiricalreview,commonmisuseof thetermtriangulationwasapparentn evaluations hat statedsuchapurposebutdid notemployan appropriate esign.In addition,relativelyew evaluations n this review ntegrated hedifferentmethodtypesat the level of data analysis. Strategies or integrateddata analysisareamongthe issuesidentifiedas prioritiesor furthermixed-methodwork.

The inevitable organizational,political,and interpersonalchallenges of programevaluationmandate he useof multiple oolsfrom evaluators' ull methodologicalreper-toire (Cook, 1985; Mathison, 1988). In re-cent years,this repertoirehas been consid-erably expanded with the acceptance ofqualitativemethods as appropriate, egiti-mate,andeven preferredora widerangeofevaluationsettingsand problems.Concom-itantly, evaluatorshave expressedrenewedinterest n mixed-methodevaluationdesigns

Anearlier ersion f thispaperwaspresentedas a panelat the 1988 AnnualMeetingof theAmericanEvaluation ssociationn NewOrle-ans. Theauthors reindebted o MelvinMarkforhisinsightfulndconstructiveomments ntheworkpresentederein, othat theconferenceand in subsequentpersonalcommunications(Mark, 988).

employingbothquantitativeandqualitativemethods (e.g., Cook & Reichardt, 1979;Madey, 1982; Rossman & Wilson, 1985;Smith & Louis, 1982). However,the terri-tory of mixed-method designs remainslargelyuncharted.Of particularneed is acleardifferentiationof alternativepurposesfor mixing qualitative and quantitativemethodsand of alternativedesigns,analysisstrategies, ndcontextsappropriateoreachpurpose(Greene& McClintock,1985).Forexample,in currentpractice,quitedifferentmixed-methoddesigns are advocated andused in variedevaluationcontexts for thecommon proclaimedpurposeof triangula-tion. Such practicemuddlesthe concept oftriangulation s originallyconstruedandre-mains insensitiveto otherpossiblebenefitsof mixed-methoddesigns(Mathison,1988).Further,ust as carefulplanningand defen-sible rationalesaccompanythe design and

255

http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp

7/29/2019 1163620

3/21

Greene, aracelli,ndGrahamimplementationof evaluation case studies,ethnographies, surveys, and quasi-experi-ments, so must similar thoughtfulnessbegiven to the designand implementationofmixed-methodstudies.Toward heseends,the presentstudycon-tributes o the developmentof a conceptualframework, hus enablingmore thoughtfuland defensiblemixed-methodevaluative n-quiry. In this study, we defined mixed-methoddesignsas thosethat includeat leastonequantitativemethod(designed o collectnumbers)and one qualitativemethod (de-signedto collectwords),whereneithertypeof method is inherentlyinked to anypartic-ularinquiryparadigm.Throughan analyticreviewof first heoreticaland thenempiricalliterature on mixed-method inquiry, thisstudy generated valuable information onmixed-methodpurposesand designcharac-teristics.Reviewprocedures ndfindings orthesetwocomponentsof ourmixed-methodconceptual frameworkthus constitute thefocus of the present discussion.Relativelylittle information was garneredrelevant toother components of this framework,in-cludingthe differentialutilizationof quan-titative and qualitative information, dataanalysisstrategiesand contextsappropriatefor mixed-method inquiries, as well asmixed-methodprojectmanagementand re-source ssues.Theseconcernsarebrieflydis-cussed at the conclusionof the presentarti-cle as issueswarrantingurtherwork.

TheoreticalBaseThis study on mixed-methodevaluationinquirywasgrounded n an initialreview offour theoreticalstartingpoints, selected fortheirconceptualattentionto one or more ofthe key issues represented n our mixed-methodconceptualframework.Triangulation.(See Campbell & Fiske,1959; Denzin, 1978; Webb, Campbell,Schwartz, & Sechrest, 1966; see also Mathi-son, 1988, for an excellent discussion oftriangulation from these same sources.)From its classicsources, riangulation efersto the designed use of multiple methods,with offsettingor counteractingbiases, ininvestigationsof the same phenomenon in

orderto strengthen he validityof inquiryresults.The corepremiseof triangulation sa design strategy s that all methods haveinherent biases and limitations, so use ofonly one methodto assessa givenphenom-enon will inevitablyyieldbiasedandlimitedresults.However,when two or more meth-ods that have offsettingbiases are used toassessa givenphenomenon,and the resultsof these methods converge or corroborateone another, then the validity of inquiryfindingsis enhanced. As noted by Greeneand McClintock(1985), this triangulationargument requires that the two or moremethodsbe intentionallyused to assessthesameconceptualphenomenon,be thereforeimplemented simultaneously,and, to pre-servetheircounteractingbiases,also be im-plemented ndependently.Multiplism. (See Cook, 1985; Mark &Shotland, 1987; Shotland& Mark, 1987.)ThomasCook'scriticalmultiplismacknowl-edgesthe decreasedauthorityof social sci-ence theory and data in a postpositivistworld and then seeks to reaffirm andstrengthenhevalidityof, andtherebyusers'confidencein, empiricalworkby extendingthe basiclogicof triangulationo all aspectsof the inquiryprocess.

Thefundamentalostulate f multiplismis thatwhen t is not clearwhichofseveraloptionsorquestionenerationrmethodchoice s "correct,"ll of themshouldbeselectedo as to "triangulate"nthemostuseful or the most likely to be true ....Multiplismimsto foster ruthby estab-lishing orrespondencescrossmanydiffer-ent,butconceptuallyelated,waysof pos-inga question ndby rulingoutwhetheranyobtainedorrespondencesreartifactsofanyepiphenomenafvalue, ubstantivetheory,or methodchoicethatmayhavebeen nadvertentlyncorporatedntoindi-vidualests. Cook,1985,pp.38and46)Congruent with the basic logic of trian-

gulation, Cook's multiplism emphasizes en-hanced validity via convergence of resultsfrom multiple methods, theoretical orienta-tions, and political or value perspectives.Cook also acknowledges that the results ofmultiple methods may serve more comple-256


7/29/2019 1163620

4/21

Toward Conceptualrameworkmentary hanconvergentpurposes,as whendifferentmethodsareused for different om-ponentsof a multitaskstudy. Elaborating nthis point, Markand Shotland(1987) offerthreedifferentpurposes ormultiple-methoddesigns: a) triangulation,which seeks con-vergenceof findings; (b) bracketing,whichseeks a range of estimates on the correctanswer(or triangulationwith a confidenceinterval); nd(c)complementarity,n whichdifferentmethodsare usedto assessdifferentstudycomponentsor phenomena,to assessthe plausibilityof identified hreats o valid-ity, or to enhancethe interpretability f as-sessmentsof a singlephenomenon-for ex-ample, via broadercontent coverageor al-ternate evelsof analysis.Mixing methods and paradigms. (SeeGuba & Lincoln, 1984; Kidder & Fine,1987;Reichardt&Cook, 1979;Rossman&Wilson, 1985;Smith, 1983;Smith&Heshu-sius, 1986.) This set of referenceswas se-lected primarilyfor their common discus-sion of the following design issue: Aremixed-methodevaluationdesigns, n whichthequalitativeandquantitativemethodsarelinked to contrasting inquiry paradigms,meaningful,sensible,and useful?Rossmanand Wilson (1985) outline a continuum ofthree stances on this issue: the purists,thesituationalists, nd the pragmatists.The purists(includingGuba & Lincoln,1984;Smith, 1983;and Smith&Heshusius,1986) answerwith an unequivocal"no"tothe issue posed. They arguethat the attri-butesof a paradigm orma "synergisticet"that cannot be meaningfullysegmentedordivided up. Moreover,differentparadigmstypicallyembody incompatibleassumptionsabout the nature of the world and what isimportant o know,forexample,realistver-sus relativistontologies. So, mixed-methodevaluationdesigns,in whichthe qualitativeand quantitativemethods are conceptual-ized and implemented within different par-adigms (characteristically, interpretive andpostpositivist paradigms, respectively), areneither possible nor sensible.In contrast, Reichardt and Cook (1979)argue pragmatically that paradigm attributesare logically independent and therefore can

be mixedand matched, n conjunctionwithmethods choices, to achieve the combina-tion most appropriate or a given inquiryproblem.Thepracticaldemandsof theprob-lem are primary; inquirer flexibility andadaptivenessare neededto determinewhatwill work best for a given problem. Or, inthe pragmaticviewof Milesand Huberman(1984), epistemologicalpuritydoes not getresearchdone.The middle-groundsituationalist posi-tion, articulatedby KidderandFine (1987),retainsthe paradigmaticntegritystanceofthe puristsbutalso argues, ike the pragma-tists, that our understandingof a given in-quiryproblemcanbesignificantly nhancedby exploringconvergences n storiesgener-ated from alternateparadigms.Congruentwith Cook's proposalfor aggressivemeta-analyses,Kidderand Fine suggest hat suchexplorationsoccuracrossstudies,in partic-ular,acrossquantitative postpositivist) ndqualitative interpretivist)tudies.Thisstrat-egy may yield "storiesthat converge"ordiscrepancies hat invoke freshperspectivesand new,more illuminatingexplanations.In a similarvein, Rossman and Wilson(1985) soughttheirown middle groundonthis issueof mixingparadigmsby outliningthree functionsfor mixed methodology: a)corroboration,as in establishingconver-gence;(b) elaboration,as in providingrich-ness and detail; and (c) initiation, which"promptsnewinterpretations,uggestsareasfor furtherexploration,or recasts he entireresearchquestion. Initiationbringswith itfresh nsightanda feelingof thecreativeeap.... Rather than seeking confirmatory evi-dence, this [initiation]design searchesforthe provocative"Rossman&Wilson, 1985,pp. 637 and 633).Mixed-method design strategies. (SeeGreene, 1987;Greene& McClintock,1985;Knapp, 1979;Madey, 1982;Mark& Shot-land, 1987; Maxwell, Bashook, & Sandlow,1986; Sieber, 1973; Trend, 1979.) This morediverse set of references was reviewed pri-marily for additional ideas on alternativemixed-method purposes and on design char-acteristics that may differentiate amongthese purposes. Building on the work of

257


7/29/2019 1163620

5/21

Greene,Caracelli,ndGrahamGreene and McClintock (1985), Greene's(1987) synthesis of these ideas with thoserepresented n the other three theoreticalstartingpoints served as a key foundationforthe presentconceptualwork(andis thusincorporatedwithin our later presentationof findings).

EmpiricalReviewBelievingthat soundconceptualworkre-quiresan interplayof theory and practice,we next conducted a comprehensivereviewof a purposivesampleof 57 mixed-methodevaluation studies. Our review guide in-cludedall of the componentsof our mixed-methodconceptual rameworkpurpose,de-sign characteristics,utilization,data analy-sis, contexts, management,and resources),with directedemphasisfromthe theoreticalstartingpointson the firsttwo.' The samplewas purposive n that we aimed to identifystudies in which the mixed-methodaspectof the design was prominent and thus inconcert with our researchobjectives.The

searchwas limited to studiesreported rom1980 to 1988. We also soughta broadrep-resentationof differentapproaches o eval-uation, differentkinds of evaluands, anddifferent types of evaluation documents.Our final sample,which included 18 pub-lished evaluationstudies, 17 evaluation re-ports,and 22 evaluationpapers,met all ofour samplingcriteriaexcept representationacrossevaluands.Comparedwithotherdatabases employed during sampling, ERICyielded many more appropriatestudies;hence, our sample tilted toward mixed-method evaluations conducted on educa-tionalprograms.Our reviews of these selected literatureson thetheoryandpracticeof mixed-methodevaluationyieldedmost importantlya set offive differentmixed-methodpurposes andseven relevantdesigncharacteristics.Theseresultsare presentedin the following twosections.

Results for Mixed-MethodPurposesTheory.The five mixed-methodpurposesgeneratedfrom our theoreticalreview are

presented n Table 1 and brieflyelaboratedbelow.A mixed-methoddesignwith a triangula-tion intent seeksconvergence n the classicsense of triangulation.The use of both aqualitative interview and a quantitativequestionnaire to assess program partici-pants'educationalaspirationsllustrateshistriangulationintent. In conjunction withthis intent, Shotlandand Mark(1987) cau-tion that differentmethodsmay be biasedin the same direction or, in fact, may beaskingdifferentquestions.Variationswithinthis triangulation urpose ncludeCampbelland Fiske's (1959) advocacy of multiplemethodsto evaluatediscriminantas well asconvergent validity, and Mark and Shot-land's(1987)ideaof usingmultiplemethodsto bracketrather han convergeon the cor-rect answer.This idea of triangulationwitha confidence interval is drawn from Rei-chardtand Gollob(1987).In a complementarity mixed-methodstudy,qualitativeand quantitativemethodsare used to measureoverlappingbut alsodifferentfacets of a phenomenon, yieldingan enriched, elaboratedunderstandingofthat phenomenon. This differs from thetriangulationntentin that the logic of con-vergencerequires hat the differentmethodsassess the same conceptual phenomenon.The complementarity ntent can be illus-tratedby the use of a qualitative nterviewto measurethe natureand level of programparticipants'ducationalaspirations, swellas influenceson theseaspirations, ombinedwitha quantitativequestionnaireo measurethe nature, level, and perceived rankingwithin peer group of participants'educa-tionalaspirations.The two measures n thisexampleareassessing imilar,as well as dif-ferent,aspectsof the aspirationsphenome-non. One variationwithinthis complemen-tarityintent is the use of differentmethodsto assessdifferent evels of a phenomenon(Mark& Shotland, 1987), which we char-acterizedwith the analogy of peeling thelayersof an onion.Sieber(1973) and Madey (1982), for so-ciological and evaluationcontexts, respec-tively, provide many creativeexamples of

258


7/29/2019 1163620

6/21

0 00c ld 00 00O- O's00~

~ON(US00 0r- 0000QO's0(7..,d

0sc

V5 Vs-h U a,-

00

so o0

>- s-ov(U 00

0,~

c-c

000 V '

co00

~~, Qd OCq

E V cO

Q1 k C001c Q

--l +1 ;

> 9z

0 0VL Q Z~cdCtj )C 3'Ct O(u (u cl cl M u0(4-4 0 POz O C

~ CVs - . 4 ~ VCE

j c l ko ~E o 0 +.A 0

2All 00 Lfrl U -4 , -W to C C

259


7/29/2019 1163620

7/21

Greene,Caracelli,andGrahammixing methods for developmentpurposes.All involve the sequentialuse of qualitativeand quantitativemethods, where the firstmethodis used to help informthe develop-ment of the second.For example,a quanti-tative surveyof programparticipants' du-cationalaspirations ould be used to identifya purposivesamplefor more in-depth nter-viewsabouttheseaspirations.For a given mixed-methodstudy, initia-tion as the discoveryof paradoxand freshperspectivesmay well emerge ratherthanconstitute a planned intent. However, incomplex studies, as well as across studies,bothconsistenciesanddiscrepanciesn qual-itativecomparedwith quantitative indingscan be intentionallyanalyzedfor freshin-sights invoked by means of contradictionand paradox.A mixed-method tudywithan expansionintent is a "multitask"study in Cook's(1985)multiplism ramework r a studythataims for scope and breadthby includingmultiple components. In evaluation con-texts,thismixed-method xpansionpurposeis commonlyillustratedby the use of quali-tative methodsto assessprogramprocessesand by quantitativemethodsto assesspro-gramoutcomes.Practice.Ourempiricalreviewresults ub-stantiallyconfirmedthis conceptualizationof mixed-methodpurposes.For all studieswith a discernible ationale ormixingmeth-ods, this rationalematchedone or more ofthese five purposes.Hence, we offerthis setof purposesasrepresenting oththepracticeand potential of mixed-methodevaluationstrategies seealsoSmith, 1986)and asprog-resstowarda common parlance orconcep-tualizingand describingmixed-method ra-tionalesin programevaluation.2In the empirical review, mixed-methodpurposeswere tabulated both accordingtothe studyauthors'statementof purposeandby ourdefinitions.As shown in Table2, theauthors'stated primaryor secondary pur-pose for using a mixed-methoddesignwasoften triangulation (23%) or expansion(26%). However,in a similarproportionofevaluations, no purpose for the mixed-methoddesignwasstatedorcouldbe readily

inferred.By our definitions, four fifths ofthe primarypurposesand one halfof the 70total purposeswere eithercomplementarity(not triangulation) r expansion.The moreinteresting inding n Table2 isthe backwardZpattern ormedbythiscross-tabulation.The diagonal representsagree-ment betweenthe authors' statementsandour definitionsof mixed-methodpurposes.For example, five of the empiricalstudiesreviewedhad a primaryor secondarypur-poseof triangulation upper eftcell)accord-ingto bothdeterminations.Whentherewassuch agreementabout purpose,the authorswere usually very explicit in their statedrationalefor the particularmixed-methoddesignchosen. For example, in an evalua-tion of a physical education project, "thedata were examined ... and presented em-ploying the processesof triangulationandcorroborationn order o arriveat validandreliable statements" (Moody, 1982, Ab-stract).Additional llustrationsof this diag-onal,or instancesof our mixed-methodpur-posesin evaluationpractice, ollow.Theevaluationnstrumentseredesignedtogiveoverlappingcomplementarity]ndcrosscheckingtriangulation]ssessmentsof theperceptionsf those nvolved.Pe-ters,Marshall, Shaw,1986,p. 16)Overall,hemethodologiessedconfirmedthat any paper-and-pencilnstrumentought o be supplementedy qualitativemethods.Thiswouldenrichandprovidedeptho thestatisticalataobtained.Mar-tin, 1987,pp.14-15) complementarity]Quantitativeethods anestablishhede-gree o whichperceptionsreshared, utuncovering he perceptionshemselvesmustbe [first] onenaturalistically.Gray&Costello, 987,p. 12)[development]Qualitativen additionto quantitativemethodswere ncluded o the evaluationcould"tell hefullstory."Hall, 1981,p.127)[expansion]The whole is greater han the sum of thepartswhenqualitativeandquantitative p-proaches and methods are combined.(Smith,1986,p. 37) [initiation]The horizontal lines forming the top andbottom of the backward Z show disagree-

260


7/29/2019 1163620

8/21

6 - 0,--- - r,-,, " e

0-O0 ,--.1,,0 0 D0U,,

cn?a .c1oo t-3

0

10-

03 +-# 0 " " V-

rO

'(d0

(

S0

.

W-6

UIi

B~C.>r4 g6

261


7/29/2019 1163620

9/21

Greene,Caracelli,andGrahamment between the authors'statedintentionsforthe mixed-methoddesignand ourdeter-mination of purposes.Thisdiscrepancys ofone of two types:Eitherthe authorsstatedtriangulationas the purposefor the mixed-methoddesignwhen it was not, or the au-thorsdid not state a purposewhenwe wereable to identifyone. The latterdiscrepancyisdifficult o illustratebecauseoftheabsenceof a statedpurposeby the authors. An ex-cerptfromMoran(1987) illustrates he firstdiscrepancy.This evaluator statedtriangu-lationas the purposefor the mixed-methoddesign,when we identifiedprimaryandsec-ondary purposesof developmentand initi-ation,respectively.

Some researchersmaintain hat a two-tieredmethodologys not reallya triang-ulation.Greeneand McClintock1985)contend that ... "a nonindependent,se-quentialmixed-methodtrategyosesthecapacityortriangulation.nthisstrategy,the methodsare deliberatelynteractive,notindependent,ndtheyareappliedin-glyover imeso that heymayormaynotbe measuringhe same phenomenon."Giventhe dynamicnatureof the publicservice,t is difficulto discernhowanyevaluationoutine ouldmeet he[se] rit-icisms .... The ideabehind an interactivesequentialmethodologys notto measurethe samephenomenon t the sametime,but to use the indingsof one methodologyto inform he issues to be addressed n thesubsequentevaluation. Under this con-struct, ualitativeataareemployedoen-sure hat hequantitativetudy s current.Quantitativeata n turnareused o refor-mulatethe issuesfor the qualitative tudy.(Moran,1987, pp. 623-624, emphasesadded)

Resultsfor Mixed-MethodDesignCharacteristicsThe seven characteristics of mixed-method designs presented in Table 3 repre-

sent an integration of results from our the-oretical and empirical reviews. Although theempirical review did not alter the initial setof theoretically-derived design characteris-tics,3 it did serve to refine and clarify ourconceptualization of each. Nonetheless, we

do not considerthis set of mixed-methoddesigncharacteristicso be exhaustive,butratherwe anticipate uture refinementsandadditions. Brief descriptions of the sevenmixed-methoddesigncharacteristicsener-ated in this study follow. Empiricalresultsforthesedesigncharacteristicsrepresentedin the nextsection,differentiatedyprimarymixed-methodpurpose.Methods.The methodscharacteristicep-resentsthe degreeto which the qualitativeand quantitative methods selected for agiven studyare similarto or different romone another n form,assumptions, trengths,and limitations or biases (as argued byCampbell&Fiske,1959).A scaledquestion-naireandstructurednterviewwouldbecon-sidered imilar,whereasan achievement estand open-ended nterviewwouldbe consid-ered different.Mid-rangepositionscan oc-cur when the methods sharesome charac-teristics,especiallybiases,but not others,asin the combineduseof aquantitativewrittenquestionnaireand a qualitativecritical nci-dent (alsowritten)diary.Phenomena. The term phenomena refersto the degreeto which the qualitativeandquantitativemethodsareintendedto assesstotally differentphenomenaor exactly thesame phenomenon. When differentmeth-ods areimplemented o assessdifferentphe-nomena, the methodsare usuallyrespond-ing to different questions. To illustrate,quantitative measures like standardizedachievementtests are often used to assess

the degreeof successof an educationalpro-gram,and qualitativemeasuressuch as in-terviewsandobservations reusedto under-standhow and why a program s successfulor unsuccessful.Mid-range phenomena positions occurwhen qualitativeand quantitativemethodsoverlap n theirintent,yet alsocapitalizeonthe strengths of one or both methods tosecure additional information. For example,Smith and Robbins (1984) used quantitativesurveys to provide a detailed picture of thenature, causes, and consequences of parentalinvolvement in four different federal pro-grams. A qualitative site review, which in-cluded interviews, observations, and docu-

262


7/29/2019 1163620

10/21

0 CISS-4 0~. '0

$-(0 0 4)

V) ,r +It~i ) t) '54220 5 l $-4

00 -0s0 '00-A 0 00 .- .' 0

c : C13clEac0 0d

o 0 cc ' ) 4) -0 to 0 L. 0,E c

1111+.AI II:)o0 '00

's toiQ) t4., .7.4.-0 a c0 0

~~ 0

0'- 00

0x 0

0 c l 4)l l a 0

~>'00

0~

"S

ul,

lr~$04 Q

263


7/29/2019 1163620

11/21

Greene,Caracelli,andGrahamment analysis,was intendedto securesimi-larinformationon parental nvolvement,aswell as additional nformationabout the ef-fects of parental nvolvementin variedpro-gramsettings e.g., ruralvs. urban).Paradigms.The design characteristica-beled paradigms refers to the degree towhich the differentmethodtypesareimple-mented within the same or differentpara-digms.We recognizethat any given pairofquantitativeand qualitativemethodseitheris or is not implemented within the sameparadigm,rendering his designcharacteris-tic dichotomous.Evaluationpractice,how-ever, commonlyincludesmultiplemethodsof both types. Thus, the ratings n Table 3areintended to be holistic,representinghedegree o whichthe whole set of methodsisconceptualized,designed,and implementedwithin the same ordifferentepistemologicalframeworks.Assessmentsof this designele-ment should be made independentlyof therelative number and status of qualitativeversusquantitativemethods.Status. This characteristic epresents hedegree to which a study's qualitativeandquantitativemethods have equally impor-tant or central roles vis-a-vis the study'soverallobjectives.In contrast o paradigms,the status design characteristic hould di-rectly reflectthe relativeweight and influ-ence of the qualitative and quantitativemethodswithrespect o theirfrequencyandtheircentrality o studyobjectives.Implementation: ndependence.The de-greeto which the qualitativeand quantita-tive methodsare conceptualized,designed,andimplemented nteractively r independ-entlycanbe viewedon a continuum.Some-times a study includes both components,representinga mid-rangeposition. For ex-ample, in part of Louis's (1981) study,mixed-method implementation was inde-pendent: Standardizeddata collection bycentralproject taffoccurred imultaneouslywith the development of 42 miniethnogra-phies by field staff, who worked withoutknowledgeof the central staffs emergingfindings.Part of Louis's study was also in-teractive:During analysis and interpreta-tion, everyindividualwho contributedas a

major author or analyst to the study wasfamiliarwith all dataavailable.Implementation:Timing. Although werepresent hischaracteristics a continuum,we againrecognize hata given pairof meth-ods is typicallyimplemented concurrentlyor sequentially,not in between.Yet, a shortquantitativemethodcouldbe pairedwith alongerqualitativemethod, or pre-posttestscould be implementedbeforeand afterpar-ticipantobservationillustrating,romTable3, "concurrent" nd "bracketed"iming,re-spectively).Variationon thisdesignelementalsoarises romthe use of multiplemethodswithin a mixed set. With reflectionwe re-fined this characteristicby dividingit intocategories(see Table 3) that could be as-sessedfor a whole set of mixed methodsor,as appropriate,or eachpairof methods.Study.The final designcharacteristica-beled study is essentiallycategorical.Theempiricalresearcheitherencompassedonestudyor more thanone study.Althoughourown review yielded little variationon thisdesigncharacteristicallbutfourevaluationsrepresenteda single study), it remains animportantconsideration or continueddis-cussion of mixed-method designs (Cook,1985;Kidder&Fine, 1987).

Mixed-MethodPurposesx DesignCharacteristics:RecommendedDesignsTo review, the long-rangegoal of thisstudy is the developmentof a conceptual

frameworkhat could informand guidethepractice of mixed-method inquiry. Suchguidancewouldincludea descriptionof thekind of design(and analysis, context, etc.)most appropriateora givenmixed-methodpurpose. For this reason,we analyzedtheempirical review resultson mixed-methoddesign characteristicseparately or studiesgroupedby our definitionof primary pur-pose.This analysis s presentedn Figure1.Each row of Figure 1 representsone ofour five mixed-methodpurposes;each col-umn presentsa singledesigncharacteristicandthe scaleby whichit wasrated.Thefivepoints in these scalescorrespond o the fol-lowing ratings of these design elementswhich we viewed as continua during our264


7/29/2019 1163620

12/21

4-.0

11n1000

C4 C4

04

.1mgg f?8ag mn

vrggg vr vr n vr tam fa Ia ?a ?a

N N

FW v c2C4 N

IICIO

U,,

r-0E

?a ?tas ?a ?a?a

V-4 -2 0co

~Br r vr V I C?a ?a ? ?aI;

265


7/29/2019 1163620

13/21

Greene,Caracelli,andGrahamempiricalreview: at either end (1 and 5),near eitherend (2 and 4), nearthe middle(3). Each cell entryin FigureI thusdisplaysthe distributionof our ratingson a singledesign characteristicfor a given mixed-methodpurpose.For example,the graph nthe upper eft-handcell showsthatthe qual-itativeand quantitativemethods wererateddifferentascoreof 5) in all threeevaluationswitha triangulationpurpose.Incorporatinghese empiricalreview re-sults on design characteristics,his sectionheuristically resentsarecommendeddesignforeachof the identifiedmixed-methodpur-poses. There are three caveats to keep inmind as these recommendationsare pre-sented.First,the importanceof these sevencharacteristics o mixed-methoddesigns isgenerally upportedbyourempiricalreview.Nonetheless,we havegreaterconfidenceinour definitionsof mixed-methodpurposesandconsiderelements of mixed-methodde-sign choice an open area of investigation.Second, mixed-methodstrategiesare oftenguided by more than one purpose. Thus,designswill not appearas pristineon thesecharacteristicsn practice as we have setthem forthhere.Third,we acknowledgehatsome departures rom these recommendeddesigns can be readily defended. Mark(1988), for example, suggestedthat for atriangulationdesign, the differentmethodsneed not be implementedsimultaneouslyfthe phenomenon of interest is stable overtime. In short, we present these recom-mended designs to underscore the impor-tance of design element choice in mixed-methodframeworks, ut we present hem asworking ideas rather than prescriptivemodels.Figure2 profilesour five recommendedmixed-methoddesigns.In this figure,eachletter representsa different mixed-methodpurpose.Individual ettersdenote a recom-mended position on a designcharacteristicfor a particularpurpose.Letterswith barsindicatethatthe recommendedpositioncanrange somewhat.The omission of a lettermeans that a specific position on a charac-teristic s not warranted.

Triangulation (T) design. The combined

use of quantitativeand qualitativemethodsfor the purposeof triangulationdominatescurrentdiscussionsaboutmixed-method a-tionales.Yet, as indicatedby our empiricalreview(see Table 2), methodological rian-gulationin its classicsense is actuallyquiterare n mixed-methodpractice.Our recom-mendedtriangulationdesign s basedon thelogicof convergence mbedded n theclassicconceptualization f triangulation.This logic requiresthat the quantitativeand qualitativemethods be differentfromone another with respectto their inherentstrengths and limitations/biasesand thatbothmethodtypesbeusedto assess hesamephenomenon. Methods that are biased inthe same directionorthatask/answerdiffer-ent questionscan underminethe triangula-tion logic and resultin spurious nferences(Shotland & Mark, 1987). Relatedly, themethods need to be conceptualized,de-signed, and implementedwithin the sameparadigmaticframework(Greene & Mc-Clintock, 1985; Kidder & Fine, 1987).Strong between-methods triangulation isalso enhancedwhen the statusof the differ-ent methods-that is, their relativeweightand influence-is equalandwhenthequan-titative and qualitativestudy componentsare implemented ndependentlyand simul-taneously.Across mixed-methodpurposes,the recommendedindependentimplemen-tation of the differentmethods is uniquetotriangulation.ComplementarityC) design.One appar-entlycommon purpose orcombiningqual-itative and quantitativemethods is to usethe results from one method to elaborate,enhance, or illustratethe results from theother.The recommendedcomplementaritydesign depicted n Figure2 is similarto thetriangulationdesign,with the exceptionofthe phenomenaand implementation-inde-pendence characteristics.The phenomenacharacteristichas a slight range, indicatingthat the quantitativeand qualitativemeth-ods shouldbe used to examineoverlappingphenomena or differentfacets of a singlephenomenon.In complementaritydesigns,the paradigmaticramework or both typesof methodsshould also be similar,and in-

266


7/29/2019 1163620

14/21

TowardA Conceptual rameworkIC

METHODS Similar -D- T DifferentI -D-PHENOMENA Different E -C- T Same

DCPARADIGMS Different I T SameIDCSTATUS Unequal T Equal

IMPLEMENTATION:ID

Independence InteractiveC T Independent

CTiming SequentialD T SimultaneousEC

STUDY > Onestudy T OnestudyFIGURE2. Recommendedmixed-methoddesignsNote. T = triangulation;C = complementarity;D = development; = initiation;E = expansion.

terpretability s best enhanced when themethods are implemented simultaneouslyand interactivelywithina single study.In our empirical review there were 18mixed-methodevaluations mplemented orthe primarypurposeof complementarity.Acomparisonof the designcharacteristics fthese 18 studies(see Figure1)with our rec-ommended complementaritydesign yieldsconsiderablecongruence.On each designcharacteristicwith the exception of status,approximatelyhree fourthsof these mixed-method studieswere udgedto be at or closeto our recommended position. Somewhatmore variabilitywasevidentforstatus.This

congruenceof theorywithpracticesupportsand encouragesboth.Development (D) design. The salient fea-ture of our recommendeddevelopmentde-sign is the sequentialtiming of the imple-mentation of the different methods. Onemethod is implemented irst,andthe resultsare used to help select the sample,developthe instrument,or inform the analysisforthe other method. By definition, then, im-plementation s alsointeractive,andthe dif-ferent methods are used to assess the sameor similar phenomena, conceptualizedwithin the sameparadigm.Wefurthermain-tain that strong development designs use

267


7/29/2019 1163620

15/21

Greene, aracelli,ndGrahamdissimilarmethods of equal status. Mixed-method studies with a developmentintentcan occur within a single study or acrossstudies,conductedsequentially o capitalizeon the benefits derivedfrom each methodtype.Like the theory-to-practicecomparisonfor complementarity, he design character-istics of the seven empiricalmixed-methodstudies conducted for purposesof develop-ment (see Figure 1) were quite congruentwith this recommendeddesign.The designsof fiveor six of these studieswere at or closeto the recommendedposition on all char-acteristicsexcept phenomena. Surprisingly,for this design element, three studies usedmixed methods to assess different ratherthan similarphenomena.An interesting ar-iation was found in one study that imple-mented the different methods simultane-ously ratherthan sequentially(Bower,An-derson,& Thompson, 1986).In this case, asmall "prefatorynaturalistic study" pro-vided a descriptive base of information,which was then used for threesuccessivelylarger"waves"of data collection, each ofwhichincludedboth quantitativeand qual-itativemeasures.GrayandCostello(1987) stretchour con-ceptions about this design by advocatingmixing methods, as well as paradigmsfordevelopment purposes. Their main argu-ment is that the use of naturalisticqualita-tive methods to assesscontext firstdoes notpreclude the use of positivist quantitativemethods for other purposes later in thestudy. Gray and Costello'swork also sup-ports our call for a more thoroughunder-standingof the contextsappropriateorvar-ious mixed-methodpurposesand of the in-fluencethatcontextualfactorsmayhave onmixed-methoddesigns.Initiation(I) design. In a mixed-methodstudy with an initiation intent, the majoraim of combining qualitativeand quantita-tive methods is to uncover paradox andcontradiction.Jick (1983) discussedsimilarpurposesin outlininghis "holistictriangu-lation"design.Rossmanand Wilson(1985)demonstratedthat iterative use of bothmethodtypescanintentionally eek areasof

nonconvergence n orderto "initiate nter-pretationsandconclusions,suggestareas orfurtheranalysis,orrecast he entireresearchquestion"(p. 633).Nonetheless, purposeful initiation maywell be rare in practice.One excellent ex-ampleof a more emergent nitiationdesignfrom our empiricalreviewis Louis's(1981)evaluation of the Researchand Develop-ment Utilization program (RDU). Thiseight-million-dollardemonstrationprojectwas fundedby NIE (National InstituteforEducation)between 1976 and 1979 to pro-mote theadoptionof new curricula ndstaff

developmentmaterialsn 300 local schools.Louis discusseskey featuresand examplesof the "cyclicalinteraction"model devel-oped duringthe course of this evaluation,includingthe following:1. Purposivesamplingof particular aseswas combined with random samplingforsurveyor other structureddata collectioninorder to maximizeboth discoveryand gen-eralizability.2. An iterativeapproach o instrumenta-tion for bothfield datacollection and morestandardized instruments was achievedthroughongoinginteractionbetweenquali-tativeandquantitativeanalyses.3. Analysis beganwith the firstdata col-lection and occurred at periodic intervalsthroughoutthe project.The same staffen-gaged n simultaneousanalysisof bothqual-itative and quantitativedata. Testing andverificationof both types of data sourcesincreasedreliabilityand validity.A second example of a more emergentinitiation designis Maxwellet al.'s (1986)evaluationof the use of "medicalcare eval-uationcommittees"n physicianeducation.In thisunusualstudy,ethnographicmethodswere employed within an experimentalframework. nitiationfeatureswereevidentin the authors'commentsregardinghe ad-vantages of the ethnographic approach: "Itallowed us to discover aspects of the com-mittees' educational functioning that we hadnot anticipated and would have missed hadwe relied on quantitative methods" (p. 138).Specifically, the qualitative data prompted arecasting of how medical care evaluation

268


7/29/2019 1163620

16/21

Toward Conceptualrameworkcommittees nfluencedphysicians'perform-ance. In the originalhypothesis,committeeparticipationwas expected to directly in-crease a physician'sknowledgeand therebyenhancehis/her performance.The data in-dicated, however,thatcommitteeparticipa-tion servedto increasethe physician'scon-fidence to apply knowledgehe/she alreadyhad,and this enhancedconfidenceunderlayperformance hanges.Drawing in part on these empiricalex-amples, our recommended design for amixed-methodevaluationwith an initiationintent incorporateswo distinctivefeatures.First,the phenomenainvestigatedwith ini-tiation-orientedmixed methods couldcovera broad range. Second, to maximize thepossibilityof unlikely findings,mixingpar-adigms n thisdesignis acceptableand evenencouraged.This advocacyof mixedepiste-mological frameworks is congruent withCook's (1985) call for multiple theoreticaland value frameworksn appliedsocial in-quiry.Expansion (E) design. In our empiricalreview, the most frequentlycited mixed-method purposewas expansion.This sug-geststhatmanyevaluatorsaremixingmeth-ods primarily o extend the scope, breadth,andrangeof inquirybyusingdifferentmeth-ods for different nquirycomponents.Typi-cally, in the empirical studies reviewed,quantitativemethods were used to assessprogram outcomes, and qualitativemeas-uresto assess mplementation.Figure2 rec-ommends only two elements for a mixed-method expansion design. The empiricalworkwouldbe encompassedwithina singlestudy,and,uniqueto expansiondesigns, hephenomenainvestigatedwould be distinct.Oursampleof mixed-methodexpansionde-signs is fairlycongruentwith these recom-mendations(seeFigure1).The decision to "expand"an evaluationto include both process and product com-ponents is undoubtedly motivated by thedesire to produce a more comprehensiveevaluation. However, in many of the evalu-ations of this genre that we reviewed, therewas a paramedic quality to the qualitativecomponent. That is, qualitative data often

appeared n the emergencyroom of reportwritingas a life-savingdeviceto resuscitatewhat was eithera failedprogramor a failedevaluation.Problematicprogramsor evalu-ations with insufficient(quantitative)con-trols or statisticalpower were discussedinterms of (qualitative) participant experi-ences, implementation impediments, andrecommendations for program improve-ment.What is at issue here is how qualitativeand quantitativemethodsin an expansiondesigncanbe mixedmeaningfully ndeffec-tively.Even in the strongerexpansionstud-iesreviewed, hequalitativeandquantitativemethods were kept separate throughoutmostphasesof theinquiry.Thetermparalleldesign(Louis, 1981) may appearmore ap-propriate.Yet, weprefer he termexpansionbecause we believe it more accuratelyre-flects the "multitask"ntent of such studiesin Cook's multiplismframework.4We alsobelievethat mixed-methodexpansionstud-ies have not yet tested the limits of theirpotential. For example, in a higher orderexpansiondesign,a more integrateduse ofmethods could be achieved by employingcombinationsof qualitativeandquantitativemethods to assessboth implementationandoutcomes.Such a study may well incorpo-rateelements of triangulationand comple-mentarityinto its design, becoming,in ef-fect,a multipurposetudy.Ora higherorderexpansiondesigncould use a mix of differ-ent methods, each creativelydesigned toassessconceptualstrandsthat span or linkprogram implementation and outcomes.The majorbenefitof such higherorderde-signs would be strengthenednferences.Incontrast,our reviewsuggested hat the cur-rent normativeexpansiondesignkeepsthedifferentmethods separatedand thus doesnot realizesuchbenefits.In summary,Figure3 presentsa funnelarray of recommended design options forthe various mixed-method purposes. Thisarray indicates that design options are rela-tively constrained and narrow for somemixed-method purposes but more flexibleand wider for others. The order from mostto least constrained design options for

269


7/29/2019 1163620

17/21

Greene,Caracelli,andGraham

Constrained-NarrowT R I A N G U L A T I O N

C O M P LEM E NT A RI TY

DEVELOPMENTINITIATIONEXPANSION

Open-WideFIGURE3. Flexibilityof designoptions or mixed-method urposes

mixed-methodpurposeis as follows: trian-gulation, complementarity, development,initiation,and expansion.Results forMixed-MethodData Analyses

Our empirical review also assessed thenature and degreeof qualitativeand quan-titative integrationattained by the studiesreviewed during their data analysis andinterpretation/reportingtages.The resultswere groupedin four categories: a) no in-tegration-both analysesand interpretationwereconductedseparately;b) analyseswereconductedseparately,but some integrationoccurredduring interpretation;c) integra-tion occurred during both analyses andinterpretation;nd(d)analysesnotreported.A crosstabulationof these analysis resultswith mixed-methodpurposesis shown inTable 4. These results revealthat althoughnearlyequalnumbersof thestudiesreviewedattained some degree of qualitative andquantitative ntegrationas did not (23 and25, respectively),only 5 studies achievedsuch integrationduringthe analysisprocessitself. The resultsfurthersuggestthat rela-tively low levels of integrationmay charac-terizestudieswith an expansion intent,andperhapsrelatively high levels may accom-panystudies withan initiationintent.

TowardFurtherDevelopmentof Mixed-MethodTheoryandPracticeInthis analysisof selectedtheoreticalandempirical iterature,we havebegunto chartthe territoryof mixed-method evaluation

designs.Our focus has been on clearlydif-ferentiatingalternativepurposes or mixingqualitativeand quantitativemethods. De-sign characteristics relevant to mixed-method strategiesand appropriateor spe-cific purposeswere also explored.In addi-tion to further refinement of these mixed-method purposesand design elements,webelieve severalother issues within our over-all conceptual frameworkrepresent highprioritiesfor futurework. These issues in-clude the relationship of mixed-methodstrategieso evaluationpurpose,continuingparadigmquestions,procedures or mixeddata analysis,utilization,and relevantcon-textual factors.Withrespect o the first ssue,wesurmisedthat important distinctions in mixed-method purposesand designs might arisewith differentevaluation intents-that is,formativeversussummativeor processver-sus product.An analysisof ourown sampleof empirical studies (which included 11process studies, 12 productstudies,and 30evaluationswith both processand productcomponents)yieldedno markeddifferencesin mixed-methodpurpose.Nonetheless, herelationshipbetween evaluation purposesand mixed-methodstrategies s an impor-tant area forfurtherresearch.To the probabledismayof purists, n thisstudy we sidesteppedthe knotty paradig-matic issues involved in mixed-method n-quiry.Yet, a comprehensivemixed-methodframeworkmusteventuallyaddresswhetherit is appropriate o mix paradigmswhen

270


7/29/2019 1163620

18/21

TowardA Conceptual rameworkTABLECrosstabulationf mixed-method nalysesandpurposes

Purpose TotalsTriangula-Complement- Develop- Initia- Expan-Analysis ategory tion arity ment tion sion No. %No integration 6 2 17 25 44

Integrationuringnterpreta-tion 3 8 2 5 18 32Integrationuring nalysisand nterpretation 1 1 2 1 5 9Notreported 3 2 4 9 16mixingmethods. Our own thinkingto datesuggestshat the notion of mixing paradigmsis problematicordesignswithtriangulationor complementarity purposes, acceptablebut still problematic or designswith a de-velopmentorexpansion ntent,andactivelyencouragedfor designs with an initiationintent.Futureresearchalso need to addresstheissue of data analysisstrategies or mixed-method evaluations. In our empirical re-view,only five studies ntegratedqualitativeand quantitativedata during the analysisprocess.The creativeand promisingstrate-gies used by these evaluatorsare reportedmore fully in a separatearticle (Caracelli,Greene,&Graham,1989).In 18 additionalstudiesthat we reviewed,some measure ofintegrationof the different data sets wasattainedduring nterpretationndreporting.Typically,in these studies,qualitativedatawerebrought n to supportor explain quan-titativefindings, o fleshout conclusions,orto makerecommendations.However,whendata mismatchesoccurred,there was littlediscussionn anyof thesestudiesaboutthesediscrepancies,nor were there effortsto re-solve them. Both Trend (1979) and Jick(1983)discussthe importanceand the chal-lengeof reconcilingnonconvergentindings."When different methods yield dissimilarresults, heydemandthatthe researcher ec-oncile the differences omehow. In fact, di-vergencecan oftenturnout to be an oppor-tunity for enrichingthe explanation" Jick,1983, p. 143). Shotlandand Mark (1987)

also underscore he importanceof the "em-piricalpuzzles"Cook,1985)thatarisewhenresultsdo not converge,and they call for amore systematicexplorationof the possiblecauses of such inconsistentresults.An additionalimportantarea of inquiryconcernsutilizationspecificallyas it relatesto mixed-method trategies.The fundamen-tal issue here is this: In what common anddifferentways s quantitative ndqualitativeinformation used? And what implicationsdo these utilization processes have formixed-method approachesto evaluation?Further, attention to contextual factorskeepsus mindfulof an importantquestion:Is the problemguidingour choice of meth-ods, or vice versa?These identified areas of future mixed-methodinquiry-the roleof evaluationpur-pose, paradigm ssues, data analysisstrate-gies, and utilization-as well as others ofparticularnterest o other nquirers, refun-damental o the inherentaim of the researchpresentedherein.Carefulplanningand de-fensible rationalesmust accompanythe de-signandimplementationof mixed-methodsevaluations.This goalcan be achievedonlywith a morecomprehensiveheoryto guideuse of mixed methods n evaluationpractice.

NotesInordero beable odescribeurrentmixed-methodpractice,we also extractedromeachselectedtudya descriptionf theevaluandndof the evaluationpproach,urposee.g., orm-ativeorsummative),ime rame, ndqualitative271


7/29/2019 1163620

19/21

Greene,Caracelli,and Grahamand quantitativemethods used. This descriptiveinformationand reference ist for our sampleofmixed-method evaluation practice is availablefrom the authorsupon request,as are the com-plete details of our samplingand reviewproce-dures.

2 Sevenof the empirical tudiesreviewedcitedas a secondaryrationalethe inclusion of eitherqualitative or, more commonly, quantitativemethods,not for methodologicalor theoreticalreasons,but rather n anticipationof studyaudi-ences' knownpreferences r needs for this formof information.This politicalresponsivenessn-tent for mixingmethods can be viewed,in part,as a tacticalmaneuver o increase he utilizationof evaluationresults.However, n contrast o theother mixed-methodpurposes,a responsivenessintent is unlikelyto invokeany significant ffortat integration, itherat the level of methods,or,more importantly,with respect o the inferencesdrawn(Mark, 1988). For this reason,we viewresponsiveness s conceptuallydifferent rom theotherfivemixed-methodpurposes.

3 An eighth design characteristic dentifiedfromthe theoretical iteraturewasdeletedduringthe pilot testing of the empiricalreview guide.From Cook (1985) and Shotland and Mark(1987), this characteristicwas the following:Arethe criteriaused to decide which phenomenatoassesswith multiplemethods(i.e., what to makemultiple) derived from theory (substantiveormethodological)or from the context? Thoughdeleted as a design element, this concern wasretainedn thedescriptive ectionof theempiricalreviewguide.

4 Alternatively,Mark 1988) suggestedhatex-pansionbe viewed,in conjunctionwith comple-mentarityand triangulation, s a continuumofmixed-methodpurposes.This continuumis es-sentially our phenomena design characteristic,representingheuse of differentmethods o assessdifferent,related,similar,or the same phenom-ena.References

Bower, . C.,Anderson, .N., &Thompson, .(1986, April). Triangulationn the evaluationofschoolsysteminterventions. aperpresentedat the annual meeting of the AmericanEdu-cationalResearchAssociation,SanFrancisco.Campbell,D. T., &Fiske,D. W. (1959).Conver-gentanddiscriminant alidationby the multi-trait-multimethodmatrix.PsychologicalBul-letin, 56, 81-105.Caracelli,V. J., Greene,J. C., & Graham,W. F.

(1989).Mixed-method ataanalysis:Strategiesand issues.Manuscriptn preparation.Cook, T. D. (1985). Postpositivistcriticalmul-tiplism. In R. L. Shotland & M. M. Mark(Eds.),Socialscienceand socialpolicy(pp.21-62). BeverlyHills,CA:Sage.Cook, T. D., & Reichardt,C. S. (Eds.).(1979).Qualitativeand quantitativemethods n evalu-ation research.BeverlyHills,CA:Sage.Denzin,N. K. (1978). The research ct:An intro-duction osociologicalmethods chap. 10).NewYork:McGraw-Hill.Gray,P.,&Costello,M.(1987,October).Contextas determinant f methodology n multi-clientserviceorganizations.Paper presentedat theannual meeting of the AmericanEvaluationAssociation,Boston.Greene,J. C. (1987). Uses and misusesofmixed-method evaluationdesigns. Proposalfor the1988 annualmeetingof the AmericanEduca-tion ResearchAssociation,New Orleans.Greene,J. C., & McClintock,C. (1985). Trian-gulation n evaluation:Designand analysis s-sues.EvaluationReview,9, 523-545.Guba,E.G., & Lincoln,Y. S. (1984).Do inquiryparadigms imply inquiry methodologies?Copyrightedmanuscript.

Hall, J. N. (1981). Evaluationand comparison:Social learningcurriculumand instrumentalenrichment. Final report (Contract No.G008001869,U.S. Departmentof Education).Nashville, TN: George Peabody College forTeachers. (ERIC Document ReproductionServiceNo. ED 244 484)Jick,T. D. (1983). Mixing qualitativeand quan-titativemethods:Triangulationn action.In J.VanMaanen (Ed.), Qualitative methodology(pp. 135-148). BeverlyHills,CA:Sage.Kidder,L.H., &Fine,M.(1987).Qualitative ndquantitativemethods:Whenstoriesconverge.In M. M. Mark& R. L. Shotland Eds.),Mul-tiplemethods n programevaluation:NewDi-rections or ProgramEvaluation 35 (pp. 57-75). SanFrancisco: ossey-Bass.Knapp,M.S. (1979).Ethnographicontributionsto evaluationresearch. n T. D. Cook & C. S.Reichardt Eds.),Qualitativeand quantitativemethods n evaluationresearch pp. 118-139).BeverlyHills,CA:Sage.Louis, K. S. (1981, April).Policy researcher ssleuth:New approacheso integratingqualita-tiveandquantitativemethods.Paperpresentedat the annual meeting of the AmericanEdu-cational ResearchAssociation, Los Angeles.(ERIC Document ReproductionServiceNo.ED 207 256)

272


7/29/2019 1163620

20/21

TowardA Conceptual rameworkMadey,D. L. (1982).Some benefitsof integratingqualitativeand quantitativemethods in pro-gram evaluation, with illustrations.Educa-tional EvaluationandPolicyAnalysis,4, 223-236.Mark, M. M. (1988, October).Discussant re-markson "Toward theoryof mixed-methodevaluationdesigns."Panelpresentedat the an-nual meetingof the AmericanEvaluationAs-sociation,New Orleans.Mark,M. M., & Shotland,R. L. (1987). Alter-nativemodelsforthe use of multiplemethods.In M. M. Mark& R. L. Shotland Eds.).Mul-tiplemethods n programevaluation:New Di-rections or ProgramEvaluation35 (pp. 95-100).SanFrancisco: ossey-Bass.Martin,G. 0. (1987, October).The use of quali-tativeand quantitative valuationmethods orprogramevaluation. The CaliforniaInterna-tionalStudiesProject.A casestudy.Paperpre-sented at the annualmeetingof the AmericanEvaluationAssociation,Boston.Mathison, S. (1988). Why triangulate?Educa-tionalResearcher, 7(2), 13-17.Maxwell,J. A., Bashook,P. G., &Sandlow,C. J.(1986). Combiningethnographicand experi-mentalmethodsin educationalevaluation.In

D. M. Fetterman& M. A. Pitman(Eds.),Ed-ucationalevaluation:Ethnographyn theory,practice,and politics (pp. 121-143). BeverlyHills,CA:Sage.Miles,M. B., & Huberman,A. M. (1984). Qual-itative data analysis: A sourcebookof newmethods.BeverlyHills,CA:Sage.Moody, P. R. (1982). PittMeadowsphysicaled-ucationproject:Evaluationof an implementa-tion project(Report No. 82:13). Vancouver:EducationalResearchInstituteof BritishCo-lumbia.(ERICDocumentReproductionServ-ice No. ED 222 564)Moran,T. K. (1987). Researchand managerialstrategiesfor integratingevaluation researchinto agencydecision making.EvaluationRe-view,11, 612-630.Peters,M., Marshall, ., & Shaw,R. (1986). Thedevelopmentand trials of a decision makingmodel:An evaluation.EvaluationReview,10,5-27.Reichardt,C. S., & Cook, T. D. (1979). Beyondqualitativeversusquantitativemethods. In T.D. Cook & C. S. Reichardt Eds.),Qualitativeand quantitativemethods in evaluation re-search pp.7-32). BeverlyHills,CA:Sage.Reichardt,C. S., & Gollob,H. F. (1987). Takinguncertainty nto account when estimatingef-fects. In M. M. Mark& R. L. Shotland Eds.),

Multiplemethods n programevaluation:NewDirections or ProgramEvaluation35 (pp. 7-22). San Francisco: ossey-Bass.Rossman,G. B., & Wilson,B. L. (1985). Num-bers and words:Combiningquantitativeandqualitativemethods n a single arge-scale val-uationstudy.EvaluationReview,9, 627-643.Shotland,R. L., & Mark,M. M. (1987). Improv-ing inferencesfrom multiplemethods.In M.M. Mark & R. L. Shotland(Eds.). Multiplemethods in programevaluation:New Direc-tionsfor ProgramEvaluation35 (pp. 77-94).San Francisco: ossey-Bass.Sieber,S. D. (1973).Theintegration f fieldworkand surveymethods.AmericanJournalof So-ciology, 78, 1335-1359.Smith, A. G., & Louis, K. S. (Eds.). (1982).Multimethodpolicyresearch:ssuesandappli-cations [Special issue]. AmericanBehavioralScientist,26.Smith, A. G., & Robbins,A. E. (1984). Multi-methodpolicyresearch:A casestudyof struc-tureand flexibility. n D. M. Fetterman Ed.),Ethnography n educationalevaluation (pp.115-130). BeverlyHills,CA:Sage.Smith,J. K. (1983). Quantitativeversusqualita-tive: An attemptto clarifythe issue. Educa-

tionalResearcher, 2, 6-13.Smith, J. K., & Heshusius,L. (1986). Closingdown the conversation:The end of the quan-titative-qualitativedebate. Educational Re-searcher,15(1), 4-12.Smith,M. L. (1986).Thewhole is greater:Com-bining qualitativeandquantitative pproachesin evaluationstudies.In D. D. Williams Ed.),Naturalistic evaluation: New DirectionsforProgramEvaluation30 (pp.37-54). SanFran-cisco:Jossey-Bass.Trend, M. J. (1979). On the reconciliationofqualitativeand quantitativeanalyses:A casestudy.In T. D. Cook &C. S. Reichardt Eds.),Qualitative nd quantitativemethods n evalu-ation research pp. 68-86). BeverlyHills, CA:Sage.Webb,E. J., Campbell,D. T., Schwartz,R. D.,& Sechrest,L. (1966). Unobtrusivemeasures(chap.1),Chicago:RandMcNally.

AuthorsJENNIFERC. GREENE,Associate Professor,Human Service Studies, Cornell University,N136B MarthaVan RensselearHall, Ithaca,NY 14853. Specializations:programevalua-tion, appliedsocialresearchmethodology.273


7/29/2019 1163620

21/21

Greene,Caracelli,andGrahamVALERIEJ. CARACELLI, ocialScience Ana-lyst,GeneralGovernmentDivision,U.S. Gen-eralAccountingOffice,441 G St., N.W.,Wash-ington, DC 20548. Specializations: mixed-methods evaluationresearch;adolescent andadultdevelopment.

WENDY F. GRAHAM, Director of Planningand InformationSystems,CornellUniversity,Collegeof HumanEcology,MarthaVan Ren-selearHall, Ithaca,NY 14853.Specializations:programevaluation, educationaladministra-tion, surveyresearchmethods.

274

1163620

Documents