shavelson, richard j. et al_ on the science os education design studies.pdf

8/12/2019 SHAVELSON, Richard J. et al_ On the science os education design studies.pdf

1/5

On the Science of Education Design StudiesAuthor(s): Richard J. Shavelson, D. C. Phillips, Lisa Towne, Michael J. FeuerSource: Educational Researcher, Vol. 32, No. 1, Theme Issue: The Role of Design in EducationalResearch, (Jan. - Feb., 2003), pp. 25-28Published by: American Educational Research AssociationStable URL: http://www.jstor.org/stable/3699932Accessed: 19/05/2008 18:32

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at

http://www.jstor.org/page/info/about/policies/terms.jsp . JSTOR's Terms and Conditions of Use provides, in part, that unless

you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you

may use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at

http://www.jstor.org/action/showPublisher?publisherCode=aera .

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed

page of such transmission.

JSTOR is a not-for-profit organization founded in 1995 to build trusted digital archives for scholarship. We enable the

scholarly community to preserve their work and the materials they rely upon, and to build a common research platform that

promotes the discovery and use of these resources. For more information about JSTOR, please contact [email protected].
http://www.jstor.org/stable/3699932?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/action/showPublisher?publisherCode=aerahttp://www.jstor.org/action/showPublisher?publisherCode=aerahttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/3699932?origin=JSTOR-pdf


2/5


3/5

terms the Committee concluded hat (a) design studies are cre-ative research ndeavors hat hold promise especially or devel-

oping and elaborating conjectures and addressing ssues ofresearch-based ractice, nd (b) principles f research hat applyto all scientific work must apply also to knowledge laims rom

design tudies. Our position echoes Steffe and Thompson 2000,p. 277), who underscored he scientific ntent of their design-study work: We use experiment n 'teaching experiment' n ascientific ense .... What is important s that the teaching xper-iments are done to test hypotheses s well as to generate hem.

In this article we focus on the scientific tatus of the conclu-sions reached ia design tudies. To this end, we review he Com-mittee's guiding principles or scientific research with emphasison their applicability o design studies, as we understand designstudies. Our main goal is to scrutinize he knowledge claimsfrom design studies through he lens of the guiding principles.We next take up the key epistemological ssue-that of warrantsfor knowledge. At issue s how we come to know about the ef-fects of educational nvironments n students' and researchers')learning and construction of meaning. Finally, we consider an

important practical question about the conduct of design stud-ies, namely whether specific designs are well matched o the re-search questions hey are ntended o elucidate.

Guiding Principles of Scientific Research

The Committee argued hat all the sciences, ncluding cientificeducational esearch, hared a set of epistemological r funda-mental guiding principles. The Committee argued hat all sci-entific endeavors hould

* pose significant uestions hat can be investigated mpirically,* link research o relevant heory,* use methods hat permit direct investigation f the questions,* provide a coherent and explicit chain of reasoning,

* attempt o yield findings hat replicate nd generalize crossstudies, and* disclose research ata and methods o enable and encourage

professional crutiny and critique.These principles o not constitute an algorithm, hecklist, or

how-to guide; no single study is likely to encompass hem all(although well-designed nd coordinated program f scientificresearch ould). Rather, he Committee viewed he principles sprofessionally nternalized orms that reflect dedication o theprimacy of evidence; to healthy skepticism about knowledgeclaims; o ruling out all alternative explanations competitive r-

gumentation ; ee Schoenfeld, Smith, & Arcavi, 1993); to eluci-dating and reducing biases hat might affect he research process;to disciplined, creative, and open-minded hinking; and to thefree flow of constructive riticism.

Although all scientists hare at least implicitly) hese norms,how they are nstantiated n different ieldsof science varies. Eachfield has features hat influence he questions posed and the de-sign, conduct, nterpretation, nd generalization f research. al-ues and democratic deals n schools, volition and diversity ofpeople (teachers, tudents, administrators), ariability nd orga-nization f school processes curriculum, nstruction, overnance)across educational ettings, roles of tools and technologies, andthe need for collaboration ith practitioners nd school commu-nities all shape scientific research n education. These features,

though not individually nique rom other ields, are singular ntheir combination and require lose attention o powerful con-textual actors n the research process. Scholars working n a par-ticular area ollectively-such as the community of design-studyresearchers-establish he scientific raditions nd standards orhow most appropriately o apply he guiding principles o theirarea of study.

On Design Studies and MethodsDesign studies have been characterized, ith varying mphasisdepending on the study, as iterative, process ocused, nterven-tionist, collaborative, multileveled, utility oriented, and theorydriven (e.g., Cobb, Confrey, diSessa, Lehrer, & Schauble, hisissue). Design studies are iterative n that they involve tightlylinked design-analysis-redesign ycles that move toward both

learning and activity or artifact improvement. They are processfocused n that they seek to trace both an individual's or group'sor school system's; ee Cobb et al., this issue) earning by under-

standing successive patterns n the reasoning and thinking dis-played and the impact of instructional rtifacts n that reasoningand learning. They are interventionist n testing theory and in-structional rtifacts y designing and modifying real-world et-

tings. They are collaborative n that hey depend on the knowledgeand co-work of practitioners. hey are often multileveled n thatthey ink classroom ractices o events or structures n the school,district, and community. They are utility orientedwith he intentof improving he effectiveness f instructional ools to supportlearning. And they are theory riven n the sense of testing plac-ing them in harm's way ; f. Cobb et al., this issue) and advanc-ing theory hrough he design-analysis-redesign f instructionalactivities nd artifacts.

Researchers conducting design studies (e.g., Brown, 1992;Collins, 1999) have contrasted heir intense, iterative, deo-

graphic methods with traditional psychological xperimentationthat nvolves ontrol of variables nd focus on causal ffects: De-sign experiments .. attempt o carry xperimentation nto real-life settings n order to find out what works in practice. Thismeans giving up the notion of controlling ariables nd necessi-tates the development of a new methodology to carry out re-search Collins, 1999, p. 290). Such contrasts are helpful in

clarifying he nature of design tudies, but we argue hat n someinstances capitalizing n the respective trengths f design stud-ies and other methods-including traditional experiments-inan integrated, omprehensive esign would enhance nquiries(cf. McCandliss, Kalchman, & Bryant, his issue).

Because of their emersion n complex, multivariable settingsand their ocus on the changes r learning hat takes place n par-ticipants be they students, eachers, r administrators), nd thedevelopment f instructional ctivities nd tools, design-study e-search soften distinguished ythe generation f a comprehensiverecord of the evolving design process. This documentation ypi-cally nvolves xtensive ideo recordings f student engagement,teacher ctivity, nd the like supplemented y other extual mate-rial, periodic nterviews, nd questionnaires. uch documentationserves s the basis or a retrospective nalysis f what happened nthe design study. The central challenge for the retrospective analy-ses is to systematically ork through his extensive longitudinaldata o arrive t and communicate rigorous, mpirically rounded

2| EDUCATIONAL ESEARCHER


4/5

claims and assertions Cobb et al., this issue) and to enhance helikelihood of replicability Hoadley, 2002). The retrospectiveanalysis roduces situated, arrative ccount which sstory-like,including actors, actions, ntentions) of learning nd how it canbe supported and organized. And, to establish generalizability,the narrative analysis laces he study n a broad heoretical on-text to show if and how the study is a paradigmatic ase of thephenomenon under investigation. The narrative ccount, hen,provides a structure or conveying a series of related events, aplot.... Important gents, events, causes and results are relayed... [and thereby] can help make explicit some of the implicitknowledge he designer .. used to understand nd implementthe intervention (Hoadley, p. 2).

Design Studies, the Human Sciences, and Warrantsfor KnowledgeIn recognition f the complex, multivariate, multilevel, terative,and interventionist nature of design studies, those working nthis research mode have argued or intensive, ongitudinal tud-ies that race he design process nd capture meaning onstructedby individual subjects over time. Naturally enough, reports ofthis kind of work usually ake he form of narratives, hether ornot the researchers oncerned recognize his; but in recent yearsthere has been considerable explicit discussion of, and advocacyon behalf of, the use of narrative ccounts. Moreover, many dis-tinctions hat design-study esearchers ake between raditionalexperimental methods and design-study methods (e.g., narrativeaccounts nd interpretive rameworks) eflect he broader ebateamong psychologists nd philosophers f science (e.g., Bruner,1990; Donald, 2001; Toulmin, 2001; Polkinghorne, 988) onpositivist nd postpositivist cience (Phillips & Burbules, 000)and the need for a new epistemology hat meets the needs of

human ciences.

The premise f those who take he narrative urn s that o un-derstand human action, acultural psychology s needed, onethat must venture beyond the conventional aims of positivistscience with ideals of reductionism, ausal explanation nd pre-diction Bruner, 1990, p. xiii). Bruner adds rhetorically, Arenot plausible nterpretations referable o causal explanations,particularly hen the achievement f a causal xplanation orcesus to artificialize sic]what we are studying o a point almost be-yond recognition s representative f life? (p. iii).

The point is that it can be argued hat to understand phe-nomena such as student learning and to document how this de-velops during he course of a design study, t is necessary o takeinto account he desires, beliefs, goals, reasoning processes, nd

so forth of the students over time, and that this is best done inthe form of a narrative. he narrative an also capture he socio-cultural setting n which the learning occurs and which helps toshape he students' behaviors Bruner, 1990, and for some cau-tions, see Olson, 1992).

Design studies, hen, often rely on narrative ccounts as datafor modifying heory and the design of artifacts teratively vertime. They also rely on narrative o communicate heir researchfindings although he use of the narrative ormat s often not ex-plicitly commented upon). The argument for the validity of suchan account ies in the reasonableness f the argument explana-tory ramework ) sjudged by, for example, xpert practitioners

and researchers communities f practice ): We know fromour own experience n telling consequential tories about our-selves hat there s an ineluctably human' ide to making ense(Bruner, 1990, p. 55). To the extent that the narrative vokesscript and scene schema similarly mong hose well steeped n thephenomena under study, the narrative s claimed o be reason-able, credible, and generalizable.

For us the issue s that with so many confounding ariables na design tudy, can the knowledge laims be warranted? ur con-cern s not alleviated y the narrative urn suggested y some the-orists and by some members f the design-study ommunity. TheNRC Committee's guiding principles ead us to certain questions:To what extent can rival narrative ccounts of the same action beruled out? To what extent would another narrator replicate heaccount? To what extent does the narrative generalize o othertimes and places? There s nothing in the use of narrative orm,by itself, that guarantees he veracity of the content of the ac-count or which vitiates he need for the usual epistemic warrantsused n science (see Phillips, 2000). As Olson (1992, p. 30) putit in his review of Bruner's ecent books on mind and narrative,

justbecause doctrine s 'full of comfort' s not an

argumentor

its truth. n concrete erms, o what extent were decisions aboutthe design of instructional rtifacts based on information hatruled out other possible alternatives? o what extent would an-other narrative f the use of the artifact n learning eplicate heone on which decisions were being made? How can it be deter-mined that the narrative being used s complete, or does not mis-represent vents? To what extent would the tool developed none intensively ngineered etting generalize o another setting?

The basis of narrativist knowledge laims-for example, hatpractitioners nd researchers re able to recognize and under-stand the explanatory ramework f the narrative ased on thescript and scene schema the narrative vokes-is provocative.

But, who selects the practitioners nd researchers? We couldimagine a probability ampling basis or making ome knowledgeclaims rom narratives, electing epresentative ractitioners ndresearchers, nd seeing if the narrative vokes the same under-standing of human actions and meaning among them.4 How-ever, we also stress hat understanding n explanatory rameworkis one thing; whether he framework s well warranted r not isquite another matter. Even f practitioners nd researchers agreein their interpretation f the narrative, n what grounds couldthe narrative e said o correspond o what actually ranspired rwould the narrative rise rom the needs and imagination f thestoryteller? or Bruner 1990, p. 44) this is not a problem; nar-rative can be 'real' or 'imaginary' without loss of its power as a

story. For him (but not for us),5 plausibility ather han verifia-bility s the key validity ssue.

There s more at stake here han understanding ctions. Somenarratives will inevitably (perhaps unknowingly) make causalclaims (e.g., we did x and it produced ). But narratives ackingmore raditional ontrols on extraneous ariables ill not be ableto warrant he causal claim. As we have said before, we believethat the principles or scientific esearch apply o design studies.To ensure hat they are scientifically igorous, t is incumbent onthose carrying ut design studies to create a culture of sciencethat ncludes ruling out competing hypotheses, hat fosters skep-ticism about knowledge claims (including heir own), and that

JANUARY/FEBRUARY 003 |


5/5

encourages powerful tests of rival conjectures. We now turn to

question-driven methods that might apply to various phases of

design studies.

Design Studies, Research Questions, and Methods

Our Committee was careful to point out that research questionsshould drive the choice of research methods. The questions driv-

ing design studies probably depend on what stage in the designevolution they are posed. In early stages, in the so-called contextof discovery, open-ended exploration is common to design stud-ies, just as it is in any other branch of science. This wide-rangingexploration turns into systematic descriptions and evolves intowell-formulated questions, creating a context for verification.The design-study question-and, consequently, the researchmethod-most likely depends upon where in the evolution ofthe design it is posed. The Committee identified three genericquestions within which design studies might be appropriategiven their developmental stage and linked various types of re-search designs to each:

* What is happening? This question invites description of var-ious kinds including, for example, properly characterizing a

population of students with a statistical sample; understand-ing the scope and severity of a problem through survey,ethnographic, or case study methods; or identifying changesover time from both an insider's and outsider's perspectiveby case study, interview, ethnographic, and other methods.In design studies, such questions most likely arise at the be-

ginning of the evolutionary process. The rich descriptive in-formation collected in design studies certainly can use both

quantitative and qualitative methods for description, as wellas qualitative narrative accounts based on warranted knowl-

edge claims. These claims, if clearly identified, might thenbe combined with information about context and motiva-tions into a narrative o communicate to practitioners. Such

a narrative might suggest possible explanations and ways forredesigning learning environments or instructional artifacts.

* Is there a systematic efect? This question is about the intentto establish cause and effect. The Committee concludedthat randomized experiments, when feasible, are the best ap-proach to ferreting out cause-and-effect relationships. Webelieve that experiments, quasi-experiments, and causal mod-els have a role to play in design studies. The role of thesemethods might differ depending on where in the design evo-lution they are applied. As well-formulated questions arise,for example, about which alternative activities or changes inan applet are most likely to lead to a desired outcome, asmall, randomized trial might be used within a classroom.

To be sure, experiments should not be brought out willynilly anytime a design decision is made. Rather they shouldbe reserved or choices among important design alternatives.Moreover, experiments are particularly apt at the scaling upstage of a design study. An artifact or program that has beenshown to work locally would be transported to and testedin other locales. The use of experimental studies combinedwith case (and other) studies of implementation seems ap-propriate to test the generalizability (and limits) of effects.

* Why or how is it happening? This question seeks a causal agentor mechanism: How does x cause y? It might be asked oncea systematic effect between x and y has been established, or,

alternatively, underlying theory might drive the question. In

design studies theory often drives the design of activities orartifacts with a tentative causal explanation or mechanism.

Through iterative tryout-redesign-tryout, claims for under-

standing the mechanism are advanced, and the question of

replicability and generalizability then comes into play.

Final Comments

We believe that the wide range of questions posed in educationalresearch calls for a healthy diversity of scientific methods. The

questions and methods may range from pre-science exploration towell-warranted descriptive, causal, and mechanism-driven studies.We believe it is possible for those doing design studies to incor-

porate our guiding principles into their enterprises and, indeed,

many already have. And we believe that coupling scientificallywarranted knowledge and rich contextual information in somenarrative form might lead to increased understanding and use ofscientific research n practice.

NOTES

We thank Maria Araceli Ruiz-Primo and two anonymous reviewers for

their helpful comments on the article. Errors of commission and omis-sion, of course, are ours.

1We use the term design study to refer to this genre of research rec-

ognizing reasons others prefer terms that include experiment. We avoidthe term experiment o as not to confuse it with other writing on exper-iments as randomized trials in social and behavioral science.

2 Design studies focus on the evolution of learning. The learningmight be that of a student, a teacher, or an organization.

3 Although we draw on our experiences with the Committee, this ar-ticle reflects the views of the authors and not other members of theCommittee or the NRC.

4 Such a strategy might satisfy Toulmin's (2001) call for a middle

ground between the conventional view of rationality and his preferencefor reasonableness n research.

5 Or for Olson (1992, p. 30).

AUTHORS

RICHARD J. SHAVELSON is a professor at Stanford University, Schoolof Education, 485 Lasuen Mall, Stanford, CA 94305-3096; [email protected]. His research interests include social science researchmethods and science teaching, learning, and assessment.

D. C. PHILLIPS s a professor of education and (by courtesy) philoso-phy at Stanford University, 485 Lasuen Mall, Stanford, CA 94302-3096; [email protected]. His research areas of interest include

philosophical and methodological issues in social science and educa-tional research and constructivist positions in psychology, sociology,and science (including science education).

LISA TOWNE is a senior program officer at the Center for Educationof the National Research Council of the National Academies, 500 5thStreet, NW, Washington, D.C. 20001; [email protected]. Her current in-terests include the conduct, use, and infrastructure of education research.

MICHAEL J. FEUER s the executive director of the Division of Behav-ioral and Social Sciences and Education at the National Research Coun-cil of the National Academies, 500 5th Street, NW, Washington, D.C.

20001; [email protected]. His research interests include public policy,social science theory and practice and education.

Manuscript eceived June 3, 2002Revisions received November 6, 2002

Accepted November 7, 2002

2i EDUCATIONAL ESEARCHER

shavelson, richard j. et al_ on the science os education design studies.pdf

Documents