Working Together and Seeking Generalized Results: Secondary Analyses and Syntheses*

Larry D. YoreUniversity of VictoriaTaipei, Taiwan March 6, 2008

(*see Rossman & Yore, chapter in progress, Springer)

2

Key IdeasBased on medical and health sciences research and some work done in the social sciences.Clear Definitions of Target GoalsAssessment of Alignment and Selection CriteriaQuilting as Quantitative Secondary Analyses

Meta-analysesSecondary Reanalyses

Crazy Quilting as Qualitative SynthesesSystematic ReviewsSecondary ReanalysesCase-to-case ComparisonsMetasyntheses

3

Looking Outside of Science Education (also see Yore & Boscolo, in press)

NCLB and Gold Standard for Educational Research

Partially based on the public’s disappointment with education research to address pressing issuesPartially based on the skepticism that politicians and bureaucrats have toward education research

Advances in seeking and finding generalizations in medicine, health care, and social sciences

4

Clarifying Targets

Mathematics, Scientific, or Technological LiteracyNature of Mathematics, Science, and TechnologyLanguage in Doing, Learning, and Teaching ScienceLearning Outcomes, Measures, and Methods

5

Interacting Senses of Mathematical Literacy — Cognitive Symbiosis (Yore, Pimm, & Tuan, 2007)

Fundamental Sense Derived Sense

Cognitive and Metacognitive Abilities

Understanding the Big Ideas, Strands, and Substrands of Mathematics

Mathematical Thinking and Quantitative Reasoning

Nature of Mathematics

Emotional Dispositions/ Habits of Mind

Knowledge about Problem Solving

Language of Mathematics (including proofs as arguments)

Real-world Problems

Information Communication Technologies (ICT)

6

Interacting Senses of Scientific Literacy — Cognitive Symbiosis (Yore, Pimm, & Tuan, 2007)

Fundamental Sense Derived Sense

Cognitive and Metacognitive Abilities

Understanding the Big Ideas and Unifying Concepts of Science

Critical Thinking/Plausible Reasoning

Nature of Science

Habits of Mind Scientific Inquiry

Scientific Language (including mathematical language)

Technological Design

Information Communication Technologies (ICT)

Relationships among Science, Technology, Society, and Environment (STSE)

7

Interacting Senses of Technological Literacy — Cognitive Symbiosis (Pacific Crystal Group, in progress)

Fundamental Sense Derived Sense

Cognitive and Metacognitive Abilities

Understanding the Big Ideas of Technology: Design Principles, Real-world Problems, and Constraints

Critical and Creative Thinking

Nature of Technology

Emotional Dispositions/ Habits of Mind

Knowledge about Research and Development, Design, and Creative Problem Solving

Language of Technology Technology and Engineering in Daily Life and Democratic Society

Information Communication Technologies (ICT)

8

Modern View of Science: Naïve Realist Ontology and Evaluativist Epistemology(Yore, Hand, & Florence, 2004)

Science knowledge is a temporary explanation that best fits the existing evidence, established knowledge, and current thinking about reality as we know it.

9

Modern View of Science (continued)

Science knowledge claims are open to repeated public evaluation

Language is not a exact transcription of scientific inquiry — Science reports are not records of actual actionsLanguage shapes as well as reports science ideas — Scientists use metaphors to capture their mental images, and the metaphor starts to influence their thinkingLanguage must reflect scientific inquiry and the tentative and temporary nature of claims — Science does not ‘prove’; it only rejects or supports hypotheses

10

Language of ScienceThe 3–Language Problem: Home language (L1), school language (L2), and science language (L3) frequently do not match(Yore & Treagust, 2006)

Metalanguage of Science and scientific epistemology and ontology: Theory, hedges, physical causality, explanations, etc.Instruction must start with the student’s home language and move toward the languages of instruction and science

11

Alignment of Outcomes and Methods

Learning OutcomesMeasures: Items, questions, and tasks

CommonUnique

MethodsInterviewsWork samples, artifacts, etc.Observations

12

Learning Matrix (Shymansky, n.d.)

Meaningful

Level ofLearning

Rote Drill Argument

Process of Learning

13

Oral Language that Promotes Cognitive Symbiosis

Learning to talk/argue and talking/arguing to learnLittle meaningful oral discourse occurs in most science classrooms:

Most oral discourse in classrooms does not reflect scientific discourse and metalanguageIt is social, not focused and purposeful

14

Talking Science

Patterns of Verbal Interactions(Flanders, 1964; Shymansky, 1978)

Traditional Science Lesson• One-way: lecture• Two-way and unidirectional: teacher to

student (t-s)

Inquiry Science Lesson• Two-way and multi-directional: t-s, s-t, s-s

15

Classroom QuestioningTeacher questioning needs to reflect the phase and purpose of inquiry

Wait-time: 3 seconds among question, response, and further questions (Rowe, 1996)

Use specific types of questions for specific purposesChained Questions: Response and rationale

Debating Science, Technology, Society, and Environment IssuesArgumentation: The process of argument

16

Producing and Interpreting Science Text

Writing to readWrite and represent ideas first (knowledge construction), reading what is written (making sense of text) in early grades

Writing-to-learn and Reading-to-learn Science

Science Writing HeuristicVisual Representations and Transformation between Modes of RepresentationMetacognitive Awareness and Executive Control of Science Reading and Making Sense of Text

17

Debates and Arguments Involving Science, Technology, Society, and Environment Issues

STSE issues provide ill-structured problems, multiple solutions, and rich contemporary contexts involving trade-offs among science, technology, and societal valuesStructured Controversy: Apply authentic debating procedures (Johnson & Johnson, 1985)

King’s College London Project (Osborne, Erduran, & Simon, 2004; Simon, Erduran, & Osborne, 2006)

18

Structured Controversy(Johnson & Johnson, 1985)

DebateEvaluateChange Position and Debate AgainDraw Consensus and Plan Action

19

Scientific Arguments(Osborne, Erduran, & Simon, 2004)

Elements of ArgumentationClaimsEvidenceWarrantsBackingsCounter-claimsQualificationsRebuttals

20

Extended Pattern of Argumentation (Toulmin, 1958)

Evidence Qualifiers and Claims Counter-claims

Warrants Rebuttal

Backings

21

Example of an Extended Argument

Examination of:AIDS and HIV in HIVhealthy some causespatients people AIDS

HIV was found Peoplein all AIDS with weakpatients and some immunehealthy patients systems

22

Research into Language and Science Understanding (Tippett, in press)

Links between argumentation ability, quality of arguments, and science understandingUse of software scaffolds for improving argumentation and learningInterpretative frameworks and scoring rubrics to evaluate arguments and explore their effects

23

Developing Research Results and Reports to Persuade Politicians, Policy Makers, and Decision Makers

Academic demands of universities and funding agencies appears to influence research approaches toward short ‘one-shot’ inquiriesPopular beliefs that learning and teaching is situated and total unique has encouraged case studies and contextual focused studiesPoliticians and bureaucrats view these studies as not applying to their problems and constituents

24

Selection CriteriaAvailable studies focus on similar research questions, outcomes, measures, methods, etc.

Look for common tests, items, interview questions, performance tasks, instruction artifacts.

Collect published and unpublished reports of studies from journals, ERIC, theses, dissertations, client reports, etc.

Screening for quality should not be left to publication status.Follow-up via email with authors to clarify quality issues and elaborations of results.

25

Quantitative Secondary Analyses as Quilting

Cat Patternhttp://www.nmia.com/~mgdesign/

26

Quantitative Meta-analysis (Shymansky, Hedges, & Woodworth, 1990)

Summarization of results: Differential effects between control-treatment studies producing an average effectSummation of individual standardized differences in experiment-control designs: Unweighted or weighted sum of difference in gains/standard deviation (1 to n) divided by the number of studies (n)

Effect size #1 + Effect size #2 + … + Effect size #n divided by number of studies (n) orN1 x ES #1 + … + Nn x ES #n / N1 + … + Nn

27

Quantitative Reanalysis(Gunel, Hand, & Prain, 2007)

Combined datasets for similar individual studies focused on same outcome, using similar designs, or anchored measuresStandardize outcome measuresDevelop a collective dataset of the standardized dataAnalyze the combined dataset with combined sample size

28

Qualitative Summarization as Crazy Quilting

http://www.nmia.com/~mgdesign/qor/styles/crazy/crzayqlt.htm

29

Qualitative Systematic Review

Value AddedSelection CriteriaCritical Summary and Interpretation

New developmentsTheory and theoretical frameworkFuture lines of research

Summary is recursive, grounded in the data and interpretative framework

30

Case-to-case ComparisonAnalytical Generalization

Theory-driven predictions systematically verified against the results of several qualitative case studies to demonstrate valid applications and to map the problem space (Academies for Young Scientists)

Case-to-case SynthesisMultiple case studies of co-authoring (Florence & Yore, 2004)

Multiple case study of science teachers’ beliefs, instructional decisions, and implementation of literacy events in science (Dillon, O’Brien, Moje, & Stewart, 1994)

31

Qualitative ReanalysisCombined qualitative information from similar studies focused on same outcome, using similar designs, or anchored measures (Hand & McDermott, 2008)

Develop a collective text file or video file of these qualitative dataAnalyze the combined text or video files with a new or unified interpretative framework or method (discourse analysis, text analysis software, video analysis systems)

32

Qualitative Synthesis (Finfgeld, 2003)

Using the results of qualitative studies on similar issues, using similar data sources and collection techniques, and anchor itemsSystematically combine the results of selected studiesNational Literacy Panel (August & Shanahan, 2006)

33

References (not included in handout)Flanders, N. A. (1964). Some relationships among teacher influence, pupil attitudes, and achievement. In B. J. Biddle & W. J. Ellons (Eds.), Contemporary research on teacher effectiveness (pp. 196-231). New York: Holt, Rinehart & Winston.

Johnson, R. T., & Johnson, D. W. (1985). Using structured controversy in science classrooms. In R. W. Bybee (Ed.), Science technology society: 1985 yearbook of the National Science Teachers Association (pp. 228-234), Washington, DC: National Science Teachers Association.Norris, S. P., & Phillips, L. M. (2003). How literacy in its fundamental sense is central to scientific literacy. Science Education, 87, 224-240.

34

References (continued)

Novak, J. D., & Gowin, B. D. (1984). Learning how to learn. Cambridge, UK: Cambridge University Press.Osborne, J., Erduran, S., & Simon, S. (2004). Enhancing the quality of argumentation in school science, Journal of Research in Science Teaching, 41, 994-1020.Rowe, M. B. (1996). Science, silence, and sanctions. Science and Children, 34(1), 35-38.

35

References (continued)

Shymansky, J. A. (1978). Assessing teacher performance in the classroom: Pattern analysis applied to interaction data. Studies in Education Evaluation, 4, 99-106.

Toulmin, S. (1958). The uses of argument. Cambridge, UK: Cambridge University Press.Wellington, J., & Osborne, J. (2001). Language and literacy in science education. Philadelphia, PA: Open University Press.

36

References (continued)

Tippett, C. (in press). Argumentation: The language of science. Journal of Elementary Science Education.Rossman, G.B. & Yore, L.D. (in progress). Stitching the pieces together to reveal the generalized patterns: Systematic research reviews, secondary reanalyses, case-to-case comparisons, and metasyntheses of qualitative research studies. In M. Shelley, B. Hand, & L.D. Yore (Eds.). Education Research Meets the “Gold Standard”: Evaluation, Research Methods, and Statistics after No Child Left Behind (pp. tba), Springer.

37

References (continued)

Yore, L.D. & Boscolo, P. (in progress). Why “Gold Standard” Needs Another “s”: Results from the Gold Standard(s) in Science and Literacy Education Research Conference. In M. Shelley, B. Hand, & L.D. Yore (Eds.). Education Research Meets the “Gold Standard”: Evaluation, Research Methods, and Statistics after No Child Left Behind (pp. tba), Springer.Yore, L.D. & Lerman, S. (in press). A call for systematic reviews, secondary analyses, generalizations, and meta-syntheses. International Journal of Science and Mathematics Education.

38

References (continued)

Yore, L.D., Hand, B.M., & Florence, M.L. (2004). Scientists’ views of science, models of writing, and science writing practice. Journal of Research in Science Teaching, 41, 338-369.Yore, L.D. & Treagust, D.F. (2006). Current realities and future possibilities: Language and science literacy--empowering research and informing instruction. International Journal of Science Education, 28, 291-314.Yore, L.D., Pimm, D., & Tuan, H-L. (2007). The literacy component of mathematical and scientific literacy. International Journal of Science and Mathematics Education, 5, 559-589.