construct driven assessment design: nclt reality or lake michigan pipe dream?

Download Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Post on 14-Jan-2016




0 download

Embed Size (px)


Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?. Jim Pellegrino, Joe Krajcik Shawn Stevens & Namsoo Shin. Overview of the Next Two Days. Why we’re here & what we’’ll do Construct-driven assessment - an introduction Activity 1: Unpacking the “Construct” - PowerPoint PPT Presentation


  • Construct Driven Assessment Design: NCLT Reality orLake Michigan Pipe Dream?

    Jim Pellegrino, Joe Krajcik Shawn Stevens & Namsoo Shin

  • Overview of the Next Two DaysWhy were here & what well doConstruct-driven assessment - an introductionActivity 1: Unpacking the ConstructActivity 2: Translation into Claims & EvidenceReporting out of progressActivity 3: Creating an Assessment BlueprintReporting out of progressWrapup & Next StepsTasks and toolsFinal thoughts

  • The NeedTherefore, we need a consistent, principled way for developing assessmentsDeveloping good assessments for evaluating student understandingis critical for all aspects of Work in the NCLT

  • Examples of assessmentas a driver of NCLT WorkConcepts & Concept Inventories using assessments as diagnostic aids for instructors

    Learning Progression assessing student knowledge to understand how students knowledge builds over time develop assessments from learning progressions

    Curriculum Development how do you know that your curriculum materials, technology tool, etc. was successful? how do you provide feedback for learners?

    Professional Development how do we assess teacher learning? how do we provide feedback for learners?

  • Workshop Activities & ObjectivesFor a few of the big ideas in nanoscience and nanotechnology:

    Unpack each one to define the construct to be assessed and the relevant claim space about student knowledge and understanding

    Decide what would serve as appropriate evidence that a student has the desired knowledge.

    Design an assessment blueprint that includes the range of tasks, questions and/or situations needed to provide the types of evidence you want and need for a particular assessment purpose.

  • Layers in the assessment enterprise What is important about this domain? What work and situations are central in this domain? What KSAs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Design structures: domain model (claims), evidence, and task models How do we choose and present tasks, and gather and analyze responses? How do students and tasks actually interact? How do we report examinee performance?

  • Construct-Driven AssessmentWhat complex of knowledge, skills, or other attributes should be assessed?

    What behaviors or performances should reveal those constructs?

    What tasks or situations should elicit those behaviors? (Messick, 1994)

  • Assessment as a Process of Reasoning from Evidencecognition model of how students represent knowledge & develop competence in the domain

    observationstasks or situations that allow one to observe students performance

    interpretationmethod for making sense of the dataobservationinterpretationcognitionMust be coordinated!

  • Scientific Foundationsof Educational AssessmentAdvances in the Sciences of Thinking and Learning -- the cognition vertexinforms us about what observations are important and sensible to make

    Contributions of Measurement and Statistical Modeling -- the interpretation vertexInforms us about how to make sense of the observations we have made

  • An Example: Wilsons Approach

  • Building Block 1: Construct MapDevelopmental perspectiveassessment system should be based on a developmental perspective of student learningProgress variable as a representationVisual metaphor for how the students develop knowledge & understanding and how we think about how their item responses might change

  • Example: Why things sink and floatExample provided by:

  • Building Block 2: Items designInstruction & assessment matchthere must be a match between what is taught and what is assessedItems design a set of principles that allows one to observe the students under a set of standard conditions that span the intended range of the item contextsExample-Please answer the following question. Write as much information as you need to explain your answer. Use evidence, examples and what you have learned to support your explanations.

    Why do things sink and float?

  • Building Block 3: Outcome spaceManagement by teachersthat teachers must be the managers of the system, and hence must have the tools to use it efficiently and use the assessment data effectively and appropriatelyOutcome spaceCategories of student responses must make sense to teachers

  • Example: Why things sink and float


    What the Student Knows


    Relative Density




    Mass and Volume






    Productive Misconception


    Unconventional Feature


    Off Target


    No Response

  • Building Block 4: Measurement modelEvidence of qualityreliability and validity evidence, evidence for fairnessMeasurement modelmultidimensional item response models, to provide links over time both longitudinally within cohorts and across cohorts

  • Example: Evaluate progress of a group


  • Evaluate a students locations over timeEmbedded Assessments

  • Why Models of the Development of Domain Knowledge are Critical Tell us what are the important aspects of knowledge that we should be assessing.Give deeper meaning and specificity to standardsGive us strong clues as to how such knowledge can be assessedSuggest what can and should be assessed at points proximal or distal to instructionCan lead to assessments that yield more instructionally useful information -- diagnostic & prescriptiveCan guide the development of systems of assessments -- work across levels & time

  • Integrated Knowledge Course GoalMultipart Framework for the Domain AnalysisAP Redesign Process Design

  • ModelsSystemsContinuity and ChangeScaleStructure/FunctionScience Explains the Real WorldUnifying ConceptsScientific Inquiry and ReasoningAsking testable questions, drawing conclusions based on evidence, and generating useful representationsApplying the scientific way of knowing through reasoning based on evidence.Organizing and communicating ideasExperimental design, execution and data analysisIntegrated KnowledgeCourse GoalChemistry Domain AnalysisMatter is made from discrete, fundamental units called atoms.Chemical and physical properties of materials can be explained by the structure and the arrangement of atoms, ions or molecules and the forces between them.Changes in matter involve the rearrangement and/or reorganization of atoms and/or the transfer of electrons.The laws of thermodynamics explain and predict the direction of changes in matter.Any bond or intermolecular attraction that can be formed can be broken. These two processes are in a dynamic competition, sensitive to initial conditions and external perturbations.Big Ideas of ChemistryRates of chemical reactions are determined by details of the molecular collisions.Using symbolic and graphical representations of relationshipsGoals: Traditional emphasis on calculation and description shifts to an emphasis on the underlying concepts and the reasoning from which they emerge. Traditional emphasis on teacher-directed procedures is an early scaffold that supports subsequent experiences involving guided inquiry.

  • Chemistry Levels 1 & 2

    Big Idea 1: Matter is made from discrete, fundamental units called atoms.

    1.A Everything is comprised of a limited number of immutable building materials called atoms.

    1.B The atomic theory of matter describes elements as comprised from atoms with unique structures.

    1.C Elements have unique properties that exhibit periodic patterns

    1.D Atoms can be changed in nuclear processes

    1.E Atoms are so small that they cannot be observed directly but can be inferred from experimental data.

    1.F Elements combine chemically to form pure substances called compounds.

    1.G Atoms are conserved in physical and chemical processes

    1.H Atoms interact via their outer shell electrons.

    Big Idea 2: Chemical and physical properties of materials can be explained by the structure and the arrangement of atoms, ions or molecules and the forces between them.

    2.A A portion of matter with the same chemical and physical properties is called a phase or state of matter.

    2.B Forces of attraction between noble gas atoms and covalent molecules determine how the physical state of the molecular substance changes with temperature.

    2.C The strong electrostatic forces of attraction holding atoms together in a unitare called chemical bonds.

    2.D The type of bonding can be deduced from the properties of the solid state.

    2.E The reactivity of a material depends on the relative bond strengths of the products and reactants.

    Big Idea 3: Changes in matter involve the rearrangement and/or reorganization of atoms and/or the transfer of electrons

    3.A. Chemical changes are represented by a balanced chemical reaction that identifies the ratios with which reactants react and products form.

    3.B Chemical changes can be classified by considering what the reactants are or what the products are, or how they change from one into the other, such as precipitation, acid-base and oxidation-reduction reactions.

    3.C. Chemical change involves the breaking of ionic, covalent or metallic bonds.

  • Chemistry Levels 1 & 2 (cont)

    Big Idea 4: Rates of chemical reactions are determined by details of the molecular collisions.

    4.A Reaction rates are temperature dependent and are determined by measuring changes in concentrations of products or reactants over time.

    4.B A reaction proceeds along a specific pathway, with at least one transition stat