Transcript
Page 1: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Construct Driven Assessment Design:

NCLT Reality orLake Michigan Pipe

Dream?

Jim Pellegrino, Joe Krajcik

Shawn Stevens & Namsoo Shin

Page 2: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Overview of the Next Two Days

• Why we’re here & what we’’ll do• Construct-driven assessment - an introduction• Activity 1: Unpacking the “Construct”• Activity 2: Translation into Claims & Evidence

– Reporting out of progress

• Activity 3: Creating an Assessment Blueprint– Reporting out of progress

• Wrapup & Next Steps– Tasks and tools– Final thoughts

Page 3: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

The “Need”

ConceptInventory

ProfessionalDevelopment

Self-Assembly

LearningProgressionNanoconcepts

Therefore, we need a consistent, principled way for developing assessments

Developing good assessments for evaluating student understandingis critical for all aspects of Work in the NCLT

Page 4: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Examples of assessmentas a driver of NCLT Work

Concepts & Concept Inventories• using assessments as diagnostic aids for instructors

Learning Progression• assessing student knowledge to understand how students’ knowledge builds over time• develop assessments from learning progressions

Curriculum Development• how do you know that your curriculum materials, technology tool, etc. was successful?• how do you provide feedback for learners?

Professional Development• how do we assess teacher learning?• how do we provide feedback for learners?

Page 5: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Workshop Activities & Objectives

For a few of the “big ideas” in nanoscience and nanotechnology:

• Unpack each one to define the “construct” to be assessed and the relevant “claim space” about student knowledge and understanding

• Decide what would serve as appropriate “evidence” that a student has the desired knowledge.

• Design an “assessment blueprint” that includes the range of tasks, questions and/or situations needed to provide the types of evidence you want and need for a particular assessment purpose.

Page 6: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Layers in the assessment enterprise

Assessment Delivery

• What is important about this domain?• What work and situations are central in this domain?• What KSAs are central to this domain?

• How do we represent key aspects of the domain in terms of assessment argument. • Design structures: domain model (claims), evidence, and task models

• How do we choose and present tasks, and gather and analyze responses?

• How do students and tasks actually interact? • How do we report examinee performance?

Assessment Implementation

Conceptual Assessment Framework

Unpacking the domain

Page 7: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Construct-Driven Assessment

What complex of knowledge, skills, or other attributes should be assessed?

What behaviors or performances should reveal those constructs?

What tasks or situations should elicit those behaviors?

(Messick, 1994)

Page 8: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?
Page 9: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Assessment as a Process of Reasoning from Evidence

• cognition – model of how students

represent knowledge & develop competence in the domain

• observations– tasks or situations that allow

one to observe students’ performance

• interpretation– method for making sense of

the data

observation interpretation

cognition

Must be coordinated!

Page 10: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Scientific Foundationsof Educational Assessment

• Advances in the Sciences of Thinking and Learning -- the cognition vertex– informs us about what observations are

important and sensible to make

• Contributions of Measurement and Statistical Modeling -- the interpretation vertex– Informs us about how to make sense of the

observations we have made

Page 11: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?
Page 12: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

An Example: Wilson’s Approach

MeasurementModel

(Interpretation)

ItemScores

(Interpretation)

ItemsDesign

(Observation)

ConstructMap

(Cognition)

Page 13: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Building Block 1: Construct Map

• Developmental perspective– assessment system should be based on a

developmental perspective of student learning

• Progress variable as a representation– Visual metaphor for

• how the students develop knowledge & understanding and • how we think about how their item responses might change

Page 14: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Example: Why things sink and float

Levels of Understanding

Buoyancy depends on thedensity of the object relative

to the density of the medium.

Lessons

12: Relative Density

Assessment Activities

Reflective Lesson @10

Reflective Lesson @7

Reflective Lesson @6

Post test

Buoyancy depends on thedensity of the object.

11: Density of Medium

10: Density of Object

Buoyancy depends on themass and volume of the object.

7: Mass and Volume

Reflective Lesson @11

Buoyancy depends on thevolume of the object.

Buoyancy depends on themass of the object.

6: Volume

Reflective Lesson @4

Pretest

4: Mass

1: Introduction

Example provided by:

Page 15: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Building Block 2: Items design

• Instruction & assessment match– there must be a match between what is taught and

what is assessed

• Items design

– a set of principles that allows one to observe the students under a set of standard conditions that span the intended range of the item contexts

Example-Please answer the following question. Write as much information as you need to explain your answer. Use evidence, examples and what you have learned to support your explanations.

Why do things sink and float?

Page 16: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Building Block 3: Outcome space

• Management by teachers– that teachers must be the managers of the

system, and hence must have the tools to use it efficiently and use the assessment data effectively and appropriately

• Outcome space– Categories of student responses must make

sense to teachers

Page 17: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Example: Why things sink and float

Level What the Student Knows

RD Relative Density

D Densi ty

MV Mass and Volume

M V Mass Vo lume

PM Pro ductiv e Mis co ncep tio n

UF Unc o nven tio nal Featu re

OT Off Target

NR No R e sp o nse

Page 18: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Building Block 4: Measurement model

• Evidence of quality– reliability and validity evidence, evidence for

fairness

• Measurement model– multidimensional item response models, to

provide links over time both longitudinally within cohorts and across cohorts

Page 19: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Example: Evaluate progress of a group

OT UF PM M V MV D RD

Page 20: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Evaluate a student’s locations over time

Embedded Assessments

Page 21: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Why Models of the Development of Domain Knowledge are Critical

• Tell us what are the important aspects of knowledge that we should be assessing.– Give deeper meaning and specificity to standards

• Give us strong clues as to how such knowledge can be assessed– Suggest what can and should be assessed at points

proximal or distal to instruction

• Can lead to assessments that yield more instructionally useful information -- diagnostic & prescriptive

• Can guide the development of systems of assessments -- work across levels & time

Page 22: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Page 23: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Unifying Concepts

Big Ideas ofXXXXX

Scientific Inquiryand Reasoning

Integrated Knowledge

Scientific Investigation

Technology Design

Problem Solving

Critical Thinking

Models

Continuity and Change

Systems

Scale

Application to the Real World

Science ExplainsThe Real World

Structure/Function

Course Goal

Multipart Framework for the Domain Analysis

AP RedesignProcess Design

Page 24: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Models

Systems

Continuity and Change

Scale

Structure/Function

Science Explains the Real World

Unifying Concepts

Scientific Inquiry and Reasoning

Asking testable questions, drawing

conclusions based on evidence, and

generating useful representations

Applying the scientific way of knowing

through reasoning based on evidence.

Organizing and communicating ideasExperimental design, execution and data

analysisIntegrated Knowledge

Course Goal

Chemistry Domain Analysis

Matter is made from discrete, fundamental units

called atoms.

Chemical and physical properties of materials can be explained by the structure and the arrangement of atoms, ions or molecules and the

forces between them.

Changes in matter involve the rearrangement and/or

reorganization of atoms and/or the transfer of electrons.

The laws of thermodynamics explain and predict the direction of changes in

matter.

Any bond or intermolecular attraction that can be formed

can be broken. These two processes are in a dynamic

competition, sensitive to initial conditions and external

perturbations.

Big Ideas of Chemistry

Rates of chemical reactions are determined by details of

the molecular collisions.

Using symbolic and graphical

representations of relationships

Goals:

Traditional emphasis on calculation and description shifts to an emphasis on the underlying concepts and the reasoning from which they emerge.

Traditional emphasis on teacher-directed procedures is an early scaffold that supports subsequent experiences involving guided inquiry.

Page 25: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Big Idea 1: Matter is made from discrete, fundamental units called atoms. 1.A Everything is comprised of a limited number of immutable building materials called atoms. 1.B The atomic theory of matter describes elements as comprised from atoms with unique structures. 1.C Elements have unique properties that exhibit periodic patterns 1.D Atoms can be changed in nuclear processes 1.E Atoms are so small that they cannot be observed directly but can be inferred from experimental data. 1.F Elements combine chemically to form pure substances called compounds. 1.G Atoms are conserved in physical and chemical processes 1.H Atoms interact via their outer shell electrons. Big Idea 2: Chemical and physical properties of materials can be explained by the structure and the arrangement of atoms, ions or molecules and the forces between them. 2.A A por tion of matter with the same chemical and physical properties is called a phase or state of matter. 2.B Forces of attraction between noble gas atoms and covalent molecules determine how the physical state of the

molecular substance changes with temperature. 2.C The strong electrostatic forces of attraction holding atoms together in a unit

are called chemical bonds. 2.D The type of bonding can be deduced from the properties of the solid state. 2.E The reactivity of a material depends on the relative bond strengths of the products and reactants. Big Idea 3: Changes in matter involve the rearrangement and/or reorganization of atoms and/or the transfer of electrons 3.A. Chemical changes are represented by a balanced chemical reaction that identifies the ratios with which

reactants react and products form. 3.B Chemical changes can be classified by considering what the reactants are or what the products are, or how they

change from one into the other, such as precipitation, acid-base and oxidation-reduction reactions. 3.C. Chemical change involves the breaking of ionic, covalent or metallic bonds.

Chemistry Levels 1 & 2

Page 26: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Big Idea 4: Rates of chemical reactions are determined by details of the molecular collisions. 4.A Reaction rates are temperature dependent and are determined by measuring changes in concentrations of

products or reactants over time. 4.B A reaction proceeds along a specific pathway, with at least one transition state as well as possible intermediate

states. 4.C For an effective collision to occur, reacting particles need sufficient energy and proper orientation. 4.D. Reaction rates may be increased by the use of a catalyst Big Idea 5. The laws of thermodynamics explain and predict the direction of changes in matter 5.A. Energy is the fundamental currency of all chemical and physical processes. 5.B. Energy is conserved. Energy is neither created nor destroyed but only converted form one form to another. 5.C Breaking bonds requires energy and making bonds releases energy. 5.D. The natural tendency is to transfer heat from the system to the surroundings and/or toward a more disordered

state (dispersed). Big Idea 6: Any bond or intermolecular attraction that can be formed can be broken. These two processes are in a dynamic competition, sensitive to initial conditions and external perturbations. 6.A Equilibrium is a dynamic, reversible state in which rates of opposing processes are equal. 6.B Systems at equilibrium are responsive to external perturbations. The presence of an external perturbation

implies that the system is no longer a closed system. The composition of the system will change. 6.C The magnitude of the difference in Gibbs free energy between reactants and products can be used to predict the

equilibrium constant or cell potential, which can be used to predict product yield.

Chemistry Levels 1 & 2 (cont)

Page 27: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Models

Systems

Continuity and Change

Scale

Structure/Function

Science Explains the Real World

Objects and systems have properties such as mass,

charge, and internal structure.

Fields existing in space can be used to explain

interactions.

The interactions of one object or system with

another is described by dynamics.

The interactions of one object or system with

another can be described by conservation laws.

Interactions among systems can result in

changes to those systems.

Unifying Concepts

Scientific Inquiry and Reasoning

The evolution of a complex system is determined by the

probability of its configuration.

The interactions of one system with another

system can be mediated by waves.

Integrated Knowledge

Course Goal

Beneath the seven big ideas of content lie 40 enduring understandings whose scope has been defined with 143 level-3 concepts describing what is and is not in the course.

Physics Domain Analysis

Strategic skills that support the

identification and solution of problems

Constructing and interpreting visual and

graphical representations of

relationships

Experimental design

Measurement and interpretation of

measurement

Applying mathematical reasoning

Drawing conclusions based on evidence

Big Ideas of Physics

Page 28: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

1. Objects (anything from a fundamental particle to the universe depending on scale) and Systems (a group of one or more objects at the appropriate scale) have properties such as mass, charge and internal structure.

1.A. The internal structure of an object or system determines other properties of the object or system. 1.B. Electric charge is a property of an object that affects its interactions with other objects containing

charge. 1.C. Objects and systems have properties of inertial mass and gravitational mass that are experimentally

verified to be the same.

1.D. Classical mechanics cannot describe all properties of objects.

1.E. Materials have macroscopic properties which are averages of microscopic properties.

2. Fields existing in space can be used to explain interactions. 2.A. A field is a concept used by physicists in which every point in space has a well-defined value. It is a

useful model for describing forces which act at a distance as well as a wide variety of other physics phenomena.

2.B. A gravitational field is a property of space caused by a massive object. It is the field most familiar to students.

2.C. An electric field is a property of space that is caused by a charged object. 2.D. A magnetic field is a property of space that is caused by a magnet or a moving charged object. The

magnetic field causes a force on a moving charged object that is perpendicular to the direction of motion of the object and to the magnetic field. Magnetic fields observed in nature always seem to be caused by dipoles or combinations of dipoles and never by single poles (monopoles).

2.E. A field line map is a model used by physicists to visualize the properties of a field. 2.F. Physicists often construct a map of "iso" lines (or surfaces) connecting points of equal value for the

quantity related to the field, and use these maps to help visualize fields.

3. The interactions of an object with other objects can be described by forces which can cause a change

of motion .

3.A. All forces have universal properties

3.B. The forces on an object can be visualized using a free body diagram 3.C. If the mass of the object is constant, the motion of an object interacting with other objects is found by

setting the vector sum of the forces equal to the mass times acceleration.

3.D. Certain types of forces are important at the macroscopic level.

3.E. A force on an object transfers momentum to or from that object.

3.F. A force on an object can transfer energy to or from that object.

3.G. Force on an object can cause a torque on that object.

3.H. Certain types of forces are considered fundamental

Physics Levels 1 & 2

Page 29: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

4. Interactions between systems can result in changes in those systems.

4.A. Changes in the motion of the system are caused by a net Force acting on the system, where Fnet = ma.

4.B. Interactions with other objects or systems can change the total linear momentum of a system.

4.C. Interactions with other objects or systems can change the total energy of a system 4.D. A net torque exerted on a system by other objects or systems will change the angular momentum of the

system. 4.E. The electrical and magnetic properties of a system can change in response to the presence of, or

changes in other systems

5. The interaction of one object or system with another is governed by conservation laws. 5A. A qua ntity is conserved if the change of that quantity for a system is equal to the transfer of that

quantity to or from the system by interactions with other systems.

5B. The energy of a system is conserved.

5C. The electric charge of a system is conserved.

5D. The linear momentum of a system is conserved.

5E. The angular momentum of a system is conserved.

5F. The mass of a system is conserved.

6. Some phenomena may be described as waves (or How is a wave different from a particle?) 6.A. A wave can be described by its amplitude, frequency, wavelength, phase difference, speed, and

energy.

6.B. Waves are not physical objects but a means of transferring energy and momentum.

6.C. Only Waves exhibit interference & superposition.

6.D. Matter exhibits wave properties.

6.E. The direction of a light ray can be changed by interaction with a medium

6.F. The electromagnetic spectrum can be modeled as waves or as particles.

6.G. Interference & superposition lead to standing waves & beats

7. The evolution of complex systems over time is described by probability.

7.A. The properties of an ideal gas are determined by probability.

7.B. The tendency of isolated systems to move toward disorder is described by probability.

7.C. The wave properties of matter lead to a probability description for the microscopic world.

Physics Levels 1 & 2 (cont)

Page 30: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

The structure of a single Big Idea with 8 learning objectives

Page 31: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Each learning objective is given meaning with descriptions at one more level of detail

Page 32: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

AP Domain Reporter QuestionsLevel 2

2.1 What prior knowledge, skills, and abilities must students bring to the study of this concept from previous courses?

2.2 What prior knowledge, skills, and abilities must be developed in this course to support understanding of this concept?

2.3 What areas of emerging research should be used to provide an engaging and meaningful context for this concept?

2.4 What misconceptions and preconceptions are students likely to bring to the study of this concept?

2.5 What teaching methods do you see as most successful for the acquisition of the knowledge, skills, and abilities here?

2.6 What minimum resources are needed to support instruction in this concept?

2.7 What must be done to ensure that the redesign increases access of underrepresented students to this concept?

Level 3 3.1 What statement describes this Level 3 concept?

3.2 What would mastery of the concept look like? Include here evidence and representations that would serve as evidence of student mastery of the concept. This should be a short list of good examples.

3.3 How should scientific inquiry and reasoning be applied to the study of this concept? This should include a key term from the scientific inquiry and reasoning categories with an indication of how that term best applies to this concept.

3.4 How does this concept relate to the unifying concepts as they are expressed in your discipline? This should include a key term from the unifying themes with an indication of the nature of the relationship to the concept.

Page 33: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

2.0 What statement describes this second level concept? 2.A Matter can be described by it’s physical properties. The physical properties generally depend on the spacing between the particles (atoms, molecules, ions) and the forces of attraction between the particles.

2.1 What prior knowledge, skills, and abilities must students bring to the study of this concept from

previous courses? Students need to be able to describe each of the states of matter in terms of their volume and shapes

(elementary). Students need to be able to describe the arrangement and spacing of the particles at a molecular level

in each state (middle school). Students need to be able to distinguish compounds, e lements and mixtures (first year chem or even

middle level). Students are familiar with the Periodic Table and the common elements in the first 3 periods (middle

and first year chem). Students must understand that a gas has mass and takes up space (late elementary or middle). Students will have separated mixtures by filtration, use of magnet, and evaporation (middle and elem) Student need to understand that gasses have mass and take up space. Students need to understand air

pressure. They will have some prior experience with study of weather so terms pressure and humidity will be

part of their experience. Like charges repel and unlike charges attract. (middle) Electrons are negative and outside nucleus, protons positive and inside nucleus. Ions form by gaining

or losing electrons. (middle, first year chem.) Student will have learned that energy is ability to do work and have used terms kinetic and potential

energy in previous course work. He probably is only familiar with gravitation potential energy. Students will be familiar with need for heating to change from solid to liquid or liquid to gas. Student will have been asked to draw or interpret heating curves for water and usually will construct

one for BTU or naphthalene in first year chemistry.

Page 34: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

3.1 What statement describes this level 3 concept? 2.A.1 Solids and liquids consist of particles (atoms, molecules, ions) that are held together by attractive forces between the particles. Solid and liquid differ in the orderliness of the arrangement of their particles.

3.2 What would a student say or do to demonstrate understanding at the appropriate depth? Rather than

assessments provide a short list of good examples showing what a student could say or do using this understanding. -The difference in volume is usually very small for solids and liquids at the melting point. The difference in orderliness of the particles (atoms, molecules, ions) between the liquid and solid state is more significant. -The particles in a crystalline solid have a regular 3-D structure with limited motion of the individual particles. The structure of the liquid contains small, particle size holes that enable the particles of the liquid to flow. The student will be able to draw pictures of both the solid and liquid state. Amorphous solids are similar to liquids because they have a random structure with no long-range order but unlike liquids they do not flow. - The boiling point is dependent on the attractions between the particles (molecules, ions, atoms) of the liquid. It is possible to identify that two clear liquids are different by observing their evaporation rate (an example would be water and ethanol) and this can be related to differences in the strength of attractions between the particles of the liquids. Students will be able to distinguish between the types of bonds that will need to be broken for various liquids and to rank these liquids in terms of boiling points and vapor pressure. -Energy must be added to break the forces of attraction between the particles in the solid and liquid. When energy is added at the melting point the temperature of a pure substance remains the same until all the solid has become a liquid. Student can label heating curve and perform calculations for changing temperature and state of a sample of a pure substance. -Student will be able to explain why the heat of fusion for any substance is always much less than the heat of vaporization in terms of the differences in the attractive forces in the solid, liquid, and gas states.

Page 35: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Example: What does it mean to understand the particle model of matter for solids?

- that all matter is made of particles- that in solids, the particles (atoms) are arranged in an ordered compact way- that the particles (atoms) are in constant motion; the degree of motion is dependent on temperature- that the particles (atoms) are the fundamental building blocks of matter- that all particles (atoms) of the same type are the same shape and size- that the arrangement of particles (atoms) determines the substance and can affect its properties

Page 36: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Your First Major Assignment• For your “big idea,” begin to define the construct

– What does it mean to “know & understand” this?– What’s appropriate for students at Level X to understand

(as contrasted with domain experts)– What is the possible prerequisite or co-requisite

knowledge– What are some possible student misconceptions and/or

difficulties

• Use the “big ideas” materials already provided• Try to think in terms of Learning Progressions• Do it verbally, graphically, or however it seems to

make sense for your group.• Be prepared to upload your work to the wiki

Page 37: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Exactly what knowledge do

you want students to have and how do you

want them to know it?

claim spaceOR

construct

evidence task

What task(s) will the

students perform to

communicate their

knowledge? How will you analyze and interpret the evidence?

Evidence-Centered Design

Frase, L.T., Chudorow, M., Almond, R.G., Burstein, J., Kukich, K., Mislevy, R.J., Steinberg, L.S., & Singley, K. (in press). Technology and assessment. In H.F. O'Neil & R. Perez (Eds.), Technology applications in assessment: A learning view.

What will you accept as

evidence that a student has the desired knowledge?

Page 38: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Possible claims about what a student understands about the particle model

The Student understands……- that all matter is made of particles- that in solids, the particles (atoms) are arranged in an ordered compact way- that the particles (atoms) are in constant motion; the degree of motion is dependent on temperature- that the particles (atoms) are the fundamental building blocks of matter- that all particles (atoms) of the same type are the same shape and size- that the arrangement of particles (atoms) determines the substance and can affect its properties

Page 39: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

EvidenceWhat would you accept as appropriate evidence that a student has the desired knowledge?

What does it mean to know that?

What kind of behaviors or performances are necessary for the students to demonstrate that he or she has that knowledge?

To do this you need to use words like:state analyzeexplain applyevaluate model

Page 40: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

AP Translation Process

The following illustrates the construction of a student claim using the claim template.

The student will be able to [reason about]

A field is a concept used by physicists in which every point in space has a well-defined value. It is a useful model for describing forces that act at a distance as well as a wide variety of other physics phenomena.

Within these [contexts] derived from (select the concept(s))

2.A.1. A vector field is a property of space such that the values are vectors. The usual example is a force field; electric and magnetic fields are also vector fields.

2.A.2. A scalar field ascribes magnitudes to every point in space.

Page 41: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Elaboration of a Claim

The student will be able to (you can choose as many as three) Construct and interpret visual representations and Formulate strategies

This claim involves categories of the content that provide a Context: : type of field source, type of field, symmetry of source What is the elaborated claim? Student claim: The student will be able to construct visual representations of a field within the space surrounding a point charge of either sign or a point mass.and state that the direction of the field lines is a convention and state the convention.

Page 42: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

AP RedesignModel of Knowing & Learning

A claim has the general form:

The student is able to <Verb & verb clauses>

the enduring understanding

within <contexts derived from the Commission’s scope-delimiting report>

Evidence supporting a claim is expressed as a work product whose features are defined by these reports.

Page 43: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Elaboration of EvidenceEvidence that supports this claim would have

these important categories of features. Essential observable features of a work product: symmetry of field lines, direction of field lines, statement of direction as convention Assignment of values to these observable features:: field lines have radial symmetry, field lines point at the mass source, field lines point at the negatively charged source, field lines point away from the positively charge source, explanation of why the field lines have this orientation Which of these features would support the claim as A work? Evaluation of A work: correct symmetry is displayed, correct orientation for each type of source is displayed, explanation indicates convention Which of these features would support the claim as C work? Evaluation of C work: correct symmetry is displayed, orientation of field lines may be reversed but are consistent, orientation has a physical rationale

Page 44: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

AP RedesignModel of Knowing & Learning

Each concept description generates multiple claims: What is the elaborated claim? Student claim: The student will be able to construct visual representations of force vectors.

Student work can provide evidence to support the claim. The features of this evidence are taken from the concept descriptions:

B. Evidence that supports this claim would have these important categories of features. Essential observable features of a work product: vectors have magnitude and direction, vectors have perpendicular compenents, vectors act at a single poi nt on an object, vectors are added component by component, adding vector quantities is a different process than adding scaler quantities. Assignment of values to these observable features : force vectors originate from a single point on the object, A single force is drawn to scale and directed correctly. The components of a single vector are correctly drawn and labeled (no more than two -dimensional representations), correctly combining vector components to determine the vector sum of several co-planer forces, correctly representing the resultant force vector and correctly labeling the magnitude and direction including quadrant

Page 45: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Your Next Major Assignment

• For the “big idea” that you have begun to unpack and elaborate now try to specify:– What possible claims about student knowledge, skill, and

understanding you would want and/or need to make.– For each claim state it as precisely as you can.– What are major claims and minor claims?– For whom is such a claim appropriate?– State as precisely as you can the nature of the evidence that

is needed to support a claim– Be sure to consider the mapping between claims and

evidence

• Do this as systematically as possible.• Be prepared to upload your work to the wiki

Page 46: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

• From Mislevy & Riconscente, in press

Assessment DeliveryAssessment DeliveryHow do students and tasks actually interact? How do we report examinee performance?

How do students and tasks actually interact? How do we report examinee performance?

Assessment Implementation

Assessment Implementation

Conceptual Assessment Framework

Conceptual Assessment Framework

Domain ModelingDomain Modeling

Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

How do we represent key aspects of the domain in terms of assessment argument.

How do we represent key aspects of the domain in terms of assessment argument.

Design structures: Student, evidence, and task models

Design structures: Student, evidence, and task models

How do we choose and present tasks, and gather and analyze responses?

How do we choose and present tasks, and gather and analyze responses?

Layers in the assessment enterpriseLayers in the assessment enterprise

Page 47: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

• From Mislevy & Riconscente, in press

Assessment DeliveryAssessment DeliveryHow do students and tasks actually interact? How do we report examinee performance?

How do students and tasks actually interact? How do we report examinee performance?

Assessment Implementation

Assessment Implementation

Conceptual Assessment Framework

Conceptual Assessment Framework

Domain ModelingDomain Modeling

Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

How do we represent key aspects of the domain in terms of assessment argument.

How do we represent key aspects of the domain in terms of assessment argument.

Design structures: Student, evidence, and task models

Design structures: Student, evidence, and task models

How do we choose and present tasks, and gather and analyze responses?

How do we choose and present tasks, and gather and analyze responses?

Cog Psych, expertise studies,domain research

Cog Psych, expertise studies,domain research

Page 48: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

• From Mislevy & Riconscente, in press

Assessment DeliveryAssessment DeliveryHow do students and tasks actually interact? How do we report examinee performance?

How do students and tasks actually interact? How do we report examinee performance?

Assessment Implementation

Assessment Implementation

Conceptual Assessment Framework

Conceptual Assessment Framework

Domain ModelingDomain Modeling

Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

How do we represent key aspects of the domain in terms of assessment argument.

How do we represent key aspects of the domain in terms of assessment argument.

Design structures: Student, evidence, and task models

Design structures: Student, evidence, and task models

How do we choose and present tasks, and gather and analyze responses?

How do we choose and present tasks, and gather and analyze responses?

Assessment ArgumentAssessment Argument

Page 49: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

• From Mislevy & Riconscente, in press

Assessment DeliveryAssessment DeliveryHow do students and tasks actually interact? How do we report examinee performance?

How do students and tasks actually interact? How do we report examinee performance?

Assessment Implementation

Assessment Implementation

Conceptual Assessment Framework

Conceptual Assessment Framework

Domain ModelingDomain Modeling

Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

How do we represent key aspects of the domain in terms of assessment argument.

How do we represent key aspects of the domain in terms of assessment argument.

Design structures: Student, evidence, and task models

Design structures: Student, evidence, and task models

How do we choose and present tasks, and gather and analyze responses?

How do we choose and present tasks, and gather and analyze responses?

• Explicit connection to domain & purpose• Generative structures for recurring kinds

of proficiencies (e.g., inquiry cycles, troubleshooting) across projects

• PADI Design patterns

Page 50: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

• From Mislevy & Riconscente, in press

Assessment DeliveryAssessment DeliveryHow do students and tasks actually interact? How do we report examinee performance?

How do students and tasks actually interact? How do we report examinee performance?

Assessment Implementation

Assessment Implementation

Conceptual Assessment Framework

Conceptual Assessment Framework

Domain ModelingDomain Modeling

Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

How do we represent key aspects of the domain in terms of assessment argument.

How do we represent key aspects of the domain in terms of assessment argument.

Design structures: Student, evidence, and task models

Design structures: Student, evidence, and task models

How do we choose and present tasks, and gather and analyze responses?

How do we choose and present tasks, and gather and analyze responses?

Generative Design SchemasGenerative Design Schemas

Page 51: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

• From Mislevy & Riconscente, in press

Assessment DeliveryAssessment DeliveryHow do students and tasks actually interact? How do we report examinee performance?

How do students and tasks actually interact? How do we report examinee performance?

Assessment Implementation

Assessment Implementation

Conceptual Assessment Framework

Conceptual Assessment Framework

Domain ModelingDomain Modeling

Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

How do we represent key aspects of the domain in terms of assessment argument.

How do we represent key aspects of the domain in terms of assessment argument.

Design structures: Student, evidence, and task models

Design structures: Student, evidence, and task models

How do we choose and present tasks, and gather and analyze responses?

How do we choose and present tasks, and gather and analyze responses?• Explicit connection of argument to machinery

• Generative structures for recurring task situations (e.g., item shells)

• Re-usable / interoperable data structures• PADI templates

Page 52: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

• From Mislevy & Riconscente, in press

Assessment DeliveryAssessment DeliveryHow do students and tasks actually interact? How do we report examinee performance?

How do students and tasks actually interact? How do we report examinee performance?

Assessment Implementation

Assessment Implementation

Conceptual Assessment Framework

Conceptual Assessment Framework

Domain ModelingDomain Modeling

Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?

How do we represent key aspects of the domain in terms of assessment argument.

How do we represent key aspects of the domain in terms of assessment argument.

Design structures: Student, evidence, and task models

Design structures: Student, evidence, and task models

How do we choose and present tasks, and gather and analyze responses?

How do we choose and present tasks, and gather and analyze responses?

Specific implementations. Surface elements.

Specific implementations. Surface elements.

Page 53: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Assessment BlueprintWhat particular tasks, questions or situations will bring about a response that provides the type of evidence you want?

A single task may provide evidence for more than one claim

Multiple tasks may be necessary to assess a single claim

For example-multiple-choice interviewopen-ended writtenshort answer performance

Page 54: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Your Next Major Assignment

• For your claims and evidence now try to specify:– The range of tasks and situations that would provide the

relevant evidence– How different task types would yield the relevant evidence – What types of “task templates” might be created– What range of contexts would be applicable– How the tasks might be scored -- rubrics that relate to the

evidence needed– If possible, generate sample tasks or items and show how

they fit aspects of your blueprint

• Do this as systematically as possible.• Be prepared to upload your work to the wiki

Page 55: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Group Appraisal of Construct-Driven Assessment for the Work of NCLT

• Practical & Useful for NCLT: ?????– Depends on your definition of practical and/or useful as well as

what aspects of the assessment design still need to be developed

• Realistic: appears to be doable in other contexts– There are multiple examples at varying grain sizes– Examples cover simple to complex domains of learning and

instruction so NCLT contexts are within the ballpark

• Pipe Dream: definitely– Gotta keep on smokin’ & tokin’

Page 56: Construct Driven Assessment Design: NCLT Reality or Lake Michigan Pipe Dream?

Top Related