proficiency in science: assessment challenges & opportunities jim pellegrino
DESCRIPTION
Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino. Chinese Curse? . - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/1.jpg)
Proficiency in Science:
Assessment Challenges & Opportunities
Jim Pellegrino
![Page 2: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/2.jpg)
Chinese Curse?
• There is a Chinese curse which says 'May he live in interesting times.' Like it or not we live in interesting times. They are times of danger and uncertainty; but they are also more open to the creative energy of men than any other time in history.
– Robert Kennedy, 1966.
![Page 3: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/3.jpg)
![Page 4: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/4.jpg)
![Page 5: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/5.jpg)
![Page 6: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/6.jpg)
New Definition of Competence
• The NRC Science Framework & NGSS have proposed descriptions of student competence as being the intersection of knowledge involving: – important disciplinary practices – core disciplinary ideas, – and crosscutting concepts with– performance expectations representing the
intersection of the three.• They view competence as something that develops
over time & increases in sophistication and power as the product of coherent curriculum & instruction
![Page 7: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/7.jpg)
Goals for Teaching & Learning• Coherent investigations of
core ideas across multiple years of schooling
• More seamless blending of practices with core ideas
• Performance expectations that require reasoning with core disciplinary ideas – explain, justify, predict,
model, describe, prove, solve, illustrate, argue, etc.
Core Ideas
Practices
Crosscutting Concepts
![Page 8: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/8.jpg)
![Page 9: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/9.jpg)
Aligning Curriculum,Instruction & Assessment
Standards
![Page 10: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/10.jpg)
Challenges for Assessment
• To develop assessment tasks that integrate the three dimensions.
• To develop tasks that can assess where a student can be placed along a sequence of progressively more complex understandings of a given core idea, and successively more sophisticated applications of practices and crosscutting concepts.
• To develop assessment tasks that measure the connections between the different strands of disciplinary core ideas (e.g. using understandings about chemical interactions from physical science to explain phenomena in biological contexts).
![Page 11: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/11.jpg)
5 Things for Us to Talk About
1. assessment contexts, purposes and uses,
2. the nature of assessment and the importance of research on learning,
3. assessment design processes,
4. affordances of technology, and
5. systems of assessment
![Page 12: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/12.jpg)
Contexts and Purposes
• Educational assessment typically occurs in multiple contexts: – Small scale: individual classrooms – Intermediate-scale: districts– Large-scale: states, nations
• Within and across contexts it can be used to accomplish differing purposes:– Assist learning (formative)– Measure individual achievement (summative)– Evaluate programs (accountability)
![Page 13: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/13.jpg)
A Multiplicity of Actors & Needs
The reason we have so many different forms and types of assessment is that “One size does not fit all”
– Educators at different levels of the system need different information at different time scales
– They have differing priorities, they operate under different constraints, & there are tradeoffs in terms of time, money, and type of information needed
![Page 14: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/14.jpg)
5 Things for Us to Talk About
1. assessment contexts, purposes and uses,
2. the nature of assessment and the importance of research on learning,
3. assessment design processes,
4. affordances of technology, and
5. systems of assessment
![Page 15: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/15.jpg)
Assessment as a Process of Reasoning from Evidence
• cognition – theory, models, and data about
how students represent knowledge & develop competence in the domain
• observations– tasks or situations that allow
one to observe students’ performance
• interpretation– methods for making sense of
the data
observation interpretation
cognition
Must be coordinated!
![Page 16: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/16.jpg)
Why Models of Development of Domain Knowledge are Critical
• Tell us what are the important aspects of knowledge that we should be assessing.
• Give us strong clues as to how such knowledge can be assessed
• Can lead to assessments that yield more instructionally useful information– diagnostic & prescriptive
• Can guide the development of systems of assessments intended to cohere– across grades & contexts of use
![Page 17: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/17.jpg)
![Page 18: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/18.jpg)
The Goldilocks Problem
• Not too big, not too little, just right!!!!
• Not too many things, not too few things, just the right grain size!!
![Page 19: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/19.jpg)
5 Things for Us to Talk About
1. assessment contexts, purposes and uses,
2. the nature of assessment and the importance of research on learning,
3. assessment design processes,
4. affordances of technology, and
5. systems of assessment
![Page 20: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/20.jpg)
Issues of Assessment Design & Development
• Assessment design spaces vary tremendously & involve multiple dimensions– Type of knowledge and skill and levels of sophistication– Time period over which knowledge is acquired– Intended use and users of the information– Availability of detailed theories & data in the domain– Distance from instruction and assessment purpose
• Need a principled process that can help structure going from theory, data and/or speculation to an operational assessment– Evidence Centered Design
![Page 21: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/21.jpg)
Exactly what knowledge do
you want students to have and how do you
want them to know it?
claim space evidence task
What task(s) will the
students perform to
communicate their
knowledge? How will you analyze and interpret the evidence?
Evidence-Centered DesignWhat will you
accept as evidence that a student has the desired knowledge?
Need to consider what this might mean when it comes to performance expectations integrating Disciplinary Core Ideas & Practices
![Page 22: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/22.jpg)
AP Content & Science Practices
Core Ideas Science practices
1. Use representations and models to communicate scientific phenomena and solve scientific problems.
2. Use mathematics appropriately.
3. Engage in scientific questioning to extend thinking or to guide investigations within the context of the AP course.
4. Plan and implement data collection strategies in relation to a particular scientific question.
5. Perform data analysis and evaluation of evidence.
6. Work with scientific explanations and theories.
7. Connect and relate knowledge across various scales, concepts, and representations in and across domains.
![Page 23: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/23.jpg)
Illustrative Claims and EvidenceAP Biology
Big Idea 1: The process of evolution drives the diversity and
unity of life.
EU 1A: Change in the genetic makeup of a population over time is
evolution.
L3 1A.3: Evolutionary change is driven by genetic drift and
artificial selection.
The Claim: The student can make predictions about the effects of natural selection versus genetic drift on the evolution of both large and small populations of organisms.
The Evidence: The work will include a prediction of the effects of either natural selection or genetic drift on two populations of the same organism, but of different sizes; the prediction includes a description of the change in the gene pool of a population; the work shows correctness of connections made between the model and the prediction and the model and the phenomena (e.g. genetic drift may not happen in a large population of organisms; both natural selection and genetic drift result in the evolution of a population).
Suggested Proficiency Level: 4
Skill 6.4: The student can make claims and predictions about natural phenomena based on scientific theories and models.
![Page 24: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/24.jpg)
Connecting the Domain Model toCurriculum, Instruction, &
Assessment
![Page 25: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/25.jpg)
Sample AP Bio Task
![Page 26: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/26.jpg)
5 Things for Us to Talk About
1. assessment contexts, purposes and uses,
2. the nature of assessment and the importance of research on learning,
3. assessment design processes,
4. affordances of technology
5. systems of assessment
![Page 27: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/27.jpg)
A Claim We Need to Embrace• Much of what is new, different, and
important in the NRC Science Framework and NGSS cannot be adequately assessed by conventional methods, items, and measurement models– The capacity to engage in the practices as
connected to important domain-specific ideas and understandings
– We are interested in the processes of thinking as well as the products of those processes
![Page 28: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/28.jpg)
Advantages of Technology for Science Assessment
• Present authentic, rich, dynamic environments• Present phenomena difficult or impossible to observe and
manipulate in classrooms• Represent temporal, causal, dynamic relationships “in
action”• Allow multiple representations of stimuli and their
simultaneous interactions (e.g., data generated during a process)
• Allow overlays of representations, symbols• Allow student manipulations/investigations, multiple trials• Allow student control of pacing, replay, reiterate• Capture student responses during research, design,
problem solving
![Page 29: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/29.jpg)
SimScientists: Levels of Analysis and Understanding
![Page 30: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/30.jpg)
Life Science SimulationStudents
– view animation to observe relationships among organisms– draw food web illustrating those relationships .
![Page 31: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/31.jpg)
In the experiment that you just analyzed, the amount of alewife was set to 20 at the beginning.
Another student hypothesized that the result might be very different if she started with a larger or smaller amount of alewife at the beginning.
Run three experiments to test that hypothesis. At the end of each experiment record your data by taking pictures of the resulting graphs.
After three runs, you will be shown your results and asked if it makes any difference if the beginning amount of alewife is larger or smaller than 20.
Life Science Simulation
![Page 32: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/32.jpg)
Embedded Assessments with Formative Feedback
![Page 33: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/33.jpg)
5 Things for Us to Talk About
1. assessment contexts, purposes and uses,
2. the nature of assessment and the importance of research on learning,
3. assessment design processes,
4. affordances of technology, and
5. systems of assessment
![Page 34: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/34.jpg)
Desired end product is a multilevel systemEach level fullfills a clear set of
functions and has a clear set of intended users of the assessment information
The assessment tools are designed to serve the intended purpose• Formative, summative or accountability• Design is optimized for function served
The levels are articulated and conceptually coherentThey share the same underlying
concept of what the targets of learning are at a given grade level and what the evidence of attainment should be.
They provide information at a “grain size” and on the “time scale” appropriate for translation into action.
![Page 35: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/35.jpg)
What Such a System Might Look Like
Multilevel Assessment System
An Integrated System
Coordinated across levels
Unified by common learning goals
Synchronized by unifying progress
variables
![Page 36: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/36.jpg)
The Key Design Elements of Such a Comprehensive System
The system is designed to track progress over time At the individual student level At the aggregate group level
The system uses tasks, tools, and technologies appropriate to the desired inferences about student achievement Doesn’t force everything into a fixed testing/task
model Uses a range of tasks: performances, portfolios,
projects, fixed- and open-response tasks as needed
![Page 37: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/37.jpg)
Example 1: Aggregating Information Across Levels
![Page 38: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/38.jpg)
Example 2: Multi-levelTask Correspondence
![Page 39: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/39.jpg)
Five Questions forUs to Ponder
1. What are some “Grand Challenges” for science assessment
2. Where do we stand in meeting them?
3. What’s left to do?
4. Will we have tests worth teaching to?
5. Can we avoid the sins of the past?
![Page 40: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/40.jpg)
Science Assessment: Grand Challenges
![Page 41: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/41.jpg)
Where Do We Stand inMeeting these Challenges?
• We have a much better sense of what the development of competence should mean and the possible implications for designing coherent science education
• We have examples of thinking through in detail the juxtaposition of disciplinary practices and core content knowledge to guide the design of assessment– AP Redesign Project– Multiple Assessment R&D Projects
![Page 42: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/42.jpg)
Connecting the Domain Model toCurriculum, Instruction, &
Assessment
![Page 43: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/43.jpg)
![Page 44: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/44.jpg)
Science Practices:Science Inquiry &
ReasoningEssential
Knowledge
Learning Objectives
Enduring Understandings
Structure of the AP Biology Curriculum Framework
4 Big Ideas
![Page 45: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/45.jpg)
Curriculum Framework:Big Ideas
The unifying concepts or Big Ideas increase coherence both within and across disciplines. A total of Four Big Ideas:
The process of evolution drives the diversity and unity of life.B I G I D E
A 1
Living systems retrieve, transmit, and respond to information essential to life processes.B I G I D E
A 3Biological systems interact, and these interactions possess complex properties.B I G I D E
A 4
Biological systems utilize energy and molecular building blocks to grow, reproduce, and maintain homeostasis.
B I G I D E A 2
![Page 46: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/46.jpg)
Immediate Impacts of AP Biology Changes
1. ~10,000-12,000 high school biology teachers across the country all changed the way they taught AP Biology…at the same time
2. For many teachers, they had to replace all their laboratory investigations. For all of them, they had to incorporate inquiry activities throughout the course, not just use inquiry in a few labs. For some, incorporation of mathematical skills is a challenge.
3. In May ~180,000 students took the new AP Biology exam
![Page 47: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/47.jpg)
No test items will focus on low cognitive level/declarative knowledge/recall
For each exam item, students will either produce the evidence (CR) or engage with the evidence (SR/MC)
ECD and the New AP Exam
► explain► justify► predict► evaluate► describe► analyze
► pose scientific questions
► construct explanations
► construct models► represent
graphically► solve problems► select and apply
mathematical routines
![Page 48: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/48.jpg)
What’s The Impact Of Curriculum Changes On New AP Biology Exam?
Because of use of Big Ideas….in 2008, 12% of questions had something to do with evolutionIn new exam, 35% of questions have something to do with evolution
Because of emphasis on science practice and mathematical skills…new types of questions are being asked, e.g., grid-ins
Because of use of evidence…the number of Multiple Choice questions was reduced from 100 questions on last year’s exam to 63 on this year’s exam.
![Page 49: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/49.jpg)
Lessons Learned from the AP Redesign Project
• No Pain -- No Gain!!! -- this is hard work• Backwards Design and Evidence
Centered Design are challenging to execute & sustain–Requires multidisciplinary teams–Requires sustained effort and negotiation–Requires time, money & patience
• Value-added -- Validity is “designed in” from the start as opposed to “grafted on”–Elements of a validity argument are contained
in the process and the products
![Page 50: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/50.jpg)
What’s Left to Do? – A LOT!!!
• We need to translate the standards into effective models, methods and materials for curriculum, instruction, and assessment.– Need clear performance expectations– Need precise claims & evidence statements– Need task models & templates
• We need to use what we know already to evaluate and improve the assessments that are part of current practice, e.g., classroom assessments, large-scale exams, etc.
![Page 51: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/51.jpg)
• Desires of the policy community often conflict with the capacities of the R&D community– Need for better coordination and communication
• USDoE, States, IES & NSF, R&D Community, Teachers, Administrators, & Professional Education Groups
• Standards are the beginning not the end – not a substitute for the thinking and research needed to define progressions of learning that can serve as a basis for the integration of curriculum, instruction and assessment.
Will We Have AssessmentsWorth Teaching With & To?
![Page 52: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/52.jpg)
Assessment Should not be the “Tail Wagging the Science Education Dog”
Assessment
![Page 53: Proficiency in Science: Assessment Challenges & Opportunities Jim Pellegrino](https://reader035.vdocuments.us/reader035/viewer/2022062315/568162ca550346895dd356d0/html5/thumbnails/53.jpg)
“The perfect is the enemy of the good."
Voltaire