assessment for learning in science: issues in learning to interpret student work
DESCRIPTION
Assessment for learning in science: Issues in learning to interpret student work. Center for the Assessment and Evaluation of Student Learning (CAESL) University of California, Berkeley University of California, Los Angeles Lawrence Hall of Science. UCLA Shaunna Clark Joan Herman - PowerPoint PPT PresentationTRANSCRIPT
1
Assessment for learning in Assessment for learning in science: science:
Issues in learning to interpret Issues in learning to interpret student workstudent work
Center for the Assessment and Evaluation of Student Learning (CAESL)
University of California, BerkeleyUniversity of California, Los Angeles
Lawrence Hall of Science
2
WestEdDiane
CarnahanKaren Cerwin
Kathy DiRannaJo Topps
U.C. BerkeleyMaryl Gearhart
Jennifer Pfotenhauer
Cheryl Schwab
UCLAShaunna ClarkJoan Herman
Sam NagashimaEllen OsmundsonTerry Vendlinski
Lawrence Hall of Science
Lynn BarakosCraig Strang
3
PresentationPresentation
• Program and participants
• Research framework • Design and methods• Selected findings• Implications
3
4
Situation • assessments in materials of variable qualityassessments in materials of variable quality• teachers lack expertise to reviseteachers lack expertise to revise• professional practices not well establishedprofessional practices not well established
Argument • science education reform (NRC/NSES) science education reform (NRC/NSES) • known benefits of classroom assessment known benefits of classroom assessment
(e.g., Black & Wiliam, 1998; Sloane & Wilson, (e.g., Black & Wiliam, 1998; Sloane & Wilson, 2000)2000)
• value of reflective practice and long term value of reflective practice and long term collaboration (Garet et al, 2001)collaboration (Garet et al, 2001) 4
Impetus for programImpetus for program
5
Principles • integrated with practice• long term• collaborative
• reflective practice
Core strategy• assessment portfolio
5
CAESL Leadership AcademyCAESL Leadership Academy7/03 - 12/047/03 - 12/04
6
N = 19 (23)
Years teaching 14.6
Education (percentage)MA + 53
Recent participation in (percentage)professional organizations 89assessment 95
Perceived [weak (1) – very strong (5)]content knowledge 4.5knowledge of standards 4.5knowledge of assessment 4.3confidence in teaching science 4.7
Academy participantsAcademy participants
6
7
Interwoven structures• district vertical teams w/ administratorsdistrict vertical teams w/ administrators• cross district grade level teamscross district grade level teams• independent classroom implementationindependent classroom implementation
Series of portfolios • repeated opportunities to build expertiserepeated opportunities to build expertise
7
Program organizationProgram organization
8
Portfolio: I. PlanPortfolio: I. Plan
Establish learning goals• analyze ‘conceptual flow’ of materialsanalyze ‘conceptual flow’ of materials• align with standardsalign with standards
Select assessments• choose key assessments to track choose key assessments to track
progress: pre -> junctures -> post progress: pre -> junctures -> post • identify the concepts assessed identify the concepts assessed • anticipate ‘expected student anticipate ‘expected student
responses’responses’ 8
9
Portfolio: II. Portfolio: II. ImplementationImplementation
Interpret student work• refine assessment criteriarefine assessment criteria• scorescore• chart and identify trendschart and identify trends
Use evidence• document instructional follow-document instructional follow-
up and feedbackup and feedback9
10
Portfolio: III. Evaluate & Portfolio: III. Evaluate & reviserevise
10
Evaluate using student work• alignment with goals and alignment with goals and
instructioninstruction• quality of tasks and criteriaquality of tasks and criteria• methods of analysismethods of analysis
Revise and strengthen
11 11
Rated for completeness Complete:Complete: I, II (some student I, II (some student
work), IIIwork), III
Partial: Partial: I I oror III, II (some) III, II (some)
Minimal: Minimal: I or III onlyI or III only
None (but participating)None (but participating)
Portfolio Completion
12
Fall 2003 Spring 2004 Fall 2004
Level (22) (23) (19)
Complete .50 .39 .47
Partial* .32 .30 .16
Minimal .18 .26 .16
None .00 .00 .21*but includes student work
Portfolio Completion
12
13
StudyStudy
13
Focus • Growth in understanding and practice• Supports and barriers
Longitudinal, nested design• 18 months = 3 portfolios• Cohort: Surveys, focus groups, portfolios
• Cases: Interviews and observations
14
Framework for Framework for classroom assessment classroom assessment
expertiseexpertiseUnderstanding of assessment concepts
Facility with assessment practices
14
UNDERSTANDING ASSESSMENT CONCEPTS
QUALITY GOALS FOR STUDENT LEARNING AND
PROGRESS
QUALITY USE
QUALITY
TOOLS
15
UNDERSTANDING ASSESSMENT CONCEPTS
QUALITY GOALS FOR STUDENT LEARNING AND
PROGRESS
QUALITY USE
QUALITY
TOOLSSOUND
INTERPRETATION
16
ESTABLISH GOALS & DESIGN ASSESSMENT PLAN
ASSESS
USE INFORMATION TO GUIDE TEACHING &
LEARNING
REVISE ASSESSMENTS
PROVIDE INSTRUCTION
INTERPRET STUDENT WORK17
CLASSROOM ASSESSMENT PRACTICES
ASSESS
USE INFORMATION TO GUIDE TEACHING &
LEARNING
REVISE ASSESSMENTS
PROVIDE INSTRUCTION
INTERPRET STUDENT WORK
Soundness of interpretation:- criteria capture student understanding?- scoring consistent?- interpretation appropriate to purpose?
ESTABLISH GOALS & DESIGN ASSESSMENT PLAN
18
USING CONCEPTS TO GUIDE PRACTICE
ASSESS
USE INFORMATION TO GUIDE TEACHING &
LEARNING
REVISE ASSESSMENTS
PROVIDE INSTRUCTION
INTERPRET STUDENT WORK19
ESTABLISH GOALS & DESIGN ASSESSMENT PLAN
Soundness of interpretation:- criteria capture student understanding?- scoring consistent?- interpretation appropriate to purpose?
USING CONCEPTS TO GUIDE PRACTICE
20
20
Selected findingsSelected findings
Portfolios: Patterns of change• assessment criteria• analysis of whole class patterns• alignment
Exit survey and focus groups• perceived growth• supports, barriers, needs
21
Source• series of 2 or 3 portfolios (n ≈ 10)
Issues & constraints• burden of documentation
• paper & pencil assessments
• professional choice
Patterns in portfoliosPatterns in portfolios
21
22
Assessment criteria
22
+from global/holistic to more specific, differentiated, and assessable
+from focus on surface features to efforts to capture student understanding
+from dichotomous (right/wrong) to attention to qualitative levels of understanding
=but … quality variable despite teacher interest (example: reliance on content analysis or notes)
23
Whole class analysis
23
+from few to efforts at systematic analysis using charting or content analysis
+from global patterns toward more differentiated analysis (item analysis, item clustering) and efforts to coordinate group & individual patterns
+efforts to analyze progress (espec. pre-post)
=but … information often unintegrated, inferences unsystematic, comparisons inappropriate
24
Alignment
24
+efforts to align interpretations with learning goals, tasks, and criteria
+efforts to revise criteria to strengthen alignment
+fewer inferences about ancillary skills not assessed
=but … problematic alignment of assessments and inferences to track progress
25
Understanding of CAESL• 1 (none) <--> 5 (full)
Implementation of CAESL
• 1 (none) <--> 4 (full) <--> 5 (beyond)
25
Exit survey
26
Item (examples) Mean[None (1)– Full (5)]
Percentagebelow full
(5)
INTERPRETATION OF WORK
Analyzing whole class sets 4.3 .50
Comparing pre and post 4.4 .45
Comparing pre and juncture 4.4 .50
UNDERSTANDING
26
27
Item (examples) MeanNone
(1)–Full+(5)
Percentagebelow full (4)
INTERPRETATION
Evaluating and developing criteria
Evaluating criteria – content quality 3.5 .40
Developing using ESRs 3.6 .40
Refining using stud. work 3.9 .25
Scoring
Using criteria to score 4.0 .20
Checking reliability of scoring 3.8 .30
Charting scores 3.2 .55
Charting and analyzing patterns
Charting conceptual understanding 3.2 .65
Analyzing whole class patterns 3.8 .30
IMPLEMENTATION
27
28
Exit focus groupsExit focus groups
28
ESTABLISH GOALS & DESIGN ASSESSMENT PLAN
ASSESS
USE INFORMATION TO GUIDE TEACHING &
LEARNING
REVISE ASSESSMENTS
PROVIDE INSTRUCTION
INTERPRET STUDENT WORK29
PRACTICES STRENGTHENED MOST?
ESTABLISH GOALS & DESIGN ASSESSMENT PLAN
ASSESS
USE INFORMATION TO GUIDE TEACHING &
LEARNING
REVISE ASSESSMENTS
PROVIDE INSTRUCTION
INTERPRET STUDENT WORK30
PRACTICES STRENGTHENED LEAST?
ESTABLISH GOALS & DESIGN ASSESSMENT PLAN
ASSESS
USE INFORMATION TO GUIDE TEACHING &
LEARNING
REVISE ASSESSMENTS
PROVIDE INSTRUCTION
INTERPRET STUDENT WORK31
PRACTICES MOST IMPORTANT?
32
Portfolio• establishing goals• revision and re-application• resource for collaboration
Resources• CAESL framework • articles and books• grade level teams & facilitators
• long term program
Supports
32
33
Portfolio• assessment development• only paper and pencil• focus on perf. assessments• time
Resources• weak assessments• limited frameworks• no clear models for progress• gaps in teacher knowledge
Barriers
33
Context• standards• testing• school & district
34
Portfolio• assessment development• only paper and pencil• focus on perf. assessments• time
Resources• weak assessments• limited frameworks• no clear models for progress• gaps in teacher knowledge
Barriers
34
Context• standards• testing• school & district
Unnamed challenges• inquiry• ancillary skills
35
Portfolio …if….• streamline• focus on goals, interpretation, and use
• refinement not development
• expand assessment types
Resources•embedded assessments
•handbook•conceptual development
•grade level collaboration
•coaching and facilitation
Context•align with district and state assessments
35
Requests
36
Strengthen materials & resources
Expand to unit assessment systems
Align assessment content and quality
Modify program organization 36
Implications