california k-12 assessment update
DESCRIPTION
California K-12 Assessment Update. California Education Research Association November 30, 2012 Patrick Traynor, Ph.D. Director, Assessment Development and Administration Division Eric Zilbert, Ph.D. Administrator, ADAD Psychometric Unit. Presentation Overview. - PowerPoint PPT PresentationTRANSCRIPT
CALIFORNIA DEPARTMENT OF EDUCATIONTom Torlakson, State Superintendent of Public Instruction
California Education Research Association
November 30, 2012
Patrick Traynor, Ph.D.Director, Assessment Development and Administration Division
Eric Zilbert, Ph.D.Administrator, ADAD Psychometric Unit
California K-12 Assessment Update
TOM TORLAKSONState Superintendent of Public Instruction
Presentation Overview
• Recent and Upcoming SBAC Developments
• Scoring Technologies
• Alternate Assessment Participation (1% Population)
• Transitional Activities
2
TOM TORLAKSONState Superintendent of Public Instruction
Recent and Upcoming SBAC Developments
June-October 2012 Cognitive Labs
Oct 15-Nov 2, 2012 Small Scale Trials
October 9, 2012 Released Items
Nov 27, 2012-Jan 15, 2013
Draft Achievement Level Descriptors
Spring 2013 Pilot Test
Spring 2014 Field Test 3
TOM TORLAKSONState Superintendent of Public Instruction
SBAC Cognitive Labs
PurposeTo examine the impact of various item formats, tools, and accommodations on the ability of students to demonstrate in depth understanding of content measured.
Sampling– 700 students – ≈ 85 students in California
– Focus on SWD, ELL, and low SES students.
4
TOM TORLAKSONState Superintendent of Public Instruction
SBAC Cognitive Labs
Methodology •Research questions related to technology-based assessment
•A trained facilitator – administers test items– conducts a interview (≈ 90-120 minutes).
•Computer provided by the facilitator
•Labs (two approaches)– Think aloud as work through an item– First solve, then respond to questions about
approach
5
TOM TORLAKSONState Superintendent of Public Instruction
SBAC Small Scale TrialsPurposeTo inform automated and human scoring
Sampling•Stratified random sampling
– ≈ 900 schools across member states – ≈ 230 schools in California
•Goal: to be representative of SBAC students
•Schools – randomly select one to two classrooms– assigned grade (4, 7, or 11)
6
TOM TORLAKSONState Superintendent of Public Instruction
SBAC Small Scale Trials
Methodology
•Each assessment – 15-18 selected-response and constructed response
items – ≈ 60-90 minutes of student time
•School computers
•Each participating school designates a school coordinator who is trained to administer the assessment
•Schools are not required to assess all selected students at the same time or on the same day
7
TOM TORLAKSONState Superintendent of Public Instruction
Smarter Balanced Pilot Testing
• Smarter Balanced will conduct a Pilot computer-based administration of their assessment system beginning in February 2013.
• Items will be aligned to the Common Core State Standards and will include selected response, constructed response, and performance tasks
• Participation in the Pilot Test will be open to all schools in the Consortium and will be administered to students in grades 3–8 and grade 11.
8
TOM TORLAKSONState Superintendent of Public Instruction
Smarter Balanced Pilot Testing
• The Pilot Test entails two approaches (or components) in its implementation:
1) “Volunteer” component that is open to all schools in Smarter Balanced states and will ensure that all schools have the opportunity to experience the basic functionality of the system
2) “Scientific” component that targets a representative sample of schools and yields critical data about the items developed to date, as well as how the system is functioning
9
TOM TORLAKSONState Superintendent of Public Instruction
Pilot Participation Important Dates
The table below presents upcoming key activities related to the Smarter Balanced Pilot Test:
Activity Dates
Interested schools volunteer to participate via the online survey located at: https://www.surveymonkey.com/s/SmarterBalancedPilot
November 2012 – January 2013
Scientifically selected schools contacted to confirm their participation in the Pilot Test
Late-November 2012 – January 2013
Online training test is available Mid-January 2013
Smarter Balanced Pilot Test window
February 20 – May 10, 2013
10
TOM TORLAKSONState Superintendent of Public Instruction
Smarter Balanced Draft Achievement Level
Descriptors (ALDs)
• “Describe” four levels of achievement: “deep command,” “sufficient command,” “partial command,” or “minimal command” of knowledge, skills, and processes in both English–language arts/literacy and mathematics
• First draft ALDs for public comment from November 27, 2012 through January 15, 2013
• A full description of the ALDs and an online survey for providing feedback are available on the Smarter Balanced achievement level descriptors Web page at http://www.smarterbalanced.org/achievement-level-descriptors-and-college-readiness/
• 11
Smarter BalancedSample Items and Performance Tasks
12
Purpose of Sample Items and Performance Tasks
• Demonstrate rigor and complexity of ELA/literacy and mathematics questions
• Showcase variety of item types:• Selected response• Constructed response• Technology enhanced• Performance tasks
• Help teachers to begin planning for the shifts in instruction
Page 13
Overall Claim for Grades 3-8
Overall Claim for Grade 11
Claim #1 - Reading
Claim #2 - Writing
Claim #3 - Speaking and Listening
Claim #4 - Research/Inquiry
Claims for the ELA/Literacy Summative Assessment
Page 14
Overall Claim for Grades 3-8
Overall Claim for Grade 11
Claim #1 - Concepts & Procedures
Claim #2 - Problem Solving
Claim #3 - Communicating Reasoning
Claim #4 - Modeling and Data Analysis
Claims for the Mathematics Summative Assessment
Page 15
Exploring the Sample Items
16
Sample Items and Tasks Landing Page to Launch Oct. 9
Page 17
Sample Items and Tasks Navigation
View mathematics or ELA/literacy items
Advance to next item, or go back to previous
Page 18
Sample Items and Tasks Navigation
Content Claim
Grade band
Page 19
Sample Items and Tasks Navigation
Filter by item type, themes
Page 20
Item Metadata
About this item
Page 21
Exploring the Sample Items
Selected response and technology enhanced items are machine scorable
Page 22
Accessibility and Accommodations
• Sample items do not include accessibility and accommodations features
• As part of the development of accessibility and accommodations policies, the Consortium commissioned research on best practices for assessing English language learners and students with disabilities
• When fully operational, Smarter Balanced will be providing translation accommodation options for all math items
• For the Pilot, Smarter Balanced will be providing full Spanish translations and translated Spanish pop-up glossaries that are customized at the item-level for a specific form for each of three grades (almost 70 items each). Each grade-level (elementary, middle, and high school) will have one grade with the special form.
• The Consortium will need ELLs to participate in the Pilot to make sure they are represented in the student sample.
23
Accessibility and Accommodations
• Full range of accessibility tools and accommodations options under development guided by:– Magda Chia, Ph.D., Director of Support for Under-Represented
Students– Accessibility and Accommodations Work Group– Students with Disabilities Advisory Committee
• Chair: Martha Thurlow (NCEO)– English Language Learners Advisory Committee
• These teams of experts will ensure that the assessments provide valid, reliable, and fair measures of achievement and growth for both English learners and Students with Disabilities
• Smarter Balanced will also continue working with educators and experts in the field to design and test the assessment system
• Learn more online:– http://www.smarterbalanced.org/parents-students/support-for-
under-represented-students/24
TOM TORLAKSONState Superintendent of Public Instruction
Scoring Technology• Templates• Optical Scanning
– Scantron
– Electronic image based scoring (E.g. Pearson e-Pen)
– Scan to Score
• Traditional machine scoring– Dichotomous (correct/incorrect) scoring most common
– Exact word, number, or grid matches
– No partial credit
• Automated Scoring– Allows scoring of short answer and essay questions
– Require set of human scored papers to develop the scoring model
– Can give partial credit, or multiple point scores
25
TOM TORLAKSONState Superintendent of Public Instruction
How Automated Scoring Works
• Uses a set of human scored examples to develop a statistical model used to analyze answers
• Generally examine overall form and specific combinations of words
• Has an extensive library of possible meanings for words
26
TOM TORLAKSONState Superintendent of Public Instruction
What can be scored?
• Written responses– Prompt Specific Essays– Prompt Independent Essays– Short Answers– Summaries
• Spoken language – Correctness– Fluency
• Responses to simulations– Diagnosis of a patient’s illness– Landing a plane
27
TOM TORLAKSONState Superintendent of Public Instruction
How good is automated scoring?
• Consistent with the scores from expert human graders
• The way automated scores are produced is understandable and meaningful
• Fair• Validated against external measures in the
same way as is done with human scoring• The impact of automated scoring on reported
scores is understood
According to ETS, Pearson and the College Board in the recent, report “Automated Scoring for the Common Core Standards:”
28
TOM TORLAKSONState Superintendent of Public Instruction
Autoscoring Performance Response Assessment
Prompt Material
N Machine-Human
Correlation
Human-Human
Correlation
Source
Written 81 published essay prompts (grade 6-12)
400 0.89 0.86 Prentice Hall
18 research-leveled essay
prompts (grade 4-12)
635 0.91 0.91 MetaMetrics
5 synthesizing memos from
multiple sources
1239
0.88 0.79 Council for Aid to
Education
Spoken 2000 spoken English items
50 0.97 0.98 Balogh & et al. (2005)
3000 spoken Arabic items
134 0.97 0.99 Bernstein et al. (2009)
9 Oral Reading Fluency Passage
Grades 1-5
248 0.98 0.99 Downey et al. (2011)
Source: Streeter et. al. Pearson’s Automated Scoring of Writing, Speaking, and Mathematics, Pearson, May 2011 29
Example Essay Feedback
Source: Streeter et. al. Pearson’s Automated Scoring of Writing, Speaking, and Mathematics, Pearson May 2011
30
Data Requirements for Various Types of Automated Scoring
31
I tem Type Response Length in
Words
Typical Data Requirements for
development
Measures Returned
Prompt-Specific Essays
100-500 200-250 double-scored student essays
Overall score, trait scores, grammar & mechanics feedback
Prompt I ndependent Essays
(general models)
100-500 Approximately 1000 essays per grade
Overall score, select trait scores, grammar & mechanics feedback
Short Answers ~10-60 500 double-scored student answers
Total or partial-credit content score
Summaries 50-250 Readings to be summarized divided by major sections
Content coverage score for each section; checks copying, length, redundancy and irrelevance.
Source: Streeter et. al. Pearson’s Automated Scoring of Writing, Speaking, and Mathematics, Pearson, May 2011
TOM TORLAKSONState Superintendent of Public Instruction
Alternate Assessment Participation
• California recently joined the National Center and State Collaborative (NCSC) as a Tier II state
• Representing a Tier II state, the California team will:
– Dedicate a staff member to coordinate the work– Work directly with members of the Special Education Administrators of
County Offices of Education (SEACO) and with directors of special education local plan areas (SELPA) to build a community of practice
– Meet directly with the field implementers every other month with technology supported meetings in between and as needed
– Deliver electronically to California stakeholders the comprehensive curriculum, instruction, and professional development modules available from the NCSC on the CCSS expected by fall 2012
32
TOM TORLAKSONState Superintendent of Public Instruction
STAR In-Transition Activities
• Considerations– SBAC– CAT– ELA and Math Only– System – Formative, Interim, and
Summative
• Specific Planned Activities
33
TOM TORLAKSONState Superintendent of Public Instruction
Re-take option
Optional Interim assessment system—
Summative assessment for accountability
Last 12 weeks of year*
DIGITAL CLEARINGHOUSE of formative tools, processes and exemplars; released items and tasks; model curriculum units; educator training; professional development tools and resources; scorer training modules; and teacher collaboration tools.
Scope, sequence, number, and timing of interim assessments locally determined
* Time windows may be adjusted based on results from the research agenda and final implementation decisions.
Source: http://www.ets.org
Computer AdaptiveAssessment andPerformance Tasks
PERFORMANCETASKS
• Reading• Writing• Math
END OF YEARADAPTIVE
ASSESSMENT
English Language Arts and Mathematics, Grades 3–8 and High School
Computer AdaptiveAssessment andPerformance Tasks
BEGINNING OF YEAR
END OF YEAR
INTERIM ASSESSMENT INTERIM ASSESSMENT
34
Formative Assessments
TOM TORLAKSONState Superintendent of Public Instruction
35
Computer Adaptive Testing
TOM TORLAKSONState Superintendent of Public Instruction
36
Assessments to Consider (by content area and grade level)
TOM TORLAKSONState Superintendent of Public Instruction
Relationship of Assessments
37
Perie, Marion, Gong, Wurtzel, 2007
TOM TORLAKSONState Superintendent of Public Instruction
3838
STAR In-Transition Activities
Alignment to Common Core State Standards
•Planning to provide CCSS-aligned results on STAR Student Reports
•Planning to align CST released test questions (RTQs) with California’s Common Core State Standards (CCSS)
TOM TORLAKSONState Superintendent of Public Instruction
39
Proposed Web Site RedesignHome Page
TOM TORLAKSONState Superintendent of Public Instruction
40
Proposed Web Site RedesignCurrent Testing Standards—Filtering Feature
TOM TORLAKSONState Superintendent of Public Instruction
41
Proposed Web Site RedesignCurrent Testing Standards—Information About the Standard
TOM TORLAKSONState Superintendent of Public Instruction
42
Proposed Web Site RedesignCurrent Testing Standards—Sample Question
TOM TORLAKSONState Superintendent of Public Instruction
43
Proposed Web Site RedesignCurrent Testing Standards—Related CCSS Information
TOM TORLAKSONState Superintendent of Public Instruction
44
Proposed Web Site RedesignCommon Core—Filtering Feature
TOM TORLAKSONState Superintendent of Public Instruction
45
Proposed Web Site RedesignCommon Core—Standards Information
TOM TORLAKSONState Superintendent of Public Instruction
46
Proposed Web Site RedesignCommon Core—Comparison to Current Testing Standards
TOM TORLAKSONState Superintendent of Public Instruction
47
Proposed Web Site RedesignCommon Core—Example Related Sample Question
TOM TORLAKSONState Superintendent of Public Instruction
Contact Information• Patrick Traynor, PhD, Director
Assessment Development and Administration Division – E-mail: [email protected]
• Jessica Valdez, Administrator, Transition OfficeAssessment Development and Administration Division– Phone: 916-319-0332– E-mail: [email protected]
• Jessica Barr, SBAC and Reauthorization Lead Consultant, Transition OfficeAssessment Development and Administration Division– Phone: 916-319-0364– E-mail: [email protected]
48