transitioning to the smarter balanced assessment system north carolina superintendent association...

Transitioning to the Smarter Balanced Assessment System

North Carolina Superintendent Association

Susan GendronSenior Fellow

International Center June 17 2012

Key to Effective Instruction Is Alignment

• Organizational Leadership• Instructional Leadership• Teaching

Teaching

Organ

izational

Lead

ersh

ipInstructional

Leadership

Student Achievement

What will our students need to:

Know Do

4

PISA 2009

1 Shanghai-China 556

2 Korea 539

3 Finland 536

4 Hong Kong-China 533

5 Singapore 526

6 Canada 524

7 New Zealand 521

8 Japan 520

9 Australia 515

10 Netherlands 508

17 United States 500

20 Germany 497

21 Ireland 496

22 France 496

25 United Kingdom 494

33 Spain 481

43 Russian Federation 459

48 Mexico 425

53 Brazil 412

57 Indonesia 402

Overall Reading

Scale

Significantly Above OECD Average

Not Significantly Different

(OECD Average 493)

Significantly below OECD Average

PISA 2009

Overall Math Scale

Significantly Above OECD Average

Not Significantly Different

(OECD Average 496)

Significantly below OECD Average

1 Shanghai-China 600

2 Singapore 562

3 Hong Kong-China 555

4 Korea 546

6 Finland 541

9 Japan 529

10 Canada 527

11 Netherlands 526

13 New Zealand 519

15 Australia 514

16 Germany 513

22 France 497

28 United Kingdom 492

31 United States 487

32 Ireland 487

34 Spain 483

38Russian Federation

468

51 Mexico 419

57 Brazil 386

61 Indonesia 371

PISA 2009

Overall Science

Scale

Significantly Above OECD Average

Not Significantly Different

(OECD Average 501)

Significantly below OECD Average

1 Shanghai-China 575

2 Finland 554

3 Hong Kong-China 549

4 Singapore 542

5 Japan 539

6 Korea 538

7 New Zealand 532

8 Canada 529

10 Australia 527

11 Netherlands 522

13 Germany 520

16 United Kingdom 514

20 Ireland 508

23 United States 502

27 France 498

36 Spain 488

39 Russian Federation 478

50 Mexico 416

53 Brazil 405

60 Indonesia 383

Reading Risk

Mapping State Proficiency Standards onto NAEP Scales, IES August 2011

NC

ProficiencyProficiencyGrade 4 Reading 2009Grade 4 Reading 2009

ProficientRequired

NAEP Score

North Carolina 69 % 204

Massachusetts 54 % 234

Missouri 47 % 229

New Hampshire 74% 211

Oregon 84 % 177

Washington 73 % 205

Vermont 70% 214

Reading Risk

Mapping State Proficiency Standards onto NAEP Scales, IES August 2011

NCNC

ProficiencyProficiencyGrade 8 Reading 2009Grade 8 Reading 2009

ProficientRequired

NAEP Score

North Carolina 66% 246

Minnesota 67% 259

Missouri 50% 267

Vermont 69% 259

Oregon 69% 250

Washington 68% 253

Math Risk

Mapping State Proficiency Standards onto NAEP Scales, IES August 2011

NCNC

ProficiencyProficiencyGrade 4 Mathematics 2009Grade 4 Mathematics 2009

ProficientRequired

NAEP Score

North Carolina 81 % 220

Massachusetts 48 % 255

Hawaii 50 % 239

New Hampshire 73 % 237

New Mexico 77 % 224

Washington 52 % 243

Missouri 45 % 246

Math Risk

Mapping State Proficiency Standards onto NAEP Scales, IES August 2011

NCNC

ProficiencyProficiencyGrade 8 Mathematics 2009Grade 8 Mathematics 2009

ProficientRequired

NAEP Score

North Carolina 80 % 253

Massachusetts 49 % 300

Missouri 47 % 287

Hawaii 39 % 286

Minnesota 58 % 287

Oregon 71 % 266

Washington 51 % 288

Elbow partner

• What are your risks?• How will you prepare your school for

a potential dip in scores?• Do you have a communication

strategy and a clear vision for the future?

16

Organ

izational

Lead

ersh

ip

StudentAchievement

Culture

Organizational Leadership

Levin and Elmore• Everyone needs to collaborate to

ensure that daily teaching and learning practices are the focus of the school

• All responsible for success• Principals and teachers are

fundamentally evaluators• Leaders responsible for cultural

changes – by displacing specific norms, structures, and processes by others

How will you create a culture in your schools of ALL students “college and career ready”?

20

Rigor and Relevance

Teaching

Rigor/Relevance Rigor/Relevance For For

All StudentsAll Students

22

A B

DC

1.1. AwarenessAwareness2.2. Comprehension Comprehension 3.3. ApplicationApplication4.4. AnalysisAnalysis5.5. Synthesis Synthesis 6.6. EvaluationEvaluation

Knowledge TaxonomyKnowledge Taxonomy

23

Application ModelApplication Model1.1. Knowledge in one disciplineKnowledge in one discipline

2. Application within one 2. Application within one disciplinediscipline

3. Application across disciplines3. Application across disciplines

4. Application to real-world 4. Application to real-world predictable situationspredictable situations

5. Application to real-world 5. Application to real-world unpredictable situationsunpredictable situations

24

25

Curriculum-Instruction-Assessment • What should students know and to be able to do?

•What should students learn?

•What should students be taught?

• What are students being taught? • How are students being taught?

• What have students learned? • What haven’t students learned?

Curriculum

Students

Instruction Assessment

Curriculum-Instruction-Assessment Connections

ObservationObservation InterpretationInterpretation

CognitionCognition

“AssessmentTriangle”

Evidence-Based Design Framework

Models of Cognition

• Describe how students acquire knowledge and develop competence in a particular area

• Reflect recent and credible scientific evidence of typical learning processes and informed experiences of expert teachers

• Describe typical learning progression toward competence, including milestones (benchmarks)

31

Observation Models

• A set of specifications for assessment tasks that will elicit illuminating responses from students

• The tasks or situations are linked to the cognitive model of learning and should prompt students to say, do, or create something that provides evidence to support inferences about students’ knowledge, skills, and cognitive processes

Interpretation

• Interpretations use the evidence from observations to make claims about what students understand and can do

• Claims– Frame a manageable number of learning goals

around which instruction can be organized– Guide the specification of appropriate evidence– Provides a basis for meaningful reporting to

different interested audiences

An Overview of SBAC’s Approach

Content Specifications …– Create a bridge between standards and assessment and,

ultimately, instruction

– Organize the standards around major constructs & big ideas

– Express what students should learn and be able to do

Each claim is described for assessment

• Rationale for each claim– Why is this learning goal important for College & Career

Readiness (CCR)?– What does the research say about learning in this area?

• What does ‘sufficient’ evidence look like?– What types of items/tasks?– What content/texts will be emphasized?

• What are some suggested reporting categories?

Summative Assessment Targets

• Indicate proposed prioritized content for the summative assessment- link CCSS to the kinds of items/tasks students will respond to

• Show how one or more (or parts) CCSS addresses the target – ‘bundles’ CCSS (examples on next slide)

– Standards or parts of standards that relate to same type of understanding & comparable rigor/DOK demands

– Several similar CCSS from different strands

Assessment Claims for English Language Arts/Literacy

Reading

Writing

Speaking/Listening

Research/Inquiry

(a/o Round 2 – released 9/20/11)

Draft Overall Assessment Claims for English Language Arts/Literacy

OVERALL 3-8

OVERALL 9-12

(a/o Round 2 – released 9/20/11)

Students can demonstrate

progress toward college and

career readiness in English

language arts and literacy.

Students can demonstrate

college and career readiness in

English language arts and

literacy.

English Language Arts and Literacy Standards “Roadmap”

READING WRITINGSPEAKING & LISTENING LANGUAGE

10 Anchor Standardsfor College and Career Readiness

10 Anchor Standardsfor College and Career

Readiness

6 Anchor Standards for CCR

6 Anchor Standards for CCR

ELA Standards

K-12

ELA Standards

K-12

ELA Standards

K-12

Literacy Standards

6-12

ELA Standards

K-12

Literacy Standards

6-12

Literary Text

Hist. / S.S.

Sci. / Tech Subj.

Inform Text

1K

2

3

4

5

6

7

8

9-10

11-12

9-10

11-12

6-8

9-10

11-12

6-8

11-12

1K

2

3

4

5

6

7

8

9-10 9-10

11-12

6-8

1K

2

3

4

5

6

7

8

9-10

11-12

1K

2

3

4

5

6

7

8

9-10

11-12

K

1

2

3

4

5

9-10

11-12

6

7

8

Found-ational Skills

1

2

3

4

5

K → → → → → → → → → → →→ → → → → → → → → → →→ → → → → → → → → → →

→ → → → → → → → → → →

→ → → → → → → → → → →→ → → → → → → → → → →

→ → → → → →

→ → → → → →

→ → → → → →

→ → → → → →

→ → → → → →

→ → → → → →

40

English Language Arts and Literary Shifts

engageny

Smarter BalancedAssessment Consortium

The Assessment Challenge

How do we get from here...

...to here?

All studentsleave high

school college and career

ready

All studentsleave high

school college and career

ready

Common Core State

Standards specify K-12 expectations

for college and career

readiness

Common Core State

Standards specify K-12 expectations

for college and career

readiness...and what can an

assessment system do to help?

Next Generation Assessments

• More rigorous tests measuring student progress toward “college and career readiness”

• Have common, comparable scores across member states, and across consortia

• Provide achievement and growth information to help make better educational decisions and professional development opportunities

• Assess all students, except those with “significant cognitive disabilities”

• Administer online, with timely results

• Use multiple measures

Source: Federal Register / Vol. 75, No. 68 / Friday, April 9, 2010 pp. 18171-85

Smarter BalancedBackground

A National Consortium of States

• 28 states representing 44% of K-12 students

• 21 governing, 7 advisory states

• Washington state is fiscal agent

Smarter BalancedApproach

Seven Key Principles

1. An integrated system2. Evidence-based approach3. Teacher involvement4. State-led with transparent

governance5. Focus: improving teaching and

learning6. Actionable information – multiple

measures7. Established professional standards

A Balanced Assessment System

Common Core State Standards specify

K-12 expectatio

ns for college and

career readiness

Common Core State Standards specify

K-12 expectatio

ns for college and

career readiness

All students

leave high

school college

and career ready

All students

leave high

school college

and career ready

Teachers and schools have information and tools

they need to improve

teaching and learning

Interim assessments Flexible, open,

used for actionable feedback

Summative assessments

Benchmarked to college and career

readiness

Teacher resources for

formative assessment

practicesto improve instruction

A Balanced Assessment System

School Year Last 12 weeks of the year*

DIGITAL CLEARINGHOUSE of formative tools, processes and exemplars; released items and tasks; model curriculum units; educator training; professional development tools and resources; scorer training modules; and teacher collaboration tools.

English Language Arts and Mathematics, Grades 3-8 and High School

Computer Adaptive

Assessment andPerformance

Tasks

Computer Adaptive

Assessment andPerformance

TasksScope, sequence, number and timing of interim assessments locally determined

*Time windows may be adjusted based on results from the research agenda and final implementation decisions.

SummativePerformance

TasksFor

Accountability• Reading• Writing• Math Re-take

option

Summative EndOf Year AdaptiveAssessment forAccountability

Optional Interim

Assessment

Optional Interim

Assessment

Using Computer Adaptive Technology for Summative and

Interim Assessments

Technology

IT readiness survey: Available January 2012

IT readiness survey: Available January 2012

System architecture: Available January 2012

System architecture: Available January 2012

Vendors start building the system: February 2012

Vendors start building the system: February 2012

Improve the technology throughout pilot and field test: 2012 and 2013

Improve the technology throughout pilot and field test: 2012 and 2013

Technology

New Purchases (as of April 2012)– Hardware – 1GHz processor, 1 GB RAM, 9.5 inch

screen size (10 inch class), screen resolution of 1024 x 768

• Must have tools to temporarily disable features ( i.e.. web browser, Bluetooth connections, application switching)

– Operating Systems – Windows 7, Mac 10.7, Linux (Ubuntu 11.10, Fedora 16), Chrome, iOS, Android 4.0

• Will consider older versions and Linux after survey data and cognitive labs

– Network – Must be able to connect to the Internet

Technology

New Purchases (as of April 2012)

– Form Factors –Desktops, laptops, netbooks, thin-client, and tablets ) iPad, Windows and Android) that meet the above specifications

– Additional Accessories –• Headphones may be required for audio support• Physical keyboards (as opposed to virtual) and/or

mice may be required for use with tablets

Technology

Anticipate by 2016-17 up to 25% of the mathematics assessment will comprise items/tasks requiring student-produced illustrations or calculations.

Students will need access to tablet (or other device) that employs a stylus for user input

Future required use will be described in the August 2012 guidelines

Technology Plan

• What is your vision?

• Designing an infrastructure – CoSN, SETDA, Horizon report, Maine Learning Technology Initiative

• Define the type of learning you want for your students.

• Professional Development – Teachers, Principals, Technology Coordinators

57

2012 Horizon Time to Adoption

• One Year or Less:– Mobiles and Apps– Tablet Computing

58

2012 Horizon Time to Adoption• Two to Three Years:

– Game-Based Learning– Personal Learning Environments

Horizon Report 2012 44

Personal Learning Environments

Horizon Report 2012

2012 Horizon Time to Adoption

• Four to Five Years:– Augmented Reality– Natural User Interfaces

61Horizon Report 2012

Key Trends (Horizon Report 2012)• Abundance of resources• Cost of technology dropping, • More common for students to bring

their own device• Schools including online learning,

hybrid learning and collaborative models

62

Key Trends (Horizon Report 2012)• One to one computing spreading

across the world and US• People expect to work, learn, and

study whenever• Technology profoundly affects the

way we work, collaborate, communicate and succeed

• New emphasis on challenged-based and active learning

63

Challenges (Horizon Report 2012)

• Demand for personal learning is not supported by current technology

• Lack of digital media literacy for teachers• Economic pressures – new models of

education – unprecedented competition• Institutional barriers • Blending of formal and informal learning

64

Challenges (Horizon Report 2012)

• Learning that incorporates real life learning – is undervalued

• Many activities related to learning take place outside the walls of classrooms

• Putting 21st century technology into 19th century schools is a major undertaking

• We don’t use digital media for formative assessment

65

What is on your Horizon?

• Download the NMC 2012 Horizon Report

• www.CoSN.org/Horizon

66

Assessment System Components

Assessment System Components

• Optional comprehensive and content-cluster assessment to help identify specific needs of each student

• Can be administered throughout the year

• Provides clear examples of expected performance on Common Core standards

• Includes a variety of question types: selected response, short constructed response, extended constructed response, technology enhanced, and performance tasks

• Aligned to and reported on the same scale as the summative assessments

• Fully accessible for instruction and professional development

Assessment System Components

The use of

performance

measures has been

found

to increase the

intellectual challenge

in classrooms

and to support

higher-quality

teaching.

- Linda Darling-Hammond and Frank Adamson, Stanford University

“

”

Performance Task Guidelines

• Integrate knowledge and skills across multiple standards or strands

• Measure capacities such as depth of understanding, research skills, complex analysis, and identification/providing of relevant evidence

• Require student-initiated planning, management of information and ideas, interaction with other materials

70

Performance Task Guidelines

• Require production of more extended responses (e.g., oral presentations, exhibitions, product development, in addition to more extended written responses which might be revised and edited

•Reflect a real-world task and/or scenario-based

•Allow for multiple approaches

•Represent content that is relevant and meaningful to students

71

Performance Task Guidelines

• Allow for multiple points of view and interpretations

• Require scoring that focuses on the essence of the task

• Be feasible for the school/classroom environment

72

Performance Task Guidelines

• Allow for demonstration of important knowledge & skills, including those that address 21st century skills such as critically analyzing, synthesizing media texts

73

Grade 08 ELA Sample CR Item

• Assessment Target- 11. REASONING & EVALUATION: Apply reasoning and a range of textual evidence to justify inferences or interpret author’s presentation of information (author’s line of reasoning; point of view/purpose; relevance of evidence and/or elaboration to support claims, concepts, ideas)

• Standards: RI-6, RI-8, RST 6

• DOK - 3

Item Prompt

Based on the text, what inference can be made about how tests and testing should occur to ensure an accurate measurement of overall water quality? Explain your inference using details from the text.

Questions to be answered in the speech

•In two sentences, use your own words to tell what a wonder is and explain how a person who helps others can be considered a wonder. •Write 2 or 3 sentences identifying a personal quality that both Mickey and Ana display. Give an example from both the video and the interview to support your answer.

Questions to be answered in the speech

• Tell which website you think would be most useful for learning about another young person that is a wonder because he or she helps others. Cite the web site by giving the web address. Use details from the website to support your answer.

Performance Tasks Sources (2 articles, 3 videos)

Article 1 •Metro Daily News June, 2011

The Rise of the Robot Pet by Elena Soto

Video 1: Fugitsu’s cute teddy-bear robot shows what it can do, May 2010. http://www.youtube.com/watch?v=AwWeN1ARy74--This is an introduction to the Fujitsu robot teddy bear. (1:58)

Video 2: Pleo: Robot, pet or both? December 2007 http://www.youtube.com/watch?v=F6LCEFr8SxQ--The host of a technology show provides an introduction to a specific robot pet. (3:10)

Article 2 Technology Trends Quarterly Fall, 2010 Love In the Time of Robots by Frank Mullin

Video 3: Maya’s Human Interaction – Sensors (A Genibo-QD film), February 2010. http:// www.youtube.com/watch?v=9b4jx5RzqAk --The Genibo robot dog displays some of its capabilities. (1:49)

Scoring

1.Establishing a claim2.Presentation of relevant evidence3.Organization4.Conventions5.Language, tone, purpose, audience6.Content

Draft Assessment Claims for Mathematics (a/o Round 2 – released 12/9/11)

A Schematic representation of CCSSM content

“Hurdles Race”

Think of the Content involved

• Interpreting distance-time graphs in a real-world context

• Realizing “to the left” is faster

• Understanding points of intersection in that context (they’re tied at the moment)

• Interpreting the horizontal line segment

• Putting all this together in an explanation

Think of the Practices involved

• Make sense of problems and persevere in solving them.

• Reason abstractly and quantitatively.

• Construct viable arguments…

• Model with mathematics.

• Use appropriate tools strategically.

• Attend to Precision.

• Look for and make use of structure.

• Look for and express regularity in repeated reasoning.

Item Exemplars: Technology Enhanced and Constructed

Response

Item Exemplars: Technology Enhanced and Constructed

Response

Item Exemplars: Technology Enhanced and Constructed

Response

Few initiatives are

backed by evidence

that they raise

achievement.

Formative

assessment is one of

the few approaches

proven to make a

difference.- Stephanie Hirsh, Learning Forward

Assessment System Components

“

”

145

Definition:Assessment that takes place continuously during the course of teaching and learning to provide teachers and students with feedback to close the gap between current learning and desired goals.

Assessment Reform Group, 2002; Bell & Cowie, 2001; Black et al., 2003; Black & Wiliam, 1998; OECD, 2005; Sadler, 1989; Shepard, 2000)

Advantages of Formative Assessment

• Students learn faster• Teachers know what students

already know & adjust instruction

• Students aware of progress• Most powerful moderator in

student achievement• Works for at risk students

Formative Assessment Strategies

(Black, Wiliam,1998; Sadler, 1998; Stiggins, 2007;Heritage, 2007)• Pre-assessing students

• Sharing Learning goals with students

• Co-creating classroom discourse & questioning

• Rich & challenging tasks elicit student response

• Identifying gaps

Formative Assessment Strategies

(Black, Wiliam,1998; Sadler, 1998; Stiggins, 2007;Heritage, 2007)

• Providing feedback/how to improve

• Self-assessments

• Peer- assessments

• Opportunities to close the gap

• Celebrations

Feedback

Strategies

• Traffic Light feedback• Gallery Walk• Portfolio• Concept Map• Ticket out the door

What do students say

• Class discussion• Debate• Oral presentation• story/event telling• Agree/disagree• Choral reading• Think-Pair-Share• You’re the Judge• Ask a question• Make a Statement• Radio Show

• Small group talk• Play/drama• Reciting a

poem/speech• Panel discussion• Music• Interviews• Think aloud• Answer specific• Podcasts• Read aloud• Other____

Take a Three Minute Pulse

• After 10-15 discussion, reading, lecture• Reflect, discuss what they learned

using higher order thinking skills • Suggested questions: (Marzano)

– How does this information relate to you?– How does what we’ve just learned relate to..– How is what we just learned similar or different to– Identify one thing you knew and one thing that was

new to you…

Exit Sheet

One Minute Response

Grades Supported

Grades Summative Interim(Optional)

Formative Tools and Professional Learning

(Optional)

✔ ✔ ✔

1-2 Performance Tasks as Required

to Cover CCSS

✔

EOC and Comprehensive

✔

✔ ✔

EOC and Comprehensive

✔

Optional ✔

EOC and Comprehensive

✔

3 8

9 10

11

12

Data are only useful if

people are able to

access, understand

and use them… For

information to be

useful, it must be

timely, readily

available, and easy to

understand.

- Data Quality

Campaign

Assessment System Components

• Static and dynamic reports, secure and public views

• Individual states retain jurisdiction over access and appearance of online reports

• Dashboard gives parents, students, practitioners, and policymakers access to assessment information

• Graphical display of learning progression status (interim assessment)

• Feedback and evaluation mechanism provides surveys, open feedback, and vetting of materials

“

”

Support for Special Populations

• Accurate measures of progress for students with disabilities and English Language Learners

• Accessibility and Accommodations Work Group engaged throughout development

• Outreach and collaboration with relevant associations

Common-Core Tests to Have Built-inAccommodations

- June 8, 2011

“

”

Timeline

Formative Processes, Tools, and Practices Development Begins

Formative Processes, Tools, and Practices Development Begins

Writing and Review of Pilot

Items/Tasks (including

Cognitive Labs and Small-Scale Trials)

Writing and Review of Pilot

Items/Tasks (including

Cognitive Labs and Small-Scale Trials)

Field Testing of Summative and

Interim Items/Tasks Conducted

Field Testing of Summative and

Interim Items/Tasks Conducted

Content and Item

Specifications

Development

Content and Item

Specifications

Development

Pilot Testing of Summative and

Interim Items/Tasks Conducted

Pilot Testing of Summative and

Interim Items/Tasks Conducted

Preliminary Achievement

Standards (Summative) Proposed and Other

Policy Definitions Adopted

Preliminary Achievement

Standards (Summative) Proposed and Other

Policy Definitions Adopted

Operational Summative Assessment

Administered

Operational Summative Assessment

Administered

Procurement Plan

Developed

Procurement Plan

Developed

Writing and Review of Field Test Items/Tasks

(throughout the school year)

Writing and Review of Field Test Items/Tasks

(throughout the school year)

Final Achievement Standards

(Summative) Verified and

Adopted

Final Achievement Standards

(Summative) Verified and

Adopted

Common Core State Standards Adopted

by All Member States

Common Core State Standards Adopted

by All Member States

Summative Master Work Plan Developed and Work Groups

Launched

Summative Master Work Plan Developed and Work Groups

Launched

Find Out More

The Smarter Balanced Assessment Consortium

can be found online athttp://www.smarterbalanced.org