What is High-Quality Assessment? Linking Research with Practice

Santa Clara County Office of Education June 23, 2014

Karin K. Hess, Ed.D.khess@nciea.org or

karinhessvt@gmail.com

Presentation Overview• Clarify understandings of cognitive rigor/DOK –

using sample assessments & rubrics• Use the Hess Validation Tools & Protocols

(Module 3) to examine technical criteria for high quality assessments: Formative & Performance

• Review tools & strategies to discuss & plan future assessment activities and support to teachers

• Karin’s coaching tips…

Rubric Design & Formative Tools• Revisit Handout from this morning: “What I

need to do” rubric (citing evidence of proficiency)

• Handout 2a: Find a half• Handout 2b: Hess Cognitive Rigor Matrix –

Math-Science• Handout 2c: What will this formative

assessment uncover?• Work in small groups to analyze the assessment

What do we mean by high-quality performance assessment?

• At your tables, brainstorm examples of performance assessments – any content area (e.g., arts, writing, science) or real world assessments (driver’s test, marriage planning, etc.)

• Have a recorder write them down• You have only 3 minutes

Turn & talk: Select one PA from your list and answer these questions:

1. What is it actually assessing (skills & concepts)?

2. What makes it a PA? 3. What evidence is captured in the assessment

that distinguishes poor from best performances?

4. What makes it a “good” performance assessment?

5. You have 5 minutes

Let’s generalize…

• With regard to skills & concepts assessed ______

• What makes something a PA? ______• The kind of evidence that will distinguish

poor from exemplary performances _______ • What makes it a “good” performance

assessment? _________

What we know (from research) about High Quality Assessment:

• Defined by agreed-upon standards/ expectations • Measures the individual’s learning & can take

different forms/formats• Measures the effectiveness of instruction and

appropriateness of curriculum • Is transparent: – Students know what is expected of them and how they will

be assessed– Assessment criteria are clear and training is provided to

educators and reviewers/raters.• Communicates information effectively to students,

teachers, parents, administration and the public at large

Simply put, HQ assessments have…• Clarity of expectations• Alignment to the intended expectations (skills, concepts)• Reliability of scoring and interpretation of results • Attention to the intended rigor (tasks & scoring guides)• Opportunities for student engagement & decision

making• Opportunities to make the assessment “fair” & unbiased

for all• Linked to instruction (opportunity to learn)

2. The DOK Matrix Instructional Paths

Each standard has an assigned Depth of Knowledge.

The DOK determines the cognitive level of instruction.

Recall, locate basic facts, definitions, details, events

Select appropriate words for use when intended meaning is clearly evident.

DOK 1Recall and Reproduction

Remember

Understand

DOK 2Skills and Concepts

Apply

Explain relationshipsSummarizeState central idea

Use context for word meaningsUse information using text features

DOK 3Reasoning and

Thinking

Analyze

Analyze or interpret author’s craft (e.g., literary devices, viewpoint, or potential bias) to critique a text

Explain, generalize or connect ideas using supporting evidence (quote, text, evidence)

.

Cite evidence and develop a logical argument for conjectures based on one text or problem

Evaluate

Use concepts to solve non-routine problems and justify

DOK 4Extended Thinking

Synthesize across multiple sources/ textsArticulate a new voice, theme, or perspective

Evaluate relevancy, accuracy and completeness of information across texts or sources

Analyze multiple sources or multiple textAnalyze complex abstract themes

Devise an approach among many alternatives to research a novel problem

-Explain how concepts or ideas specifically relate to other content domains.

Develop a complex model or approach for a given situationDevelop an alternative solution

.Create

Instruction & Assessment Decisions…

9

Selected ResponseSelected ResponseSelected ResponseSelected Response

Constructed ResponseConstructed ResponseConstructed ResponseConstructed ResponsePerformance Performance

TasksTasks

Performance Performance TasksTasks

GOAL: Each “validated” assessment will demonstrate:

• Clarity of expectations for the student and teacher(s)

• Alignment (task & scoring) to the intended standards: content & performance/DOK

• Provide opportunities for student engagement• Provide opportunities to make the assessment

“fair” & unbiased for ALL students

First we consider alignment…

• It’s really about validity – making decisions about the degree to which there is a “strong match” between grade level content standards + performance and the assessment/test questions/tasks

• And making valid inferences about learning resulting from an assessment score

“Validity is a matter of degree, rather than all or none.”Robert Lynn, 2008

Alignment (validity) Questions:

• Is there a strong content match between assessment/test questions/tasks and grade level standards?

• Are the test questions/tasks (and the assessment as a whole) more rigorous, less rigorous, or of comparable rigor (DOK) to grade level performance standards?

Task Validation Protocol Handout # 3(K. Hess, 2013)

• Table Groups review the technical criteria and descriptions on pages 3-4 in the protocol at your tables

• What’s one aspect you feel you (or teachers you work with) now do well in most local assessments?

• What’s one aspect you feel you (or your teachers) need to understand more deeply as you work with them?

Uses of the assessment task validation tools & protocols

• Develop new assessments • Analyze existing assessments• Validate a revised assessment or new

assessment prior to broader administration (or purchase)

• Provide OBJECTIVE feedback to assessment developers

• Promote collaboration and a shared understanding of high quality assessment

Local Validation Teams represent the diversity of the school

• Administrator/Leader/Coach• All* content areas represented• All/most grade levels (grade spans)*

represented• PLUS Representation from special education,

fine arts, HPE, CTE, foreign language, ELL, etc.

*decisions may differ depending on school configurations and staffing, but diversity in teams is critical, especially including special educators

Frequency of Validations?• Initially learning & debriefing the process together

serves as calibration - so everyone is on the same page – “developing a shared understanding” of what high quality assessment looks like

• School teams set up their schedules – once each month, every other month, as needed, highest priority, etc.

• Team members may rotate on-off so more (all) staff are involved over time

Getting ready for validation• Grade level or department teams develop the assessments

using the Basic Validation Protocol (e.g., a gr 2 team might develop a common math assessment for all gr 2 classes/schools)

• Developers put the assessment on the local (school/district) validation calendar

• Validation teams prioritize order of validations – common assessments, major assessments first, second round review after getting feedback, etc.

Validation Materials• Each team member needs (electronic) validation

protocols (Handout: Module 3, pages 3-4)

• Each person needs a copy of the cover page with the assessment and scoring rubric/answer key (Handout: Module 3, pages 5-6)

• There may be additional materials – e.g., anchor papers, examples that do not need to copied for everyone but may be helpful to see during the review

• Each person needs a content specific DOK reference sheet (Handout: Module 1, tools #1, #2, or # 3)

Validation Protocols [1]

• Each time, preview norms for working together– I am…– I am NOT…

• Choose a recorder – to keep an electronic record & provide a copy of feedback for the assessment developers

• Date and list validation panel names on the “official copy” (this can be set up ahead of time)

• Individually, take 5-10 minutes to read through & make notes before any discussion

Sample norms (Source: adapted from Powell, WY)

I AM • Keeping electronic devices on vibrate/off• Listening to understand other points of

view• Respecting everyone as a professional• Focusing on the issues• Avoiding side conversations• Encouraging everyone having a turn to

speak• Refraining from judgmental statements• Representing the best interests of all

students • Asking clarifying questions• Demonstrating a commitment to the

process (attending meetings, on time, etc.)

• Others?

I AM NOT• Using killer phrases• Preparing my next remark instead

of listening• Sounding apologetic• Engaging in unrelated activities• Using negative gestures/body

language• others?

Optional -Validation Protocols [2]

• Should the authors present the task at the start? (especially if 2nd round) – there are pros & cons to this– Go over what is on the cover page/what is included and

what the purpose of the assessment is– 2-5 minutes to explain the materials in the packet – no

interruptions from validation panel– Panel then asks any clarifying questions only– The is NOT for depth of understanding, just to know/clarify

what is there BEFORE silently reading & discussing

Validation Protocols [3]

• Make notes individually before discussion• Choose a task manager/ timekeeper to keep things

moving – reads each indicator on the Validation Protocols

• Have a process to reach consensus (fist 5, thumbs up, etc.)- be sure to involve each person!

• Choose 2 people to give feedback to the authors/developers & “rehearse” comments

• DEBRIEF! Did we honor norms? What went well/needs to be refined next time?

Giving Feedback• Use descriptive language, NOT judgmental

language• While you may wonder about instructional

pieces, comments/suggestions about instruction are probably not appropriate

• Your job is NOT to redo the assessment! Keep feedback crisp & to the point (e.g., pose a question)- it is the developer’s job to decide what to do next to strengthen the assessment tasks.

Giving Feedback (continued)

• Well-written, clear feedback guides assessment developers to make a stronger assessment in the end.

• Place your positive (and descriptive) comments under the feedback section (Module 3, page 7): What makes this a HQ (high quality) assessment?

Examples of Feedback (noted on page 7)

1. We were unable to locate…2. We think this might be DOK2, not DOK3 because…what do

you think?3. We were not clear what the student is expected to do or

to produce. Did you mean…?4. This might be better aligned to this standard …5. As hard as it will be, avoid saying “we liked…” This implies

you did not like other things and your job is NOT to like the assessment.

6. Include the “HQ” positives! The directions are clear; students have authentic choices; etc.

Debrief each time!

• Did the validation team honor the norms at all times?

• Do we need to modify/revise norms?• What went well?• What could have gone better?• What will we do differently next time?• Who/when will we meet with authors to give

feedback?