Understand and Explore Assessment Concepts

Relative to UDL

UDL InstituteCAST

Tracey E. Hall 2012

Operational Definition of Assessment

The gathering of information about a learner from his or her performance in a variety of tasks, subjects, and learning contexts to determine abilities and knowledge for the purpose of making educational decisions (Salvia & Ysseldyke, 2009).

Summative Evaluation: Measurement to determine subject performance at the end of a specific grade level, unit, year, or instructional episode. Mastery Measurement

Formative Evaluation: Assessment of progress toward a long-term goal or major objective, used primarily to diagnose what students have learned in order to plan further instruction (i.e., the on-going assessment of progress toward an objective).

Types of Assessment:

Summative Evaluation: Measurement to determine subject performance at the end of a specific grade level, unit, year, or instructional episode. Mastery Measurement

Formative Evaluation: Assessment of progress toward a long-term goal or major objective, used primarily to diagnose what students have learned in order to plan further instruction (i.e., the on-going assessment of progress toward an objective).

Types of Assessment:

Educational tests are: Indirect measure of a construct or a set of closely

related constructs.

Constructs are not observable

A test provides a sample of observable behaviors believed to be the product of the intended construct

Observable behaviors are used to make an inference about the construct

What is a test supposed to do?

2 perspectives

1.Access to the test

2.Access to the construct

Accessing Assessment

Accessing the Construct

Test construct refers to the concept or the characteristic that a test is designed to measure.

e.g., In a mathematics assessment, an item designed to evaluate students' ability to show equivalence has a test construct of numerical equivalency.

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999).

Test construct

Tests are designed to: Present information to stimulate the

construct.

Interact with the information as the construct is operating

Produce a response which becomes a physical product

Components of an assessment item

Tests are designed to: Present information to stimulate the

construct.

Interact with the information as the construct is operating

Produce a response which becomes a physical product

Components of an assessment item

How Items Function

Interact with Apply View Product Measure of Stimuli Construct of Construct Construct

If a challenge or barrier exists at any of these levels, inference about the construct will fail.

Present Information

Produce Response

Interact with

Construct

Inference to

Construct

AQuantita-

tiveScore

Michael Russell 2011

When there are construct irrelevant impediments, distractions, or barriers in the assessment methods it is essential to provide scaffolds, supports or accommodations in order to improve accuracy and validity, especially for individuals with disabilities.

Keep in mind, using those same scaffolds, supports or accommodations inappropriately – where they affect the construct relevant demands of the assessment – is likely to invalidate the measure.

Construct relevant refers to the factors (e.g., mode of presentation or response) that are relevant (related) to the construct that the test is intended to measure.

Identifying the symbols for a math calculation problem is a construct relevant change to the assessment item.

Construct relevant

Construct irrelevance is the extent to which test scores are influenced by factors (e.g., mode of presentation or response) that are irrelevant (not related) to the construct that the test is intended to measure.

For Elisa, a student with severe cognitive disabilities, changing the font size and increasing sizing of images helped her to see the assessment item for counting objects and doing so was construct irrelevant to assessing ability to count.

Construct Irrelevant

Accommodations, scaffolds, and supports should not be exclusive to the instructional environment but should be a part of the whole instructional episode including assessment.

The preservation of the distinction between what is construct relevant and what is construct irrelevant is essential when making decisions about accommodations, scaffolds, and supports in the assessment environment.

 

Envision taking a test:

Envision taking a test:

Ability to concentrate on the

problem is one of the

constituent parts of the

successful mental work

Envision taking a test:

Many students in the United States struggle in the areas of reading and writing, with difficulties emerging early and continuing into the secondary school years and beyond (National Center for Education Statistics, 2002, 2003, 2005, 2007). As student performance levels diverge, it becomes increasingly necessary to provide greater differentiation of instruction. Two approaches are particularly suited for helping teachers and practitioners to differentiate instruction.

Many students in the United States struggle in the areas of reading and writing, with difficulties emerging early and continuing into the secondary school years and beyond (National Center for Education Statistics, 2002, 2003, 2005, 2007). As student performance levels diverge, it becomes increasingly necessary to provide greater differentiation of instruction. Two approaches are particularly suited for helping teachers and practitioners to differentiate instruction.

Adapted presentation Flexible font size (magnification)

Color contrasts

Format, White space

Easy in a digital environment, no item interference. No impact on measuring the construct.

Only change is how the item is being displayed.

“These are no brainers in a computer environment” Mike Russell 09.

On paper, more difficult to make changes such as these.., for administration not student – lots of different papers

Alternate representations: in digital environment – relatively simple

water

agua

H2O

Multiple Representations of Information

Alternate means for Expression and Action: also easily accomplished in a digital environment

Engagement Example:Options for to read and respond.

Students are (a)engaged by

choice (b) often select

based on interest or prior knowledge

(c) perform better

Limitations of Accommodationsas Retrofits

Marginal efficacy Consider:◦ Extended time◦ Read-aloud accommodation vs. independent

reading and strategic test-taking May invalidate measurements

Consider:◦ Use of read-aloud for decoding task◦ Use of calculator for arithmetic task

Thus it is often difficult to level the playing field with accommodations, especially if the playing fields was initially un-level for all students.

Some Solutions: Consider Student Diversity During Test Development & Delivery

During Test Development◦ Ensure narrowly-defined constructs◦ Conduct bias review early◦ Ensure items are suitable for appropriate delivery

to all students

During Test Delivery• Provide multiple means of recognition, expression

& engagement• Ensure matching of classroom and assessment

supports

The cartoon above depicts United States frustration with A) the Good Neighbor policy B) Dollar DiplomacyC) the Spanish-American War D) the Bay of Pigs invasion

12th Grade NAEPU.S. History Item

Broadly-Define Constructs / Bias Review:

Assumptions of Prior Knowledge

Circle the picture that starts with “B”

Broadly-Defined Constructs / Bias Review:Culture and Native Language Assumptions

Circle the picture that starts with “T”

1st Grade Phonological Awareness Tasks

Ensuring Suitable Delivery: Invalidation

“Sarah must determine the diameter of a circle that she has drawn, but doesn’t know the value of П. Can she still do it?”

8th Grade Geometry: Word Problem vs. Symbol Recognition

Consider effect of read-aloud accommodation on the following test question:

MCA-IIGrade 10 Reading Sample Item

UDL Considerations for Assessment Development Structure:

Based on the UDL Principles

Research on working memory

Research on eye movement

*handout

UDL Considerations for Assessment Development Structure

.

Linguistic Complexity

Information Density

Self-Regulation and Visio-Spatial information

UDL Considerations for Assessment Development

Linguistic Complexity (LC) The relationship between the sentence processing mechanism and the available computational resources (Gibson, 1998)

LC-1 Syntax –

LC-2 Simplifying vocabulary

LC-3 Reduced sentence length

LC-4 Language translation

LC-5 Clarify anaphoric references

Linguistic Complexity

Information Density (ID) The two critical limitations on handling information in working memory are the (1) small number of pieces of information an individual’s memory can handle and (2) the short duration of time during which information can remain in memory (Sylwester & Choo, 1992). The following modification structures help to reduce working memory load and maximize use of information read when used carefully.

ID-1 Question placement ID-2 Emphasize key information ID-3 Chunking or combining tasks (questions or response options) ID-4 Guide information processing— ID-5 Contextualizing skills –skill icons ID-6 Hint ID-7 Line numbering ID-8 Passage primer

Information Density

Self-regulation is seen by many cognitive researchers as a pivot upon which students’ achievement turns. The structures noted here when used in a non-construct relevant application help to support self-regulation and attention to tasks.

SR-1 Progress map   SR-2 Self check   SR-3 Optional Workspace   SR-4 Skill Icon Preview

SR-5 Reduce Reflexive Eye Movement  

 Self-Regulation and Visio-Spatial information (SR)

Example Item 1: A Time-Distance Graph 40

Example Item 2: Features of Plant and Animal Cells 41

Example Item 3: The Boiling Points of Two Beakers of Water 42

UDL - Assessment revisions/considerations Discussion –UDL considerations

Small groups◦ Start with the task directions◦ Think UDL access to executive functioning