Universal Design and Testing

Tim O’Connor

Testing and Evaluation ServicesUniversity of Wisconsin-Madison

• Toward a Better Understanding of Testing

• Test Audits

Toward a Better Understanding of Testing

Merriam-Webster defines testing as something (as a series of questions or exercises) for measuring the skill, knowledge, intelligence, capacities, or aptitudes of an individual or group

• Tests are: A tool to measure behavior, knowledge or skills

• Tests are not: Magic, voodoo, or witchcraft

Validity

The extent that a test measures what it is meant to measure.

Reliability

The extent to which a test is consistent and accurate in measuring what it is meant to measure.

Accommodations, Reasonable

• Accommodations vs. modifications

While both accommodations and modifications are nonstandard conditions, accommodations differ in that modifications change what construct is being measured; accommodations should only change how a construct is being measured.

Most students benefit from extended time on tests.

• A review of over 40 studies concluded that students without learning disabilities also benefit from extended time, although not as much. (Sireci, Li, & Scarpati, 2003)

The most significant effect may be psychological

• One study suggests that extended time may affect psychological functioning more than performance on standardized tests, i.e., providing a more positive testing experience by reducing test anxiety and creating the opportunity to utilize good test-taking strategies. (Elliot & Marquart, 2004)

Security-Blanket Effect

• Anecdotal records of students taking the University of Wisconsin placement tests show that the majority of those receiving extended time accommodations, use no more time to complete the tests than those who are taking it under standardized conditions.

Testing is part of the Learning Process

• Summative Evaluation – Assesses student learning

• Formative Evaluation– Assesses and improves student learning

How Universal Design can Improve Testing

Research Findings and Practical Reasons for Using UD in Testing

Tests need to accessible to be effective tools of measurement

“It is a dreadful irony that students with disabilities have better access to school buildings than they do to the curricula within them.” (Dolan & Hall, 2001)

UD Principles help Everyone!

Preliminary research findings suggest that all students may perform significantly better on tests applying universal design principles than on traditionally designed tests. (Johnstone, 2003)

Tests need to accurately measure student learning to be effective

tools of measurement

“Whether assessment is embedded into teaching (e.g. curriculum-based measurement) or administered separately (e.g. large-scale assessment), it must provide students with adequate and equitable means to express their knowledge and understanding if it is to provide accurate feedback on the performance of students. “(Dolan & Hall, 2001)

“many assessments appear to have been designed under the assumption that learners are relatively homogeneous, and that the expected outcomes for all students are relatively the same. As a result, assessments are rarely free of confounding information, and thus rarely represent a truly accurate measure of student abilities.” (Dolan, 2001)

The medium of a test is a very often overlooked dimension.

“Most testing depends on a single medium - text- for the majority of learning and evaluation. As important to our culture as this medium is, it is not equally accessible to all students. For students with reading-related disabilities (among others), this dominant medium of instruction is more of a barrier than a ramp.” (Dolan & Hall, 2001)

“In light of student differences, administering assessment using a common format does not level the playing field as many educators believe. Rather, a single format tilts the playing field, favoring some and hampering others.” (Dolan, 2001)

Ability to demonstrate knowledge can be obscured by reading and text presentation difficulties. The medium of the test is not weighed into it’s reliability or validity.

The style of learning should be a consideration.

Research has shown how important the method of student learning (learning styles) is to how well learning is accomplished. Yet this is seldom a consideration when we devise our methods of assessing student learning.

Tests should not measure extraneous information.

We also know that available research indicates that the current methods of large-scale assessment are generally inadequate for students with disabilities. Particularly problematic is the issue of construct-irrelevance; many assessments measure not only the targeted construct but also unintended constructs related to accessing the test material or carrying out a response.

Examples of such unintended constructs include sensory capabilities such as sight and hearing, physical capabilities such as holding a pencil and cognitive capabilities such as memory and attention.

(Abedi, Leon & Mirocha, 2001; Helwig, Rozek-tedesco, Tindal, Heath & Almond, 1999; Parks, Suen, Zimmaro & Zappe, 1999)

“Reading is probably the most-often unintended construct in many assessments, math being one example.” (Clarkson, 1983; Clements, 1980; Newman, 1977)

While read-aloud accommodations have been repeatedly shown to improve test performance of students with reading disabilities, there are some serious issues with its use. There are three significant ways in which the human read-aloud accommodation fails to provide adequate supports for students and potentially compromises test validity.

1) human read-alouds vary in quality, and some readers may mispronounce or misread words;

2) students are reluctant to (or may be unable to) ask human readers to re-read test portions (or may be unable to); and

3) through intonation or non-scripted comments human readers may inadvertently influence students’ attention or responses. (Landau, et al., 2003)

The pace of human read-alouds, including video and audiotaped versions, also impose upon students a linear path and a set pace, the latter of which has been shown to negatively affect the test performance of students with disabilities. (Hollenbeck et al, 2000)

Technology can help make tests more universally designed.

“Unlike a human read-aloud, a text-to-speech read-aloud provides students with consistent readings free of potentially directive or misleading intonation. Consistent with UD principles, a text-to-speech read-aloud supports students’ diverse ways of recognizing, strategically interacting, and engaging with an assessment by offering individualized, independent, and self-paced multimodal access to test content, on demand.” (Dolan et al., 2005)

“Digital text and accompanying universally designed curriculum and assessment can bring down barriers to learning for all students.” (Rose, 2001)

“Technology-based accommodations have tremendous potential to improve the accessibility of tests for students with disabilities.” (Dolan & Hall, 2001)

Computer-based testing is a good medium for universally designed assessments, because it allows for built in features such as speech to text, electronic reading supports and alternative access devices. These devices reduce the need for accommodations and assistive technology devices during assessments. (Thompson, Johnstone & Thurlow, 2002)

Study on the effectiveness of screen readers in assessments found that screen readers helped students with reading difficulties achieve better test scores in content areas. (Brown and Augustine, 2000)

The mode of expression has strong effects on students without disabilities. A set of studies at Boston College (Russell, 2000) investigated the role of different modes of expression (handwriting versus keyboarding) on the standardized test scores of general education students. Results indicate that student scores were affected by the mode of responding. That is, students who had experience on computers got much higher scores on the same test if they responded with computer than with handwriting.

Seven Elements for Universally Designed Assessments:

1. Inclusive assessment population - when assessments are first conceptualized, they need to be thought of in the context of the entire population of who will be assessed.

2. Precisely defined constructs - assessments should measure what they actually intend to measure (construct validity).

3. Accessible, non-biased items - items shouldn’t potentially advantage or disadvantage any subgroup of examinees within the population being tested.

4. Amenable to accommodations - while assessments should be designed to be accessible to most students, there will usually still be some students that need accommodations.

5. Simple, clear, and intuitive instructions and procedures - assessment instructions and procedures need to be easy to understand, regardless of a student’s experience, knowledge, language skills, or current concentration level.

6. Maximum readability and comprehensibility - assessments should be a test of student knowledge and not their reading abilities or comprehension level.

7. Maximum legibility - assessments should be able to be deciphered with ease.

National Center for Educational Outcomes (Thompson, Johnstone & Thurlow, 2002)

Test Audits