Application of Universal Design for Learning Principles to Assessments

A Guide to UDL Test Item DevelopmentTracey E. Hall, Kristin H. Robinson, Mindy Johnson,

Alicia Safier, Christopher Green, and Elizabeth DaltonCAST, Inc.

Overview

This is a guide developed to support the decision-making processes for integrating the principles of Universal Design for Learning (UDL) into assessments. This work is meant to provide recommendations for improvement in development of assessment items.

The primary focus is on applying the principles of Universal Design for Learning to assessment. These UDL principles are developed and researched by CAST, Inc. (www.cast.org and http://www.udlcenter.org/search/node). The components of this implementation guide serve to organize the process of UDL item modification and provide detailed information that supports each process step. Models and explanations, as well as item non-examples, help guide the development of assessment items that reduce barriers to assessments and integrate the principles of UDL.

UDL Principles and Guidelines: Definition, relationship and rationale for assessment integration

Universal Design for Learning (UDL) principles guide the preparation of this document. UDL is an educational framework that offers great promise to help improve educational outcomes for all students, including those with disabilities and students who are “in the margins.” UDL includes the entire curriculum goals, methods, assessment, and materials. The term “universal” in UDL does not imply one optimal solution, but instead underscores the need for an inherently flexible and customizable environment. Universal Design for Learning is driven by three foundational principles:

Multiple means of representation, to give diverse learners options for acquiring information and knowledge

Multiple means of action and expression, to give diverse learners options for demonstrating and acting upon information and knowledge

Multiple means of engagement, to tap into learners' interests, offer appropriate challenges, and increase motivation

The principles of UDL are further clarified by the guidelines for UDL implementation, developed by CAST in 2008.

The principles and guidelines for UDL recognize and support the natural variability in all learners. By applying the principles of UDL to assessment, we can assure that students are provided with supports and scaffolds to meet individual learning and access needs at the point of design. Application of UDL principles to the process of assessment implies that consideration be given to multiple means of representation, multiple means of action and expression, and multiple means of engagement. Curriculum (instruction and assessment) can be designed to create flexible assessments that most accurately measure student skills and understanding. By designing assessments that integrate the UDL principles, the resulting assessments will have diminished need for additional accommodations, as variations on forms of delivery and response are integrated in the original assessment design. Design with flexibility is most easily accomplished in a digital environment. However, the recommendations made in this document may be implemented in a non-digital or other representations as well. By applying the principles of UDL to assessment, we can provide students with supports and scaffolds that allow them to demonstrate their skills and understanding (Dolan & Hall, 2001).

Components of this guide to test item modification include:

Guiding Matrix for Applying Universal Design for Learning (UDL) Principles to Test Item Development

1. UDL modified exemplary passages (or “UDL exemplars”) 2. Item detail charts (or “Item charts) 3. Glossary of terms (or “Glossary”)

The components of this guide are presented in the following table:

Table 1: Components of the UDL Guide COMPONENT DESCRIPTION IMPORTANCE USE

1. Glossary A glossary of key terms used throughout this document. The glossary is an appendix to this guide.

The Glossary provides a common base of terms and language to support a shared understanding of key terms for all users

Review glossary before working with the test item modification document, to identify any terms used in ways that diverge from current usage or understanding. While working with the document, look up key terms as needed.

2. Guiding Matrix for reading modifications

The Guiding Matrix is an overview of the modification categories and rationale considered during the development of this UDL assessment model for reading. Included in the guiding matrix are: a) descriptors of each category, b) links to individual assessment items that demonstrate the modification, and c) identification of the UDL principles met by each modification category

The Guiding Matrix provides an ‘at-a-glance’ overview for the item modification process, and offers details relating to UDL-based modifications

The Guiding Matrix is a reference tool that serves as an index to access in-depth examples that support the PSSA-M UDL-based assessment examples.

3: Context Independent Modifications

Context Independent Modifications is a summary of those modifications that are made consistently regardless of assessment content. These Modifications include, but are not limited to; progress keys, icon previews, passage primers, skill icons, and self check prompts.

The Context Independent Modifications provide an ‘at-a-glance’ graphical representation and summary of modifications applied consistently and independently of specific content.

The Context Independent Modifications is a reference graphic and table that serves as an index to frequently and/or consistently applied modifications across content and genre that support the PSSA-M UDL based assessment examples.

4. UDL Exemplars Example of how passages would look when UDL principles are used to guide the development of passages and items. Each UDL-based change is linked the specific explanation of the type of modification and the impact of each modification in the assessment process

Each exemplar provides a contextualized version of how UDL-based modifications can display within an actual assessment passage

Exemplars serve as models for constructing assessment passages based on UDL principles. Through these examples, you will see how all of the UDL-based modifications work together in the context of student use

5. Item Charts Item Charts provide in-depth information about each UDL-based modification type, by category. Test questions often contain several modifications, and each modification is explained separately. Descriptions include: a) explanation of the specific modification, b) the UDL principles and guidelines addressed by the modification, c) original PSSA item, d) UDL example and explanation, and e) non-examples

Each item chart provides detailed models and explanations that support increased understanding of the practical application of UDL principles and guidelines in a large-scale assessment context.

Allows the user to deepen their understanding of item modifications by clarifying:a) complexity of changes;b) explanations of specific changes;c) general rules for types of changes;d) non-examples, showing reasonable, but ineffective modifications

UDL Considerations for Assessment Development Structure

The designs for the assessment are based on the UDL Principles and research on working memory. From these works we created three main categories of consideration; (1) Linguistic Complexity, (2) Information Density and (3) Self-Regulation and Visio Spatial Information. Each is briefly described below with sub-structures.

The UDL exemplars are based on these categories and organized for the user applying the structure described below. Exemplars were created in the content areas of reading and mathematics from the Pennsylvania State Assessment (PSSA) released sample items. Item types were selected representing a range grade levels and anchor categories (standards).

Linguistic Complexity (LC) The relationship between the sentence processing mechanism and the available computational resources (Gibson, 1998).

LC-1 Syntax – Active subject-verb-object constructions (rather than complex causal structure); Simplify syntactical structure to clarify meaning and purpose of item

LC-2 Simplifying vocabulary Common over unusualAvoid use of contractions Provide definitions for key construct-irrelevant vocabulary words in student friendly

language LC-3 Reduced sentence length

Use basic sentence structureAvoid compound or complex sentences

LC-4 Language translation Provide translation for key construct-irrelevant vocabulary (examples in Spanish)

LC-5 Clarify anaphoric references Limit use of pronoun references from sentence to sentence Limit use of pronoun references phrase to phrase

Information Density (ID) The two critical limitations on handling information in working memory are the small number of pieces of information an individual’s memory can handle and the short duration of time during which information can remain in memory (Sylwester & Choo, 1992). The following modification structures help to reduce working memory load and maximize use of information read when used carefully.

ID-1 Question placementReduce cognitive load for recall and search for context to respond to questions by placing question

items in proximity to passage text where construct irrelevant. Visual display (arrow) to help direct student attention from passage to question.

ID-2 Emphasize key informationProvide explicit cues or prompts that assist individuals in attending to those features that matter

most while avoiding those that matter leastKey words to find in relation to item questions frequently printed with bold typeface or

underlined to assist word search.

ID-3 Chunking or combining tasks (questions and/or response options) Provide memory support by breaking or chunking questions, directions, and response options into

separate steps when several steps are required to complete the item.Use format and cues such as bulleting or numbering to indicate essential components or steps of

an item.To support working memory, combining or consolidating directions/information regarding the

task instead of separating information with images, graphics or diagrams. ID-4 Guide information processing—

Provide relevant tools (graph, number line, ruler, image, etc.) when providing such a tool does not interfere with the construct evaluated.

ID-5 Contextualizing skills –skill icons Context Independent Modification designed to focus student on the skill evaluated in the task

ID-6 Hint Information provided such as a formula or reminder to prompt problem completion or calculation.

Only information that is not construct relevant is made available, or information/support provided through the original assessment (e.g. formula sheet).

ID-7 Line numberingContext Independent Modification, which provides line by line numbers in passage

and in question stem as a reference to key non-construct relevant information.ID-8 Passage primer

Context Independent Modification Simple and short statement of topic and character or overarching event in the

passage Used as a strategy for activating prior knowledgeActivation is a research-validated approach for improving children's memory and

comprehension of text

Self-Regulation and Visio-Spatial information (SR) Self-regulation is seen by many cognitive researchers as a pivot upon which students’ achievement turns. The structures noted here when used in a non-construct relevant application help to support self-regulation and attention to tasks.

SR-1 Progress map Context Independent ModificationSupport motivation and engagementFor many students, merely recognizing that they are making progress toward greater

independence is highly motivating. Support students to recognize their own progress.

SR-2 Self check at the end of passage and items (checklist, arrow with prompt)Context Independent ModificationProcess to guide effort, in meeting standards and goalsPromote self-regulation with checklists to guide students in task completion during assessment

process

SR-3 Optional WorkspaceContext Independent Modification

Provide a workspace for students to facilitate information management and processing

SR-4 Skill Icon PreviewContext Independent ModificationProvide anticipatory set of skills evaluated in the assessmentPromote self-regulation with icon previews to guide students in approaching the assessment

process

SR-5 Reduce Reflexive Eye Movement Context Independent ModificationBorders on text-boxes and other elements causes reflexive eye movement and can interfere with

working memory and information processing.

ULD Assessment Considerations:GLOSSARY of TERMS

Accommodations (for assessment): Accommodations are a change in how an assessment is presented, administered, or in how a student is allowed to respond. The proper use of accommodations does not change academic level or performance criteria, and are made to provide equal opportunity to demonstrate knowledge.Source: Modified from: The NAEP Glossary of Terms http://nationsreportcard.gov/glossary.aspContextual Use: Alana is a student whose disabilities include visual impairment. Her IEP team determined that text-to-speech was an appropriate accommodation for classroom and formal assessments.

Assessment domain: Assessment domain refers to the specific academic content and breadth of content being tested.Source: Adapted from Tindal, & Marston (1990). Classroom-based assessment: Evaluating instructional outcomes. Contextual Use: The spring assessment for 8th grade included English Language Arts. This assessment domain includes persuasive writing.

Background Knowledge (see prior knowledge)

Barriers: Barriers pertain to anything that restrains or obstructs progress or access to the task at hand.Source: Adapted from: http://dictionary.reference.com/browse/barrier Contextual Use: Eli is unable to decode the text of his story for English class. Even though he understands the language and story structure, he is unable to read for understanding because his inability to decode words is a barrier to knowing the meaning of words.

Chunking: Chunking is a procedure of breaking up learning materials into manageable sections or components (e.g., grouping of words in sentences into short meaningful phrases).Source: Adapted from: Casteel, (1988). Effects of chunked reading among learning disabled students: An experimental comparison of computer and traditional chunked passages.Contextual Use: Instead of asking Jill to complete the 20 math problems listed on the page in one sitting, the teacher used a chunking strategy and asked Jill to complete four problems at a time.

Cognitive load: The human cognitive architecture consists of a limited working memory that interacts with a comparatively unlimited long-term memory. The limited working memory carries the risk of learners being cognitively overloaded when performing a high-complexity task. It is critical to design the presentation of information in a manner that encourages learner activities that optimize intellectual performance. Source: Cognitive load theory. The Gale Group By Kirschner, P., Kirschner , F.& Paas F. Retrieved November 5, 2010.Contextual Use: When planning instruction as well as assessment items, reducing the cognitive load usually helps the learner to maintain task engagement.

Construct-irrelevant: Construct irrelevance refers to factors that are extraneous to the construct being measured (e.g., mode of presentation or response, contextualizing information). See test construct.Source: Adapted from: American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association, p. 173. http://teststandards.org /

Contextual Use: For Elisa, a student with severe cognitive disabilities, changing the font size and increasing sizing of images helped her to see the assessment item for counting objects and doing so was construct irrelevant to assessing ability to count.

Construct relevant: Construct relevant refers to the factors (e.g. decoding, order of operations)) that are relevant (related) to the construct that the test is intended to measure. See test construct.Source: Adapted from Madaus, Russell & Higgins, (2009). The paradoxes of high stakes testing: How they affect students, their parents, teachers, principals, schools and society. Contextual Use: Ms. Briggs wanted to adapt an assessment item to make it more accessible for her student. However, she realized that identifying the symbols for a math calculation problem is a construct relevant change to the assessment item, and must be avoided.

Content specific: Content specific refers to terms, concepts or vocabulary having explicit meaning critical to understanding particular content.Source: CAST, Inc., internal definition Contextual Use: The state's standards for 6th grade history required that students demonstrate an understanding of "civic duty" by citing a content specific example of an act by historical figures.

Counter example (see non-example)

Engagement: Engagement refers to a student's level of interest and involvement with learning.Source: Adapted from: http://www.udlcenter.org/aboutudl/udlguidelines/principle3Contextual Use: To ensure that an assessment offered multiple means of engagement, student choice was built into several items as a way to provide students with a sense of autonomy.

Information Density: The two critical limitations on handling information in working memory are the number of pieces of information it can handle and the short duration of time during which information can remain and be processed there. The UDL modification structures help to reduce working memory load and maximize use of information read when used carefully.Source: Adapted from Sylwester & Choo, 1992. What Brain Research Says about Paying Attention Contextual Use: Caution should be used to ensure that the information density of assessment items is not overly complex.

Informational text (passage): Text designed to convey factual information, rather than tell or advance a narrative. Informational text may employ techniques such as lists, comparing/contrasting, or demonstrating cause/effect, and may be accompanied by graphs or charts. Most textbooks consist of primarily informational text.

Source: Glossary of Education education.com Contextual Use: The informational texts used on the assessment covered content areas of cooking, sports, and world history.

Key Concept: Information or idea identified as central/core to the question or assessment item. Source: CAST, Inc., internal definition Contextual Use: The science expert group determined that convection was the key concept for the next item on the grade assessment.

Linguistic complexity The relationship between the sentence processing mechanism and the available computational resources.Source: Adapted from Gibson, (1998). Linguistic complexity: locality of syntactic dependenciesContextual Use: When creating directions, questions and assessment items the linguistic complexity should be kept at a minimum, attend to complexity of syntax, vocabulary, sentence structure and length, as well as limit the use of pronoun references.

Modification: When a curriculum modification is made, either the specific subject matter is altered, or the performance level expected of the student is changed; a curriculum modification is made when a student is either taught something different from the rest of the class or taught the same information but at a different level of complexity.Source: Nolet & McLaughlin, (2005). Accessing the general curriculum: Including students with disabilities in standards-based reform. Contextual Use: The student's IEP team decided to make a modification to the unit on Mark Twain's Adventures of Tom Sawyer. Instead of reading the original work, the student would read an abridged version written at his individual reading level.

Narrative text (passage): The telling of a story or an account of a sequence of events. Narrative differs from expository, which can also relate a sequence of events, in that narrative need not be factual and may be written from the perspective of a character in the text.Source: Adapted from Glossary of Education education.com Contextual Use: Several of the passages on the reading assessment were narrative text, each a different story to help keep students interested.

Non- construct relevant: (see construct –irrelevant)

Non-example: An instructional strategy to illustrate the limitation in variation and quality of the concept being taught or expressed. Source: Adapted from Kameenui, Carnine, Dixon, Simmons, & Coyne, (2002). Effective teaching strategies that accommodate diverse learners 2nd Ed.Contextual Use: After showing students several examples of mountain ranges, he then showed an image of a single peak as a non-example to help students understand that a range of mountains contains many peaks.

Prior Knowledge: Information mastered and maintained from previous experiences, whether from earlier instruction or life experience. Refers to related knowledge that students must already know in order to act upon or learn a new concept, strategy, system or big idea. Source: Adapted from: Kameenui, Carnine, Dixon, Simmons & Coyne (2002). Effective teaching strategies that accommodate diverse learners.Contextual Use: To assure that her students understood the story they were preparing to read, the teacher provided them with images and information about soccer to ensure prior knowledge was activated.

Progress map: Progress maps can be used to organize assessment data to provide a picture of change or progress over time. Source: Adopted from: Pellegrino, Chudowsky, & Glaser (2001). Knowing what students know: The science and design of educational assessment.Contextual Use: Jamie filled in the progress map at the top of the page for each item she finished to see how much was complete and how many more items she had to do.

Prompts: A cue that provides assistance or guide an action during a learning task. Source: National Center on Universal Design for Learning. http://www.udlcenter.org/Contextual Use: The teacher provided a short prompt about work completion to help the student stay focused on the assessment item.

Representation: Representation refers to the ways in which information is presented to students.Source: Adapted from: http://www.udlcenter.org/aboutudl/udlguidelines/principle1 Contextual Use: The teacher ensured that there were multiple means of representation for each test item. Students could read the item or listen to the item using text-to-speech. Images or animation depicted the main ideas of the item as well.

Scaffolds: Scaffolding is the use of an external contextual support that makes a particular learning task possible but that can be gradually removed as the learner moves toward competent independent performance.Source: Adapted from: Granott, Fischer, & Parziale, (2002). Bridging to the unknown: A transition mechanism in learning and development. Contextual Use: Jason has difficulty keeping track of information and organizing his thoughts. As a type of scaffolding, the teacher offers Jason a graphic organizer to help him manage information. As Jason's executive function skills develop, this tool may not be necessary.

Self-regulation: Self-regulation is an individual's strategic modulation of emotional reaction or state in order to cope or engage with the environment more effectively.Source: Adapted from: http://www.udlcenter.org/aboutudl/udlguidelines/principle3

Contextual Use: Jessie kept a simple drawing of a thermometer on his desk as a self-regulation mechanism. He simply filled in a line each time he completed a task, so he could see how close he was to reaching his goal for the day.

Simplified syntax: Simplified syntax is the use of symbol/word selection, phrase structure or sentence structure for writing the item. Item directions and questions should be written using active subject-verb-object constructions (rather than complex causal structure) to clarify meaning and purpose of item.Source: Adapted from: http://dictionary.reference.com/browse/syntaxContextual Use: Samuel is an English Language Learner who has difficulty comprehending complex sentence structure. When administering assessments, the teacher rephrases the test directions with simplified syntax so that his limited English will not prevent him from understanding the directions.

Standards: Standards refer to specific criteria for what students are expected to learn and be able to do.Source: Adapted from: http://www.ascd.org/Publications/Lexicon_of_Learning/S.aspxContextual Use: As teachers and IEP teams prepare plans for instruction, they often take into account the district, state or national standards, to help determine what knowledge the student should be able to demonstrate.

Supports: Supports are a means to help students overcome deficits in the learning medium that prevent them from accessing or interacting with the content. Supports need not necessarily be removed as students gain proficiency in independent performance.Source: Adapted from: Rose, Hall, & Murray, (2008). Accurate for all: Universal design for learning and the assessment of students with learning disabilities.Contextual Use: Denise has a vivid imagination and can create great stories; however, her handwriting is illegible. As a support, Denise uses a tape recorder to capture her stories.

Test Construct: Test construct refers to the concept or the characteristic that a test is designed to measure.Source: American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Contextual Use: As the educators reviewed the test item, they reviewed the test construct for the item to assure their modification did not interfere with what was being measured.

Universal Design for Learning (UDL): Universal Design for Learning is an educational approach for designing curriculum for all learners. UDL is based on three principles: Multiple means of representation, to give diverse learners options for acquiring

information and knowledge Multiple means of action and expression, to give diverse learners options for

demonstrating and acting upon information and knowledge

Multiple means of engagement, to tap into learners' interests, offer appropriate challenges, and increase motivation

Source: Adapted from: http://www.cast.org/about/index.htmlContextual Use: The UDL AA-AAS Evaluation tool assists in the evaluation of alternate assessments systems with regard to presence of the principles of Universal Design for Learning.

Working memory: The ability to store and manage information in one's mind for a short period of time; in one test of working memory a person listens to random numbers and then repeats them; the average adult can hold seven numbers in their working memory; working memory is sometimes called Short-term memory.Source: Adopted from Sylwester, Choo, (1992). What Brain Research Says about Paying Attention. Contextual Use: Various supports and scaffolds can be built into curriculum and assessment tasks to help reduce heavy working memory loads and therefore help learners succeed and demonstrate knowledge.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. http://teststandards.org/

Butler, D.L., & Winne, P.H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research 65 (3) 245-281

Casteel, C. (1988). Effects of chunked reading among learning disabled students: An experimental comparison of computer and traditional chunked passages. Journal of Educational Technology Systems, 17(2), 115-21.

Center for Applied Special Technology (CAST) (2008). Universal design for learning guidelines version 1.0. Wakefield, MA: Author. Retrieved April 1, 2010 at http://www.udlcenter.org/sites/udlcenter.org/files/UDL_Guidelines_v2%200-Organizer_0.pdf

Dolan, R. P. and Hall, T. E. (2001). "Universal Design for Learning: Implications for Large-Scale Assessment." IDA Perspectives 27(4): 22-25.

Gibson, E., (1998). Linguistic complexity: locality of syntactic dependencies. Cognition. 68(1):1-76. Retrieved Aug. 19 from http://www.ncbi.nlm.nih.gov/pubmed/9775516

Granott, N., Fischer, K. W., & Parziale, J. (2002). Bridging to the unknown: A transition mechanism in learning and development. In N. Granott, & J. Parziale (Eds.), Microdevelopment: Transition processes in development and learning (pp. 131-156). Cambridge: Cambridge University Press; Guzdial, M. (1994). Software-realized scaffolding to facilitate programming for science learning. Interactive Learning Environments, 4, 1-44.

Kameenui, E. J., Carnine, D. W., Dixon, R. C., Simmons, D.C., & Coyne, M.D., (2002). Effective teaching strategies that accommodate diverse learners 2nd Ed. New Jersey: Merrill/Prentice Hall.

Kirschner, P., Kirschner , F.& Paas F. The Gale Group www.education.com/partner/articles/galegroup/ Retrieved November 5, 2010.

Madaus, G., Russell, M. & Higgins, J. (2009). The paradoxes of high stakes testing: How they affect students, their parents, teachers, principals, schools and society. Information Age Publishing, Inc.: Charlotte, NC.

Meyer, A., & Rose, D. (2006). Preface. In D. Rose, & A. Meyer (Eds.), A practical reader in universal design for learning. (pp. vii-xi). Cambridge, MA: Harvard Education Press.

New World College Dictionary (2010). Technology definition. Retrieved April 1, 2010 at http://www.yourdictionary.com/technology

Nolet, V., & McLaughlin, M. J. (2005). Accessing the general curriculum: Including students with disabilities in standards-based reform. CA: Corwin Press, Inc. (p. 87)

Pellegrino, Chudowsky, & Glaser (2001). Knowing what students know: The science and design of educational assessment. Committee on the Foundations of Assessment, Board on Testing and Assessment, Center for Education, National Research Council. Retrieved Aug. 14, 2010. http://www.nap.edu/books/0309072727/html/

Ridley, D. S. (1992). Self-regulated learning: The interactive influence of metacognitive awareness and goal-setting. Journal of Experimental Education, 60(4), 293-306.

Rose, D. H., Hall, T. E., & Murray, E. (2008). Accurate for all: Universal design for learning and the assessment of students with learning disabilities. Perspectives on Language and Literacy, 23-28.

Rose, D. H. & Meyer, A. (2006). The future is in the margins: The role of technology and disability in educational reform. In Rose, D. H., Meyer, A., & Hitchcock, C. (Eds.) The Universally Designed Classroom (pp. 13-35). Cambridge, MA: Harvard Education Press.

Sylwester, R. & Choo, J.Y. (1992). What Brain Research Says about Paying Attention. Educational Leadership, 50, 71-75.

Tindal, G.A., & Marston, D.B.,(1990). Classroom-based assessment: Evaluating instructional outcomes. Columbus, OH: Merrill Publishing Co.