Assessment Literacy Series

1

-Module 6-Quality Assurance & Form

Reviews

SAM-L Development participants will:

1.Apply a quality assurance checklist to the developed materials to ensure all documents meet the requirements.2.Verify the item is measuring the intended content and standard.3.Ensure the scoring key/rubrics are correct.

2

Objectives

Participants may wish to reference:

Guides• Handout #10 – Quality Assurance Checklist• QA Screening Tool• Performance Measure Rubric-Teacher

Other “Stuff”•“Smart Book”

3

Helpful Tools

4

Outline of Module 6

Module 6: Form Reviews

Content Review

Fairness, Sensitivity,

Bias Apply Checklist

to all forms

Alignment

Rigor

Form Review

Reviewing forms involves the following components:Content Review BiasFairnessSensitivityAccessibilityAlignment

Note: Refer to “Smart Book” for detailed explanations of these components.

5

Content Review Determine if each item/task clearly aligns to the

targeted content standard. Evaluate all items for accuracy. Judge if each item is age and grade appropriate for

students.• Reading level is appropriate.• Vocabulary is appropriate.• Required reasoning skills are appropriate.

Review each item stem and answer options to be sure they are clear and understandable.• Student response expectations are clear.

6

Item Bias Bias is the presence of some

characteristic of an item that results in the differential performance of two individuals with the same ability but from different subgroups.

Bias is not the same as stereotyping. Bias-free items/tasks provide an equal

opportunity for all students to demonstrate their knowledge and skills.

7

Fairness and SensitivityReview to ensure:Items are sensitive to different cultures, religions, ethnic and socio-economic groups, and disabilities.There is a balance of gender roles.Use of positive language, situations, and images.Items avoid text that may elicit strong emotions in specific groups of students and prevent those students from accurately demonstrating their skills and knowledge.

8

Sensitive Topics to Avoid

• Abortion• Birth control• Child abuse/neglect• Creationism• Divorce• Incest• Illegal activities• Occult/witchcraft

• Questioning parental authority

• Rape• Religious doctrine• Sex/sexuality• Sexual orientation• Weight• Suicide• Sexually transmitted diseases

*Note: The above list does not contain every sensitive topic that item developers should avoid and does not consider test-taker’s age/maturity. 8

What is Accessibility? Test accessibility is defined as the extent to which a test eliminates

barriers and permits the test-taker to demonstrate his or her knowledge of the tested content.

All items should be reviewed to ensure they are accessible to the entire population of students.

Item reviews must consider:

• Readability

• Syntax complexity

• Item presentation

• Font size

• Clarity of images, graphs, tables

• Amount of work space

10

AlignmentAlignment Categorical Concurrence

◦ The same categories of the content standards are included in the assessment◦ Items might be aligned to more than one standard

Source of Challenge◦ Students give a correct or incorrect response for the wrong reason (bias)

Balance of Representation◦ Ensures there is an even distribution of the standards across the test

Range of Knowledge◦ The extent of knowledge required to answer correctly parallels the knowledge

the standard requires Depth of Knowledge

◦ The cognitive demands in the standard must align to the cognitive demands in the test item

Source: Webb, N. L. (1997). Research Monograph No. 6: Criteria for alignment of expectations and assessments in mathematics and science education. Washington, DC: Council of Chief State School Officers

11

Quality Assurance QA Screening Tool◦ Use to examine first draft materials◦ Outlines key criteria

Quality Assurance Checklist[Handout #10]

◦ Use with final drafts◦Examines each developed component of the performance measure.

Performance Measure Rubric-Teachers◦Comprehensive self-assessment of teacher-made performance measures.

12

Process Steps1. Apply the Quality Assurance Checklist [Handout #10] to the

performance measure and supporting documents. Apply the

checklist to each individual item /task, and then evaluate the

measure in its totality.

2. Make notes directly on the paper copies of the documents.

3. Review areas that may need additional focus.

4. Print and conduct a final team review of all documents,

particularly the operational form and scoring keys.

5. Turn your packet in to the facilitator, and destroy draft

material.

13

14

Summary

Module 6: Form Reviews•Evaluated the documents created by the performance measure development process, and ensured the measures were of high-quality, rigorous, and aligned to the targeted content standards. All documents went through a quality assurance process prior to finalization.