Including Quality Assurance Within The Theory of Action

Presented to:CCSSO 2012 National Conference on Student Assessment

June 27, 2012

• Quality Assurance - A continuous cycle that improves and refines the processes necessary to produce valid assessments to the highest standards. Ensures that excellence is inherent in all components of the process. – Plan– Do– Study– Act

Quality Assurance as a Continuous Cycle

• Evaluated assessment systems in most states and the U.S. territories through the federal standards and assessment peer review process.

• Developed interpretative arguments for the validity of assessments and conducted studies to test the claims and assumptions in the interpretive argument for a number of states.

• Provided technical assistance for reviewing the technical quality of assessment and data quality of assessment systems.

Edcount’s relevant work:

• Students are appropriately tested for identified testing options

• Test scores accurately reflect student knowledge and skills

• Reports provide valid scores and are used appropriately

Quality Assurance Within the Theory of Action - Claims

• The Learner Characteristics Inventory (LCI)– A survey designed to collect information about

the population of students who participate in alternate assessments based on alternate achievement standards (AA-AAS).

– Helps states identify and evaluate “who” is participating in the alternate assessments. Provides information to help direct participation rules and guidance.

Students Are Appropriately Tested for Identified Testing Options – Special Studies

• Accommodation Studies– The extent to which accommodations are

provided on the state assessments.– Literature review on best practices for

accommodation use. – Provides information and evidence to improve

best practices, guidance, and monitoring of accommodation use.

Students Are Appropriately Tested for Identified Testing Options – Special Studies

• Performance Level Descriptor (PLD) studies– Examine the extent to which assessment cut scores

appropriately distinguish between student performance levels, and whether the performance level descriptors adequately capture meaningful distinctions in grade-level performance.

– Engage teachers in rating and feedback processes to help administrators understand how teachers use and interpret PLD’s

– Evaluate the PLD’s for coherence, consistency, and accuracy.

Test Scores Accurately Reflect Student Knowledge and Skills – Special Studies

• Performance Level Descriptor (PLD) studies cont.– Provides information to help states refine their

standard setting processes, focus their professional development and support their interpretations on the state assessments.

Test Scores Accurately Reflect Student Knowledge and Skills – Special Studies

• Alignment Studies– Evaluations of the alignment between academic

content standards and alternate assessments.– Evaluations of the alignment between academic

content standards and general assessments.– Provides information to support and improve the

design and development of assessments.

Test Scores Accurately Reflect Student Knowledge and Skills – Specialized Studies

• Cognitive Labs– Collect information about whether assessments

are eliciting the intended cognitive processes. – The cognitive labs can help states understand why

students may struggle with certain items or content areas on an assessment, and, as a result, provide feedback about ways to develop accessible assessments and ways to improve instruction.

Test Scores Accurately Reflect Student Knowledge and Skills – Special Studies

• Consequential Validity Studies– Evaluate and collect evidence about the intended

and unintended consequences of a states large scale assessment system.

– Provides evidence to inform policies and processes and focus professional development.

Reports Provide Valid Scores and Are Used Appropriately - Special Studies

• Special Studies– PLD study– Accommodation Study– Consequential Validity Evaluation– Cognitive labs

• Professional Development– Data Use Workshop– Curriculum Mapping Boot Camps

Work Within Puerto Rico

• Special Studies– PLD studies– Cognitive Process Validity Study

• Technical Assistance For:– Development of a balanced assessment system

with formative and interim assessments– Development of an accountability System – Small

school review– Development of a data management system

Work Within the Laurent Clerc Center at Gallaudet University

• Provided focus for professional development• Provided information to improve the design

and development activities for assessments and reports

• Provided context as well as empirical evidence to support policies and processes

How Did the Work Inform Quality Assurance and Processes

Contact Information• Jennifer Stegman, Senior Associate• jstegman@edcount.com

• edCount LLC• 5335 Wisconsin Avenue NW• Washington DC 20015• edCount.com