the journey toward accessible assessments karen barton ctb/mcgraw-hill validity &...
TRANSCRIPT
![Page 1: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/1.jpg)
The Journey Toward
Accessible Assessments
Karen Barton
CTB/McGraw-Hill
Validity & Accommodations:
![Page 2: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/2.jpg)
Validity• Validity: the ongoing trust in the accuracy of the test, the
administration, and interpretations and use of results• According to Messick (1995), “validity is not a property of the test . . .
as such, but rather of the meaning of the test scores . . . (that) are a function not only of the items or stimulus conditions, but also of the persons responding . . . (p. 741).
• Validation must therefore encompass the full testing environment: – test constructs– items– persons – characteristics and interactions of each
This goes beyond the validity of accommodating to the heart of assessment validity.
![Page 3: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/3.jpg)
Validity
ExamineeContext
(CRT, NCLB)ConstructTest/Items
![Page 4: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/4.jpg)
Validity
ExamineeContext
(CRT, NCLB)ConstructTest/Items
Accommodations Classification
![Page 5: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/5.jpg)
Validity
ExamineeContext
(CRT, NCLB)ConstructTest/Items
Targeted Accessible
![Page 6: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/6.jpg)
Validity
ExamineeContext
(CRT, NCLB)ConstructTest/Items
Accommodations Classification Targeted Accessible
![Page 7: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/7.jpg)
Validity
ExamineeContext
(CRT, NCLB)ConstructTest/Items
Accommodations Classification Targeted Accessible
Predetermined Pieces
![Page 8: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/8.jpg)
Confounding Variables: Persons
Persons• Disabilities• Accommodations• Access/barriers• Response
![Page 9: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/9.jpg)
Person Variables & Variation• Identification of disability• Accommodation policies, selection and
provision• Access to and instruction in varied standards
and depth/breadth of coverage • Access to test information and opportunity to
accurately respond
![Page 10: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/10.jpg)
Given the current state of the state, where diverse examinees approach the assessment platform with various accommodations and in non-standard administrations, what can be done to improve the validity of the assessments?
![Page 11: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/11.jpg)
Validity
ExamineeContext
(CRT, NCLB)ConstructTest/Items
Accommodations Classification Targeted Accessible
Predetermined Pieces
![Page 12: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/12.jpg)
What is a construct?• “A product of informed scientific imagination,
an idea developed to permit categorization and description of some directly observable behavior . . (The construct itself is) not directly observable. . . (and) it must first be operationally defined (Crocker and Algina, 1986, p. 230).
• ≠ Trait.
![Page 13: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/13.jpg)
Construct
TargetedTrait
TargetedTrait
TargetedTrait
Evidence Evidence Evidence Evidence Evidence Evidence
The operational definition includes the specification of traits and observable skills that, together, represent the unobservable construct.
The operational definition should be researched and empirically supported.
![Page 14: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/14.jpg)
Math Intelligence
ComputationProblem Solving
Numbers
Item Item Item Item Item Item
![Page 15: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/15.jpg)
Predetermined Pieces
Validity
ExamineeContext
(CRT, NCLB)ConstructTest/Items
Accommodations Classification Targeted Accessible
![Page 16: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/16.jpg)
Access
Precision
Validity
![Page 17: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/17.jpg)
Access• Student access to
– test information (directions, stimulus), – requirements (expectation of how to respond), – response capabilities (the way in which students
respond) • Item access to student ability – true
performance
![Page 18: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/18.jpg)
Improved Access• Improved student access:
– Accommodations: Access tools specific to examinees that allow for assessment such that disability or language does not misrepresent true performance.
• Improved item access:– Minimizing Construct Irrelevant Variance
(systematic error)
improved precision
![Page 19: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/19.jpg)
Precision threat: ErrorRandom error– Random or inconsistent– Inherent to the assessment– Examples – content sampling, summative “snap-shot” assessment, scoring,
distractions– Reduce usefulness of scores
Systematic error – Consistent– Inherent to examinee– Example – students with disabilities without needed accommodation(s), low item
accessibility– Reduce accuracy of scores
When error is minimized, scores are more trustworthy!
SE
RE
![Page 20: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/20.jpg)
Validity
ExamineeContext
(CRT, NCLB)ConstructTest/Items
Accommodations Classification Targeted Accessible
RE
SE
SE
SE
![Page 21: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/21.jpg)
Minimizing Error• Random: Standardization – belief that random
error can be minimized by standardizing test administrations.
• Systematic: Construct Irrelevant Variance– Constant ~ group specific– Over/underestimation of scores~ “Students potentially provide the most serious threat to
CIV.” (Haladyna & Downing, 2004. p.23)
~ This brings us back to the test and how students interact with the constructs to be measured.
![Page 22: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/22.jpg)
Accommodations• Such tools change administrations from standard to
non-standard, threatening comparability of results. • Providing either a standard or non-standard
administration requires sacrifices: – random errors in a non-standard environment– systematic errors when a test is standard and inflexible to
the access of students to test information
The question is: at what point are the sacrifices impeding measurement precision and the validity
thereof?
![Page 23: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/23.jpg)
Back to Basics: Valid Assessment Systems• Improved student data
– Improved collection, particularly in light of Peer Review, to include subgroup data
– Supporting students– Improving decisions on accommodations and
standardization of the provisions thereof– Recognizing the assumptions of policy decisions:
classifications and accommodations
• Re-conceptualization of “standardization.” A more valid conceptualization may be what is standard for each examinee.
![Page 24: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/24.jpg)
Back to Basics: Valid Assessment Systems• Well targeted to clearly and operationally defined construct
– If we can’t define what we want to know, how do we know that what we know is what we want to know?
• Balanced and aligned expectations of:– standards – skills – range of difficulties
• Improved measurement precision – Reduction in random AND systematic errors– Expanded item sampling– Increased accessibility– Flexibility– UDA
~ the Goldilocks approach
![Page 25: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/25.jpg)
Past Research • Ways to “validate” accommodations – DIF,
EFA, cluster analyses, qualitative reviews, etc.
• Inconclusive results• Difficulties in conducting research:
– Experimental designs– Concurrent accommodations vs. single
accommodations– Confounding variables
![Page 26: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/26.jpg)
Past Research• Lack of consensus on what constitutes “valid”
accommodations– Does “boost” = validity? – Isn’t it possible that a valid accommodation
might increase precision in measurement and possibly reveal student inability – no boost?
![Page 27: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/27.jpg)
Continued & Future ResearchGiven the confounding variables of both persons and tests,
accommodations can not be validated apart from an in- depth look at the assessment and what it is trying to measure, in concert with how the accommodation by the student and test items interact. (Ex. - construct irrelevant variance by researchers Abedi, Kopriva, Winters,, et. al)
It must be clear how the accommodations affect skill measurement.
Therefore, future research should focus deeply on assessment validity in light of how the wide range of students, with all their diversities (and confounding variables), approach assessments.
![Page 28: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/28.jpg)
Continued & Future Research• Re-evaluation of test constructs• Research on all students, not limited to disability classifications• Is there a way to measure individual systematic error?• Research on distractors
– What are the types of errors students make/distractors students choose
• Think aloud studies focused on access and student response preferences
![Page 29: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/29.jpg)
Continued & Future Research
• Flexibility:– New item types and acceptable student
response modes– Approach flexible item types and research
thereof as parallel item forms and formats for more than the “accommodated” sample.
![Page 30: The Journey Toward Accessible Assessments Karen Barton CTB/McGraw-Hill Validity & Accommodations:](https://reader035.vdocuments.us/reader035/viewer/2022062620/551b59ba550346ae7a8b5727/html5/thumbnails/30.jpg)
General, accommodated, alternate, and modified alternate assessments can and should be – Better aligned to clearly defined constructs– More innovative by design, – Valid for more than the middle of the bell, and – More meaningful and useful.