© caveon, llc legal issues in testing (part one) a legal framework for the appropriate and...
TRANSCRIPT
© Caveon, LLC
Legal Issues in TestingLegal Issues in Testing(Part One)(Part One)
A legal framework for the appropriate and defensible use of Caveon Data Forensics™
June 21, 2006
Bob Hunt, J.D. Ph.D.Vice President of Legal Services and General Counsel
Caveon Test Security
© Caveon, LLC
Overview …Overview …
What is “data forensics” The need for competitive fairness in
testing Adequacy of proctoring Legal rights to use statistics to:
monitor testing behavior cancel scores and take other actions
Examples of Concepts
© Caveon, LLC
Fairness in TestingFairness in Testing
Does the test objectively evaluate relevant knowledge and skills?
Does the test provide consistent
results?
Is the testing process equally fair?
© Caveon, LLC
Why is Competitive Fairness Why is Competitive Fairness Important?Important?
Without competitive fairness, tests are simply biased rituals
The validity and reliability of test results hinge on fair competition
Consistent results
(reliability)
Test Relevance (validity)
Relational Fairness(level playing field)
© Caveon, LLC
The (in)Adequacy of ProctoringThe (in)Adequacy of Proctoring
Cheating behaviors are increasingly “invisible” to proctors Theft of test material Wider use of stolen test content
obtained from the Internet Wireless communications
Human proctoring is hampered by: Scope: one to many Errors of observation, memory and bias
© Caveon, LLC
Data Forensics Promotes Data Forensics Promotes Competitive FairnessCompetitive Fairness
More comprehensive (no limits on effectiveness due to volume of cheating)
Fewer errors of observation, memory and bias (need to elaborate why scientific error is more trustworthy)
Flags “trusted” test results Reinforces other detection efforts
Confirm or clear other suspicions
© Caveon, LLC
Caveon Data Forensics Caveon Data Forensics
“Low-resolution” analyses1.Gain-score2.Response aberrance3.Latency aberrance
“High-resolution” analyses1.Copying indices2.Erasures
© Caveon, LLC
Questions?
© Caveon, LLC
The $1,000,000 Legal QuestionThe $1,000,000 Legal Question
Can statistics improve test security if they don’t provide “first-hand” evidence of cheating?
© Caveon, LLC
Answer: Cheating is OverratedAnswer: Cheating is Overrated Testing programs have broad authority to
manage the validity of test results Courts only require a general “lack of
confidence” in a test result Lots of (other) reasons to lack confidence in
test results: Disruption Illness Technology issues Mishandling, etc.
© Caveon, LLC
“We are satisfied that the relevant public and private interests are fairly accommodated by a procedure which permits ETS to cancel scores upon an adequate showing of substantial question as to their validity, without any necessity for a showing of actual cheating or other misconduct.” Scott v. ETS (N.J. Sup.1991)
Validity vs. CheatingValidity vs. Cheating
© Caveon, LLC
Courts Understand the Need for a Courts Understand the Need for a Broader SolutionBroader Solution
“To demand that ACT prove by eyewitness testimony that an individual cheated before invalidating a score would undermine ACT's primary function of providing colleges with scores that are highly reliable.”
“ACT could not possibly catch every student who cheats on its exams if it had to produce an eyewitness to confirm every instance of misconduct.”
Langston v. ACT, (11th Cir 1989)
© Caveon, LLC
Question: Does that Mean that Test Question: Does that Mean that Test Scores can be Cancelled for ANY Scores can be Cancelled for ANY
Reason? Reason?
Answer: Nearly Testing programs enjoy broad
discretion in evaluating the validity of test scores
Score cancellations based on statistical evidence have been upheld
Courts that have addressed the issue directly have stated that it is not necessary to prove wrongdoing
© Caveon, LLC
Implement Uncertainty!Implement Uncertainty!
“Your scores may be classified as indeterminate if the scores are at or above the passing level and the USMLE program cannot certify that they represent a valid measure of your knowledge or competence as sampled by the examination.”
U.S. Medical Licensing Board
Test retakes replace committee reviews, hearings, etc. as the chief due process opportunity
© Caveon, LLC
Questions?
Case Studies: Scott v. ETS: Private/Contract law
Murray v. ETS: Public/Constitutional law
© Caveon, LLC
Scott v. ETS Scott v. ETS (N.J. Sup. 1991)(N.J. Sup. 1991)
National Teachers Examination administered by ETS for the State
of New Jersey Decided on constitutional grounds
© Caveon, LLC
Scott v. ETS (cont.)Scott v. ETS (cont.)
ETS detected a score gain of 42 points Scott’s second and third attempts
ETS then detected similarity between the Scott’s test-responses and those of another examinee (not seated near her).
ETS cancelled Scott’s score but offered her an opportunity to retest.
© Caveon, LLC
Scott v. ETS (cont.)Scott v. ETS (cont.)
The issue: Whether Scott had been denied due
process by ETS’s failure to prove that she had actually cheated.
© Caveon, LLC
Scott v. ETS (cont.)Scott v. ETS (cont.)
The Decision: “We are satisfied that the relevant public and
private interests are fairly accommodated by a procedure which permits ETS to cancel scores upon an adequate showing of substantial question as to their validity, without any necessity for a showing of actual cheating or other misconduct.”
© Caveon, LLC
Scott v. ETS (cont.)Scott v. ETS (cont.) The “relevant public and private interests”
ETS: accuracy of its test results and predictions
Test-takers: no unfair advantage
School officials: reliability Public: reserving teacher certification for
those who fulfill its requirements
© Caveon, LLC
Scott v. ETS (cont.)Scott v. ETS (cont.)
Meaning of “Substantial Question of validity”:
“Here ETS questioned plaintiff's scores on the basis of a statistical analysis showing hardly more than a 4 in 10 million chance that they were fairly earned … That gave ETS, and would give any other observer, substantial grounds for doubting the reliability of the scores.”
© Caveon, LLC
Case Study: Murray v. ETS Case Study: Murray v. ETS (5(5thth Cir. 1999) Cir. 1999)
U. Texas El-Paso basketball recruit Needed an SAT score of 820 (of 1600) to
receive a scholarship Scored 700 on his first attempt in March Enrolled in “Testbusters” Scored 1300 in June
© Caveon, LLC
Murray v. ETS (cont.)Murray v. ETS (cont.)
Test-use agreement:
“ETS reserves the right to cancel any test score if … the student engages in misconduct, if there is a testing irregularity, or if ETS believes there is a reason to question the score's validity.”
SAT I Registration Bulletin
© Caveon, LLC
Murray v. ETS (cont.)Murray v. ETS (cont.)
The Decision: “The only contractual duty ETS owed to Murray
was to investigate the validity of Murray's scores in good faith.”
Good-faith means: Having a reason to lack confidence in a test
result (from statistics or some other source) Providing an opportunity to retake the test and
to present other evidence supporting scores Allowing independent review
© Caveon, LLC
Murray v. ETS (cont.)Murray v. ETS (cont.)
“ETS dutifully fulfilled its contract [ ..] by following established procedures for determining the validity of questionable scores.”
© Caveon, LLC
ConclusionsConclusions
Statistical analyses of test data like Caveon Data Forensics provide powerful test security information
Managing cheating with test-use agreements, statistics and retakes is within the legitimate power of testing programs to manage the “validity” of their tests
© Caveon, LLC