caveon webinar series - discrete option multiple choice april 2014
DESCRIPTION
It's not news that those of us in the testing world struggle with a variety of challenges. What's one of the largest? Test security - which is not likely to go away anytime soon. But, we also have measurement issues as our tests sometimes measure more than we want them to, and may not be as fair to certain subgroups in the population. In addition, costs to develop and administer tests have gone up with the use of technology, a fact opposite to how technology use has generally affected other industries. Finally, testing critics seem to be getting bolder in their claims, some of which are uncomfortably close to accurate. This webinar looks at how, through technology, changing the traditional multiple choice question can help substantially with these challenges and others. You won't want to miss this webinar, hosted by Caveon CEO David Foster. Dr. Foster will present information from his recently published white paper Compelling Test Design Technology: Discrete Option Multiple Choice.TRANSCRIPT
Caveon Webinar Series:
Solving Big Problems with DOMC, a 21st Century Technology
April 23, 2014
Dr. David Foster, CEOCaveon Test Security
2
Agenda for Today
• What is DOMC?• DOMC effects on testing
problems• Creating DOMC items• SUPER DOMC• Challenges with DOMC
3
Major Problems We Face
Security is abysmal
Quality of measurement can be better
Costs are too high
Every test is unfair to someone
4
DOMC can help…
What is DOMC?
Discrete Option Multiple Choice
Simplest Description: Options for the multiple choice question are presented randomly one at a time with YES and NO buttons.
Options stop appearing when the test taker answers the question correctly or incorrectly.
5
DOMC Examples
The best way to understand DOMC is to try it out…going to trydomc.com
We are going to look at some varieties:• simple multiple choice• multiple choice with more than one correct
answer• multiple choice scored correct with responses
to distractors only• DOMC with larger option pools (SUPER DOMC)
6
How can DOMC help solve these
problems?
7
Major Problem #1: Tests Are Stolen and Used to Cheat
• Tests are stolen routinely and shared quickly on the Internet.
• New technologies (e.g., Google Glass) will be very effective.
• Cheating by using pre-knowledge of questions is impossible to detect by proctors.
• Old fashioned cheating remains a threat.• Security problems directly affect the test score
and society’s trust in those scores.
8
Security Solution: How Does DOMC Help?
• DOMC exposes much less of the item content • Makes it more difficult to steal accurate
questions– SCIENTIFIC RESEARCH:
• 2.5 options on the average are presented compared to 4 or 5 for the traditional MC item.
• About a 50% reduction in the exposure of critical content.
– Absolutely NO MEASUREMENT LOSS– SUPER DOMC reduces the exposure more
dramatically (more on this later)
9
How Does Reduced Exposure Improve Test Security?
• You can’t capture what you can’t see!• You can’t share what you didn’t see!• If you want to cheat, you can’t depend
on what others tell you about the test!
10
RESEARCH: College Students
11
Expert Opinion
“DOMC testing may also considerably reduce the danger of unwanted copying and sharing of test items because many answer options do not have to be presented at all.”
Sonja Willing, Martin Ostapczuk, and Jochen Musch, University of Dusseldorf, in Advances in Health Science Education, 2014
12
Major Problem #2: Quality of a score from multiple
choice tests is less than it should be
What would happen to the quality of a cake if you added 10% of some totally unrelated ingredient? Chili Peppers? Vinegar? Asphalt?
• Traditional multiple choice questions are like this cake.
13
Major Problem #2: Quality of a score from multiple
choice tests is less than it should be
• SCIENTIFIC RESEARCH: TESTWISENESS adds about 10% to 15% to a person’s score.
• This is a large portion of a score (and time/money) devoted to measuring an irrelevant skill!
• It is CRITICAL that we remove as much TESTWISENESS as possible if we want to more accurately measure what is important.
14
Measurement Solution: How Does DOMC Help?
• SCIENTIFIC RESEARCH: – DOMC reduces TESTWISENESS effect
• Test takers cannot compare options, a major tactic of TESTWISENESS
• Use of TESTWISENESS is also a FAIRNESS issue. (more on this later)
15
Windfall
Test takers soon realize that “being prepared” means learning the subject matter of the test, not learning how to game the question format.
16
RESEARCH: Student Reactions
College students prefer the traditional MC format…
…because DOMC removes test taking cues
Of course, they would also prefer not taking tests at all, not going to the dentist, and not having their ID’s checked.
17
Major Problem #3: Tests cost too much to create and deliver
Programs create large numbers of items for security reasons (to foil thieves).
The cost to administer a computerized test is higher than it should be, in my opinion.
This may not be a major concern to most programs, but who wouldn’t like more cash to spend?
18
Cost Solution:How Can DOMC Help?
In at least two ways:1. DOMC test questions remain in service,
uncompromised, for much longer, perhaps indefinitely. Test development costs are cut significantly.
2. SCIENTIFIC RESEARCH: Testing seat time is reduced by 10% to 39%.
1. Less seat time means less money paid for test administration.
2. Opportunity for greater testing volumes.
19
Major Problem #4: Fairness
FAIRNESS Facts:The testing industry is getting hammered
by critics—justifiably so in some cases1. We allow TESTWISENESS in multiple choice
tests2. Our multiple choice tests are unfair to some
sub-groupsThese are, in my opinion, accurate
criticismsThe most NOBLE thing we can do is make
our tests as fair as possible.
20
How Does DOMC Help with Unfairness?Removes Some Inequality of Test Prep
A major goal of test prep courses is to teach tactics to best answer multiple choice questions (TESTWISENESS).
Test preparation courses costs money and can’t be afforded by many test takers, who are disadvantaged as a result.
DOMC removes the need to take test prep course to learn test taking skills.
21
How Else Is DOMC More Fair?It Is Simple to Do!
DOMC format is cognitively simpler making it easier for all, including:
– young children– older adults– those with test anxiety– those where the language
of the test is not their native language
– those where the culture does not include MC tests
– many others
22
Creating DOMC Items
• Item banking and test administration systems need to be modified for DOMC, just as they need to be changed for any new item type.
• Kryterion has supported a simple version of DOMC.
• Excelsoft has built the more complete version of DOMC.
• Here is a screen from their system, SARAS.
23
SARAS Test Development Screen #1: Stem
24
SARAS Test Development Screen #2: Options
25
SARAS Test Development Screen #3: Scoring/Stopping Rules
26
SUPER DOMC Item
Definition: A SUPER DOMC item has more correct and incorrect options than would be normal for the traditional version.
For example, having 3 correct and 7 incorrect options would enhance the advantages
described in these slides.
It is not always easy to do to add options, but almost always possible.
27
Example: SUPER DOMC Item
Objective: Add two 2-digit numbers.
MC Stem: Is this addition correct?
Correct Options look like this: 12 + 79 = 91There are hundreds of possible correct options
Incorrect Options look like this: 33 + 17 = 40There are thousands of possible incorrect options
28
Challenges with DOMC
• Most test takers don’t like it.• Takes only a little bit of practice.• Item banking systems and test
administration systems have to be modified.
• Test administration must use a “forward-only” design (e.g., no re-viewing items).
29
To Learn More…There is no charge to the licensee to try out DOMC or for doing research.
Caveon will provide you will the help you need to get started.
Link to DOMC white paper – http://www.caveon.com/white-papers/
If questions, email [email protected]
30
Questions?
31
Thank you!
Dave Foster, CEO
Caveon Test Security
Follow Caveon on twitter @caveonCheck out our blog www.caveon.com/blogLinkedIn Group “Caveon Test Security”