“There’s no confidence in multiple-choice testing …”
Phil Davies
School of Computing
University of Glamorgan
South Wales
Who has NO confidence?
• STAFF– “They can just guess the answer”– “It’s not the way it is in the real world”– “there are varying degree of knowledge”
(Echternacht, 1972)
– “don’t worry it’s easy, it’s multiple choice”
• Possibly more importantly -- STUDENTS
The First Test
• 40 Questions / Formative
• Time Consuming / Difficult to create sensible distracters / Rocket Science
• I thought … Average 1 minute per question
• Actual … all finished in ten minutes …
apart from one who took 50 minutes• Why? No reason to think
On-Line Assessment &
Learning … OLAL• Level Two Degree .. Computer
Communications & Networks
• Failures High … Programming to Demonstrate .. Technical Subject .. Essay-type answers in examinations
• Assess .. “Knowledge” “Understanding” “Learning Outcomes” …. Directed
On Line Assessment & LEARNING
• Four Tests
• Weeks 3,6,9,12 = 5%, 10%, 15% 20%
• Cumulative
• Two Passes of SAME test
• Learning .. Short term?
• Slowed Down .. Thought about WHEN WRONG .
What decides the QUALITY of a question?
• In the recent World cup qualifier between England and Greece, who scored the all important equalizing goal?
• David Beckham
• Victoria Beckham
• Brooklyn Beckham
Rewards?
• Know it
• On the tip of my tongue
• If I see the answers I’ll know it
• Deduction
• Guess
• Know it … oh no I’ve got it wrong .. misinformation
Allocation of Marks
• Logarithmic +3, +2, +1, -3, -2, -1
• This is not the same!! -> not seeing answer
• Students were involved / Provisional– “You can try this out but it must not have a
negative affect upon our results compared with last year”
• Reward knowledge +4, +2, +1, -2, -1, 0
• If we don’t know it, then why take off 1
Results 2000-2001 (NO CONFIDENCE TESTING)Test Number Average % Standard
DeviationOLAL TEST 1 56.49 16.11OLAL TEST 3 49.23 13.25OLAL TEST 4 59.18 14.25
Results 2001-2002 (WITH CONDFIDENCE TESTING)OLAL TEST 1 63.10 15.27OLAL TEST 3 55.55 13.41OLAL TEST 4
+4, +2, +1, -2, -1, 058.60 15.42
OLAL TEST 4+3, +2, +1, -3, -2, 0
51.33 17.32
OLAL TEST 4+3, +2, +1, -3, -2, -1
50.13 18.08
OLAL TEST 4+5, +3, +1, -3, -2, -1
56.45 16.43
OLAL TEST 4+5, +3, +1, -2, -1, 0
61.35 14.60
2002 OLAL 1/3 .. Better by approx. 6%
+5, +3, +1, -2, -1, 0
2000 +2 -1
05
1015202530
90to99
80to89
70to79
60to69
50to59
40to49
30to39
20to29
10to19
0to9
2000 +2 -1
2001 +4 +2 +1 -2 -1 0
05
1015202530
2001 +4 +2 +1 -2-1 0
2000 +2 -1
05
1015202530
90to99
80to89
70to79
60to69
50to59
40to49
30to39
20to29
10to19
0to9
2000 +2 -1
2001 +5 +3 +1 -2 -1 0
05
1015202530
2001 +5 +3 +1 -2-1 0
Ma
rks
20
00
-20
01
No
co
nfi
de
nc
e
%
20
01
-20
02
Co
nfi
de
nc
e
5,3
,1,2
,1,0
%
90-99 0 380-89 6 670-79 23 1960-69 25 2450-59 24 2740-49 12 1430-39 7 320-29 3 310-19 0 1
0-9 0 0
0
5
10
15
20
25
30
90 -99
80 -89
70 -79
60 -69
50 -59
40 -49
30 -39
20 -29
10 -19
0 - 9
No confidence
Confidence
Percentile Frequency Distribution
Average Mark Improvement, Tests 3 to 4Mark Rangevia OLAL 3
2000 (no confidence) 2001 Confidence(5,3,1,-2,-1,0)
80-89 4.3 7.0070-79 3.85 3.5960-69 7.32 3.6350-59 6.37 5.0740-49 10.52 5.8030-39 12.44 11.1020-29 20.71 19.5510-19 35.05 26.11
0-9 35.39Quartiles
75+ 5.98 4.2350+ 6.62 3.2525+ 10.78 5.260+ 16.29 10.49
AverageImprovement
9.95 11.00
Number of Questions correct (out of 60)MARKSTest 3
2000No Confidence
2001Confidence
80-89 55.00 56.0070-79 50.00 50.8660-69 48.71 47.7450-59 44.23 44.6640-49 42.52 41.5830-39 39.23 40.0020-29 38.66 39.6710-19 37.50 37.00
0-9 37.00Quartiles
75+ 48.48 50.5850+ 44.33 46.8025+ 42.93 43.550+ 39.10 40.27
Proportion of Questions Correct
• High Confidence => 84% correct
• Fairly Confident => 58% correct
• Not confident => 47% correct
• 72% of all questions answered with high confidence
Based on Test 3, who were the most confident (60 Questions)
• 75+– 52 7 1
• 50+– 44 13 3
• 25+– 42 15 3
• 0+– 33 19 8
Student Feedback Thoughts on Multiple Choice
• “useful learning tool”
• “second test is a really useful way of learning”
• “immediate feedback helps me to learn”
• “doesn’t reflect my lack of knowledge”
• “not fair, I found it difficult to tell the difference between the answers”
• “negative marking made me panic”
Student thoughts on Confidence
• “Eliminates guesswork”
• “I was waiting for a leggy blond to bring on my prize at the end”
• “separated the lucky students from me”
• “at least I didn’t lose marks for selecting no confidence”
• “are you testing my confidence or knowledge”
Quality of a multiple choice question?
• In the recent World cup qualifier between England and Greece, who scored the all important equalizing goal?
• Teddy Sheringham• Michael Owen• Robbie Fowler• QUALITY OF QUESTION NOT
DISTRACTERS
Question Format• “One or more questions were impossible to
judge”
• “I wish you’d put more detail in the question”
• “Perhaps it would have been better to show the answers, ask for confidence, and then show the question”
Conclusions?• Assess subject area not individual’s
confidence
• Quality of questions => question NOT distracters
• Allocation of Marks (based on past?)
• Student Accept / Staff Accept
• Time for tests increases “really had to think”
