the future of online testing with moocs: an exploratory analysis of current practice
TRANSCRIPT
![Page 1: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/1.jpg)
The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice
Eamon Costello (National Institute for Digital Learning, Dublin City University)
Jane Holland (Royal College of Surgeons in Ireland)
Mark Brown (National Institute for Digital Learning, Dublin City University)
![Page 2: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/2.jpg)
Background
• NIDL - DCU and Funded MOOC research– MOOCs and building regional capacities
SCORE2020 Project– HOME Project• MOOCs and the Media• Institutional Drivers
![Page 3: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/3.jpg)
Possible MOOC Futures
![Page 4: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/4.jpg)
Possible MOOC Futures
![Page 5: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/5.jpg)
Possible MOOC Futures
![Page 6: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/6.jpg)
Brightest MOOC Futures
Dillenbourg, P. (2015) Proposal for a Digital Education Strategy for Flanders Universities. “Thinkers in Residence” Programme from KVABKoninklijke Vlaamse Academie van België voor Wetenschappen en Kunsten. Available from: http://www.kvab.be/denkersprogramma/files/DP_BlendedLearning_No-time-to-lose.pdf
“Sooner or later, online tests will be as reliable or even more reliable than on campus exams”
![Page 7: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/7.jpg)
Current State of Play
• The future is taken care of• But what about the present?– How mature are (x)MOOCs?– How reliable and valid are MCQ type tests in
MOOCs?
![Page 8: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/8.jpg)
Multiple Choice Question (MCQ) Tests (Single Best Answer)
– Reliability: If we repeated this would we get the same result?
- Validity:Are we measuring what we think we are?
Don’t get “fooled by randomness”
Taleb, N. (2004). Fooled by randomness: The hidden role of chance in life and in the markets. Random House Incorporated.
![Page 9: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/9.jpg)
Best Practice
Case, S. M., & Swanson, D. B. (2003). Constructing written test questions for the basic and clinical sciences (3rd ed.). Philadelphia, PA: National Board of Medical Examiners.
![Page 10: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/10.jpg)
• Ambiguous or unclear information• Negative worded stem (not, incorrect, except)• Implausible distracters • Gratuitous information in stem• More than one or no correct answer• Longest option is correct• Logical cues in stem • Word repeats in stem and correct answer• Unfocused stem • True/false question • Use of all of the above • Vague terms (sometimes, frequently)• Absolute terms (never, always)• Use of none of the above• Fill-in-blank • Complex or K-type • Grammatical cues in sentence completion• Convergence cues
Best Practice
Tarrant, M., Knierim, A., Hayes, S. K., & Ware, J. (2006). The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments. Nurse Education Today, 26(8), 662-671
![Page 11: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/11.jpg)
• Ambiguous or unclear information• Negative worded stem (not, incorrect, except)• Implausible distracters • Gratuitous information in stem• More than one or no correct answer• Longest option is correct• Logical cues in stem • Word repeats in stem and correct answer• Unfocused stem • True/false question • Use of all of the above • Vague terms (sometimes, frequently)• Absolute terms (never, always)• Use of none of the above• Fill-in-blank • Complex or K-type • Grammatical cues in sentence completion• Convergence cues• Position of correct option
Best Practice
Tarrant, M., Knierim, A., Hayes, S. K., & Ware, J. (2006). The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments. Nurse Education Today, 26(8), 662-671
![Page 12: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/12.jpg)
Methodology
• Use Tarrant et. al. (2004)’s diagnostic tool to analyse MCQs in MOOC systematically
• Look for item writing flaws
![Page 13: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/13.jpg)
The Data
• 12 Courses – from six MOOC platforms – From 12 Universities/Providers
• Total MCQs: 115
![Page 14: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/14.jpg)
None/All of the Above
• None of the above: 1 (0.87%)• All of the above: 2 (1.74%)
The kernel is defined as:
A. The graphical user interface on top of the operating systemB. The glue between hardware and software applicationsC. the software libraries need to run the systemD. all of the above
Holsgrove, G., & Elzubeir, M. (1998). Imprecise terms in UK medical multiple choice ‐questions: what examiners think they mean. Medical Education, 32(4), 343-350.
![Page 15: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/15.jpg)
Number of Correct Options per Question
• Greater than 1 correct: 10 (8.7%)• Average: 1.39
Check the words that are used synonymously in a JMeter test plan
A. end usersB. virtual usersC. concurrent usersD. threads
![Page 16: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/16.jpg)
Number of Options per QuestionGreater than 4 options: 9 (8%)Average: 3.77
1 option 2 options 3 options 4 options 5 options 7 options0
20
40
60
80
0%
10%
20%
30%
40%
50%
60%
70%
80%
2%6%
17%
68%
6%2%
![Page 17: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/17.jpg)
Correction Option is the Longest
• In 47 of the 115 MCQs the correct option is the longest (40.87%)
![Page 18: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/18.jpg)
Position of Correct Option 105 MCQs
1st 2nd 3rd 4th 5th 0
10
20
30
0%
5%
10%
15%
20%
25%
30%
35%
21%
30%29%
19%
1%
![Page 19: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/19.jpg)
1st 2nd 3rd 4th 0
5
10
15
20
25
15%
27%
32%
26%
Position of Correct Option in 73 Four-Option MCQs
![Page 20: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/20.jpg)
Overall
• 17 (14.78%) all the questions contain a defined item writing flaw
• When counting only four item writing flaws• Two more item writing flaws are apparent in
characteristics that appear more often than they should be chance (small sample)
![Page 21: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/21.jpg)
Further workAdd qualitative analysis using the Tarrant et. al. (2004) evaluation tool
• Ambiguous or unclear information• Negative worded stem (not, incorrect, except)• Implausible distracters • Gratuitous information in stem• Logical cues in stem • Word repeats in stem and correct answer• Unfocused stem • Vague terms (sometimes, frequently)• Absolute terms (never, always)• Fill-in-blank • Complex or K-type • Grammatical cues in sentence completion• Convergence cues
![Page 22: The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice](https://reader030.vdocuments.us/reader030/viewer/2022032501/55b6d258bb61eb8c418b4748/html5/thumbnails/22.jpg)
Implications
• Validity and Reliability of MOOC Testing• Replicating unsound pedagogies• MOOC teachers/developers need evidence-lead
teaching