8.setting and maintaining examination standards
DESCRIPTION
PPTTRANSCRIPT
SETTING & MAINTAINING SETTING & MAINTAINING EXAM STANDARDSEXAM STANDARDS
Raja C. Bandaranayake
DEFINITIONSDEFINITIONS
Standard setting is a process of determining how much is good enough.
The standard or criterion level of performance is a point on the scale of measurement at which separation of competence and incompetence occurs.
Cut-score, cut-off score or passing score represents this standard on a given test for making decisions pertaining to the purpose for which the test was conducted, e.g., to certify competence.
ERROR IN MEASUREMENTERROR IN MEASUREMENT
True score is a conceptual measure indicating true extent of competence in a given subject, e.g., Anatomy.
Observed score is the score assigned as a result of taking a test, say in Anatomy.
The difference between true and observed scores is indicative of the amount of error in the measurement.
The reliability of a test and the associated standard error of measurement are estimates of the amount of error in the measurement.
DECISION ERRORSDECISION ERRORS
False positive: passing an incompetent examinee
False negative:failing a competent examinee
NORM- & CRITERION-REFERENCED NORM- & CRITERION-REFERENCED STANDARDSSTANDARDS
NORM-REFERENCED
• Relative
• Based on peer-performance
• Varies with each group
• Cut-off point not related to competence
CRITERION-REFERENCED
• Absolute
• Not related to peer performance
• Standard set prior to exam
• Referenced to a defined level of performance
METHODS OF STANDARD SETTINGMETHODS OF STANDARD SETTING
1. Test-centred methodsStandards derived from hypothetical decisions based on test content before the test is answered.
2. Examinee-centred methodsStandards derived from reviewing examinees’ performance before deciding cut-off score.
3. Compromise methods Provide flexibility for adjusting the standard based on the
examinees’ performance on the test.
NEDELSKY (1954) METHOD: NEDELSKY (1954) METHOD: ExampleExample
Consider N judges and n MCQ items of 1 in 5 type
Judge A identifies 2 options in item 1 as those which a minimally competent examinee should eliminate as incorrect.
MPL for that item for Judge A [MPLA1] = 1/(5-2) = 1/3 Similarly, in item 2 he identifies 3 options, giving an MPLA2 =
1/(5-3) = 1/2 He repeats this process for each item.
The exam MPL for Judge A [MPLA] = MPLA1 +MPLA2 + MPLA3 + ………….MPLAn
Similarly, Judge B’s MPL [MPLB] is determined
The MPL for the exam (= cut-off score) is: (MPLA + MPLB + MPLC +….MPLN) / N
ANGOFF (1971) METHODANGOFF (1971) METHODExampleExample
N judges consider 100 minimally competent examinees taking an MCQ exam of n items.
Judge A estimates that, of these examinees, 50 should answer item 1 correctly, 20 item 2 correctly, 70 item 3 correctly, and so on to item n.
The MPL for Judge A [MPLA] = (0.5 + 0.2 + 0.7 + . xn) / n X 100 = (say) A%.
Similarly, for Judges B, C, D, E, …..N, the MPLs would be B%, C%, D%, E% ……N%, respectively.
The MPL (cut-off score) for the exam is: (A% + B% + C% + D% + E% +....N%) / N
EBEL (1972) METHODEBEL (1972) METHODExampleExample
Assume that Judge A assigns items in a 200-item MCQ test to the cells of a “relevance-by-difficulty” matrix, as follows.
He then estimates the percentage of items in each cell of the matrix that a minimally competent examinee should be able to answer correctly (as indicated within the cell).
Each cell also includes the products of these two values.
EASY MEDIUM HARD
ESSENTIAL 15 x 100% = 1500 25 x 80% =2000 10 x 60% = 600
IMPORTANT 20 x 80% = 1600 40 x 60% =2400 20 x 50%
=1000 ACCEPTABLE 10 x 50% = 500 25 x 40% = 1000 5 x 10% =
50 QUESTIONABLE 10 x 30% = 300 15 x 20% = 300 5 x 0% = 0
EBEL (1972) METHOD - EBEL (1972) METHOD - contd.contd.ExampleExample
The MPL for Judge A [MPLA] is then:
(1500 + 1600 + 500 + 300 + 2000 + 1000 + 300 + 600 + 1000 + 50 + 0) / 200 = 56.25 %
Similarly, the MPL for Judges B [MPLB], C [MPLc], D [MPLD] …..N [MPLN] are determined.
The MPL for the exam (cut-off score) is: (MPLA+ MPLB+ MPLc+ MPLD + …..MPLN) / N
PROPOSED EBEL MODIFICATIONPROPOSED EBEL MODIFICATION
EASY MEDIUM HARD
ESSENT. 6x 100% = 600 12 x 80% = 960 7 x 50% = 350
IMPORT. 12 x 80% = 960 24 x 60% = 1440 19 x 40% = 760
ACCEPT. 5 x 60% = 300 12 x 50% = 600 3 x 10% = 30
MPL: =600 + 960 + 350 + 960 + 1440 + 760 + 300 + 600 + 30
=6000/100
= 60
FailureRate%
Cut-off score(%)
10
15
20
35 40 45 50
fmin
fmax
cmin cmax
A
B
HOFSTEE METHOD
HOFSTEE METHOD
Example
A plot of cut-off scores for a given exam
against resulting failure rates is given
cmin = 40%
cmax = 45%
fmin = 10%
fmax = 20%
A = point representing cmin,fmax
B = point representing cmax,fmin
Line AB intersects the curve at a cut-off point
of 42.5%
Thus, operational cut-off score = 42.5%
CUT-OFF SCORE FOR 1 IN 5 MCQCUT-OFF SCORE FOR 1 IN 5 MCQ[FRACS PART 1][FRACS PART 1]
Probability of guessing (=1 in 5) = 20%
‘Total ignorance’ score = 20%
Maximum possible score =100%
Effective range of scores = 20% to 100%
Mid-point of this range = 60%
Additional factor (as PG exam) = 5%
Nominal cut-off score (60%+5%) = 65%
•
CUT-OFF SCORES: CUT-OFF SCORES: “MARKER QUESTIONS”“MARKER QUESTIONS”
1. Comparison of exam scores
Mean score in this exam: 56.7%
Average exam mean score over last 4 years: 59.4%
Thus mean score in this exam is: 2.7% lower
Assuming this candidate group is of same standard as in last 4 yrs, this exam is: 2.7% harder
CUT-OFF SCORES: CUT-OFF SCORES: “MARKER QUESTIONS” - contd.“MARKER QUESTIONS” - contd.
2. Comparison of “marker” scores
Mean score in this exam on previously used questions (N=162): 62.5%
Mean score on same questions when they were each last used: 60.5%
Thus, when compared with previous candidates, this group of candidates, on these items, scored (62.5-60.5)% = 2.0% higher
Thus this group of candidates is: 2.0% better than previous groups
CUT-OFF SCORES: CUT-OFF SCORES:
“MARKER QUESTIONS” – contd.“MARKER QUESTIONS” – contd.
3. Estimating examination difficulty
Thus it is expected that their mean score in this exam would be: 2.0% higher
But their mean score in this exam is: 2.7% lower
Thus this exam is really: 4.7% harder
CUT-OFF SCORES: CUT-OFF SCORES:
“MARKER QUESTIONS” –contd.“MARKER QUESTIONS” –contd.
4. Determining cut-off score
The cut-off level for an average exam is: 65.0% Thus the cut-off level for this exam
should be (65 – 4.7)% = 60.3%
Cut-off score = 60.3%Cut-off score = 60.3%
FailureRate%
Cut-off score(%)
10
15
20
55 60 65 70
HOFSTEE CURVE