collaborative examination item review process in a team-taught course
TRANSCRIPT
COLLABORATIVE EXAMINATION ITEM
REVIEW PROCESS IN A TEAM-TAUGHT
COURSE
Adam Pate, PharmD; David J. Caldwell,
PharmD; Laurel Sampognaro, PharmD
Objectives
1. Describe how to implement a collaborative item review process
2. Identify potential barriers to implementation and success of a
collaborative item writing process
3. Generate ideas to establish a collaborative process at your
respective institution
Baseline information about us
A review process was created to limit variability of
content and quality of items in the course
The first meeting is crucial in setting the tone
for the semester
All instructors relied on principles from
Haladyna et. al. to define “good” items
Real quick item statistics
• Point biserial correlation (rpb)
• Difficulty (p)
Item classification guide
Item Class Item DifficultyItem Discrimination
(point biserial)Description
Level I 0.45 to 0.75 +0.20 or higherBest item statistics; use most
items in this range if possible
Level II 0.76 to 0.91 +0.15 or higher Easy; use sparingly
Level III 0.25 to 0.44 +0.10 or higher
Difficulty; use very sparingly and
only if content is essential--rewrite
if possible
Level IV <0.24 or >0.91 Any discriminationExtremely difficult or easy; do not
use unless content is essential
Faculty discussed these guidelines and
came to group consensus on usage
Guideline For (%) Uncited (%) Against (%)
Use positives, no negatives 63 19 18
Write as many plausible
distractors as you can70 26 4
Use carefully None of the Above 44 7 48
Avoid All of the Above 70 7 22
Use humor sparingly 0 85 15
Item Review Process
• Who is involved?
• Self Care I – 11 faculty
• Self Care II – 9 faculty
• Participation
Item Review Process
Deadline for exam items
Test creation and dissemination
Meeting preparation
Item Review Process
• Meeting details
• Environment
• Discussion
• Agreement
• Follow-up after meeting
Item Review Process
• What worked well?
Item Review Process
• Barriers to implementation
Self Care 1(without training or review)
Self Care 2
(without training or review)
Spring and Fall 2012, control sequence
6 exams, 272 items
(without training or review)
Spring and Fall 2013, intervention sequence
Self Care 1(with training or review)
Self Care 2
(with training or review)
6 exams, 264 items
(without training or review)
Interventions
1. Pre-semester survey
2. Presentation of item-writing
guidelines at semester start
3. Guideline review and discussion
at each exam review meeting
4. Review and editing of exam
items per guidelines
5. Post-semester survey
Comparisonsitem difficulty, discrimination, and
classification by these factors, and
student performance
All interventions were completed in both
Self Care 1 and 2; instructors teaching in
both only completed surveys in
Self Care 1
ComparisonsPre- versus post-survey
About the participating faculty
NOTABLE BASELINE REPORTS
Which of the following factors affect your
sense of success in item writing?
• Item statistics (n=9)
• Previous training in item writing (n=4)
• Student challenges to exam items (n=3)
How often have you participated in peer-
review of exam items?
• Half of the time (n=4)
• A minority of the time (n=1)
• Never (n=5)
Results
GOALS
1. To improve examination quality through a faculty development program,
followed by a longitudinal item review occurring before examination
administration
2. To improve faculty members’ self-rated confidence and success
3. To measure changes in their opinions regarding item-writing guidelines
and review.
Goal 1: What happened to item-quality?
Results
• No significant difference between
the control and intervention items,
respectively
• Mean student scores (% ± SD)
did change (p<0.001):
• Control sequence, 88.3 ± 4.5
• Intervention sequence 85.6 ± 6.0
Distribution by level
CONTROL ITEMS INTERVENTION ITEMS
Distribution by level
ITEM CLASS WITH REVIEW, n (%) WITHOUT REVIEW, n (%)
Level 1 31 (11.4) 52 (19.7)
Level 2 70 (27.5) 76 (28.8)
Level 3 7 (2.6) 3 (1.1)
Level 4 142 (52.2) 122 (46.2)
Uncategorizable 22 (8.1) 11 (4.2)
Goal 2: How did participants’ self-rated
confidence and success change?
Survey opinions: self-focused
ITEM Pre (mean) Post (mean) p-value
How would you rate your confidence at writing effective
multiple-choice test items?
(0 – very unconfident, 10 – very confident)
6.0 8.1 0.002
How would you rate your success at writing multiple-
choice test items?
(0 – very unsuccessful, 10 – very successful)
6.4 7.9 <0.001
To what degree do you feel confident that you can
properly evaluate your and your colleagues' test
questions?
(0 – very unconfident, 10 – very confident)
6.7 8.4 0.005
To what degree do you feel confident that you could
implement a formal exam item evaluation process
as a coordinator of another course?
(0 – very unconfident, 10 – very confident)
5.5 7.1 0.008
Goal 3: How did participants’ opinions of
item guidelines and review change?
Survey opinions: item-focused
ITEM Pre (mean) Post (mean) p-value
In your opinion, to what degree will peer-review of exam items affect item quality?
(0 – very negatively, 10 – very positively)
7.9 8.5 0.14
Do you plan to modify future multiple-choice items based on item-writing guidelines?
(1 – Yes, 2 – No)
9 Y
1 N
10 Y
0 N1.00
In your opinion, to what degree will voluntary application of item-writing guidelines affect item quality?
(0 – very negatively, 10 – very positively)
7.9 8.4 0.24
Other findings
Top 5 item flaws
GUIDELINES n (% of total changes)
Include the central idea in the stem instead of the choices. 37 (33.6)
Use correct grammar, punctuation, capitalization, and spelling. 17 (15.5)
Minimize the amount of reading in each stem. 13 (11.8)
Use the question, completion, and best answer versions of the conventional
multiple choice (MC), the alternate choice, true-false, multiple true-false,
matching, and the context-dependent item and item set formats, but avoid the
complex MC (Type K) format.
10 (9.1)
Keep choices independent; choices should not be overlapping. 7 (6.4)
Comparison of undesirable format frequency
Item Review Process
• Experience in other courses