consistent application pf learning outcomes in assessment
TRANSCRIPT
Consistent Application of
Learning Outcomes in
AssessmentAACOM 2016
Dr. Susan Mackintosh
Dr. Colleen Talbot
Dr. Gerald Thrush
Dr. Kathleen Satterfield
Lara McMahon
Objectives
Describe difficulties in applying/utilizing common
definitions for learning outcomes
Discuss findings from COMP/COMP-Northwest/CPM
study in faculty use patterns of learning outcomes in
assessment practices
Discuss implications and potential future considerations
from preliminary study results
Competency-Based Medical
Education
Flexner report revolutionized medical education
Biomedical model
Evolution to competency-based medical education
Institutional Learning Outcomes
Program Learning Outcomes
Course Learning Outcomes
NBME
NBOME
Learning Outcomes
WesternU Institutional Learning Outcomes (ILO)
Critical Thinking
Breadth and Depth of Knowledge in the Discipline
Interpersonal Communication Skills
Collaboration Skills
Ethical and Moral Decision Making Skills
Life Long Learning
Evidence Based Practice
Humanistic Practice
Learning Outcomes
Program Learning Outcomes
Osteopathic Philosophy and Osteopathic
Manipulative Medicine
Medical Knowledge
Patient Care
Interpersonal and Communication Skills
Professionalism
Practice-Based Learning and Improvement
Systems-Based Practice
Learning Outcomes
Board Learning Outcomes/NBME +OPP
OPP
Behavioral Science
Biochemistry
Biostats and Epidemiology
Cardiovascular
Gastrointestinal
General Principles of H&D
Genetics
Gross Anatomy
Hematopoietic and
Lymphoreticular
Histology and Cell Bio
Micro/Immuno
Musculoskeletal and CT
Nervous System/Special Senses
Nutrition
Pathology
Pharmacology
Physiology
Renal/Urinary
Repro/Endocrine
Respiratory
Curriculum Mapping
Based upon faculty tagging, elegant
curriculum mapping system demonstrates
all ILO’s, PLO’s and BLO’s are covered
both in:
Curriculum delivery
Assessment
Today we are going to focus on
assessment/tagging of exam items
ExamSoft® Exam Summary Analysis
Standard Understanding of Learning
Outcome TermsQuestions were asked:
Are faculty members consistently applying the same
definition(s) of the various competencies?
Are faculty then consistently tagging the appropriate
competency when delivering curriculum and when
assessing curriculum?
Team embarked to do a preliminary study to determine
faculty patterns in tagging of competencies for
assessment
Research hypothesis was that faculty members do not
have a consistent and standardized definition of
competencies
Study Design
Fall of 2013, two healthcare colleges (COMP and CPM)
that share certain curriculum elements began process
where faculty members tagged all exam questions with
appropriate learning outcomes using ExamSoft® CBT
software
Question arose regarding faculty practices in using
standard definitions of competencies in tagging of
questions
COMP and CPM undertook a preliminary study to begin
investigating the application of competency
terminology in assessment
IRB approved protocol
Study Design
Exam questions from the spectrum of pre-clerkship courses were reviewed and 47 questions were selected
All questions had been tagged by original author with ILO’s, PLO’s and BLO’s
Questions embedded in a survey sent to all faculty and faculty were asked to tag each question with any LOs they believed pertinent to the question
32 faculty completed the study and results were compiled in various groups
Clinician vs Basic Medical Science Faculty
College agreement with original author
Consistency amongst the faculty
Results:
Representative Tagging Data for Selected
Questions
Good agreement Poor agreement
Results:
Clinicians vs Basic Scientists
ILO PLO BLO
% M
atc
he
d O
utc
om
es
Results:
COMP vs Podiatric Medicine Faculty
ILO PLO BLO
% M
atc
he
d O
utc
om
es
Results:
Tagging Data for Questions with Multiple LOs
Author: Multiple LO’s Author: Single LO
Results:
Tagging Data for Questions with Multiple LOs
0
20
40
60
80
100
ILO PLO BLO
any LO 2 LO's
ILO PLO BLO
% M
atc
he
d O
utc
om
es
n = 5 n = 4 n = 10
A total of 13 (of 47) questions had multiple LO’s tagged by the author
Q3: Which one of the following molecules is
NOT a direct product of the TCA cycle?
A. ATP
B. CO2
C. FADH2
D. GTP
ILO PLO BLO
Author 2 2 3
Faculty
% Match97% 97% 97%
Faculty
% with Extra LO’s19% 6% 6%
Good agreement in all three LO’s between item author and faculty.
Suggests appropriately tagged item
Q5: Absorption of drug X via the oral route is known to
be saturated (zero order) at doses higher than 100 mg.
If a patient swallowed 300 mg of drug X and assuming
no expenditure of energy is involved in transporting the
drug molecules across the membrane, what would be
the most likely mode of transport for drug X across the
gut?
A. Facilitated transport
B. Active transport
C. Ion-pair transport
D. Simple diffusion
E. Filtration
ILO PLO BLO
Author 2 2 17
Faculty
% Match100% 100% 91%
Faculty
% with Extra LO’s*
53%
(1)9%
62%
(18)
Two of the LO’s had a large number of extra tags Should item have been double tagged?
ILO PLO BLO
Author 1 2 18
Faculty
% Match16% 97% 44%
Faculty
% with Extra LO’s*100%
(2)3%
87%
(14)ILO likely mis-tagged:
100% of the faculty selected ILO 2 and only 16% selected the same ILO as the author
Multiple BLOs likely appropriate:
100% of the faculty selected ILO 2 and only 16% selected ILO 1
Q10: What is the principal function of the cochlea?A. To break complex sound waves into component tones and to transduce these
tones into action potentials in primary sensory fibers.
B. To provide impedance matching by amplifying incoming sound waves as they
move from the oval window to the round window.
C. To filter sound waves so that frequencies below 1000 Hertz are represented at the
base and above 1000 Hertz are represented at the apex.
D. To localize the sound in space by
comparing arrival time in the two ears.
E. To convert efferent signals from the
superior olive into contractions in the
outer hair cells thus amplifying
incoming sound waves.
Q26: A 67-year-old Caucasian woman presents to your office with
2 month history of dry non-productive cough and 10-lbs of
unintentional weight loss. She denies fevers, chills, dyspnea, sore
throat or hemoptysis. Patient has 80- pack year history of smoking.
Family hx is unremarkable. Patients vital signs and physical exams
are unremarkable. A CXR is ordered and shows a peripherally
located solitary pulmonary nodule. Biospy of this solitary mucin
secreting nodule is as shown in image. Which of the following is
MOST LIKELY diagnosis?
A. Adenocarcinoma of the lung
B. Small cell carcinoma of the lung
C. Pulmonary Hamartoma
D. Bronchial Carcinoid
E. Mesothelioma
ILO PLO BLO
Author 2 2 21
Faculty% Match
100% 100% 81%
Faculty
% with Extra LO’s*
78%
(1)47%
(3)81%
(16)
Unanswered Questions/Confounders
Faculty training:
When is it appropriate to tag multiple LO’s
Encourage it more with BLO’s – i.e., many tagged
systems (e.g., renal) should be counter tagged with
an appropriate discipline (e.g., physiology)
How many LO’s is it appropriate to tag?
“serial taggers”
How to minimize mis-tags of LO’s
Unanswered Questions/Confounders
Faculty training:
When is a question testing the ILO’s:
Critical Thinking
Life Long Learning
Evidence Based Practice
Or the PLO’s:
Patient Care
Practice-Based Learning and Improvement
Systems-Based Practice
Next Steps/Future Considerations
Faculty training
Defining the LOs
Understanding which LOs are appropriate for MCQs
Life-long learning
Patient-centered care
Do we need to reassess our tagging?
Do we need to move towards milestones/EPAs?
Are LOs still relevant for the future of healthcare
education?
???
Questions/Discussion
Dr. Susan Mackintosh
Dr. Colleen Talbot