Nadim Lalani MD
Vanilla Sky
Tom Cruise 2001 Existential “Mind warp” Deals with cryogenics and the
possibility of living a virtual life after death
Blending of the technologic and biologic worlds “plugged in’
?Virtual [technologic] world to supplement [real world] EM medical education
Objectives Definition Background Literature Review
Med Ed Resident Ed Professional Development
What it might look like Future Directions
What is “Online learning”? Online Learning [e-learning] = is digital Evolved from CD/computer labs Everyone does it! Performance Support [ for software e.g] Web page [e.g. Uptodate] Self-paced Web-based [CME] Leader-led [ Distance Learning] Blended [or hybrid] learning
combines conventional with digital learning
Advantages of e-learning Rich environment:
Media-filled [esp in EM] transfer of difficult concepts Links to sources
Convenient, efficient & flexible Asynchronous Can be accessed from a distance
Adult learning principles: Self-paced and self-directed Flexible/ home access efficiency
Background: Why bother?
U of C Medical School current enrollment = 130 students goal 150 Mandatory EM rotation / increasing
competencies Resident numbers also increasing Result more learners in the ED Relative shortage of preceptors, relevant
clinical encounters and curricular time Will be worse when our program expands
usurp learning opportunities
Why bother
Deficiency in learning encounters = a performance gap future physicians do not have the
adequate exposure to emergent problems.
imperative we equip students, clerks & residents with the skills and training.
why bother Increased digitalisation is a key
strategic goal of the U of C Learners are unique with
mulitdimensional learning styles. Adult learning principles Attract the best candidates Provide a method of training
students and clerks at two different campuses
Provide consistency in teaching
Why bother? Provide efficient means of knowledge
transfer to residents Increasing number of competencies
[CANMEDs] Better use of academic half-day.
Provide more effective professional development: Asynchronous don’t have to be there Interactive discussion board Consistent, evidence-based standard of
practice Increased self-efficacy
E-learning not a panacea there is more to training and
education than e-learning Certain skills do not lend themselves
to e-learning The key will be selecting the best
delivery method. Cannot simply upload old material. Learner – focused no one solution [blended may work for
residents].
Process: Can it be done?
“Fail to prepare … prepare to fail”
Need to address several key questions:1. Purpose? Added value?2. What support and expertise exist?3. Ongoing upkeep?4. Stakeholders?5. Team?6. Instructional design/Pedagogy
Literature Review
Same Search terms in PUBMED Bibliographies of relevant articles scanned Missing 1 Med Ed & 1 CME [both foreign
language]
Literature general Comments More Literature exists for Med Ed
Pre 1990 Limited by lack of internal validity
Few Randomised Controlled Studies Emerging Lit wrt Resident
experience Despite lots of experience with
online CME Little Literature … mostly Descriptive
Literature General Comments Terminology inconsistent Interventions vary. ? “prototypes” of today’s
technology? Don’t address some of the uniqueness
[internet] Comparing apples to oranges
Can E-learning be used to replace/augment Traditional Methods?
E Learning & Med Ed
Study Dartmouth Med School 328 Students randomised to:
Interactive Case-based study guide on Computer*
Case-Based Printed study guide Anemia and Cardiology Courses
Outcomes: Performance on higher order MCQ tests,
exams Self-reported Efficiency
* media-rich images, blood smears and EKG’s
Results
No Difference in Test Scores No difference on board exams The vehicle is an acceptable means of
delivery
Time Spent
Cardiology Computer
4.4 ± 0.6h
Cardiology Workbook
9.4 ± 0.5h P = 0.0001
Anemia Computer
5.5 ± 0.5h
Anemia Workbook
8.0 ± 0.5h P = 0.001
Limitations Self reporting of efficiency! Confounders [other text books/practice exams/time-spent
cramming] Doesn’t really tell us about dynamic problem-solving/
clinical judgment
Board Exams
Anemia CV
Class + CAL
81.5 ± 6.5 80.0 ± 6.7 NS
CAL 82.7 ± 6.0 82.4 ± 5.6 NS
179 Paeds Clerks in [2 sites Chicago] Randomised to Lecture via:
Multimedia Text Book* Lecture Printed Text No intervention
Paeds airway diseases Outomes:
MCQ Test Score [at end of rotation & at 1 y later]
* Only different in audio/video
Results
Score / 26
MMTB Printed Book
Lecture Control
N = 89 21 19 23 26
Initial Test 16.6 [2.5] 16.5 [3.6] 15.9 [2.5] 14.2 [2.9]
Final Test 15.3 [2.8] 15.3 [2.9] 14.6 [2.3] 14.5 [3.4]
Limitations
51% Attrition rate! Clerks at one site had mail-in repeat
exam Confounders
One hour instruction embedded in a 6 week clerkship
75 Med students [Brisbane Australia] Randomised after pretest to:
Computer Tutorial Focus on knowledge Computer tutorial create ideal patient
for dx + feedback [every 10 cases] Computer Tutorial both knowledge &
decision + three different types of feedback [after every 10 cases]
Looking at diagnosing abdo pain [ 30 cases]
Outcomes and Results Outcomes:
Attained knowledge Diagnostic accuracy Decision-making confidence [self reported]
Results: Students focusing on facts did not improve
decision-making All feedback groups improved diagnostic
accuracy Type of Feedback not important.
Self reported confidence improved
Limitations
Small study Very convoluted method ?
reliability
Can E-learning be used to Teach Procedural Skills?
E- Learning & Med Ed
82 Medical Students [Toronto & Augusta] Randomised to:
Computer Tutorial + knot board Lecture with Feedback + knot board Two-Handed Knot tying
Tested right after [filmed] Outcomes:
Proportion Square/ Time to tie Knot Performance score [blinded surgeons] Student Questionnaire
Results
NO difference in “Cognitive” portion Lower performance score in CAL group 89% Students would have preferred
Lecture Session Lack of feedback cited as negative
Limitations
Apples and Oranges! ? Not controlled for hands-on feedback Maybe CAL better if it described pitfalls /
showed video of good and bad knots? Reliability of performance score [not
included]
42 Clerks U of T Randomised to:
Computer Tutorial [rich text, animations, interactive –Q&A, no audio/video]
small group seminar [also interactive] Epistaxis Management
Outcomes: Short Answer written Test Practical Test [16 point performance
scale]
Results:
Poor Prior knowledge No difference in written scores Slightly better practical skills with
CBL
Limitations:
Small numbers Examiners NOT blinded ?reliability of performance score [not
included] Practical was on dummy
? transferability
69 Medical Students [Wisconsin] After pre-test Randomised to:
Didactic Session/Q&A¥
Video-Tape Session* Computer Tutorial*
Post Intervention: MCQ test, Practical Skills test [2 blinded
obs] Repeat testing at one month¥ no feedback . * Instructor present
Outcomes: MCQ Test Scores Timed observation of skills Critical Skills evaluated via checklist Performance Quotient calculated
Results
Higher initial mean % correct / % complete in CBT group [p<0.01]
Significantly better PQ in CBT group at 1 month [p < 0.01]
Results
Didactic group better on immediate MCQ [63% vs 49% for video/CBT p < 0.01]
Difference in MCQ still there at 1 month
Results
Bigger change in PQ with CBT at 1 month [ P< 0.01]
Limitations
Small study Video vs CBT essentially the same
intervention ? Why CBT would do better than Video
? Reliability of checklist and PQ?
Can E-Learning be used for Emergency Medicine Rotations?
What About the ED Experience?
100 Clerks [Mt Sinai] Randomised by blocks
EM rotation with access to EM Website Modules [ACLS, Tox], Xrays, Pix, Paeds
Cases EM Rotation without access Outomes:
Exam Scores Student Satisfaction
Results
ONLY 28% intervention group used it. 72% Cited lack of time
NON sig difference in exam score [72.8 vs 68.2 p = 0.058]
Non sig difference in satisfaction [ 77.5% vs 66% p = 0.23]
Baseline only 26% > 1h /wk online [cf 96% next class]
Baseline 65% wanted online component
Limitations
< 30% in intervention group didn’t reach power. WAS ITT so results would have been
+ve with more participants Problems with randomising by block
rotations given away on lottery [ CARMs]
Unmotivated learners? ?generalisable to clerks in 2008
23 Clerks [U of T Sick Kids] Volunteered for study, Randomised
to Access to Web-based Modules No Access to Web-based modules
ED Procedures [lac, LP, splint] Outcome:
Performance on MCQ Test
Results
Statistically higher competence [ p = 0.0001]
Cohen’s d Effect size r = 0.79
Limitations
Small sample size Volunteers [EM /techno gung-ho] Methodology:
Unclear when test was administered in relation to rotation
?randomised to learning vs no learning? Validity of MCQ vs Observed skills
Transfer of knowledge? MCQ vs Observed skills
350 Urology Clerks [4 med schools US] Randomised [two-group crossover] to:
Web-based Tutorials [BPH,ED,PC,PSA] No Access to Web Tutorials Served as the controls for the modules they
didn’t have access to online Outcomes:
Performance on test [pre/post] [Cr = .79] Durability of learning/ Learning efficiency in
SubG
Results
Results:
Results
Learning Efficiency 0.10 vs 0.03 [p<0.001] Test scores still improved without WBT [12%
BPH, 6% ED, 24% PC, 20% PSA] Web-based alone had Cohen’s r = 24.9!
Limitations:
Volunteers with unequal participation b/w sites [93% HMS vs 52% BUSM]
High Drop out rate 210 /350 completed
? Generalisability of repeated measures
? Generalisability to EM
Summary: E-Learning & Med Ed Content can be delivered Appears to be transfer of learning of
Cognitive skills … Perhaps also Psychomotor skills
Still need for experienced clinician feedback
Increased student satisfaction Attempt to make instructional design
identical [validity] undermines uniqueness of EL Always will be apples and oranges
Summary E-Learning & EM: Controlled Interventional Literature
conflicts Learners:
Identify time constraints as a barrier in ER Want more visual aids [Ekg, XR, photo & WEB]
Effective strategies for ER teaching include: Using Resources Going beyond patient care Improving the learning environment
Adult principles/ learners may be driving force despite dearth of evidence
How Can we Augment Med Ed?
Currently Clerks Use LMS “Osler”Download PDFs of Content/CasesCore Content Reviewed with Preceptor
We can do more …
Procedures/ Anatomy Pictures Xray / Ekg interactive ppt
E Learning for Residents
109 IM Residents [US] during “clinics” year
Randomised [Crossover] to: Access to Web-based Practice Guidelines
On WebCT [rich format with links to sources] Printed Practice Guidelines
Outcomes: Format Preference Performance on Final test
Results
Strong preference for Web Based material Men > Women
NO difference on test between groups Non-Significant reduction in time spent ++ problems with WEbsite
Limitations
Volunteer Significant dropout 145 eligible
109 enrolled 51 completed all aspects
? generalisability
162 FM & IM Residents Randomised to:
Web-based tutorial [hyperlinks, graphics] Printed material ACC/AHA Guidelines Management post AMI
Outomes: Test Score Efficiency Satisfaction
Results:
Limitations
Only enrolled 30% [550 eligible] Participants not blinded to hypothesis ? Generalisable
Voluntary + monetary honorarium
22 EM residents & Staff [U of T] Noninferiority for U/S vascular access
course Randomised [after pretest] to:
Web Tutorial + practice* 1h Lecture + practice*
Outcomes: [2 weeks later ..] MCQ test 4 OSCE stations [blinded obs used checklist]*Non-precepted
Results
Same pass rate All web users logged on from home
Limitations
Small study [did have power though] No controlling of practice session
Stronger coaching the weaker? No interrater reliability for OSCE
Summary: E-Learning + Residents Learning can be delivered this way
Weak evidence of non-inferiority for learning guidelines in other specialties
Modest evidence for use prepping for a hands-on session relating to EM.
Weak evidence for Psychomotor skill acquisition
Consistent satisfaction with the method
How to Augment Resident Ed
Website / Subscriptions / RemergS HPS / AHD
Better Use of AHD …
Longitudinal course on LMS? Already Licensed by U of C
Used by Anesthesia
Interaction is key
Learner-Sources, Learner-learner, Learner-Clinician
E Learning and CME
52 Physicians [US] randomised to: Web-Based CME Normal Instruction Office Dermatologic procedures
Outcomes: Performance on test Satisfaction Self-reported performance change
Results:
More improvement in test score with WBL [13.2 vs 9.6]
General satisfaction with WBL Increased self-reported competency Fair amount of interaction in
asynchronous forums
Limitations
Small study Self-reporting ? Reliability of MCQ test
? Direct observation of procedures
99 Physicians [US] Randomised to:
Web based CME for domestic violence Regular Instruction [only did two surveys]
Outomes: Self-Efficacy
Externally validated Survey instrument [CR 0.7]
Change in screening
Results:
Increase in self efficacy + 18% intervention vs - 0.6% control [p =0.01]
Positive other endpoints in survey NO difference in DV screening
Limitations
Honoraria to WEB participants Methods
?No learning to controls? generalisability
103 physicians [US] randomised to: Web-based CME F-2-F small group CME session Cholesterol management
Outcomes: Performance on tests Chart audit [20 docs from each group 25
charts] Satisfaction survey
Results:
More improvement with Online NO difference in Satifaction between
groups Online spent 3.8 h / 3 sessions Chart review statistically signif [small
difference] in guideline adherence [but no difference in cholesterol screening]
Limitations
170 eligible only 103 included ? Different groups? [online better at
baseline] Live CME event happened before Online
[?cross contamination] Hard to reconcile change in
behaviour when no change in screening
? generalisability
87 Physicians self-selected into: Online CME [3 sessions 8h] F-2-F CME [one 8 h session] Treatment of opioid dependence
Outcomes: Post intervention test Satisfaction
Results
NO difference in improvement No difference in satisfaction scores
Limitations
30% attrition [87 entered data for 62]
NON-randomised ?reliability of MCQ test ?Generalisability
Summary: E-Learning and CME Certainly can be used to disseminate
info In keeping with adult learning principles
Transfer of cognitive component ? Psychomotor NO change in practice patterns Subjective increased self-efficacy
Enhancing CME
Combine LMS with narrated slides Provide facilitation for your topic
Narrated Options for Quizzes
Future Directions
International Emergency Medicine Disseminate info before experts hit the
ground Collaboration with other programs
Online Electives? EBM Toxicology Radiology
Thoughts?
Appendix: How you get there
Discerning the Context
Learning Context Power Dynamics
PGME Regionalisation
People dynamics Resistance change Relatively few champions of Med Ed
Supporting Cast
Stakeholders: University UME /PGME RMES
Planning Committee: Content experts Educational specialists/Instructional
design IT [graphic design/ media] End-users
Identifying Program Ideas Core content
Literature Current material
Needs analysis Narrow focus Not redundant
Sorting and Prioritising
Prioritise content Important? Feasible?
Developing Objectives
Needs to be competency based Reflect goals of UME/ Colleges
Instructional Plan
How are you going to deliver this? Need interaction:
Content, Instructors & peers Dedicated ED computer WEBSITE LMS
Instructional Plan
Instructors: few innovators Clinicians Well suited “many hats” Need to “train the trainers”
Learners: Self-directed & Computer savvy Clinical/ procedural Skills Blended approach
Instructional Plan Variety & content:
Anatomy and radiology PowerPoint, Flash files, video streaming. e-case modules LMS online discussion
Logisitics: LMS powerful tool
Free ones available STARS uses Moodle U of C has licenses & tech Support Regionalisation