medbiquitous annual conference 2011 may 11, 2011 baltimore, maryland
DESCRIPTION
Formative Evaluation of the Implementation of the Medical Education Metrics Standard for Continuing Education. MedBiquitous Annual Conference 2011 May 11, 2011 Baltimore, Maryland. Presenters and Disclosures. Francis Kwakwa , MA - PowerPoint PPT PresentationTRANSCRIPT
Formative Evaluation of the Implementation of the Medical Education Metrics Standard for
Continuing Education
MedBiquitous Annual Conference 2011
May 11, 2011 Baltimore, Maryland
Presenters and Disclosures
• Francis Kwakwa, MAAssistant Director, Data Management , Radiological Society of North AmericaDoes not have an interest in selling anything to CME professionals
• Andrew Rabin, BSChief Technology Officer, CECity Does have an interest in selling a program or service to CME professionals
• Sean Hayes, PsyD Vice President of AXDEV GroupDoes have an interest in selling a program or service to CME professionals
• Valerie Smothers, MA Deputy Director, MedBiquitousDoes not have an interest in selling anything to CME professionals
2
Objectives of Today’s Session
• Increase knowledge about use of Medical Education Metrics Standard (MEMS) in CME
• Share experience of using MEMS as part of Program and Activity Reporting System
• Present formative findings collected regarding the experience of those who have implemented MEMS
• Offer case study findings as to how to assess implementation of new systems
3
MedBiquitous Metrics Working Group
Mission
To develop XML standards & Web services
requirements and descriptions for the exchange
of aggregate evaluation data and other key
metrics for health professions education.
4
MEMS: Medical Education Metrics
• Technology standard for core evaluation data
• Users– Educators want best
practices, ability to compare– Funders want to measure
reach and efficacy– Accreditors want to
measure success of activity and provider
MEMS Data
• Activity Description– What’s being evaluated
• Participant Activity Evaluation– What did participants think
• Participation Metrics– How many people participated
• Learner Demographics
Program and Activity Reporting System (PARS)
7
• Manual data entry
• Comma delimited upload
• XML upload
Formative Evaluation
• Provides immediate feedback to program participants
• Fosters reflection and self-regulation
• Provides feedback about the program itself to designers and sponsors
• Oriented to improving the focus of the program
• Based upon a dynamic model
Relationship Between Formative and Summative Evaluation
Across Life of an Innovation
Rel
a tiv
e E
mp
has
is
Program Life
Summative Evaluation
Formative Evaluation
Worthen 1987
Focus Group
• MEMS implementers using XML in association with PARS
• 4 organizations represented
• Measured – Success and satisfaction– Resources for implementation– Characteristics of implementers– Decision factors
10
Results
• Implementers felt satisfied, but limited (not many customers implementing)
• Limited effort and resources required; change management important
• Implementers all fluent in XML • ACCME adoption key; implementers
seeking to better support customers, show competitive advantage, make links between different types of data
Case Study
• Implemented MEMS for PARS upload
• Experience implementing MedBiquitous standards
• Integrated new data into user interface and workflow
• Provided user training
Lessons Learned
16
• Standards implementation can bring about efficiencies when integrated into workflow and properly supported
• Validation tools needed
• Web services needed to realize full benefits, one click data transfer
Contact Us
17
• Valerie will forward questions to the working group