Department of Graduate Medical Education (GME)
Now That We Have an Institutional Report Card---How Do We Use It?
Conference Session: SES010
2012 ACGME Annual Education Conference
Ann Dohn, MA, DIO,
and
Nancy Piro, PhD,
Program Manager/Education Specialist
Department of Graduate Medical Education
Stanford Hospital & Clinics
Department of Graduate Medical Education (GME)
Disclosure
• No conflicts of interest to report
Department of Graduate Medical Education (GME)
Session Objectives
At the end of this session, participants will be able to:
1) Effectively use objective data in Balanced Report Cards to make Institutional and Programmatic decisions.
2) Understand strategies for gaining Institutional Administration (C-Suite) support for DIO/GMEC Decisions based on balanced report cards
3) Predict future ACGME review outcomes based on empirical data from balanced report cards
Department of Graduate Medical Education (GME)
Setting the Stage
• How does GME administration function in 2012?
• What are the barriers?
• What are the challenges?
• How can we be heard by the C-Suite?
• What language should we speak?
• How do we optimize success?
Department of Graduate Medical Education (GME)
How does GME Administration Function?
• How do we know?
• What do we measure?
• What management strategies do we use?
Department of Graduate Medical Education (GME)
Historically…..
• GME has not used objective data in making many decisions
• Instead, decisions were often made based on:– Politics– Financial considerations– Special deals– Hunches– Ability to scream loudly and frequently– Fear
Department of Graduate Medical Education (GME)
But what is going on in the environment?
• Evidenced based– Balanced score cards– Increased accountability for resources– Transparency
Department of Graduate Medical Education (GME)
Fitting GME within the overall business culture
• Remember…your “C-Suite” colleagues often have that magical MBA …..
• They think differently
• They use a different language
Department of Graduate Medical Education (GME)
Fitting GME within the overall business culture
• We need to understand how the C-Suite thinks
• AND learn to think the same way…
AND speak their language…
To gain their support!!!
Department of Graduate Medical Education (GME)
What Considerations does the C-Suite have with Respect to Data?
• Data must have:– An Organizational focus– Regular Means of Data Collection – Include External Assessments as well as Internal
Assessments
• Data used must be– Standardized (comparable) / lean– Capable of transformation – Aligned with mission statement
Department of Graduate Medical Education (GME)
DIOs need to be able to make decisions based on comparative data
• Data must:– Be Organizationally Focused– Have Regular Means of Data Collection – Include External Assessments as well as Internal
Assessments– Standardized– Capable of transformation – Aligned with mission statement
Department of Graduate Medical Education (GME)
Conclusion
• CEOs and DIOs have the same requirements with respect to their data needs
Department of Graduate Medical Education (GME)
But what models for data do we have?
• Few data models exist for GME
• The concept of Institutional Accountability is relatively new. – Until the ACGME Outcome Project, there was no
centralized curriculum oversight in GME, unlike medical schools / UME.
Department of Graduate Medical Education (GME)
The “Report Card” Vision
• In 2005, Stanford hired its first PhD in GME– The vision was to
develop tools to construct evidence based decision-making for Graduate Medical Education consistent with our mission
– “We needed a Report Card”…
Department of Graduate Medical Education (GME)
SIGH….
• It wasn’t as easy as first thought!
Department of Graduate Medical Education (GME)
Our First Attempt …
Department of Graduate Medical Education (GME)
Background on InstitutionalReport Cards
• Government and Industry Models– Multiple models exist and can be used as per specific
purpose:• GRPA (Government Performance and Results Act)• Organizational Report Cards• Balanced Scorecard• Benchmarking• Program Evaluations• Social Indicators• No Child Left Behind• C BEST• College Performance Testing
Department of Graduate Medical Education (GME)
Which Model Should We Use?
• We needed a model that:– was organizationally focused and managed – had a track record of effective use– fit our existing structure with multiple programs and
organizations– was flexible enough to be adapted for use on an
annual basis – not an accreditation cycle-Regular Data Collection
– was “Easily Digestible” – had internal and external measurement dimensions
Department of Graduate Medical Education (GME)
Our Choice
Balanced Scorecard Framework in an Organizational and/or
Report CardTool
Best of Both Worlds
Department of Graduate Medical Education (GME)
The Balanced Scorecard Approach
• The Balanced Scorecard is a performance measurement and performance management system developed by Robert Kaplan and David Norton (1992, 1996) – adopted by a wide range of leading edge
organizations, both public and private.
(“The Balanced Scorecard--Measures That Drive Performance,” Harvard Business Review, Jan-Feb 1992; and “The Balanced Scorecard-Translating Strategy into Action,” Harvard Business School Press, 1996)
Department of Graduate Medical Education (GME)
Stanford’s Report Card – Three Programs
Department of Graduate Medical Education (GME)
Key to the Report Card
Department of Graduate Medical Education (GME)
Balanced Scorecard Strategic Perspectives
Institutional / Financial Growth
MissionVision
Strategy
Learning
Program Processes
Resident
Do we continue to improve (outcomes)?
Are our programs excelling?
How do our residents see us?
Are we putting our resources in the right
places?
Department of Graduate Medical Education (GME)
Selection of Balanced Report Card Measures
PRE PERI POSTINTERNAL EXTERNAL
RESIDENT PERCEPTIONS“Voice of the Resident”
PROGRAM PROCESSES
• Faculty Eval of program• Duty Hr Violations• # Internal Citations
LEARNING OUTCOMES
FINANCIAL / GROWTH
• Resources
• GME Internal HS Survey Recommendation of Program• GME Internal HS Survey Overall Satisfaction • Resident Evaluation of Program
• Match Depth• ACGME Survey Responses
• Teaching• Supervision• Scholarship• Non Hostile/Intimidating Environment• Address Concerns Confidentially
• Training Exams • Board Pass Rates• Career Placement
• Grants
• ACGME Cycle Length• # ACGME Citations
Department of Graduate Medical Education (GME)
Balanced ScoreCards “Report Cards”
Had– Organizational focus
– Regular Means of Data Collection
– Include External Assessments as well as Internal Assessments
Were– Standardized
– Capable of transformation
– Aligned with the Mission of Stanford Graduate Medical Education (GME)
Department of Graduate Medical Education (GME)
Stanford Hospital & Clinics Use of the “Balanced Report Card”
• The Stanford Report Card is built on the Balanced Scorecard conceptual framework for translating an organization’s vision into a set of performance indicators distributed among four perspectives adapted for GME: 1. Resident Perception Measurements2. Program Processes3. Learning Outcomes4. Financial/Growth
Department of Graduate Medical Education (GME)
Stanford Background
Stanford University Medical Center currently sponsors 85 ACGME-accredited training programs with over 1100 enrolled residents and fellows.
Department of Graduate Medical Education (GME)
Stanford University Medical Center Mission
• Dedication to pursuing the highest quality of patient care and graduate medical education, recognizing that one of its major responsibilities is the provision of organized educational programs. – Support of quality graduate medical education programs and
excellence in residency training and research. – Guidance and supervision of the resident while facilitating the
resident’s professional and personal development and ensuring safe and appropriate care for patients.
– Provision of adequate funding of graduate medical education to ensure support of its faculty, residents, ancillary staff, facilities, and educational resources to achieve this important mission.
– Ensuring that all of its graduate medical education programs meet or exceed the Institutional and Program Requirements promulgated by the Accreditation Council for Graduate Medical Education.
Department of Graduate Medical Education (GME)
Translating the Stanford Hospital & Clinics’ GME Mission to the Balanced Report Card
1. Resident Perception Measurements“Guidance and supervision of the resident while
facilitating the resident’s professional and personal development and ensuring safe and appropriate care for patients.”
2. Program Processes“Ensuring that all of its graduate medical education
programs meet or exceed the Institutional and Program Requirements promulgated by the Accreditation Council for Graduate Medical Education.”
Department of Graduate Medical Education (GME)
Stanford Hospital & Clinics Report Card
3. Learning Outcomes
“Support of quality graduate medical education programs and excellence in residency training and research.”
4. Financial/Growth
“Provision of adequate funding of graduate medical education to ensure support of its faculty, residents, ancillary staff, facilities, and educational resources to achieve this important mission.”
Department of Graduate Medical Education (GME)
Stanford Hospital & Clinics Balanced Report Card
• Indicators are designed to measure SHC’s progress toward achieving its vision; other indicators are designed to measure the long term drivers of success.
• Through the balanced report card SHC :– Monitors its current performance (finances, resident
satisfaction, learning outcomes and program process results)
– Monitors its efforts to improve processes, educate residents– Enhances its ability to grow, learn and improve the quality
of its fellowship and residency educational programs.
Department of Graduate Medical Education (GME)
Why GME Thinks We Need This
• ACGME and Institutions are increasingly holding DIOs and GME Committees accountable for their utilization of institutional resources.
• Actions / decisions must be based on documented real-time analyses of needs.
Department of Graduate Medical Education (GME)
Selection of Balanced Report Card Measures
PRE PERI POSTINTERNAL EXTERNAL
RESIDENT PERCEPTIONS“Voice of the Resident”
PROGRAM PROCESSES
• Faculty Eval of program• Duty Hr Violations• # Internal Citations
LEARNING OUTCOMES
FINANCIAL / GROWTH
• Resources
• GME Internal HS Survey Recommendation of Program• GME Internal HS Survey Overall Satisfaction • Resident Evaluation of Program
• Match Depth• ACGME Survey Responses
• Teaching• Supervision• Scholarship• Non Hostile/Intimidating Environment• Address Concerns Confidentially
• Training Exams • Board Pass Rates• Career Placement
• Grants
• ACGME Cycle Length• # ACGME Citations
Department of Graduate Medical Education (GME)
Findings
• After several years of data collection….– Some data is more valuable – Some data drives change
Department of Graduate Medical Education (GME)
VOICE OF THE RESIDENT
Internal Measures
• Resident Overall Evaluation of the Program
• Duty Hour Violations
• GME/HS Survey - Overall Satisfaction ?
• GME/HS Survey - Recommend Program ?
External Measures
• Match Depth• % Top Medical Schools• ACGME Survey - %
Compliant Responses•Teaching
•Supervision
•Scholarship
•Non Hostile/Intimidating Environment
•Address Concerns Confidentially
Department of Graduate Medical Education (GME)
PROGRAM PROCESSES
Internal
• Faculty Overall Evaluation of the Program
• # Internal Citations
External
• ACGME Cycle Length
• # ACGME Citations
Department of Graduate Medical Education (GME)
Case Study - Stanford
• How the DIO uses the Balanced Report Card:– A Tale of Three Programs…– Starting from the bottom….– Assuming that being at the “bottom” may mean
“needs more help”
Department of Graduate Medical Education (GME)
How Do We Use this Data?
• Look at Indicators that are Resident Driven – “Voice of the Resident”– Is there a discrepancy between the voice of the
resident and the other indicators?• Would the majority of the residents not choose the
program again yet the program receives a five year accreditation cycle with one or no citations?
Department of Graduate Medical Education (GME)
How Do We Use this Data?
• How Do the Programs Compare Against Each Other?
• How do they compare against their ACGME Cycles?
Department of Graduate Medical Education (GME)
Stanford’s Report Card – Three Programs
Department of Graduate Medical Education (GME)
What does this tell the DIO? (Program A)
• Setting the stage:– 2008-ACGME gave a 5 year cycle with 1
citation
– 2009-ACGME resident survey shows
“tanking program”
– 2009-GME House Staff survey shows
“program in trouble”
– Duty Hour violations are > 15 times greater than the Institutional Average
» Over a three year period
Department of Graduate Medical Education (GME)
Stanford’s Program A
Department of Graduate Medical Education (GME)
Stanford’s Program A
Department of Graduate Medical Education (GME)
What does this tell the DIO?
– Educational Milieu• Intimidation and Fear of Retaliation - Increasing
• Lack of confidence in the program to confidentially deal with problems or concerns - Increasing
• Overall resident satisfaction with the program – decreasing
• Would recommend: extremely low in 2008/09 (lowest of any SHC program)
Definitely an early warning….
Department of Graduate Medical Education (GME)
How Did We Use the Data
• Validated data with resident interviews (single and group) You know you have a problem when the residents call and ask to
meet you at Starbucks…
Department of Graduate Medical Education (GME)
Setting up an “Action Plan”
• Met with program leadership – Shared data– Discussed “their” interpretation of the data– “Brain-stormed” with program leadership
• “How can GME help you?”
Department of Graduate Medical Education (GME)
Program View on Situation
• Need more help…more residents…more MD extenders….
• Residents are the problem…”not tough” and they “whine”….
Department of Graduate Medical Education (GME)
GME Evaluation of the Situations
• Qualitative analysis of every comment on the internal house staff survey
• Trend analysis of surveys
• DIO used her training in conflict resolution
• Report developed to define the problems– Shared with C-Suite
Department of Graduate Medical Education (GME)
Action Plan
• Used data regarding duty hours with the C-Suite to:– justify hiring additional physician extenders– reconfigure rotations
• Used resident program satisfaction data with the Dean to:– leverage program change and “motivate”
program leadership
Department of Graduate Medical Education (GME)
Results - Success ?
• Duty Hours have improved
• Waiting on resident sat from 2012 survey (May)
Department of Graduate Medical Education (GME)
Program B
• Setting the stage:– Internal Review in 2008-09 found a program substantially
out of compliance» Problems were largely identified as structural ie:
formal policies/curriculum» Acting program director» Small, internationally known fellowship» Very competitive» Trainees pleased with program» Trainees interviewed 2 months after start of the program
Department of Graduate Medical Education (GME)
Program B
Department of Graduate Medical Education (GME)
Stanford’s Program B
Department of Graduate Medical Education (GME)
• Assigned a GME staff professional to work with the program director on the missing “structural elements” in the program.
• DIO monitored progress
Setting up an “Action Plan”
Department of Graduate Medical Education (GME)
Progress…
• Internal review citations corrected
• 2010 ACGME review gave 5 years and a single citation
Department of Graduate Medical Education (GME)
However….the voices of the residents
• ACGME survey begins to show concerns over training environment
• DIO starts receiving individual concerns from trainees
• Program becomes large enough to qualify for its own internal survey (smaller programs are grouped)– Internal survey shows major concern areas on
Report Card…a sea of “RED”
Department of Graduate Medical Education (GME)
Power of the Data
• Concerning data is shared with program leadership
Department of Graduate Medical Education (GME)
Report Card Speaks
• Changes are implemented– Service is reduced– Associate program director appointed– Protected time for education enforced– Department leadership engaged– Administration support in place
Department of Graduate Medical Education (GME)
Disconnect
• Report card allowed GME to implement intervention which successfully addressed the areas of concern
• BUT…..ACGME (not having access to the most recent data) called for an early site visit.
Department of Graduate Medical Education (GME)
Program “C”: Setting the Stage
• The program was already experiencing difficulties before their most recent ACGME site visit– Tough transition to a new program director
when former director abruptly left
• Problems identified on both the ACGME and internal GME house staff surveys
Department of Graduate Medical Education (GME)
Program “C”
Department of Graduate Medical Education (GME)
Stanford’s Program C
Department of Graduate Medical Education (GME)
Action Items
• GME assigned a staff member to work closely with program– Note: program director readily accepted any
and all assistance-knew he had “inherited” a “train wreck”.
Department of Graduate Medical Education (GME)
Progress?
• Program continued to spiral down……– Lack of C-Suite “buy in”
Department of Graduate Medical Education (GME)
ACGME Outcomes
• ACGME site visit– Proposed probation which resulted in:
• HUGE wake-up call for the C-Suite, residents, and four major affiliates
• Data invaluable in presenting problems
Department of Graduate Medical Education (GME)
Action Items
• Data presented to– Department chair– Dean– DIOs at major affiliates– CEOs– RESIDENTS
Department of Graduate Medical Education (GME)
Consensus
• Decision to “save the program”– C-Suite– Program leadership– GME
Department of Graduate Medical Education (GME)
Results
• Appealed the probation– 2 year cycle– Program continues to improve
Department of Graduate Medical Education (GME)
What does this tell the DIO?
Report card allows for easy monitoring of multiple factors
With longer ACGME accreditation cycles in our futures…imperative to be able to monitor programs frequently
Department of Graduate Medical Education (GME)
Lessons Learned…
• Three very different programs– But data drives change in all– In all three cases the final outcome was data
driven– In all three cases the C-Suite was needed to
resolve issues-and was “moved” to help ONLY after being presented with data
Department of Graduate Medical Education (GME)
Lessons Learned (continued)
• As good as we think our “hunches” are…– The C-Suite
• wants facts, figures, stats, etc
• You need to “prove” your case
• The C-Suite is VERY sensitive to the “norm”, average, scores of other programs, benchmarks, etc.
• Your argument needs to be evidenced-based
• Don’t forget the outcomes!!
Department of Graduate Medical Education (GME)
A Tale of Three Programs
Department of Graduate Medical Education (GME)
Questions