direct observation of clinical skills during patient care new insights – reynolds meeting 2012...
TRANSCRIPT
Direct Observation of Clinical Skills During Patient Care
NEW INSIGHTS – REYNOLDS MEETING 2012
Direct Observation Team:J. Kogan, L. Conforti, W. Iobst, E. Holmboe
2
Medical Education Trend 2000- present
Competency BasedEducation
Variable length, defined outcome
Fixed length, variable outcome Structure/Process•Knowledge acquisition•Single subjective measure•Norm referenced evaluation•Evaluation setting removed•Emphasis on summative
Competency Based•Knowledge application•Multiple objective measures•Criterion referenced•Evaluation setting: DO•Emphasis on formative
Caraccio et al 2002
3
In-Training Performance AssessmentAssessment in authentic situations
Learners’ ability to combine knowledge,skills, judgments, attitudes in dealing with realistic problems of professional practice
Assessment in day to day practice Enables assessment of a range of essential
competencies, some of which cannot be validly assessed otherwise
Govaerts MJB et al. Adv Health Sci Edu. 2007;12:239-60
4
Observation to Test AssumptionsDirect observation can test assumptions4 observations needed to detect outliers
Shared responsibility
Detect Outliers Feedback/development
TIME/TASKEarly Late
5
Observation and Safe Patient Care
Importance of appropriate supervisionEntrustment
Trainee performance* X Appropriate level of supervision**
Must = Safe, effective patient-centered care
* a function of level of competence in context
**a function of attending competence in context
6
Types of SupervisionRoutine oversight
Clinical oversight planned in advance (i.e. what we normally do)
Responsive oversight: Clinical activities that occur in response to trainee or
patient specific issues (i.e. you do more than usual)Direct patient care:
When supervisor moves beyond oversight to actively providing care for the patient
Backstage oversight: Clinical oversight which the trainee is not aware of
Kennedy TJT et al. JGIM 2007.22:1080-85.
7
Your Supervision
How do you usually supervise?When do you supervise more closely?How do you change your supervision to ensure
patients get safe, effective, patient-centered care?What did you learn observing that will change how
you supervise going forward?
REMEMBER: SUPERVISION ALSO FOR FEEDBACK
8
Entrustment
“A practitioner has demonstrated the necessary knowledge, skills, and attitudes to be trusted to independently perform this activity.”
Ten Cate O, Scheele F. Acad Med 2007;82:542-7
9
Problems with Performance Assessment
Poor accuracy
Focus on different aspects of clinical performance
Differing expectations about levels of acceptable clinical performance
Rating errors Halo effect/ “Horn” effectLeniency/stringency effect
10
Factors That May Impact Ratings
Minimal impact of demographicsAge, gender, clinical and teaching
experienceFaculty’s own clinical skills may matter
Faculty with higher history and patient satisfaction performance scores provide more stringent ratings.
Kogan JR. et al. Acad Med. 2010;85(10 Suppl):S25-8
11
Factors Influencing Faculty RatingsDifferent frameworks for judgments/ratings
Self-as-reference (predominant) Trainee levelAbsolute standardPracticing physicians
12
Faculty OSCE Clinical Skills
Competency Mean (SD) Range Generaliz-ability
History Taking 65.5% (9.6%) 34% - 79% 0.80
Physical Exam 78.9% (13.6%)
36% - 100% 0.52
Counseling 77.1% (7.8%) 60% - 93% 0.33
Patient Satisfaction1
5.62 (0.48) 4.43 – 6.63 0.60
1On 7-point scale
Kogan JR. et al. Acad Med. 2010;85(10 Suppl):S25-8
N=44
13
Other Factors Influencing Ratings
Factors external to resident performanceEncounter complexity Resident characteristicsInstitutional culture
Emotional impact of constructive feedback
Role of inference
14
Definition of Inference
a. The act or process of deriving logical conclusions from premises known or assumed to be true.b. The act of reasoning from factual knowledge or evidence.
15
Affix meaning
Concrete data
(resident actions)
ConclusionsAssumptions
1.
2.
3.
16
The Problem with InferenceInferences are not recognized
Inferences are rarely validated for accuracy
Inferences can be wrong
17
Types of Inference about ResidentsSkills
Knowledge Competence Work-ethic
Prior experiences Familiarity with scenario
Feelings Comfort Confidence Intentions Ownership
PersonalityCulture
18
High Level Inference
19
Direct Observation: A Conceptual Model
Kogan JR, et al. Med Educ. 2011
20
International Comparative Work Yeates (UK)
Differential salience Criterion uncertainty Information integration
Govaerts (Netherlands) Use of task-specific and person schemas
Substantial inference in person schema Rater idiosyncrasy
Gingerich (Canada) Impact of social models: clusters; person; labels
21
Achieving Accurate, Reliable Ratings
Form is not the magic bullet
Assessment requires faculty trainingSimilar frameworksAgreed upon levels of competenceMove to criterion referenced assessment
22
Questions