workplace based assessments for undergraduates: training for assessors leighton sealresponsible...
TRANSCRIPT
Workplace Based Assessments for Undergraduates:
Training for Assessors
Leighton Seal Responsible Examiner for WBA and Consultant Endocrinologist
Dr Judith Ibison: Senior Lecturer Primary Care & MBBS Personal and Professional Development Theme lead,
Dr Elizabeth Miles: Senior Lecturer Staff Development and CD PGCHEDr Clare Shoults: T Year Lead and Consultant Acute Medical Unit
Workplace Based Assessments for Undergraduates:
Training for Assessors1. Context of assessment for students
2. WPBA3. How to supervise and assess WPBA4. Common pitfalls5. Video – try rating one6. Feedback
Outcomes 1 − The doctor as a scholar and a scientist
The graduate will be able to – Apply to medical practice biomedical scientific
principles, method and knowledge relating to: anatomy, biochemistry, cell biology, genetics, immunology, microbiology, molecular biology, nutrition, pathology, pharmacology and physiology.
– Apply psychological principles, method and knowledge to medical practice.
– Apply social science principles, method and knowledge to medical practice.
– Apply to medical practice the principles, method and knowledge of population health and the improvement of health and healthcare.
– Apply scientific method and approaches to medical research.
Outcomes 2 − The doctor as a practitioner
The graduate will be able to – Carry out a consultation with a patient.– Diagnose and manage clinical presentations.– Communicate effectively with patients and
colleagues in a medical context.– Provide immediate care in medical emergencies.– Prescribe drugs safely, effectively and
economically.– Carry out practical procedures safely and
effectively.– Use information effectively in a medical context.
Outcomes 3 − The doctor as a professional
The graduate will be able to – Behave according to ethical and legal
principles.– Reflect, learn and teach others.– Learn and work effectively within a
multiprofessional team.– Protect patients and improve care.
Why use workplace based assessments in medical
students?
Importance of clinical skillsDeficiencies in clinical skillsProblems with traditional assessments
Effectiveness of feedback
Importance of clinical skills
Interview and PE are primary source of diagnostic data
Several studies document deficienciesData gathering errors are the most common source of diagnostic errors
Good communication skills improve health status of patients, increase efficiency of care and REDUCE likelihood of litigation
Feedback
Feedback is sought after and valued by medical students and trainees
Feedback is a strong motivator for behaviour change and has been shown to improve confidence and clinical performance
In meta-analyses of the impact of various educational strategies on medical education outcomes, feedback alone effective is effective in 71% of studies (Veloski et al. 2006)
How to use the MBBS Workplace-based
Assessment (WBA) resource
All users of the MBBS WBA materials on this website are strongly recommended to:
watch the “Introduction to WBA” video watch the “Giving Effective Feedback” cluster of 3 videos (which set the scene for effective student learning through formative assessment)
How to use the MBBS Workplace-based
Assessment (WBA) resource
Select additional resources to view by following the guidance after the statement (1-3)
1. “I am unsure about WBA and what is expected of the assessor and the MBBS student.”
2. “I already know about/have taken part in WBA at postgraduate level and want to know what standard to expect of the MBBS students.”
3. “I already know about WBA and what level to expect, but want to check about formative assessment and giving feedback to MBBS students”.
How to use the MBBS Workplace-based
Assessment (WBA) resource
Watch each of the three clusters of clips – WBA– WBA Examples– Feedback
(there are points and questions to consider beside each clip): – a. Case-based Discussion – b. Mini Clinical Examination– c. Directly Observed Procedural Skill
How to use the MBBS Workplace-based
Assessment (WBA) resource
There are downloads attached to clips and under the Documents tab: – “Professional Behaviour & Clinical Practice
Assessment Guide” – “Workplace-based Assessment” PowerPoint – “Changes to Assessment of St. George’s
Students on Clinical Attachments” – WBA assessment forms
Effectiveness of feedback
Critical to learning Significant influence on achievement – meta-analyses show efficacy of feedback alone in improvement
But traditionally students are rarely observed….
Promoting effective feedback
To facilitate the student’s own self-reflection.
Prevent the student having misconceptions about performance.
Ascertain whether there is disagreement or resistance to the observer’s comments.
https://intranet.sgul.ac.uk/teach/Clinical%20skills%20videos/cbd/cbd_bad.wmv
Problems that can occur with giving feedback
include:Non-specific feedback ‘clerk more patients’
Giving personal or emotional response to performance
Fear of criticising the student.
Fear of emotional responses from student.
No time
https://intranet.sgul.ac.uk/teach/Clinical%20skills%20videos/cbd/cbd_good.wmv
Promoting effective feedback
Optimise the value of feedback by:
Start with the learner’s self-assessment of what went well and what could be improved.
Reinforce important areas and points of specific feedback.
Review understanding and feeling in response to those areas and points.
Formulate a specific and realistic plan with clear, direct goals.
Workplace based assessments
Miller’s pyramid
Single Best Answer
Short Answer Question / Extended Matching Item
Objective Structured Clinical Examination
WORK BASED ASSESSMENTS
WPBA are one component of a comprehensive assessment strategy
WPBA: ‘the
learning moment’
Constructive feedback by assessor and rating
Assessor facilitates
self appraisal by
learner
Self assessment by learner
Suitable WPBA tool
Agreement between
learner and assessor on
focus of assessment
Agreement of learning
objectives and action plan for learner
Learner demonstrates skill by on
or about real patient
WPBA: ‘the
learning moment’
Constructive feedback by assessor and rating
Assessor facilitates
self appraisal by
learner
Self assessment by learner
Suitable WPBA tool
Agreement between
learner and assessor on
focus of assessment
Agreement of learning
objectives and action plan for learner
Learner demonstrates skill by on
or about real patient
WPBA: ‘the
learning moment’
Constructive feedback by assessor and rating
Assessor facilitates
self appraisal by
learner
Self assessment by learner
Suitable WPBA tool
Agreement between
learner and assessor on
focus of assessment
Agreement of learning
objectives and action plan for learner
Learner demonstrates skill by on
or about real patient
WPBA: ‘the
learning moment’
Constructive feedback by assessor and rating
Assessor facilitates
self appraisal by
learner
Self assessment by learner
Suitable WPBA tool
Agreement between
learner and assessor on
focus of assessment
Agreement of learning
objectives and action plan for learner
Learner demonstrates skill by on
or about real patient
WPBA: ‘the
learning moment’
Constructive feedback by assessor and rating
Assessor facilitates
self appraisal by
learner
Self assessment by learner
Suitable WPBA tool
Agreement between
learner and assessor on
focus of assessment
Agreement of learning
objectives and action plan for learner
Learner demonstrates skill by on
or about real patient
WPBA: ‘the
learning moment’
Constructive feedback by assessor and rating
Assessor facilitates
self appraisal by
learner
Self assessment by learner
Suitable WPBA tool
Agreement between
learner and assessor on
focus of assessment
Agreement of learning
objectives and action plan for learner
Learner demonstrates skill by on
or about real patient
WPBA: ‘the
learning moment’
Constructive feedback by assessor and rating
Assessor facilitates
self appraisal by
learner
Self assessment by learner
Suitable WPBA tool
Agreement between
learner and assessor on
focus of assessment
Agreement of learning
objectives and action plan for learner
Learner demonstrates skill by on
or about real patient
What is a workplace based assessment?
WBA are observed assessments of clinical practice in the workplace, with feedback to the learner on the strengths and weaknesses of his or her performance.
The key is observed practice and developmental feedback.
Workplace based assessments
Vehicles for offering direct, timely and relevant feedback
High validity, High acceptability, High educational impact. ? Cost –effectiveness
Poor reliability
Key components of a WPBA
Designed to assess a specific skill/taskUse rating scales – to provide information on specific behaviours and the quality and fitness of their performance (vs checklists)Opportunity for feedback to improve subsequent performance
Skills
HistoryExaminationDiagnostic ReasoningManagement – investigations and interventionsEthicsPractical proceduresAcute or chronic contextsGeneral or specificPersonal behaviours relevant to clinical practice
Learner Training programme Public/Employer
Gain feedback Demonstrating trainee progression
Ensuring trainees are competent
Motivation through positive feedback
Contributing evidence for summative decision making
Encouraging excellence
Encourage aspiration to mastery
Identifying doctors in difficulty
Promoting learning and informing learning objectives
Providing data to support decisions involving trainees in difficulty
Providing evidence to inform annual review of competence progression
Providing information when designing remediation packages
Demonstrating progression Identifying patterns of behaviour
Academy of Medical Colleges 2009
WBA: Assessment challenges: reliability
Inter-observer variability
Intra-observer reliability
Case-specificity
Therefore we are using them formatively
Mini CEX
Should be based on – observed focused history taking,– observed examination – observed higher communication skills
Main domains – history taking, – physical examination skill, – clinical judgement and synthesis, – humanistic skills
Also overall ratingDifferent details in different schemes – number of items and rating scales
How many mini-CEX do you need?
In T year2 CEX per long clinical attachmentSurgeryGP Medicine1 In AMU week
How many mini-CEX do you need?
In P year2 CEX per long clinical attachment
1 Short Attachments
Exceptions
– O+G – Psychiatry Medicine– Surgery– Senior Medicine– Paediatrics
– AMU week– Speciality
– Neurology 2 CEX (Including GCS)
How many mini-CEX do you need?
In F year1CEX per long firmSurgery AHOMedicine AHOGP AHO
Direct Observation of Procedural Skills
DOPs allows students to move from procedures learnt in a skills lab to performance in real patients.
Performance of practical procedures involves technical skills but also communication, patient approach and attitudes to analgesia and patient comfort
DOPS 4P’s
Preparation
Performance
Practice Based Discussion
Progression
DOPS Assessment
Preparation for DOPSProficient in SIMSObservation in work place of a competent practioner
Consent
DOPS Assessment
PerformanceSee Clinical Skills LOBSPre clinical and T year LOBS for clinical skills
DOPS Assessment
Practice Based DiscussionIndicationSafetyInterpretation of result
DOPS Assessment
ProgressionReflection on performance of the skill in a clinical setting
Learning neededFurther assessment
DOPS
DOPS compulsory but assessment formative.
Practical procedures must be supervised and assessed by F1 or above, or non-medic who performs procedure routinely in clinical practice e.g. nurse or e.c.g. technician
Standard: Procedurally correct and safe
Supervising DOPS
Are you confident to do the procedure? If not refuse…….
Is the student ready to perform the procedure on a real patient?
Talk through whether the student knows how to do the procedure…..can they accurately describe the detail of the process……if not refuse!
End of Firm Assessment
Previously had problems Benchmarking results
Clinical Practice Grades by attachment
AHO Medicine
AHO Surgery A&E CCA
AHO GP Public Health
Excellent 214 (73%) 165 (56%) 77 (26%) n/a 160(54%) n/a
Acceptable 80 129 217 294 133 294
Clinical Practice: AHO Surgery by site
No. of students
Clinical Practice: AHO medicine by site
No. of students
Clinical Practice: A+E grades by clinical site
No of students
GP: Clinical Practice grades by site
No of students
Old Assessment Criteria
Ex - Excellent A - Acceptable CC - Cause for Concern U - Unacceptable
Excellent T year knowledge of basic and clinical science with ability to apply to unfamiliar problems.
Adequate T year knowledge of basic and clinical science, applied to clinical practice.
□ Very limited knowledge of basic and clinical science.
□ Lacks knowledge of basic and clinical science
Good thorough history- taking and consistent fluent examination of all major systems.
Competent basic examination and adequate history-taking, not necessarily with fluency.
□ Unsystematic examination with limited skill and incomplete histories without diagnosis.
□ Unable to do basic examination or formulate sensible diagnoses or rational management.
Demonstrates an excellent for T year, broad analytical approach to diagnostic problems
Adequate T year standard approach to common diagnostic problems.
□ Able to describe key features of common disease but cannot explain them.
□ Unable to explain disease processes.
Excellent, above average T year, consideration of a broad variety of treatment options and able to explain preferred option.
Adequate T year approach to considering treatment options
□ Aware of common treatments but not rationale.
□ Not aware of very basic treatment groups
Excellent awareness and articulation of wider social and societal issues.
Adequate T year awareness of broader psychosocial issues. □ Limited awareness of
psychosocial issues. □ Cannot identify psychosocial issues.
Other..........................................................
New End of Firm Sign Off
Doctor as a scholar
Knowledge of basic and clinical science, including pharmacology, as it relates to the clinical practice
Correct identification of important psychosocial and ethical aspects of clinical medicine:
Doctor as a practioner
Information Gathering–History content and technique–Examination content and technique–Patient centredness: communication,
attention to modesty and pain, disability.
Doctor as a practioner
Synthesis of Clinical Problem–Case synthesis: Correctly identifies and
prioritises differential diagnoses–Reasoned and detailed approach to
investigation choices, or interpretation–Treatment Plan: Correct content and
prioritisation with reasoned justification. –Structured and concise medical record
keeping
Doctor as a practioner
Clinical Management–Understand how to prescribe drugs
safely, effectively and economically–Carry out practical procedures safely
and effectively–Communicates effectively with patients –Communicates and colleagues in a
medical context
Advantages
Aligns with Training DoctorsAligns with postgraduate benchmarking
Allows a more tangible assessment of what stage the student is at in the course
Gives more extensive structured feedback
Summary
Clinical Skills and professionalism are core to a doctors practice
DaP is the key to formally assessing this by peer evaluation
Students appreciate effective and timely feedback
Benchmarking is problematic
Summary
The new assessmentsUse assessment tools based on validated post
graduate training assessment– Trainers/trainees already familiar with them
New system will be electronic– Rapid feedback– Easier – More secure
Benchmarking on expected performance for stage of training– Should allow more standarization of assessments with
better QA