the ecvim specialist examination in internal medicine · the ecvim specialist examination in...
Post on 18-Apr-2018
277 Views
Preview:
TRANSCRIPT
The ECVIM Specialist Examination
in Internal Medicine
K. GommerenUniversity of Liège
O. DossinUniversity of Toulouse
1
Outline
• Presentation of the examination
• Results of the on-line survey (July 2012)
• Implemented changes
• Planned actions
2
Why do we need an examination?
• EBVS requirement
• Maintain the credibility of the title▫ Promote utilization of specialty services by the public/profession
▫ Improved veterinary services available to the public
• Performance assessment for minimal competencies
• Compensate for the diversity of the training programs by applying standards….
3
How do you become ECVIM-CA board
certified in Internal Medicine?
• Complete internship and residency
• Pass the MCQ general exam
• Pass credential examination
• Pass the specialist examination in Internal Medicine
Recredentialing every 5 year: 100 points sytem
4
Exam organization
• 3 parts
▫ MCQ : 100 questions. 1 stem: 1 correct answer and 4 distractors
▫ Essay questions: 4 EQ, 4 broad topics, short answer Q
▫ Case management session
� 3 oral case management (OCM)
� 1 practical part: images assessment, written part
5
How is the exam prepared?
• MCQ▫ Call to diplomates to submit questions?� Recommendations� Return rate variable� 20 to 50 diplomates
� 40 to 150 questions
� Less than 20% usable
� Less that 15 % fulfilling recommendations
▫ Prepared by EC� Reviewed by 6 persons� Consultation with experts if problem
6
How is the exam prepared?
• Essay questions
▫ Call to diplomates to submit ideas?
� 1 response this year!
▫ Prepared by EC
� Reviewed by 6 persons
� Consultation with experts if problem
� Memo with answers prepared and reviewed by EC
7
How is the exam prepared?
• Case management session▫ Prepared by EC/reviewed by 5-6 persons▫ OCM
� 3 cases with clinical data and laboratory/procedures results
� Minimal pathway� Problem oriented approach/problem solving skills/critical
thinking� But not a predefined scenario: multiple ways are possible!!!
▫ Practical� Images/graphics� Clinical scenario� Memo with answers prepared and reviewed by EC
8
How is the exam marked?
• MCQ
▫ Electronic marking
▫ Statistical evaluation for
� reliability,
� difficulty,
� definition of the pass mark
9
How is the exam marked?
• MCQ
▫ Pass mark
� According to
� Performance/difficulty of exam across years in new candidates
� Exam reliability: Cronbach alpha
� Difficulty index for each question
� Discrimination of Q
10
• MCQ
▫ Evaluation by diplomates according to Angoff method
� What percentage of minimally qualified candidates should answer this question correctly?
� Comments on each questions
70% (±10, range 53-88%)
11
How is the exam marked?
• Essay questions▫ Double marking according to a model answer▫ Triple or quadruple marking if more than 15 % difference
▫ Pass mark is 15 points per essay (total of 60/100)
• Practical▫ Double marking according to a memo▫ Multiple marking if problem▫ This part is designed for
� Evaluation of image interpretation skills� Possible compensation of borderline OCM
12
How is the exam marked?
• OCM▫ 2 examiners ± 1 observer▫ Exam is tape-recorded▫ Marking
� According to a minimal pathway� Principles of problem oriented approach
� Clinical reasoning/Critical thinking
▫ If disagreement between examiners or doubt� Recorded file is evaluated by 3rd/4th person
▫ Final decision is fail or pass▫ Practical can modestly compensate for borderline candidates
13
How did I ever end up here???
• Newsletter Vol 20, no.2, 2010
(K. Gommeren)
▫ Stress
▫ Reading list
▫ Work load
▫ Pass rates
▫ ACVIM vs ECVIM
• Newsletter Vol 21, no.1, 2011(E. Teske)
▫ Pass rates
▫ Reading list
▫ Work load
▫ ACVIM vs ECVIM
▫ Stress
• Newsletter Vol 22, no.1, 2012(A. nonymous)
▫ Work ethics
▫ Deadlines
▫ Case failure
▫ Transparency
14
Snow Meeting
Purpose of the examination• Knowledge
• Clinical skills/management
• Problem solving/Communication
• End/Validation
• Assessment of programmes
15
Executive Board
Credentialing Process Survey• Case reports
▫ Stopped/Reduced
▫ Feedback
▫ Resubmit sooner to avoid delay
• Publications
▫ Reduce requirements
� Accept case reports/series
� Sit exam before acceptance
16
Credential & examinationSurvey Construction
Laurence MichielsMaud JacquetAlix McBreartyAstrid WehnerMiguel CamposKris GommerenOlivier Dossin
• Telematic Teaching Service Ulg
• Diplomates
17
Survey Results
• 204 responses▫ 93 ECVIM exam
▫ 24 A/ECVIM
▫ 30 De Facto
▫ 29 Residents
▫ 28 Ex-residents
▫ 147 Diplomates (58%)
▫ 57 Non-diplomates (?)
18
Diplomates 146 Ex-residents 28
• Top 10% 53
• Top 25% 46
• Top 50% 21
• Bottom 50% 3
• Top 10% 4
• Top 25% 11
• Top 50% 6
• Bottom 50% 0
Residents 29
• Top 10% 7
• Top 25% 8
• Top 50% 10
• Bottom 50% 0
Top 10%
Top 25%
Top 50%
Bottom 50%
Resident and Diplomates
What type of students
19
Motivation residency
• Interest/passion 108
• Referral practice 23
• Job opportunity 16
• Research 8
• Offered/invited 6
• First opinion boring 4
• Logical step 2
• More money 1
• Don’t know 1
20
Motivation programme
• Same institution 51
• I got accepted 45
• Ideal location 25
• Programme quality 22
• Clinical guidance 14
• Recommended 9
• Feline 3
21
Competences
DipECVIM-CA• Case management 154
• Knowledge 119
• Practical skills 59
• Communication 55
• Teach 54
• Perform research 45 • Critical thinking 42
• Be ethical/caring 16
• Have experience 14
• Out of the box 10
• Self-awareness 10
• Stress resistant 10
22
Did you feel prepared for….?
at the end of your residency (1-5)?
Referral
medicine
Guidance of
residents
Clinical
research
All 3,89 3,59 3,05
ACVIM Dipl 4,56 4,31 3,5
ECVIM trained 3,81 3,50 2,99
- ECVIM Dipl 3,91 3,54 2,99
- Ex-residents 3,61 3,43 3,17
- Residents 3,65 3,43 2,83
23
Time management
Time off/year• 0 4
• 1-4 38
• 4-8 59
• 9-12 56
• 13-16 3
• 17-20 4
• >20 4
24
Time management
General examAll candidates Not enough Those who failed Not enough
• 0 7 2
• 1-4 92 16
• 4-8 50 2
• 9-12 11
• >20 1
• 0 1 1
• 1-4 20 7
• 4-8 13 1
0
1-4
5-8
9-12
>20
25
Time management
CredentialsAll candidates Not enough Those who failed Not enough
• 0 53 16
• 1-4 72 9
• 4-8 15 3
• 9-12 6
• >20 1
• 0 16 5
• 1-4 18 2
• 4-8 6 2
• 9-12 3
• >20 1
0
1-4
5-8
9-12
>20
26
Time management
Certifying examAll candidates not enough Those who failed not enough
• 0 7 3
• 1-4 34 8
• 4-8 54 11
• 9-12 24 4
• 13-16 2
• >20 1
• 0 4 1
• 1-4 14 5
• 4-8 24 9
• 9-12 12 2
• >20 0
0
1-4
5-8
9-12
13-16
>20
27
Test results…
General % Credentials % Certifying %
Passed 1st attempt 124 78 94 60 72 55
Passed xnd attempt 26 17 48 31 36 28
Refused 8 5 14 9 22 17
General Credentials Certifying
28
Male vs Female
General Credentials Certifying
Passed 1st
attempt
Gender A 79 81% 69 75% 41 63%
Gender B 50 88% 39 78% 31 72%
29
Language issue
General Credentials Certifying
Passed 1st attempt B1 3 60% 2 50% 0
B2 18 82% 13 62% 5 50%
C1 24 80% 19 76% 9 47%
C2 82 85% 72 80% 57 74%
General Credentials Certifying
Passed 1st attempt English 47 87% 42 79% 32 73%
Lang B 23 74% 15 65% 11 61%
Lang C 10 91% 9 100% 4 57%
Lang D 10 83% 7 78% 4 40%
30
Accredited vs Alternative
General Credentials Certifying
Passed 1st attempt Acc. Prog. 114 84% 97 79% 61 66%
Alt. Prog. 14 74% 11 58% 10 71%
31
Region General Credentials Certifying
Country A 40 83% 38 78% 26 72%
Prog. A 8-4 8 100% 8 89% 4 100%
Prog. B 7-6 5 71% 4 67% 6 100%
Prog. C 6-5 4 67% 5 83% 4 80%
Prog. D 6-4 5 83% 2 40% 3 75%
Prog. E 10-8 8 80% 7 78% 5 63%
Prog. F 7-7 7 100% 9 90% 4 57%
Prog. G 4-4 3 75% 4 100% 2 50%
Prog. H 9-6 7 78% 5 63% 3 50%
Accredited programmes
32
Test impressions
Relevance % Quality Difficult %
General 112/166 67 3.36 126/166 76
Credentials 113/157 72 3.17 112/157 71
Certifying 3.59-3.27 115/124 93
ID failure % Feedback %
General 19/25 76 NA
Credentials 24/48 50 17/48 35
Certifying 34/58 59 14/58 24
33
Difficulties
Residency programme• Time 104• Stress 81• Administration 68• Daywork 62• Supervisor 51• Guide work 43• Family 38• Training spec 37• Guide research 37• Finding cases 34• On calls 32• Expectations 21• Colleagues 18• Scientific 18• Guide teaching 13• Language 12
34
What benefits to expect
as a DipECVIM-CACardiology + Oncology 11 Internal Medicine 69
• More opportunities 8
• Changed job 8
• More money 10
• Better job 6
• More opportunities 46
• Changed job 23
• More money 33
• Better job 21
35
What should we do to improve our exam?
• Evaluation by specialist in medical education and assessment of performance evaluation tests for clinicians
• Define clearly what is a minimally competent internist: the College’s responsibility
▫ Impact on residency programs
▫ Impact on residents studying
▫ Impact on exam preparation and evaluation
36
What has been changed?
• Credentialing process
• Rating of the MCQ S by diplomates
• Reviving the General Examination Committee
37
What should we do to improve our exam?
• Get more feed-back from diplomates▫ MCQs (quantity and quality)
▫ Essay question proposals
▫ Angoff methods for standards setting
• Consider format changes?▫ Short answers EQ w/o a defined topic for each EQ
▫ Computerized essays (hand writing!!!!!!!!!!!!!!!!!!!)
▫ Mark EQ over 100 with a PM instead of 4 times 25
▫ Improve the format of the OCM/computerized exam?
38
What should we do to improve our exam?
• Better information on ECVIM web site
• Organize mock exams for OCM (EQ?)
• Reduce the reading list
39
Reading list issue
0
100
200
300
400
500
600
2001 2011
JVIM
JAVMA
AJVR
JSAP
40
Future considerations
• Standardize the residency programmes
▫ Standard off-clinic and exam prep times
▫ Evaluate programmes more, candidate less
� Credentials
� Shared responsibility
• Create more benefits for Diplomates
▫ Public (professional) awareness
▫ Recognizition by Universities
41
Responsibility of the
CredentialsResponsibility
• Cred Comm 64 yes 140 no
• Supervisor 61 yes 143 no
• Candidate 42 yes 162 no
• Colleagues 7 yes 197 no
• ECVIM 94
▫ 41x cred committee
▫ 37x supervisor
▫ 29x cand
▫ 5x colleagues
• (Ex-)Residents 57
▫ 23x supervisor
▫ 20x cred committee
▫ 10x cand
▫ 2x colleagues
42
Thank you…
• The College
• The Board
• Survey responders
• People of the examination committeeI Aroch
D Batchelor
T Glaus
P Kook
J Schoeman
G Segev…….
B KohnN LuckschandlerE O’NeillA RidyardP Watson …….
I Aroch
D Batchelor
T Glaus
P Kook
J Schoeman
G Segev…….
Acknowledgements
B KohnN LuckschandlerE O’NeillA RidyardP Watson …….
44
Future considerations
• Standardize the residency programmes
▫ Only ECVIM programmes on European soil
▫ Average times off clinic and preparation for exam
▫ Evaluate the programme more, the candidate less
� Credentials
• Push to create more benefits for Diplomates
▫ Recognizition by universities
▫
45
Snow Meeting
Current examinationPositive Problems
• Read literature (focus)
• International standard
▫ Consistency
▫ Variable training
• Comprehensive assessment
▫ Range of
� Knowledge
� Clinical skills
� Problem solving
• Sifting procedure
• ACVIM▫ Different▫ Low pass rate
• ECVIM▫ No validation▫ Lack of transparency
� Requirements� Compensation
• Reading list• The exam
▫ OCM prep time/difficulty▫ MCQ too specific▫ Essays too focused
• Language (dis)advantage
46
Executive Board
ECVIM-CA vs ACVIM• ACVIM easier than ECVIM-CA?
• Mutual recognition
▫ AVMA objects
▫ ACVIM recognized if
� Practice in Europe
� Publication requirements
▫ Future => full credential package???
47
Snow Meeting
Current examination => Improvements• OCM
▫ Restructured format
� Serial fashion
� On day 2 case 3 for ½s
▫ Candidate numbers ↑
� Alternatives e.g. pc based
• Essays
▫ Broader/More topics
▫ One score
• MCQs
▫ 1+3 rather than 1+4
▫ Obligatory for Diplomates
• External validation
• Examination rating
• Revise
▫ Reading list
▫ Expected knowledge base
• Improve awareness of supervisors on recent exams
▫ Publish past essays
48
Executive Board
Credentialing Process Survey• Survey for diplomates (115; 45%)
▫ >50% overall approval
� >95% presentation log
� >75% case log
� >75% activity log
� >65% procedure log
49
Executive Board
AGM Proposals• Case reports
▫ 5 becomes 3
� First
� General Exam
� Help from supervisor
� Feedback from exam committee
• Publications
▫ First author on an original scientific study
▫ Second paper
� Co-author of original scientific study
� First author of a Review/Case report/Case series in a peer reviewed journal
• Activity log
▫ Replaced by signed testament from the supervisor
▫ Unaltered for alternative residencies
50
What was true/you should have known
before last night…• Participants
▫ 110 married
▫ 8 divorcees
▫ 43 « in a relationship »
▫ 42 singles
• Civil state
▫ 12 broke up while 13 got married
▫ 13 started with children
& 15 made them along the way
51
What to expect of a
Residency Programme• Finances 169
▫ Saved money 15 4 Xtra work▫ Enough 64 12Xtra work▫ Financial aid 68 16Xtra work▫ Extra income 22 19Xtra work
• 51 performed Xtra work▫ Non-veterinary work 3▫ PhD 4▫ Locums 44
52
Time managment
On clinics/day ; On call nights/weekHours on clinics/day /wk Nights on call/week
• ACVIM 11.3 (10-14) 56.5
• ECVIM 10.4 (3-15) 52
• Ex-residents 11.4 (6-14) 57
• Residents 10.7 (8-14) 53.5
• A-students 10.8 (7-15) 54
• ACVIM 3 (0-7)
• ECVIM 1.8 (1/2-7)
• Ex-residents 1.8 (0-7)
• Residents 1.8 (1/2-5)
• A-students 1.9 (1/2-7)
53
Resident and Diplomates
What type of studentsDiplomates 147 Ex-residents 28
• A-students 107
• Resit 36
• Double 3
• A-students 13
• Resit 15
• Double 0
Residents 29
• A-students 17
• Resit 12
• Double 0
54
Objectives
Residency ProgrammeTraining Requirements
• Case management 67
• Practical skills 52
• Knowledge 49
• Research 33
• Critical thinking 21
• Communication 14
• Teaching 7
• Support/Supervise 64
• Provide cases 40
• Provide resources 24
• Provide structure 12
• Other 27
55
GOAL
Credential/Examination Process
Ascertain; Be/Make/Ensure
Check; Evaluate; Test; Verify
Confirm; Demonstrate;
Establish; Identify
Final Step; Determine; Proof;
Validate
Assess; Differentiate
Create; Set a Standard
56
Credential/Examination Process
Results ~Capacities• All participants 2.82
▫ ACVIM 3.53
▫ De Facto 3.12
▫ ECVIM 2.95
▫ Residents 2.35
▫ Ex-residents 2.04
57
Resident and Diplomates
Previous working experience
Diplomates
146
ECVIM
93
Ex-residents
28
Residents
29
PhD 37 26 5 6
Private 74 60 24 18
Internship 66 51 20 19
Industry 1 1 1 3
Publish 36 29 6 4
58
Test results…
93 diplomatesOverall (93) % Time-effect
• Perfect score 40 43
• Failed GE 13 14
• Failed CR 25 27
• Failed CE 35 38
• 1998-2005 (26) %
▫ Perfect score 12 46
▫ Failed GE 3 12
▫ Failed CR 6 23
▫ Failed CE 8 31
• 2006-2011 (67)
▫ Perfect score 28 42
▫ Failed GE 10 15
▫ Failed CR 19 28
▫ Failed CE 27 40
59
Test results…
57 (ex-)residentsOverall (57) Residents (29) % Dipl
• Failed GE 24/57 15 OK
• Failed CR 23/45 13 OK
• Failed CE 17/17
• Failed GE 11/29 38 14
• Failed CR 9/20 45 27
• Failed CE 5/5
60
Region General Credentials Certifying
Country B 19 79% 15 79% 8 67%
Prog. I 4-3 2 50% 2 50% 2 67%
Prog. J 4-0 3 75% 1 100% 0 -
Prog. K 9-5 8 89% 6 86% 3 60%
Prog. L 5-3 5 100% 4 80% 2 67%
Prog. M 2-2 1 100% 2 100% 1 100%
Country C 22 96% 15 79% 7 41%
Prog. N 6-6 6 100% 4 67% 2 33%
Prog. O 9-6 9 100% 6 75% 1 17%
Prog. P 8-5 7 88% 5 100% 4 80%
Accepted programmes
61
Test impressionsTopic Relevance %
Credentials
Case log 97/157 62
Article 89/157 57
5 cases 86/157 55
Oral presentation 81/157 52
External rotations 51/157 32
Certifying
EQ 107/124 86
PS 106/124 85
OCM 104/124 84
MCQ 88/124 71
62
Test impressions
• Problems
▫ General
� Poor guidelines70
� Stress 45
▫ Credentials
� Case selection 70
� Time 69
� Case level 58
� Write cases 54
� Supervision 43
• Problems
▫ Certifying
� Quantity 80
� Time 56
� Transparency 53
� Poor guidelines 41
� Hours 38
63
1st attempt
Xnd attempt
Refused
General exam
• Sat it 164
• Passed 158
▫ Passed 1st attempt 124
▫ Passed xnd attempt 26
▫ Refused GE 8
Credential and Examination
Failure
64
1st attempt
Xnd attempt
Refused
• Submitted 156
• Passed 142
▫ Passed 1st attempt 94
▫ Passed xnd attempt 48
▫ Refused credentials 14
Credentials
Credential and Examination
Failure
65
1st attempt
Xnd attempt
Refused
Certifying exam n %
• Sat it 130 100
• Passed 1st attempt 72 55
• Fail 1st attempt 58 45
▫ Not succeeded 22 17
▫ Passed 2nd try 29
▫ Passed 3rd try 7
Credential and Examination
Failure
66
Residency
InsufficienciesCase load Guidance
• NONE 78• Infectious 23• Cardiology 22• Neurology 21• Respiratory 19• Immunology 16• Endocrinology 16• ECC 12• Oncology 9• Urology 9• Hematology 4• Hepat/Pancreatic 4• GE 1
• NONE 45
• ECC 42
• Neurology 40
• Cardiology 33
• Infectious 28• Immunology 27
• Respiratory 25
• Oncology 22
• Urology 15
• Hematology 13
• Endocrinology 12
• GE 10
• Hepat/Pancreatic 5
67
Improvements
• Exam evaluation by diplomates 133
• Monitor residency programs 129
• Check covered subjects exam 124
• Common case-mistakes on website 122
• Communication 119
• More examples on website (essay and OCM) 117
• More common responsibility 115
• Check exam pass rates 105
• Evaluate language influence 104• Objectives residency and definition specialist 93
• Contact person outside of programme 92
• Objectives of each exam/credentials part 87
• Yearly resident monitoring 76
• Selection criteria for veterinarians 64
68
OTHER Improvements
• Exams before credentials
• Supervisors that have passed the exams
• Homogenous residency programmes
69
top related