the ecvim specialist examination in internal medicine · the ecvim specialist examination in...

The ECVIM Specialist Examination

in Internal Medicine

K. GommerenUniversity of Liège

O. DossinUniversity of Toulouse

1

Outline

• Presentation of the examination

• Results of the on-line survey (July 2012)

• Implemented changes

• Planned actions

2

Why do we need an examination?

• EBVS requirement

• Maintain the credibility of the title▫ Promote utilization of specialty services by the public/profession

▫ Improved veterinary services available to the public

• Performance assessment for minimal competencies

• Compensate for the diversity of the training programs by applying standards….

3

How do you become ECVIM-CA board

certified in Internal Medicine?

• Complete internship and residency

• Pass the MCQ general exam

• Pass credential examination

• Pass the specialist examination in Internal Medicine

Recredentialing every 5 year: 100 points sytem

4

Exam organization

• 3 parts

▫ MCQ : 100 questions. 1 stem: 1 correct answer and 4 distractors

▫ Essay questions: 4 EQ, 4 broad topics, short answer Q

▫ Case management session

� 3 oral case management (OCM)

� 1 practical part: images assessment, written part

5

How is the exam prepared?

• MCQ▫ Call to diplomates to submit questions?� Recommendations� Return rate variable� 20 to 50 diplomates

� 40 to 150 questions

� Less than 20% usable

� Less that 15 % fulfilling recommendations

▫ Prepared by EC� Reviewed by 6 persons� Consultation with experts if problem

6


• Essay questions

▫ Call to diplomates to submit ideas?

� 1 response this year!

▫ Prepared by EC

� Reviewed by 6 persons

� Consultation with experts if problem

� Memo with answers prepared and reviewed by EC

7


• Case management session▫ Prepared by EC/reviewed by 5-6 persons▫ OCM

� 3 cases with clinical data and laboratory/procedures results

� Minimal pathway� Problem oriented approach/problem solving skills/critical

thinking� But not a predefined scenario: multiple ways are possible!!!

▫ Practical� Images/graphics� Clinical scenario� Memo with answers prepared and reviewed by EC

8

How is the exam marked?

• MCQ

▫ Electronic marking

▫ Statistical evaluation for

� reliability,

� difficulty,

� definition of the pass mark

9


• MCQ

▫ Pass mark

� According to

� Performance/difficulty of exam across years in new candidates

� Exam reliability: Cronbach alpha

� Difficulty index for each question

� Discrimination of Q

10

• MCQ

▫ Evaluation by diplomates according to Angoff method

� What percentage of minimally qualified candidates should answer this question correctly?

� Comments on each questions

70% (±10, range 53-88%)

11


• Essay questions▫ Double marking according to a model answer▫ Triple or quadruple marking if more than 15 % difference

▫ Pass mark is 15 points per essay (total of 60/100)

• Practical▫ Double marking according to a memo▫ Multiple marking if problem▫ This part is designed for

� Evaluation of image interpretation skills� Possible compensation of borderline OCM

12


• OCM▫ 2 examiners ± 1 observer▫ Exam is tape-recorded▫ Marking

� According to a minimal pathway� Principles of problem oriented approach

� Clinical reasoning/Critical thinking

▫ If disagreement between examiners or doubt� Recorded file is evaluated by 3rd/4th person

▫ Final decision is fail or pass▫ Practical can modestly compensate for borderline candidates

13

How did I ever end up here???

• Newsletter Vol 20, no.2, 2010

(K. Gommeren)

▫ Stress

▫ Reading list

▫ Work load

▫ Pass rates

▫ ACVIM vs ECVIM

• Newsletter Vol 21, no.1, 2011(E. Teske)

▫ Pass rates

▫ Reading list

▫ Work load

▫ ACVIM vs ECVIM

▫ Stress

• Newsletter Vol 22, no.1, 2012(A. nonymous)

▫ Work ethics

▫ Deadlines

▫ Case failure

▫ Transparency

14

Snow Meeting

Purpose of the examination• Knowledge

• Clinical skills/management

• Problem solving/Communication

• End/Validation

• Assessment of programmes

15

Executive Board

Credentialing Process Survey• Case reports

▫ Stopped/Reduced

▫ Feedback

▫ Resubmit sooner to avoid delay

• Publications

▫ Reduce requirements

� Accept case reports/series

� Sit exam before acceptance

16

Credential & examinationSurvey Construction

Laurence MichielsMaud JacquetAlix McBreartyAstrid WehnerMiguel CamposKris GommerenOlivier Dossin

• Telematic Teaching Service Ulg

• Diplomates

17

Survey Results

• 204 responses▫ 93 ECVIM exam

▫ 24 A/ECVIM

▫ 30 De Facto

▫ 29 Residents

▫ 28 Ex-residents

▫ 147 Diplomates (58%)

▫ 57 Non-diplomates (?)

18

Diplomates 146 Ex-residents 28

• Top 10% 53

• Top 25% 46

• Top 50% 21

• Bottom 50% 3

• Top 10% 4

• Top 25% 11

• Top 50% 6

• Bottom 50% 0

Residents 29

• Top 10% 7

• Top 25% 8

• Top 50% 10

• Bottom 50% 0

Top 10%

Top 25%

Top 50%

Bottom 50%

Resident and Diplomates

What type of students

19

Motivation residency

• Interest/passion 108

• Referral practice 23

• Job opportunity 16

• Research 8

• Offered/invited 6

• First opinion boring 4

• Logical step 2

• More money 1

• Don’t know 1

20

Motivation programme

• Same institution 51

• I got accepted 45

• Ideal location 25

• Programme quality 22

• Clinical guidance 14

• Recommended 9

• Feline 3

21

Competences

DipECVIM-CA• Case management 154

• Knowledge 119

• Practical skills 59

• Communication 55

• Teach 54

• Perform research 45 • Critical thinking 42

• Be ethical/caring 16

• Have experience 14

• Out of the box 10

• Self-awareness 10

• Stress resistant 10

22

Did you feel prepared for….?

at the end of your residency (1-5)?

Referral

medicine

Guidance of

residents

Clinical

research

All 3,89 3,59 3,05

ACVIM Dipl 4,56 4,31 3,5

ECVIM trained 3,81 3,50 2,99

- ECVIM Dipl 3,91 3,54 2,99

- Ex-residents 3,61 3,43 3,17

- Residents 3,65 3,43 2,83

23

Time management

Time off/year• 0 4

• 1-4 38

• 4-8 59

• 9-12 56

• 13-16 3

• 17-20 4

• >20 4

24

Time management

General examAll candidates Not enough Those who failed Not enough

• 0 7 2

• 1-4 92 16

• 4-8 50 2

• 9-12 11

• >20 1

• 0 1 1

• 1-4 20 7

• 4-8 13 1

0

1-4

5-8

9-12

>20

25

Time management

CredentialsAll candidates Not enough Those who failed Not enough

• 0 53 16

• 1-4 72 9

• 4-8 15 3

• 9-12 6

• >20 1

• 0 16 5

• 1-4 18 2

• 4-8 6 2

• 9-12 3

• >20 1

0

1-4

5-8

9-12

>20

26

Time management

Certifying examAll candidates not enough Those who failed not enough

• 0 7 3

• 1-4 34 8

• 4-8 54 11

• 9-12 24 4

• 13-16 2

• >20 1

• 0 4 1

• 1-4 14 5

• 4-8 24 9

• 9-12 12 2

• >20 0

0

1-4

5-8

9-12

13-16

>20

27

Test results…

General % Credentials % Certifying %

Passed 1st attempt 124 78 94 60 72 55

Passed xnd attempt 26 17 48 31 36 28

Refused 8 5 14 9 22 17

General Credentials Certifying

28

Male vs Female


Passed 1st

attempt

Gender A 79 81% 69 75% 41 63%

Gender B 50 88% 39 78% 31 72%

29

Language issue


Passed 1st attempt B1 3 60% 2 50% 0

B2 18 82% 13 62% 5 50%

C1 24 80% 19 76% 9 47%

C2 82 85% 72 80% 57 74%


Passed 1st attempt English 47 87% 42 79% 32 73%

Lang B 23 74% 15 65% 11 61%

Lang C 10 91% 9 100% 4 57%

Lang D 10 83% 7 78% 4 40%

30

Accredited vs Alternative


Passed 1st attempt Acc. Prog. 114 84% 97 79% 61 66%

Alt. Prog. 14 74% 11 58% 10 71%

31

Region General Credentials Certifying

Country A 40 83% 38 78% 26 72%

Prog. A 8-4 8 100% 8 89% 4 100%

Prog. B 7-6 5 71% 4 67% 6 100%

Prog. C 6-5 4 67% 5 83% 4 80%

Prog. D 6-4 5 83% 2 40% 3 75%

Prog. E 10-8 8 80% 7 78% 5 63%

Prog. F 7-7 7 100% 9 90% 4 57%

Prog. G 4-4 3 75% 4 100% 2 50%

Prog. H 9-6 7 78% 5 63% 3 50%

Accredited programmes

32

Test impressions

Relevance % Quality Difficult %

General 112/166 67 3.36 126/166 76

Credentials 113/157 72 3.17 112/157 71

Certifying 3.59-3.27 115/124 93

ID failure % Feedback %

General 19/25 76 NA

Credentials 24/48 50 17/48 35

Certifying 34/58 59 14/58 24

33

Difficulties

Residency programme• Time 104• Stress 81• Administration 68• Daywork 62• Supervisor 51• Guide work 43• Family 38• Training spec 37• Guide research 37• Finding cases 34• On calls 32• Expectations 21• Colleagues 18• Scientific 18• Guide teaching 13• Language 12

34

What benefits to expect

as a DipECVIM-CACardiology + Oncology 11 Internal Medicine 69

• More opportunities 8

• Changed job 8

• More money 10

• Better job 6

• More opportunities 46

• Changed job 23

• More money 33

• Better job 21

35

What should we do to improve our exam?

• Evaluation by specialist in medical education and assessment of performance evaluation tests for clinicians

• Define clearly what is a minimally competent internist: the College’s responsibility

▫ Impact on residency programs

▫ Impact on residents studying

▫ Impact on exam preparation and evaluation

36

What has been changed?

• Credentialing process

• Rating of the MCQ S by diplomates

• Reviving the General Examination Committee

37


• Get more feed-back from diplomates▫ MCQs (quantity and quality)

▫ Essay question proposals

▫ Angoff methods for standards setting

• Consider format changes?▫ Short answers EQ w/o a defined topic for each EQ

▫ Computerized essays (hand writing!!!!!!!!!!!!!!!!!!!)

▫ Mark EQ over 100 with a PM instead of 4 times 25

▫ Improve the format of the OCM/computerized exam?

38


• Better information on ECVIM web site

• Organize mock exams for OCM (EQ?)

• Reduce the reading list

39

Reading list issue

0

100

200

300

400

500

600

2001 2011

JVIM

JAVMA

AJVR

JSAP

40

Future considerations

• Standardize the residency programmes

▫ Standard off-clinic and exam prep times

▫ Evaluate programmes more, candidate less

� Credentials

� Shared responsibility

• Create more benefits for Diplomates

▫ Public (professional) awareness

▫ Recognizition by Universities

41

Responsibility of the

CredentialsResponsibility

• Cred Comm 64 yes 140 no

• Supervisor 61 yes 143 no

• Candidate 42 yes 162 no

• Colleagues 7 yes 197 no

• ECVIM 94

▫ 41x cred committee

▫ 37x supervisor

▫ 29x cand

▫ 5x colleagues

• (Ex-)Residents 57

▫ 23x supervisor

▫ 20x cred committee

▫ 10x cand

▫ 2x colleagues

42

Thank you…

• The College

• The Board

• Survey responders

• People of the examination committeeI Aroch

D Batchelor

T Glaus

P Kook

J Schoeman

G Segev…….

B KohnN LuckschandlerE O’NeillA RidyardP Watson …….

I Aroch

D Batchelor

T Glaus

P Kook

J Schoeman

G Segev…….

Acknowledgements

B KohnN LuckschandlerE O’NeillA RidyardP Watson …….

44

Future considerations

• Standardize the residency programmes

▫ Only ECVIM programmes on European soil

▫ Average times off clinic and preparation for exam

▫ Evaluate the programme more, the candidate less

� Credentials

• Push to create more benefits for Diplomates

▫ Recognizition by universities

▫

45

Snow Meeting

Current examinationPositive Problems

• Read literature (focus)

• International standard

▫ Consistency

▫ Variable training

• Comprehensive assessment

▫ Range of

� Knowledge

� Clinical skills

� Problem solving

• Sifting procedure

• ACVIM▫ Different▫ Low pass rate

• ECVIM▫ No validation▫ Lack of transparency

� Requirements� Compensation

• Reading list• The exam

▫ OCM prep time/difficulty▫ MCQ too specific▫ Essays too focused

• Language (dis)advantage

46

Executive Board

ECVIM-CA vs ACVIM• ACVIM easier than ECVIM-CA?

• Mutual recognition

▫ AVMA objects

▫ ACVIM recognized if

� Practice in Europe

� Publication requirements

▫ Future => full credential package???

47

Snow Meeting

Current examination => Improvements• OCM

▫ Restructured format

� Serial fashion

� On day 2 case 3 for ½s

▫ Candidate numbers ↑

� Alternatives e.g. pc based

• Essays

▫ Broader/More topics

▫ One score

• MCQs

▫ 1+3 rather than 1+4

▫ Obligatory for Diplomates

• External validation

• Examination rating

• Revise

▫ Reading list

▫ Expected knowledge base

• Improve awareness of supervisors on recent exams

▫ Publish past essays

48

Executive Board

Credentialing Process Survey• Survey for diplomates (115; 45%)

▫ >50% overall approval

� >95% presentation log

� >75% case log

� >75% activity log

� >65% procedure log

49

Executive Board

AGM Proposals• Case reports

▫ 5 becomes 3

� First

� General Exam

� Help from supervisor

� Feedback from exam committee

• Publications

▫ First author on an original scientific study

▫ Second paper

� Co-author of original scientific study

� First author of a Review/Case report/Case series in a peer reviewed journal

• Activity log

▫ Replaced by signed testament from the supervisor

▫ Unaltered for alternative residencies

50

What was true/you should have known

before last night…• Participants

▫ 110 married

▫ 8 divorcees

▫ 43 « in a relationship »

▫ 42 singles

• Civil state

▫ 12 broke up while 13 got married

▫ 13 started with children

& 15 made them along the way

51

What to expect of a

Residency Programme• Finances 169

▫ Saved money 15 4 Xtra work▫ Enough 64 12Xtra work▫ Financial aid 68 16Xtra work▫ Extra income 22 19Xtra work

• 51 performed Xtra work▫ Non-veterinary work 3▫ PhD 4▫ Locums 44

52

Time managment

On clinics/day ; On call nights/weekHours on clinics/day /wk Nights on call/week

• ACVIM 11.3 (10-14) 56.5

• ECVIM 10.4 (3-15) 52

• Ex-residents 11.4 (6-14) 57

• Residents 10.7 (8-14) 53.5

• A-students 10.8 (7-15) 54

• ACVIM 3 (0-7)

• ECVIM 1.8 (1/2-7)

• Ex-residents 1.8 (0-7)

• Residents 1.8 (1/2-5)

• A-students 1.9 (1/2-7)

53


What type of studentsDiplomates 147 Ex-residents 28

• A-students 107

• Resit 36

• Double 3

• A-students 13

• Resit 15

• Double 0

Residents 29

• A-students 17

• Resit 12

• Double 0

54

Objectives

Residency ProgrammeTraining Requirements

• Case management 67

• Practical skills 52

• Knowledge 49

• Research 33

• Critical thinking 21


• Teaching 7

• Support/Supervise 64

• Provide cases 40

• Provide resources 24

• Provide structure 12

• Other 27

55

GOAL

Credential/Examination Process

Ascertain; Be/Make/Ensure

Check; Evaluate; Test; Verify

Confirm; Demonstrate;

Establish; Identify

Final Step; Determine; Proof;

Validate

Assess; Differentiate

Create; Set a Standard

56

Credential/Examination Process

Results ~Capacities• All participants 2.82

▫ ACVIM 3.53

▫ De Facto 3.12

▫ ECVIM 2.95

▫ Residents 2.35

▫ Ex-residents 2.04

57


Previous working experience

Diplomates

146

ECVIM

93

Ex-residents

28

Residents

29

PhD 37 26 5 6

Private 74 60 24 18

Internship 66 51 20 19

Industry 1 1 1 3

Publish 36 29 6 4

58

Test results…

93 diplomatesOverall (93) % Time-effect

• Perfect score 40 43

• Failed GE 13 14

• Failed CR 25 27

• Failed CE 35 38

• 1998-2005 (26) %

▫ Perfect score 12 46

▫ Failed GE 3 12

▫ Failed CR 6 23

▫ Failed CE 8 31

• 2006-2011 (67)

▫ Perfect score 28 42

▫ Failed GE 10 15

▫ Failed CR 19 28

▫ Failed CE 27 40

59

Test results…

57 (ex-)residentsOverall (57) Residents (29) % Dipl

• Failed GE 24/57 15 OK

• Failed CR 23/45 13 OK

• Failed CE 17/17

• Failed GE 11/29 38 14

• Failed CR 9/20 45 27

• Failed CE 5/5

60

Region General Credentials Certifying

Country B 19 79% 15 79% 8 67%

Prog. I 4-3 2 50% 2 50% 2 67%

Prog. J 4-0 3 75% 1 100% 0 -

Prog. K 9-5 8 89% 6 86% 3 60%

Prog. L 5-3 5 100% 4 80% 2 67%

Prog. M 2-2 1 100% 2 100% 1 100%

Country C 22 96% 15 79% 7 41%

Prog. N 6-6 6 100% 4 67% 2 33%

Prog. O 9-6 9 100% 6 75% 1 17%

Prog. P 8-5 7 88% 5 100% 4 80%

Accepted programmes

61

Test impressionsTopic Relevance %

Credentials

Case log 97/157 62

Article 89/157 57

5 cases 86/157 55

Oral presentation 81/157 52

External rotations 51/157 32

Certifying

EQ 107/124 86

PS 106/124 85

OCM 104/124 84

MCQ 88/124 71

62

Test impressions

• Problems

▫ General

� Poor guidelines70

� Stress 45

▫ Credentials

� Case selection 70

� Time 69

� Case level 58

� Write cases 54

� Supervision 43

• Problems

▫ Certifying

� Quantity 80

� Time 56

� Transparency 53

� Poor guidelines 41

� Hours 38

63

1st attempt

Xnd attempt

Refused

General exam

• Sat it 164

• Passed 158

▫ Passed 1st attempt 124

▫ Passed xnd attempt 26

▫ Refused GE 8

Credential and Examination

Failure

64

1st attempt

Xnd attempt

Refused

• Submitted 156

• Passed 142

▫ Passed 1st attempt 94

▫ Passed xnd attempt 48

▫ Refused credentials 14

Credentials


Failure

65

1st attempt

Xnd attempt

Refused

Certifying exam n %

• Sat it 130 100

• Passed 1st attempt 72 55

• Fail 1st attempt 58 45

▫ Not succeeded 22 17

▫ Passed 2nd try 29

▫ Passed 3rd try 7


Failure

66

Residency

InsufficienciesCase load Guidance

• NONE 78• Infectious 23• Cardiology 22• Neurology 21• Respiratory 19• Immunology 16• Endocrinology 16• ECC 12• Oncology 9• Urology 9• Hematology 4• Hepat/Pancreatic 4• GE 1

• NONE 45

• ECC 42

• Neurology 40

• Cardiology 33

• Infectious 28• Immunology 27

• Respiratory 25

• Oncology 22

• Urology 15

• Hematology 13

• Endocrinology 12

• GE 10

• Hepat/Pancreatic 5

67

Improvements

• Exam evaluation by diplomates 133

• Monitor residency programs 129

• Check covered subjects exam 124

• Common case-mistakes on website 122


• More examples on website (essay and OCM) 117

• More common responsibility 115

• Check exam pass rates 105

• Evaluate language influence 104• Objectives residency and definition specialist 93

• Contact person outside of programme 92

• Objectives of each exam/credentials part 87

• Yearly resident monitoring 76

• Selection criteria for veterinarians 64

68

OTHER Improvements

• Exams before credentials

• Supervisors that have passed the exams

• Homogenous residency programmes

69

the ecvim specialist examination in internal medicine · the ecvim specialist examination in...

Documents