the quebec system of monitoring and enhancement of physicians performance

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

The Quebec system of Monitoring and Enhancement of Physicians PerformanceAndré Jacques, MD, Director

Practice Enhancement Division

Collège des médecins du Québec

Expect the Unexpected: Are We Clearly Prepared?


Potential conflict of interests disclosure

Dr André Jacques declare:

“ I do not have any affiliation or financial interest of any nature in

a business corporation, or I do not receive remuneration,

royalties or research grants from a business corporation”


Mission of the College

The mission of the

Collège des médecins du Québec

is to promote a quality

medicine in order to protect the

public and to contribute to

enhance the health of Quebecers


Governance of the College

Bureau(28 Governors)

Executive Committee(5 Governors)

President

ProfessionalInspectionCommittee

Committeeon Revision

Committeeon Discipline

Committeeon Admission

to practice

Committeeon CPD

Committeeon Budget


Management structure

Legal ServicesDivision

External AffairsDivision

Administrative Services Division

General Division

MedicalEducationDivision

PracticeEnhancement

DivisionInquiry Division


Role of the College of physicians

• Admission to practice - registration

– Accreditation of training programs and sites

– Licensure examination for family physicians (written and oral examinations, OSCE and SOO)

– Specialist certification examination (written and oral and/or clinical examination)



• Supervision of physicians’ practice– Assessment of practice of physicians in :

• Offices • Hospital• Nursing homes• Etc.

– Interventions to promote quality of practice • CME activities• Training programs• Coaching• Etc.



• Investigation of complaints related to both physicians and illegal practitioners

– Disciplinary process (for physicians)

– Legal proceedings against illegal practitioners

– Alternate resolution of complaints



• Political role

– Physicians and medical organizations

– Public and media's

– Government and official bodies

– Other health professions


Human and financial resources

• Staff of the College:– 23 full time physicians– 78 support staff

• Budget :– 18.6 millions (Canadian dollars)


Quebec medical manpowerDistribution by type of practice (N = 18 746)

49,30% 50,70%

GPs ( N= 9 242) Specialists (N = 9 504)

Based on data of December 2005


Quebec medical manpower

Distribution by sex (N = 18 746)

64,60%

35,40%

Male (N = 12 120)Female (N = 6 626)

Based on data of December 2005


Code of ethics


• A physician must practice his profession in accordance with the highest current standards; to that end he must improve and keep his knowledge up to date.

(Section 44)

Code of Ethics of physicians


Professional Code


By-laws related to supervision of physicians practice

• A Professional Inspection Committee is established within each order.

(Professional Code, section 109)


• The committee shall supervise the practice of the profession by the members of the order.


By-laws related to supervision of physicians practice


• The committee may recommend that it require a member of the order to serve a period of refresher training or take a refresher course or both and that it restrict or suspend his right to engage in professional activities during the training period or course.


By-laws related to supervision of

physicians practice


• It is forbidden to hinder in any way a member of the committee, an inspector or an expert in the performance of the duties conferred upon him by this code.


By-laws related to supervision of

physicians practice


Professional Inspection CommitteeFunctions

• Inquiries into the professional competence of a physician

• Inquiries into the the quality of the medical care provided in establishments


Practice Enhancement

Division


Practice Enhancement DivisionMandate

The mandate of the Practice Enhancement Division of the College of physicians of Quebec is the monitoring

of the practice of physicians and its enhancement in order to protect the public and contribute to enhance the

health of Quebecers


Practice Enhancement DivisionHuman and Financial Resources

• Staff of the Division :– 10 physicians

•1 director•2 assistant directors •6 inspectors-investigators•1 physician for sick MD

– 1 coordinator– 1 research agent– 11 secretaries

• Budget 2006-2007 : 4.3 M $


Practice Enhancement DivisionFunctions

• Evaluation of the quality of physicians' practice in office as well as in hospital

• Evaluation of the quality of the practice of medicine in health establishments

• Support to the Professional Inspection Committee


Practice Enhancement DivisionFunctions

•Intervene to enhance the physicians' performance in office as well as in hospital

•Support the Continuing Professional Development (CPD) Committee

•Maintain the secretariat of the Conseil québécois du développement professionnel continu des médecins


Who will be the winner !


Study of a representative sampling of 100 GPs

• Observational analysis of medical records randomly selected

• Assessment of some aspects of the quality of general practitioners’ professional practice

• Period : 1995-1996



• Professional inspection visits to 100 family physicians

• Random selection among Quebec’s family physicians :– practicing either entirely or partially in a

private office– in the greater metropolitan Montreal

region



• Two random sample groups :

– 52 family physicians licensed before 1980

– 48 family physicians licensed between 1980 and 1996



• Professional inspection visits made by– 5 inspectors/investigators from the College– Average of 10 years of experience in

assessment of competence

• Study of at least 30 patients records randomly chosen

• Using of a standard grid listing the criteria, including a scores scale


Sampling of 100 GPs Criteria of evaluation

• Office practices

• Record-keeping

• Quality and quantity of CME

• Quality of practice– Investigation– Diagnosis accuracy– Treatment plan– Relevance of care


Sampling of 100 GPs Demographic data

Distribution of physicians by sex

0% 25% 50% 75% 100%

Board of Collège desmédecins du Québec

Physicians of thestudy

Male Female


Sampling of 100 GPs Demographic data

Distribution by licensing period

0% 25% 50% 75% 100%

Board of Collège desmédecins du Québec

Physicians of the study

License obtained after 1980 (Group 1)

License obtained before 1980 (Group 2)


Sampling of 100 GPs Results

• Comparisons between physicians of groups 1 and 2 regarding :

– Sex – Age– Status in hospital – University of issuance of the medical degree – Average number of consults per week



• No significant statistical difference detected between the two groups (except for record-keeping)

• No link between demographic variables and quality of care

• Strong link between :– record-keeping and quality of care– CME activities and quality of care



0% 20% 40% 60% 80% 100%

Office practices

Record-keeping

Quality and quantityof CME

Combined results (Groups 1 and 2)

Satisfactory Unsatisfactory



0% 20% 40% 60% 80% 100%

Treatment plan

Investigation plan

Diagnosis accuracy

Relevance of care

Combined results (Groups 1 and 2)

Satisfactory Unsatisfactory


Sampling of 100 GPs Conclusion of the study

• Results enable the College to reassure the public with regard to the overall quality of the professional practices of family physicians in Quebec

• Study confirms the PIC decision to evaluate the practice of physicians who present a potential risk of deviant profile, based on quality indicators


Monitoring and enhancement of physician’s performance

Screening of all physicians

Monitoring

Monitoring

Feedback to all physicians

Enhancement

Enhancement

CME programs for groups and/or

individuals

Assessment of physicians in need

or at risk

Remediation programs

Detailed needs assessment

Monitoring Enhancement

Step 3

Step 2

Step 1


Step


Development of indicators for

individual physicians

• Choice of indicator– By Professional Inspection Committee

and Practice enhancement division, after consultation with different organizations

• Responsibility of the development– One staff physician


Development of indicators for

individual physicians• Elaboration with :

– Group of «experts» in the domain – Group of interested physicians in a hospital – Research group

• Pilot process evaluating feasibility• Validation of the indicator• Diffusion of the indicator with

appropriate guidelines• Implementation of the indicator


The use of PSA levels as a means of

detecting prostate cancer

The use of PSA level as a means of detecting prostate cancer …

When available data does not always allow to document practices and

changes inducted by an intervention




• Medical controversy about extended use of PSA as a means of detecting prostate cancer

• Publication of an evidence-based guideline



detecting prostate cancer • Publication of an information

booklet for patients (in French)• Local CME activity (in a single

region)• Analysis of physicians'

perceptions : – 90% say they are aware of it– 76% agree with it– 58% say it has influenced their practice




Evolution of the number of PSA tests (all of Quebec excluding Montreal)

0

50000

100000

150000

200000

250000

93-94 94-95 95-96 96-97 97-98 98-99

Follow-up PSA

Screening PSA

Average number of PSA tests(excluding cases of prostate cancer)

0

5

10

15

Period 1 Period 2 Period 3

Physicians who attended a workshopAll physiciansPhysicians who did not attend a workshop

Evolution of prescription profiles in Québec and in the region where a continuing medical intervention was conducted

After dissemination

After intervention

Dissemination of

guideline



detecting prostate cancer• Data analysis …

– Significant decreasing of the use of PSA levels as a means of detecting prostate cancer

– Incomplete and impossible to analyze data •Especially in Montreal area

– Data to be collected locally by archivists (in patient records)•Hard and long job


Step


Professional Inspection CommitteeComposition

• 2 physicians from the Practice Enhancement Division :

•The Director•One Assistant director

• 6 physicians in practice, representative of both medical and surgical specialties

• 10 meetings per year


Professional inspection visits Usual procedures

• Sending of a notice and a questionnaire to be completed before the visit by the physician or the establishment

• Inspection visit, with or without an expert

• Writing of a report and of a project of recommendations

• Submission of the report to the Professional Inspection Committee


Professional inspection visits’ algorithm

Recommendations

Meeting with PIC secretary CME Syndic Administrati

ve Committee

PAM

No control Control

With expert

Without expert

Structured oral

interview

New recommendatio

ns

Voluntary

training

Imposed

training

No limitation of practice

Limitation of practice

No control Control


Professional inspection visits’ algorithm

Sending of recommendations

Meeting with PIC secretary

Transfert to Syndic

Transfert to the PAMQ

No control visit Control visit With expert

Without expert

SOI

New recommendations

Volontary training program

Imposed training program

Without limitation of practice

With limitation of practice

No control visit Control visit

Good practice Unsatisfactory practice

Felicitations from the PIC

Good practice Unsatisfactory practice


Individual inquiries

49

24 25

0 0 0 0 0 0 0 0 0

10

18

7

0

12

7

10

0

5

10

15

20

25

30

35

40

45

50

Number of visited physicans

Requests from Syndic OfficeReceived informationDiplomees >35 years>25% of psychoth. (GP's)>50% house calls (GP's)>40% outside of specialtyWalk-in clinics (GP's)Requests from Exec. Comm.NSAID's prescriptionWitness groupAesthetic Medicine (GPs)>30% diagn.-ther. ActsSubstitute doctorsWork in office onlyMethadoneFailure at licence examPursuit of evaluationRestrictive permitOther programs

152 individual inquiries


54,4%

46,7%

73,6%

61,2%

69,7%

84,2%

72,8%

0%

20%

40%

60%

80%

100%

Office practices

Record Keeping

CME

Medical Examination

Accuracy of Diagnosis

Pertinencen of care

Treatment

Results of inquiries

Adequacy of :


144

35

7

0

6

7

7

7

18

6

30

15

0 20 40 60 80 100 120 140 160

Congratulations letter -Good practiceMeeting with PICsecretaryReferral to Syndic office

Recommendation of atraining programReferral to ExecutiveCommitteeRecommendations tophysicians' councilReferral for a SOI

Referral for CME

Request for a reportfrom hospitalRequest for a reportfrom physicianControl visit

Recommendations tophysician

Decisions of PIC


Practice enhancement activities

• Sending of recommendations from the Professional Inspection Committee

• Follow-up of recommendations by inspectors

• Organization of CME group activities (workshops on patient-doctor relationship, workshops on periodical health examination, courses)


Enhancement activities 1991-2003 (n=533)

11

4

12

31

51

39

78

64

46

52

66

79

0

20

40

60

80

'91-92 '92-93 '93-94 '94-95 '95-96 '96-97 '97-98 '98-99 '99-00 '00-01 '01-02 '02-03


Workshops on Physician-patient relationship 1992-2000

(n=296)Number of workshops

75

70

67

84

0 20 40 60 80 100

Better communication

Right distance

Difficult relationship

Breaking bad news


Workshops on Physician-patient relationship 1992-2000

(n=296)

Attendance to workshops

1036

913

823

1217

0 300 600 900 1200 1500

Better communication

Right distance

Difficult relationship

Breaking bad news


Other courses and workshops 1992-2000 (n=125)

Number of sessions

84

37

4

0 20 40 60 80 100

Record-keeping

Pharmacotherapy

Periodical examination


Step


Detailed needs assessment

• Structured oral interview– Specific tool developed for GP’s– Individual evaluation of the performance– 25 clinical cases to resolve– 5 cases with simulated patients– Goal of the evaluation : personalized

remedial CME prescription

• Evaluation training period– Used for needs assessment of specialists


Structured oral interview

• Since 1990 : clinical cases– Assessment of knowledge

• In 1997 : addition of OSCE cases– Assessment of skills and attitudes

• In 2000 : addition of Script concordance tests– Assessment of clinical reasoning


Structured oral interview

Domains assessedDomains assessed

• «SOI» (20 cases)–Knowledge (data collection)–Problems solving–Interpretation of lab tests

• «OSCE» (5 cases)–Knowledge (data collection)–Physical examination (skills)–Interpretation of lab tests–Physician patient relationship

• Script concordance test –Knowledge (data collection)–Clinical reasoning–Interpretation of lab tests

Content of the reportContent of the report

• Analysis of resolved cases• Analysis of referred cases• Detail of strengths and

weaknesses• Analysis of performance :

–Emergency cases –Pharmaceutical treatments–Follow-up

• Interpretations–Responsibility level able to handle–Knowing of its limits–Inappropriate but not dangerous conducts

–Harmful of dangerous conducts–Informed of recent conducts


Structured oral interviews 1995-2005 (n=101)

10 10

6

9

6 6

11

10

8 8

17

0

2

4

6

8

10

12

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005


Structured Oral Interviews 1995-2001

Personnal request26%

Request from Syndic31%

Recommendation of PIC (16)

28%

Others7%

Re-entering the roll3%

Recommendation from Committee on

discipline5%


Individualized CME programs

• Clinical training programs• Focused readings• Tutorial chart recalls


68

2 9

0

10

20

30

40

50

60

70

Clinical training programs Focused reading Tutorial chart recalls

Individualized CME


Personalized CME (1992-2000) Statistics for all physicians

(n=240)

Who are the candidates ?

199

41

177

63

0 20 40 60 80 100 120 140 160 180 200

Specialists

General practitioners

Female

Male



(n=240)

Who referred candidates ?

59

97

62

18

4

0 20 40 60 80 100

Executive Committee

Committee on Discipline

Syndic office

PIC

Personal request



(n=240)

Why do candidates need remedial CME ?

177

41

19

3

0 20 40 60 80 100 120 140 160 180 200

Striking off the Roll

Reorientation

Return to practice

Quality problems



(n=240)What problems are encountered ?

16,70%

35,80%

31,80%

4,60%

8,70%

2,40%

Record-keeping

Diagnosis accuracy

Treatment plan

Judgement

Skills

Communication


Personalized CME (1992-2000)Training programs (n=225)

12

3

1

3

23

21

17

13

6

66

15

15

0 10 20 30 40 50 60 70

General Practice

Anaesthesia

Emergency

General Surgery

Geriatrics

Medical Specialties

Obs.-Gynaecology

Petiatrics

Psychiatry

Public Health

Radiology

Surgical Specialties


82,0%

15,0%3,0%

Successful activities

Abandon retraining

Non successful activities

Personalized CME (1992-2000)Results of remedial CME


Monitoring and enhancement for improving performance in

hospitals

Screening of all health establishments

Monitoring

Monitoring

Comparative feedback to all health establishments

Enhancement

Enhancement

Feedback to the establishment

Screening of establishments in

need or at risk

Meeting with authorities

Control visit

Monitoring Enhancement

Step 3

Step 2

Step 1


Step


Monitoring of Health establishmentsChoosing of the indicators

• In collaboration with the concerned professional Association :

– Selection of experts of the working group– Proposition of indicators by the working group– Evaluation of the feasibility by a pilot– Validation of indicators– Extension of the project to a group of hospital


Monitoring with indicators

Content of the pathological report of

breast cancer


Why choosing the pathology ?

• Request from the Professional Inspection Committee

• Interest of the Quebec's Pathologists Association for implementing a monitoring program with quality indicators


And why the breast cancer pathology ?

• Integral part of most of pathologists

• Media coverage of unhappy events in an hospital

• Launching of Provincial program of breast cancer screening


Main steps of the program

• Fall 1998 : – Nomination of the experts' group

• Winter 1998-1999 : – Completion of a pilot project

– Final choice of quality indicators

• Summer 1999 : – Data collection (First study)



• Winter 1999-2000 : – Analyze and evaluation of data by the experts'

group

• Spring 2000 : – Feedback to hospitals

• Summer and Fall 2000 : – Enhancement activities by the Quebec’s

Pathologists Association



• Spring 2003 : – Data collection (Second study)

• Fall and winter 2003-04 : – New feedback to hospitals


Experts' group

• Mandate : Identify quality indicators

– Easy to monitor – Non expensive – Valid – Allowing to get a global view of the practice

of pathologists


Experts' group

• Composition : – The 5 members of the Quality-insurance

Committee of Quebec's Pathologists Association

– One member of College of Physicians' Practice Enhancement Division


• Evaluation of the last 20 medical records of breast cancer surgery in each of participating hospital

• Essential information for physicians in order to plan the treatment and establish the prognosis

Quality indicators (Spring 1999)


• Number of cases with research of hormonal receptors

• Indication - in the report - of dimension of the tumor

• Mention in the report of the distance between lesion and at least one resection margin



• Mention of histological type in final diagnosis

• Mention of histological / nuclear grade

• Mention of the presence or absence of lymphatic or vascular invasion

• Mention of histological evaluation of resection margins



The first study (Summer 1999)

• Questionnaire sent to 53 hospitals

• Rate of answers : 100 %

• Number of records evaluated : 1040 (from the 53 hospitals)

• Number of pathologists aimed at the program : 247


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Hormonal receptors (94,0 %)

Size of tumor (95,5 %)

Lesion / margin distance (68,7 %)

Histological type (99,8 %)

Histological or nuclear grade (85,5 %)

Vascular / lymphatic invation (61,6 %)

Histological evaluation of margins (90,2 %)

Results - Percentage of appearance of each element


• Feedback made by the Professional Committee in March 2000 at each of the 51 hospitals aimed at the program

• Feedback made to each hospital :– with tables and graphics– about results of the study for all hospitals aimed

at the program– about their individual performance

Feedback to hospitals


Feedback to hospitalsHistological evaluation of

margins

0

5

10

15

20

0 5 10 15 20 25 30 35 40 45 50 55

Serial number of hospital

Num

ber

of consi

stent re

cord

s


Feedback to hospitalsVascular / lymphatic invasion

0

5

10

15

20

0 5 10 15 20 25 30 35 40 45 50 55


Num

ber

of consi

stent re

cord

s


Feedback to hospitalsDistance between lesion and

margin

0

5

10

15

20

0 5 10 15 20 25 30 35 40 45 50 55


Num

ber

of consi

stent re

cord

s


Enhancement activities

•By Quebec's Pathologists Association

– Production of a CD-ROM (prototype only) about breast cancer pathology

– Presentation of the CD-ROM prototype at Quebec's Pathologists Association's annual meeting


Enhancement activities

•By Quebec's Pathologists Association

– Production and implementation of guidelines about elements to include in the pathological report of breast cancer in order to help physicians and surgeons to plan treatment and evaluate the prognosis.


The second study

• Period of the study : Fall / winter 2002-03

• Questionnaire sent to the 51 hospitals aimed at the first study

• Rate of answers : 100 %


The second study

• Number of record evaluated : 999– 1 hospital excluded (not enough records)– 1 hospital sent 19 records

• Number of pathologists aimed at the study : 246


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Hormonal receptors (97,1 %)

Size of tumor (96,2 %)

Lesion / margin distance (86,8 %)

Histological type (99,7 %)

Histological or nuclear grade (92,1 %)

Vascular / lymphatic invation (81,6 %)

Histological evaluation of margins (92,7 %)

Results - Percentage of appearance of each element


Compared resultsRate of consistency per

criteria

0%

20%

40%

60%

80%

100%

First study 90,2% 61,6% 85,5% 99,8% 68,7% 95,5% 94,0%

Second study 92,7% 81,6% 92,1% 99,7% 86,8% 96,2% 97,1%

. p=0,09 p<0,01 p<0,01 p=1,00 p<0,01 p=0,57 p<0,01

Margin evaluation

Vscular / lymphatic Invasion

Histological / nuclear Grade

Histological Type Lesion / margin

Distance Size of tumor

Hormonal receptors


Compared resultsEvaluation of lesion/margin

distance

0

5

10

15

20

0 5 10 15 20 25 30 35 40 45 50 55


Num

ber

of consi

stent re

cord

s


Performance enhancementEvaluation of lesion/margin

distance

-100%

-80%

-60%

-40%

-20%

0%

20%

40%

60%

80%

100%

0 5 10 15 20 25 30 35 40 45 50 55


Per

form

ance

enhance

men

t ra

te


Step


20

21 1

0

5

10

15

20

25

30

Hospitals Nursing homes CLSC Others

Visits in Health Establishments2002-2003 (n=24)


Activities of practice enhancement in

health establishments• Sending of recommendations by the

Professional Inspection Committee and coaching by one of the College staff physician

• Organization of courses for answering specific needs

• Workshops on evaluation of quality given in health establishments


Step


Control visits with an expert

• In 1998-1999 :

– 10 control visits of hospitals

– 1 control visit of a nursing home

– 1 control visit in another establishment


Meetings with authorities in 1998-1999

On 3 occasions, meeting with administrative authorities of an establishment (1 hospital and 2 nursing homes)

On 8 occasions, meeting with the “Régie régionale” (in regard with 2 hospitals and 4 nursing homes)

On 4 occasions, meeting with Health Minister (in regard with 1 hospital and 3 nursing homes)


The Future !


Competency

AttitudeHability to

practice

Use of

ressources

Components of physician’s performance

Professionnalism

Patient

Patient

Patient Patient

TEAM TEAM


What are the conditions of success of that system?

4 key words who support that system:

CommitmentInvestmentTeam workPartnership


First key word: Commitment

Commitment from my organization to the mandate of the Practice Enhancement Division:

Assessment and Remediation at all levels More formative then punitive Research agenda


Second key word: Investment

Support from my organization to the mandate of the Practice Enhancement Division:

Budget from 860,000 $ to 4.3 M $ 10 full time physicians dedicated to the

mandate 1 research agent (4 days/week) 22 full time persons


Third key word: Team work

Merging, in 1997, the CPD Division with the Peer Assessment Division to the new Practice Enhancement Division with a new mandate of assess physicians with a goal of remediate those in need.

Developing the new levels (level 1) to enhance a wider scope of physicians.


Fourth key word: Partnership

This system cannot work without collaboration and partnership with:

The four Universities in Quebec The two Unions (FMOQ-FMSQ) The two others Colleges (RCPSC-CFPC) The major medical associations


Questions and Exchanges

the quebec system of monitoring and enhancement of physicians performance

Documents

family physicians

college of physiciansadmission

professional activities

professional codebylaws

training period

code of ethicscode

period of refresher

medical organizationspublic