A KETHO3 OF ASSESSMENT OF A PiIYSICIAX'S COMPETENCYEiY HIS PERFCIRMANCE

;'roposal of .3 ReSealTCh Project

The past feds months ha:;e s4e1.I :~apiij.b i5XWiXC3 ciCCe;2tar.Ce Of theconcept of recertification 3s a c:esira.i3le approaah to demonstration

--T----of physicians accoczbxity. ?Gost of tr;e twenty-two specialtyboards have expressed formal approval of voluntary, periodic re-examination for the purpose of assurirlg clinical competence. Oneboard requires periodic rece.rtificatior:. Four boards have setdates fGr initiation of recertification programs.

Cn addktion,medical

intensive study is being given the entire system of--1_- licensure and to possible approaches to periodic relicensure.Ainong the proposals being seriouc: -iy considered is Lhat of basiy--rel.ice;rsuwe upon specialty board recertification " It seems apparentthat whatever approach or dpproack~es are ultimateLy adopted, theyiv L I_ ., be greatly influenced by the te~p~;-:i,-uer; l~sed in the r-cert-fica-tion process.

---assure tiie cr,nsumer and the profession with validity the clinical- - - - - - - - - --_I_--_competency of the physician." I-Iowever,"What is the most

the answer to the question,valid technique that can be devised to accomniish

this goal?"' has not yet been determined. The American Society&ofInternal Medicine (ASIlvi) is interested in pursuing this question todetermine if it can be answered.

ASIM 4.3 a federation of 51 state component societies having amembership of o:rer i2,OOC; internists in private practice and academicmedicine. The American Society of Irzterrsal Medicine is an InternalRevenue Service Code 501 (c) (6j crganizatkon. XIM has an affiliated.Socic-Economic R e s e a r c h and Fiducation Foundation which has beendesignated as 3 "Public 501 (cj (.3) U Foundation by the Internalp.e\ren-Je Service .

The 1555 ASIK House of Deie5ates expressed the following to beaIilOr;c; the primary objectives of the Society:

1, 'To develop and maintain standards for the practice ofinternal medicine zonsistent with high qluality medicalcare f and

2. 'To assure the CCi!'*l.TikinCj professional competence of theinternist in the practice of his specialty.

In pursuit of these gcais, many of ASIFi' s activities over the pastten years have been directed toward developing, t,tlT.+ating and encouraging

the acceptance of effective peer review methods as a valid approachto pubiic accountability. The Society has published manuals inpeer review and in Professional Standards Re:riew Organizationdevelopment through peer review methods. ASIM and the AmericaniMedical Assoziation cosponsored the first major nationai conferencefor dissemination of information on peer review. Valuable basicexperience i~i the urder-explored field of office practice qualityeval:la-Lion was gained in the ASIM project funded ($148,555.00)*&ro;g~, ;i. contract -ti: th the Department of Health, Education andWelfare.

ASIM participates in the Kellogg Foundation-funded FrivateInitiative in FSRO with ,the American Coliege of Physicians, AmericanHospital Association, American Medical Association and the AmericanAssociation of Foundations for Medical Care. ASIM has representa-tion on the Board of Regents of the Institute on ProfessionalStandards which administers an HEW contract for training personnelfor PSRO's.

The Problem- -

Two basic approaches to the assessment 3f a physician'scompetence are now under consideration:

1 . Evaluation of his knowledge by examination.

2 . Evaluation of the quality of the care provided in hisdaily practice of medicine.

1. Evaluation of knowledge. The formal examination based on recallof knowme=prises-all or part of the recertification processplans of all Boards at the present time. The first of these,that of the American Board of Internal Medicine, offered inOctober 1974, is designed to make maximum use of the popularAmerican College of Physicians Medical Kno?wledge Self-Assess-ment Program. Particular attention is given to the practicalclinical applications of the material addressed by therecertification exam itself, in order to counter the frequentlyexpressed criticism that self-assessment tests have emphasizedknowledge in areas of little use to the practicing physician.

iirlly the American Board of Family Practice presently requiresmandatory recertification, to be accomplished through a writtenexamination every six years, 3OG hours of continuing education,and a review of patient records selected by the member,

Inci:;ded in the plans of several of the Boards, in addition tothe trad;tional written examinations, is the use of computerassisted patient management simulation tests whose purpose isto dispiay the decrsion-making logic processes of the testce.

Proponents of the written examination, while readily admittingthe inherent weaknesses cf present testing techniques, point

out its similarity in form to initial certification. They

-3-

believe that it will provide an impetus to the physician'scontinuing education activities, since knowledge is animportant and necessary component in competence. Whileacknowledging the importance of evaluation of practicecompetence, they point out that, as yet, no wholly satis-factory technique has been developed.

Opposition to recertification solely by examination is basedon

A.

B.

C .

D.

E .

F.

T O

the following concerns:

The real issue of recertification (and perhaps eventualrelicensure) is the actual quality of care provided.

There is little evidence that any examination based onknowledge recall can demonstrate competency. The self--_--defined limi-ts of a physician's day-to-day practicefurther complicate the examination approach.

Knowledge is only one component of competency; otherimportant elements include cognitive skills, interpersonalrelationships, efficiency, adequacy of laboratoryfacilities and availability are highly important components.

There is considerable evidence that many (perhaps most) ofthe deficiencies in care detected by peer review methodsare not the result of insufficient knowledge.

There is considerable doubt that patient care simulationtests accurately display the management decisions of theoffice setting.

Undisputed adoption of the examination approach mayestablish precedent over and preempt use of more validmethods as such methods are developed in the future.

summarize it is only necessary to quo-te C. Barber Mueller'sletter to the Editor of the New England Journal of Medicine onJune 17, 1971.

"Whenever reexamination, relicensure or recertification ismentioned among medical practitioners and educators, theensuing conversation invariably proceeds in a manner thatassumes that the reexamination process will consist of acombination of continuing education courses followed byperiodic examinations conducted in a fashion not unlikethat of Part III of the National Boardslrz, or some ofthe specialty-board examinations. 'These examinationswould thus involve simulated patient situations, simulatedproblem solving, a computer-based examination, a series ofmultiple-choice questions and oral or written examinations.I contend that such examination practices are inappropriatetools for continued assessment of a physician in practiceand are not able to provide the clarity or rationalerequired to assess continuing physician performance in theline of duty. It is a physician's performance 'at work ina responsible setting' that must be assessed. Performanceat work (what a man does), rather than performance in

-4-

examination (what a man knows) is the important element."

2 . Assessment by Performance. Recognizing the need for moreobjective documentation of physician competence and thelikelihood of eventual recertification and possibly relicen-sure, ASIM's Committee on Quality Evaluation and its TaskForce on Assessment by Performance, in early 1974 made adetailed study of possible evaluative techniques and con-cluded that the only approach which addresses itself to the- -critical issue of quality of care delivered is an assessmentof clinical competence by an objective evaluation of a physi-cianss performance in day-to-day practice utilizing peer reviewmethods. We believe that this review should clearly displaythe physician's ability to obtain data in a logical and completemanner, and also to validate the clinical decisions derived fromthis data.

An awareness of the relatively primitive state of the art, andof several specific bottlenecks standing in the way of a viablecompetence evaluation caused the ASIM Board of Trustees toassign several committees specific tasks aimed at correctionof these deficiencies:

We assigned high priority to the task of assuring adequatedocumentation of care in a manner which clearly displaysthe physician's ability to obtain data in a logical andcomplete manner and the validity of his clinical decisions.Believing that the problem-oriented approach to record-keeping fulfilled these qualifications, as well as offeringother advantages over traditional methods, the Society hasactively promoted the use of the Weed System among itsmembership.

We have developed a data collection system suitable forcompletion by a non-physician and also suitable for com-puterization. The system seeks to display in a "time-oriented" manner the initial data base, diagnostic, thera-peutic services and continued care, This instrument hasbeen subjected to test by members of the ASIM Board of Trusteesand the Quality Evaluation Committee. Appropriate modifica-tions have been made.

We have also developed critical process and outcome criteriafor the majority of problems and diagnoses encountered byinternists - these to be used for comparison with actualperformance.

The ASIM Committee on Assessment by Performance (the Committeeon Quality Evaluation name changed in May 1974) believes that progresshas been made toward achievement of the above tasks and that the basictools are now available for the analysis of the practice patterns ofa limited population of internists. It is felt that refinement ofthese techniques and final determination of feasibility of approachawaits actual trial. For this reasonl the Committee proposed, andthe ASIM Board of Trustees approved earlier this year, that twodemonstration projects be initiated by January 1, 1975, with the

-5-

following goal:

"Assessment of clinical competency by an objective evaluationof the physician's investigative, therapeutic and exploratorydecisions based upon valid clinical data, and leading to anacceptable outcome. These decisions should be properlydocumented and arrived at efficiently."

The Model

The over-all conceptual model for the proposed assessment byperformance process may be expressed as follows:

RECERTIFYOR

RELICENSE

t

PASS

4

M.D. ACCEPTSASSESS. BY ---~.PERF.

I- - ---

CONTINUINGEDUCATION

A

--.- -

.-- --

-._ -~-

SURVEY FOR. ----

kPROFILE OF _---_PRACTICE

COMPARISON WITHCRITERIA AT

DECISION POINTS

+_---_.--~--.-----_-l

LOCAL PANEL 4--_--~--_- FAIL

!FAIL -- ----~.-~ / I _-~I~-+ PASS

I4

AUDIT

RECERTIFYiv) 3R

RELICENSURE

The model describes the general process of assessment byperformance. This process begins with a physician agreeing toparticipate in the program and accepting the proposed methodologyand results. The physician's practice is then surveyed to determinea profile of the practice, i.e., a frequency distribution of variousproblems and diagnoses that are treated by the physician. Certainpatient charts are then abstracted for specific information. Thisinformation is compared with predetermined criteria that are related

to various stages in the clinical decision process. This comparisonis concluded by a decision that the physician passes or fails theassessment process. Those who pass this stage of the assessmentare automatically recertified, and those who fail are referred tocontinuing education programs, and are assigned another date forrepetition of the screening/audit procedure.

?'his modeli if successfui, could provide a more meaningfulalternative to examination as one approach to recertification orrelicensure, whether voluntary or mandatory, and lend fullcredibility to our assurance to the public of high quality care.Also, if successful, this technique with its emphasis on logicbranching evaluation and outcome analysis may open the door tosignificant practical advances in the art of quaiity assessmentand consequently have considerabie impact on PSRO and relatedactivities.

Methodoloa

Twc demonstration sites have been selected for this study. Onewill be centered at the Xew Haven, Connecticut area and the otherin Columbus, Ohio. Daniel Hamaty, M.D., will coordinate theConnecticut demonstration and William Millhon, Y,D*, will coordinatethe Ohio group. Each demonstration group will consist of id to 20internists in private practice who are expected to volunteer theirparticipation in the project.

The number of participants in each group was determined asfollows. According to the AMA, there are approximately 30,000 in-ternists in private practice who are either self-designated orboard eligible or board certified in Internal Medicine. If all ofthese internists wish to participate in future assessment programs,it will be necessary to assess 3,000 internists every year ifthere is a 10 year interval between assessments. The assessmentprocess is expected to require six months, thus two assessment periodscan be designated in every year. The 3,000 internists to be assessedevery year could be divided into two groups of 1,500 each. Theassessment is expected to be conducted in a rlumber of regions.Assuming approximately 200 regions (possibly reiated to the PSROregions) the assessment will involve no more than 20 physicians perregion per assessment period.

'The Columbus, Ohio and the n'ew Haven, Connecticut regions wereselected because of the previous experience of Dr. Hamaty and be-cause of strong leadership and commitment expressed by both theConnecticut and the Ohio groups.

Observation and Data Collection_----

We believe that the most economical method for data collectioninvolves the utilization of a standardized form to be completed bythe physician himself, his staff, or a trained health accountant.If either staff or a health accountant is used for the data collec-

-7-

tion, the physician must certify that the required information isavailable in the patient's chart. Each physician may train hisown staff to collect the information o.r he may request the aid ofa health accountant.

Health Accountants-__I___

The experience of Dr. Hamaty with the Connecticut AmbulatoryCare Study of the Connecticut State Medical Society will be tappedfor the purpose of training health accountants. Personnel of theConnecticut Ambulatory Care Project will train a team of healthaccountants for the Assessment by Performance Project in theConnecticut area. A medical record librarian or a registerednurse from Ohio will spend approximately two weeks of intensivetraining in Connecticut and will then be charged with the trainingof the Ohio team of health accountants. It is expected that threehealth accountants will be needed in each region.

The heaith accountants will have two major tasks: The firstwill be to review the participants' practices and determine apractice profile for each participant. The second task will be toreview patient charts within each participant's practice and completethe testing instruments for designated patients in the most reliableand accurate way possible.

The Sample- - -

In constructing the practice profiles for each participant, thehealth accountants will review the appointment books and select atrandom 20% of the appointments made on 35 different days, selectedat random from a one year period. Patient charts will be retrievedfor the selected cases to confirm the patients' diagnoses andcomplaints. The outcome of this phase will be a profile of eachpractice in the form of a distribution of the most common problemsand diagnoses seen and treated by each participant.

'cising these profiles as guides, the sample of patients" recordsto be abstracted in each practice will be selected at random inproportion to the respective profiles of each participant. No morethaii iO0 patients will be reviewed per practice during a period ofsix months; however, the precise sample size needed will depend onthe accuracy and validity of the testing instrument. At the presenttime, it is difficult to assess the error rate that can be expectedto be generated by the use of the instrument itself. In additionto errors generated by the testing instrument, the classical erroranalysis must be considered.

Since we are dealing with a testing procedure, we must accepta certain error level. Two types of errors are usually recognizedin this type of testing activity, the false negatives and the falsepositives. For statistical purposes, it is convenient to describethese two errors as follows: a) The Type I or Alpha error, which

-8-

specifies the fraction of good performers that may be rejected aspoor ones; b) The Type II or Beta error, which specifies thefraction of cases of poor performance that may be accepted asgood performance. These two types of errors are the main factorsto be considered in the sampling of cases for review, In additionto these two parameters ito be desiqnated by the Committee), it isalso necessary to identify an acceptable level of poor performancein each practice. That is to say, we must accept the fact that ashuman beings we are all bound to commit some errors in judgmentand even in the practice of the best clinician one may detect somecases of poor performance. This may be given as a percent of casesthat one may identify as poor care within any practice withoutconcluding that the clinician is a poor performer. In addition,it is necessary to identify that critical poor performance level.

'These parameters are not readily available and it is expectedthat the first few experimentations will yield some informationthat may assist in identifying these values.

The Testing Instrument

The objectives of the instrument are:

1 . To aid in an cbjective evaluation of the quality,efficiency and outcome of the various steps inpatient care.

2 . To aid in assuring credibility for provider.

3 . To provide uniformity in the methodology so that itscredibility is maintained regardless of the practicesetting in which it is used.

In developing this testing instrument, the following are someof the specifications which must be satisfied:

1 .

2 .

It should be problem oriented.

It must have a problem-diagnosis specificity. (i.e.,it must relate to both problems and diagnoses and tothe logical relationship between them)

3 .

4.

It must be computer compatible for volume processing.

A representative sample of cases for each problem-diagnosis must be audited periodically.

5 . The instrument must be adaptable to either retrospectiveor concurrent review.

6 . It must reflect both inpatient and ambulatory care.

7 . It must identify specific outcomes in the process ofmedical care.

-9-

b” ii- should be related to tracer problem-diagnosisset.s, rather than attempting to deal with all pro-blems and diagnoses that a physician may encounter.

'i'iia main premise of the testing instrument is that medicalcare should be a iogiccl,l process consisting of several definablesteps, ea c h s t (3 p e I? d in y in a Jpr-edietable decision point. (seeattachment #I). Each decision point can then be validated by ananalysis of the data that has been accumulated prior to that point.Each decision point will then justify the process of medical carethat follows. In the diagram, each decision point is ,a locus forthe evaluation of the outcome of patient care given the informationat that ;)oint. For example, at the end of history and physical,the evaluation may yield an identified problem that will lead thephysician to order problem solving ancillary services. If, however,no problem has been identified, the physician will order problemseeking ancillary services. After the results of the anciliarvinformation is available, a higher level of problem resolution or-stablished diagnosis will result. This will lead the physicianto decide whether to prescribe therapeutic management for the treat-ment of the specific diagnosis that has been established or to dis-charge the patient because he has been found to bc in good health.Similarly, after the treatment has been instituted, an assessmentof the outcome of care may lead the physician to decide to, (1)discharge the patient because the disease has been eliminated,(2) to change the treatment regime because of an unsatisfactorytherapeutic outcome, (3) to progress to continuing care because thedisease has been adequately controlled, (4) to re-evaluate the patientbecause of inadequate problem resoiuti.on.

The decision point criteria are being perfected by the ASIPINorms Committee. These criteria will be validated by specialistsin each field. Two types of criteria can be identified:

1 . Criticai criteria: Those procedures or treatments that--____.will be present with a 100% incidence except whenspecific exclusions are noted.

2 . Relevant criteria: Those procedures and treatments thattire indi:icati?J. only when specific findings have beenobtained duririg the invcstlyative management, at whichtime rcleL7ant criteria become c:rFtical criteria.

Since physicians see problems and complaints which must beresolved into diagnoses, the Norms Committee devised an approach foridentifying most of the possible diagnoses resulting from a certaincomplaint. The attached example [Attachment #Zj is the possibleresolution of the problem "chest pain."

The Norms Committee has identified a number of common problemsand diseases in the practice of internal medicine for which theyhave developed quality care criterial. Attachment ir3 is a sampleof the form ,dsed by the Committee to develop a model treatment

-IO-

yuideiine Zor specific problems and diagnoses.

The ASIM Norms Committee, in conjunction with the Committeeon Assessmenl: by Performance, has developed a testing instrumentthat 1s eased on the previously described philosophy and criteria.(At-tachment $4)

The proposed pilot experiment will reveal the feasibility ofusing this testing instrument in ciinical practice including thel.evel of physician satisfaction.

Evaluation of the Assessment Data

T h e health accountants will be provided templates of critical-.---- -criteria for specific problems and diagnoses. These templateswill be in the form of an overlay to fit over the data collectionform. They will be trained to retrieve from the patient chartthe specific items required by each template. Fi.rst evaluation willir:vo!ve a simple coun t of all critical criteria to assure 100%compliance. Further analysis must evaluate the appropriateness ofdecisions as related to previously collected Information. Finalanaiysis will be made of outcomes predicted and :~hiev.z:~.

Data Processins Key_uirements

It is planned to do most of the processing of the data from thedemonstration project manually, not involving a computer. This isnecessary to establish ail the evaluation routines and identify alli;ossible pitfalls. It is, however, our plan to engage a computerurocessing specialist to advise on hardware and software require-ments for large processing of this information. Assuming that thedemonstration will result in a feasible approach to assessment byperformance, it may be necessary to use a computer for the screeningof a large volume of cases that must be assessed during everytesting period. It is expected that in full operation the systemmay be required to review approximately 313,000 patient records per\ i ‘.' : i ,, . The design of the testing instrument and the evaluation routines-.:. :< i ompatihie with computer processing and it is therefore expectedT blat this task may be accomplished efficiently by a computer.

Pr~opsed 'Time-table- --~ ----~ _- ---

-:‘ne pilo t experiment for the testing instruments began as:;c:h~du! eci 012 June 15 , 19 7 4 . Evaliratlon ar,d modifications of thistest i.::,? instrument, along with completion of the critical criteria:~ 1-s ts have been completed. Training of health accountants couldbe accomplished within six weeks. We are hoE;inq to inj-tiate thetwo demonstration programs by Jcinuary 1975, proceeding with coll.ectionof data and evaluation unti L Sune 1975. Evaluation of the totalprogram and preparation of the final reports should be completed byDecember 31, 2975.

-11-

Cost Controi

The project will closely monitor the costs of the assessment;JTocess * Research costs will be separated from the costs ofdsses:xerit efforts. It is expected that as a result of this carefulxost analysis it may be possible to determine the cost per assessedp:lysiciari. This information is needed for the establishment of afuture charge per assessment. This may be the best indication forthe feasibility and practicality of the proposed assessment mechanismbecause it will show whether or not such an assessment program canbecome self-sufficient,

If this proposed research demonstrates that the assessmentprocedure is feasible, applicants may be charged a fee for recer-tification, It is expected that these fees wil.1 cover the costsCJf the i)rogram, making it independent and self-sufficient.

Consultation.._---__

Consultation from recognized authorities in the field ofmedrcal care quaiity evaluation will be in the final developmentalphase of the project and later in confirmation oE the validi-tyof resuits. The academic expertise of such possible consultantsas Drs. John Williamson, Beverly Paine or Paul Sanazaro will beparticularly valuable in complementing the clinical experience ofthe Committee.

Identified

ANC1LLARY

0 D:sease

::ontrol.l.edeclslon

i------/

i

Disease

JControlled-- Decision

Point--&.---

I+

t

iLong Term

I Con I: liluing C a r e&-- _.. --..-.- DECISION FLOW DIAGRAM

Eliminated

I

/ Attach. 3

Cfforr theRt psi” ;i ;j j! Dear+ ‘i__---__ I / - -I!

- -‘i /I

-15- Attach. 3

ISCHEMIC HEART DISEASE, ANGINA PECTORIS

Tests & Procedures:-~

ECGPost prandiai blood sugaror glucose tolerance test

Lipid profile

Stress test

Halter monitor

X-raJs:

Chest x-ray Critical, no exceptionsCoronary angiography Relevant:

Critical, no exceptions

Critical, no exceptionsRelevant:

1) exclude if patient less than 60years old

Relevant:1) for diagnosis if resting ECG is

normal2) for prognosis or exercise

prescriptionsRelevant:

1) suspicion of dysrhythmia

1) for consideration of surgery2) suspicion of ischemic heart disease

in absence of other objectivefindings

Management:----.

Nitroglycerine or analogueWeight controlBeta adrenergic blocking agents

Exercise program

Critical, no exceptionsRelevant, if obesity is presentRelevant:

1) uncontrolled angina pectoris in theabsence of congestive heartfailure or asthma

Relevant, after multistage exercisetesting has established tolerance

-16-Attdch. 4

*,.

I

,

, ,