assessment of students competence in health professionals education

1

Assessment of Students Competence in Health Professionals Education

Professor Hossam HamdyUniversity of Sharjah

2

Five Basic Questions

1. WHY?2. WHAT?3. HOW?4. WHEN?5. WHOM?

3

Purpose of Assessment

• Providing feedback

• Guiding student learning

• Insuring medical competence

4

WHY?

• Consider assessment as a diagnostic test for the disease “medical incompetence”

• False negative = failing a competent student

• False positive = passing a student who is incompetent

5

TRUTH

CompetentIncompetent

PassT.PF.P

FailF.NT.N

TEST

6

Professional Competence

The Habitual and Judicious use of:

• Communication

• Knowledge

• Technical Skills

• Evidence-based decision-making

• Emotions

• Values and reflection to improve the health of the individual patient and the community

7

WHAT Should Be Assessed?

ACGME Competencies- Patient care- Knowledge- Practice-based learning and

improvement - Interpersonal and communication skills- Ethics & Professionalism- System-based practice

8

Blueprinting

Test content matches objectives

• Assessment matches competencies learnt

• Assessment matches format of learning, “PBL”

9

Aim of Blueprinting

Reduce:A. Construct under-representation:

Biased samplingFew items covering a topic e.g. OSCE

B. Construct irrelevant variance:Flawed items formatInadequate sampling of student Behaviour.

10

Blueprinting

• Identify clinical problems that examinees should be able to handle.

• For each problem, define the clinical tasks in which the examinee is expected to be competent.

• Define level of performance / resolution

11

Blueprinting

• Prepare a blueprint to guide the selection of problems to be included in the assessment.

• Let the characteristics of the clinical problems define the tasks to be included in the assessment.

A Simple BlueprintCompetenceCategories

CVSRS…GI..…MSSEndo/Metab

Haem/ Oncology

.…

History Taking

Physical Exam

Tests and Procedures

…..

Management

Communication and Patient Education

Pre-Clerkship Phase Theme B.Sc. Exam Blueprint 2003-2004

UNITS

Inheritance,

growth &

development;

degeneration & aging

Nurture,

environment

& nutriti

on

Structure:

function;

positional

relationship

Control, regulation &

communication within the body,

Homeostasis: metabolie, circulatory, acid/base & electrolyte

Response and

adaptation

systems &

mechanisms

Injury, inflammati

on & infection;

loss, repair and substitutio

n; neoplasia

Immunological

& defense mechan

isms

Investigative

procedure &

interpretation

Intervention

prevention &

management

principles

Human behavior, ethical & medico legal

principles

1234567891011

UNIT VIII:Integrated Multi-System

UNIT VII:

Nervous System, Special Sense & Human Behavior

UNIT VI:

Integumentary & Muscular-Skeletal

UNIT V:

Hematopoietic & Immune Systems

UNIT IV:

Endocrine, Metabolism & Reproductive System

UNIT III:

GIT & Renal System

UNIT II:

Respiratory & CVS

UNIT I:

Concepts & Principles & Community Health

14

15

HOW?Choice of Method

U = W R x W V x W E x W A x W C 1 2 3 4 5

U = Utility R = ReliabilityV = Validity E = Educational Impact A = Acceptability C = Cost/Available

Resources W = Weight

16

Reliability

• Reliability = reproducibility

• Inter-rater: Multiple examiners across different cases

• Inter-case: “multi use format (OSCE)”

17

Reliability

Reliability is matter of sampling

• Across content

• Across Tasks

• Across Problems

18

Domain of Interest

Test Sample

Test Sample

19

“Reliability”Practical Implications

• Sample adequately across content

• Clinical competencies are inconsistent across different tasks

• Test length is crucial if high-stakes decisions are required

• Use as many examiners as possible

20

Validity

• Validity = measures what it intends to measure

• Select appropriate test formats for the competencies to be tested.

This action invariably results in a composite examination.

21

A Simple Model of Competence

Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

Knows

Shows how

Knows how

Does

Pro

fess

iona

l aut

hent

icity

Pro

fess

iona

l aut

hent

icity

Written, Oral orComputer based assessment

Performance or hands on assessment

22

Validity Climbing the Pyramid . . .

Knows

Shows how

Knows how

Does

Knows Factual tests: MCQ, essay type, oral…..

Shows howPerformance assessment in vitro:OSCE, SP-based test…..

DoesPerformance assessment in vivo: Masked SPs, Video, Audits…..

Knows how(Clinical) Context based tests:MCQ, essay type, oral…..

23

Educational impact

24

WHOM

• Self

• Peers

• Supervisors

• 360º “Different Stakeholders”

25

Written Assessment Methods

Stimulus Format

Response Format

Context-free questions

Context-rich questions

EMQ, KF, A & R “MCQ”

Open ended questions types

SAQ, MEQ

MCQ types

26

The MCQ’s Family

Alive SickDead

One Best Answer key Features Extended Matching

One single answer True & False x Type

Long PMPs

K TypeC Type

27

Written Assessment of Clinical Competence

• MCQs 1) Best Single Response• 2) Extended Matching

• Key Features Questions

• Constructed Response Questions

28

Components of an A-type Question

A 35-year old man has had a stomach ache all afternoon. He ate the following lunch: two big McDonalds hamburgers, an ice cream shake, large fries.

Which is the most likely diagnosis? Lead-in

Vignette, Scenario, Or Stem

29

A. Abdominal aneurysmB. AppendicitisC. Bowel obstructionD. CholecystitisE. Colon cancer F. PancreatitisG. Too much lunch

Options A, B, C, D, E & F are

distracters

Option G is the key

Components of an A-type Question

Sample Good Question & ‘Cover The Options’ Rule

A 32-yo man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 100 F, BP 130/80, pulse 94, respirations 42 and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Which of the following is the most likely diagnosis?

A. Acute disseminated encephalomyelitisB. Guillain-Barré syndromeC. Myasthenia gravisD. PoliomyelitisE. Polymyositis

31

Sample Good Question & ‘Cover The Options’ Rule

A 32-yo man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 100 F, BP 130/80, pulse 94, respirations 42 and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Which of the following is the most likely diagnosis?

A. Acute disseminated encephalomyelitisB. Guillain-Barré syndromeC. Myasthenia gravisD. PoliomyelitisE. Polymyositis

32

D C A E B

Least MostCorrect CorrectAnswer Answer

Note: Options must be homogeneous (eg, all diagnoses, all muscles). You must be able to rank-order the options on the same dimension.

Diagram of Rank-Ordered Options

for a Good One-Best-Answer Item

33

A. It occurs frequently in women.

B. It is seldom associated with acute pain in a joint.C. It may be associated with a finding of chondrocalcinosis.D. It is clearly hereditary in most cases.E. It responds well to treatment with allopurinol.

Sample Flawed Question & ‘Cover The Options’ Rule

Which of the following is true about pseudogout?

34

D InheritanceRx E

Gender A

AssociationsFalse True

B C

Sample Flawed One-Best-Answer Item

Which of the following is true about pseudogout?A. It occurs frequently in women. B. It is seldom associated with acute pain in a joint.C. It may be associated with chondrocalcinosis.D. It is clearly hereditary in most cases.E. It responds well to treatment with allopurinol.

35

Test application of Knowledge not just recall of isolated facts

Basic Science Recall of Isolated Facts Item:

What area is supplied with blood by the posterior inferior cerebellar artery?

Basic Science Application of Knowledge Item:

A 62-year-old man develops left-sided limb ataxia, Horner's syndrome, nystagmus, and loss of appreciation of facial pain and temperature sensations. What artery is most likely to be occluded?

36

“Old” Style Biochemistry QuestionTesting Recall of an Isolated Fact

Acute intermittent porphyria is the result of a defect in the biosynthetic pathway for

A. collagen

B. corticosteroid

C. fatty acid

D. glucose

E. heme

F. thyroxin E

37

An otherwise healthy 33-yo man has mild weakness & occasional episodes of steady, severe abdominal pain with some cramping but no diarrhea. His aunt and a cousin have had similar episodes. During an episode, his abdomen is distended, and bowel sounds are decreased. Neurological exam shows mild weakness in the upper arms. A defect in which of the following biosynthetic pathway is the most likely cause of the patient’s problem?

A. collagenB. corticosteroidC. fatty acidD. glucoseE. hemeF. thyroxine

Application of Knowledge Item: Biochemistry/Genetics

E

38

Item Assessing Recall of an Isolated Fact

What is arterial blood gas analysis most likely to show in patients with cardiogenic shock?

A. Hypoxemia with normal pHB. Metabolic acidosisC. Metabolic alkalosisD. Respiratory acidosisE. Respiratory alkalosis

39

A 74-year-old woman is brought to the emergency department because of crushing chest pain. She is restless, confused, and diaphoretic…..

Which of the following arterial blood gas analysis results is most likely?

A. Hypoxemia with normal pHB. Metabolic acidosisC. Metabolic alkalosisD. Respiratory acidosisE. Respiratory alkalosis

Alternate Item Assessing

Application of Knowledge: Diagnosis

40

Summary

• Stimulus – Context rich “Patient Vignette”

• One best answer, not one single correct option

• Cover the options rule

41

Summary

Avoid:

• All / non of the above

• Except

• Negatively worded items. “Which one of the following statements is incorrect.”

42

Extended Matching R-Set

Options:A. Left anterior cerebral artery E. Right anterior cerebral

arteryB. Left posterior cerebral artery F. Right posterior cerebral arteryC. Left middle cerebral artery G. Right middle cerebral artery Left lenticulostriate arteries H. Right lenticulostriate arteries

Lead-in:

For each patient with neurologic abnormalities, select the artery that is most likely to be involved.

Vignette: “Items”

1. A 72-year-old right-handed man has weakness and hyperreflexia of the right lower limb, an extensor plantar response on the right, normal strength of the right arm, and normal facial movements.

Ans: A

2. A 68-year-old right-handed man has right spastic hemiparesis, an extensor plantar response on the right, and paralysis of the lower two-thirds of his face on the right. His speech is fluent, and he has normal comprehension of verbal and written commands.

Ans: G

Theme: Neuroanatomy: Blood Supply of the BrainTheme: Neuroanatomy: Blood Supply of the Brain

44

Theme: Abdominal Pain

A. Abdominal aneurysm K. Kidney stoneB. Appendicitis L. Mesenteric adenitisC. Bowel obstruction M. Mesenteric artery thrombosisD. Cholecystitis N. Ovarian cyst - rupturedE. Colon cancer O. PancreatitisF. Constipation P. Pelvic inflammatory disease G. Diverticulitis Q. Peptic ulcer diseaseH. Ectopic pregnancy – ruptured R. Perforated peptic ulcerI. Endometriosis S. PyelonephritisJ. Hernia T. Torsion

For each patient with abdominal pain, select the most likely diagnosis.

Options and Lead-in: DiagnosisOptions and Lead-in: Diagnosis

1. A 25-year-old woman has sudden onset of persistent right lower abdominal pain that is increasing in severity. She has nausea without vomiting. She had a normal bowel movement just before onset of pain. Examination shows exquisite deep tenderness to palpation in right lower abdomen with guarding but no rebound; bowel sounds are present. Pelvic examination shows a 7-cm, exquisitely tender right sided mass. Hematocrit is 32%. WBC is 18,000/mm3. Serum amylase activity is within normal limits. Test of the stool for occult blood is negative.

Ans: B

2. An 84-year-old man in a nursing home has increasing poorly localized lower abdominal pain recurring every 3-4 hours over the past 3 days. He has no nausea or vomiting; the last bowel movement was not recorded. Examination shows a soft abdomen with a palpable, slightly tender, lower left abdominal mass. Hematocrit is 28%. WBC is 10,000/mm3. Serum amylase activity is within normal limits. Test of the stool for occult blood is positive.

Ans: E

Sample Diagnosis Items: Abdominal Pain Sample Diagnosis Items: Abdominal Pain

46

Key – Feature Questions

1. Any single problem is not equivalent to any other problem in assessing C.P.S. and C.D.M. Problem solving skills are specific to the case or problem encountered.

2. C.P.S. and C.D.M. need to be measured over many problems before general inferences could be made.

3. Clinical problem solving and decision making skills are dependent on the effective manipulation of those few elements of the problem that are critical to its resolution – “key features”

4. Assessment should focus on the problems key features “Bordage and Page, 1991”

47

Question Formats

• Write-in “WI” - Diagnosis and Treatment

• Short-Menu “SM” - History, Physical “2-45 Options” Examination and

Laboratory

Case 2Paul, a 56 year old man, consults you in the outpatient clinic because of pain in his left leg which began two days ago and has been progressively worse. He states his leg is tender below the knee and swollen around the ankle. He has never had similar problems. His other leg is fine.

Question 1What diagnosis would you consider at this time? List up to three?

Question 2With respect to your diagnosis, what elements of his history would you particularly want to elicit?

1. Activity at onset of symptoms 11. Hormonetherapy 21. Previous back problems2. Alcohol intake 12. Impotence 22. Previous neoplasia3. Allergies 13. Intermittent claudication 23. Previous urinary tract infection4. Angina pectoris 14. Low back pain 24. Recent dental procedure5. Anti-inflammatory therapy 15. Nocturia 25. Recent immobilization6. Cigarette smoking 16. Palpitations 26. Recent sore throat7. Colour of stools 17. Parenthesis 27. Recent surgery8. Cough 18. Paro .. Noctural dyspnea 28. Recent work environment9. Headache 19. Polydipia 29. Wounds on foot10. Hemetemesis 20. Previous knee problems 30. Wounds on hand

49

Constructed Response Questions

• A written question

• A statement, journal article or report

• A problem scenario: clinical, basic science or community

• Photographic or other investigative data

The response generated by the student may be:

50

Constructed Response Questions

• A very short answer – one or several words

• A short answer – notes or lists

• A long answer – essay, dissertation, referral letter or report

• A physical task – a project or another complex performance such as cardiopulmonary resuscitation or conducting an investigation

A 72-year-old, right-handed man presents with a 6-month

history of increasing tremor of his right hand that causes him

severe embarrassment, such that he avoids going out. On

examination, the tremor is most marked at rest and

decreases on maintaining a posture and during movement.

There is no intention tremor.

Example questions:

• The name of this type of tremor• The anatomical site of the pathological lesion• Drugs causing this phenomenon as a side-effect• Additional signs to look for on physical examination• How to treat the condition

52

• Uninterrupted and observed 30 – 45 minute interview and examination on a selected patient

• Candidates present only their findings to the examiner

The Traditional Long Case Examinations (LCE)

53

• It assesses the integrated interaction between the student/doctor and the real patient

• Authenticity high

• More valid than the task given in an OSCE

• Little is known about the construct validity and consequential validity “Educational Impact”

)LCE (Validity

54

• Poor intercase reliability

• Content specificity is the most crucial issue in the assessment of clinical competence

• Broad sampling across cases is essential “Multiple Biopsies”

• Logistics will be difficult

)LCE (Reliability

55

• Three hour examination of four 45 minutes observed long cases

• Real patient selected from a predetermined blue print of common diseases

• Check List – clinical encounter domains

• Reliability 0.8 (Cronbach’s alpha)

(Hamdy et al, Med Educ, 2003)

Direct Observation Clinical Encounter Examination

The DOCEE