Measuring Outcomes

Geoffrey T. MillerAssociate Director, Research and Curriculum Development 

Division of Pehospital and Emergency Healthcare

Gordon Center for Research in Medical Education

University of Miami Miller School of Medicine

Session aims

• Discuss the importance of outcomes evaluation and challenges to traditional assessments 

• Discuss the importance of validity, reliability and feasibility as it relates to assessment

• Discuss types of assessments and their application in healthcare education

A little terminology…

• Assessment and evaluation are often used interchangeably

• However for our purposes…– Assessment = learner outcomes

– Evaluation = course/program outcomes

Why is assessment important?

Because… assessment:

• “Drives learning”

• Allows measures of individual and programmatic progress

• Fundamental to outcomes‐ or competency‐based education

• Assures public that providers are competent

• Credentialing, privileging, licensure, board certification – high stakes for practitioner and patient/society– All involve assessment of competence

Formula for the effective use of simulation

TrainingResources

TrainedEducators

CurricularInstitutionalizationX X =

EffectiveSimulation‐

basedHealthcareEducation

Issenberg, SB. The Scope of Simulation‐based Healthcare Education. Simulation in Healthcare. 2006.

Formula for effective outcomes measurement

DefinedOutcomes

Instruments& TrainedEvaluators

AppropriateSimulatorX X =

EffectiveOutcomes 

Measurement

What are some challenges totraditional methods of assessment

for healthcare providers?

Challenges in traditional assessments

• Ethical issues: “using” real pts (substitutes)– Invasive procedures (patient safety)

– Sensitive tasks (cultural concerns, pt modesty)

– Problems using cadaveric tissue models

– Animal welfare issues

Challenges in traditional assessments

• Real patients for evaluation of physical exam skills

• Feasibility issues for large‐scale examinations

• Standardized, perceived fairness issues in high‐stakes settings

• Standardized patients (SPs) improve reliability, but validity issues exist: cannot mimic many physical findings

Challenges in traditional assessments

• Wide range of clinical problems, including rare and critical events

• Availability

• Cost

• Reliability, validity, feasibility

Developing outcome measurements

“Any road will get you there, when you don’t know where you are going”

Curricula development 

• Analysis– Define expected outcomes

• Design 

• Development

• Implementation 

• Evaluation

Defining outcomes

• Learners are more likely to achieve competency and mastery of skills if the outcomes are well defined and appropriate for the level of skill training

• Define  clear benchmarks for learners to achieve

• Plain goals with tangible, measurable objectives

• Start with the end‐goal in mind and the assessment metrics, then the content will begin to develop itself

Curricula/assessment process

CurricularDevelopment

‐DefineOutcomes

Teaching andLearning

+/‐ Refinement Assessment andEvaluation

Use of assessments in healthcare simulation

InformationInformation

DemonstrationDemonstration

PracticePractice

Rosen, MA et al. Measuring Team Performance in Simulation‐Based Training: Adopting Best Practices for Healthcare.Simulation in  Healthcare 3:2008;33–41.

FeedbackFeedback RemediationRemediation

MeasurementMeasurement DiagnosisDiagnosis

Preparing assessments

• What should be assessed?– Every aspect of curriculum considered essentialand/or has significant designated teaching time

– Should be consistent with learning outcomes that are established as the competencies students should master/perform at a given phase of study 

Blueprinting

Global Global ObjectiveObjective

Recognize a potential terrorist incident and initiate incident operations

UM-ERTModule Obj.

2.3 Recognize and describe scene hazards and appropriate personal protective measures

FloridaFloridaObjective(sObjective(s))

Tier 1: I (L), III (D), (F), (N), IV (J), V (A), (D), VI (B)

LearningLearningOpportunityOpportunity

LectureLecture TabletopTabletopVideoVideo

ExerciseExerciseSkillSkill OSCEOSCE

X X X XAssessment

PrePreMCQMCQ

PostPostMCQMCQ

SkillSkill OSCEOSCE

5, 23 6, 19, 20 X

Clinical competence and performance

• “Competent performance” = requires acquisition of basic knowledge, skills & attitudes

• Competence = – Application of specific KSAs

• Performance = – “Translation of competence into action”

“Can they do it? Do they do it?”

Possible outcome competencies

• Patient care• Medical knowledge• Practice‐based learning and improvement

• Interpersonal and communication skills

• Professionalism • Systems‐Based Practice 

Knowledge

SkillsAttitudes

Knowledge competencies 

KnowledgeCognitive knowledge•(factual) Recall•Comprehension•Application•Analysis•Synthesis•Evaluation

Knowledge

Skills

Skills•Communication•Physical Exam•Procedures•Informatics•Self Learning•Time Management•Problem Solving

Skill competencies

Knowledge

Skills

Attitudes• Behavior• Teamwork• Professionalism• Key Personal Qualities

• MotivationAttitudes

Attitude competencies

Continuous process

Knowledge

SkillsAttitudes

Relating Miller’s pyramid of competence to learning and assessment

Miller’s Pyramid of Competence

Miller GE. The Assessment of Clinical Skills / Competence / Performance, Academic Medicine, 65:9, S63-S67.

Does

Shows

Knows How

Knows

LearningOpportunity• Reading /

Independent Study

• Lecture• Computer-based• Colleagues /

Peers

Teaching and Learning “Knows”

Does

Shows

Knows How

Knows

Assessment of “Knows”

Factual TestsDoes

Shows

Knows How

Knows

The Tools of “Knows”

• Multiple Choice Questions (MCQs)• Short Answer• True / False• Matching (extended)• Constructed Response Questions

Example - MCQ

Information Input (Facts)

Information Input (Facts)

Factual Output (Answers)Factual Output (Answers)

“Wheezes are continuous, musical, whistling sounds during difficult breathing such as in asthma, croup and other respiratory disorders.”FACT

Q. Whistling sounds associated with an asthmatic patient are called?

A. Rales B. RhonchiC. Wheezes D. Vesicular ANSWER

Learning Opportunity

Assessment

Computer-based model

Choose the best description of the patient’s finding:

A. MyoclonusB. Partial SeizureC. TicD. FasciculationsE. Tremor

Click on picture to play video

LearningOpportunity• Problem-based Ex.• Tabletop Exercises• Direct Observation• Mentors

Teaching and Learning - “Knows How”

Does

Shows

Knows How

Knows

Clinical ContextBased Tests

Assessment of “Knows how”

Does

Shows

Knows How

Knows

• Multiple-choice question

• Essay• Short answer• Oral interview

The Tools of “Knows How”

Example – Clinical Context MCQ

• 64-year-old man• No past medical Hx• 1 week of intermittent

– Headache– Double vision

• R pupil dilated

Which of the following is most likely the patients problem?

A. MigraineB. Myasthenia gravisC. Multiple SclerosisD. Ischemic StrokeE. Cerebral aneurysm

LearningOpportunity• Skill-based

Exercises•Repetitive practice

• Small Group• Role Playing

Teaching and Learning - “Shows”

Does

Shows

Knows How

Knows

Assessment of “Shows”

PerformancePerformanceAssessmentAssessment

Does

Shows

Knows How

Knows

• Objective Structured Clinical Examination (OSCE)

• Standardized Patient-based

The Tools of “Shows”

Variables in Clinical Assessment

ClinicalAssessment

Examiner

Patient

Student

Control as many variables as possible

LearningOpportunity• Experience

Teaching and Learning - “Does”

Does

Shows

Knows How

Knows

Assessment of “Does”

PerformanceAssessment

Does

Shows

Knows How

Knows

• Undercover / Stealth / Incognito Standardized Patient-based

• Video• Portfolio• Service ratings (customer satisfaction)

The Tools of “Does”

Influences on clinical performance

Does

Performance

CompetenceSystem related Individual related

Cambridge Model for delineating performance and competence

Rethans JJ, et al. The relationship between competence and performance: implications for assessing practice performance, Medical Education, 36:901-909.

Assessments types

• Choose the appropriate assessment method:– Formative

– Summative

– Self

– Peer 

Assessment• Formative Assessment

– Lower stakes 

– One of several, over time of course or program

– May be evaluative, diagnostic, or prescriptive

– Often results in remediation or progression to next level

• Summative Assessment– Higher stakes 

– Generally final of course or program 

– Primary purpose is performance measurement

– Often results in a “Go, No‐Go” outcome

Formative assessment example

Assessments ‐ self

• Encourages responsibility for the learning process, fosters skills in making judgments as to whether work is of an acceptable standard – it improves performance.

• Most forms of assessment can be adapted to a self‐assessment format (MCQs, OSCEs, and short answers)

• Students must be aware of standards required for competent performance.

Individual self‐learning and assessment

Assessments ‐ peer

• Enables learners to hone their skills in their ability to work with others and professional insight 

• Enables faculty to obtain a view of students they do not see 

• An important part of peer assessment is for students to justify the marks they award to others

• Justification can also be used as a component when faculty evaluates attitudes and professionalism. 

Assessments ‐ standard setting

• Should be set to determine competence

• Enables certification to be documented, accountable and defensible

• Appropriately set standards for an assessment will pass those students who are truly competent• Standards should not be two low (false positives) to pass those who are incompetent, nor too high (false negative) to fail those who are competent.

Assessments ‐ standard setting

• Those responsible in setting standards must also have a direct role in teaching students at the level being examined and assist in providing examination material

Assessments ‐ standard setting

• Standards should be set around a core curriculum that includes the knowledge, skills and attitudes required of all students

• When setting a standard the following should be considered:– What is assessed must reflect the core curriculum– Students should be expected to reach a high standard in the core components of the curriculum (For instance an 80‐90% pass mark of for the important core and 60‐80% for the less important aspects.)

– Students should be required to demonstrate mastery of the core in one phase of the curriculum before moving on to the next part of the curriculum 

Choosing appropriate assessment methods

• When choosing the assessment instrument, the following should be answered:– Is it valid

– Is it reliable

– Is it feasible

Assessments ‐ validity

• Are we measuring what we are supposed to be measuring

• Use the appropriate instrument for the knowledge, skill, or attitude you are testing 

• The major types of validity should be considered (content, predictive, and face)

Assessments ‐ reliability

• Does the test consistently measure what it is supposed to be measuring

• Types of reliability:– Inter‐rater (consistency over raters)

– Test‐retest (consistency over time)

– Internal consistency (over different items/forms)

Both archers are equally reliable.

Archer 1 hits bulls eye every time.

Archer 2 hits outer ring in same spot every time.

Reliability as Consistency

Inter‐rater Reliability

• Multiple judges code independently using the same criteria 

• Reliability = raters code same observations into same classification

• Examples• Medical record reviews

• Clinical skills

• Oral examinations

Factors Influencing Reliability

• Test length• Longer tests give more reliable scores

• Group homogeneity• The more heterogeneous the group, the higher the reliability

• Objectivity of scoring• The more objective the scoring, the higher the reliability

Both archers are equally reliableValidity = quality of archer’s hits

Archer 1 hits bulls eye every time

Archer 2 hits outer ring in same spot every time

Validity is accuracy

Reliable and Valid

Reliable, not valid

Not reliable, not valid

Reliability and Validity

Improving reliability and validity

• Base assessment on outcome/objectives‐ event triggers‐ observable behavior‐ behavioral rating‐assess against competence

• Define:– Low‐medium‐high performance

– Use of rubric or rating metric

– Use (video) training examples of performance

– Employ quality assurance/improvement system

Assessments ‐ feasibility

• Is the administration and taking of the assessment instrument feasible in terms of time and resources

• The following questions should be considered: – How long will it take to construct the instrument?– How much time will be involved with the scoring process?– Will it be relatively easy to interpret the scores and produce the results?

– Is it practical in terms of organization?– Can quality feedback result from the instrument?– Will the instrument indicate to the students the important elements within the course?

– Will the assessment have a beneficial effect in terms of student motivation, good study habits and positive career aspirations?

Practicality

• Number of students to be assessed

• Time available for the assessment 

• Number of staff available

• Resources/equipment available

• Special accommodations

Assessment instruments

Assessments ‐ instruments

• Be aware of the types of assessment instruments available as well as the advantages and disadvantages of each

• It is important, if feasible, to use more than one assessment instrument and more than one assessor when looking at skills and attitudes

Assessments – knowledge instruments

• Objective tests (short answer, true/false, matching, multiple choice)

• Objective Structured Clinical Evaluations (OSCEs)

• Constructed response questions

• Rating scales (used on clerkships)

Assessments – skill instruments

• Objective tests (Simulation based)

• OSCEs

• Constructed response questions

• Critical reading papers (interpreting literature)

• Checklists

• Rating Scales

• Portfolios (self‐evaluation, time management)

Assessments – attitude instruments

• Portfolios

• Essays / Modified essay questions

• OSCEs

• Checklists

• Rating scales

• Patient management problems

• Short/long case assessments 

Assessment Metrics

• Procedural or Check List assessment

• Global Rating assessment

Assessment Metrics

• Procedural or Check List assessment

BCLS Y N

Open Airway

Check Breathing

BCLS Y NOpen Airway(< 5 sec of LOC)

Check Breathing(< 5 sec of Airway)

BCLS Y NOpen Airway

Check Breathing

A

Rating Score +1 -1 0 *Assist

Assessment Metrics

• Global Rating assessmentCode Blue P F

CPR and

ACLS

Code BlueCPR<1(low) - 5(Hi)> points

ACLS<1(low)- 5(Hi)> points

Code Blue H M

CPR

ACLS

L

Rating Score +1 0 -1

Pts.

Review

• Assessment drives learning

• Clearly define the desired outcome, ensure that it can be measured

• Consider the “threats” to the effectiveness of the measurement

• Feedback to individual candidates

• Feedback to training programs

Questions and discussion