The The ValueValue in E in Evaluvaluationation

Erica FriedmanErica Friedman

Assistant Dean of EvaluationAssistant Dean of Evaluation

MSSMMSSM

FORMATIVE SUMMATIVE

Written ObservedObserved

MCQ

MEQ

Application

Essay

PracticalPractical

Note review

SP

OSCEOSCE

OralOral

Longitudinal Clinical Longitudinal Clinical ObservationObservation

KNOWS

KNOWS HOW

SHOWS HOW

DOES

Session GoalsSession Goals Understand the purposes of assessmentUnderstand the purposes of assessment Understand the framework for selecting and Understand the framework for selecting and

developing assessment methodsdeveloping assessment methods Recognize the benefits and limitations of different Recognize the benefits and limitations of different

methods of assessmentmethods of assessment

Conference ObjectivesConference Objectives Review the goals and objectives for your course or Review the goals and objectives for your course or

clerkship in the context of assessmentclerkship in the context of assessment Identify the best methods of assessing your goals Identify the best methods of assessing your goals

and objectivesand objectives

Purpose of EvaluationPurpose of Evaluation To certify individual competenceTo certify individual competence To assure successful completion of goals/objectivesTo assure successful completion of goals/objectives To provide feedbackTo provide feedback

To studentsTo students To faculty, course and clerkship directorsTo faculty, course and clerkship directors

As a statement of values (what is most critical to As a statement of values (what is most critical to learn)learn)

For Program Evaluation- evaluation of an aggregate, For Program Evaluation- evaluation of an aggregate, not an individual (ex. average ability of students to not an individual (ex. average ability of students to perform a focused history and physical)perform a focused history and physical)

Consequences of evaluationConsequences of evaluation

Steering effect- exams “drive the learning”- Steering effect- exams “drive the learning”- students study/learn for the examstudents study/learn for the exam

Impetus for change (feedback from Impetus for change (feedback from students, Executive Curriculum, LCME)students, Executive Curriculum, LCME)

Definitions- ReliabilityDefinitions- Reliability

The consistency of a measurement over time or by The consistency of a measurement over time or by different observers- ( ex. a thermometer always different observers- ( ex. a thermometer always reads 98 degrees C when placed in boiling, distilled reads 98 degrees C when placed in boiling, distilled water at sea level)water at sea level)

The proportion of variability in a score due to the The proportion of variability in a score due to the true difference between subjects (ex. The difference true difference between subjects (ex. The difference between Greenwich time and the time on your between Greenwich time and the time on your watch)- watch)-

Inter-rater reliability (correlation between scores of 2 raters)Inter-rater reliability (correlation between scores of 2 raters) Internal reliability (correlation between items within an Internal reliability (correlation between items within an

exam)exam)

Definitions-ValidityDefinitions-Validity The ability to measure what was intended (the The ability to measure what was intended (the

thermometer reading is reliable but not valid)thermometer reading is reliable but not valid) Four types- Four types-

Face/contentFace/contentCriterionCriterionConstruct/predictiveConstruct/predictiveInternalInternal

Types of validityTypes of validity Face/content- Would experts agree that it assesses what’s Face/content- Would experts agree that it assesses what’s

important?-(driver’s test mirroring actual driving situation and important?-(driver’s test mirroring actual driving situation and conditions)conditions)

Criterion- draw an inference from test scores to actual performance. Criterion- draw an inference from test scores to actual performance. Ex. if a simulated driver’s test score predicts the road test score, the Ex. if a simulated driver’s test score predicts the road test score, the simulation test is claimed to have a high degree of criterion validity.simulation test is claimed to have a high degree of criterion validity.

Construct/predictive- does it assess what it intended to assess (ex. Construct/predictive- does it assess what it intended to assess (ex. Driver’s test as a predictor of the likelihood of accidents- results of Driver’s test as a predictor of the likelihood of accidents- results of your course exam predict the student’s performance on that section your course exam predict the student’s performance on that section of Step 1)of Step 1)

Internal- do other methods assessing the same domain obtain Internal- do other methods assessing the same domain obtain similar results (similar scores from multiple SPs assessing history similar results (similar scores from multiple SPs assessing history taking skills)taking skills)

Types of Evaluations- Formative Types of Evaluations- Formative and Summative Definitions:and Summative Definitions:

Formative evaluation- provide feedback so the Formative evaluation- provide feedback so the learner can modify their learning approach- learner can modify their learning approach- “When the chef tastes the sauce, that’s formative “When the chef tastes the sauce, that’s formative evaluation”evaluation”

Summative evaluation- done to decide if a Summative evaluation- done to decide if a student has met the minimum course student has met the minimum course requirements (pass or fail)- usually judged requirements (pass or fail)- usually judged against normative standards- “when the customer against normative standards- “when the customer tastes the sauce, that’s summative evaluation”tastes the sauce, that’s summative evaluation”

Conclusions about formative Conclusions about formative assessmentsassessments

Stakes are lower (not determining passing or Stakes are lower (not determining passing or failing, so lower reliability is tolerated)failing, so lower reliability is tolerated)

Desire more information, so they may require Desire more information, so they may require multiple modalities (it is rare for one assessment multiple modalities (it is rare for one assessment method to identify all critical domains) for validity method to identify all critical domains) for validity and reliabilityand reliability

Use evaluation methods that support and reinforce Use evaluation methods that support and reinforce teaching modalities and steer students’ learningteaching modalities and steer students’ learning

May only identify deficiencies but not define how May only identify deficiencies but not define how to remediateto remediate

Conclusions about summative assessmentsConclusions about summative assessments Stakes are higher- students may pass who are Stakes are higher- students may pass who are

incompetent or may fail and require remediationincompetent or may fail and require remediation Desire high reliability (>0.8) so often require Desire high reliability (>0.8) so often require

multiple questions/problems or cases (20-30 multiple questions/problems or cases (20-30 stations/OSCE, 15-20 cases for oral presentations, stations/OSCE, 15-20 cases for oral presentations, 700 questions for an MCQ)700 questions for an MCQ)

Desire high content validity (single cases have low Desire high content validity (single cases have low content validity and are not representative)content validity and are not representative)

Desire high predictive validity (correlation with Desire high predictive validity (correlation with future performance), which is often hard to future performance), which is often hard to achieveachieve

Consider reliability, validity, benefit and cost Consider reliability, validity, benefit and cost (resources, time and $) in determining the best (resources, time and $) in determining the best assessment toolsassessment tools

How to Match Assessment to Goals How to Match Assessment to Goals and Teaching Methodsand Teaching Methods

Define the type of learning (lecture, small Define the type of learning (lecture, small group, computer module/self study, etc)group, computer module/self study, etc)

Define the domain to be assessed Define the domain to be assessed (knowledge, skill, behavior) and the level of (knowledge, skill, behavior) and the level of performance expected (knows, knows how, performance expected (knows, knows how, shows how or does)shows how or does)

Determine the type of feedback requiredDetermine the type of feedback required

Purpose of feedbackPurpose of feedback

For students: To provide a good platform to For students: To provide a good platform to support and enhance student learningsupport and enhance student learning

For faculty: To determine what works (what For faculty: To determine what works (what facilitated learning and who were facilitated learning and who were appropriate role models)appropriate role models)

For students and faculty: To determine For students and faculty: To determine areas that require improvementareas that require improvement

Types of FeedbackTypes of Feedback

QuantitativeQuantitative Total score compared to other students, providing Total score compared to other students, providing

the high, low and mean score and minimum the high, low and mean score and minimum requirement for passing graderequirement for passing grade

QualitativeQualitative Written personal feedback identifying areas of Written personal feedback identifying areas of

strength and weaknessstrength and weakness Oral feedback one on one or in a group to discuss Oral feedback one on one or in a group to discuss

the areas of deficiency to help guide further the areas of deficiency to help guide further learninglearning

Evaluation Bias-PitfallEvaluation Bias-Pitfall

Can occur with any evaluation requiring Can occur with any evaluation requiring interpretation by an individual (all methods other interpretation by an individual (all methods other than MCQ)than MCQ)

Expectation bias (halo effect)- prior knowledge or Expectation bias (halo effect)- prior knowledge or expectation of the outcome influences the ratings expectation of the outcome influences the ratings (especially a global rating)(especially a global rating)

Audience effect- a learner’s performance is Audience effect- a learner’s performance is influenced by the presence of an observer (seen influenced by the presence of an observer (seen especially with skills and behaviors)especially with skills and behaviors)

Rater traits- the training of the rater or the rater’s Rater traits- the training of the rater or the rater’s traits affect the reliability of the observationtraits affect the reliability of the observation

Types of assessment tools-Types of assessment tools-WrittenWritten

Does not require an evaluator to be present during Does not require an evaluator to be present during the assessment and can be open or closed bookthe assessment and can be open or closed book

• Multiple choice question (MCQ)Multiple choice question (MCQ)• Modified short answer essay question (MEQ)- Modified short answer essay question (MEQ)-

Patient management problem is a variation of thisPatient management problem is a variation of this• EssayEssay• Application testApplication test• Medical note/chart reviewMedical note/chart review

Types of assessment tools-Types of assessment tools-Observer Dependent InteractionObserver Dependent Interaction

Usually requires active involvement of an Usually requires active involvement of an assessor and occurs as a single eventassessor and occurs as a single event

• PracticalPractical• Medical record reviewMedical record review• Standardized Patient(s) (SP)Standardized Patient(s) (SP)• Objective Structured Clinical Examination Objective Structured Clinical Examination

(OSCE)(OSCE)• Oral examination- chart stimulated recall; triple Oral examination- chart stimulated recall; triple

jump or direct observationjump or direct observation

Types of assessment tools- Types of assessment tools- Observer Dependent Observer Dependent

Longitudinal InteractionLongitudinal Interaction Continual evaluation over timeContinual evaluation over time• Preceptor evaluation – either completion of Preceptor evaluation – either completion of

a a critical incident report or structured a a critical incident report or structured rating form based on direct observation rating form based on direct observation over timeover time

• Peer evaluationPeer evaluation• Self evaluationSelf evaluation

MCQMCQ Definition: A test composed of questions on which each stem is Definition: A test composed of questions on which each stem is

followed by several alternative answers. The examinee must select followed by several alternative answers. The examinee must select the most correct answer.the most correct answer.

Measures: Knows and Knows howMeasures: Knows and Knows how Pros: Efficient; cheap; samples large content domain (60 Pros: Efficient; cheap; samples large content domain (60

questions/hour); high reliability; easy objective scoring, direct questions/hour); high reliability; easy objective scoring, direct correlate of knowledge with expertisecorrelate of knowledge with expertise

Cons: Often a recall of facts; provides opportunity for guessing Cons: Often a recall of facts; provides opportunity for guessing (good test-taker); unrealistic; doesn’t provide information about the (good test-taker); unrealistic; doesn’t provide information about the thought process; encourages learning to recallthought process; encourages learning to recall

Suggestions: Create questions that can be answered from the stem Suggestions: Create questions that can be answered from the stem alone; avoid always, frequently, all or none; randomly assign alone; avoid always, frequently, all or none; randomly assign correct answers; can correct for guessing (penalty formula)correct answers; can correct for guessing (penalty formula)

MEQMEQ

Definition: A series of sequential questions in a linear format based Definition: A series of sequential questions in a linear format based on an initial limited amount of information. It requires immediate on an initial limited amount of information. It requires immediate short answers followed by additional information and subsequent short answers followed by additional information and subsequent questions. (patient management problem is a variation of this type)questions. (patient management problem is a variation of this type)

Measures: Knows and Knows howMeasures: Knows and Knows how Pros: Can assess problem solving, hypothesis generation and data Pros: Can assess problem solving, hypothesis generation and data

interpretationinterpretation Cons: Low inter-case reliability; less content validity; harder to Cons: Low inter-case reliability; less content validity; harder to

administer; time consuming to grade and variable inter-rater administer; time consuming to grade and variable inter-rater reliabilityreliability

Suggestions: Use directed (not open ended) questions; provide Suggestions: Use directed (not open ended) questions; provide extensive answer keyextensive answer key

Open ended essay questionOpen ended essay question Definition: Question allowing a student the freedom to Definition: Question allowing a student the freedom to

decide the topic to address and the position to take- it decide the topic to address and the position to take- it can be take homecan be take home

Measures: Knows, Knows howMeasures: Knows, Knows how Pros: Assesses ability to think (generate ideas, weigh Pros: Assesses ability to think (generate ideas, weigh

arguments, organize information, build and support arguments, organize information, build and support conclusions and communicate thoughts; high face conclusions and communicate thoughts; high face validityvalidity

Cons: Low reliability; time intensive to grade; narrow Cons: Low reliability; time intensive to grade; narrow coverage of contentcoverage of content

Suggestions: strictly define the response and the rating Suggestions: strictly define the response and the rating criteriacriteria

Application testApplication test

Definition: Open book problem solving test incorporating Definition: Open book problem solving test incorporating a variety of MCQs and MEQs. It provides a description of a variety of MCQs and MEQs. It provides a description of a problem with data. The examinee is asked to interpret a problem with data. The examinee is asked to interpret the data to solve the problem. (ex. Quiz item 3)the data to solve the problem. (ex. Quiz item 3)

Measures: Knows and knows howMeasures: Knows and knows how Pros: Assesses higher learning; good face/content Pros: Assesses higher learning; good face/content

validity; reasonable reliability; useful for formative and validity; reasonable reliability; useful for formative and summative feedbacksummative feedback

Cons: Harder to create and gradeCons: Harder to create and grade

Practical ExamPractical Exam Definition: Hands on exam to demonstrate and apply knowledge Definition: Hands on exam to demonstrate and apply knowledge

(ex. Culture and identify the bacteria on the glove of a Sinai (ex. Culture and identify the bacteria on the glove of a Sinai cafeteria worker, or performance of a history and physical on a cafeteria worker, or performance of a history and physical on a patient)patient)

Measures: Know, knows how , and ? shows how and doesMeasures: Know, knows how , and ? shows how and does Pros: Can test multiple domains, actively involves the learner (good Pros: Can test multiple domains, actively involves the learner (good

steering effect); best suited for procedural/technical skills; higher steering effect); best suited for procedural/technical skills; higher face validityface validity

Cons: Labor intensive (creation and grading); hard to identify gold Cons: Labor intensive (creation and grading); hard to identify gold standard, so subjective grading; high rate of item failure standard, so subjective grading; high rate of item failure (unanticipated problems with administration)(unanticipated problems with administration)

Suggestions: Pilot first; adequate, specific instructions and goals; Suggestions: Pilot first; adequate, specific instructions and goals; specific, defined criteria for grading and train raters; for direct specific, defined criteria for grading and train raters; for direct observation, require multiple encounters for higher reliabilityobservation, require multiple encounters for higher reliability

Medical record/note reviewMedical record/note review

Definition: Examiner reviews learner’s previously created Definition: Examiner reviews learner’s previously created document; can be randomdocument; can be random

Measures: Knows how and DoesMeasures: Knows how and Does Pros: Can review multiple records for higher reliability; Pros: Can review multiple records for higher reliability;

high face validity; less costly than oral (done without high face validity; less costly than oral (done without learner and at examiner’s convenience)learner and at examiner’s convenience)

Cons: Lower inter-rater reliability; less immediate Cons: Lower inter-rater reliability; less immediate feedback; unable to determine basis for decisionsfeedback; unable to determine basis for decisions

Suggestions: Create a template with specific ratings for Suggestions: Create a template with specific ratings for skillsskills

Standardized PatientsStandardized Patients

Definition: Simulated patient/actor trained to present Definition: Simulated patient/actor trained to present history in reliable, consistent manner and to use a history in reliable, consistent manner and to use a checklist to assess students skills and behaviorschecklist to assess students skills and behaviors

Measures: Knows, Knows how, Shows how and DoesMeasures: Knows, Knows how, Shows how and Does Pros: High face validity; can assess multiple domains; can Pros: High face validity; can assess multiple domains; can

be standardized; can give immediate feedbackbe standardized; can give immediate feedback Cons: Costly; labor intensive; must use multiple SPs for Cons: Costly; labor intensive; must use multiple SPs for

high reliabilityhigh reliability

OSCE (Objective Structured Clinical OSCE (Objective Structured Clinical Exam)Exam)

Definition: Task oriented, multi-station Definition: Task oriented, multi-station exam; stations can be 5-30 minutes and exam; stations can be 5-30 minutes and require written answers or observation (ex. require written answers or observation (ex. Take orthostatic VS; perform a cardiac exam; Take orthostatic VS; perform a cardiac exam; smoking cessation counseling; read and smoking cessation counseling; read and interpret CXR or EKG results; communicate interpret CXR or EKG results; communicate lab results and advise a patientlab results and advise a patient

Measures: Knows, Knows how, Shows how Measures: Knows, Knows how, Shows how and Does and Does

OSCE (Objective Structured Clinical OSCE (Objective Structured Clinical Exam)Exam)

Pros: Assesses clinical competency; tests a wide Pros: Assesses clinical competency; tests a wide range of knowledge, skills and behaviors; can give range of knowledge, skills and behaviors; can give immediate feedback; good test-retest reliability; immediate feedback; good test-retest reliability; good content and construct validity; less patient good content and construct validity; less patient and examiner variability than with direct and examiner variability than with direct observationobservation

Cons: Costly (manpower and $); case specific; Cons: Costly (manpower and $); case specific; requires > 20 stations for internal consistency; requires > 20 stations for internal consistency; weaker criterion validityweaker criterion validity

Oral ExaminationOral Examination

Definition: Method of evaluating a Definition: Method of evaluating a learner’s knowledge by asking a series of learner’s knowledge by asking a series of questions. The process is open ended with questions. The process is open ended with the examiner directing the questions. –(ex. the examiner directing the questions. –(ex. chart stimulated patient recall or a triple chart stimulated patient recall or a triple jump)jump)

Measures: Knows, Knows how, sometimes Measures: Knows, Knows how, sometimes Shows how and doesShows how and does

Oral ExamOral Exam

Pros: Can measure clinical judgement, interpersonal Pros: Can measure clinical judgement, interpersonal skills (communication) and behavior; high face skills (communication) and behavior; high face validity; flexible; can provide direct feedbackvalidity; flexible; can provide direct feedback

Cons: Poor inter-rater reliability (dove vs hawk and Cons: Poor inter-rater reliability (dove vs hawk and observer bias); content specific so low reliability observer bias); content specific so low reliability (must use > 6 cases to increase reliability); labor (must use > 6 cases to increase reliability); labor intensiveintensive

Suggestions: multiple short cases; define questions Suggestions: multiple short cases; define questions and answers; provide simple rating scales and train and answers; provide simple rating scales and train ratersraters

Triple JumpTriple Jump

Definition: Three step written and oral exam- written, Definition: Three step written and oral exam- written, research and then oral part- (ex. COMPASS 1)research and then oral part- (ex. COMPASS 1)

Measures: Knows, knows how, shows how and doesMeasures: Knows, knows how, shows how and does Pros: Assesses hypothesis generation, use of resources, Pros: Assesses hypothesis generation, use of resources,

application of knowledge to problem solve and self application of knowledge to problem solve and self directed learning; provides immediate feedback; high face directed learning; provides immediate feedback; high face validityvalidity

Cons: only for formative assessment (poor reliability); Cons: only for formative assessment (poor reliability); time/faculty intensive; too content specific and time/faculty intensive; too content specific and inconsistent rater evaluationsinconsistent rater evaluations

Clinical ObservationsClinical Observations

Definition: Assessment of various domains Definition: Assessment of various domains longitudinally by an observer- either longitudinally by an observer- either preceptor, peer or self (small group preceptor, peer or self (small group evaluations during first two years and evaluations during first two years and preceptor ratings during clinical exposure)preceptor ratings during clinical exposure)

Measures: Knows, knows how, Shows how Measures: Knows, knows how, Shows how and Doesand Does

Pros: Simple; efficient; high face validity; Pros: Simple; efficient; high face validity; formative and summativeformative and summative

Clinical ObservationsClinical Observations Cons: low reliability (only recent encounters Cons: low reliability (only recent encounters

often influence grade); halo effect (lack of domain often influence grade); halo effect (lack of domain discrimination); more often a judgement of discrimination); more often a judgement of personality and “Lake Woebegone” effect (all personality and “Lake Woebegone” effect (all students are necessarily above average); students are necessarily above average); unwillingness to document negative ratings (fear unwillingness to document negative ratings (fear of failing someone)of failing someone)

Suggestions: Frequent ratings and feedback; Suggestions: Frequent ratings and feedback; increase the number of observations; multiple increase the number of observations; multiple assessors (with group discussion about specific assessors (with group discussion about specific ratings)ratings)

Peer/Self EvaluationPeer/Self Evaluation Pros: Useful for formative feedbackPros: Useful for formative feedback Cons: Lack of correlation with faculty evaluations; Cons: Lack of correlation with faculty evaluations;

same cons as others (measure of “nice guy”, low same cons as others (measure of “nice guy”, low reliability, halo effect- peer evaluations have friend reliability, halo effect- peer evaluations have friend effect or fear of retribution or desire to penalizeeffect or fear of retribution or desire to penalize

Suggestions: limit the # of behaviors assessed; Suggestions: limit the # of behaviors assessed; clarify the difference between evaluation of clarify the difference between evaluation of professional and personal aspects; develop professional and personal aspects; develop operationally proven criteria for rating; provide operationally proven criteria for rating; provide multiple opportunities for students to do this and multiple opportunities for students to do this and provide feedback from facultyprovide feedback from faculty

Knows

Knows How

ShowsHow

Does

Erica Friedman’s Educational Pyramid

MCQ

MEQ,Essay

OSCE, T JumpOral, SP, PracticalChart review

Direct Observation,Practical

Type of assessment Content areas tested

Potential Reliability

Potential Validity

WrittenWritten

MCQMCQ Knows, Knows HowKnows, Knows How ++ ++

MEQMEQ Knows, Knows HowKnows, Knows How + ++

Application test, PracticalApplication test, Practical Knows, Knows Knows, Knows How, Shows HowHow, Shows How

+ +

EssayEssay Knows, Knows HowKnows, Knows How 00 00

Medical Note reviewMedical Note review Knows How, DoesKnows How, Does ++ ++++

ObservedObserved

SPSP Shows How, Does ++ ++

OSCEOSCE Shows How, Does ++ ++

OralOral Shows How, Does ++ ++

Longitudinal clinical experienceLongitudinal clinical experience Shows How, Does 00 00

Critical factors for choosing an Critical factors for choosing an evaluation toolevaluation tool

Type of evaluation and feedback desired: Type of evaluation and feedback desired: formative/summativeformative/summative

Focus of evaluation:Focus of evaluation:

Knowledge, skills, behaviors (attitudes)Knowledge, skills, behaviors (attitudes) Level of evaluation:Level of evaluation:

Know, Knows how, Shows how, DoesKnow, Knows how, Shows how, Does Pros/Cons:Pros/Cons:

Validity, Reliability, Cost (time, $ resources)Validity, Reliability, Cost (time, $ resources)

How to be successfulHow to be successful

Students should be clear about the course/clerkship Students should be clear about the course/clerkship goals and the specifics about the types of goals and the specifics about the types of assessments used and the criteria for passing (and if assessments used and the criteria for passing (and if relevant, just short of honors and honors)relevant, just short of honors and honors)

Make sure the choice of assessments is consistent Make sure the choice of assessments is consistent with the values of your course and the schoolwith the values of your course and the school

Final judgments about students’ progress should be Final judgments about students’ progress should be based on multiple assessments using a variety of based on multiple assessments using a variety of methods over a period of time (instead of one time methods over a period of time (instead of one time point)point)

Year/Year/

# courses # courses or or clerkshipsclerkships

MMCC

QQ

MMEEQQ

EssayEssay Prac-Prac-tical tical

handshands-on-on

SPSP OSCEOSCE OralOral Small Small group or group or preceptorpreceptor

# # using using 1 tool1 tool

# # using using > 1 > 1 tooltool

1 n=111 n=11 99 55 11 33 11 -- -- 55 44 77

2 n=132 n=13 1212 44 44 -- 11 11 11 33 44 99

3 n=83 n=8 77 --1 1 Simu-Simu-

latorlator22 -- 77 88 -- 88

4 n=44 n=4 11 -- --1 1 Simu-Simu-latorlator

-- -- -- 44 22 22

Number of courses or clerkships using a specific assessment tool-assessing our assessment methods

Why assess ourselves?Why assess ourselves?

Assure successful completion of our course Assure successful completion of our course goals and objectivesgoals and objectives

Assure integration with the mission of the Assure integration with the mission of the schoolschool

Direct our teaching/learning-(determine Direct our teaching/learning-(determine what worked and what needs changing)what worked and what needs changing)

How we currently assess ourselvesHow we currently assess ourselves Student evaluations (quantitative and qualitative)- Student evaluations (quantitative and qualitative)-

most often summativemost often summative Performance of students on our exam and specific Performance of students on our exam and specific

sections of USMLEsections of USMLE Focus and feedback groups (formative and Focus and feedback groups (formative and

currently done by Dean’s office)currently done by Dean’s office) Peer evaluations of course/clerkship- by ECCPeer evaluations of course/clerkship- by ECC Self evaluations- yearly grid completed by course Self evaluations- yearly grid completed by course

directors and core facultydirectors and core faculty Consider peer evaluation of teaching and teaching Consider peer evaluation of teaching and teaching

materialsmaterials