The Irkutsk State Medical Academyfor Postgraduate Education

(Tempus IV 159328)

STEERING COMMITTEE MEETING & INTERNATIONAL FINAL CONFERENCE

Evaluation of Medical Student: Knowledge, know-how

and life skills

Pr Sylvain MeurisFaculté de Médecine

Université Libre de Bruxelles

Objective :To Reach Professional Competence !

Is the habitual and judicious use of:

Communication

Knowledge

Evidence-based decision-making

Technical Skills

Emotions

Values and reflection to improve the health of the individual patient and the community

Who Assess and What Should Be Assessed ? Transmission of knowledge, know-how & life skills

between teacher and student Who :

Teacher - During the training process Academic steeplechase (Education Minister)!

Peers - End of the training process Accreditation Council for Graduate Medical Education

(Health Minister)

What : Numerous and Cumulative Competencies :Knowledge + Practice-based learning and improvement + Interpersonal and communication skills + Ethics & Professionalism + System-based practice + Patient care,…

Is Assessment Necessary ? No formal assessment before the

Renaissance in Europe Chinese invention (VI century) to

Select Imperial officers Imported by Portuguese Jesuits to

Select Representatives for the Portuguese Colonial Empire

Widely adopted in European administrations and Universities !!!

Probably useful

Is Assessment Necessary ?

A useful tool to : Train students and Assume graduation responsibility

Two questions: WHY do we need to assess ? WHAT do we want to measure ?

Is Assessment Necessary ?

Answers depends on the position

Teacher verify if the minimum level of “knowledge” is reached Learning strategy

Student prepare examination to reach the minimum level to be authorized to continue… Strategy to pass

WHY do we assess ?

To ensure safety of patientsour responsibility to the public

To achieve of a minimum standardresponsibility

to the candidate and to the University !

In principle…

WHY do we assess ?

To ensure competence

As a means of academic competition

In practice: the scope…

WHY do we assess ?

Formative: to give feedback and advice When the cook tastes the soup, that's

formative Summative: to grade

When the guests taste the soup, that's summative

Qualificative: to graduate

In practice: the purpose

When should we assess ?

WHEN should we assess ?

Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

Knows

Shows how

Knows how

Does

Pro

fess

iona

l aut

hent

icity

Written, Oral orComputer based assessment

Performance or hands on assessment

WHEN should we assess ?

C+

Pass/Fail

AA-

85% F

S

Unsatisfactory

67% DC

B

93%

4/5

7/10

14/20

Certified

Qualified

Assessment is continuous (Life-Long-Learning)

HOW do we assess ? Assessment

The process of measuring something with the purpose of assigning a numerical value.

Scoring The procedure of assigning a numerical value to

assessment task. Evaluation

The process of determining the worth of something in relation to established benchmarks using assessment information.

Assessment Concerns Validity -- Is the test assessing what’s

intended? Are test items based on stated objectives? Are test items properly constructed?

Difficulty -- Are questions too easy or too hard? (e.g., 30% to 70% of students should answer a given item correctly)

Discriminability -- Are the performance on individual test items positively correlated with overall student performances? (e.g., only best students do well on most difficult questions)

Evaluation Types

Criterion-referenced evaluation -- student performance is assessed against a set of predetermined standards

Norm-referenced evaluation -- student performance is assessed relative to the other students

Criterion-Referenced Eval’s Based on a predetermined (minimum) set of

criteria. For instance,

90% and up = A 80% to 89.99% = B 70% to 79.99% = C 60% to 69.99% = D 59.99% and below = F

Criterion-Referenced Eval’s

Pros:- Sets minimum performance expectations.- Demonstrate what students can and cannot do in relation to important content-area standards

Cons:- Some times it’s hard to know just where to set boundary conditions- Lack of comparison data with other students and/or schools.

Norm-referenced Evaluation Based upon the assumption of a

standard normal (Gaussian) distribution with n > 30.

Employs the z score: A = top 10% (z > +1.28) B = next 20% (+0.53 < z < +1.28) C = central 40% (-0.53 < z < +0.53) D = next 20% (-1.28 < z < -0.53) F = bottom 10% (z < -1.28)

ECTS = European Credit Transfer System

Norm-referenced Evaluation Pros:

- Ensures a “spread” between top and bottom of the class for clear grade setting- Shows student performance relative to group

Cons: Top and bottom performances can sometimes be very close Dispenses with absolute criteria for performance

Being above average does not necessarily imply “A” performance

In a group with great performance, some will be ensured an “F.”

Criterion and Norm Evaluation Norm-Referenced

Ensures a competitive classroom atmosphere

Assumes a standard normal distribution

Small-group statistics a problem

Assumes “this” class like all others

Criterion-ReferencedAllows for a cooperative classroom atmosphere

No assumptions about form of distribution

Small-group statistics not a problem

Difficult to know just where to set criteria

Academic Grading !

In Sweden

1 Country = 1 Grading Scale

Need to be harmonized

WHAT do we measure ?

To test not only presence of knowledge

…but also the application of knowledge

In principle…

Model of clinical assessment

Certification of competence - pass / fail a state (and legal) requirement

Grading in rank orderfor employment / placement purposes

A competition for the award of a prize

In principle…a three-fold aim

Model of clinical assessment

Measurement of:

adequacy of basic clinical skills ability to interpret clinical findings facility of communication in practical

settings ability to think analytically about diagnosis ability to discuss management logically

In practice…

Practical steps for assessment At least 6 encounters with different clinical

situations (Internal Medicine, Surgery, Pediatrics, Ob-Gyn,…)

At least 2 examiners at every encounter, each examiner giving an individual assessment

Highly structured examination and detailed assessment of skills

Examiners from other Universities for process evaluation and quality control

Set-points for Testing

Attitude to patient

Actual examination skills

Presentation of findings

Clinical judgment

Clinical examination of patients

Methodology-related Issues

Reliability Validity Practicality Transparency

Organisational Issues

Resources available Student numbers Patient availability Invasive procedures Scheduling

Role of Grading

Is there need for grading ? Selection process for employment Selection for postgraduate training Quality of assessment method and

performance of candidates

Final question :What happens to candidates who fail ?

Review of performance – a formative exercise

Counselling at a personal level Specific attention and individual training Repeat assessment after a period of

time

In conclusionI. Assessment of student (a future peer) is a continuous

and complex process

II. Assessment of clinical skills is Teacher, Medical School, National and Culture-Dependent

III. There is place for widespread application of agreed standard methods to assess

- clinical competences in core curricula - additional competences essential to individual practice

IV. Teachers must give example of what areKnowledge, Know-How and Life-Skills