Course Objectives

1. Recognize the scheme of integrated student assessment

2. Construct a standardized efficient MCQ

3. Analyze the quality of MCQ in the post exam

Evolution of Student Assessment Following Implementation of an Integrated Medical Curriculum: Contribution to Improved Educational OutcomesGregory K. Asimakis, Michael A. Ainsworth, Judith F. Aronson, Ann W. Frye, Steven A. Lieberman & Jeffrey P. Rabek

The University of Texas Medical Branch, School of MedicineGalveston, Texas, USAMedical Science Educator Volume 21 : No. 2 - articles 

lessons learned “Curriculum Change without Assessment Change Equals Little Change in Learning”

“modifying existing multiple choice “

“moved from a departmentally-based to centrally coordinated curriculum, “

“workshops to train the course faculty in test blue-printing, item writing, developing integrative questions, and analyzing item statistics”

Welcome Assessment coordinators

Objectives

1-Align the learning objectives of all levels (module ,year ,college) 2-Identify appropriate assessment methods. 3-Characterize efficient and effective assessment processes. 4-Describe the way of Implementation of the assessment plan 5-revise the plan by data analysis

Methods of assessment

Knowledge much ? level?

Skills level Attitudes with student patient staff Circumstances

Methods of assessments

• Ideal test pass all those deserve to pass and fail all those who deserve to fail

• So○ Use multiple methods to assess each

learning outcome. ○ Include qualitative as well as quantitative

measures.

MODIFIED ESSAY QUESTION MEQ

Please answer all the questions in sequence.

Mrs. Sami, a 50 years old lady complains of lethargy, weight loss and excessive vaginal discharge, has been admitted to the gynae ward on which you work.

Q.1. What are the three most likely diagnosis. a.b.c.

Q.2. List three specific questions which would help you distinguish between these possibilities.

a. b.c.

Q.3. A pap smear ordered by her general practitioner 2 month back revealed CIN II. Write down the clinical examinations you want to do.

a.b.c.

EMQLow Blood Pressure

a. Adrenaline f. Glyceryl trintitrateb. Anaphylaxis g. Gram negative sepsisc. Atenolol h. Hypovolaemic shockd. Autonomic neuropathy i. Morphinee. Cardiogenic shockj. Noradrenaline

The following patients are all found to have low blood pressure on examination. Choose the most likely underlying cause for each case. Each answer option may be used once, more than once or not at all.

1. A 64-year old man presents with a 2-hour history of sudden onset of severe abdominal pain radiating to his back. He has a history of treated hypertension and angina. On examination he is drowsy and pale, has cold extremities with a weak radial pulse of 136/min and a BP of 66/38. (h)

2. A 72-year old woman with a history of ischaemic heart disease and chronic atrial fibrillation undergoes an emergency laparotomy for a perforated sigmoid colon. Twenty four hours after surgery, her conscious level begins to fall and her urine output decreases to less than 15ml/hour. On examination she has warm and clammy hands and feet, and has a bounding radial pulse of 124/min and BP of 76/48. Prior to surgery her pulse and blood pressure were normal. (g)

3. A 35-year old man is admitted with a right lower lobe pneumonia. He is treated with intravenous Amoxicillin and oral Erythromycin. Shortly after the drugs have been administered he complains of difficulty breathing and develops a florid rash over his whole body. He becomes markedly hypotensive and you notice that his lips are swollen. (b)

Quality assurance principles

• Reliability

• Validity.

• Educational impact

• Feasibility

• Acceptability

• Formative. For,Self. As ,summative. Of

How to increase reliability

• Increase number of items

• Increase time of testing

• Large range of score

• Using high discriminatory item

How to improve validity?

• By choosing the correct questions and form of question.

Knowledge

• MCQ. ESSAY..

• Validity. Low High.

• Reliability. High. Low.

Attitude

• Low reliability (behavioral expression

• Variable validity best is workplace assessment

Way of implementation of assessment plan

Data Analysis

• Let the data sing

• Difficulty index

• Discrimination test

• Test item analysis

summary

• integrated assessment is a team based work

• planning :concept ,objectives,methods,quality quide ,analysis revision ,and making decisions

How to Construct Effective Multiple-ChoiceTests

March 22, 2010

Objectives

• List what can be tested with multiple-choicetests.

• Define the anatomy of a multiple-choice item.• Explain guidelines for constructing multiple-

choice items.• Review examples of effective and ineffective

multiple choice tests.• Evaluate MCQs items after exam(post tests)• Identify MCQs technical flaws(practical)

When to Use

• To test a variety of levels of learning

• When you have a large number of individualstaking the test

• When you have time to construct the test items

• When time is limited for scoring(Clegg & Cashin, 1986)

Strengths Weaknesses

Easily marked Very hard to set well

testing factual knowledge not effective for higher orderthinking & skills…..

run on-line. guessing

save time and resources Female students

Formative +summative Danger of assessing

Planning a Test

• Use a test blueprint.

• Identify major ideas and skillsrather than specific details.

• Choose format : best answer correct answer (Nilson, 2010)

Test Matrix

Anatomy of a Multiple-Choice item

Each item has:• stem

• Lead- in question

• Answer and alternativescorrect or keyed optionseveral distractor options

1.2.3.4.

Choose an important conceptWrite the stemWrite the correct answer (key)Develop distractors

common misconceptionserrors that could be madeplausible, yet less important informationsimilar in style, length to the keyevery distractor should be reasonable

(Clegg & Cashin, 1986)

Format stems as

• problems, scenario or tasksClearcentral ideaAvoid window dressing (excessive verbiage).Positively structured

• Include most of informations in the stem sothat the options can be short

The Stem• USE familiar language

• PROVID sufficient information

• DO NOT create tricky items by omitting essentialinformation

• DO NOT add extraneous information

• AVOID statements that fail to present a completethought or question.

• ELIMINATE excessive wording and irrelevantinformation

Writing StemsWhen using incomplete statements avoid beginning with theblank space.___ is the least severe form of behavior disorder.

A. PsychosisB. Panic disorderC. NeurastheniaD. Neurosis

Better:The least severe form of behavior disorder is ___ .

(Ory & Ryan, 1993)

Please check

Lead -in Question• Different clinical tasks can be tested

• Can be done with the same stem (cloning ofquestion)

•••••

HistoryDiagnosisInvestigationsManagement/Treatment/Drug therapyCounseling

Lead-in Question• Try asking questions that lead to clinical decision

making

Which one of the following…………?– … is the most likely diagnosis?– … investigations would you now order?

– …is the next step in the work-up andmanagement of this patient?

– … is the most important step in the initialmanagement of this patient?

EXCEPT

Item testing clinical decision makingA 62 year-old woman with a history of confusionand constipation comes to the office for a follow-up visit. Laboratory investigations reveal a serum

calcium of 2.9mmol/L, a creatinine of 146µmol/L, and a hemoglobin of 108g/L. Which one

of the following is the most likely diagnosis?1. Hyperparathyroidism2. Chronic renal failure3. Multiple myeloma4. Vitamin D intoxication5. Renal cell carcinoma

Item testing clinical decision making

• A 62 year-old woman with a history of confusionand constipationcomes to the office for a follow-up visit. Laboratory investigations reveal a serumcalcium of 2.9mmol/L, a creatinine of 146µmol/L, and a hemoglobin of 108 g/L. Which oneof the following would help confirm the diagnosis?

1. Parathyroid hormone2. Serum protein electrophoresis3. 25-OH vitamin D4. Serum creatinine

5. Abdominal ultrasound

Writing Item Alternatives• Three - five is adequate.

• only one is the right answer.

• Vary the location .

• logical or numerical order.

• independent, not overlapping.

Writing Item Alternatives• homogeneous

• grammatical structure.

• equal in length

• None-of-the-above carefully.

• Avoid All-of-the-above.

Writing Item Alternatives• AVOID• negatives such as NOT // I don’t know

• clues to the right answer, such as:o specific determinates including always, never, completely,

and absolutely.o clang associations, choices identical to or resembling

words in the stem.o conspicuous correct choice.o pairs or triplets of options that clue the test-taker to the

correct choice.o blatantly absurd, ridiculous options.

• Logical cues• A 47-year old man present with an acute

episode of psychosis. Which one of thefollowing treatment would you consider

prescribing?• 1. Alprazolam• 2. Lorazepam• 3. Haloperidol*

• 4. Diazepam

What is a “bad” MCQ?

If students do not answer the question correctlybecause it is:

• too difficult or• too easy or

• unclear,edited or deleted

Thank you