STRATEGIESTO EVALUATE COMPETENCE

Christina B. DeBiase, EdDWVU School of Dentistry

STEPS IN EVALUATION

• Define the objectives of the learning activity – a test should evaluate the degree of student learning

• Develop a test blueprint

TEST BLUEPRINTS

• How much emphasis should be given to each objective?

• What type of question would best assess the students’ attainment of objectives?

• How difficult should the questions be to adequately discriminate the stronger students from the weaker students?

• What considerations should be given to test format and administration?

Basic Forms of Evaluation

1. Objective – data gathered represents some predetermined standard

2. Subjective – no prior standard of performance has been specified

• TESTS• CBL / PBL SCENARIOS• WRITTEN & ORAL

PRESENTATIONS• SIMULATIONS• PRACTICUMS• PORTFOLIOS• OSCE• ETC.

METHODS FOR EVALUATING STUDENTS

OPTIONS FOR TEST ITEMS

• Stand-alone multiple-choice items• Case-based multiple-choice items• True/False• Fill-in the blank/ Short-answer/Completion• Essay• Listing• Oral• Rating sheets

COMMON TEST-WRITING ERRORS

MULTIPLE-CHOICE COMPOSITION

• STEM: SET UP THE PROBLEM OR SET OF CIRCUMSTANCES; SHOULD HAVE AT LEAST ONE VERB; CAN BE WRITTEN IN QUESTION FORM OR AS A PARTIAL STATEMENT

• DISTRACTORS: ONE CORRECT OR BEST RESPONSE (tests higher level learning) AND 3-4 ALTERNATIVES; ALL OF THE ABOVE OR NONE... ARE NOT SPECIFIC OPTIONS OR SOLUTIONS AND SHOULD BE AVOIDED

KEEP IT SIMPLE!• DO NOT PUT EXTRA, UNNECESSARY

INFORMATION IN THE STEM: Mary, a 16-year old female with cerebral palsy, presents with a bruxing habit. Bruxism is the rhythmic grinding of teeth that can cause overdevelopment of a muscle of mastication. Which of the following muscles may be overdeveloped in a patient with bruxism?

POSITIVE/NEGATIVE STEMS

• POSITIVE - Usually more effective because the item tests what examinees know to be correct. Which of the following antibiotics is recommended for SBE coverage?

• NEGATIVE - Sometimes appropriate to distinguish the incorrect response among several correct alternatives. All of the following congenital cardiac defects require antibiotic premedication prior to dental procedures except one. The EXCEPTION is ___?

RULES TO FOLLOW WHEN CREATING ALTERNATIVES

• INCLUDE REPITITIOUS WORDS IN THE STEM TO AVOID REPEATING WORDS IN EVERY ALTERNATIVE.

• ALTERNATIVES SHOULD BE SIMILAR IN CONSTRUCTION AND OF APPROXIMATELY EQUAL LENGTH

• DISTRACTORS SHOULD BE PLAUSIBLE & MAY REPRESENT COMMONLY HELD MISCONCEPTIONS ABOUT A SUBJECT

• AVOID RESPONSE PATTERNS

The correct answer is longer or more specific than the distractors:All of the following determine the latent period’s

degree of biologic effect except one. The EXCEPTION is ___?

a. The exposure time

b. The type of radiation

c. The type of tissue irradiated

d. The ability to cause substances to fluoresce *

HINT- Put “The” in the stem!

ADDITIONAL PRACTICES TO TO AVOID WHEN

DESIGNING ALTERNATIVES FOR MULTIPLE-CHOICE

QUESTIONS

The alternatives are not mutually exclusive (overlapping alternatives);if numbers, always list in ascending or descending order & if ranges, intervals should be equivalent:Which of the following I.Q. ranges is descriptive of a person with moderate mental retardation?a. 50-70b. 35-50c. 20-35d. <20

Distractors are not plausible:

Which of the following primary molars is a smaller replica of the permanent first molar:

a. First

b. Second

c. Third

Grammatical clues in the stem suggest the correct answer:

Which of the following drugs is an antihypertensive agent?

a. Procardia

b. Prednisone

c. Cyclosporine and Imuran

d. Amoxicillin and Erythromycin

Information in the stem is also found in the correct answer:

Which of the following forms of diabetes requires insulin for effective management?

a. Insulin - dependent

b. Non-insulin dependent

c. Hyperglycemia

d. Hypoglycemia

CAUSE AND EFFECT ITEMS• (Statement)Prior to cardiac surgery, a patient should

receive a dental cleaning, (Reason) BECAUSE in a plaque-infected oral cavity the likelihood of bacteremia is greatly increased.

• a. Both the statement and reason are correct and related.• b. Both the statement and the reason are correct, but NOT

related.• c.The statement is correct, but the reason is NOT.• d.The statement is NOT correct, but the reason is correct.• e. NEITHER the statement NOR the reason is correct.

PAIRED TRUE & FALSE ITEMS

• I. Medical record entries should be completed in non-erasable ink. II. If a mistake is made when making a notation, the incorrect information should be covered with correction fluid.

• A. Both statements are true.

• B. Both statements are false.

• C. The first statement is true, the second is false.

• D. The first statement is false, the second is true.

ADVANTAGES OF MULTIPLE-CHOICE ITEMS

• Flexible• Measure student’s knowledge in a subject area without

the influence of writing skills, spelling, or grammar• Scored quickly• Content of course can be adequately sampled• Comparable results can provide diagnostic information

about class • The best answer does not have to be the only correct

response, but should be defensible• Item- analysis is very feasible

DISADVANTAGES

• Questions are difficult to develop with sufficient plausible alternatives

• Cannot measure higher level learning unless case-based; stand-alone items usually measure factual recall only

• Guessing the correct answer can make the test ineffective

GUIDELINES FOR TRUE & FALSE STATEMENTS

AVOID:

• Negatively worded statements

• Testing trivial detail

• Specific determiners that act as clues – usually, never, always, all

GUIDELINES FOR MATCHING ITEMS

• Specify whether alternatives can be used once or more than once

• It is recommended to provide more alternatives than premises, but the number of alternatives should not greatly exceed the premises

• List premises or phrases on the left and alternatives on the right

GUIDELINES FOR COMPLETION QUESTIONS

• Include sufficient information to tell student what type of answer is expected

• Omit as few words as possible

• Use a uniform blank space

• Avoid grammatical clues

• Place the blank toward the end of the sentence

GUIDELINES FOR ESSAY QUESTIONS

• Cover the objectives• Prepare a structured response with elements students should

include for credit• Give clear directions so that the student knows what is

expected• Ensure the test is short enough to allow time to complete the

essay(s)• Avoid testing trivial details• Avoid giving students the option to answer 5 of 7 essays , for

example• Grade by reading all students’ responses to one essay before

going on to the next question