![Page 1: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/1.jpg)
1
Measuring What
Students Know:
Writing Effective Essay
and Oral Tests
![Page 2: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/2.jpg)
2
What Can Student
Assessments Do? • Communicate important content and skills
• Provide basis for grading, passing,
graduation
• Identify students with inadequate knowledge
• Identify areas of inadequate knowledge for
remediation
• Motivate students to master content and
skills in school and promote life-long learning
in practice
![Page 3: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/3.jpg)
3
Implications of Assessment
• Assessment is a critical tool in meeting
curriculum goals
• Thus, there should be clear mapping
between:
Assessments;
Knowledge, skills, and performance to be
mastered;
Course goals and objectives; and
Curriculum goals and objectives
![Page 4: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/4.jpg)
4
Basic Test Item
Formats
![Page 5: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/5.jpg)
5
Two General Categories
• Constructed-response formats
• Short-answer Items
• Essay Questions
• Oral Exams
• Defined response formats
• True/False
• Multiple Choice Questions
• Pick-N
• Extended matching
![Page 6: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/6.jpg)
6
Constructed-Response
Formats
![Page 7: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/7.jpg)
7
Short-Answer Items
• Strengths
• Less cueing than defined-response
formats
• Don’t have to come up with distractors
• Weakness
• Stem must be very clear and
unambiguous (can’t rely on the choices
to clarify the question)
• Typically must be hand-scored
![Page 8: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/8.jpg)
8
Rules for Writing Short-
Answer Items • State item so only a single, brief answer is
possible
• Harder than you think
• Requires precision in wording
• Have a colleague or student review
![Page 9: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/9.jpg)
9
Rules for Writing Short-
Answer Items • Start with a direct question, switch to
incomplete statement if greater
conciseness is possible
• What is another name for true-false
items?
• True-false items are also
called___________?
![Page 10: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/10.jpg)
10
Rules for Writing Short-
Answer Items • Leave only one blank and make it the
main point
• In terms of type of response, the
_________item is mot like the
___________________item.
• In terms of type of responses, which
item is most like the matching item?
___________
![Page 11: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/11.jpg)
11
Rules for Writing Short-
Answer Items • Place the blanks at the end of the
statement
• _____________is likely to increase
when a test is lengthened.
• When a test is lengthened, reliability is
like ________.
![Page 12: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/12.jpg)
12
Rules for Writing Short-
Answer Items • Avoid extraneous cues to the answer
• The constructed response item used to
measure the ability to organize and
integrate material is called
an__________
• Constructed response items used to
measure the ability to organize and
integrate material are called
_______________
![Page 13: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/13.jpg)
13
Rules for Writing Short-
Answer Items • For numerical answers, indicate degree of
precision expected and relevant units
• What is the typical retailer mark-up from
the wholesale price for a Canon D60
SLR camera (to the nearest whole
percent)?_________
![Page 14: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/14.jpg)
14
Essay Questions
• Best suited to higher level cognitive skills
(synthesis, analysis, creativity)
• Least cueing of correct answer
• Clear question becomes critical
(ambiguous question can largely negate
the value of an essay question)
• Most difficult to score
• Hand scoring
• Defining ‘model’ answer is difficult
![Page 15: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/15.jpg)
15
Rules for Writing Essay
Questions • Use essay questions to measure complex
learning outcomes only
• Relate questions to intended learning
outcomes as directly as possible
• Formulate questions that present a clear
task
• Suggest a time limit for each question
![Page 16: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/16.jpg)
16
Outcomes and Terms for
Question Construction • Comparing: compare, classify, describe,
distinguish between, explain, outline,
summarize
• Interpreting: convert, draw, estimate,
illustrate, interpret, restate, summarize,
translate
• Inferring: derive, draw estimate, extend,
extrapolate, predict, propose, relate
![Page 17: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/17.jpg)
17
Outcomes and Terms for
Question Construction • Creating: compose, design, devise, draw,
formulate, make up, present, propose
• Synthesizing: arrange, combine, construct,
design, rearrange, regroup, related, write
• Generalizing: construct, develop, explain,
formulate, generate, make propose, state
• Evaluating: appraise, criticize, defend,
describe, evaluate, explain, judge, write
![Page 18: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/18.jpg)
18
Outcomes and Terms for
Question Construction • Applying: arrange, compute, describe,
demonstrate, illustrate, rearrange, relate,
summarize
• Analyzing: break down, describe, diagram,
differentiate, divide, list, outline, separate
![Page 19: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/19.jpg)
19
Scoring Essays
• Evaluate in terms of learning outcomes
• Develop a model answer
or
• Define criteria for rating answer quality
• Grade all responses to a question before
grading the next question
• Blind the identity of the student
• Have another person grade the same
answers
![Page 20: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/20.jpg)
20
Oral Examinations
• One of the earliest examination formats (at
least back to the medieval universities)
• Largely replaced by defined-response
formats
• Strengths
• Assesses higher level cognitive skills
• synthesis, analysis of alternatives,
etc.
• Permits flexibility and probing
![Page 21: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/21.jpg)
21
Oral Examinations (cont’d)
• Weaknesses
• Less thorough than written forms of
examination
• Possibility of favoritism/interpersonal
biases
• Judging the ease or difficulty of the
questions is questionable
• Renewed interest in improved versions of
oral format
![Page 22: Measuring What Students Know: Writing Effective Essay …faculty.medicine.umich.edu/.../designing_essay_oral_tests.pdf · 1 Measuring What Students Know: Writing Effective Essay and](https://reader031.vdocuments.us/reader031/viewer/2022022505/5aba0b5c7f8b9aa6018e8f79/html5/thumbnails/22.jpg)
22
Rules and Scoring Oral
Examinations • See those for essay questions
• Unless audio/video recorded, there is no
permanent record
• Scoring is in real time
• Multiple scorers need to be there ->
scheduling logistics