Preparing Effective Essay Questions

Bryan Bradley

Center for Teaching and Learning

January 21, 2010

http://ctl.byu.edu/home/about/contact/

Focus:

• Focus on essays as a way to assess thinking and reasoning;

(we will Not be focusing on essays as a way to assess writing ability.)

Objectives:

• Review definition of an essay question• Evaluate when an essay question

should be used• Construct effective essay questions• Discuss how to prepare students to

take essay exams• Review methods for scoring student

responses

Learning-Centered ObjectivesLearning-Centered Objectives

FOCUS ON HIGHER-LEVEL LEARNING GOALSFOCUS ON HIGHER-LEVEL LEARNING GOALS

LEVEL 1 LEVEL 1

RememberRemember

Retrieve Retrieve relevant relevant knowledge knowledge from long-from long-term term memorymemory

LEVEL 2 LEVEL 2

UnderstandUnderstand

Construct Construct meaning meaning from from instructional instructional messages, messages, including including oral, written oral, written and graphic and graphic

LEVEL 3 LEVEL 3

ApplyApply

Carry out Carry out or use a or use a procedure procedure in a given in a given situationsituation

LEVEL 4 LEVEL 4

AnalyzeAnalyze

Break Break material into material into parts and parts and relate to relate to one another one another and to and to larger larger structure / structure / purposepurpose

LEVEL 5 LEVEL 5

EvaluateEvaluate

Make Make judgments judgments based on based on criteria and criteria and standardsstandards

LEVEL 6 LEVEL 6

CreateCreate

Put Put elements elements together to together to form a form a coherent / coherent / functional functional whole; whole; reorganize reorganize into new into new patternspatterns

L.W. Anderson and D. R. Krathwohl (eds). L.W. Anderson and D. R. Krathwohl (eds). A Taxonomy for Learning, Teaching and AssessingA Taxonomy for Learning, Teaching and Assessing, 2001., 2001.

5

Effective Effective Alignment Alignment of of Course Design ComponentsCourse Design Components

Learning Learning OutcomesOutcomes

(Verb)(Verb)

Learning Learning Activities Activities

(Verb)(Verb)

Learning Learning Assessment Assessment

(Verb)(Verb)

1. Remember1. Remember 2.Understand2.Understand

3. Apply 4. Analyze3. Apply 4. Analyze5. Evaluate 6. Create5. Evaluate 6. Create

Categories of In-Class Categories of In-Class AssessmentsAssessments

Which level of cognitive activity would be most Which level of cognitive activity would be most valuable for students and instructors during class?valuable for students and instructors during class?Remember Remember

Tell, list, Tell, list, describe, describe, name, repeat, name, repeat, remember, remember, recall, identify, recall, identify, state, select, state, select, match, know, match, know, locate, report, locate, report, recognize, recognize, observe, observe, choose, who, choose, who, what, where, what, where, whenwhen

Understand Understand

Change, explain, Change, explain, restate, find, restate, find, describe, review, describe, review, relate, define, relate, define, clarify, illustrate, clarify, illustrate, diagram, outline, diagram, outline, summarize, summarize, interpret, interpret, paraphrase, paraphrase, transform, transform, compare compare similarities and similarities and differences, differences, derive main ideaderive main idea

ApplyApply

Apply, Apply, practice, practice, employ, solve, employ, solve, use, use, demonstrate, demonstrate, illustrate, illustrate, show, report, show, report, paint, draw, paint, draw, collect, collect, dramatize, dramatize, classify, put in classify, put in orderorder

AnalyzeAnalyze

Analyze, Analyze, dissect, detect, dissect, detect, test, test, deconstruct, deconstruct, discriminate, discriminate, distinguish, distinguish, examine, focus, examine, focus, survey, survey, compare, compare, contrast, contrast, classify, classify, investigate, investigate, outline, outline, structure, structure, categorize, categorize, same/different, same/different, solve, solve, determine determine evidence and evidence and conclusionsconclusions

EvaluateEvaluate

Coordinate, Coordinate, judge, judge, select/choose, select/choose, decide, debate, decide, debate, evaluate, evaluate, justify, justify, recommend, recommend, verify, monitor, verify, monitor, measure, the measure, the best way, what best way, what worked, what worked, what could have could have been different, been different, what is your what is your opinion, testopinion, test

CreateCreate

Create, Create, hypothesis, hypothesis, design, design, construct construct invent, imagine, invent, imagine, compose, compose, pretend, pretend, predict, predict, organize, plan, organize, plan, modify, modify, improve, improve, suppose, suppose, produce, set produce, set up, what if, up, what if, propose, propose, formulate, solve formulate, solve (more than one (more than one answer)answer)

What is an Essay Question?

• It reflects the cognitive (thinking) behaviors noted in the course goals and modeled in the readings, discussions and other course activities.

• It clearly defines and establishes the task within the problem situation.

• It presents a reasonable task to students.• It is generally written as an imperative statement

or as one or more questions. • It reflects the major areas of emphasis for which

students will be scored.

Evaluate the following essay question. Does it meet the

criteria?

List the Alcoholics Anonymous 12-steps for alcoholism recovery. List them in their correct progressive order.

Evaluate the following essay question. Does it meet the

criteria?Explain in what ways a person’s failure to apply step 5 of the 12-steps for alcoholism recovery will impact his/her ability to continue on the path to sustained sobriety. Provide an example that illustrates this impact.

Common Misconceptions

• Essay questions assess higher-order or critical thinking skills regardless of how the responses are written.

• Essay Questions are easy to construct.

• The use of essay questions eliminates the problem of guessing.

• Essay questions benefit all students by placing emphasis on the importance of written communication skills.

• Essay questions encourage students to prepare (study) more thoroughly.

When Should You Use Essays?

It is appropriate to use essay questions for the following purposes:

• To assess students' understanding of subject-matter content.• To assess students' abilities to reason with their knowledge of a subject.

When Should You Use Essays?

Use essay questions for intended learning outcomes that require complex thinking and constructed responses.

When Should You Use Essays?

If an intended learning outcome could beeither assessed through objective items or essay questions, use essay questions for the following situations:

• Your skill in writing objective items is poor, but your resources and time for grading are high (e.g., small classes, grading assistants).• Student reasoning needs to be evaluated.

How to Construct Essay Questions1. Clearly define the intended learning outcome to be assessed by the item.

2. Avoid using essay questions for intended learning outcomes that are better assessed with other kinds of assessment.

3. Clearly define the task (thinking processes and content to use in answering the question) and situate it in a novel problem situation.

a. Clearly define the task (thinking processes and content).b. Delimit the scope of the task.c. Clearly develop the problem or problem situation.

4. Present a reasonable task to students.

5. The tasks can be written as a statement or a question.

6. Specify the relative point value and the approximate time limit in clear directions.

How to Construct Essay Questions

7. State the criteria for grading.

8. Know the trade-offs between focused and less-focused essay questions.

9. Avoid the use of optional questions.

10. Improve the essay question through preview and review.a. Predict the student responses.b. Write a model answer.c. Ask a knowledgeable colleague to critically review the essay question, the

model answer and the intended learning outcome for alignment. d. Review student responses to the essay questions.

Match the Assessment to the Context of the Learned Skill

Supporting Students in Preparing for Essay Questions

• Emphasize Alignment• Highlight the required skills students need• Ask probing questions• Model test taking strategies• Link learning goals and class activities• Give feedback to students

– Frequent– Immediate– Discriminating– Lovingly

Two Approaches to Scoring Essays

Holistic• Prepare an ideal response as a

standard. Essay responses representing different degrees of lesser quality may also be used as anchor points

• The grader rapidly reads each essay and forms a general impression of it. The general impression can be recorded in pencil next to the response.

• The grader focuses on the whole essay as an integrated unit rather than focusing on specific features that a good essay should include

• The essay as a whole is compared to the model answer (and other anchors, if any) and given a single overall rating

Analytical• List the components that should be

contained in an ideal answer. This should include major ideas but also specific features such as accuracy and factual information, pertinent examples, relevant reasons, unified organization etc.

• The grader reads each essay and searches for component ideas or desired features

• Points are assigned based upon the degree to which each desired component is present

• Each component should be scored independently

Scoring Student Responsesto Essay Questions

• Decide whether to use the holistic or the analytic approach.

• Develop the scoring key/model answer(s).

• Decide in advance how to handle irrelevant and inaccurate info, bluffing, or technical problems.

• Improve the scoring key/model answer.

• Evaluate responses.

• Don't look at the student's name.

• Apply the answer key to all students.

• Grade only one question at a time.

• Be aware of typical student errors.

• Look for reoccurring mistakes in student responses.

• Randomly shuffle the order of papers each time before grading a new question.

• If possible score responses twice.

• If possible ask another person to score a few of the items to check for reliability.

Rubrics

• http://sat.collegeboard.com/scores/sat-essay-scoring-guide• http://www.pz.harvard.edu/Research/RubricsSelfPE.htm• http://www.rcampus.com/rubricshowc.cfm?sp=true&code=B37C97&

Conclusion

• Questions

• Reflections

• Final Thoughts