Writing Effective Multiple Choice Questions

Carlinda D’Alimonte

Thanks to Alice Aspinall for her help with preparing this presentation.

Rationale for Training EQAO Test Questions

Poor performance of students on MCQs

Incongruence

Practice for standardized tests

An opportunity to respond to well constructed multiple-choice questions improves performance

2

Our Focus

Student’s Process

Structure of the Question

Cognitive Process

Process

It is helpful to consider how students should approach multiple choice questions on a test/exam.

Three-Step Process

Before they attempt any of these, read the passage for basic understanding.

Step One

Step Two

Read the first question.

Does the student know the answer?

If yes, read through all the choices and make sure they select the the most correct answer. Do not choose the first correct answer they see.

Step Three

If they do not know the answer right away:

- read the four choices- cross out obviously wrong answers- highlight key words in the question and scan the

text for those words to try to find the right answer

Be aware that some answers may seem correct, but there is only one best answer.

Further Instructions

Now go back to any questions that they circled and left unanswered.

If time permits, repeat the steps to answering multiple choice questions and review the passage.

Always make a selection. Never leave an answer blank.

Repeat this three-step process with each question in the passage.

Review – Answering MC Questions

Structure

Anatomy of a Multiple-Choice Question

Varieties of MC Questions

Single Correct Answer Single Best Answer – Our Focus Today Negative – Which of the following is NOT

… Multiple Response

Structure: How the Question is Constructed Use questions instead of incomplete statements Base each item on a specific problem/issue stated clearly in the

stem. The student should be able to answer the question before reading the possible answers or have a clear, specific idea of what is being asked

Include as much of the item as possible in the stem, but do not include irrelevant material

Use plausible distracters and word them clearly and concisely Keep the alternatives mutually exclusive Keep alternatives homogeneous in content Keep options at a similar length Be grammatically correct Avoid clues to the answer Use only “most correct” answer Do not use “all of the above,” “none of the above,” or negative

questions or answers Give clear instructions

Multiple Choice Questions

Think of two types of questions.

1. Questions whose answers are easily determined. In the case of a reading passage the answer is found right in the text.

- easiest to write- measure knowledge only

2. Questions whose answers require students to infer – that is, read between the lines or extend beyond what is written in the passage

- take time to construct - measure higher level thinking

16

Making an Inference Means…..

• Drawing conclusions• Making deductions • Making conjectures• Speculating • Making presumptions • … and more

17

Cognitive ProcessEmphasize

Higher-Level Thinking

Think of Bloom’s Taxonomy

Bloom’s TaxonomyCompetence Skills Question Cues

Knowledge - know major ideas, dates, events, places- master subject matter - observe and recall of information

collect, define, describe, examine, identify, label, list, name, outline, quote, show, state who, when, where, etc., tabulate, tell

Comprehension - understand/grasp information/meaning- translate knowledge into new context- interpret facts, compare, contrast- order, group, infer causes- predict consequences

associate, contrast, convert, describe, differentiate, distinguish, discuss, estimate, extend, infer, interpret, paraphrase, predict, summarize, translate

Application

- use information, methods, concepts, theories in new situations- solve problems using required skills

apply, demonstrate, calculate, complete,compute, illustrate, show, solve, examine,modify, relate, classify, experiment, discover

Analysis - see patterns or organize parts- recognize hidden meanings- identify components

analyze, separate, order, explain,connect, classify, distinguish, arrange, divide, compare, select, explain, infer

Synthesis - use old ideas to create new ones- generalize from given facts- relate knowledge from several areas- predict, draw conclusions

combine, integrate, modify, rearrange,substitute, plan, create, design,invent, what if, compose, formulate,prepare, generalize, rewrite, summarize, explain

Evaluation - compare and discriminate between ideas- assess value of theories, presentations- chose reasoned arguments- verify value of evidence- recognize subjectivity

assess, compare, conclude, contrast, decide, rank, grade, test, measure, recommend, convince, select,judge, explain, discriminate,support, conclude, compare

"Big Dog & Little Dog's Performance Juxaposition." Bloom's Taxonomy of Learning Domains. N.p., 26/05/2009. Web. 16 Apr 2010. <http://nwlink.com/~Donclark/hrd/bloom.html>.

"Bloom's Taxonomy." Univeristy of Victoria Student Services. University of Victoria, 17/12/2009. Web. 16 Apr 2010. <http://www.coun.uvic.ca/learning/exams/blooms-taxonomy.html>.

What MCQ’s Can Assess

20

QUESTION TYPE BLOOM’S COMPETENCEGeneralization Comprehension, SynthesisSummarizations Comprehension, SynthesisConclusion Evaluation

Justification of Motives or Reasons EvaluationJudgment or Prediction Comprehension, Analysis

Showing Relevance or Significance EvaluationJustification/Evaluation of Methods, Procedures, Solutions

Comprehension, Application, Analysis, Synthesis, Evaluation

Support ComprehensionAuthorial Choices/Text Features Analysis

Application ApplicationComparison Comprehension, Analysis,

EvaluationInterpretation/Explaining Cause-Effect Relationships

Analysis, Evaluation

Generalizations:A General Thought, Idea or Premise Arrived at as a Result of or Based on Specific Information

1. Which word best describes the main character? the central conflict? the initial problem?

2. What is most likely the theme of the story?a. Nice guys often finish lastb. Kindness is not always its own rewardc. Kindness is its own rewardd. Time really does heal

3. What are paragraphs 3 and 4 mainly about? or What is the passage/selection/paragraph mainly about?

4. Which statement best describes the message of the article?a. The Iraqi people are upset that the American soldiers are

here.b. The situation in Baghdad has resulted in violence

erupting between Iraqi’s and US soldiers.c. Leadership in the capital has disappeared.d. The situation in Baghdad is still not secure.

Summarizations:A Statement Which Specifies the Main Idea or Main Points___________________________

1. Which is the best summary of the article/paragraph/excerpt?a. There are many types of jobs and many people looking for work.

It is important to find the best person suited to the job.b. Unemployment is on the rise. There are many unemployed

people who are applying for different types of jobs.c. Job openings are on the rise. It is important that the unemployed

work hard to try to find a job.d. Unemployment is on the rise. Job openings are on the decline

due to many economic factors

2. Which statement best summarizes the main character’s thoughts/actions/problem-solving processes, etc. when ___________?

1. What is this passage mostly about?

Conclusions:A Judgement Formed After Critical Thought___________________________

1. What might the reader conclude based on John’s actions?a. John is unable to control his angerb. John is very passionatec. John cannot accept the loss of his brotherd. John needs the support of his family.

2. Which author/playwright would be most likely to make this statement about his text? “I believe that the common man is as apt a subject for tragedy in its highest sense as kings were.”

a. Millerb. Orwellc. Shakespeared. Checkov

3. Based on the data in the chart/graph/data set, what might one conclude?

Motives or Reasons:Consider Actions, Ideas, Principles, etc.___________________________

1. What is the most likely reason Tom walked out of the meeting?a. He was afraid of being confronted about his previous actionsb. He did not feel prepared and needed time to consider how he would handle his

presentationc. He was angry about what had happened in his boss’s office that morningd. He remembered that he needed to make an urgent phone call

2. Which of the following best justifies Fred Scott’s decision to sue his employer?

3. According to the article, why are teens texting more and blogging less?

a. Teens like texting betterb. There is an explosion in social networkingc. In 2006, 28% of teen internet users bloggedd. Older people are becoming more comfortable with blogging

4. Which statement best describes the role a circle plays in being able to find the area of a minor segment?

Judgements/Predictions:Determinations Related to What Might Happen or Occur Next___________________________

1. What will most likely be the effect of Fortinbras’ leadership in Norway?

a. He will be a stabilizing influenceb. He will involve Norway in more warsc. He will be greedy in his approach to leadingd. He will have a destabilizing influence

2. Based on Anne’s past actions what is she most likely to do if she …

Relevance or Significance:___________________________

1. What does the robe symbolize in the passage?a. Loveb. Greedc. Charitye. Hope

2. How is the river important to the story?

3. How has postmodernism shaped film over the past twenty years?

Justification/Evaluation:Examination of Methods, Procedures, Solutions___________________________

1. Which statement best supports the ideas contain in paragraph 9?

2. Which statement is the best assessment of …? 3. How would you evaluate the strength of a thesis

statement?a. It is meaningful, defensible, and controversialb. It is meaningful, direct, and broad in its scopec. It is meaningful, specific, and phrased as a questiond. It is meaningful, defensible, and specific

4. Which of the following is the best procedure to use to determine the effectiveness of supporting points?

Support:Reasoning or Evidence Behind Ideas___________________________

1. Which statement best supports the idea that…?

2. Which of the following actions best supports the idea of John’s impulsiveness?

a. He skipped school oftenb. He ignored his responsibilitiesc. He was led to believe he could trust Jamesd. He often regretted his choices

3. What statement best illustrates ideas explored in Theatre of the Absurd?

a. Life is meaninglessb. People avoid taking responsibility for their livesc. Events are essentially out of our controld. Life’s absurdity if fun

Authorial Choices/Text Features:Ideas About the Writer’s Purpose, Techniques or Use of Format___________________________

1. What is the most likely reason the author uses slang/a diary format/subtitles/italics/graphs/etc?

2. In Macbeth, what purpose is best served by the repetition of the word tyrant in Act 5?

a. It expresses the sentiments of those under Macbeth’s powerb. It emphasizes the transformation in Macbeth’s characterc. It indicates Macbeth’s pending demised. It supports the idea that, like Macbeth, we are all tyrants

3. What purpose is best served by the semicolon/colon/brackets/bold dash/quotation marks/etc. in paragraph 5?

4. What is the most likely reason that the author included paragraph 4 in this passage?

a. It provides a counter argumentb. It provides more background informationc. It shows that she is being faird. It creates doubt in the reader’s mind

Application:The Use of Previously Learned Information___________________________

1. Which set of questions would best guide your writing of a character analysis?

2. Which of the following statements uses the verb to lie correctly?a. I’m going to lay down for a while.b. She likes to lay around all day.c. Let’s lie in the grass all afternoon.d. I’m going to lie this book on the table for now.

3. Which of the following shows the best use of a semi-colon?a. By telling the story from the perspective of the protagonist, Walker create4s empathy for the main

character; we empathize with the narrator’s history, weaknesses, and intentions.b. By telling the story from the perspective of the protagonist, Walker creates empathy for the main

character; she also sets the story in a run down house in southern United States.c. By telling the story from the perspective of the protagonist, Walker creates empathy for the main

character; the main character struggles with her decision at the climax of the story.d. By telling the story from the perspective of the protagonist, Walker creates empathy for the main

character; the tone is both bitter and nostalgic, creating a tense mood.

Comparisons:___________________________

1. To what popular children’s story might Hamlet best be compared?

a. Little Red Riding Hood

b. Cinderella

c. The Lion King

d. Chicken Little

2. To what might _________ best be compared?

3. Which of these characters best contrasts Hamlet?

a. Julius Caesar

b. King Lear

c. Romeo

d. Macbeth

Interpretations/Explanations of Cause and Effect Relationships:Describing Outcomes or Impact of Events or Situations ___________________________

1. Which of the following best describes the effect of _________on _________?

2. What was the cause of Hamlet’s inability to kill Claudius in the church?

a. Over-thinkingb. Cowardicec. Angerd. Grief

3. Based on the context of the word “shenanigans” in the article, what would be its best definition?

a. Tricky or questionable conductb. Athletic abilityc. High energy leveld. Fair and generous practices

Improving Questions1. What marks the climax of a Shakespearean tragedy?

a. When the tragic hero commits an error of judgement

b. When the tragic hero recognizes his errorc. When the tragic hero first responds to the

generating incidentd. When the tragic hero dies

Suggestions for improvement: Include as much information in the stem so as to avoid unnecessary repetition in the answers

A better way to phrase this question might be as follows:

It is the behaviour of the tragic hero that marks the climax in a Shakespearean tragedy. Which of the following best describes the point at which the hero’s fortunes begin to decline?

Improving Questions

2. A word used to describe a noun is called an

a. Adjectiveb. Conjunctionc. Pronound. Verb

The word “an” gives the answer away. Also it would be better for the student to apply knowledge of adjectives to a passage.

This question could be more effectively phrased in this way:

In the following sentence the words that add descriptive detail to the images are what part of speech? He remembered the quaint East India Tea House at the fair, the polished sandalwood, the tightly draped turbans, and the loose robes, the cool interior and the sweet smell of India tea.

Improving Questions3. Hamlet

a. Is the Thane of Denmarkb. Is awkward when he rehearses with the playerc. Never demonstrates his leadership abilitiesd. Is clearly angry with Ophelia

Suggestions for improvement: The stem is vague and broad. The answers are not homogenous. The student should know what specifically is being asked after reading the stem.

A way to improve this question would be as follows: Hamlet is one of Shakespeare’s most complex characters. Which of

the following statements best reflects his complexity?a. He is indecisive when he rehearses with the players, yet he

is decisive with the pirates who take him captiveb. He is confident and forthright when he rehearses with the

actors, yet he is indecisive with the pirates take him captivec. He is accepting of his mother’s marriage, yet he cannot

accept Claudius as his stepfatherd. He is confident and forthright with the players and with the

pirates, yet he is indecisive about killing Claudius

Your Task Work in a small group to write two higher-level

multiple-choice questions that will measure a part of any two objectives.

Use the Planning Chart to write your questions and determine which higher order thinking task you are tapping.

Refer to the sample questions. Write structurally solid questions that force the

student to use higher order thinking .

Worksheet

37

Example

38

Which of the following best describes the reason it is important to follow the directions listed in the passage?

a. To be able to accurately complete income tax forms

b. To be able to learn to file taxes before deadline

c. To ensure the government will not do an audit

d. To ensure the correct information is on all forms

X

X

x

Share Your Favorite Question

Tell …. What higher order thinking skills you

are tapping, your question and answer choices.

39

ClassMarker

http://classweb101.pbworks.com/Online-Testing-Sites

http://www.classmarker.com/ http://walkliteracy.pbworks.com (Download

this and other material here.)

Works CitedBurton, Steven. Brigham Young University, 1991. Web. 16 Apr 2010.

<http://testing.byu.edu/info/handbooks/betteritems.pdf>.

Scott, Jana L. Making Inferences. University of Missouri, n.d. Web. 16 Feb 2010.

<www.av.k12.mo.us/main/files/Making_an_Inference.ppt >.

14 Rules for Writing Multiple-Choice Questions. Brigham Young University. Web. 12 Jan. 2010.

“Bloom's Taxonomy of Learning Domains.” Big Dog & Little Dog's Performance Juxaposition.".

N.p., 26/05/2009. Web. 16 Apr 2010. <http://nwlink.com/~Donclark/hrd/bloom.html>.

"Bloom's Taxonomy." Univeristy of Victoria Student Services. University of Victoria, 17/12/2009.

Web. 16 Apr 2010. <http://www.coun.uvic.ca/learning/exams/blooms-taxonomy.html>.