Item Development Principles
Well constructed multiple-choice items can test a
range of knowledge and skills involving higher
levels of thinking. They need not be limited to
recall items testing student knowledge of concrete
information. As well, multiple-choice examination
items can provide useful diagnostic information
about individual achievement as all students
experience the same task.
Guidelines for Building MC Items
Use the Program of Studies as your reference. Focus on a
specific outcome prior to writing the item.
Aim to write items that cover a range of difficulty.
Ensure that the item has both curricular and instructional
validity.
Ensure that the item tests something worthwhile.
Multiple-Choice Item Types
There are two types of multiple-choice items:
Discrete Item – a stem is provided and the student
seeks to select the correct response from the
alternatives provided.
Source-Based Item – an item that derives from
the consideration of details in a specific source.
Sources include written texts, graphs, charts and
photographs.
Anatomy of a Multiple-Choice Question
Which of the following celestial bodies emits
light?
A. Star
B. Moon
C. Planet
D. Comet
These are the “alternatives”,
“choices”, or “options”.
This is the “keyed response” or correct choice.
These are the “distracters”,
(alternatives that are incorrect).
This is the stem.
Building Multiple-Choice Questions:
The Stem
A stem may be worded in one of two different ways.
As a question:
What is the area of a rectangle that has a length of 3 cm and a width
of 1 cm?
As a statement:
The area of a rectangle that has a length of 3 cm and a width of 1 cm
is
The stem of the question must be focused. A good stem
provides enough context to enable the student to visualize
a correct response prior to reviewing the alternatives.
Building Multiple-Choice Questions:
Appropriate Diction
Use vocabulary that is appropriate to the reading
level of the students being tested and that is
consistent with the expectations of the
curriculum.
Avoid ambiguous word choices. Be sure to avoid
colloquial expressions, idioms, slang and jargon
that may be an obstacle to the students’
understanding of the question or the alternatives.
Building Multiple-Choice Questions:
Alternatives & Distracters
Distracters should be logical, for example, based on common misconceptions that students may have.
Directing words must be bolded (most directly, mainly, etc)
Alternatives should all be alike in kind (length, complexity, grammatical structure).
Be sure that the keyed answer does not stand out from the alternatives because it is more detailed or more qualified than the distracters.
Avoid creating a question that implies that wrong information is correct.
Make sure that a question contains only one correct answer.
Building Multiple-Choice Questions:
Alternatives & Distracters
Avoid absolutes such as "all" and "every" because
there has to be only one exception to make the
alternative wrong.
Avoid “all of the above” and “none of the above” as
distracters.
Building Multiple-Choice Questions:
Alternatives & Distracters
Quality Multiple-Choice Questions
Which of the following
sentences contains an example
of a correctly used apostrophe?
A. I di’dnt do my chores today.
B. I should’nt talk during class.
C. I couldn’t find my homework.
D. I woul’dnt want to go to the zoo.
Which of the following
words contains a
correctly used apostrophe?
A. should’nt
B. woul’dnt
C. couldn’t
D. di’dnt
SO: recognize various uses of apostrophes, and use them appropriately in own
writing (4.2c).
IO: Students will use apostrophes correctly in the their short stories. (applying
taxonomy level)
Building Multiple-Choice Questions:
Alternative Order
Alternatives must be pyramided, which means they are
arranged by length (shortest to longest, or longest to
shortest)
The exception to this rule is when the alternatives contain
direct quotations from a source. In this case, the
alternatives must be placed in the order that the quotations
appear in the source
Building Multiple-Choice Questions:
Not Pyramided
According to democratic theory, active interest groups
and a free press ensure that
A. bureaucratic decisions are given greater credibility
B. minority rights do not become entrenched C. elections reflect the wishes of mainstream political parties D. tyranny of the majority is prevented
Building Multiple-Choice Questions:
Pyramided
According to democratic theory, active interest groups
and a free press ensure that
C. bureaucratic decisions are given greater credibility
B. minority rights do not become entrenched
D. elections reflect the wishes of mainstream political parties
A. tyranny of the majority is prevented
Building Multiple-Choice Questions:
Reverse Pyramid
According to democratic theory, active interest groups
and a free press ensure that
B. bureaucratic decisions are given greater credibility
C. minority rights do not become entrenched
A. elections reflect the wishes of mainstream political parties
D. tyranny of the majority is prevented
Building Multiple-Choice Questions:
Quotation Order
The lack of communication between the speaker and the companion
is most clearly suggested in the quotation
A. “My shadow darkens your face” (line 8)
B. “removed / from my words, the touch / of my fingers” (lines 9-11)
C. “What caves do you move through?” (line 12)
D. “all you believe / falls away” (lines 17-18)
What other characteristics of good multiple
choice items does this question demonstrate?
Building Multiple-Choice Questions:
Your Turn
The clocks in the house are
A. in the hall
B. in the living room
C. in the kitchen
D. ticking quietly
What errors do you see? How would
you improve this question?
Building Multiple-Choice Questions:
Pitfalls
These data could most appropriately be modeled using
an
A. exponential function
B. logarithmic function
C. sinusoidal function
D. quadratic function
Building Multiple-Choice Questions:
Pitfalls
Another example:
In lines 46 to 48, the repetition of the word upright mainly
emphasizes Fine Man’s
A. good posture
B. sense of tenacity
C.high moral values
D.persevering attitude
Building Multiple-Choice Questions:
Pitfalls
Which of the following Shakespearean plays is an example of the author’s mastery of the tragic form?
A. Macbeth
B. The Misanthrope
D. The Massacre at Paris C. A Man for All Seasons
Building Multiple-Choice Questions:
Pitfalls
The quintessential woodworker would have which
of the following implements in the domicile in
which he pursues his noble vocation?
A. Knife and fork
B. Inclined plane
C. Chainsaw
D. Veritas maple woodbench with attached vise
The Strengths of Multiple-Choice Items
Strengths:
Can help identify student misconceptions and errors
(helpful for providing diagnostic information for both individual
test-takers and for a group)
Useful when there are multiple possible distracters
A relatively large number of items can be answered in a
typical testing period
Scoring is easy, objective and reliable
Possible to test a range of knowledge and skills including
higher level thinking
Eliminate bluffing and writing skill influences
Guessing becomes a factor
Difficult to write items that are above the knowledge
taxonomy level that are free of ambiguity
Difficult and time consuming to create
Poorly constructed items provide unreliable results and
invalid inferences, creating frustration and confusion
Potential for superficial or recall items that test trivia and
test-wiseness
The Limitations of Multiple-Choice Items
Limitations:
Multiple-Choice Item
Quality Checklist
An item is technically
sound when…
the item is free of cues
to the answer
the stem is focused on a
single, meaningful problem
key words in the stem are
emphasized as needed
distracters are all plausible
alternatives are in an order
that is logical and easily
understood
the item has correct spelling
and grammar
An item is equitable when
the item…
is free from cultural and sexual
bias
is free of irrelevant material
is stated in appropriate and
clear language
is free from cultural references
that would not be familiar to all
students
Correct choice:
• Possesses required
information or skill
• Uses partial information
to get the answer
• Uses cues in the item
and/or test
• Makes a blind, lucky
guess
Incorrect choice:
• Lacks required
information or skill
• Uses partial
information or
misinformation that
favours a distracter
• Makes a blind,
unlucky guess
Common Reasons for Selecting Correct
or Incorrect Multiple-Choice Answers
Building Multiple-Choice Questions:
Test-wiseness
Test-wiseness can be a huge problem!
– Verbal associations that give away the answer
– Grammatical inconsistencies that eliminate wrong answers
– Specific determiners that make certain answers probable and
others improbable
– Length or location of correct answers
– Material in one item that aids in answering another item
– Implausible alternatives given the context of the question
What Constitutes Quality Source Material?
Quality source material • must relate to the learning outcomes.
• must be novel (new to the students).
• must be at the appropriate length and reading level for the subject and grade level.
• stimulates thought about an issue or topic.
• be authentic and not directly taken from the prescribed resources for the course.
• provide adequate and appropriate context for questions.
• can come from a published source (beware of copyright issues).
• can be created by you.
Be Sure the Source is Required!
Where did the United Nations send its first
peacekeeping force?
A. Hungary
B. the West Bank
C. West Germany
D. the Suez Canal
Be Sure the Source is Required!
The newspaper headline identifies an
effort to resolve an international crisis
situation through the use of
A. collective security
B. mutual deterrence
C. appeasement
D. detente
What Constitutes a Quality Source-
Based Multiple-Choice Item?
Test the knowledge and skills you desire, NOT just the information presented explicitly in the source • The source MUST be required to answer the items
• When choosing or creating your own source, be careful not to include extraneous information
• The item must be at the “Applying” or above taxonomical level
• Follow the same guidelines for discrete multiple-choice item construction
Highlighted Sentences from Jenny’s Short Story:
The Day the Dogs Howled All Night
Sentence #1 I couldn’t believe the sound that came from all the howling dogs.
Sentence #2 I wanted to cover my ear’s but the sound was too loud.
Sentence #3 Ralph, my brown little terrier, started to bark just like my friend
Jessie’s dog.
Sentence #4 The dog didn’t hear me yelling but ran to the gate and escaped.
Use the following information to answer question 1.
Which of the sentences from Jenny’s short story contain an
apostrophe error?
A. Sentence #1
B. Sentence #2
C. Sentence #3
D. Sentence #4
Movie Admission Prices
Year Adult Ticket
2001 $12.25
2002 $12.75
2003 $13.25
2004 $13.75
6. If the pattern shown in the table above continues, then in which
year will the price of one adult ticket have increased to $15.25?
A. 2007
B. 2008
C. 2009
D. 2010
Use the following table to answer question 6.
Strengths and Limitations of
Source-Based MC Items
Strengths An efficient means of
measuring the interpretation of information in various forms (e.g., written, charts, graphs, maps, pictures).
Can measure complex learning outcomes.
Introductory material provides a common basis for responding.
Scoring is easy, objective, and reliable.
Limitations Difficult to construct quality
items using real-world sources.
Written material is highly dependent on reading skills.
Highly subject to cueing.
Ineffective in measuring the ability to originate, organize, and express ideas.
Source-Based Multiple Choice
Quality Checklist An item is technically
sound when…
the item is free of cues to the answer
the item uses source material that is
familiar but new to the student
the source material is an appropriate length
and at an appropriate reading level
the source material is properly cited
the items cannot be answered without the
source material
the items are at the applied taxonomy level
or above
the item has correct spelling and grammar
the items meet the technically sound
checklist for multiple choice items
An item is equitable when
the item…
is free from cultural and sexual
bias
is free of irrelevant material
is stated in appropriate and
clear language
is free from cultural references
that would not be familiar to all
students
Consider the Benefits of Open-Ended
Constructed Response Items
The following question was given to two hundred applicants for a job…
You are driving along on a stormy night in your little two-seater car. Passing a bus stop you see three people waiting:
A. an elderly woman clearly in need of immediate medical attention
B. an old friend who once saved your life
C. the perfect mate you have dreamed of your entire life
Who would you pick up? Share your choices with a partner.
Given the Following Response…
“I would give the car keys to my friend and let him take
the elderly woman to the hospital. Then I would stay
behind and wait for the bus with the person of my
dreams.”
…What does this response tell us about a benefit of
open-ended responses?
Structure of a Restricted-Constructed
Response Item
During an exam, it is obvious that the person
next to you is copying from the person sitting
in the row in front of you. List 3 reasons why it
is appropriate to report this behaviour.
Answers may be in point form or up to three
sentences in length. (3 marks)
A context is provided
Guiding words limit the
response and define the
subject matter
Scoring information is provided
Directions “bound” the
requirements for the
answer
A Quality Restricted-Constructed
Response Item
Choose the guiding words
carefully so writers create
the responses for which
you hoped.
Compose clear
directions
Point to an answer (few
sentences max)
Provide a meaningful
context
Explain
Predict
Show all calculations
Refer to specific details
Create a diagram
A Quality Restricted-Constructed
Response Item
Here are the directions and the context for the response item
Read the information below and answer the questions that follow. Marks
for each question are provided.
You have finally saved enough money to purchase your
first cell phone. This is an important purchase, so you
want to make sure you make an informed decision. You
decide to conduct a valid and reliable survey and have
placed your survey data in the following chart.
A Quality Restricted-Constructed
Response Item
Type of Cell
Phone
Number of Users Satisfaction
Rating (*****)
TouchIt 115 ****
Neptune S3 61 *****
TalkaLot 42 ***
SwishZ 34 ****
Samurai1000 18 **
1. Based on the data collected, formulate a survey question that
participants of this survey would have been asked. (1 mark)
2. Carefully analyzing the data provided, identify TWO valid conclusions that
can be made. Answer using complete sentences or point form. (2 marks)
Restricted-Constructed Response:
Your Turn
We have been studying the importance of the carbon
cycle and how it works. Briefly describe why we need
to know about it, how it works, and its location. Your
answer will be assessed using the following criteria:
Explain why understanding the carbon cycle is important
Identify at least 3 major carbon reservoirs
Describe at least 6 ways that carbon is transferred from one
place to another
3 points
3 points
6 points
A Quality Restricted-Constructed
Response Item
On a par four hole, Andy takes a practice swing and accidentally nudges his ball off the tee. He replaces it and hits his drive, hooking it into the rough. When he finds the ball, he moves it to a small clump of grass so he can get the club under it. He drops his next shot into the sand trap at the front of the green. He lifts the ball out of its crater and moves it to a level bit of sand. His shot lands two feet in front of the pin and rolls into the cup. Andy records a birdie for the hole.
Identify two violations of the rules of golf Andy has committed on this hole. (2 marks)
If Andy had played by the rules, what score should Andy have recorded on this hole? (1 mark)
Read the situation below and answer the questions that follow in point form. Marks for each question are indicated.
A Restricted-Constructed
Response Item
Describe three ways the artist uses colour in this painting. (3 marks)
Name two additional artistic techniques the artist obviously demonstrates a
loving fondness for in this painting. (2 marks)
Identify the likely time period in which this artist worked. (3 marks)
Describe two characteristics of the period in which this artist worked.
(2 marks)
Bonus: Identify the artist of this painting. (1 mark)
A Quality Restricted-Constructed
Response Item
Describe three ways the artist uses colour in this painting. (3 marks)
Name two additional artistic techniques the artist demonstrates in this painting.
(2 marks)
Identify the likely time period in which this artist worked. (1 mark)
Describe two characteristics of the art in the time period in which this artist
worked. (2 marks)
Each student who correctly identifies the artist who painted this work will get
to choose a treat from The Jar.
Restricted-Constructed Response
Strengths Guessing is less likely than in
selection-type items.
A range of outcomes can be
measured.
Directions impose boundaries
and limitations such that the
response is limited in scope
Easier to score than
extended items
Limitations Scope of answer is limited
There is a need for
establishing and sharing
scoring criteria ahead of time.
Less freedom of expression
than extended items
Restricted-Constructed Response
Quality Checklist
An item is technically sound
when…
the item contains appropriate guiding words
the item presents a context that is
meaningful and engaging for students
the item points to the answer so
that it is bounded
the item is above the “Remembering” taxonomical level
the item has correct spelling and
grammar
An item is equitable when the
item…
is free from cultural and
sexual bias
is free of irrelevant material
is stated in appropriate and
clear language
is free from cultural
references that would not be
familiar to all students
How to contact us Rite line: 310-0000 then (780) 427-0010
Director of Diploma Programs - Assessment Sector – [email protected]
English Language Arts 30-2 – [email protected] (Team Leader)
– [email protected] (Examiner)