Questions à choix multiples – Principes généraux

General Suggestions

1. Do not write the test in one day. Spread the work out over time. Questions demanding high-level thinking take longer to craft-professional item writers often write only 3 or 4 per day. Write one or two questions after each class, so it becomes a simple matter of assembling them into an exam. Some teachers keep a rubber-banded stack of note cards in their desk for this purpose.

2. If students are to hand-write the letters of their chosen answers, ask them to use CAPITAL LETTERS. The handwritten, lower-case letters "a" and "d" and "c" and "e" can be difficult to distinguish when scoring.

[Top]

Writing the Stem

1. Phrase stems as clearly as possible-confusing questions can generate wrong answers from students who do understand the material. For example, a confusing stem like:

"According to Tuckman's model, groups develop through several stages over time. Furthermore, it contradicts Poole's activity-track model which has groups switching among several different linear sequences. Which of the following is not one of the stages identified in Tuckman's model?"

could be cleaned up to read:

"Tuckman's model of group development includes: [Select all that apply]

2. Avoid extra language in the stem. Some think extraneous details make a question more complex. However, they most often just add to the students' reading time. This reduces the number of questions you can put on a test, therefore reducing the reliability of the test. For example, in the Tuckman question above, the information on Poole's model had nothing to do with the information sought by the question.

3. Include any language in the stem that you would have to repeat in each answer option. For example, a stem such as "Biology is defined as the scientific study of:" keeps you from having to repeat "is the scientific study of" at the beginning of each option.

[Top]

Answer Options

1. Avoid lifting phrases directly from text or lecture. This becomes a simple recall activity for the student. Use new language as frequently as possible.

2. Most literature recommends writing the correct answer before writing the distracters. This makes sure you pay enough attention to formulating the one clearly correct answer.

3. Answer options should be about the same length and parallel in grammatical structure. Too much detail or different grammatical structure can give the answer away.

For example, the specificity and grammatical structure of the first option here are dead give-aways:

The term "side effect" of a drug:

a) refers to any action of a drug in the body other than the one the doctor wanted to drug to have.b) is the chain effect of a drug.c) additionally benefits the drug.

4. Limit the number of answer options. Research shows that three-choice items are about as effective as four-choice items. Four choice items are the most popular, and never give more than five alternatives.

5. Distracters must be incorrect, but plausible. If you can, include among the distracters options that contain common errors. Students will then be motivated to listen to your explanations of why those options are incorrect.

6. To make distracters more plausible, use words that should be familiar to students.7. If a recognizable key word appears in the correct answer, it should appear in some or all

of the distracters as well. Don't let a verbal clue decrease the accuracy of your exam.

For example, someone with no biology background would not have to think very hard to make a correct guess on this question:

Every organism is made of cells and every cell comes from another cell. This is the:

a) Relativity Theoryb) Evolution Theory c) Heat Theory d) Cell Theory

8. Help students see crucial words in the question. For example: "Which of the following is NOT an explicit norm?" Likewise, when you ask a similarly-worded question about two different things, always highlight the difference between the questions.

9. It is often difficult to come up with 3 or 4 plausible distracters, and teachers will sometimes add some that are not plausible, or even humorous. Be careful. If it is too easy to eliminate one or two options, then the question loses much of its measurement value. If energy or time is limited and you must come up with one more distracter, consider either

offering a true statement that does not answer the question and/or a jargon-ridden option that is meaningless to someone who understands the concept.

10. Use Rarely: o Extreme words like "all," "always" and "never" (generally a wrong answer). o Vague words or phrases like "usually," "typically" and "may be" (generally a

correct answer). o "All of the above" - eliminating one distracter immediately eliminates this, too. o "None of the above" - use only when the correct answer can be absolutely correct,

such as in math, grammar, historical dates, geography, etc.. Do not use with negatively-stated stems, as the resulting double-negative is confusing. Studies do show that using "None of the above" does make a question more difficult, and is a better choice when the alternative is a weak distracter.

Effective Multiple Choice Questions:Guidelines for Writing Multiple Choice Questions:

Constructing good multiple choice items requires plenty of time for writing, review, and revision. If you write a few questions after class each day when the material is fresh in your mind, the exam is more likely to reflect your teaching emphases than if you wait to write them all later. Writing questions on three-by-five index cards or in a word-processing program will allow you to re-arrange, add, or discard questions easily.The underlying principle in constructing good multiple choice questions is simple: the questions must be asked in a way that neither rewards "test wise" students nor penalizes students whose test-taking skills are less developed. The following guidelines will help you develop questions that measure learning rather than skill in taking tests.Writing the Stem:The "stem" of a multiple-choice item poses a problem or states a question. The basic rule for stem-writing is that students should be able to understand the question without reading it several times and without having to read all the options.

a. Write the stem as a single, clearly-stated problem. Direct questions are best, but incomplete statements are sometimes necessary to avoid awkward phrasing or convoluted language.

b. State the question as briefly as possible, avoiding wordiness and undue complexity. In higher-level questions the stem will normally be longer than in lower-level questions, but you should still be brief.

c. State the question in positive form because students often misread negatively phrased questions. If you must write a negative stem, emphasize the negative words with underlining or all capital letters. Do not use double negatives--e.g., "Which of these is not the least important characteristic of the Soviet economy?"

Writing the Responses:Multiple-choice questions should usually have either four or five options to make it difficult for students to guess the correct answer. The basic rules for writing responses are:

a. Students should be able to select the right response without having to sort out complexities that have nothing to do with knowing the correct answer and

b. They should not be able to guess the correct answer from the way the responses are written.

c. Therefore: Write the correct answer immediately after writing the

stem and make sure it is unquestionably correct. In the case of "best answer" responses, it should be the answer that authorities would agree is the best.

Write the incorrect options to match the correct response in length, complexity, phrasing, and style. You can increase the believability of the incorrect options by including extraneous information and by basing the distractors on logical fallacies or common errors, but avoid using terminology that is completely unfamiliar to students.

Avoid composing alternatives in which there are only microscopically fine distinctions between the answers, unless the ability to make these distinctions is a significant objective in the course.

Avoid using "all of the above" or "both A & B" as responses. These options make it possible for students to guess the correct answer with only partial knowledge.

Furthermore, these types of responses make it all but impossible to randomize responses.

Use the option "none of the above" with extreme caution. It is only appropriate for exams in which there are absolutely correct answers, like math tests, and it should be the correct response about 25% of the time in four-option tests.

Again, you cannot randomize responses when using the "none of the above" response option.

Avoid giving verbal clues that give away the correct answer. These include: grammatical or syntactical errors; key words that appear only in the stem and the correct response; stating correct options in textbook language and distractors in everyday language; using absolute terms--e.g., "always, never, all," in the distractors; and using two responses that have the same meaning.

General Issues

a. Base each question on student learning objectives, not trivial information.

b. All questions should stand on their own. o Avoid using questions that depend on knowing the answers to

other questions on the test. o Also, check your exam to see if information given in some items

provides clues to the answers on others. c. Develop a "randomly generated " exam from a question pool.

o The question pool should contain at least three questions for each question used in the exam.

So, a 15-question exam should be generated from a pool of at least 45 questions.

d. Randomize the position of the correct responses. o Placing responses in alphabetical order will usually do the job. o Remember, you can not randomize responses containing the

following options: All of the above. None of the above. A and B above. Etc.

C2.1 Level 1: KnowledgeAt this level, one simply requires the recall of acquired knowledge. WARNING! A test at this

level can easily become a "Trivial Pursuit" exercise!

Example C2.1.1Which one of the following persons is the author of "Das Kapital"?

1. Mannheim2. Marx3. Weber4. Engels5. Michels

Note that the responses are internally consistent - they are all the names of Germans whose

written work have been major contributions on social issues.

Example C2.1.2In the area of physical science, which one of the following definitions describes the term "polarization"?

1. The separation of electric charges by friction.2. The ionization of atoms by high temperatures.3. The interference of sound waves in a closed chamber.4. The excitation of electrons by high frequency light.5. The vibration of transverse waves in a single plane.

Simple recall of the correct definition of polarization (#5) is required. Internal consistency and

plausibility are maintained in that all responses are actual physical phenomena.

Example C2.1.3 According to the microgenesis of perception concept, the threshold of awareness consists of a hierarchy of thresholds. Which one of the sequences shown below is correct?

1. Recognition thresholds > physiological thresholds > detectionthresholds.

2. Physiological thresholds > detection thresholds > recognition thresholds.

3. Physiological thresholds > recognition thresholds > detection thresholds.

4. Recognition thresholds > detection thresholds > physiological thresholds.

In this example, nothing more is required than the recall of the order of certain pieces of related

information. The correct answer is #2.

C2.2 Level 2. ComprehensionAt this level, knowledge of facts, theories, procedures etc. is assumed, and one tests for

understanding of this knowledge.

Example C2.2.1 Which one of the following describes what takes place in the so-called PREPARATION stage of the creative process, as applied to the solution of a particular problem?

1. The problem is identified and defined.2. All available information about the problem is collected.3. An attempt is made to see if the proposed solution to the problem is acceptable.4. The person goes through some experience leading to a general idea of how the problem can be solved.5. The person sets the problem aside, and gets involved with some other unrelated activity.

In this question, the knowledge of the five stages of the creative process must be recalled

(KNOWLEDGE), and one is tested for an understanding (COMPREHENSION) of the meaning

of each term, in this case, "preparation".

Note that this question violates the rule that the answer and distractors should all be of about the

same length. It is difficult to get around this one here, so the text is edited so that each line is

about the same length.

C2.3 Level 3: Application

In order to classify a question into this group, ask yourself if prior knowledge of the background

to the question is assumed to be both known and understood, and whether one is merely expected

to apply this knowledge and understanding. Calculations based on known formulae are good

examples of this, as shown in the example below:

Example C2.3.1 Which one of the following values approximates best to thevolume of a sphere with radius 5m?

a. 2000m³b 1000m³c. 500m³d 250m³e. 125m³

In order to answer this question, the formula 4[pi]r³ /3 must be known (recall of knowledge) and

the meaning of the various symbols in the formula understood (comprehension) in order to

answer this question. The correct answer is #3.

Example C2.3.2 Which one of the following memory systems does a piano-tunermainly use in his occupation?

1. Echoic memory.2. Short-term memory.3. Long-term memory.4. Mono-auditory memory.5. None of the above.

This is clearly a case of testing for the application of previously acquired knowledge (the various

memory systems), which is also understood, as the meaning of each term must be clear before

the student can decide whether it is applicable to the given situation. The correct answer is #1.

Note that students may not necessarily know what a piano- tuner is or does. Watch out for

cultural bias!

The next example is more difficult to classify:

Example C2.3.3 You are the sole owner and manager of a small enterpriseemploying 15 workers. One of these, Alfred, (who has beenworking for you for the past year and has somewhat of ahistory of absenteeism), arrives late for work one Wednesdaymorning, noticeably intoxicated. Which one of the followingactions is the most appropriate in the circumstances?

1. You terminate Alfred's employment on the spot, paying him the wages still due to him. 2. You parade Alfred in front of the other workers, to teach them

all a lesson. 3. You give Alfred three weeks' wages in lieu of notice, and sack him. 4. You wait until Alfred is sober, discuss his problem, and give him a final written warning, should it be required. 5. You call Alfred's wife to take him home and warn her that this must not happen again.

Note that this this question is classified as APPLICATION as in order to answer it, the relevant

labour legislation should be known and understood. One could made a case for it to have a

higher classification such as EVALUATION, on the grounds that one is asked to evaluate which

one of the proposed actions is the best in the circumstances, or ANALYSIS, on the grounds that

in order to select the most appropriate answer, one should analyse the possible outcomesof each

decision. For both these levels, one would expect a greater amount of information as to Alfred's

situation, the relationship between Alfred and his co-workers, union involvement in the

enterprise etc., and have a more sophisticated set of distractors. Here, option #4 is clearly the best

both on legal and human terms. Note that the figure of speech "on the spot" may not be

understood by second- language students. Use suitable language!

C2.4 Level 4: AnalysisExample C2.4.1 "The story is told of the famous German Organic ChemistAuguste Kékulé who was struggling with the problem of howthe six carbon atoms of benzene were linked together. He wasgetting nowhere with the problem, and one day fell asleep infront of the fireplace while he was pondering on it. Hedreamt of molecules twisting and turning around like snakes.Suddenly, one of the snakes swallowed its own tail androlled around like a hoop. Kékulé woke up with a start, andrealized that his problem could be solved if the six carbonatoms of benzene were attached to each other to form a ring.Further work showed that this was entirely correct."

The above passage illustrates a particular phase of thecreative process. Which one is it?

1. preparation 2. incubation 3. orientation 4. illumination 5. verification

In the above example, the student is expected to know and understand the five stages of the

creative process, and to apply this knowledge to an important factual example of creative

thinking (the elucidation of the chemical structure of the benzene molecule). The ability to

analyse the data (i.e. the given text) in terms of each of the five stages is what is being tested.

The correct answer, by the way, is #4.

Example C2.4.2 (Assume the question below is asked in a philosophy test.) Read carefully through the paragraph below, and decide whichof the options 1-5 is correct.

"The basic premise of pragmatism is that questions posed byspeculative metaphysical propositions can often be answeredby determining what the practical consequences of theacceptance of a particular metaphysical proposition are inthis life. Practical consequences are taken as the criterionfor assessing the relevance of all statements or ideas abouttruth, norm and hope."

1. The word "acceptance" should be replaced by "rejection". 2. The word "often" should be replaced by "only". 3. The word "speculative" should be replaced by "hypothetical". 4. The word "criterion" should be replaced by "measure".

This question requires prior knowledge of and understanding about the concept of pragmatism.

The paragraph, seen in this light, contains one word which vitiates its validity, and the student is

tested on his/her ability to analyze it to see whether it fits with the accepted definition of

pragmatism. With this in mind, #2 is correct. Option #1 would degrade the paragraph further,

while #3 and #4 would simply result in changing to acceptable synonyms. Note that this question

does not address Level 6 (Evaluation), as one is not asked to pass a value judgement on the text.

This must be considered as a very difficult question, and will obviously require a high level of

reading skills. Bear in mind that there will be a significant time factor involved.

Example C2.4.3 Look at the following table and indicate which countries'statistics are being reported in rows A, B and C.

GNP per capita 1991 ($ USA)

Growth rate of GNP per capita p.a. 1980-91

Population growth rate 1980-91

Structures of total employment 1980-85 (percentages) Agriculture Industry Services

A 500 2,5% 1,5% 51 20 29 B 1570 5,8% 1,6% 74 8 8

S.A. 2560 0,7% 2,5% 17 36 36 C 25110 1,7% 0,3% 6 32 32

Choose your answer from the following list of possibleanswers:

1. A is South Korea; B is Kenya; C is Canada. 2. A is Sri Lanka; B is Germany; C is Thailand. 3. A is Sri Lanka; B is Thailand; C is Sweden.

4. A is Namibia; B is Portugal; C is Botswana.

In order to answer this question, students must be able to recall the relative economic rankings of

various countries (KNOWLEDGE) and understand the basis for such a ranking

(COMPREHENSION). They must be able to apply these concepts when information is supplied

to them (APPLICATION), and they must be able to ANALYZE the given information in order

to answer the question. Students did not like this question when they were faced with it in a class

test, as their immediate reaction was that "it was impossible to remember the statistics for all the

countries that were discussed in class and given to them in handouts". They were surprised when

told that such detailed knowledge was in fact not expected of them, but that they were to

examine the table and perform a ranking on the basis of concepts that they should have mastered.

The correct answer is 3.

C2.5 Level 6: EvaluationAt this level, one is asked to pass judgement on, for example, the logical consistency of written

material, the validity of experimental procedures or interpretation of data.

Example C2.5.1 A student was asked the following question: "Briefly listand explain the various stages of the creative process".

As an answer, this student wrote the following:

"The creative process is believed to take place in fivestages, in the following order: ORIENTATION, when theproblem must be identified and defined, PREPARATION, whenall the possible information about the problem iscollected, INCUBATION, when there is a period where nosolution seems in sight and the person is often busy withother tasks, ILLUMINATION, when the person experiencesa general idea of how to arrive at a solution to theproblem, and finally VERIFICATION, when the persondetermines whether the solution is the right one for theproblem."

How would you judge this student' s answer?

1. EXCELLENT (all stages correct in the right order with clear and correct explanations) 2. GOOD (all stages correct in the right order, but the explanations are not as clear as they should be). 3. MEDIOCRE (one or two stages are missing OR the stages are in the wrong order, OR the explanations are not clear OR the explanations are irrelevant) 4. UNACCEPTABLE (more than two stages are missing AND the order is incorrect AND the explanations are not clear AND/OR they are irrelevant)

In the above question, one is expected to make value judgment on the content of the given text

(KNOWLEDGE of the subject is required), the meaning of the terminology used

(COMPREHENSION of the subject matter), and its structure (ANALYSIS of the answer for the

right order of events. The correct answer here is #1, but suitable modification of the putative

student answer could provide a small bank of questions with other correct answers

Example C2.5.2

Another example is the "Assertion/Reason" question, in which two statements linked by

"BECAUSE" have to be evaluated in the light of certain criteria:

Judge the sentence in italics according to the criteriagiven below:

"The United States took part in the Gulf War against IraqBECAUSE of the lack of civil liberties imposed on the Kurdsby Saddam Hussein's regime."

a. The assertion and the reason are both correct, and the reason is valid. b. The assertion and the reason are both correct, but the reason is invalid. c The assertion is correct but the reason is incorrect. d. The assertion is incorrect but the reason is correct. e. Both the assertion and the reason are incorrect.

The correct answer is "b", since while it is true that the United States took part in the Gulf War, it

is also true that the Kurds in Iraq did not (and still do not) enjoy an abundance of civil liberties,

but the threat to the US's oil supply as a result of Iraq's invasion of Kuwait was a much more

pertinent reason for the United States joining in the fray. A knowledge and understanding of

Middle East politics is assumed. What is tested here is the ability to evaluate the between cause

and effect in the sentence in terms of predefined criteria.

When designing items you should ask yourself which of the followinglevels your ILO addresses and design the item to fit.1. Remembering: Test of fact, basic concepts, recall of specificinformationKey words: What, select, where….This is the simplest item to devise2. Understanding: Test of understanding - more than recall,advanced conceptsKey words: Why, how, when.

3. Applying: applies knowledge in a new relevant situation/contextKey words: Choose, select, identify, which, what….These can be scenario or case questions.4. Analysing: Tests ability to make a judgement on the basis ofinformation presentedKey words: Decide, determine, evaluate, recommend, interpret,prioritise, confirm.These also can be scenario or case questions.5. Evaluating: Tests ability to come to a conclusion giveninformation presentedKeywords: choose, conclude, decide, determine, evaluate, judge,rate, assess, value, estimate.

Before writing, ensure that• the MCQ format suits the purpose for assessment• each item matches with one or more ILOs for the subject• you use clear, direct and simple languageIn writing stems, ensure that they• are clear and unambiguous and so do not rely on high levelcomprehension skills just to understand the question• have only relevant text• do not use negatives if at all possible• have as much of the question as possible, so it is notduplicated in the options. It is a good idea to write stems asfull sentences.In writing the options and the distracters, ensure• there is only one correct answer• the distracters are credible but not too close to the correctanswer to be confusing• to avoid ‘all are correct’ ‘none is correct’ choices• they are culturally appropriate• they are research/evidence-based, representing typicalmisconceptions, preconceptions, or wrong answers.

The following are suggestions for writing successful questions.

DO DON'T

Before you start writing each question, identify what it is you are trying to test - knowledge? understanding?

Give away answers to a previous question with a later question - look at your test as a whole.

Consider the use of alternative question formats if the simple format doesn't serve your purpose

Include superfluous or irrelevant information in the stem.

Phrase questions as clearly and concisely as possible, avoiding complex language.

Compose stems which give clues to the key.

Write a stem which clearly specifies what the question is asking for.

Use negatives in the stem too often.

Include common information in the stem rather than repeating it in the options.

Write a key that is longer, more complex or more specific than the distracters.

Ensure any negatives in the stem are clearly highlighted.

Have more than one key (unless this is specified to students).

Make your distracters as plausible as possible.

Forget to randomise the location of the key within the options.

Focus on common student mistakes as areas for questions and/or distracters.

Use questions without piloting them on colleagues first.

General tips

There is no set rule about the number of options you should include. The greater the

options the smaller the mathematical chance of correct guesswork, but research shows

that questions with more options are not necessarily harder (see McKenna and Bull,

1999, p8), so you may be making unnecessary extra work for yourself. Be cautious

about using "all of the above" or "none of the above" to increase the number of

distracters. "All of the above" gives too much help to students with limited knowledge,

because it must be the correct answer if two of the distracters are true. "None of the

above" is less problematic (although it does not test that students actually know the

correct answer, only that they can eliminate wrong ones), but avoid over-using it and

make sure it is sometimes the correct answer, rather than an add-on because you are

stuck for a final distracter!

Avoiding common errors

The following are some 'poor' questions with suggestions as to how they can be

improved. For a thorough guide to drafting MCQs see McKenna and Bull (1999) and

Kehoe (1995).

Question 1: Which one of the following sections provides for the appointment of new trustees?

a. Section 40 of the Trustee Act 1925 b. Section 39 of the Trustee Act 1925 c. Section 36 of the Trustee Act 1925 d. Section 53 of the Trustee Act 1925

Here information is repeated in all the options needlessly. An improved question would

place common information ("of the Trustee Act 1925") in the stem rather than the

options. Note how the stem emphasises that there is only one correct answer.

Question 2: Equity looks toa. Overrule the common law b. Intent rather than form c. Establish its superiority d. The common law

This stem is vague. The purpose of the question could be made clearer by changing the

stem to "Complete the following equitable maxim. Equity looks to"

Question 3: Which one of the following is NOT an equitable maxim?a. Equity looks to intent rather than form b. Equity deserves obedience c. He who comes to Equity must come with clean hands d. Delay defeats equity

An acceptable question in itself but gives clues to the previous question. Always check

your test as a whole for hidden give-aways. Note how the negative in the stem is clearly

identified.

Question 4: Bill secretly wants to provide financially for his mistress Brenda when he is dead without his wife finding out. Which of the following types of trust is the BEST way for Bill to achieve his wishes?

a. A resulting trust b. A secret trust c. A constructive trust d. A purpose trust

This question gives clues to the answer in the stem of the question.

Question 5: In which one of the following cases was Mr Hunter the settlor?a. Oughtred v IRC b. Grey v IRC c. Vandervell v IRC d. Broadway Cottages v IRC

This is a clearly phrased factual knowledge-based question. But how important is it that

students know the answer to this? Questions should relate to the intended learning

outcomes. In addition, an alert student might well have noticed by now the question-

setter's preference for (b) as the key and correctly choose Grey v IRC! Putting your

options in strict alphabetical order achieves randomisation and minimises the potential

for students to 'calculate' mathematically whether the answer 'must be' a particular

option.

Drafting questions for different skillsKnowledge-based questions:

useful for testing recall of factual points like terms, case names, section numbers and procedures, for example: Which statement best defines the term...? Which case is authority for the principle that...? Which statement best expresses the principle of...?

can be combined with comprehension/application questions or used on their own to test 'basics'

particularly useful as introductory questions in an MCQ test can vary in difficulty, but avoid increasing the difficulty by setting questions on

obscure points beyond the intended learning outcomes

Comprehension questions:

useful for testing students' ability to interpret or summarise primary and secondary material

can be used to encourage students to distinguish cases on their facts (see example A)

the stem can be in the form of an analogy (see example B) to test students' ability to compare and contrast material

especially valuable as a formative tool with feedback

Application questions:

useful to test students ability to apply knowledge to given scenarios (see example C)

can be used in sequence from one problem scenario with increasing complexity (see example D)

can vary in difficulty from comparatively straightforward to complex situations

can be used effectively in both formative and summative assessments

Example A: In relation to each of the cases listed below, how would you classify the detriment on which the claimant relied?

a. Some direct financial contributions, but mainly sought to rely on indirect financial contributions

b. Some indirect financial contributions, but mainly sought to rely on non-financial contributions

c. Indirect financial contributions only d. Non-financial contributions only

1. Burns v Burns 2. Midland Bank v Cooke 3. Eves v Eves 4. Lloyd's Bank v Rosset 5. Grant v Edwards 6. Cooke v Head

Example B: Within the Law of Property Act 1925, Section 53(1)(b) is to declarations of trusts of land as Section 53(1)(c) is to:

a. Declarations of trusts of personality b. Declarations of implied interests c. Dispositions of secret trusts d. Dispositions of equitable interests

Example C: Bill and Ben are trustees of a large trust fund. The beneficiaries, Charles and Camilla are contingently entitled to the fund. Bill and Ben wish to use income from the fund to pay for Charles and Camilla to attend private school. Which section of the Trustee Act 1925 would allow them to do so?

a. Section 31 b. Section 32 c. Section 52 d. Section 53

Example D: Raj and Sophie, who have never married, have two children; Ben aged 8 and Shazia aged 2. Raj and Sophie's relationship has ended, and Sophie has married Carlton. Raj has agreed that the children should live with Sophie and Carlton for the time being. For questions 1 - 8, the options are:

a. Raj and Sophie only b. Raj, Sophie and Carlton c. Sophie and Carlton only

d. Sophie only e. Raj only

1. Who has parental responsibility for the children at present?2. If section 8 orders are required in respect of the children, who could apply as of right (without leave) for any section 8 order?3. And who would be able to apply as of right (without leave) for a residence or contact order?4. If Raj obtained a contact order to see the children every week, who would have parental responsibility for the children? It is two years later. Ben is now aged 10 and Shazia is aged 4. Raj obtains a residence order so that Ben can live with him. Sophie and Carlton adopt Shazia. 5. Who now has parental responsibility for Ben?6. Who now has parental responsibility for Shazia? It is three years later. Sophie and Carlton divorce. Sophie and Raj resume their relationship and get married. 7. Who now has parental responsibility for Ben?8. Who now has parental responsibility for Shazia? Alternative question formats

Variations on the simple MCQ can provide more scope for testing particular skills and/or

to increase difficulty. With alternative question formats it is especially important to give

students the opportunity to 'practice' before completing a summative assessment. The

most common variation is the combined response question, for example:

Which two of the following requirements are necessary to create a valid fully secret trust?

1. The terms must be communicated to the trustee 2. The beneficiaries must be named in the terms of the will 3. The terms must be communicated to the beneficiaries 4. The trustee must accept the terms of the trust

a. 1 and 2 b. 3 and 4 c. 1 and 4 d. 2 and 3

For other alternative question formats, see McKenna and Bull (1999) and Kehoe (1995).

Writing Stems

We will first describe some basic rules for the construction of multiple-choice stems, because they are typically, though not necessarily, written before the options.

1. Before writing the stem, identify the one point to be tested by that item. In general, the stem should not pose more than one problem, although the solution to that problem may require more than one step.

2. Construct the stem to be either an incomplete statement or a direct question, avoiding stereotyped phraseology, as rote responses are usually based on verbal stereotypes. For example, the following stems (with answers in parentheses) illustrate undesirable phraseology:

What is the biological theory of recapitulation? (Ontogeny repeats phylogeny)

Who was the chief spokesman for the "American System?" (Henry Clay)

Correctly answering these questions likely depends less on understanding than on recognizing familiar phraseology.

3. Avoid including nonfunctional words that do not contribute to the basis for choosing among the options. Often an introductory statement is included to enhance the appropriateness or significance of an item but does not affect the meaning of the problem in the item. Generally, such superfluous phrases should be excluded. For example, consider:

The American flag has three colors. One of them is (1) red (2) green (3) black versus One of the colors of the American flag is (1) red (2) green (3) black

In particular, irrelevant material should not be used to make the answer less obvious. This tends to place too much importance on reading comprehension as a determiner of the correct option.

4. Include as much information in the stem and as little in the options as possible. For example, if the point of an item is to associate a term with its definition, the preferred format would be to present the definition in the stem and several terms as options rather than to present the term in the stem and several definitions as options.

5. Restrict the use of negatives in the stem. Negatives in the stem usually require that the answer be a false statement. Because students are likely in the habit of searching for true statements, this may introduce an unwanted bias.

6. Avoid irrelevant clues to the correct option. Grammatical construction, for example, may lead students to reject options which are grammatically incorrect as the stem is stated. Perhaps more common and subtle, though, is the problem of common elements in the stem and in the answer. Consider the following item:

What led to the formation of the States' Rights Party? a. The level of federal taxation b. The demand of states for the right to make their own laws c. The industrialization of the South d. The corruption of federal legislators on the issue of state taxation One does not need to know U.S. history in order to be attracted to the answer, b.

Other rules that we might list are generally commonsensical, including recommendations for independent and important items and prohibitions against complex, imprecise wording.

Writing Options

Following the construction of the item stem, the likely more difficult task of generating options presents itself. The rules we list below are not likely to simplify this task as much as they are intended to guide our creative efforts.

1. Be satisfied with three or four well constructed options. Generally, the minimal improvement to the item due to that hard-to-come-by fifth option is not worth the effort to construct it. Indeed, all else the same, a test of 10 items each with four options is likely a better test than a test with nine items of five options each.

2. Construct distractors that are comparable in length, complexity and grammatical form to the answer, avoiding the use of such words as "always," "never," and "all." Adherence to this rule avoids some of the more common sources of biased cueing. For example, we sometimes find ourselves increasing the length and specificity of the answer (relative to distractors) in order to insure its truthfulness. This, however, becomes an easy-to-spot clue for the testwise student. Related to this issue is the question of whether or not test writers should take advantage of these types of cues to construct more tempting distractors. Surely not! The number of students choosing a distractor should depend only on deficits in the content area which the item targets and should not depend on cue biases or reading comprehension differences in "favor" of the distractor.

3. Options which read "none of the above," "both a. and e. above," "all of the above," _etc_., should be avoided when the students have been instructed to choose "the best answer," which implies that the options vary in degree of

correctness. On the other hand, "none of the above" is acceptable if the question is factual and is probably desirable if computation yields the answer. "All of the above" is never desirable, as one recognized distractor eliminates it and two recognized answers identify it.

4. After the options are written, vary the location of the answer on as random a basis as possible. A convenient method is to flip two (or three) coins at a time where each possible Head-Tail combination is associated with a particular location for the answer. Furthermore, if the test writer is conscientious enough to randomize the answer locations, students should be informed that the locations are randomized. (Testwise students know that for some instructors the first option is rarely the answer.)

5. If possible, have a colleague with expertise in the content area of the exam review the items for possible ambiguities, redundancies or other structural difficulties. Having completed the items we are typically so relieved that we may be tempted to regard the task as completed and each item in its final and permanent form. Yet, another source of item and test improvement is available to us, namely, statistical analyses of student responses.

Duplication

Put as much of the question in the stem as possible, rather than duplicating material in each of the options. (Gronlund 1988)

A : Weak question Theorists of pluralism have asserted which of the following?

a. The maintenance of democracy requires a large middle class. b. The maintenance of democracy requires autonomous centres of

contervailing power. c. The maintenance of democracy requires the existence of a multiplicity of

religious groups. d. The maintenance of democracy requires a predominantly urban population.

e. The maintenance of democracy requires the separation of governmental powers.

B : Improved question Theorists of pluralism have asserted that the maintenance of democracy requires

a. a large middle class b. autonomous centres of contervailing power c. the existence of a multiplicity of religious groups d. a predominantly urban population

e. the separation of governmental powers

Basic Tips for Writing Effective Multiple Choice Questions (MCQ’s):A Compilation of the Most Useful AdviceJennifer Murdock, Department of Economics, U of TTo define terms, here is a sample multiple choice question where the correct answer is (E):(1) List the alternatives vertically and in a logical order. These two points are illustrated bythe sample multiple choice question above. Students who know the correct answer canquickly find it. Also, a jumbled order can be distracting to some students.(2) Make the question as direct and clear as possible. Questions can degenerate into a testof logic if you use double negatives and complex structures.(3) Make sure there is only one unambiguously correct response.(4) Create plausible distracters that would sound right to an incompetent student but areclearly wrong. You can include common misconceptions, common mistakes, andtechnical jargon to make distracters more effective.(5) Instruct students to choose the “best answer” rather than “the correct answer.” This helpsavoid argument and allows you to skip specifying rounding conventions (for example, theanswer to the sample question above was 23.1678…).(6) Write the question stem such that students can quickly and easily deduce what thequestion is asking. Students should not have to read all of the alternatives to figure thisout. When possible, write stems as you would an open-ended question such that if aproficient student were given the question stem s/he could write out the correct response.(7) Don’t get too fancy with your English and the subtleties of the English language. If you do,students who do not have English as their first language may get your question wrongeven though they knew the concept you were trying to test.(8) Use negatives sparingly and emphasize them if used. For example, “For which of thefollowing population density functions is the population mean NOT equal to the populationmedian?” There is some evidence that ESL students are particularly disadvantaged bynegatives.(20) If marks are normally distributed with mean68 and standard deviation 15, what percent ofstudents have a mark of 79 or higher?(A) 15(B) 17(C) 19(D) 21(E) 23Question StemDistractersAnswerAlternatives(9) While you may want to emphasize words in the question stem to help comprehension,emphasis in the alternatives should be avoided.(10) Keep the question stem and alternatives as short as possible. Use few words. Avoid

repeating words from the question stem in the alternatives.(11) Before writing a question, think about what it is that you want to test. Lecture notes,textbook readings, assigned problems, and other course materials can be inspiration.(12) Use “None of the above” with caution. You can make straightforward numeric calculationquestions more challenging by including “None of the above” as an option (may need tospecify rounding conventions). For other types of questions, you need to think carefullyabout whether there are some plausible arguments a proficient student could make tosupport choosing this alternative when you intended it as a wrong answer. If you want tosave time writing questions don’t use “None of the above.”(13) Do not include alternatives such as “Both (A) and (D)” or “All but (C)” as these complicatethe structure of the question and tend to confuse students and/or slow them down. If youwant to convince yourself, look at someone else’s questions that use these and see howmuch harder it is to focus on what you’re supposed to be doing.(14) Be aware of the difficulty level of each question. Make sure you have a sufficient numberof easy and more challenging questions so that you will be able to separate “F” studentsfrom “D” students, “D” from “C” students, “C” from “B” students, and “B” from “A” students.Easier questions test a student’s knowledge. For example, do they know what selectionbias is. Medium difficultly questions test comprehension. Does a student understandunder what conditions selection bias might arise? Harder questions test a student’s abilityto apply concepts and do analysis. For example, give students a scenario where theyneed to realize that selection bias would be a concern (without being told) and tounderstand the implications of that bias on inference in that case.(15) Try to make the first few multiple choice questions relatively quick and easy to help calmstudent down so they can focus on the more challenging questions to come.(16) Avoid the temptation to test many things in one question. If it is possible, try to write morethan one multiple choice question rather than test multiple concepts in one question.Testing too many things in one question reduces your ability to discriminate amongststudents with differing levels of understanding. Further, students get upset because thereis no partial credit.(17) Ask more than one question when a fair amount of information must be provided as ittakes time for students to carefully read and understand the information you provide in atest. For example, you could give them a table of results, a graph, or a scenario and thenask two or three different multiple choice questions about it.(18) Five alternatives (A) – (E) are recommended. You cannot include more than five with ourScantron forms. You could include only three or four, but this increases the expectedvalue of guessing. There is no reason all of your questions have to have the same numberof alternatives.Basic Tips for MCQ’s (July 20, 2006) (19) This table shows some common strategies test-wise students, who are skilled test takersbut not proficient in the course material, can use to guess correct answers and how youcan respond. Some are adapted from Russell A. Dewey, “Writing Multiple Choice Itemswhich Require Comprehension” http://www.psywww.com/selfquiz/aboutq.htm.(20) Do not try to write the entire test in one day: it takes time, creativity and thought to writegood multiple choice questions. You could write a few each week as you teach thematerial or as you get ideas based on students’ questions or performance on homework.(21) Come back to the questions you’ve written a day or two later with a fresh eye.(22) Ask trusted TA or a colleague to try your questions to sort out any ambiguities (especiallyif you’re using these questions for a final examination).

Some of the most important tips for writing effective multiple-choice questions are:

Have most of the relevant information in the stem or “question”, and only list short answers - one word or short phrases - as options. In fact, the question should be answerable without looking at any of the options. Write questions which require students to reason out a solution rather than simply recall facts and figures. Use plausible distractors for the different options - the aim should be that each option would be selected by at least some students. Make sure all the options follow on grammatically from the question. Have an answer which would be agreed upon by a group of experts. If a group of experts would have difficulty reaching consensus on the correct answer, the question is not valid. Do not use vague terms such as rarely, often, commonly, frequently, usually - experts can’t agree on what these mean, let alone students.