writing test questions

49
Writing Assessment Questions 12/14/09 Melissa Chiaramonti

Upload: mgenta

Post on 13-May-2015

212 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Writing test questions

Writing Assessment Questions

12/14/09

Melissa Chiaramonti

Page 2: Writing test questions

Questions Improve the Learning Process

Slide 2

Know

led

ge/S

kill

Time

Reduce

Forg

ett

ing

Reduce

Forg

ett

ing

Cre

ate

s In

trig

ue

Str

ength

en

s M

em

ory

Questions….

Page 3: Writing test questions

Slide 3

Dr. Will Thalheimer research:

Learning Benefits from Questions Learning Benefit

Min Max

Pre-questions (to focus attention) 5% 40%

RepetitionsInitial

Subsequent30%15%

110%40%

Feedback 15% 50%

Retrieval Practice 30% 100%

Questions Spaced Over Time 5% 40%

100% 380%

Page 4: Writing test questions

Guiding Principles for Evaluation

1. Match the right question type to the right skill

2. Carefully construct your questions

3. Provide feedback to guide learning

4. Organize questions in a centralized item bank

5. Randomize choices and questions

6. Align delivery environment to assessment stakes (*)

7. Accommodate participant needs: blended delivery (*)

8. Analyze results and sharing with stakeholders

Page 5: Writing test questions

Questions to Ask Yourself

• What objectives will (should) I be assessing?

• What types of items will be included in the test? – 2i– 2a– 2b– 2c

• How long will the test be in terms of time and number of items?

Page 6: Writing test questions

Common question types

• True/False / Yes/No - the participant selects one of two potential choices

• Multiple choice - the participant selects one choice from a group of possible answers

• Multiple response - Similar to multiple choice except the participant can select one or more choices

• Numeric questions - a participant is prompted to enter a numeric value

• Fill in the blanks - participant is presented with a statement where one or more words are missing

Slide 6

Page 7: Writing test questions

Question Types verses Cognitive Levels of Learning

KnowledgeComprehension

Application AnalysisSynthesis Evaluation

2i, 2a 2b 2c

Multiple Choice (MC)True/False (TF)MatchingCompletion Short Answer

MCShort AnswerProblems EssayPerformance

MCShort AnswerEssay

Page 8: Writing test questions

Planning

1. Outline learning objectives or major concepts to be covered by the assessments• Test should be representative of objectives and material covered• Major Learner complaint: Tests don’t fairly cover the material that was supposed to be

canvassed on the test

2. Create a test blueprint – Currently working on

3. Create questions based on blueprint – Why we are here today– Match the question type with the appropriate level of learning

4. Organize questions and/or ideas by item types

5. Eliminate similar questions and organize questions logically

6. Validate the tests

7. Analyze the results

Page 9: Writing test questions

Don’ts

Avoid… Why

Avoid negatives, especially double negatives.

Can make questions confusing to the participant.

Avoid "All of the above" and use of "None of the above" with caution.

Make it much easier for a participant to eliminate obvious choices and “guess” an answer

Avoid vague pronouns (e.g. it, they). Can make questions ambiguous and confusing to the participant.

Avoid unnecessary modifiers, especially absolutes (e.g. always, never, all, none).

“Always” and “Never”, etc., can make it easier guess an answer

Slide 9

Page 10: Writing test questions

Don’ts

Avoid… Why

Avoid providing cues to correct answer in the stem.

Giving the answer away means the question will not effectively measure the skill/knowledge it was intended to measure

Avoid providing clues to the answer of one question in another question.

Giving the answer away means the question will not effectively measure the skill/knowledge it was intended to measure

Avoid conflicting alternatives. Can make questions ambiguous and confusing to the participant.

Slide 10

Page 11: Writing test questions

Do’s

Do Why

Keep stems and statements as short as possible and use clear, concise language.

Provides clarity to what is being asked; minimizes ambiguity

Use questions whenever possible (What, Who, When, Where, Why and How)

Provides clarity to what is being asked; minimizes ambiguity

Maintain grammatical consistency to avoid cueing (Parallel Structure).

Helps to avoid providing cues to the correct answer

List choices in a logical order. Helps to avoid providing cues to the correct answerOptional – in many cases you may wish to shuffle choices.

Slide 11

Page 12: Writing test questions

OVERVIEW OF THE QUESTION TYPES

Page 13: Writing test questions

Multiple Choice Items • Advantages:

– Extremely versatile-can measure the higher level mental processes (application, analysis, synthesis and evaluation)

– A compromise between a short answer/essay and T/F item – Can cover a wide range of content can be sampled by one test

• Disadvantages – Difficult to construct plausible alternative responses

Page 14: Writing test questions

14

Multiple Choice Question Construction

Stem: A 65-year-old man has difficulty rising from a seated position and straightening his trunk, but he has no difficulty flexing his leg.

Lead-in: Which of the following muscles is most likely to have been injured?

Options: A. Gluteus maximus* Answer

B. Gluteus minimus

C. Hamstrings

D. Iliopsoas

E. Obturator internus Distractors

Page 15: Writing test questions

15

Have a Distinctive Shape

Stem

Lead In

Option A (distractor)

Option B (answer)

Option C (distractor)

Option D (distractor)

Option E (distractor)

Page 16: Writing test questions

16

Poorly Shaped MC Question

Short stem & lead-in

a. Long option

b. Long option

c. Long option

d. Long option

Page 17: Writing test questions

Consider this question:

• Which of the following are not disadvantages of hybrid cars when compared to similar gasoline powered vehicles? (select all that apply)

A. Lower sticker priceB. Better fuel economyC. Lower carbon emissionsD. Better audio systems

Slide 17

Page 18: Writing test questions

Consider this question:

• Which of the following are not disadvantages of hybrid cars when compared to similar gasoline powered vehicles? (select all that apply)

A. Lower sticker priceB. Better fuel economyC. Lower carbon emissionsD. Better audio systems

Slide 18

Double Negative

Page 19: Writing test questions

Better wording...

• Which of the following are advantages of hybrid cars when compared to similar gasoline powered vehicles? (select all that apply)

A. Lower sticker priceB. Better fuel economyC. Lower carbon emissionD. Better audio systems

Slide 19

Page 20: Writing test questions

Consider this question:

• The President and Vice President of the United States are chosen by the Electoral College.

A. TrueB. False

Slide 20

Page 21: Writing test questions

Consider this question:

• “The President and Vice President of the United States are chosen by the Electoral College.”– Ambiguous question because...

• This is true to some degree, BUT...• You could make a strong argument for “False”, making the point that the

Electoral College is mainly a formality because electors are generally “pledged” (or “instructed who to vote for”) based on the results of the popular vote in their state.

Slide 21

Page 22: Writing test questions

Better wording:

• “The Electoral College meets every four years to cast the official votes for President and Vice President of the United States.”– There is no ambiguity here.

Slide 22

Page 23: Writing test questions

1. Write the stem first: a) Be sure the stem asks a clear question b) Stems phrased as questions are usually easier to write c) Stems should not contain a lot of irrelevant info.d) Be sure the stem is grammatically correcte) Avoid negatively stated stems (double negative risk)

How to Write Multiple Choice Questions

Page 24: Writing test questions

How to Write Multiple Choice Questions

2. Writing the correct response 1. Use same terms/reading level2. Avoid too many qualifiers 3. Assign a random position in the answer sequence

3. Read the stem and correct response together

4. Generate the distractors/alternative responses

Page 25: Writing test questions

Multiple Choice Items

• Other Tips for Constructing MC Items:– Items should have 3-4 alternatives. – Stem should present a single, clearly formulated problem– Simple, understandable, exclude extraneous words from

both stem and alternatives – Include in the stem any word that are repeated in each

response – Avoid all of the above (can answer based on partial

information)– Avoid none of the above

Page 26: Writing test questions

26

Consider this question:

• The primary purpose of the stam is to remove the– carm– denton– menace– stam bar

Page 27: Writing test questions

27

Consider this question:

• The primary purpose of the stam is to remove the– carm– denton– menace– stam bar

“Word repeats” in stem and answer

Page 28: Writing test questions

28

Consider this question:

• Which of the following pairs has won the greatest number of Abby awards?– Jones and Smith– Smith and Taylor– Smith and White– White and Allen

Page 29: Writing test questions

29

Consider this question:

• Which of the following pairs has won the greatest number of Abby awards?– Jones and Smith– Smith and Taylor– Smith and White– White and Allen

Convergence: correct answer has the most in common with other choices

Page 30: Writing test questions

30

Consider this question:

• How many pounds of pressure are exerted by a callam?

– 2.6– 150– 260– 2600

Page 31: Writing test questions

31

Consider this question:

• How many pounds of pressure are exerted by a callam?

– 2.6– 150– 260– 2600

Convergence: Correct answer has the most in common with other choices

Page 32: Writing test questions

32

Consider this question:

• The stanon is aided by a– instel– immon– octal– port

Page 33: Writing test questions

33

Consider this question:

• The stanon is aided by a– instel– immon– octal– port

Grammatical cues: only “port” follows grammatically from the stem

Page 34: Writing test questions

34

Consider this question:

• The stanon frequently overheats because– all grestels are bilious– no immon are directly fectitious– octals are usually casable– ports are always critical

Page 35: Writing test questions

35

• The stanon frequently overheats because– all grestels are bilious– no immon are directly fectitious– octals are usually casable– ports are always critical

Absolute terms: never found in the correct answer

Consider this question:

Page 36: Writing test questions

36

Consider this question:

• Stammation normally occurs when the– anstels rupture– immon falls and the denton is in place– octal rotates easily– ports pass over the carm

Page 37: Writing test questions

37

Consider this question:

• Stammation normally occurs when the– anstels rupture– immon falls and the denton is in place– octal rotates easily– ports pass over the carm

Longest answer is usually the correct answer

Page 38: Writing test questions

True-False Test Quedstions

• Best suited for testing 3 kinds of info.: • Knowledge level learning • Understanding of misconceptions• When there are two logical responses

• Advantages: – Sample a large amount of learning per unit of Learner

testing time

• Disadvantages: – Tends to be very easy– 50-50 chance of guessing – Tends to be low in reliability

Page 39: Writing test questions

Tips for Constructing True/False Items

–Tips for constructing True-False Items• Avoid double negatives• Avoid long or complex sentences• Specific determiners (always, never, only, etc.) should be

used with caution• Include only one central idea in each statement• Avoid emphasizing the trivial• Exact quantitative (two, three, four) language is better

than qualitative (some, few, many) • Avoid a pattern of answers

Page 40: Writing test questions

Short-Answer Items

• Two Types: (Question and Incomplete Statement)

• Advantages: – Easy to construct– Excellent format for measuring who, what, when, and where info.– Guessing in minimized– Student must know the material- rather than simply recognize the

answer

• Disadvantages: – Grading can be time consuming/difficult– More than one answer can be correct

Page 41: Writing test questions

Short Answer Items

• Tips for Constructing Short Answer Items – Better to supply the term and require a definition– For numerical answers, indicate the degree of precision

expected and the units in which they are to be expressed.– Use direct questions rather than incomplete statements– Try to phrase items so that there is only one possible

correct response– When incomplete statements are used, do not use more

than one blank within an item.

Page 42: Writing test questions

Essay Questions

Types of Essay Questions

• Extended Response Question– Great deal of latitude on how to respond to a question. – Example: Discuss essay and multiple-choice type tests.

• Restricted Response Question– More specific, easier to score, improved reliability and validity– Example: Compare and contrast the relative advantages of

disadvantages of essay and multiple choice tests with respect to: reliability, validity, objectivity, & usability.

Page 43: Writing test questions

Essay Items

• Advantages: – Measures higher learning levels (synthesis, evaluation) and

is easier to construct than an objective test item– Learners are less likely to answer an essay question by

guessing– Require superior study methods– Offer Learners an opportunity to demonstrate their abilities

to: • Organize knowledge• Express opinions• Foster creativity

Page 44: Writing test questions

Essay/Scenario Items

• May limit the sampling of material covered• Tends to reduce validity of the test

• Disadvantages– Subjective unreliable nature of scoring

• “halo effect” – good or bad student’s previous level of performance• Written expression• Handwriting legibility• Grammatical and spelling errors

• Time Consuming

Page 45: Writing test questions

Essay/Scenario Items • Give Learners a clear idea of the scope & direction intended

for the answer– Might help to start the question with the description of the required

behavior (e.g., compare, analyze)

• Appropriate language level for Learners

• Construct questions that require Learners to demonstrate a command of background info, but do not simply repeat that info.

• If question calls for an opinion, be sure that the emphasis is not on the opinion but on the way its presented or argued.

• Use a larger number of shorter, more specific questions rather than one or two longer questions so that more information can be assessed.

Page 46: Writing test questions

Essay/Scenario Items • You might

– Give Learners a pair of sample answers to a question of the type you will give on the test.

– Sketch out a rubric (grading scheme) for each question before reading the papers OR randomly select a few to read and make up the grading scheme based on those answers

– Give Learners a writing rubric – Detach identifying information and use code numbers instead to avoid

letting personality factors influence you. – After grading all the papers on one item, reread the first few to make

sure you maintained consistent standards – Be clear to student the extend to which factors other than content (e.g.,

grammar, handwriting, etc.) will influence the grade.

Page 47: Writing test questions

Essay Questions

• Tips for constructing Essay Questions – Provide reasonable time limits for each question

• “thinking and writing time” – Avoid permitting Learners a choice of questions

• Will not necessarily get a representative sample of student achievement. Only be requiring all Learners to answer all questions can their achievement be compared

– A definite task should be put forth to the student• Critical words: compare, contrast, analyze, evaluate, etc.

Page 48: Writing test questions

Scoring Essay Items

• Write an outline of the key points (use outline to design a rubric)

• Determine how many points are to be assigned to the question as a whole and to the various parts within it.

• If possible, score the test without knowledge of the student’s name – Face Sheet

• Score all of the answers to one question before proceeding to the next question – Consistent standard

Page 49: Writing test questions

Questions?