day2 ses1_test items and rubrics

7/27/2019 Day2 Ses1_Test Items and Rubrics

1/48

Creating a Table ofSpecifications A table of specifications is simply a means of connecting learning

objectives, instructional activities and assessment. The followingsteps will guide you in the creation of a table of specifications:

Develop learning objectives based on the taxonomy ofeducational objectives

Identify instructional activities that target the learning objectives Implement instructional activities

Reflect on instructional activities and identify relevant learningobjectives that will be assessed based on the instructionalexperience

Determine the relative importance and weighting of eachobjective

Generate test items based on the designated learning objectives

http://www.clemson.edu/assessment/assessmentpractices/referencematerials/documents/Blooms%2

0Taxonomy%20Action%20Verbs.pdf


2/48

Developing TestItems and Rubrics

Day 2 Session 1


3/48

Warmer: Find someone who

You cannot use the same name twice!

Find someone who:

1. drank coffee this morning. _____________

2. watched a TV movie yesterday. _____________

3. grew up in another country. _____________

4. played badminton last week. _____________

5. skipped lunch today. _____________

6. sang at a karaoke last Saturday. _____________

7. cut his/her hair in the last month. _____________

8. had a date in last weekend. _____________

9. took a trip out of town last holiday _____________

10. bought a mobile phone a few weeks ago. _____________


4/48

Review/Preview (5 min)

1. Name one type of closed-ended item and list the prosand cons for this type of item.

_____________________________________________

2. Two components of a test blueprint are __________

and _________________.3. The degree to which the test actually measures what it

is intended to measure is called reliability (T/F)

4. The systematic assignment of numbers to attributes iscalled

A. TestB. Evaluation

C. Assessment

D. Measurement


5/48

Types of Test Items

objectivesubjective

Open-

ended

Closed-

ended


6/48

Writing Objective Test Items

Multiple-choice Questions (MCQ)

True/False (TF)

Matching

Supply-Type


7/48

Multiple-choice questions

are the hardest type of objective questions to write forclassroom teachers

Although many people believe MCQs are simplistic,actually the format can be used for intellectuallychallenging tasks

Multiple-choice test items consist of a stem or question and

three or more alternative answers

the correct answer sometimes called the keyed response and the incorrect answers called distractors.

Always keep in mind the guiding principles whenwriting MCQs


8/48

The good and the bad of MCQs

Pros Cons

more answer options (4-5)

reduce the chance of

guessing that an item iscorrect

many items can aid in

student comparison and

reduce ambiguity

greatest flexibility in type ofoutcome assessed:

knowledge goals,

application goals, analysis

goals, etc.

reading time increased with

more answers

reduces the number ofquestions that can be

presented

difficult to write four or five

reasonable choices

takes more time to writequestions


9/48

Tips for Writing MCQs

Avoid responses that are interrelated.

Avoid negatively stated items.

It is easy to miss the negative word "not." If you use

negatives, bold-face the negative qualifier to ensurepeople see it.

Avoid making your correct response different from

the other responses, grammatically, in length, or

otherwise.


10/48


Avoid the use of "none of the above." When a students guesses "none of the above," you still

do not know if they know the correct answer.

Avoid repeating words in the question stem in your

responses. For example, if you use the word "purpose" in the

question stem, do not use that same word in only one of

the answers, as it will lead people to select that specific

response.

Use plausible, realistic responses.


11/48


Create grammatically parallel items to avoid giving

away the correct response. For example, if you have four responses, do not start

three of them with verbs and one of them with a noun.

Always place the "term" in your question stem and

the "definition" as one of the response options.


12/48

Example 1

Stem Presented as a Question. This form is generally better than the incomplete stem

because it is simpler and more natural.

Who is primarily responsible for maintaining anaircraft in an airworthy condition?

A. Pilot in command or operator.B. Owner or operator of the aircraft.

C. The lead mechanic responsible for that aircraft.


13/48

Example 2

Stem as an Incomplete Statement. When using this form, care must be exercised to avoid

ambiguity, giving clues, and using unnecessarily complex orunrelated alternatives.

VFR cruising altitudes are required to be maintainedwhen flying..

A. at 3,000 feet or more AGL, based on true course.

B. more than 3,000 feet AGL, based on magneticcourse.

C. at 3,000 feet or more above MSL, based on magneticheading.


14/48

Example 3

Stem Supplemented by an Illustration.

This form is useful for measuring the ability to read

instruments, or identify objects.

(Refer to figure 1.) The acute angle A is the angle of

A. Attack.B. Dihedral.

C. Incidence.


15/48

Example 4

Multiple Response is Required.

This form is a variation of the previous forms in that it contains more than one correct

answer, and students are instructed to select all correct answers.

Which of the following statements is/are generally true regarding the charging of

several aircraft batteries together?

1. Batteries of different voltage (but similar capacities) can be connected in series with

each other across the charger, and charged using the constant current method.

2. Batteries of different ampere-hour capacity and same voltage can be connected in

parallel with each other across the charger, and charged using the constant voltage

method.

3.Batteries of the same voltage and same ampere-hour capacity must be connected inseries with each other across the charger, and charged using the constant current

method.

A. 3.

B. 1 and 2.

C. 2 and 3.


16/48

Example 5

Negative Variety Type. This form is not suggested but, if used, always

emphasize the negative word.

Which of the following is NOT considered a methodof heat transfer?

A. Diffusion.B. Conduction.

C. Convection.


17/48

Example 6

Association Type. This form is useful if a limited number of associations are

to be made.

Which aircraft has the right-of-way over the otheraircraft listed?

A. Airship.B. Gyroplane.

C. Aircraft towing other aircraft.


18/48

Example 7

Definition Type.This form is used to determine knowledge of a specificdefinition.

Aspect ratio of a wing is defined as the ratio of the

A. wingspan to the wing root.B. wingspan to the mean chord.

C. square of the chord to the wingspan.


19/48

True/False (TF)

Advantage: Easier and quicker to write than MCQs

Remember the guidelines for writing effective test

items when creating these T/F items


20/48

The good and the bad of TF

Pros Cons

can present many items

at once

easy to score used to assess popular

misconceptions, cause-

effect reactions

most difficult question

to write objectively

ambiguous terms canconfuse many

few answer options (2)

increase the chance of

guessing that an item iscorrect; need many

items to overcome this

effect


21/48

Tips for Writing True/False Items

Do not use definitive words such as "only," "none,"

and "always," that lead people to choose false, or

uncertain words such as "might," "can," or "may,"

that lead people to choose true. Do not write negatively stated items, as they are

confusing to interpret: "Thomas Jefferson did not

write the Declaration of Independence." True or

False?


22/48

Tips for Writing True/False Items

People have a tendency to choose "true," so

design at least 60% of your T/F items to be "false"

to further minimize guessing effects.

Use precise words (100, 20%, half), rather thanvague or qualitative language (young, small, new,

beautiful, many).

Avoid making the correct answer longer than the

incorrect answer (a give-away).


23/48

Matching

Matching is an extended form of MCQ that draws

upon the students ability to make connections

between ideas, vocabulary and structure

advantage over MCQs is that the student has moredistractors per item.

writing items in the matching format is somewhat

easier for teachers than either MCQs or TFNs

some important points to bear in mind


24/48

The good and the bad of

Matching Items

Pros Cons

efficient

used to assess student

understanding ofassociations,

relationships,

definitions

difficult to assess

higher-order outcomes

(i.e., analysis,synthesis, evaluation

goals)


25/48

Tips for Writing Matching Items

Include more items in the answer group than in thequestion group

Never write items that rely on direct 1-on-1matching. The consequence is that if a student gets one item

wrong, at least two are wrong by default.

By contrast, all previous items right, the last item is aprocess of elimination freebie.

Matching can be used very effectively with relateditems for gap-fill paragraphs instead of two lists. In this way, students focus on meaning in context and

attend to features such as collocation.


26/48


If a two-column format is used for matching,

number the questions and letter the answer

options.

Leave a space for students to write the letter of thechosen answer.

This prevents lines drawn from Q to A columns.

Two-column matching formats should be used

sparingly for word association tasks.


27/48


When this is the specific testing objective, be sure

that the syntax between the two columns is correct

and unambiguous.

Avoid extraneous clues such as using an whenthe correct answer starts with a vowel.


28/48

Matching-Equal Columns

When using this form, providing for some items in the response column to beused more than once, or not at all, can preclude guessing by elimination.

Directions: In the blank before each electrical term in the left-hand column, writethe letter corresponding to the unit of measurement which is most closelyassociated with that term. Each unit of measurement may be used more than

once and some units may not be used at all.

1. ____ Electromotive force A. Watt

2. ____ Electrical power, apparent B. Volt

3. ____ Electrical power, true C. Ampere

4. ____ Resistance D. Coulomb

5. ____ Capacitance E. Ohm6. ____ Inductance F. VAR

7. ____ Current G. Farad

8. ____ Impedance H. Henry


29/48

Matching-Unequal Columns

Generally preferable to equal columns.

Directions: In the blank before each phrase in the left-hand column, write the letter(s)corresponding to the type(s) of drag which is/are most closely associated with that phrase. Eachtype of drag may be used more than once, and some types may not be used at all.

1. ____ Occurs when varied currents over an airplane meet and A. Form drag

interact. B. Induced drag

2. ____ Results from the turbulent wake caused by the separation C. Skin friction drag

of airflow from the surface of a structure.

D. Static drag

3. ____ Caused by the roughness of the airplane's surfaces.

E. Interference drag4. ____ Generated by the airflow circulation around the airfoil as

F. Rolling drag

it creates lift.

G. Sliding drag


30/48

Supply-Type

may be required where a selection-type cannot be

devised to properly measure student knowledge

valuable in measuring the students' generalized

understanding of a subject


31/48

The good and the bad of Supply-

Type

Pros Cons

chances of guessing

reduced

measures knowledgeand fact outcomes well,

terminology, formulas

scoring is not objective

can cause difficulty for

computer scoring


32/48

Writing Subjective Test Items

Short-answer

Problem sets

Oral exams

Performance tests

Essay tests

Guided and Free Writing


33/48

Short-answer Qs

depending on objectives set, these

questions can call for one or two sentences

or a long paragraph

easier to write, though they take longer to

score compared to multiple-choice tests


34/48

Problem sets

Normally used in Mmathematics and the

Sciences

Tip = allow students ten minutes to solve a

problem you can do in two minutes


35/48

Oral exams

common at the graduate level, rarely used forundergraduates except in foreign language classes

are usually time-consuming, too anxiety provokingfor students, and difficult to score unless the

instructor tape-records the answers

However, a math professor has experimented with individual thirty-minute oral tests in a small seminar class.

Students receive the questions in advance and are allowed to drop

one of their choosing.During the oral exam, the professor probes students' level ofunderstanding of the theory and principles behind the theorems.

He reports that about eight students per day can be tested.


36/48

Performance tests

ask students to demonstrate proficiency in conducting an experiment,

executing a series of steps in a reasonable amount oftime, following instructions,

creating drawings, manipulating materials or equipment, or

reacting to real or simulated situations

can be administered individually or in groups

seldom used in colleges and universities logistically difficult to set up,

hard to score, and

the content of most courses does not necessarily lenditself to this type of testing


37/48

Tips for performance tests

in classes that require students to demonstrate theirskills (for example, health fields, the sciences,education).

Anderson (1987, p. 43) recommends the following:

1. Specify the criteria to be used for rating or scoring(for example, the level of accuracy in performing thesteps in sequence or completing the task within aspecified time limit).

2. State the problem so that students know exactly what

they are supposed to do (if possible, conditions of aperformance test should mirror a real-life situation).

3. Give students a chance to perform the task morethan once or to perform several task samples.


38/48

Essay tests - Pros

relatively easy to write enable you to judge students' abilities to

organize,

integrate,

interpret material, and

express themselves in their own words

Research indicates that (McKeachie, 1986) students study more efficiently for essay-type examinations than for selection (MCQ)

tests

students preparing for essay tests focus on broad issues, general concepts, andinterrelationships rather than on specific details, and

this studying results in somewhat better student performance regardless of the type ofexam they are given

also give you an opportunity to comment on students' progress,

the quality of their thinking,

the depth of their understanding, and

the difficulties they may be having


39/48

Essay tests - Cons

because essay tests pose only a few questions,

their content validity may be low.

the reliability of essay tests is compromised by

subjectivity or inconsistencies in grading difficulty also lies in clearly specifying the task for

the student so that grading is fair and equitable to

all students


40/48

Tips for essay tests

writing good subjective items is an interactive,

negotiated process

one best approach is to write a sample answer and

then analyze the elements of that answer OR ask a colleague to write a sample answer and

critique the prompt


41/48

Guided vs. Free Writing

literature generally addresses two types of writing:

free writing

requires students to read a prompt that poses a situation and write a

planned response based on a combination of background knowledge and

knowledge learned from the course

guided writing

requires students to manipulate content that is provided in the prompt,

usually in the form of a chart or diagram

a bridge between objective and subjective formats

requires teachers to be very clear about what they expect students to do

decide in advance whether mechanical issues like spelling, punctuation

and capitalization matter when the task focuses on comprehension


42/48

Guided vs. Free Writing

goal for teachers is to elicit comparable products

from students of different ability levels


43/48

Writing Assessment Scales

literature generally recognizes two

different types of writing scales for

assessing student written proficiency:

1. holistic marking and

2. analytical marking


44/48

Alternative Testing Modes

Take-home tests

Open-book tests

Group exams Paired testing

Portfolios


45/48

Administering Tests


46/48

Timing

Item Type Time Allocated

True/False 30 seconds

MCQ 1 min

MCQ with Higher Learning Objectives 1.5 mins

Short Answer 2 minutes

Completion 1 min

Matching 30 secs per response

Short Essay 10-15 mins

Extended Essay 30 min

Visual Image 30 seconds


47/48

Checklist

Is this item an appropriate measure of my learningobjective?

Is the item format the most effective means of measuringthe desired knowledge?

Is the item clearly worded and easily understandable by thetarget student population?

Are items of the same format grouped together?

Are various item types included in the assessment?

Do students have enough time to answer all test items?

Are test instructions specific and clear? Does the number of questions targeting each objective

match the importance weighting of that objective?

Are scoring guidelines clearly available to students?


48/48

In closing

Tests are just tools (Hanna & Dettmer, 2004; Kubiszyn & Borich, 2007)

Test items are part of those tools

Like the tools in a carpenters toolbox, you

need to choose from tests toolbox what

works best for the task at hand

Remember, "Ifthe only tool you have is a

hammer, you tend to see everyproblem as a

nail

day2 ses1_test items and rubrics

Documents