Designing (More)Effective Tests & Quizzes

@kristobolsloan@StEds_CTE

Thursday, February 12, 3:30-4:45Fleck 314

Assumptions

Effective assessments can drive, propel even student learning.

Effective assessment and evaluation leads to more clear course outcomes.

Effective assessment and evaluation can better help a professor determine their own teaching effectiveness

Assumptions

Most importantly:

To better ensure greater access to academic success of a diverse group of learners, I must offer my students an array of assessments and evaluations to better understand the fuller range of what my students know and can do vis-à-vis my course outcomes

Time Management

Electronic management of scoring/grading

Provides immediate feedback/correct misconceptions

Can offer students opportunities for input on test creation

Socrative

Pre-assessments example

Using student questions as a source of student-to-student talk and input into test creation.

(Educational) LiteracyThe questions that will be graded on the following

three criteria:

Explicit connection to course content—not reflective of outside/internet sources

Specificity (question could only be answered by the term below)

Relevance to (contemporary educational context).

Provide the correct answer (with citation of where correct answer can be found)

Limit of 20 words!

What’s “wrong” with these ?s

1. A ______ is used in a house to help prevent the ________of the electrical wiring.

2. _____is the largest export product in Brazil?

3. When did the English first arrive in North America?

4. Discuss what you learned about the British parliament and the U.S. congress.

Instructor-created assessments

What sorts of assessments items do you offer on a typical instructor-created tests & quizzes?

Which types of items are most common?

Which are easiest/most difficult to create?

What are some possibilities/limitations of each type of item?

Instructor-Created Quizzes/Tests

Validity/Reliability concerns:

What is the difference between reliability and validity?

Instructor-Created Quizzes/Tests

Validity: Does the quiz/test item actually measure what it is intended/supposed to measure relative to the content objectives?

Reliability: Does the quiz/test item consistently estimate what the learner knows and can do relative to the content objectives?

Two general categories of items

Supply (“Completion”) itemssentence completionfill-insshort answeressay-response items

“Objective items”true/false multiple choicematching

Avoiding Traps….

Taking care not to get what you ask for….

Supply items

What qualities do you believe makes a good supply item, or “fill-in-the-blank?”

Supply items

Tip #1: Use only one blank per item

Poor: A ______ is used in a house to help prevent the ________of the electrical wiring.

Supply ItemsTip #1: Use only one blank per item

The overheating of electrical wiring in a house may be prevented if _____ are installed.

Supply Items

Tip #2: Blanks should appear either at the end or near the end of an item.

Poor: _____is the largest export product in Brazil?

Supply Items

Tip #2: Blanks should appear either at the end or near the end of an item.

The largest export product in Brazil is ______?

Supply Items

Tip #3: Use precise language, or language/terms that cannot be construed in multiple and correct ways

The largest export product in Brazil is ______?

Supply Items

Tip #3: Use precise language

The largest export product in Brazil is ______?

By “largest” do you mean is size (e.g., jackfruit, palm trees), biggest revenue producer (e.g. $?), or by “units” (e.g., number of bananas, pounds of coffee)

Supply Items

The highest revenue-producing export of Brazil is ______?

Instead of…

The largest export product in Brazil is ______?

Supply Items

Tip #4: Include a word or phrase that indicates the category to which the answer must belong.

Poor: When did the English first arrive in North America?

Supply Items

Tip #4: Include a word or phrase that indicates the category to

which the answer must belong.

What year did the English first arrive in North America?

True/False Items

What qualities do you believe makes a good “true/false” question?

True/False ItemsTip #1: Write more false than

true

items.

Researchers have concluded that teachers tend to do the reverse and students tend to mark items they are unsure about as “true”

True/False Items

Tip #2: Avoid words like “most” (which usually indicates a “true” statement) and “never” (which usually suggests a false statement).

True/False Items

Tip #3: Keep wording clear and specific.

Avoid broad, general statements such as always, never, all, may seldom, usually, etc

Poor: __ Birds eat more than mammals.

True/False Items

Tip #3: Keep wording clear and specific.

____Relative to their body weights, birds eat more than mammals.

True/False Items

Tip #4: Avoid simply taking statements directly from text (even if modified slightly) because it may encourage rote memorization with little understanding.

True/False

Tip #5: Generate binary-choice items in pairs not singly.

Because True/False questions need to be mutually exclusive when reversed, check it.

True/False Items

Tip #6:

Restrict T/F items to ONE, or a single concept!

Multiple Choice Items

What qualities do you believe makes a good Multiple Choice Item?

Multiple Choice

Which of these is the most valid question?

How old is Bob?How many years ago was Bob born?In what year was Bob born?

Multiple ChoiceWhich of these is the most valid question?

How old is Bob?How many years ago was Bob born?

In what year was Bob born?

Time sensitive questions are tricky because the answers come in many different forms and can change

Multiple Choice ItemsTip #1: Stem should present enough

information so that answer choices can be relatively short.

Tip #2: Avoid negatives (e.g., not) as well as words like all, some, often, usually, etc

If you do use word not in stem, italicize or underline it.

Multiple Choice Items: Stems and Choices

Tip #3: All choices should be feasible

Tip #4: Don’t expect students to make narrow distinctions (e.g., “Freezing”: 30’F, 31’F, 32’F, 33’F)

Multiple Choice Items

Tip #5: All choices should be grammatically consistent

The largest tree-borne fruit is a:

A. Banana

B. Apple

C. Jackfruit

D. Strawberries

Multiple Choice Items

Poor:

An example of:

A. An animal that is a mollusk is a whale.

B. An animal that is a mollusk is a clam.

C. An animal that is a mollusk is a crab.

D. An animal that is a mollusk is a lobster.

Multiple Choice Items

An example of a mollusk is a:

A. Whale

B. Clam

C. Crab

D. lobster

Multiple Choice Items

Tip #6: Do not overuse “none of the above”/ “All of the above”

&

variations of “A and B” or “C and D, but not A.”

Multiple Choice Items Strategies to find the elusive“higher-level”* multiple choice question:

Analogy based items:

Bandura is to social cognitive theory as _____________ is to social constructivist theory:

a. Piagetb. Sieglerc. Vygotskyd. Skinner

*Bloom’s Taxonomy

Knowledge-Level Items

Identify meaning of term:

Reliability is the same as:

A. consistency.

B. relevancy.

C. representativeness.

D. usefulness.

Comprehension-Level Item

Identifies an example of a concept or principle .

Which of the following is an example of a criterion-referenced interpretation?

A. Derik earned the highest score in science.

B. Erik completed his experiment faster than his classmates.

C. Edna’s test score was higher than 50 percent of the class.

D. Tricia set up her laboratory equipment in five minutes.

Application-Level ItemsDistinguish between properly and improperly stated outcomes

Which one of the following learning outcomes is properly stated in terms of student performance?

A. Develops an appreciation of the importance of testing.

B. Writes valid test questions.

C. Realizes the importance of validity.

Analysis-level ItemsAnalyze whether a word fits with the accepted definition of pragmatism.

Directions: Read carefully through the paragraph below, and decide which of the options A-D is correct.

“The basic premise of pragmatism is that questions posed by speculative metaphysical propositions can often be answered by determining what the practical consequences of the acceptance of a particular metaphysical proposition are in this life. Practical consequences are taken as the criterion for assessing the relevance of all statements or ideas about truth, norm and hope.”

A. The word “acceptance” should be replaced by “rejection.”

B. The word “often” should be replaced by “only.”

C. The word “speculative” should be replaced by hypothetical.”

D. The word “criterion” should be replaced by “measure.”

Synthesis-level ItemsDirections: Read the following comments a teacher made about testing. Then answer the questions that follow by circling the letter of the best answer.

“Students go to school to learn, not to take tests. In addition, tests cannot be used to indicate a student’s absolute level of learning. All tests can do is rank students in order of achievement, and this relative ranking is influenced by guessing, bluffing, and the subjective opinions of the teacher doing the scoring. The teacher-learning process would benefit if we did away with tests and depended on student self-evaluation.”

Identifies relationships:

Which one of the following propositions is most essential to the final conclusion?

A. Effective self-evaluation does not require the use of tests.

B. Tests place students in rank order only.

C. Test scores are influenced by factors other than achievement.

D. Students do not go to school to take tests.

Multiple Choice Items

Advantages:Efficient way of testing students’ ability to recall facts

Demonstrates understanding

DisadvantagesHard to create, time consuming

Inference Questions

Ask students draw conclusions

Ask you to find assumptions made in the passage.

Assesses your ability to go beyond what is on the page.

Inference Question Stems….

The author suggests which of the following…

It can be inferred from the passage that…

The author assumes that…

Inference Question Stems….

The main point of the passage is that...

Which of the following statements about... is best supported by the statements above?

Which of the following best states the author's conclusion in the passage above?

Inference Question Stems….

Which of the following conclusions can be most properly drawn from the data above?

Which of the following is [implied, must be true, implicit, most reasonably drawn] in the passage above?

Which of the following conclusions can most properly be drawn if the statements above are true?

Making Inferences/Drawing Conclusions

Which conclusion is best supported by the sentence: “The Golden Gate Bridge spans the San Francisco Bay, connecting northern California to the peninsula of San Francisco.”

a. The bridge enables many people to go from SF to northern California

b. The bridge has the longest span of any other suspension bridge

c. The cables are best

d. The bridge is the most popular bridge in the bay area

Making Inferences/Drawing Conclusions

Joe woke up and could hear the roosters crowing.

We can infer the time of the day is__________.

a. At noon

b. Early in the morning

c. In the afternoon

d. At night

Making Inferences/Drawing Conclusions

Which of the following statements would make the most accurate conclusion to the passage

a. Bacteria are often an indication that a person’s health has been seriously compromised.

b. Both measures would help protect the health of countless people throughout the country.

c. Researchers are always looking at ways to provide people with better dental health care.

d. Sensors are so incredibly small that they can easily be placed on teeth or even tinier spots.

Matching

1. ___The Seine river divides this city into two famous banks.

2. ___The capital city of the Island of Taiwan.

3. ___Home city to the Statue of Liberty.

4. ___The Kremlin is located in this city.

…….

A. Paris

B. Moscow

C. Taipei

D. New York City

E. Hawaii

F. Newark

……………….

MatchingAdvantages:

especially suitable for who, what, when, and where questions.Can ask students to discriminate among, and to apply concepts.efficient use of space, easy to score

Disadvantages:Poor for diagnosing strengths and weaknesses

Matching

Tip: Include more responses than premises!

This increases the difficulty and decreases the likelihood that correct answers are guesses.

Essay Items/Prompts

What qualities do you believe makes a good essay question, or essay prompt?

Essay Items/Prompts

Tip #1:Clearly indicate a task. Be precise about the information you expect, and how much information you expect

Poor: Discuss what you learned about the British parliament and the U.S. congress.

Essay Items/Prompts

Tip #1:Clearly indicate a task. Be precise about the information you expect, and how much information you expect

Identify and describe two major differences and one similarity between the BP and U.S. congress.

Compare and contrast….??

Essay Items/PromptsIdentify and describe at least three

assumptions about B.F. Skinner’s version of behaviorism as it relates to educational psychology.

Define Vygotsky’s notion of the Zone of Proximal Development AND provide a clear and convincing example that one might see in a typical classroom. Your example should include specific content or specific examples of a teaching strategy

Essay Items/Prompts

Tip #2: Try to avoid using “why” questions*

Poor: If you were a Catholic priest in Texas in the early 1800’s, why would you be Texas?

*unless your purpose is to have your students produce a teleological response.

Essay Items/Prompts

Tip #2: Try to avoid using “Why” questions

Identify and describe at least three activities a typical Catholic priest would have been engaged in during the early 1800’s in the Spanish Texas.

Essay Items/Prompts

Using Rubrics to direct students toward desired content/quality

Culturally Response Assessment Literacy

Tests are culturally loaded

Tests require language

Culturally Response Assessment Literacy

More equitable, culturally responsive assessment practices requires teachers to……

Culturally Response Assessment Literacy

The use of an array of assessments/evaluations

Evaluation of students against their own records (vs. comparisons with other students)

Inclusion of socioemotional measures as well as measures of academic content (look for both verbal/nonverbal information)

Culturally Response Assessment Literacy

Items should measure only the content of interest, not one’s knowledge of the item context.

Write items to measure what students know, not what they do not know.

Never couch correct answers around the notion of “intelligence” (See Claude Steele’s notion of “Stereotype Threat”)