david dibattista, ph.d. brock university psychology department encouraging higher-order thinking...

David DiBattista, Ph.D.Brock University

Psychology Department

Encouraging Higher-order Thinking Using Multiple-choice Questions

Toronto, OntarioFebruary, 2012

©D. DiBattista 2012

OVERVIEW Using MCQs for summative purposes The Revised Bloom’s Taxonomy Getting beyond remembering with MCQs Using MCQs for formative purposes $ The rationale $ Some strategies



What is Summative Assessment?

A summative assessment is a test that is given primarily:a) to assess a student’s achievement in a course, andb) to assign a grade that reflects the student’s level of achievement.

MCQs are the most commonly used form ofselected-response technique on classroom tests.

Selected quotes from the media• All who take part in MC tests are wasting their time.• MC tests are a stupid way to test what students know.• MC testing is not a good indicator of conceptual

understanding.• [MC tests] are a lazy, one-size-fits-all substitute for

actually assessing what students have learned.• Results [of MC tests] tend to depend on luck, not on

intelligence and hard work.• Multiple choice tests are games with a premium on

strategic guessing.


MC questions: Controversial!

• The real value of MC items…is their applicability in measuring higher-level objectives, such as those based in comprehension, application, and analysis. (Burton, et al., 1991)

• [Constructed-response items] take more examinee time and resources to measure essentially the same thing as [MCQs, but they do so] more poorly than the latter. (Wainer & Thissen, 1993)


But on the other hand…

Let's look at one of the most common criticisms of MCQs.

“It seems quite evident from many studies that the basic behavior evaluated (by MCQs) is pure recall of information.” (Srivastava, et al., 2004)

Overemphasis on memory-based MCQs

Poor content-related validity of the test

andPromotion of surface learning

at the expense of deep learning


We need a framework that lets us discuss this issue in depth.

Factual

Knowledge Dimension

Conceptual Procedural Metacognitive

CognitiveProcess

Dimension

Remember

Understand

Apply

Analyze

Evaluate

Create

The Revised Bloom’s Taxonomy Anderson and Krathwohl, 2001


This chart is in today’s handout!

Factual

Knowledge Dimension


CognitiveProcess

Dimension

Remember

Understand

Apply

Analyze

Evaluate

Create


These are all ACTION verbs—i.e., things students can DO with their knowledge.


Good tests allow us to determine what our students are capable of.

…but they can be used effectively to assess all of the other cognitive processes.

MC questions are notuseful for assessing creativity…

Factual

Knowledge Dimension


CognitiveProcess

Dimension

Remember

Understand

Apply

Analyze

Evaluate

Create


Can you remember this?

Can you understand this?

Can you apply this?

Can you evaluate this?

Can you create this?

Can you analyze this?


These are all ACTION verbs—i.e., things students can DO with their knowledge.

Factual

Knowledge Dimension


CognitiveProcess

Dimension

Remember

Understand

Apply

Analyze

Evaluate

Create



Some thoughts about REMEMBER…

ALL assessment tasks involve usingmemory to at least some degree.

BUT“If assessment tasks are to tap higher-order cognitive processes, they must require that students cannot answer them correctly by relying on memory ALONE.”

—Anderson and Krathwohl, 2001, page 71

A simple, unfortunate fact:Creating MC items that rely on memory alone is far easier than creating higher-level items.


In classical conditioning, what name is given to a stimulus that elicits a particular response even in the absence of any prior training?A. conditioned stimulusB. unconditioned stimulusC. activational stimulusD. discriminative stimulus

“Do you remember…?”


Right after a rat smells menthol, it is always given Drug X, which causes osmometric drinking. Eventually, the rat drinks whenever it smells menthol, even when it is not injected with Drug X. In this situation, what is the role of Drug X?A. conditioned stimulusB. unconditioned stimulusC. activational stimulusD. discriminative stimulus

Note the important role of NOVELTY.


To learn about the attitudes of Canadian voters toward a government bill limiting auto emissions, Gary gave a questionnaire to 200 members of a group called Friends of a Cleaner Environment. He found that every single one of them strongly supported the bill. Based on this research, which of the following is the most appropriate conclusion?


A. The questionnaire is extremely reliable.B. The questionnaire items are not written

very clearly.C. The majority of Canadian voters strongly

support the bill.D. The results tell us little about the attitudes

of Canadian voters.


This item deals with sampling bias,but sampling bias is never mentioned.

Note again the importance of novelty.

Your friend Anne reads that census data indicate that people are having fewer children nowadays than they did 40 years ago. Anne does not believe this because the young couple who live next door to her are both under 30 and already have four children. If Keith Stanovich were told about this, what might you reasonably expect him to say?


A. The census data must be wrong.B. Anne’s comment illustrates valid probabilistic reasoning.C. Anne’s comment illustrates the use of person-who statistics.D. The young couple provide an exception that actually serves to prove the rule.


Suppose that you have performed a Gram stain on a mixed culture of Gram-positive and Gram-negative cells. Afterwards you realize that you omitted the iodine step. What do you expect to see when you look at the slide under the microscope?


An MCQ from the test bank for Russell, et al., Biology: Exploring the Diversity of Life.

A. colourless Gram-positive cells and pink Gram-negative cells

B. colourless Gram-positive cells and colourless Gram-negative cells

C.pink Gram-positive cells andpink Gram-negative cells

D. purple Gram-positive cells andpink Gram-negative cells


And see the handout for more examples…Now let’s see how MCQs can

be used in a formative fashion.


What is Formative Assessment?

A formative assessment is one that is primarily intended to promote students’ learning rather than to assign a grade.


“The key idea underlying our research is that frequent classroom testing (and student self-testing) can greatly improve education from kindergarten through university. This is a bold claim that runs counter to current wisdom in educational circles…. [We] emphasize testing as an aid to learning, a practice that should be part and parcel of a good educational system.”

Roediger, et al., 2006

Formative Uses of MCQs


MCQs during class

MCQs outside of class$ Aplia$ Online quizzes

The so-called “10-minute rule”


Attributed to Hartley & Davis, 1978

Some classic data on attention…


“More students will maintain interest for a longer time when the topic is interesting and delivered with clarity, enthusiasm, and drama, with a cognitive break now and then.”

Wilson & Korn, 2007


“Change grabs attention.”Dan Willingham

University of Virginia

“More students will maintain interest for a longer time when the topic is interesting and delivered with clarity, enthusiasm, and drama, with a cognitive break now and then.”

Wilson & Korn, 2007


PRESENT NEW MATERIAL

MCQPRESENT

NEW MATERIALDEMO

PRESENT NEW MATERIAL

MCQ

A role for clickers!

BUTClickers work well only when the

questions asked are used effectively.

Lecture time

62

55

18

0

20

40

60

80

Course 1 Course 2 Course 3

Per

cen

t ag

reei

ng


Because we use clickers, we havebetter discussions of class material.

Dol & DiBattista, 2011


82

55 57

0

10

20

30

40

50

60

70

80

90

100


Per

cen

t ag

reei

ng

The questions asked when we use clickersencourage me to think more deeply about the material.



The clicker questions allowme to learn from my mistakes.


92

53

62

0

20

40

60

80

100


Per

cen

t ag

reei

ng


• Retrieval-practice questions Remember or higher-level

• Fuzzy best-answer questions More than one defensible answer

• Student perspective questions Opinions and experiences

• Misconception questions Directly address errors of thought

Four Types of Clicker Questions






Which of the following defines microevolution?

Which of the following provides an example of microevolution?







What research strategy would be most effective for answering the question that Burton has posed?







Which of the following best describes what you believe about the origins of human beings?



Would you expect the vapour pressure ofthese two flasks of water to differ? If so, how?

A B






Formative Uses of MCQs


MCQs during class

MCQs outside of class$ Aplia$ Online quizzes

• Animations and videos clarify complex biological processes.

• Active learning opportunities include automatically graded questions and detailed explanations.

• Questions focus on data analysis, experimental and observational approaches to research, real-world applications, global problems, and more.

Outside of Class:Aplia

APLIA


Outside of Class:Online Quizzes


The course

$ PSYC 2F23: Statistics and Research Design

$ Full year, 4 hr/wk of contact

$ Required course; unpopular topic

$ Tests and exams: 78% of course grade

The online quizzes

$ Purpose: Progress check and exam prep

$ Both REMEMBER and HIGHER-LEVEL MCQs

$ Six quizzes via Sakai; 12 items/quiz

$ Low stakes (2 marks per quiz)

$ 10-day window for each quiz

$ Unlimited attempts

$ Only highest mark counts

$ Feedback after each quiz attempt


The research

$ Students completed an extensive survey at end of course.

$ Participation rate: 69% (287 out of 413 students)

$ Obtained self-reports re:Course liking, Quiz liking, Quiz Motivation, and Learning Value of Quizzes


Major findings

$ 94%: Quizzes cover important material.

$ 87%: Quizzes help me know what to study more.

$ 17%: Did not like having the quizzes.

$ 79%: Would like to have them in other courses.$ 53%: Would not even look at quizzes if they did not count for any marks.


$ Total quiz score and Exam grades: r = +0.32(Partialed out: GPA, Course Liking, Quiz Liking, Quiz Motivation)

$ 79%: Quizzes have Learning Value. The online quizzes help me to learn the material relating to…

Strongly related to Quiz Motivation

When I take an online quiz,I work hard to answer every single item correctly.

If the online quizzes did not count for any marks,I doubt that I would even look at them. (R)


Learning Value YES NO

QuizMotivation

Positive

Negative

$ 97% of those with positive Quiz Motivation agreed that the quizzes promoted their learning.

$ In contrast, only 65% of those with negative Quiz Motivation felt this way.

$ They also liked the course less, and they did less well on exams.


152

84 46

5


One final thought…

We must remember to berespectful of students’ time.

Two Major Conclusions

In a summative context, MCQs can be used to assess students’ learning both at the remember level and at higher cognitive levels.

Well-chosen MCQs can be used in a formative fashion to promote students’ learning both inside and outside of the classroom.


David DiBattista, Ph.D.Brock University

Psychology Department

Encouraging Higher-order Thinking Using Multiple-choice Questions

Toronto, OntarioFebruary, 2012


david dibattista, ph.d. brock university psychology department encouraging higher-order thinking...

Documents

mc questions

results of mc tests

david dibattista

good tests

mc testing

common criticisms of

summative purposes

classroom tests