comparing check-all and forced-choice question … · comparing check-all and forced-choice...

Comparing Check-All and Forced-Choice

Question Formats in Web Surveys: The Role of Satisficing, Depth of Processing,

and Acquiescence in Explaining Differences

Jolene D. Smyth

Don A. Dillman

Leah Melani Christian

Michael J. Stern

Technical Report #05-029

1

ABSTRACT

In an experimental comparison between the check-all and forced-choice formats Rasinski,

Mingay and Bradburn (1994) found that in paper surveys the forced-choice question format

produced significantly more affirmatively marked answers than did the check-all question

format. Their research raises fundamental questions about whether different response processes

are at work in the two question formats and about the implications of those processes for survey

responses. Using results of experiments from two web surveys and a paper survey comparison of

random samples of students at Washington State University, we explore whether or not the use

of the forced-choice format increases the number of options selected affirmatively in web, as

well as paper surveys. In addition we explore three theoretical explanations for these differences

that have been proposed in previous research: satisficing (Krosnick 1991; 1999), depth of

processing (Sudman and Bradburn 1982), and acquiescence (Schuman and Presser 1981).

Finally, we report limited information on item nonresponse differences in the forced-choice

format. Our test uniformly supports the hypothesis that the forced-choice format results in more

options being selected. In addition, the forced-choice format appears to invoke deeper

processing and to eliminate satisficing behavior that occurs among some respondents to the

check-all format. However, virtually no evidence was found to support the presence of

acquiescence in the forced-choice question format. Finally, our exploration of item nonresponse

in forced-choice questions indicated that it is a fairly insubstantial problem.

Jolene D. Smyth, Leah Melani Christian and Michael J. Stern are graduate research assistants in the Department of

Sociology and the Social and Economic Sciences Research Center at Washington State University, Pullman,

Washington. Don A. Dillman is Regents Professor and the Thomas S. Foley Distinguished Professor of

Government and Public Policy in the Departments of Sociology, Department of Community and Rural Sociology

and the SESRC. This research was financed in part by funds provided to the Washington State University SESRC

under a Cooperative Agreement with the USDA-National Agricultural Statistics Service supported by the National

Science Foundation Division of Science Resource Statistics.

2

INTRODUCTION

A common question format in web surveys is the check-all question for which respondents are

asked to mark all that apply from among a list of response options (e.g. Please indicate which of

the following sources of information you have used to find employment in the last month.). One

factor that may encourage its use in web surveys is the fundamental choice between using radio

buttons that allow only one item to be selected or html boxes that allow multiple items to be

marked.

In telephone surveys check-all questions are considered awkward and seldom asked. Instead, a

forced-choice format is typically used where respondents provide an answer (e.g. yes/no) for

each item in the list. However, on self-administered questionnaires there is concern that

respondents will treat forced-choice questions as check-all questions by marking answers only in

the “yes” category and ignoring the “no” category. For this reason some self-administered

questionnaire designers avoid the forced-choice alternative. While the ability to require

responses to each item on web surveys could override that concern, error messages requiring an

answer to be marked may irritate respondents and cause them to terminate their participation in

the survey (Best and Krueger, in press).

In an experimental comparison between the check-all and forced-choice formats Rasinski,

Mingay and Bradburn (1994) found that in paper surveys the forced-choice question format

produced significantly more affirmatively marked answers than did the check-all question

format. Their research raises fundamental questions about whether different response processes

are at work in the two question formats and about the implications of those processes for survey

responses.

Using results of experiments from two web surveys and a paper survey comparison, we aim in

this research to answer the question of whether the use of the forced-choice format increases the

number of options selected affirmatively in web, as well as paper surveys. In addition we

explore three theoretical explanations for these differences that have been proposed in previous

research: satisficing (Krosnick 1991; 1999), depth of processing (Sudman and Bradburn 1982),

and acquiescence (Schuman and Presser 1981). Finally, we report limited information on item

nonresponse differences between check-all and forced-choice formats. The data analyzed here

are from 10 experimental comparisons using eight different questions (one of which was

replicated across all three surveys) collected from random samples of Washington State

University students. The deliberate use of different types of questions allows the examination of

these issues for a wide range of check-all vs. forced-choice situations.

THEORETICAL BACKGROUND

Does the Forced-choice Format Produce More Affirmative Responses?

The only experimental comparison we were able to locate between the check-all that apply

format and the forced-choice format was conducted by Rasinski et al. (1994) in a paper survey

field test for round three of the 1988 National Educational Longitudinal Study. In this

3

experiment half of the respondents were assigned a version of the survey in which three

questions were formatted as check-all that apply questions and the other half were assigned a

version in which these questions were formatted as forced-choice questions with yes/no

categories. For all three items the mean number of options marked per respondent in the forced-

choice version was significantly greater than the mean number marked per respondent in the

check-all version (3.03 vs. 2.86, p = .002; 2.47 vs. 1.53, p = .001, and 1.18 vs. 0.96, p = .001).

Rasinski and colleagues’ findings, although consistent, are limited to behavioral and

factual questions. Specifically, they asked if one’s school provided any of a list of four resources

for students planning to attend college, for which of 12 reasons students decided not to continue

their education right after high school, and which of 20 events had happened in their family in

the last two years. We also include behavioral and factual questions in our surveys (e.g.

resources used at WSU, student group participation, and food vendors used on campus). In

addition to these we seek to extend the work by Rasinski et al. (1994) by including a number of

opinion-based questions (e.g. descriptions of WSU Pullman campus, admittance criteria, and

university budget adjustments) to ascertain whether or not the effects of switching to a forced-

choice format are related to the type of question being asked.

Further, although Rasinski et al. (1994) provided evidence that the forced-choice format

leads to more response options being marked affirmatively on paper surveys, this finding has yet

to be extended to web surveys. The visual nature of both modes suggests that patterns of

responses will be similar across them, and a growing amount of experimental research has found

that other question formats react similarly across paper and web modes (see Christian and

Dillman 2004; Christian 2003). Nevertheless, because the types of specific respondent actions

required in completing a paper survey are somewhat different from those required to complete a

web survey (i.e. the use of the computer, the internet, and the mouse and screen which are

spatially separate as opposed to pencil and paper), the possibility of differences exists. However,

given previous research findings as well as the three theories presented below, we expect to

verify the finding that the forced-choice format results in more response options being marked

affirmatively and to extend that finding to web surveys.

Why Might the Forced-choice Format Produce More Affirmative Responses?

Weak Satisficing. One explanation, proposed by Rasinski et al. (1994), for the increased

affirmative responses is that the check-all that apply format encourages, or at least allows for, a

satisficing response strategy. A satisficing response strategy consists of avoiding expending the

effort required to answer the question optimally (Krosnick 1991; 1999). Satisficing can be

conceptualized as lying on a continuum ranging from strong satisficing to optimizing, with weak

satisficing falling in the middle. One example of a weak satisficing behavior is that respondents

might choose the first response options they can reasonably justify and then move on to another

question without giving adequate attention to the remaining response options (Krosnick 1991;

Krosnick and Alwin 1987). If respondents are satisficing in this way, a pattern of primacy,

where options are selected more often when they appear near the top of the list than when they

appear near the bottom of the list, is likely to occur.

The check-all that apply format may be more prone to this and other satisficing strategies

because it does not communicate the expectation that respondents should look at and come to a

4

judgment about each response option individually to the extent that the forced-choice format

does. Rasinksi et al. (1994) tested this explanation by comparing the endorsement of items in the

check-all format when they appeared in the top half of lists with endorsement of those same

items when they appeared in the bottom half (the response options were reversed in some

treatments) and found no evidence of primacy-related satisficing in their data. Similar to

Rasinski et al. (1994) we test for weak satisficing behavior by looking for primacy effects within

the check-all formatted questions. If primacy-based satisficing is occurring we expect to find

more options selected when they appear in the first half of the list than are selected from the

same response options when the order of presentation is reversed. In addition, comparing the

amount of change across formats for items that appear in the top half of the list to the amount of

change across formats for items that appear in the bottom half can shed light on whether or not

the forced-choice format leads respondents to give more attention to those options appearing

later in the lists than does the check-all format, thus reducing primacy-based satisficing.

Primacy effects may indicate the presence of one type of satisficing, but others may also occur in

check-all questions. For example, respondents may avoid expending sufficient effort for an

optimal answer through other forms of satisficing such as randomly selecting responses or giving

minimal consideration to each response option. These strategies, which most likely do not

produce accurate data, would not necessarily result in primacy effects.

Depth of Processing. Depth of processing may serve as a mechanism underlying all forms of

satisficing. Sudman and Bradburn (1982) proposed that the forced-choice format encourages

deeper processing of the question and the response options. Such deeper processing may reduce

satisficing because with more consideration respondents are likely to recognize that more of the

response options apply to them (Sudman, Bradburn, and Schwarz 1996). Thus, although

Sudman and Bradburn (1982) recognize that the forced-choice format is more “cumbersome”

than the check-all that apply format, they see it as providing “better responses” because it

requires the respondent to consider each option and come to a decision about it (p. 168).

However, they have not empirically tested this proposition.

In this paper we use client side paradata (Heerwegh and Loosveldt 2002; Stern et al. 2004) to

examine depth of processing by looking at the amount of time respondents spend on the check-

all format as opposed to the forced-choice format. If respondents are processing the forced-

choice formatted questions and response options more deeply than the check-all formatted

questions and response options, then they should be taking significantly longer to respond to

them. We recognize that the mechanical aspects of answering forced-choice questions are likely

to require more time as respondents need to mark an answer for every option instead of only

some options, but if the forced-choice format takes substantially more time we can fairly

confidently conclude that there is more going on than simply additional answer marking.

Acquiescence. Another explanation for the finding of more options marked affirmatively in the

forced-choice format compared to the check-all format is that the forced-choice format is prone

to an acquiescence or agreeing response bias (Schuman and Presser 1981) that is absent in the

check-all format. This explanation suggests that respondents have an aversion to rejecting

response options by answering them negatively. In check-all questions respondents can leave

options blank which doesn’t necessarily indicate rejection (additional discussion on this point is

5

below), but when forced to answer “yes” or “no” for each response option, those who are neutral

may be more likely to check “yes” than to check “no” because their tendency is to avoid the

outright rejection that the “no” answer choice implies. Leaving a check box blank on a check-all

question may not carry the negative connotation that is implied by checking “no” on a forced-

choice item. Thus, an acquiescence effect would lead us to expect respondents to select the

affirmative category more often on forced-choice questions.

We test for the existence of an acquiescence effect on two questions by comparing the results of

a forced-choice question with only two categories (i.e. yes/no) to one with three categories (i.e.

yes/no/don’t know or yes/no/neutral). If respondents to the typical forced-choice question (two

categories) are avoiding the “no” category and thus using the “yes” category more frequently, as

would be predicted if acquiescence is occurring, then we would expect to see significantly less

use of the “yes” category in the three-category format as respondents are provided with another

outlet. Respondents who are undecided or neutral should move from “yes” to the third category

that is more fitting for them.

To further test for an acquiescence effect in the forced-choice format, we also included

experimentation testing the effects of using more active category labels (i.e. fan/not a fan versus

yes/no). We hypothesize that the use of more active category labels, intended to remove the

inquiry from an agree/disagree context, would reduce acquiescence by giving the respondent a

more explicit and tangible way to categorize themselves.

The three explanations for the finding of more affirmative responses in the forced-choice

format—satisficing, depth of processing, and acquiescence—will be examined in this paper in an

effort to determine which one(s) explain the findings. This is important given the implications of

each of the explanations. If satisficing lies at the roots of previous findings, then one could argue

that the forced-choice format yields “better answers” than the check-all that apply format. The

same conclusion applies if it is found that respondents are processing the forced-choice format

more deeply. However, if the difference is due to acquiescence, then the forced-choice format

may not yield improved accuracy in responses and the recommendations for using it in the place

of the check-all format may be unfounded. Although we discuss the implications of these three

explanations separately throughout the paper for analytical purposes, we want to stress that these

processes are not necessarily independent of each other and that evidence for or against one

explanation does not automatically confirm or disqualify another.

Item Nonresponse in the Forced-Choice Question Format

In addition to comparing the check-all and forced-choice formats and considering the

explanations for findings, we examine the frequency of item nonresponse in the forced-choice

format. Sudman and Bradburn (1982) point out that there are a number of reasons a respondent

may leave an option unchecked in a check-all question: the option doesn’t apply to them, they

are neutral or undecided about it, or they overlooked it. The forced-choice format, they argue,

eliminates some of this uncertainty by providing a “no” category for respondents to mark if the

option does not apply to them. However, nonresponse in the forced-choice format may occur if

respondents treat it as a check-all question by ignoring the “no” or otherwise negative category

6

and only marking within the “yes” category. If a large proportion of respondents to the forced-

choice questions treat them as check-all questions, thus increasing item nonresponse, the data

may not be more interpretable than the check-all data.

PROCEDURES

We explore these issues using up to four experimental variations of substantively different

questions from two web surveys and one paper survey, all designed to assess the undergraduate

experience at Washington State University (WSU). The first web survey was conducted from

March 11 to May 9, 2003, a year after the paper survey. The sample consisted of 3,004

randomly selected WSU undergraduate students from the Pullman campus. Of these students

1,591 completed the survey for a response rate of 53 percent. Three of the 21 questions from this

survey are analyzed in the current paper. All of the questions appeared on their own page in

black text against a colored background. Answer spaces appeared in white so as to provide

contrast between the answer spaces and the background. All screens were constructed with

HTML tables using proportional widths in order to maintain the visual aspect of the screen

regardless of individual user window sizes. In addition, font size and style were automatically

adjusted using Cascading Style Sheets to accommodate differing user screen resolutions.

The second web survey, conducted in the fall of 2003, was administered to a sample of 3,045

WSU undergraduates, 1,705 of whom completed the entire survey (56 percent response rate).

This survey was designed to assess students’ experiences both on and off campus in Pullman so

as to allow for variation in the substance of the questions. Many design aspects of this survey

were similar to the first including the page-by-page construction; the contrast between the

colored background, black text, and white answer spaces; and the precautions taken to minimize

the extent to which the appearance of the questions would vary across browsers or hardware

configurations. Seven questions from this second web survey are analyzed in the current paper.

Contact letters and follow-ups were sent using both postal mail and electronic mail for each of

the web surveys. All respondents received an initial postal mail contact with a two-dollar

incentive as well as their individual identification number they could use to access the survey.

Those respondents for whom we were able to obtain e-mail addresses also received an e-mail

containing a direct hotlink to the survey. For each web survey four versions were constructed

using the same questions substantively, but varying their formats. A random number generator

was used to assign one of the versions to each respondent when they entered the survey. In the

first web survey 27.3 percent completed version one, 27.5 percent completed version two, 23.1

percent completed version three, and 22.1 percent completed version four. On the second web

survey 23.1 percent completed version one, 26.2 percent completed version two, 24.8 percent

completed version three, and 26 percent completed version four.

The paper survey was administered by mail from March to April 2001 to a random sample of

1,800 WSU (Pullman campus only) undergraduate students. The item from this survey used in

the current analysis was embedded on the last page of a four page questionnaire that was printed

in a two-column format on 8-1/2 x 11 inch pages with a colored background that contrasted with

white answer spaces. Four variations or versions of the questionnaire were mailed randomly to

7

equal subsamples of students in the sample; however, the experimental questions reported here

were identical in two of the four versions. A two-dollar incentive was enclosed with the first

mailing and a follow-up postcard and replacement questionnaire were mailed. The response rate

on this survey was 57.9 percent (1,042).

Statistical tests made in the analyses include chi-square tests for differences between the percent

of respondents using various response behaviors and t-tests for differences in the mean number

of response options checked between formats or in the mean time it took to complete formats.

FINDINGS

Does the Forced-choice Format Produce More Affirmative Responses?

Results in Table 1 from both web experiments and the paper experiment unequivocally support

the expectation that the forced-choice format yields more options marked affirmatively than the

check-all format. Overall in the check-all formatted versions an average of 4.1 options were

marked per question. In the forced-choice version the average number of options marked per

question was significantly higher at 5.0 (t = -18.57, p = .000). Fifteen of the sixteen comparisons

were significantly different in the expected direction and the sixteenth approached significance

(p = .054). Additionally, 91 percent of response options were marked affirmatively more often

when they appeared in the forced-choice format. These results confirm the findings of Rasinski

et al. (1994) that the forced-choice format leads to more options being marked affirmatively than

the check-all format, but more importantly, they extend those findings from paper surveys to web

surveys and across different types of questions ranging from behavioral and factual to opinion

based.

Why Might the Forced-choice Format Produce More Affirmative Responses?

Weak Satisficing. If the observed difference between the check-all and forced-choice formats is

due to weak satisficing we would expect to find a pattern of primacy in the check-all questions;

that is, items would be checked more often when they appear early in the list of response options

than when they appear later in the list. We test for primacy in the check-all format by comparing

the mean number of response options marked in the top and bottom half of the lists when those

lists are presented in original and reverse order.

The top panel of Table 2 shows such comparisons for five questions. For each of these questions

one of the treatments involved presenting the response options in the “original” order and

another involved presenting them in the “reverse” order. Only one question shows a significant

pattern of primacy. For question Q11 the average number of response options checked for

options A - E is 3.89 when they are located in the top half of the table and only 3.75 when they

are located in the bottom half of the table. Similarly, more of options F - J are marked when they

are located in the top as opposed to bottom half of the list (2.07 vs. 1.64). Both of these

differences are significant. In addition to this question seven of the eight remaining comparisons

8

are in the expected direction (more options are endorsed when they appear in the top of the list),

but none of these are significant.

The bottom panel of Table 2 shows similar comparisons, but in this case the analysis is limited to

only the first and last three response options in the list. Limiting the analysis to only the first and

last three options should provide a more refined test for primacy effects as it eliminates any

obscuring effects of the middle categories. Results indicate that four of the ten comparisons

show significant primacy patterns when analyzed in this way. In addition, all but two

comparisons are in the expected direction. Overall then, we are left with mixed evidence of

primacy in the check-all format.

The results in Table 1 tell us that the forced-choice format leads to a higher mean number of

response options marked affirmatively, but do not specify which response options are marked

more often in the forced-choice format. However, if the forced-choice format is eliminating

satisficing (in the form of primacy) from the check-all format, we would expect to see a greater

increase in the bottom half of the list of response options than we see in the top half when we

compare these two question formats. Therefore, an additional test of satisficing consists of

comparing the mean number of options marked in the top and the bottom half of lists of response

options across question formats.

Table 3 shows the mean number of response options marked in the top (and bottom) half of the

forced-choice formatted versions expressed as a proportion of the mean number of response

options marked in the top (and bottom) half of the check-all formatted version for each question.

For example, the number 104 in the first row of Table 3 indicates that for question 11 in the first

web experiment, the top half of the forced-choice version produced a mean number of response

options marked that was 4 percent higher than the mean number in the top half of the check-all

version. The bottom half of the forced-choice version produced a mean number of response

options marked that was 16 percent higher than the mean number in the bottom half of the

check-all version. The difference between the 4 percent increase in the top half and the 16

percent increase in the bottom half when the question is converted to a forced choice format is in

the expected direction, but is not statistically significant. In fact, 10 of the 16 comparisons were

in the expected direction, but only five were significant. In addition, six comparisons were not in

the expected direction, four of which occurred in versions where the response options were

reversed from the original order (which yielded results in the expected direction). These findings

suggest that the mean number of response options marked by location may be more closely

linked to the options themselves than to respondent answer strategies such as satisficing.

Overall, the difference between the mean proportional increase in the top half (22%) and the

mean proportional increase in the bottom half (33%) was only approaching significance (t =

1.58, p = .068). Thus, we are left with minimal evidence that the forced-choice format corrects

for any primacy that may occur in the check-all format.

The lack of primacy and correction for primacy found in these questions is evidence against the

hypothesis that the increased affirmative responses in the forced-choice format are the result of

satisficing in the check-all format. These findings are consistent with those reported by Rasinski

et al. (1994).

9

Depth of Processing. If the explanation for the increased affirmative selection of response

options in the forced-choice format is deeper processing, then we would expect respondents to

spend more time on the forced-choice formatted versions. Paradata results shown in Table 4

indicate that respondents overwhelmingly spent more time on the questions when they were

formatted as forced-choice questions than when they are formatted as check-all questions. This

finding occurs across all 15 comparisons from the two web surveys. Some of this additional

time spent in the forced-choice format is undoubtedly spent marking the “no” category, a step

that is not required on the check-all format; however, the magnitude of the time differences

between the formats suggests that respondents are spending more time on the forced-choice

format independent of this extra response step. At minimum, respondents spent 45 percent more

time on the forced-choice format than on the check-all format and on average they spent two and

a half times as much time on the forced-choice formatted questions as on the check-all formatted

questions.

Further analysis of the time spent on these two question formats can be found in Table 5 which

shows the mean number of response options marked affirmatively by those taking the mean time

or less to complete each version of each question and those taking over the mean time to

complete each version. This table shows that on the check-all formatted versions, respondents

who spent above the mean amount of time on the questions checked more answers on average

than those who spent the mean amount of time or less, suggesting that those spending more time

were processing the response options more deeply and thus finding more response options that

were relevant to them. These differences were significant on all 14 check-all treatments.

Further analysis (not shown) indicates that the mean number of options marked by the

respondents to the check-all version who used greater than the mean amount of time approached

the mean number of options marked affirmatively by all respondents on the forced-choice

version. In fact, for 11 of the 15 web comparisons the mean number of items marked by the

check-all respondents who took above the mean time to answer surpassed the mean for the

forced-choice version, while a 12th

comparison matched. Overall, respondents to the check-all

version who used greater than the mean amount of time marked 5.4 options on average while all

respondents to the forced-choice version marked an average of 5.1 options (two-sided t = 1.91, p

= .077).

The pattern of differences in the mean number of response options marked based on amount of

time spent answering found in the check-all versions (Table 5) did not carry over into the forced-

choice versions of these questions. In 15 of 19 forced-choice treatments there was no significant

difference in the mean number of response options marked affirmatively between those taking up

to the mean amount of time and those taking above the mean amount of time to complete the

question.1 These findings suggest that the additional time spent on the forced-choice format that

we see in Table 4 is sufficient enough for respondents to more deeply process all of the response

options such that spending even more time does not lead to more options being marked.

1 Since this table relies on measures of central tendency which in turn rely on data reduction, OLS regressions were

conducted in which the number of response options marked affirmatively was regressed on the time taken to answer

the question for each treatment of each question. The results (not shown) confirm those in Table 6 such that time

was positively and significantly related to the number of options marked in the check-all format, but not in the

forced-choice format.

10

Taken together, the findings in Tables 4 and 5 as well as the additional analysis of the check-all

responses by time support the explanation that the increase in the mean number of response

options marked in the forced-choice format compared to the check-all format is the result of

deeper processing. In addition they suggest that there is some level of “optimal” processing

which respondents to the forced-choice format and those using over the mean amount of time in

the check-all format seem to be reaching.2 As such they raise concerns about the use of the

check-all format because a substantial proportion of respondents spent at or below the mean

amount of time (range 47% - 76%, mean across questions = 66%) on the questions in this format

and therefore may not have reached that “optimal” processing level.

With this data in mind we returned to the issue of primacy as an indicator for satisficing in the

check-all questions and the question of whether or not the deeper processing in the forced-choice

format is eliminating primacy effects that we see in the check-all format. We tested for primacy

separately for those spending up to the mean amount of time on the check-all questions and those

spending above the mean amount of time. Results in Table 6 indicate that for those spending up

to the mean amount of time, five of the ten comparisons of the mean number of options selected

when they appeared in the top and bottom halves of lists show significant primacy effects.

However, eight of the ten comparisons show significant primacy effects for this same group

when the analysis is limited to the top and bottom three response options in the lists rather than

the top and bottom halves of options (thus eliminating the obscuring effects of the middle

categories which don’t significantly change position in the reversal). For those spending above

the mean amount of time on these same questions there is virtually no evidence of primacy

effects (Results not shown, but available on request). Only one comparison is significant

regardless of whether all the options are considered or only the top and bottom three (Q11,

options A-C in the top three/bottom three analysis).

Acquiescence. The hypothesis that the forced-choice format leads to more affirmative responses

because respondents are apprehensive about outright rejecting response options by marking “no”

and thus are more likely to agree is examined for two questions from the second web experiment

(Fall 2003). The first question presented respondents with descriptors of the WSU Pullman

campus and asked them which they thought applied. Among the experimental treatments was

one in which a third category, “don’t know,” was offered in addition to “yes” and “no”

categories. Similarly, a third category, “neutral,” was offered on one treatment for a question

asking respondents to report which Cougar varsity sports they were a fan of. If acquiescence is

at work we expect to see the “don’t know” or “neutral” category drawing those apprehensive

respondents predominantly from the “yes” category.

As Tables 7 and 8 illustrate, in neither of these questions did the third category draw respondents

predominantly from the “yes” category as expected. Instead, for the campus descriptors question

(Table 7), the “don’t know” category was not used sufficiently enough to draw significant

numbers of respondents from either the “yes” or the “no” categories. If anything, it appears as if

the addition of this third category increased the use of the “no” category, although there were

2 Respondents to the check-all version who used over the mean amount of time still used significantly less time than

respondents to the forced-choice version on 12 of the 15 comparisons. This suggests that the forced-choice format

may be more burdensome than what is “optimal”, but the additional burden may be a small price to pay for a gain in

accuracy.

11

few significant differences among individual response options. However, it did significantly

reduce both the mean number of options marked “yes” and the mean number of options marked

“no.” This finding serves as evidence against acquiescence as an explanation. On the sports fan

questions the findings are a little clearer, but no more in support of an acquiescence explanation.

Table 8 indicates that the addition of the third category drew significantly from the “no” category

for each item except for one, men’s football, which was overwhelmingly popular on all three

surveys. The third category did not result in significant reductions of the use of the “yes”

category for any one response option and it did not significantly reduce the mean number of

options marked yes overall. As a result this question also provides evidence against the

acquiescence explanation.

Table 9 reports the results of two experiments from the first web survey that tested the results of

the use of more active category labels to replace the standard yes/no labels. In the first question

respondents to one version were asked to respond either “yes” or “no” to indicate whether or not

they were fans of each of the cougar varsity sports. Respondents to another version were asked

the same question except the yes/no categories were replaced with “fan” and “not a fan”

categories. In the second question the yes/no categories were replaced with “participated” and

“have not participated” on a question asking respondents if they had participated in various types

of student groups. Consistent with the lack of acquiescence found above, results from both of

these questions indicate that removing items from the agree/disagree context through the use of

more active labels does not produce significantly different response distributions. Across both

questions only one response option was marked by a significantly different percent of

respondents when the active labels were provided. In addition, there was no difference on either

question in the mean number of response options marked by respondents across the treatments.

The finding of no difference between the use of yes/no categories and more active category

labels provides further evidence against acquiescence as an explanation.

Item Nonresponse in the Forced-Choice Question Format

Results shown in Table 10 indicate that between zero and about 11 percent of respondents

treated the forced-choice questions as check-all that apply questions by marking only within the

“yes” category,3 thus introducing item nonresponse. Across all 24 forced-choice treatments

included in all three surveys, the mean percent treating them as check-all questions was 2.68,

suggesting that this response pattern is not detrimental to the forced-choice format in general.4

However, the high percentages on some of the questions do raise alarms. In addition, the

extremely low percentages on other questions lead us to investigate what makes these particular

questions less likely to produce check-all response patterns.

One possible explanation is that respondents treat forced-choice question as check-all questions

more often when they mark the first response option affirmatively because they then continue to

3 Respondents who marked all of the options “yes” were excluded from these percentages as we assume they

sincerely meant “yes” on all options and were not treating the question as a check-all. 4 An additional question, Q24, is included in this analysis that is not included in previous analyses. There is no

check-all treatment for this question which precludes its inclusion in previous analyses, but that limitation is not

relevant for the current analysis.

12

focus on the “yes” category and do not notice other categories. This explanation is suggested by

the two questions in which the percent of respondents treating them as check-all questions are

significantly different in the original and reverse order formats. However, the data did not

support this explanation as only about 62 percent of respondents who treated a forced-choice

formatted question as a check-all question marked the first response option affirmatively (across

all questions). Thus, it is more likely that such response behavior is linked to the particular type

of question being asked. Specifically, questions based on opinions may require more careful

consideration by the respondents and therefore discourage treatment as check-all whereas

behavior- and factual-based questions may not require such careful consideration and as such

may facilitate “quick clicking” which may result in this answering pattern. In this data there are

eight questions that are behavior- or fact-based (designated by “B” in Table 10) and four that are

opinion-based (designated by “O”). The mean percent of respondents treating the forced-choice

questions as check-all questions is 3.47 for the behavior/fact-based questions and only 1.58 for

the opinion-based questions. This difference is approaching significance (one-sided t = -1.55, p

= .067), suggesting support for this explanation. Further support is suggested by the mean time

data in Table 4. On average the behavior/fact-based questions took 23.15 seconds to complete

while the opinion-based questions took 42.16 seconds to complete (one-sided t = 4.79, p =

.002).5 Together, these findings suggest that respondents gave more consideration to (or at least

took longer to process) the opinion-based questions which discouraged their treatment of them as

check-all questions compared to the behavior/fact-based questions.6

DISCUSSION

In this paper we have explored whether posing questions as forced-choice items rather than the

often used check-all format influences the number of options marked affirmatively in self-

administered surveys. We have also explored three theoretical explanations for why these

formats may influence the number of responses chosen as well as whether item nonresponse is a

significant issue/problem in the forced-choice format.

Consistent with experimental results from a mail self-administered survey reported by Rasinski

et al. (1994) our test of 10 items in two web experiments and a mail comparison uniformly

support the hypothesis that the forced-choice format results in more options being selected. Our

results included item order reversals, replication of one item across all three surveys, and opinion

as well as behavioral items of the nature explored by Rasinski et al (1994). Together with

previous findings these data strongly suggest that on self-administered surveys when respondents

are presented with the forced-choice format instead of the check-all format, they will select a

greater number of options.

Three possible explanations for the tendency to choose more items under the forced-choice

condition were evaluated using subsets of items. The most convincing explanation for the

5 This time data should be interpreted with caution because there are significant differences in the length of the

question stems across these two types of categories. Thus, the time differences could be due to increased reading

and comprehending requirements. 6 Use of the more active labels did not significantly influence item nonresponse in the forced-choice format (see

Table 9)

13

increases in options marked affirmatively in the forced-choice format appears to be that the

forced-choice format leads respondents to more deeply process the response options. This was

evident in the differences in the time it took to complete the two question formats as well as in

exploration of the mean number of response options marked by those taking up to and those

taking over the mean amount of time to complete each question. In the check-all format those

respondents who took more time to complete the questions marked more answers, but the same

was not found in the forced-choice format. This finding in conjunction with the overall

differences in the mean time and the mean number of response options marked across the two

formats suggests that the forced-choice format results in deeper processing of response options

among all respondents, not just those who take above the mean amount of time to complete

questions.

Evidence of satisficing in the form of primacy effects in check-all questions was weak when all

respondents were included in the analysis; however, when analyses were conducted separately

for those taking up to the mean amount of time to complete the questions and those taking above

the mean amount of time, fairly strong evidence of primacy was found for those who answered

the questions relatively quickly. Taken together, the evidence suggests that respondents who

take longer to answer questions in the check-all format and respondents to questions in the

forced-choice format reach a more “optimal” level of processing than do the approximately 66

percent of respondents who only take up to the mean amount of time to answer questions in the

check-all format. This deeper processing seems to eliminate satisficing behavior in that no

evidence of primacy was found for respondents to the check-all format who took above the mean

amount of time to complete the questions. Thus, while the evidence suggests that most

respondents in the forced-choice format are adequately processing and responding to each

question and its response options, the same cannot be said for the check-all format.

Virtually no evidence was found to support the presence of acquiescence in the forced-choice

question as the addition of a third category (“don’t know” or “neutral”) did not draw significant

numbers of respondents from the use of the “yes” category. In addition, the use of more active

category labels, thought to reduce acquiescence by eliminating the agree/disagree context of the

question and replacing it with more tangible options, had no significant effects on response

distributions. However, since our analysis was limited to only two items, further testing of this

explanation is needed.

Finally, our exploration of item nonresponse in forced-choice questions indicated that this is not

an enormous impediment to the forced-choice question format. The proportion of respondents

who treated the forced-choice questions as defacto check-all questions tended to be small,

usually less than five percent. However, the occurrence of this response pattern seemed to vary

across different types of questions. With regard to these findings, it is important to note that the

wording of all of the questions tested here avoided a kind of prose that sometimes leads

respondents to mark only “yes” answers (e.g. “Please check which of these sports you are a fan

of”) by including the positive and negative categories as part of the question stem. Question

wording that uses such prose may encourage the occurrence of this response pattern in forced-

choice questions and therefore should be avoided. Nevertheless, in the cases we presented here

we do not see the relatively small number of people who chose not to mark negative answers as a

major barrier to the use of a forced-choice format.

14

CONCLUSION

Using results of experiments from two web surveys and a paper survey comparison we have

confirmed the finding of Rasinski et al. (1994) that the forced-choice format produces more

affirmatively marked response options than the check-all format and we have extended this

finding in three important ways. First, after verifying Rasinski and colleagues’ finding in a paper

survey, we extended it to web surveys. Second, this finding holds across a wide range of

question types (including both behavior/fact-based questions and opinion-based questions) as

well as across a variety of numbers of response options (9 to 14). The third and most important

extension of Rasinski and colleagues’ finding is our finding that the forced-choice format

appears to invoke deeper processing and to eliminate weak satisficing behavior that occurs

among some respondents to the check-all format. It appears as if the increased marking of

options affirmatively is due in large part to deeper processing of the forced-choice format,

confirming the speculation of several researchers (Sudman and Bradburn 1982; Dillman 2000).

We find no evidence in support of acquiescence in the forced-choice format.

It appears that the use of the forced-choice question format is a desirable alternative to the use of

the check-all question format for multiple answer questions in web surveys because it seems to

promote deeper processing of the question and response options and allows for finer

differentiation of meaning for options marked negatively. Although the evidence that the forced-

choice format produces more accurate responses is increasing, like Rasinski et al. (1994) we lack

external validation checks for our data and therefore cannot say with absolute certainty that the

forced-choice format produces more accurate responses. As such, this is an issue in need of

further research.

In addition to external validation checks, an important next step in this research is to compare the

use of forced-choice formats in aural (e.g. telephone) surveys where the check-all format is

rarely used to its use in self-administered surveys. Although we do not yet know whether the

same number of items are likely to be chosen across modes, the forced-choice format, which is

similar in wording to that used in aural surveys, produces selection of significantly more items

than the number chosen in the check-all format on self-administered surveys. This finding gives

ample reason to be concerned about the current common practice of automatically converting

those items from forced-choice to check-all formats when switching from telephone to web or

other self-administered modes.

15

Table 1: Comparison of Mean Number of Response Options Marked Affirmatively

Between the Check-All-That-Apply and Forced-Choice Format

One-Sided

Test of

Significance

Mean:

Check-All

Version

Mean:

Forced-

choice

Version

Difference in

Means T p

WEB EXPERIMENT #1: Spring 2003

Q11: Resources used at WSU (10)

†

Check-All vs. Used/Not Used 5.4 5.7 -0.3 -3.41 .000*

Check-All (R) vs. Used/Not Used (R)††

5.6 6.2 -0.6 -5.43 .000*

Q13: Cougar varsity sports fan (15)

Check-All vs. Yes/No 2.6 3.6 -1.0 -4.43 .000*

Check-All vs. Fan/Not a Fan 2.6 3.9 -1.3 -5.87 .000*

Q16: Student group participation at WSU (11) Check-All vs. Yes/No 1.9 2.6 -0.7 -5.01 .000*

Check-All vs. Part./Not Part. 1.9 2.4 -0.5 -3.67 .000*

WEB EXPERIMENT #2: Fall 2003

Q3: Descriptions of WSU Pullman Campus (12) Check-All vs. Yes/No 4.4 6.6 -2.2 -17.68 .000*

Q6: Admittance Criteria (14) Check-All vs. Yes/No 5.0 6.1 -1.1 -6.27 .000*

Check-All (R) vs. Yes/No (R) 5.2 5.9 -0.7 -4.46 .000*

Q11: University Budget Adjustments (14) Check-All vs. Yes/No 3.5 4.6 -1.1 -7.24 .000*

Q14: Cougar varsity Sports Fan (15) Checked vs. Yes/No 3.1 4.5 -1.4 -6.19 .000*

Q16: Food vendors on campus (9) Checked vs. Yes/No 4.4 5.0 -0.6 -3.56 .000*

Checked (R) vs. Yes/No (R) 4.6 5.1 -0.5 -3.79 .000*

Q20: Possessions in Pullman (13) Check-All vs. Yes/No 6.4 6.7 -0.3 -1.94 .026*

Check-All (R) vs. Yes/No (R) 6.6 6.9 -0.3 -1.61 .054

PAPER EXPERIMENT: Spring 2002


Check-All vs. Yes/No 2.6 3.8 -1.2 -5.94 .000*

OVERALL MEAN 4.1 5.0 -0.9 -18.57 .000* † The number of response options offered for each question is displayed in parenthesis.

† † “(R)” denotes treatments in which the options were presented in reverse order (inverted).

* p ≤ .05

16

Table 2: Primacy Effects in the Check-All Format: Mean Number of Options Checked

when Options Appeared in (1) Top vs. Bottom Half and (2) Top Three vs.

Bottom Three Positions in List

Mean Number of Options Checked by Location: Top and Bottom Half of List

ORIGINAL

ORDER

REVERSE

ORDER t p

WEB EXPERIMENT #1

Q11: Resources Used at WSU (10)

†

Options A - E 3.89 T 3.75 B 1.92 .027*

Options F - J 1.64 B 2.07 T 5.89 .000*

Q13: Cougar Varsity Sports Fan (15)

Options A - G 1.63 T 1.71 B -0.91 .817

Options I - O 0.96 B 1.03 T 0.73 .233

WEB EXPERIMENT #2

Q6: Admittance Criteria (14)

Options A - G 2.17 T 2.14 B 0.32 .374

Options H - N 2.44 B 2.55 T 0.95 .171

Q16: Food Vendors on Campus (9)

Options A - D 2.41 T 2.32 B 0.89 .184

Options F - I 1.31 B 1.37 T 0.67 .252

Q20: Possessions in Pullman (13)

Options A - F 2.41 T 2.38 B 0.41 .341

Bottom H - M 2.61 B 2.79 T 1.57 .058

Mean Number of Options Checked by Location: Top and Bottom Three Positions of List

WEB EXPERIMENT #1


Options A - C 2.32 T 2.16 B 2.89 .002*

Options H - J 0.51 B 0.95 T 8.13 .000*


Options A - C 0.71 T 0.79 B -1.18 .882

Options M - O 0.55 B 0.61 T 0.91 .183

WEB EXPERIMENT #2


Options A - C 2.06 T 2.04 B 0.29 .385

Options L - N 0.68 B 0.81 T 2.10 .018*


Options A - C 2.08 T 1.98 B 1.55 .060

Options G - I 0.82 B 0.89 T 1.09 .137


Options A - C 0.92 T 0.92 B -0.01 .504

Bottom K - M 1.30 B 1.43 T 1.98 .024*

† The number of response options offered for each question is displayed in parenthesis. * p ≤ .05

17

Table 3: Mean Number of Options Marked by Location in the Forced-Choice Version

Expressed as a Proportion of the Mean Number of Options Checked by

Location in the Check-All Version.1

Proportion

Test of

Significance

Top Bottom X

2 p

WEB EXPERIMENT #1


†

Check-All vs. Used/Not Used 104 116 3.45 .063

Check-All (R) vs. Used/Not Used (R)††

115 110 0.61 .436


Check-All vs. Yes/No 115 163 21.81 .000*

Check-All vs. Fan/Not a Fan 125 183 16.86 .000*

Q16: Student group participation at WSU (11) Check-All vs. Yes/No 144 117 4.41 .036

Check-All vs. Participated/Not Participated 139 105 8.13 .004

WEB EXPERIMENT #2

Q3: Descriptions of WSU Pullman Campus (12) Check-All vs. Yes/No 139 164 5.83 .016*

Q6: Admittance Criteria (14) Check-All vs. Yes/No 112 126 3.22 .073

Check-All (R) vs. Yes/No (R) 126 115 0.74 .391

Q11: University Budget Adjustments (14) Check-All vs. Yes/No 128 135 0.58 .448

Q14: Cougar varsity Sports Fan (15) Checked vs. Yes/No 137 185 13.23 .000*

Q16: Food vendors on campus (9) Checked vs. Yes/No 107 115 1.07 .301

Checked (R) vs. Yes/No (R) 127 112 3.25 .072

Q20: Possessions in Pullman (13) Check-All vs. Yes/No 100 108 1.47 .225

Check-All (R) vs. Yes/No (R) 111 105 0.78 .378

PAPER EXPERIMENT


Check-All vs. Yes/No 125 170 17.19 .000*

MEAN PROPORTIONAL CHANGE 122 133 t = -1.58 .068

1 The proportional change is calculated by: (χfc/χck) *100 Where χfc is the mean number of options marked in

the top (or bottom) half of the forced-choice version and χck is the mean number of options checked in the top

(or bottom) half of the check-all version. * p ≤ .05 and in the expected direction. † The number of response options offered for each question is displayed in parenthesis.


* p ≤ .05

18

Table 4: Comparisons of the Mean Time (seconds) Spent Answering Questions Between

the Check-All and Forced-Choice Formats

Mean Time One-Sided Test

of Significance

Check-

All

Forced-

choice Difference

Percent

Increase T p



†

Check-All vs. Used/Not Used 15.88 24.97 -9.09 57 -13.42 .000*

Check-All (R) vs. Used/Not Used

(R)††

19.22 27.89 -8.67 45 -14.45 .000*


Check-All vs. Yes/No 14.12 30.46 -16.34 116 -13.92 .000*

Check-All vs. Fan/Not a Fan 14.12 27.98 -13.86 98 -17.73 .000*

Q16: Student group participation at

WSU (11)


Check-All vs. Part./Not Part. 16.48 27.02 -10.54 64 -13.16 .000*


Q3: Descriptions of WSU Pullman

Campus (12)


Q6: Admittance Criteria (14) Check-All vs. Yes/No 13.04 42.39 -29.35 225 -20.35 .000*

Check-All (R) vs. Yes/No (R) 13.16 44.15 -30.99 235 -20.64 .000*

Q11: University Budget Adjustments

(14)


Q14: Cougar varsity Sports Fan (15) Checked vs. Yes/No 5.50 27.06 -21.56 392 -28.02 .000*

Q16: Food vendors on campus (9) Checked vs. Yes/No 5.18 12.48 -7.3 141 -19.57 .000*

Checked (R) vs. Yes/No (R) 5.58 12.56 -6.98 125 -22.30 .000*



Check-All (R) vs. Yes/No (R) 7.99 21.72 -13.73 172 -22.79 .000*

OVERALL MEAN 12.03 29.07 -17.04 159 -6.58 .000*

† The number of response options offered for each question is displayed in parenthesis.


* p ≤ .05

Time outliers were removed at 2 standard deviations above the mean.

19

Table 5: Mean Number of Response Options Marked Affirmatively by Those Taking

Longer Than the Mean Time Per Treatment and Those Taking Less than or

Equal to the Mean Time Per Treatment

One-Sided Test

of Significance

Above

Mean Time

Mean Time

and Below

Difference

in Means t p


Q11: Resources used at WSU (10)†

Check-All 5.8 5.2 0.6 3.84 .000*

Check-All (R)††

5.8 5.5 0.3 1.74 .041*

Used/Not Used 5.8 5.7 0.1 0.40 .347

Used/Not Used (R) 6.2 6.2 0.0 0.05 .479


Check-All 4.3 1.5 2.8 13.52 .000*

Check-All (R) 4.1 1.7 2.4 10.63 .000*

Fan/Not a Fan 5.1 3.4 1.7 4.26 .000*

Yes/No 4.2 3.6 0.6 0.68 .249

Q16: Student group participation at WSU (11) Check-All 2.4 1.7 0.7 4.18 .000*

Participated/Not Participated 2.9 2.2 0.7 2.98 .002*

Yes/No 3.2 2.4 0.8 3.52 .000*


Q3: Descriptions of WSU Pullman Campus (12) Check-All 5.4 3.7 1.7 9.73 .000*

Yes/No 6.5 6.6 -0.1 -0.76 .775

Yes/No/Don’t Know 5.7 6.2 -0.5 -2.67 .996

No/Yes 6.3 6.7 -0.4 -1.85 .968

Q6: Admittance Criteria (14) Check-All 6.2 4.2 2.0 10.29 .000*

Check-All (R) 6.2 4.0 2.2 12.26 .000*

Yes/No 5.9 6.1 -0.2 -1.06 .854

Yes/No (R) 5.7 6.1 -0.4 -1.61 .946

Q11: University Budget Adjustments (14) Check-All 4.8 2.9 1.9 10.67 .000*

Yes/No 4.8 4.4 0.4 1.56 .059

Q14: Cougar varsity Sports Fan (15)

Check-All 5.2 2.1 3.1 16.49 .000*

Yes/No 5.3 4.1 1.2 2.90 .002*

Yes/Neutral/No 4.0 3.6 0.4 1.05 .148

Yes/No/Neutral 4.3 4.0 0.3 0.70 .241

Q16: Food vendors on campus (9)

Check-All 6.0 3.9 2.1 9.46 .000*

Check-All (R) 5.9 3.9 2.0 10.37 .000*

Yes/No 4.4 5.1 -0.7 -2.43 .992

Yes/No (R) 4.9 5.3 -0.4 -1.58 .942


Check-All 7.9 5.8 2.1 7.98 .000*

Check-All (R) 7.8 6.1 1.7 7.66 .000*

Yes/No 6.3 6.8 -0.5 1.53 .936

Yes/No (R) 6.5 7.0 -0.5 1.95 .974


† † “(R)” denotes

treatments in which the options were presented in reverse order (inverted). * p ≤ .05

Time outliers were removed at 2 standard deviations above the mean.

20

Table 6: Primacy Effects in Check-All Format For Those Spending The Mean Amount of

Time or Below

Mean Number of Options Checked by Location: Top and Bottom Half of List

ORIGINAL

ORDER

REVERSE

ORDER t p

WEB EXPERIMENT #1


†

Options A - E 3.81 T 3.68 B 1.46 .072

Options F - J 1.51 B 2.07 T 6.37 .000*


Options A - G 1.16 T 1.26 B 0.97 .833

Options I - O 0.38 B 0.60 T 2.49 .007*

WEB EXPERIMENT #2


Options A - G 2.05 T 1.82 B 2.52 .006*

Options H - N 2.10 B 2.15 T 0.34 .368


Options A - D 2.52 T 2.40 B 1.29 .098

Options F - I 1.15 B 1.32 T 1.85 .032*


Options A - F 2.48 T 2.45 B 0.39 .348

Bottom H - M 2.59 B 2.88 T 2.30 .011*

Mean Number of Options Checked by Location: Top and Bottom Three Positions of List

WEB EXPERIMENT #1


Options A - C 2.25 T 2.11 B 2.07 .019*

Options H - J 0.39 B 0.96 T 9.15 .000*


Options A - C 0.40 T 0.50 B -1.30 .903

Options M - O 0.23 B 0.37 T 2.38 .009*

WEB EXPERIMENT #2


Options A - C 1.91 T 1.68 B 2.72 .003*

Options L - N 0.51 B 0.68 T 2.15 .016*


Options A - C 1.97 T 1.76 B 2.52 .006*

Options G - I 0.61 B 0.72 T 1.67 .048*


Options A - C 0.83 T 0.81 B 0.36 .359

Bottom K - M 1.15 B 1.30 T 2.01 .022*


* p ≤ .05

21

Tab

le 7

: E

ffec

ts o

f ad

din

g a

“D

on

’t K

now

” C

ate

gory

to a

Fo

rced

-Ch

oic

e Q

ues

tion

Q3

: H

ere

are

so

me

wa

ys

that

the

WS

U P

ull

ma

n C

am

pu

s h

as

bee

n d

escr

ibed

. P

lea

se c

hec

k e

ach

des

crip

tio

n t

ha

t y

ou

th

ink

acc

ura

tely

des

crib

es t

his

cam

pu

s.

V

ER

SIO

N 2

V

ER

SIO

N 3

TE

ST

S O

F S

IGN

IFIC

AN

CE

YE

SE

S

TE

ST

S O

F S

IGN

IFIC

AN

CE

NO

S

Yes

N

o

Yes

N

o

DK

1

Dif

f.

X2

p

Dif

f.

X2

p

Saf

e C

am

pus

85

.7

5

.2

77

.5

11

.3

3.1

8

.2

9.5

5

.00

2**

-6.1

1

1.0

9

.00

1*

Int.

Sti

mu

lati

ng

67

.5

23

.1

66

.0

23

.2

2.6

1

.5

0.2

3

.63

2

-0.1

0

.00

.9

79

Peo

ple

Fri

end

ly

84

.8

6

.1

82

.7

8

.7

0.5

2

.1

0.6

5

.42

2

-2.6

2

.31

.1

29

Fac

ult

y I

nac

cess

ible

3

7.0

5

3.8

3

0.7

5

7.4

3

.5

6.3

3

.80

.0

51

-3

.6

1.1

6

.28

1

Buil

din

gs

Mo

der

n

66

.8

24

.2

65

.0

24

.8

1.9

1

.8

0.3

1

.57

5

-0.6

0

.04

.8

35

Cla

sses

to

o L

arge

32

.7

58

.3

27

.7

62

.6

1.4

5

.0

2.6

5

.10

4

-4.3

1

.72

.1

90

Eq

uip

. U

p t

o D

ate

74

.7

16

.1

68

.8

20

.3

2.8

5

.9

3.6

9

.05

5

-4.2

2

.56

.1

10

Dri

ver

s U

nfr

iend

ly

28

.3

62

.6

24

.8

65

.5

1.7

3

.5

1.3

1

.25

3

-2.9

0

.81

.3

69

Po

liti

call

y A

ctiv

e 4

5.3

4

3.9

3

8.1

4

5.2

8

.5

7.2

4

.67

.0

31

*

-1.3

0

.13

.7

20

Cam

pus

Co

nges

ted

2

6.7

6

3.2

2

4.3

6

5.2

2

.1

2.4

0

.62

.4

31

-2

.0

0.3

9

.53

5

Lac

ks

Div

ersi

ty

28

.3

62

.3

27

.4

63

.4

1.2

0

.9

0.0

7

.78

6

-1.1

0

.10

.7

55

Too

Sp

ort

s O

rien

ted

2

2.0

6

8.4

1

8.9

7

1.4

1

.2

3.1

1

.25

.2

64

-3

.0

0.9

3

.33

4

Mea

n

6.6

5.4

6.0

5.6

0

.3

4.0

-2

.7

N (

1,7

05

) 4

46

42

3

Dif

f. o

f m

ean

s

t =

4.8

6, p

= .

00

0

D

iff.

of

mea

ns

t =

-2

.38,

p =

.0

18

1 D

K s

tand

s fo

r d

on’t

kno

w.

* p

≤ .0

5

22

Tab

le 8

: E

ffec

ts o

f ad

din

g a

“N

eutr

al”

Cate

go

ry t

o a

Fo

rced

-Ch

oic

e Q

ues

tion

Q1

4:

Wh

ich

of

the

foll

ow

ing

Co

ug

ar

va

rsit

y s

po

rts

wo

uld

yo

u c

on

sid

er y

ou

rsel

f to

be

a f

an

of?

P

lea

se C

hec

k-a

ll t

ha

t a

pp

ly.

V

ER

SIO

N 2

V

ER

SIO

N 4

TE

ST

S O

F S

IGN

IFIC

AN

CE

YE

SE

S

TE

ST

S O

F S

IGN

IFIC

AN

CE

NO

S

Y

es

No

Y

es

No

N

eutr

al

Dif

f.

X2

p

Dif

f.

X2

p

Men

’s b

aseb

all

35

.4

52

.2

30

.9

36

.1

26

.0

4

.5

2.0

3

.15

4

16

.1

23

.43

.00

0*

Wo

men

’s b

asket

bal

l 2

3.8

6

2.8

1

9.2

4

4.9

2

8.7

4.6

2

.76

.0

96

1

7.9

2

8.5

3

.0

00

*

Men

’s b

asket

bal

l 4

3.7

4

3.3

4

0.0

2

9.3

2

3.7

3.7

1

.30

.2

55

1

4.0

1

8.6

4

.0

00

*

Wo

men

’s c

ross

co

untr

y

11

.2

74

.4

10

.4

57

.6

24

.4

0

.8

0.1

6

.69

1

16

.8

28

.22

.00

0*

Men

’s c

ross

co

untr

y

10

.1

75

.8

10

.8

56

.4

25

.3

-0.7

0

.13

.7

16

1

9.4

3

7.1

6

.0

00

*

Men

’s f

oo

tbal

l 7

7.4

1

1.7

7

7.0

1

0.4

5.9

0.4

0

.02

.8

93

1.3

0

.37

.54

4

Wo

men

’s g

olf

7.8

7

8.3

6.3

6

2.3

2

4.2

1.5

0

.79

.3

75

1

6.0

2

7.0

8

.0

00

*

Men

’s g

olf

9.2

7

6.7

9.0

5

9.4

2

4.2

0.2

0

.01

.9

33

1

7.3

3

0.6

4

.0

00

*

Wo

men

’s r

ow

ing

18

.2

67

.7

13

.5

54

.9

24

.6

4

.7

3.5

5

.06

0

12

.8

15

.49

.00

0*

Wo

men

’s s

occ

er

34

.3

52

.0

37

.0

35

.7

20

.1

-2.7

0

.71

.3

98

1

6.3

2

4.1

3

.0

00

*

Wo

men

’s s

wim

min

g

15

.7

71

.1

14

.7

53

.3

24

.8

1

.0

0.1

8

.67

1

17

.8

29

.96

.00

0*

Wo

men

’s t

ennis

1

4.1

7

2.0

1

2.6

5

4.9

2

5.3

1.5

0

.42

.5

16

1

7.1

2

8.0

9

.0

00

*

Wo

men

’s t

rack

and

fie

ld

22

.6

63

.0

24

.4

45

.4

23

.0

-1.8

0

.37

.5

42

1

7.6

2

7.8

4

.0

00

*

Men

’s t

rack

and

fie

ld

26

.2

59

.9

27

.3

42

.4

22

.8

-1.1

0

.13

.7

16

1

7.5

2

7.0

2

.0

00

*

Wo

men

’s v

oll

eyb

all

47

.5

40

.1

45

.6

30

.7

16

.5

1

.9

0.3

3

.56

3

9

.4

8.6

5

.0

03

*

Mea

n

4.5

1

0.1

4.1

7.2

3.6

1

.2

15

.2

N (

1,7

05

) 4

46

44

3

Dif

f. o

f M

ean

s

t =

1.5

4, p

= .

12

4

Dif

f. o

f M

ean

s

t =

8.4

1, p

= .

00

0

No

tes:

* p

≤ .

05

23

Table 9: Effects of More Active Category Labels in Forced-Choice Questions

Q13:

Which of the following Cougar varsity sports would you consider yourself to be a fan of?

YES/NO

FORMAT

FAN/NOT A

FAN FORMAT

SIGNIFICANCE TESTS

YES VS. FAN

Yes No Fan Not a Fan X2 p

Men’s baseball 32.8 61.8 37.1 57.5 1.46 .228

Women’s basketball 14.5 79.8 19.9 72.8 3.61 .057

Men’s basketball 35.6 60.1 37.9 55.6 0.40 .530

Women’s cross-country 7.7 86.6 9.2 82.0 0.57 .450

Men’s cross-country 8.3 85.5 10.4 81.5 0.93 .335

Men’s football 78.9 19.4 80.4 16.1 0.24 .626

Women’s golf 10.0 84.0 9.3 81.7 0.10 .748

Men’s golf 13.7 80.3 12.8 79.0 0.12 .731

Women’s rowing 17.1 77.2 16.9 74.4 0.01 .943

Women’s soccer 35.0 60.7 34.6 58.0 0.02 .902

Women’s swimming 11.4 83.2 15.5 75.5 2.63 .105

Women’s tennis 10.5 83.5 15.3 76.3 3.54 .060

Women’s track and field 19.9 74.6 22.9 69.8 0.92 .336

Men’s track and field 22.5 72.4 27.2 65.7 2.16 .142

Women’s volleyball 39.9 55.8 43.6 49.3 1.02 .314

Mean # Marked Affirmatively 3.58 3.93 t = 1.32 .186

N 351 367

Q16: What types of student groups, if any, have you participated in while a student at WSU?

YES/NO

FORMAT

PARTICIPATED/

NOT PART.

FORMAT

SIGNIFICANCE TEST

YES VS. PART.

Yes No Part. Not Part. X2 p

Academic Orgs. 39.3 58.4 39.8 56.7 0.02 .898

Community Service Orgs. 38.5 59.5 36.2 58.9 0.38 .538

Entertainment and Social Orgs. 38.2 59.3 38.4 57.8 0.00 .947

Ethnic and Cultural Orgs. 15.4 81.5 17.4 77.7 0.55 .458

Fraternity or Sorority 25.1 73.2 18.5 76.6 4.52 .034*

Intramural and Rec. Sports Clubs 51.3 47.3 44.4 51.5 3.39 .066

Performing and Fine Arts Groups 11.4 85.5 12.0 82.3 0.06 .805

Religious Orgs. 19.4 77.5 17.2 77.4 0.59 .444

Student Gov. and Political Orgs. 14.2 82.9 12.5 82.3 0.45 .501

Women’s and Sexuality Orgs. 4.8 91.7 4.1 90.2 0.24 .624

Mean 2.58 2.41 t = -1.14 .255

N 351 367

* p ≤ .05

24

Table 10: Treatment of the Forced-Choice Format as a Check-All-That-Apply Format

SIGNIFICANCE TESTS ORIG.

ORDER

REV.

ORDER X2 p

QUESTION

TYPE

WEB EXPERIMENT # 1


† B

1

Used/Not Used 10.9 3.7 13.59 .000*

Q13: Cougar varsity sports fan (15) B

Fan/Not a Fan 6.5 -- -- --

Yes/No 4.3 -- -- --

Q16: Student Group Participation (11) B

Participated/Not Participated 3.0 -- -- --

Yes/No 2.0 -- -- --

WEB EXPERIMENT # 2

Q3: Describe WSU Pullman Campus (12) O

2

Yes/No 0.0 -- -- --

Yes/No/Don’t Know 0.0 -- -- --

No/Yes 0.0 -- -- --

Q6: Admittance Criteria (14) O

Yes/No 0.0 0.0 -- --

Q11: University Budget Adjustments (14) O

Yes/No 6.1 -- -- --


Yes/No 3.4 -- -- --

Yes/Neutral/No 0.5 -- -- --

Yes/No/Neutral 0.5 -- -- --

Q16: Food vendors on campus (9) B

Yes/No 0.7 1.6 1.44 .230

Q20: Possessions in Pullman (13) B

Yes/No 0.2 0.0 1.05 .306

Q24: WSU descriptors (10/5)

Yes/No (Long List) 4.8 1.6 7.42 .006* O

Yes/No (Short List) 2.4 0.9 2.90 .088

PAPER EXPERIMENT


Yes/No 11.3 -- -- --


* p ≤ .05 1 “B” stands for behavior/fact based question

2 “O” stands for opinion based question

25

WEB EXPERIMENT # 1 (SPRING 2003) Question #11 Question #13 Question #16

Ver

sio

n 1

Ver

sio

n 2

Ver

sio

n 3

Ver

sio

n 4

26

WEB EXPERIMENT # 2 (FALL 2003) Question #3 Question #6 Question #11

Ver

sio

n 1

Ver

sio

n 2

Ver

sio

n 3

Ver

sio

n 4

27

WEB EXPERIMENT # 2 (FALL 2003) Question #14 Question #16 Question #20

Ver

sio

n 1

Ver

sio

n 2

Ver

sio

n 3

Ver

sio

n 4

28

WEB EXPERIMENT # 2 (FALL 2003) PAPER SURVEY (SPRING 2001)

Question #24 Question #5

Ver

sio

n 1

Ver

sio

n 2

Ver

sio

n 1

Ver

sio

n 3

Ver

sio

n 4

Ver

sio

n 2

29

REFERENCES

Best, Samuel J. and Brian Krueger. In press. Internet Data Collection. Sage Publications.

Christian, Leah Melani. 2003. “The Influence of Visual Layout on Scalar Questions in Web

Surveys.” Unpublished Master’s Thesis, Washington State University.

Christian, Leah Melani and Don A. Dillman. 2004. “The Influence of Graphical and Symbolic

Language Manipulations on Responses to Self-Administered Questions.” Public Opinion

Quarterly. 68:58-81.

Dillman, Don A. 2000. Mail and Internet Surveys: The Tailored Design Method. 2nd

ed. New

York, NY: John Wiley and Sons, Inc.

Heerwegh, D. & G. Loosveldt. 2002. “Describing Response Behavior in Websurveys Using

Client Side Paradata.” Paper Presented at the International Workshop on Web Surveys,

Mannheim, Germany, October 2002.

Krosnick, Jon A. 1991. “Response Strategies for Coping with the Cognitive Demands of Attitude

Measures in Surveys.” Applied Cognitive Psychology. 5: 213-236.

Krosnick, Jon A. 1999. “Survey Research.” Annual Review of Psychology. 50: 537-567.

Krosnick, Jon A. and D. F. Alwin. 1987. “An Evaluation of a Cognitive Theory of Response-

Order Effects in Survey Measurement.” Public Opinion Quarterly. 51: 201-219.

Rasinski, Kenneth A., David Mingay, and Norman M. Bradburn. 1994. “Do Respondents Really

‘Mark All That Apply’ on Self-Administered Questions?” Public Opinion Quarterly. 58:

400-408.

Schuman, Howard and Stanley Presser. 1981. Questions and Answers in Attitude Surveys

Experiments on Question Form, Wording, and Context. New York, NY: Academic Press.

Stern, Michael J, Don A. Dillman, Jolene D. Smyth, and Leah Melani Christian. 2004. “The

Uses of Paradata for Evaluating Alternative Versions of Web Survey Questions.” Paper

presented at the 57th

annual conference of the World Association for Public Opinion

Research. Phoenix, Arizona. May 11-13, 2004

Sudman, Seymour and Norman M. Bradburn. 1982. Asking Questions. San Francisco, CA:

Jossey-Bass.

Sudman, Seymour, Norman M. Bradburn, and Norbert Schwarz. 1996. Thinking About Answers:

The Application of Cognitive Processes to Survey Methodology. San Francisco, CA: Jossey-

Bass.

comparing check-all and forced-choice question … · comparing check-all and forced-choice...

Documents