crafting good survey questions - naval postgraduate...
TRANSCRIPT
1
Crafting Good Survey Questions!
Professor Ron Fricker!Naval Postgraduate School!
Monterey, California!
3/26/13
Reading:!Dillman et al. chapters 4 & 5!
Goals for this Lecture!
• Describe basic types of questions!– Open-ended !– Closed-ended!– Partially-closed !
• Guidelines for writing effective questions!– Lots of dos and don’ts!– Mistakes and errors to avoid when writing
questions!• Guidelines for writing effective response
scales!– More dos and don’ts!
2 3/26/13
Design Should Minimize Nonresponse, Measurement, and Other Errors!
• Instrument should be easy to understand, to navigate, and to complete!– Encourage response!
• Questions must be:!– Clear!– Unambiguous!– Minimize possible bias!
• E.g., questions with socially desirable answers!• The harder it is for the respondents, the
poorer the data quality, the higher the survey nonresponse rate, etc.!
3/26/13 3
How Hard Can It Be?!
• That’s everyone’s first reaction!• Turns out (and you will likely experience it in this
course), it’s harder than it seems!• Here are some reasons (and there are more):!
– Devil’s in the details: Differences in words and question wording matter – sometimes a lot!
– It’s not just asking questions, but asking the right questions with the proper response scales!
– The questions need to be robust to lots of factors such as how different people (mis)interpret the questions (including response scales, question ordering, etc.) – consistency is key!
ü The reality: It’s really hard to write questions that are perfectly clear and unambiguous, yet reasonably simply worded, and that get at what you want to know!
3/26/13 4
Types of Questions!
• There are only three basic types:!– Open-ended!– Closed-ended with ordered categories!– Closed-ended with unordered categories!
• Also a hybrid: partially closed-ended!– It’s a list with “Other, please specify” at end!– Useful when complete list of response options
really long!– However, these can be harder to analyze!
3/26/13 5
Question Components!
3/26/13 6
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Open-ended Questions!
• Useful for allowing respondents to provide additional data not asked for in the survey!– Can mitigate possible frustration w/ survey!
• Also good for!– Understanding reasoning behind other answers!– When possible responses exceeds reasonable list!– May be easiest way to collect information about
complicated phenomenon/issue/problem!• But, use sparingly:!
– Generally hard to analyze!– If overused, puts large burden on respondents!
3/26/13 7
Closed-ended Questions!
• Advantages:!– Minimizes respondent burden!– Easy to tabulate and quantify responses!
• Disadvantages!– Categories may be leading to respondents!– May make it too easy to answer without thinking!– May limit spontaneity !
3/26/13 8
Closed-ended Questions!
• Two types:!– Unordered responses!– Ordered responses!
• Closed-ended with unordered responses often cognitively more difficult to answer than closed-ended with ordered reponses!– But sometimes there is no natural ordering to the
response categories!
3/26/13 9
Closed-ended Question with Unordered Response Categories!
3/26/13 10
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Closed-ended Question with Unordered Response Categories!
3/26/13 11
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Closed-ended Question with Unordered Response Categories!
3/26/13 12
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Examples of Closed-ended Questions with Ordered Scale!
3/26/13 13
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
• Choose an appropriate scale length!
• In general, limit scales to four or five categories!
• For bi-polar scales, include a neutral unless it’s imperative to force a choice!
Open- vs. Closed-ended Questions!
• Open-ended questions good when response options not well-known/structured or when want to get at logic/reasoning behind answer!– Use sparingly: Hard to analyze and can put
excess burden on respondent!– Don’t use because you’re too lazy to determine
appropriate response categories!• Closed-ended questions good when response
options well-known and mutually exclusive!– Easier to analyze then open-ended responses!– If done poorly, can frustrate respondents!
3/26/13 14
Partially-closed Questions!
3/26/13 15
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
• Partially-closed questions sometimes a useful hybrid!– But respondent’s
more likely to choose a provided option than write in an “other”!
– Also can be harder to analyze!
Guidelines for Writing Questions!
• Lots of pithy dos and don’ts: use simple words, keep it short, be specific, etc.!– Often they are too simple and/or contradictory!
• That said, here are some of Dillman’s guidelines that he has… !
“…found useful in making the wording and structural changes necessary for turning an initial draft of respondent queries into acceptable survey questions.”!
3/26/13 16
Use Wording that Specifies What You’re Asking for Clearly!
3/26/13 17
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Make Sure Question is Accurate!
3/26/13 18
Source: Mail and Internet Surveys: The Tailored Design Method, 2nd edition, by Don A. Dillman, 2006.
Ask One Question at a Time!
3/26/13 19
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Avoid Double Negative Phrasing!!
3/26/13 20
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Make Sure Questions Apply to All Respondents!
• All questions should require an answer of all respondents!– Else, use a
screening question and skip!
• This is not the same as saying all respondents must answer all questions!!
3/26/13 21
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
State Both Positive and Negative Sides in Question Stem!
3/26/13 22
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Integrate Instructions into Question Stem!
• Put instructions at the point where respondents need them!– Avoid putting them in a
separate section up front!• For questions, integrate
into the question stem!• For other instructions,
integrate then into the survey where they are relevant!
3/26/13 23
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
3/26/13 24
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Avoid Bias From Unequal Comparisons!
3/26/13 25
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Avoid Bias From Unequal Comparisons (cont’d)!
A Bit About Response Scales!
• There are many possible ordinal scales:!
• In structuring response scales, must balance:!– Analytical needs (precision of information
required)!– Ease of respondent recall and answer!
3/26/13 26
Fowler, F.J., Jr., Improving Survey Questions: Design and Evaluation, SAGE Publications.
The “Likert” Scale is Very Common!
• Wikipedia: “A Likert scale (pronounced 'lick-ert') is a type of psychometric response scale often used in questionnaires, and is the most widely used scale in survey research. When responding to a Likert questionnaire item, respondents specify their level of agreement to a statement.”!
• Five-point scale is frequently used, but it can vary!• Example: How much to you agree or disagree with the
following statement: Ice cream is good for breakfast!q Strongly disagree !q Disagree !q Neither agree nor disagree !q Agree !q Strongly agree !
3/26/13 27
http://en.wikipedia.org/wiki/Likert_scale
Provide Balanced, Evenly-Spaced Response Scales!
3/26/13 28
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Constructing Response Scales!
• Number of points in scale should be determined by how you plan to use the data!
• Respondents can generally only remember a maximum of 5 responses unless visual cues are used!
• With scales with few points, every scale can be labeled!– In longer scales, only label midpoint and endpoints!
• Graphic images such as thermometers and ladders can be effective!
3/26/13 29
Poorly Designed Response Categories Drive Respondents Nuts!
• Guidelines:!– Answer categories
must include all (reasonable) possible answers!
– Answer categories must be mutually exclusive!
3/26/13 30
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Be Sure Question Specifies (and Matches) Response Task!
• Ensure question stem clearly states the response task!1. “Do you favor or oppose expanding the
designated faculty parking at NPS?”!• And make sure it matches the response scale!
1. “Do you favor or oppose expanding the designated faculty parking at NPS?”!q Agree!q Neutral!q Disagree!
3/26/13 31
Huh???!
Use Construct-specific Questions and Responses Whenever Possible!
3/26/13 32
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Another Example!
3/26/13 33
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
And Another Example!
3/26/13 34
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Answer Ordering Matters!
3/26/13 35
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
• Randomize response options if concerned about order effects!
Layout and Spacing Matter!
3/26/13 36
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
• Guidelines:!– Maintain spacing
between answer categories consistent with measurement intent!
– If using “NA” type response, visually separate it from an ordered response scale !
Separate Non-substantive Response Items !
3/26/13 37
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
• Separate “Don’t know” and “No opinion” from response scale items!
• Helps ensure respondents do not visually misinterpret the scale!
Take Care with Verbally Labeling All Response Scale Options!
3/26/13 38
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
• Issue is words influence how respondents interpret scale!
• Consider only labeling the endpoints!– Particularly for
longer scales!
Don’t Forget “Don’t Knows”!
• Lack can frustrate some respondents!– If respondents forced to just skip answering a
question (item nonresponse) can make analysis more difficult!
• Solutions:!– Ask a screening question first!
• Appropriately allows instrument to build in skip patterns for respondents to skip over questions that do not apply!
– Include a “no opinion” or “don’t know” response!– Force respondent to answer the question (tricky to
do well)!3/26/13 39
No Matter How Good the Design, For Some There’s Nothing You Can Do…!
3/26/13 40
What We Have Covered!
• Described basic types of questions!– Open-ended !– Closed-ended!– Partially-closed !
• Guidelines for writing effective questions!– Lots of dos and don’ts!– Mistakes and errors to avoid when writing
questions!• Guidelines for writing effective response
scales!– More dos and don’ts!
3/26/13 41