crafting good questions - facultyfaculty.nps.edu/rdfricke/oa4109/lecture 2-2 -- crafting good...
Post on 21-Sep-2020
8 Views
Preview:
TRANSCRIPT
1
Crafting Good Questions!
Professor Ron Fricker!Naval Postgraduate School!
Monterey, California!
6/24/12
2
Goals for this Lecture!
• Guidelines for writing effective questions!– Lots of dos and don’ts!– Mistakes and errors to avoid when writing
questions!• Guidelines for writing effective response
scales!– More dos and don’ts!
Ø Goal is to advance our understanding of what makes a good survey question and how to write such questions!
6/24/12
Question Components!
6/24/12 3
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Guidelines for Writing Questions!
• Lots of pithy dos and don’ts: use simple words, keep it short, be specific, etc.!– Often they are too simple and/or contradictory!
• That said, here are some of Dillman’s guidelines that he has… !
“…found useful in making the wording and structural changes necessary for turning an initial draft of respondent queries into acceptable survey questions.”!
6/24/12 4
Use Simple and Familiar Words!
6/24/12 5
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Example!
6/24/12 6
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Don’t Be (Needlessly) Wordy!
6/24/12 7
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Keep the Questions as Short as Possible!
6/24/12 8
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Use Complete Sentences!
6/24/12 9
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Use Wording that Specifies What You’re Asking for Clearly!
6/24/12 10
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Make Sure Question is Accurate!
6/24/12 11
Source: Mail and Internet Surveys: The Tailored Design Method, 2nd edition, by Don A. Dillman, 2006.
Ask One Question at a Time!
6/24/12 12
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
13
Only Ask One Question at a Time (cont’d)!
6/24/12
Source: Mail and Internet Surveys: The Tailored Design Method, 2nd edition, by Don A. Dillman et al., 2009.
Only Ask One Question at a Time (cont’d)!
6/24/12 14
Source: Mail and Internet Surveys: The Tailored Design Method, 2nd edition, by Don A. Dillman et al., 2009.
Avoid Double Negative Phrasing!!
6/24/12 15
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Make Sure Questions Apply to All Respondents!
• All questions should require an answer of all respondents!– Else, use a
screening question and skip!
• This is not the same as saying all respondents must answer all questions!!
6/24/12 16
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
State Both Positive and Negative Sides in Question Stem!
6/24/12 17
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Integrate Instructions into Question Stem!
• Put instructions at the point where respondents need them!– Avoid putting them in a
separate section up front!• For questions, integrate
into the question stem!• For other instructions,
integrate then into the survey where they are relevant!
6/24/12 18
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
6/24/12 19
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Avoid Bias From Unequal Comparisons!
6/24/12 20
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Avoid Bias From Unequal Comparisons (cont’d)!
A Bit About Response Scales!
• There are many possible ordinal scales:!
• In structuring response scales, must balance:!– Analytical needs (precision of information
required)!– Ease of respondent recall and answer!
6/24/12 21
Fowler, F.J., Jr., Improving Survey Questions: Design and Evaluation, SAGE Publications.
The “Likert” Scale is Very Common!
• Wikipedia: “A Likert scale (pronounced 'lick-ert') is a type of psychometric response scale often used in questionnaires, and is the most widely used scale in survey research. When responding to a Likert questionnaire item, respondents specify their level of agreement to a statement.”!
• Five-point scale is frequently used, but it can vary!• Example: How much to you agree or disagree with the
following statement: Ice cream is good for breakfast!q Strongly disagree !q Disagree !q Neither agree nor disagree !q Agree !q Strongly agree !
6/24/12 22
http://en.wikipedia.org/wiki/Likert_scale
Provide Balanced, Evenly-Spaced Response Scales!
6/24/12 23
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Constructing Response Scales!
• Number of points in scale should be determined by how you plan to use the data!
• Respondents can generally only remember a maximum of 5 responses unless visual cues are used!
• With scales with few points, every scale can be labeled!– In longer scales, only label midpoint and endpoints!
• Graphic images such as thermometers and ladders can be effective!
6/24/12 24
Poorly Designed Response Categories Drive Respondents Nuts!
• Guidelines:!– Answer categories
must include all (reasonable) possible answers!
– Answer categories must be mutually exclusive!
6/24/12 25
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Be Sure Question Specifies (and Matches) Response Task!
• Ensure question stem clearly states the response task!1. “Do you favor or oppose expanding the
designated faculty parking at NPS?”!• And make sure it matches the response scale!
1. “Do you favor or oppose expanding the designated faculty parking at NPS?”!q Agree!q Neutral!q Disagree!
6/24/12 26
Huh???!
Use Forced-Choice Vice Check-All-That-Apply Questions!
6/24/12 27
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Use Forced-Choice Vice Check-All-That-Apply Questions (cont’d)!
6/24/12 28
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Respondents Tend to Engage More with Forced Choice!
6/24/12 29
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Use Construct-specific Questions and Responses Whenever Possible!
6/24/12 30
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Another Example!
6/24/12 31
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
And Another Example!
6/24/12 32
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Answer Ordering Matters!
6/24/12 33
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
• Randomize response options if concerned about order effects!
Layout and Spacing Matter!
6/24/12 34
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
• Guidelines:!– Maintain spacing
between answer categories consistent with measurement intent!
– If using “NA” type response, visually separate it from an ordered response scale !
Separate Non-substantive Response Items !
6/24/12 35
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
• Separate “Don’t know” and “No opinion” from response scale items!
• Helps ensure respondents do not visually misinterpret the scale!
Take Care with Verbally Labeling All Response Scale Options!
6/24/12 36
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
• Issue is words influence how respondents interpret scale!
• Consider only labeling the endpoints!– Particularly for
longer scales!
Don’t Forget “Don’t Knows”!
• Lack can frustrate some respondents!– If respondents forced to just skip answering a
question (item nonresponse) can make analysis more difficult!
• Solutions:!– Ask a screening question first!
• Appropriately allows instrument to build in skip patterns for respondents to skip over questions that do not apply!
– Include a “no opinion” or “don’t know” response!– Force respondent to answer the question (tricky to
do well)!6/24/12 37
What We Have Covered!
• Guidelines for writing effective questions!– Lots of dos and don’ts!– Mistakes and errors to avoid when writing
questions!• Guidelines for writing effective response
scales!– More dos and don’ts!
Ø Goal is to advance our understanding of what makes a good survey question and how to write such questions!
6/24/12 38
top related