crafting good questions - facultyfaculty.nps.edu/rdfricke/oa4109/lecture 2-2 -- crafting good...

38
1 Crafting Good Questions Professor Ron Fricker Naval Postgraduate School Monterey, California 6/24/12

Upload: others

Post on 21-Sep-2020

8 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

1

Crafting Good Questions!

Professor Ron Fricker!Naval Postgraduate School!

Monterey, California!

6/24/12

Page 2: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

2

Goals for this Lecture!

•  Guidelines for writing effective questions!–  Lots of dos and don’ts!–  Mistakes and errors to avoid when writing

questions!•  Guidelines for writing effective response

scales!–  More dos and don’ts!

Ø Goal is to advance our understanding of what makes a good survey question and how to write such questions!

6/24/12

Page 3: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Question Components!

6/24/12 3

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 4: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Guidelines for Writing Questions!

•  Lots of pithy dos and don’ts: use simple words, keep it short, be specific, etc.!–  Often they are too simple and/or contradictory!

•  That said, here are some of Dillman’s guidelines that he has… !

“…found useful in making the wording and structural changes necessary for turning an initial draft of respondent queries into acceptable survey questions.”!

6/24/12 4

Page 5: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Use Simple and Familiar Words!

6/24/12 5

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 6: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Example!

6/24/12 6

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 7: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Don’t Be (Needlessly) Wordy!

6/24/12 7

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 8: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Keep the Questions as Short as Possible!

6/24/12 8

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 9: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Use Complete Sentences!

6/24/12 9

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 10: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Use Wording that Specifies What You’re Asking for Clearly!

6/24/12 10

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 11: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Make Sure Question is Accurate!

6/24/12 11

Source: Mail and Internet Surveys: The Tailored Design Method, 2nd edition, by Don A. Dillman, 2006.

Page 12: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Ask One Question at a Time!

6/24/12 12

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 13: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

13

Only Ask One Question at a Time (cont’d)!

6/24/12

Source: Mail and Internet Surveys: The Tailored Design Method, 2nd edition, by Don A. Dillman et al., 2009.

Page 14: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Only Ask One Question at a Time (cont’d)!

6/24/12 14

Source: Mail and Internet Surveys: The Tailored Design Method, 2nd edition, by Don A. Dillman et al., 2009.

Page 15: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Avoid Double Negative Phrasing!!

6/24/12 15

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 16: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Make Sure Questions Apply to All Respondents!

•  All questions should require an answer of all respondents!–  Else, use a

screening question and skip!

•  This is not the same as saying all respondents must answer all questions!!

6/24/12 16

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 17: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

State Both Positive and Negative Sides in Question Stem!

6/24/12 17

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 18: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Integrate Instructions into Question Stem!

•  Put instructions at the point where respondents need them!–  Avoid putting them in a

separate section up front!•  For questions, integrate

into the question stem!•  For other instructions,

integrate then into the survey where they are relevant!

6/24/12 18

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 19: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

6/24/12 19

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Avoid Bias From Unequal Comparisons!

Page 20: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

6/24/12 20

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Avoid Bias From Unequal Comparisons (cont’d)!

Page 21: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

A Bit About Response Scales!

•  There are many possible ordinal scales:!

•  In structuring response scales, must balance:!–  Analytical needs (precision of information

required)!–  Ease of respondent recall and answer!

6/24/12 21

Fowler, F.J., Jr., Improving Survey Questions: Design and Evaluation, SAGE Publications.

Page 22: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

The “Likert” Scale is Very Common!

•  Wikipedia: “A Likert scale (pronounced 'lick-ert') is a type of psychometric response scale often used in questionnaires, and is the most widely used scale in survey research. When responding to a Likert questionnaire item, respondents specify their level of agreement to a statement.”!

•  Five-point scale is frequently used, but it can vary!•  Example: How much to you agree or disagree with the

following statement: Ice cream is good for breakfast!q Strongly disagree !q Disagree !q Neither agree nor disagree !q Agree !q Strongly agree !

6/24/12 22

http://en.wikipedia.org/wiki/Likert_scale

Page 23: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Provide Balanced, Evenly-Spaced Response Scales!

6/24/12 23

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 24: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Constructing Response Scales!

•  Number of points in scale should be determined by how you plan to use the data!

•  Respondents can generally only remember a maximum of 5 responses unless visual cues are used!

•  With scales with few points, every scale can be labeled!–  In longer scales, only label midpoint and endpoints!

•  Graphic images such as thermometers and ladders can be effective!

6/24/12 24

Page 25: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Poorly Designed Response Categories Drive Respondents Nuts!

•  Guidelines:!–  Answer categories

must include all (reasonable) possible answers!

–  Answer categories must be mutually exclusive!

6/24/12 25

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 26: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Be Sure Question Specifies (and Matches) Response Task!

•  Ensure question stem clearly states the response task!1.  “Do you favor or oppose expanding the

designated faculty parking at NPS?”!•  And make sure it matches the response scale!

1.  “Do you favor or oppose expanding the designated faculty parking at NPS?”!q  Agree!q  Neutral!q  Disagree!

6/24/12 26

Huh???!

Page 27: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Use Forced-Choice Vice Check-All-That-Apply Questions!

6/24/12 27

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 28: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Use Forced-Choice Vice Check-All-That-Apply Questions (cont’d)!

6/24/12 28

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 29: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Respondents Tend to Engage More with Forced Choice!

6/24/12 29

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 30: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Use Construct-specific Questions and Responses Whenever Possible!

6/24/12 30

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 31: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Another Example!

6/24/12 31

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 32: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

And Another Example!

6/24/12 32

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

Page 33: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Answer Ordering Matters!

6/24/12 33

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

•  Randomize response options if concerned about order effects!

Page 34: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Layout and Spacing Matter!

6/24/12 34

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

•  Guidelines:!–  Maintain spacing

between answer categories consistent with measurement intent!

–  If using “NA” type response, visually separate it from an ordered response scale !

Page 35: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Separate Non-substantive Response Items !

6/24/12 35

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

•  Separate “Don’t know” and “No opinion” from response scale items!

•  Helps ensure respondents do not visually misinterpret the scale!

Page 36: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Take Care with Verbally Labeling All Response Scale Options!

6/24/12 36

Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

•  Issue is words influence how respondents interpret scale!

•  Consider only labeling the endpoints!–  Particularly for

longer scales!

Page 37: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

Don’t Forget “Don’t Knows”!

•  Lack can frustrate some respondents!–  If respondents forced to just skip answering a

question (item nonresponse) can make analysis more difficult!

•  Solutions:!–  Ask a screening question first!

•  Appropriately allows instrument to build in skip patterns for respondents to skip over questions that do not apply!

–  Include a “no opinion” or “don’t know” response!–  Force respondent to answer the question (tricky to

do well)!6/24/12 37

Page 38: Crafting Good Questions - Facultyfaculty.nps.edu/rdfricke/OA4109/Lecture 2-2 -- Crafting Good Questio… · Crafting Good Questions! Professor Ron Fricker! Naval Postgraduate School!

What We Have Covered!

•  Guidelines for writing effective questions!–  Lots of dos and don’ts!–  Mistakes and errors to avoid when writing

questions!•  Guidelines for writing effective response

scales!–  More dos and don’ts!

Ø Goal is to advance our understanding of what makes a good survey question and how to write such questions!

6/24/12 38