introduction to questionnaire design principles and guidelines to writing questions improving...
TRANSCRIPT
![Page 1: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/1.jpg)
Introduction to Questionnaire Design
• Principles and Guidelines to Writing Questions
• Improving Response Rates
• Designing the Questionnaire’s Appearance
• Ways to Test Survey Instruments
![Page 2: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/2.jpg)
Questionnaire Modes
• Face-to-face interview
• Telephone survey
• Self-administered survey (i.e. mail survey)
![Page 3: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/3.jpg)
The information obtained by each question will be specific to the information you will need in your analysis. Therefore, before you compose any questions:
•Think through your research questions and objectives
•Think about how you will conduct your analysis
![Page 4: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/4.jpg)
I. Selection Categories
• Scales are always relative to respondent’s experience• Scale should allow for maximum variability • Use a balanced scale • Be careful about responses of ‘neutral’ or ‘no opinion’ versus ‘don’t know’ • Use item-in-a-series response categories carefully• Place response categories vertically
Principles of Writing Questions
![Page 5: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/5.jpg)
Principles of Writing Questions...
II. Questions should ask for only 1 piece of information, so avoid:
• Asking two questions at once• Asking questions that contain
assumptions• Asking questions that have
hidden contingencies
![Page 6: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/6.jpg)
Principles of Writing Questions…
III. A questionnaire gets people to express their feelings, perceptions, behaviors, and experiences, both past & present; therefore, avoid:
• Asking for secondhand information
• Asking hypothetical questions• Asking about causation• Asking for solutions to complex
issues
![Page 7: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/7.jpg)
Principles of Writing Questions…
IV. Question wording should ensure that every respondent will be answering the same thing, so avoid:
• Ambiguous wording or wording that means different things to different respondents
• Using terms for which the definition can vary. (If it is unavoidable, provide the respondent with a definition.)
• Being ambiguous about the time period the respondent should consider
• Asking complex questions (double-barreled)
![Page 8: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/8.jpg)
Do:
• Use simple wording• Be brief• Be specific
![Page 9: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/9.jpg)
Do not:
• Be vague• Be condescending or talk down to
respondent• Use biased wording• Use abbreviations or scientific jargon• Use objectionable questions• Be redundant
![Page 10: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/10.jpg)
One of the most important aspects of designing a questionnaire is improving your Response Rate, which requires providing the respondent with the motivation to complete the questionnaire. So consider:
1. Increasing perceived
rewards
2. Decreasing perceived costs
3. Establishing trust
![Page 11: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/11.jpg)
Ways to increase perceived reward
• Show positive regard• Say thank you• Ask them for “advice”• Give social validation• Give a tangible reward• Make the questionnaire interesting• State an upcoming deadline
![Page 12: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/12.jpg)
Ways to decrease perceived cost
• Avoid subordinating language• Do not embarrass the respondent• Do not inconvenience the respondent• Make questions appear short and easy• Avoid asking personal information (if it
must be asked, use ‘soft’ wording and inform about confidentiality)
• Keep subsequent requests similar
![Page 13: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/13.jpg)
Ways to establish trust
• Provide a token of appreciation• Provide a sense of legitimate
authority• Make completing the questionnaire
seem important• Remind respondent of previous
relationship with sponsor (if applicable)
![Page 14: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/14.jpg)
caveats…
• Do not provide an opportunity for the respondent to get a reward from not completing the survey
• Too much repetition of appeals diminishes their effectiveness
• Questions should not be too short – reduces credibility
• Do not send letters too far in advance
![Page 15: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/15.jpg)
Questionnaire Layout
• Graphics and Words Must Work in Concert, With Respect to
• Information Organization• Navigational Guides
![Page 16: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/16.jpg)
Question order
• Questions should be ordered so as to seem logical to the respondent
• First questions should be relevant and easy
• Questions are effectively ordered from most salient to least salient
• Demographic questions should not be covered at the beginning
• Potentially objectionable questions are placed near the end
![Page 17: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/17.jpg)
Question order
• Be sure that the order of the questions does not produce bias in responses
“Norm of even-handedness”
“Anchoring”“Addition (carryover) effect”“Subtraction effect”
![Page 18: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/18.jpg)
Create a navigational path
• Instructions should be placed right where they are needed
• Matrices are confusing: it is best to order questions from top to bottom, and left to right
• Use larger font to attract attention
• Use color shading to attract attention or show groupings
• Use spacing and similarities to show groupings
![Page 19: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/19.jpg)
Create a navigational path…
• Use symbols to identify the starting point
• Number questions consecutively• Use a consistent background• Write questions in bold and answer
responses in a lighter font• Avoid banking responses into more
than 1 column
![Page 20: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/20.jpg)
Questionnaire design problems
• Some possible threats to accuracy:– Questions not understood as intended– Don’t adequately capture respondent’s
experience– Pose a challenging response task
• Problems may not be visible in the actual survey data
• How can we find these before data collection?
![Page 21: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/21.jpg)
Testing the Survey Instruments
1. Focus groups discussions2. Cognitive interviews3. Field pre-testing4. Behavior coding
![Page 22: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/22.jpg)
Focus group discussions
Purposes: • To examine assumptions about the
reality about which people will be asked• To evaluate assumptions about
vocabulary and the way people understand terms and concepts
![Page 23: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/23.jpg)
Focus group discussions
Participants • Discussion leader and note-taker• Five to eight participants• Similar to the target population• Homogeneous vs. mixed groups
![Page 24: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/24.jpg)
Focus group discussions
Guiding questions: • What is covered in the question? • Can you answer the question? • Do you understand the terms?
![Page 25: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/25.jpg)
Cognitive Interviews
Purposes: • To discover what questions pose
particular difficulty in the interpretation, recalling and articulation of the information
• To create cognitive maps
![Page 26: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/26.jpg)
Cognitive Interviews
Cognitive processes:• Comprehension and interpretation• Memory/recalling• Reasoning and judgment • Response formatting/selection
![Page 27: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/27.jpg)
Cognitive Interviews
• One-on-one interview• Trained interviewer --investigators, not
just data collectors
![Page 28: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/28.jpg)
Cognitive Interviews
Participants:• represent a wide range of people
– Paid for their effort– Some demographic variety – Participants with average (or below)
knowledge and experience
![Page 29: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/29.jpg)
How many interviews should be conducted?
• 3 to 12 interviews will suffice
• Remember there are diminishing returns– Intensive labor effort– Rich data, time consuming to analyze
![Page 30: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/30.jpg)
Cognitive Interviews
Procedures:• “Think-aloud” interviews • Interviewer probing
![Page 31: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/31.jpg)
“Think-aloud” interviews
Subjects articulate their thoughts as they respond to the questions.
Advantages:
• Minimal interviewer bias
• Unanticipated answers
![Page 32: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/32.jpg)
“Think-aloud” interviews
Disadvantages:• Poor economy of time• Unnaturalness and Reactivity• Excessive burden on subject• Burdensome coding
![Page 33: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/33.jpg)
Interviewer Probing
• Concurrent Probing – Pros: Question is very fresh on the mind– Cons: Potential bias; switching of tasks can
be distracting. Early probes can affect answers to subsequent answers
• Retrospective Probing– Pros: Avoids bias and task-switching– Cons: Long gap between question and
probe
![Page 34: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/34.jpg)
Interviewer Probing
• Scripted Probing– Pros: interview is focused around objectives– Cons: might be too rigid
• Spontaneous Probing– Pros: allows more flexibility– Cons: no coordination of probing across
interviewers
![Page 35: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/35.jpg)
Cognitive processes
• Comprehension and interpretation• Memory/recalling• Reasoning and judgment • Response formatting/selection
![Page 36: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/36.jpg)
Interviewer Probing
• Comprehension and interpretation: Ask subject to define terms
and paraphrase questions
• Memory processes: Ask subject to talk about the
recalling procedures used when retrieving information from their memory
![Page 37: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/37.jpg)
Interviewer Probing
• Reasoning and judgment: Ask about the certainty and confusions that subjects had when answering questions
• Response formatting/selection: Ask for concerns about sensitivity and social desirability
![Page 38: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/38.jpg)
Interviewer Probing
Advantages:• More effective when research issues are
well-defined• Less burden on respondents
Disadvantages:
• Interviewer introduces bias• Question order effect
![Page 39: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/39.jpg)
Field Pre-test
• Small-scale study in which all the conditions of the full scale-survey are simulated
• Survey modes• Interviewer oral debriefing and written
reports
![Page 40: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/40.jpg)
Field Pre-test
Warning signs:• Variation (Skewed distributions)• Response rate • “No opinion” and “Don’t know” rates• Response Patterns• Flow of the questionnaire
![Page 41: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/41.jpg)
Behavior Coding
Observation of interviewer-respondent interactions
• Does interviewer ask the question as worded?
• Does respondent ask for clarification?• How long does it take for respondent to
answer each question?
![Page 42: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/42.jpg)
Behavior Coding
• Quantifiable data• Very reliable technique• Detects problems in questions but does
not suggest ways to solve them
![Page 43: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/43.jpg)
Summary
Techniques for testing the survey instruments
• Focus groups discussions• Cognitive interviews• Field pre-testing• Behavior coding
![Page 44: Introduction to Questionnaire Design Principles and Guidelines to Writing Questions Improving Response Rates Designing the Questionnaire’s Appearance Ways](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cca5503460f94992af3/html5/thumbnails/44.jpg)
References
Bourque, Linda and Eve Fielder. 1995. How to Conduct Self-Administered and Mail Surveys: Learning Objectives. Thousand Oaks, CA: Sage.
Converse, Jean M. and Stanley Presser. 1986. “Survey Questions: Handcrafting the Standardized Questionnaire.” QuantitativeApplications in the Social Sciences (series). Thousand Oaks, CA:Sage.
Dillman, Don A. 2000. Mail and Internet Surveys : The Tailored Design Method. New York: J. Wiley.
Fink, Arlene. 1995. How To Ask Survey Questions. Thousand Oaks, CA: Sage.
Fowler, Floyd J. Jr. 1995. Improving Survey Questions: Design andEvaluation. Thousand Oaks, CA: Sage.
Sudman, Seymore and Norman M. Bradburn. 1982. Asking Questions: A Practical Guide to Questionnaire Design.San Francisco: Jossey-Bass Inc.