february 11, 2015 survey design and administration catherine r. messina phd
TRANSCRIPT
![Page 1: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/1.jpg)
February 11, 2015
Survey Design and Administration
Catherine R. Messina PhD
![Page 2: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/2.jpg)
“The scientist is not a person who gives the right answers, he's one who asks the right questions.” ― Claude Lévi-Strauss (1908-2009), French anthropologist
![Page 3: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/3.jpg)
No matter how good your study design, your sampling strategy, your response/participation rate, etc …………
The quality of your measurement instruments will directly impact your ability to draw meaningful conclusions about relationships studied
![Page 4: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/4.jpg)
What is a survey or questionnaire?
…….. a systematic method for collecting information from a sample of individuals for the purpose of quantitatively describing the attributes of the larger population of which the sample are members.
Groves, Fowler, Couper, Lepkowski, Singer and Tourangeau, 2004.
![Page 5: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/5.jpg)
Benefits of surveysCan reach a large number of people relatively easilyProvide quantifiable answersRelatively easy to analyse
Drawbacks of surveysProvides only limited insight into problem
Information obtained is limited by questions asked Always possible that wrong questions are asked
Varying response – respondent may misunderstand or misinterpret questionsNeed to get it right first time
Can’t chase after missing data – or go back and try again
![Page 6: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/6.jpg)
Making inferences from survey data
Respondent answers question(s)
You infer….
Characteristic(s) (attribute(s)) of Respondent
Characteristic(s) (attribute(s)) of Sample
You infer….
Characteristic(s) (attribute(s)) of Population
![Page 7: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/7.jpg)
Measurement is a systematic and repeatable process that quantifies or classifies events or characteristics
Measurement tools (i.e., surveys/questionnaires) need to yield measurements that are reproducible (reliable) and accurate (valid)!
Reliability- the property of reproducability does the measure consistently yield a similar
finding? Validity- does the measure really assess (i.e.
measure) the concept/object that it is intended to? How accurate is the measure?
Making inferences from survey data
![Page 8: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/8.jpg)
Measurement is represented by – 3 related concepts: observed value (aka the “measured value”) true value and measurement error
Observed value = true value + measurement error
Well designed measurement tools (i.e., surveys) minimize measurement error so that the observed value is very close to the true value
Making inferences from survey data
![Page 9: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/9.jpg)
Creating a survey or questionnaire First, determine clearly what it is you want to
measure in order to answer your research question(s) or hypotheses Be very clear about what you want to know
Decisions about relevant variables are based on the research question (or hypothesis) Research question / hypotheses should be
stated in measureable terms
![Page 10: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/10.jpg)
Creating a survey or questionnaire List all relevant variables!
Dependent variables Independent variables
Covariates - A secondary variable (and of interest to the researcher) that is observed and which can affect the relationship between the IVs and the DV E.g. age can effect the relationship between
CRC-screening reminders and the likelihood of having a colonoscopy
![Page 11: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/11.jpg)
Identifying Existing Measures
Next, determine if questions that you want to include have already been developed (and tested) by other researchers
Review the literature!!!!… you do not have to reinvent the wheel!
Saves development time and can facilitate comparisons with other studies
![Page 12: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/12.jpg)
Identifying Existing Measures
Use standardized measures as available Review the literature for standardized measures Look for validity and reliability information,
appropriate to your sample Usually not a good idea to “adapt” or change – but if
you do, describe this is your Methods section and note this as a potential limitation in Discussion section
Be cautious if using only a subset of questions from an existing survey as this may change the meaning of the summary scores. Check whether a shorter version of the instrument exists that has also been validated
Re: primary research articles – if questions not included look for on-line supplemental info
![Page 13: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/13.jpg)
What if there are no existing measures on my topic, and I need to develop my own survey or questionnaire?
![Page 14: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/14.jpg)
Developing Surveys – General Approach Determine the mode of survey administration:
self-administered or interviewer adminsitered Each has its own advantages and
disadvantages …… and will determine your survey format
![Page 15: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/15.jpg)
Types of surveys
Self-administered•Hand out•Mailed•Email / web-based
Interviewer administered•Face to face•telephone
![Page 16: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/16.jpg)
Self-administered surveys Advantages:
Relatively inexpensive and easy to administer Preserves confidentiality (can be anonymous) Can be completed at respondent's convenience No influence by interviewer
Disadvantages: Low response rate – email surveys sift to bottem of
inbox very quickly Questions can be misunderstood No control by interviewer
![Page 17: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/17.jpg)
Interviewer-administered surveys Advantages:
Facilitates participation by people with low literacy Interviewer can clarify ambiguity
Disadvantages: May introduce interviewer bias Needs more resources – more expensive Short surveys - especially on telephone Difficult for sensitive issues
![Page 18: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/18.jpg)
Developing Surveys – General Approach Create a pool of new questions or revise existing
questions – write more questions than will be included in the final draft
Put questions in sequence and format survey draft This includes all instructions for filling out survey
and possible skip patterns! Include appropriate white space to get a good
estimate of the length
![Page 19: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/19.jpg)
Developing Surveys – General Approach Decided on operational definitions
How the researcher chooses to measure a particular variable E.g.: age – can be operationally defined as”
DOBYearsAge group
Use measure that offers most flexibility
![Page 20: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/20.jpg)
Developing Surveys
If possible, ask experts to review for content Content validity: do the items in a questionnaire
adequately represent the universe of items relating to a specific construct? How accurately does a measurement instrument tap into the various aspects of a specific construct?
![Page 21: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/21.jpg)
Developing Surveys Get feedback (e.g., advisor; peers) on your 1st draft Face validity: is the measurement logical on the
“face of it”? The degree to which the purpose of a measurement instrument is obvious to those using it or responding to it, i.e., “looks like” or appears to respondents as a measure of depression. Advantage of good face validity: respondent really
understands the point of the questions and interprets them accurately
Disadvantage of good face validity: respondent really understands the point of the questions and “tailors” their responses to “fake good” or “fake bad”
![Page 22: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/22.jpg)
Developing Surveys Get feedback (e.g., advisor; peers) on your 1st draft –
con’t Which questions measure IV(s)? Which questions measure DV(s)? Which questions measure covariate(s)? Are questions clear and “answerable”? Are any questions potentially offensive? Does format make sense? Is it logical? Length of survey?
REVISE!! Get more feedback (e.g., advisor; peers) on your
revised draft
![Page 23: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/23.jpg)
Developing Surveys Pilot-test on a small sample of respondents who are
similar to your study sample Are questions clear and “answerable”? Are any questions potentially offensive? Questions should be
culturally sensistive!! Does format make sense? Is it logical? Length of survey and how long to complete? (consider
participant burden) Cognitive interviewing and “think aloud”
Ask a few people in pilot sample to read the questions out loud to you and then explain to you, what each question means to him/her and how he/she thought about their answer
Test-drive your coding system – prelim analyses REVISE!
![Page 24: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/24.jpg)
Developing Questions If possible, pilot-test the near final draft of the survey
with new pilot sample
Pilot-test all procedures and logistics for administering survey Self or interviewer administered? Who will administer? Who is responsible for collecting completed
surveys? Where will these be stored before pick-up?
![Page 25: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/25.jpg)
Developing Surveys – Specifics
![Page 26: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/26.jpg)
Your goal is to write survey questions that •Respondents will interpret the same way (that you interpret them) •Respondents are able to respond accurately •Respondents are willing to answer
![Page 27: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/27.jpg)
Only include questions you plan to use Too many questions (especially if they seem
unrelated to the purpose of your survey) can irritate respondents (an IRB issue too)
Avoid lengthy or complex questions – keep it simple Write at a low reading level – understand the
literacy level of your respondents Avoid jargon and acronyms
Use words relevant to respondents - terms and concepts should be familiar to respondents and easy for them to understand
Writing good survey questions
![Page 28: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/28.jpg)
Instructions need to be placed exactly where needed - not at the beginning of the entire survey.
Use italics or bolding or underline to emphasize instructions and directions
Writing good survey questions
![Page 29: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/29.jpg)
Use standard demographic questions Facilitates comparing your sample to others in the
literature Examples from national health surveys Look at the way these are asked in similar studies Only ask for necessary demo information
Race / ethnicity – comprised of two questions (NIH format):1. Are you Hispanic or Latino? (yes vs. no)2. What is your race? (allow for multiple answers)
Writing good survey questions
![Page 30: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/30.jpg)
Avoid “and” “or” (double-barreled) questions Trying to respond to a question measuring two or
more ideas is confusing to respondent and their response is difficult for the researcher to interpret E.g., Do you think that increasing physical activity and
(or) losing weight is beneficial to you child’s health? E.g., Do you think that eye contact and (or) using words
you understand improve communication between you and your child’s doctor
E.g., Were you satisfied with the timeliness and quality of the health care service your child received?
Writing good survey questions
![Page 31: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/31.jpg)
Avoid leading questions
Do you think that the food in the hospital made your child sick?
Do you agree that the hospital staff were overworked?
Writing good survey questions
![Page 32: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/32.jpg)
Avoid bias by using both positive and negative sides in the question stem
“To what extent do you agree….”
vs.
“To what extent do you agree or disagree with this statement…. “
Writing good survey questions
![Page 33: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/33.jpg)
Avoid vague quantifiers
How often did you use aspirin for pain relief in the last month?
Never
Rarely
Occasionally
Regularly
Vs……..
Writing good survey questions
![Page 34: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/34.jpg)
How often did you use aspirin for pain relief in the last month?
Not at all About once in the last month Two or three times in the last month About once a week More than once a week
Writing good survey questions
![Page 35: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/35.jpg)
Ask questions that people are willing to answer! Do not be too personal or risky unless absolutely
needed Be culturally sensitive
People are more likely to answer questions about behaviors and health compared to income or education Using income categories rather than asking for
exact $ amounts is a better strategy
Writing good survey questions
![Page 36: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/36.jpg)
Ordering questions Two key considerations:
Questions should be ordered such that the response to one question does not influence the response to the question that follows it
Questions should be ordered such that respondents are more likely to complete the survey Fact-based questions before opinion based questions Start with non-threatening or least sensitive questions Most important questions should be placed first Vary question format to reduce likelihood that
respondents will “auto-pilot” response sets
Writing good survey questions
![Page 37: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/37.jpg)
Good questions follow comfortably from the previous question. Requires good writing skills Smooth transitions between questions or groups
of questions Group similar questions together
Avoid non-response (i.e., missing data) by including response options for “Don’t Know” and “Refused” or “Prefer not to answer”
Writing good survey questions
![Page 38: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/38.jpg)
Recall of past events – people have limited recall of past events (this will vary with type of event)
Improve recall by providing reasonable prompts Past year Past 6 months Past 3 months Past 2 weeks
Use addition prompts, if needed: “What season was it when your child was last
vaccinated?” “Was it near a holiday or someone’s birthday?”
Writing good survey questions
![Page 39: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/39.jpg)
FORMAT OF QUESTIONS
Two main question formats Closed format forced choice
Yes Always No Sometimes Don’t know Never
Open format free text
Please describe your child’s most distressing symptom? ________________________________________________________________________________________________
![Page 40: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/40.jpg)
Close-Ended Response Formats
the Likert-type response format “Indicate the strength of your agreement for each of the following statements, on a scale from 1 to 5, where a response of 1 indicates that you strongly disagree, a response of 5 indicates that you strongly agree, and 3 indicates that you are uncertain.” Odd number of categories is preferred – captures
respondents who are indifferent or undecided Reduces non-response No more than 5 response options
![Page 41: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/41.jpg)
Close-Ended Response Formats Semantic Differential Scale - used to assess the
meaning of a variable to the respondent uses a pair of discrete descriptor words, which name
opposing positions, as anchors for the response scale Everything in between represents the continuum of
choices between the 2 anchors
Very satisfied __I__I__I__I__I__I__I__I__ Very dissatisfied
Very confident __I__I__I__I__I__I__I__I__ Not at all confident
Strongly agree __I__I__I__I__I__I__I__I__ Strongly disagree
NOTE: odd number of spaces
![Page 42: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/42.jpg)
Advantages: Simple and quick Easy to code, record, analyse Easy to compare Easy to report results
Disadvantages: Restricted number of possible answers Loss of information
Close-Ended Response Formats
![Page 43: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/43.jpg)
Open – ended response questions Advantages
Rich and detailed response – respondents are not constrained by researchers choices
Unexpected responses possible Good for exploring knowledge and attitudes
Disadvantages Cognitively demanding for respondent and
researcher (coding is challenging and time intensive) Interviewer bias Difficult to analyse Difficult to compare groups
![Page 44: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/44.jpg)
Asking questions about: Knowledge Attitudes and beliefs Behavior
Developing Surveys
![Page 45: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/45.jpg)
Measuring Knowledge True/False and multiple choice “fact” questions Open-ended response formats
E.g., list all of the problems related to X that come to mind
Close-ended response formats (response options are provided by the survey) Quicker, less burden compared to open-ended Less detail but easier to code and analyze
If you plan to intervene on knowledge, include questions at pre and post-test that measure knowledge specifically addressed by intervention
![Page 46: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/46.jpg)
Measuring Attitudes and Beliefs
Close-ended response formats are very common when measuring attitudes / beliefs
Attitude questions can be stand-alone items but … Responses often combined (i.e., summed) as a
“Scale” which means that many items are used to assess a single attitude or belief
often using the same response format… e.g. strongly agree …… strongly disagree
![Page 47: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/47.jpg)
Methods to measure behavior Self report: “I did not smoke any cigarettes today.”
By phone, mail, computer, interview Observation - Report of respondent’s behavior by
others (family member, health professional) “I saw him/her smoke cigarettes today.”
![Page 48: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/48.jpg)
Survey design can go a long way in reducing non-response!
Consider using surveys that you think are attractive as a template Professional looking format - good use of “white space” Space response scales widely enough so that it is easy to circle or
check the correct answer without the mark accidentally including the answer above or below. Open-ended questions: the space for the response should be big enough to
allow respondents with large handwriting to write comfortably in the space. Closed-ended questions: line up answers vertically and precede them with
boxes or brackets to check, or by numbers to circle, rather than open blanks. Use larger font size (e.g., 14) and high contrast (black on white).
Proof read!! No typos! Get help if this is not your thing.
![Page 49: February 11, 2015 Survey Design and Administration Catherine R. Messina PhD](https://reader034.vdocuments.us/reader034/viewer/2022052701/56649dde5503460f94ad6887/html5/thumbnails/49.jpg)
Always conclude your survey with THANK YOU!