introduction to surveying (cont’d)faculty.nps.edu/rdfricke/oa4109/lecture 1-3... · how surveying...
TRANSCRIPT
1
Introduction to Surveying (cont’d)!
Professor Ron Fricker!Naval Postgraduate School!
Monterey, California!
12/31/12
Goals for this Lecture!
• A bit more about surveys and data!
• Discuss the various survey modes!– Pros and cons!– Trade-offs!
• Considerations in making mode selection!
12/31/12 2
How Surveying Has Changed!
3
Source: Internet, Mail and Mixed-mode Surveys: The Tailored Design Method, 3rd ed., Dillman, et al, 2009.
12/31/12
The Goal of Surveys!
• Collect information about a population!• Such information should be!
– Accurate and!– Generalizable!!as well as!
– Relevant and useful (to answer some question)!– Unavailable by other means!
4 12/31/12
For the Results to be “Accurate” and “Generalizable”…!
12/31/12 5
1. Answers people give must accurately describe their characteristics
2. The survey respondents must have characteristics similar to those of the larger population.
Source: Survey Methodology, 1st ed., Groves, et al, 2004.
From Questions to Characteristics !
6
27. Please tell me whether you would be willing to accept someone from a different ethnic group as…!
29. Please tell me whether you would be willing to accept someone from a different religious group as…!
a. A coworker!b. A neighbor!c. A close friend!d. A relative by marriage!
Survey Questions
Classify an individual as tolerant or not!
a. A coworker!b. A neighbor!c. A close friend!d. A relative by marriage!
12/31/12
7
From Sample to Population!
Population
Unobserved population statistic
Sample statistic
inference
sample
12/31/12
Which Survey Mode Should I Use?!
• You are going to conduct a research survey!– The goal is to make the best inference possible
from sample to target population within the constraints of a budget!
• To important questions:!– What is the most appropriate method to choose for
a particular research question?!– What is the impact of choosing a particular method
on the survey errors and costs?!
12/31/12 8
Traditional Survey Modes!
• Mail!– Paper questionnaire sent to respondents!– Self-administered and mailed back!
• Telephone!– Interviewers call respondents on telephone!– Interviewer-assisted!
• In-person!– Interviewers go to respondent’s home or office!– Interviewer-assisted!
12/31/12 9
Options Have Multiplied in the Age of Computers and Telecom !
CATI! TDE! IVR/T-ACASI!
CAPI!
Text- CASI!
Audio-CASI!
Video-CASI!
CAI
Methods!
12/31/12 10
Source: Survey Methodology, 1st ed., Groves, et al, 2004.
Mode Considerations!
• Modes differ by:!– Degree of interviewer involvement!– Degree of interaction with respondent!– Degree of respondent privacy!– Channels of communication!– Technology usage!
• Considerations:!– Cost!– Timeliness!– Accuracy!
12/31/12 11
Broadly speaking…!
• Interviewer-assisted surveys more costly than self-administered!– Face-to-face most expensive, then telephone, then mail,
then web!
• Interviewer-assisted surveys less prone to various errors and item nonresponse!– Particularly compared to self-administered paper survey!– More or less compared to self-administered computer-
assisted surveys depending on sophistication of program!
• Self-administered less subject to sensitivity bias!– Can combine some self-administered questions into a
predominantly interviewer-assisted mode!12/31/12 12
However, timeliness is not just about…!
• …the length of time a survey is in the field!– Really, it’s from instrument design through
analysis!• Much of that is not affected by survey mode!
– Compared to a mail survey, web-based surveys save on two main things:!
• Mailing time to send out and return surveys!• Time required to do data entry, coding, and
cleaning for paper surveys !• Yet, it can also take a lot of time to code, test,
and de-bug very complicated web (more generally, computer-assisted) survey programs!
12/31/12 13
And, cost is not just about…!
• …manhours!– On mail surveys, printing & postage costs can be significant!
• 1,500 surveys x 4 mailings at $1.50 each = $9,000!!• 1,500 $1 incentives = $1,500!
– On telephone surveys must consider fixed costs of phone and CATI equipment + variable costs!
– On face-to-face interviews, travel costs can be significant (not to mention interviewer time)!
• And in terms of manhours, don’t forget: !– instrument design time; !– in non-electronic modes, data entry and coding time; !– non-response follow-up efforts; and !– analysis time!
12/31/12 14
And accuracy is not just about…!
• … response rates and missing items!– Remember total
survey error – what mode or modes will decrease best all types of error!
• Lots of considerations!– Frame coverage!– Nonresponse rates!– Sensitivity bias, etc…!
12/31/12 15
Source: Survey Methodology, 1st ed., Groves, et al, 2004.
E.g., With Sensitive Questions, Mode Matters – As Do Other Factors!
12/31/12 16 Source: Survey Methodology, 1st ed., Groves, et al, 2004.
Cost and Frame Availability Often Drive Mode Choice!
• Survey budgets never unlimited!– Frequently cost will eliminate one or more modes!
• And likely drive other survey design decisions!– I.e., face-to-face interviews too expensive!
• On the other hand, surveys using area probability frames usually have to start out using face-to-face interviews!
• Similarly, if a complete e-mail frame is available, web survey is possible option!– But general population surveys eliminate web…!
12/31/12 17
Some Modes Logically Group (1)!
• Telephone and face-to-face surveys!– Telephone often considered an alternative to face-
to-face!– Both interviewer-assisted and can have similar
frame coverage issues!– Many longitudinal surveys start out with a face-to-
face interview with telephone follow-ups!• Mail sometimes an alternative to telephone!
– If both telephone and addresses available!
12/31/12 18
Some Modes Logically Group (2)!
• Web surveys often considered a replacement for mail surveys!– Coverage is still an issue for web surveys!– Web superior in terms of automation and speed!– However, literature consistently finds that
nonresponse rates are higher for web surveys!
12/31/12 19
Using Multiple Modes (1) !
• Reasons to use multiple modes:!– Maximize response rates!
• Reduce/eliminate selection & coverage bias!• Allow respondents to use most convenient
mode!
12/31/12 * Source: Fricker, R.D., Jr., Sampling Methods for Web and E-mail Surveys, The SAGE Handbook of Online Research Methods, N.
Fielding, R.M. Lee and G. Blank, eds., chapter 11, London: SAGE Publications, pp. 195-216, 2008.
*
20
Using Multiple Modes (2)!
• Reasons to use multiple modes:!– Reduce costs!
• Cross-sectional surveys: start with least costly mode and move to more expensive modes as necessary to get response!
• Longitudinal surveys: Use less expensive modes in follow-on survey rounds!
12/31/12 21
Considerations When Using Multiple Modes!
• Focus should be on designing instruments to ensure equivalency across modes!– Rather than optimizing for a particular mode!– E.g., make the follow-on web survey resemble its
counterpart paper survey!
• Ensure operation is set up to track cases across the modes!– Must avoid duplications and uncoordinated respondent
contact!
• Design so mode effects can be disentangled from other sample characteristics!– May have to randomize survey mode over a subset!
12/31/12 22
What We Have Covered!
• There are many considerations when choosing a survey mode (or modes)!
• In various ways, choice likely to affect survey!– Timeliness!– Accuracy!– Cost!
• And cost may drive mode choice possibilities!
12/31/12 23