overview: online surveys vasja vehovar university of ljubljana, slovenia handbook of online...
Post on 17-Dec-2015
217 Views
Preview:
TRANSCRIPT
Overview: Online Surveys
Vasja VehovarUniversity of Ljubljana, Slovenia
http://WebSM.org
Handbook of Online Research Methods Colloquium 28-29 March 2007
• Computer-assisted survey information collection
• Key methodological issues
• Related issues
• Applications
Structure
Introduction
1930s: applications of probability sampling
1960s: expansion of telephone surveys
1970s: computer technology appears in surveys
1980s: computer-assisted surveying
1990s: Internet mediated surveys
Technology and surveys
Computer-assisted survey information collection
• CATI – Computer-assisted telephone interviewing
• CAPI – Computer-assisted personal interviewing
• CASI – Computer-assisted self-interviewing
Early CASIC interviewer assisted modes
Computer-assisted survey information collection
• Reduced time and costs for data input
• Elimination of errors during data transcription
• Implementation of advanced features:
automatic skips and branching
randomization of questions and response options
control of answers
inclusion of multimedia elements…
CASIC benefits
Computer-assisted survey information collection
Different modes of CSAQ:
disk-by-mail
touch-tone data entry (TDE)
interactive voice response (IVR)
e-mail surveys
web surveys
Computerized self-administered questionnaires (CSAQ)
Computer-assisted survey information collection
Computer-assisted survey information collection
Web surveys
Internet surveys
Online, Internet, Web, CASIC, CSAQ, CADAC
Computer-assisted survey information collection
Web surveys
Internet surveys
CSAQ
Online, Internet, Web, CASIC, CSAQ, CADAC
Computer-assisted survey information collection
Web surveys
Internet surveys
CSAQ
CADAC
Online, Internet, Web, CASIC, CSAQ, CADAC
Interviewer involvement
Survey mode
Paper and pencil CASIC
Interviewer presence
Paper and pencil (face-to-face) interviewing (PAPI)
CAPI, CASI, Audio/Video CASI
Remote interviewer
Paper assisted telephone interviewing (PATI)
CATI, CAVI (computer assisted video interviewing)
No interviewerSelf-administered paper
questionnaires (mail questionnaires)
CSAQ telesurveys (web CSAQ, TDE, IVR, Virtual interviewer,...)
Interviewer-less and paper-less surveys
Computer-assisted survey information collection
Interviewer involvement
Survey mode
Paper and pencil CASIC
Interviewer presence
Paper and pencil (face-to-face) interviewing (PAPI)
CAPI, CASI, Audio/Video CASI
Remote interviewer
Paper assisted telephone interviewing (PATI)
CATI, CAVI (computer assisted video interviewing)
No interviewerSelf-administered paper
questionnaires (mail questionnaires)
CSAQ telesurveys (web CSAQ, TDE, IVR, Virtual interviewer,...)
Interviewer-less and paper-less data collection
Computer-assisted survey information collection
Asking question(INPUT)
Recording of responses (OUTPUT)
Manual recording Automatic voice recognition
Written questions
Standard CSAQ (PC, TV, PDA,
Mobile,...)Visual CSAQ with AVR
Audio questionsVideo CSAQ, Audio
CSAQ, TDE
IVR, Video IVR,TTS with AVR, Virtual
interview
CSAQ input-output technology
Computer-assisted survey information collection
Asking question(INPUT)
Recording of responses (OUTPUT)
Manual recording Automatic voice recognition
Written questions
Standard CSAQ (PC, TV, PDA,
Mobile,...)Visual CSAQ with AVR
Audio questionsVideo CSAQ, Audio
CSAQ, TDE
IVR, Video IVR,TTS with AVR, Virtual
interview
CSAQ input-output technology
Computer-assisted survey information collection
• Developments of the Internet and related technologies
• Importance of broadband Internet access
• Possibilities of distribution across various platforms and devices
• Fast and easy implementation using specialized software tools
Technological aspects
Computer-assisted survey information collection
• Two major problems of probability samples in Internet surveys:
non-coverage
sampling frame problems
• Image of Internet surveys as inherently non-probability ones
1. Probability vs. non-probability sampling
Key methodological issues
Issue of a statistical inference:probability vs. non-probability samples
Internet survey
Information-communication technologies
Mode of survey data collection
Management of the survey process
The spurious link
Key methodological issues
Samplingtype
SolicitationType of invitation
Personal invitation General invitation
Probability samples
Samples from closed population with email,
Pre-recruited with mail, telephone, f2f
Web intercept surveys
Non-probability samples
Self-recruited Internet panels,
Opt-in lists of e-mail adresses
General banner-ad invitation
2. Sampling and invitation
Key methodological issues
• Generally low and variable response rates to web surveys
• Complexity of the response process
• Measures to increase response rates:
incentives (e.g. using PayPal)
multiple follow-up contacts
other motivations
3. Non-response
Key methodological issues
Target population
Operational population
Frame population
Sampled units
Absorbed units
Complete respondents
Partial respondents
Starting units
Clickers
Aware units
Solicitation
Informed units
Involvement process
Key methodological issues
Respondent’s characteristics
Social environment
Researcher-respondent interaction
Survey design
Technological environment
No researcher’s control
Partially under researcher’s control
Participation in web surveys
Key methodological issues
• Question types and visual elements
• Advanced features of computerized questionnaires
• Inclusion of multimedia
• Problems and considerations:
standardization
impact on respondent’s answers
technical problems
measurement error
4. Questionnaire design
Key methodological issues
• Commonly utilized for correcting:
deviations from probability selections
non-coverage and non-response problems
• Importance for non-probability Internet surveys
• Promising approaches:
calibration methods
propensity score weighting
modelling and causal analysis
5. Post-survey adjustments
Key methodological issues
• The managerial process of survey implementation:
costs optimization
data quality
relations between data quality and costs
• Cost effectiveness of web surveys:
costs for increasing response rates
evaluations of costs in relation to errors
1. Costs, errors and management
Related issues
• Web surveys vs. other modes:
lower response rates (Lozar Manfreda et. al., 2007)
comparable or higher level of validity and reliability (e.g. Fricker et al., 2005; Chang & Krosnick, 2002)
2. Survey mode
Related issues
Contact with the respondent
personal, telephone, e-mail, mail,…
Interview
PAPI, CAPI, CATI, CASI,…
Centralized data
management
Solicitation
Mixing survey modes
Related issues
Response rates and behaviour
Modelling mixed modes
Data quality requirements
Willingness to participate
Structure of the data (MAR)
Cost models and impact of environment
Optimal combination
Selecting the right mixture
Related issues
Programing of the questionaire
Related issues
User-friendly interface
Standardized questionnaire
description in QML
Paper questionnaire
CAPICATICASI
Internet survey
• Quantitative and qualitative research methods
• Flexible combinations of approaches (Morgan, 1998):
qualitative preliminary
quantitative preliminary
qualitative follow-up
quantitative follow-up
• New possibilities with the Internet
3. Mixing research methods
Related issues
Qualitative method
Quantitative method
Analysis
Beginning
Conclusion
Flexible on-line mixed methods (FOMM)
Related issues
• New ethical dilemmas in survey research:
solicitation process, spam and privacy
data protection
children and minors
combining data…
• Development of new standards (e.g. ESOMAR, AoIR, MRA…)
4. Ethics, guides and standards
Related issues
• Different applications according to:
type (internet access panels, on-line voting, intercept…)
organizations (official, academic, commercial)
topics (marketing research, polling opinions, health…)
complexity (simple one-shot surveys vs. complex integrated data collection)
Applications
Current and future applications
psychological research
online experiments
paradata and online measurement
specific research fields (usability research, HCI...)
emerging areas (e-learning, e-government…)
Related areas
Current and future applications
• Expansion of CASIC modes, options and mixtures
• Integration of devices (TV, mobile,..)
• Speech recognition and touch-screens
• Multimedia and interactivity
• Integration with other CADAC (GPS, paradata, observations,...)
• Specialisation and professionalisation
Trends
Current and future applications
top related