indeok song joongbu university, korea sung kyum cho chungnam national university
DESCRIPTION
Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone , Interviewer , and Online Surveys. Indeok Song Joongbu University, Korea Sung Kyum Cho Chungnam National University. Contents. Background: 3 Surveys on Gambling Addiction - PowerPoint PPT PresentationTRANSCRIPT
Effects of Survey Mode on Responses:
An Experiment of Social Desirability Biases in Tele-phone, Interviewer, and Online Surveys
Indeok SongJoongbu University, Korea
Sung Kyum ChoChungnam National University
Contents
• Background: 3 Surveys on Gambling Addiction
• Research Question & Purpose
• Review of Previous Research
• Methods
• Results
• Discussions & Suggestions
Results of 2010’s Surveys
Online (n=581) Telephone (n=1,797)
Interview (n=1,000)0
0.51
1.52
2.53
3.54
3.83
0.32 0.5
Means on Gambling Addiction Scale (CPGI 0~27)
F (2, 3,375) = 402.72, p < .001
Results of 2010’s Surveys
Online (n=581)
Telephone (n=1,797)
Interview (n=1,000)
0%10%20%30%40%50%60%70%80%90%
100%
35.6
95.2 87.123.1
1.6 6.8
19.1
1.4 4.422.2
1.8 1.7
ProblemModerate RiskLow RiskNon-Problem
χ²= 1,130.35, df = 6, N=3,378, p < .001
Research Question What is the reason for the inconsistencies
among 3 surveys’ results? Which mode does produce the most accurate estimates?
Sampling? Survey Mode?
Previous Research• Survey modes “can be equivalent but are
not always identical” (Riva et al., 2003, p. 79)
–Sensitivity & Social desirability (Booth-Kewley, Lar-son, & Miyoshi, 2007; Couper, 2000; Huang, 2006; Joinson, 1999; Kays, Gather-coal, & Buhrow, 2011; Tourangeau & Yan, 2007, 2011 and so on)
• Mode of admin.: Self- vs. Interviewer-administered– Self-administration increases reporting of socially unde-
sirable behaviors (Corkrey & Parkinson, 2002; Currivan et al., 2004; Hochstim, 1967; Moskowitz, 2004; Mott, 1985; Richman et al., 1999, and so on)
∴ Paper-Pencil, Online > Telephone, Interviewer
Previous Research• Interviewer or 3rd party presence
– Social desirability bias is worse in telephone survey than in face-to-face interviews (Aquilino, 1994; Groves & Kahn, 1979; Holbrook et al., 2003; Johnson, Hougland, & Clayton, 1989; Leeuw & van der Zouwen, 1988, and so on)
– The presence of others affects responses to sensi-tive questions (Aquilino, Wright, & Supple, 2000; Harrison, 2001; Lombard & Ditton, 1997; Sudman, 2001; Tourangeau & Yan, 201, and so on)
– Online survey yields the lowest level of other peo-ple’s presence ∴ Online > Interviewer > Telephone
Research PurposeControlling for variances from sampling and dif-ferent groups of samples,
to examine the effects of survey modes on the responses to sensitive questions.
Methods• Pretest-posttest 3 between-group experi-
mental design
– 93 undergraduate students 66 analyzed• Out of 66, male 38 (57.6%), female 28 (42.4%),
mean age 20.61 (SD=1.97)
Paper-pencil(N=93) 2 weeks Random
Assignment
Online(n=21)
Telephone(n=24)
Interview(n=21)
Methods• Questionnaire Design
– Addiction Scales• Gambling addiction (4pt., 9 items)
– M=9.24, SD=2.54, α=.84• Internet addiction (4pt., 15 items)
– M=29.13, SD=7.24, α=.90• Game addiction (4pt., 20 items)
– M=26.80, SD=8.84, α=.96
– Dissatisfaction with self-appearance (5pt., 6 items)• M=14.43, SD=3.71, α=.66
– Sexual values (5pt., 10 items)• M=27.88, SD=7.14, α=.89
Methods– Sexual experience
• Age of the first experience– n=27, M=18.59, SD=1.67
• Number of sexual partners– n=26, M=10.81, SD=3.68
• Sexual behaviors (physical contact, cuddle, kiss, caress, intercourse) experienced in current rela-tionship
– n=19, M=3.68, SD=1.42
– Demographics• Gender, age, household-income
Results• Homogeneity among groups (pretest: paper-pen-
cil)Source df F p
Gambling Addiction (2, 63) 1.29 .28
Internet Addiction (2, 63) .09 .91
Game Addiction (2, 63) 2.13 .13
Dissatisfaction with Self (2, 63) .31 .73
Sexual Values (2, 63) .28 .76
Age of 1st Experience (2, 24) 1.89 .19
No. of Partners (2, 23) .59 .56
Sexual Behaviors (2, 16) .24 .79
Responses for Game Addiction
Paper-Pencil Posttest20
22
24
26
28
30
32
28.2430.71
24.5
21.92
28
24.91OnlineTelephoneInterview
Time * Mode: F (2, 63) = 3.35, p = .041, eta² = .10
Responses for Dissatisfaction with Self-Appearance
Paper-Pencil Posttest12
12.5
13
13.5
14
14.5
15
13.95
14.33
14.83
12.29
14.38
13.19OnlineTelephoneInterview
Time * Mode: F (2, 63) = 5.58, p = .006, eta² = .15
2 x 3 Factorial ANOVAs• Independent Variables
– 2 groups (‘Low’ & ‘High’) by median split– 3 modes (Online, Telephone, Interview)
• Dependent Variables– Changes in responses = Pretest – Posttest
DVs N M SDGambling Addiction 66 .27 2.30Internet Addiction 66 1.08 6.78Game Addiction 66 1.14 8.05
Dissatisfaction with Self 66 1.18 3.13Sexual Values 66 1.44 4.65
Age of 1st Experience 22 .68 1.49No. of Partners 22 -3.95 16.76
Sexual Behaviors 16 -.13 1.46
Changes in Responses for Gambling Addiction
Online Telephone Interview
-2-1.5
-1-0.5
00.5
11.5
22.5
3
-0.080 -0.07
1.22
2.5
-1.83
Low GroupHigh Group
Mode: F (2, 60) = 5.32, p = .007, eta² = .15G * M: F (2, 60) = 5.06, p = .009, eta² = .14
Changes in Responses for Internet Addiction
Online Telephone Interview
-5
-3
-1
1
3
5
7
-3 -4.17
0.51.82
5.586.44
Low GroupHigh Group
Group: F (1, 60) = 23.15, p < .001, eta² = .28
Changes in Responses forGame Addiction
Online Telephone Interview
-10-8-6-4-202468
10
-7.56
-0.2 -0.441.33
7.225.75
Low GroupHigh Group
Group: F (1, 60) = 18.54, p < .001, eta² = .24Mode: F (2, 60) = 5.65, p = .006, eta² = .16
Changes in Responses for Dissatis-faction with Self-Appearance
Online Telephone Interview
-2
-1
0
1
2
3
4
-1.75
1.36 1.081.44
3.54
1.33
Low GroupHigh Group
Group: F (1, 60) = 7.41, p = .008, eta² = .11Mode: F (2, 60) = 4.89, p = .011, eta² = .14
Changes in Responses for Sexual Val-ues
Online Telephone Interview-101234567
0 0.29 0
1.6
5.8
1.7Low GroupHigh Group
Group: F (1, 60) = 7.32, p = .009, eta² = .11
Changes in Responses for Age of the 1st Intercourse
Online Telephone Interview-101234567
0.330.67
0.29
6
0.5
Low GroupHigh Group
Group: F (1, 17) = 16.90, p = .001, eta² = .50Mode: F (2, 17) = 7.56, p = .004, eta² = .47G * M: F (2, 17) = 19.01, p < .001, eta² = .53
Summary of the Results• Responses for ‘Game Addiction’ & ‘Dissatisfaction
with Self-Appearance’– Increased in the online survey mode– Decreased in the telephone & interview survey modes
• Those who scored higher on the sensitive questions are more likely to change their responses in a so-cially desirable direction– Less addicted (gambling, Internet, game)– Less dissatisfied with self-appearance– Less liberal (more conservative) sexual values– Younger in the age of the 1st sexual intercourse
Discussions & Suggestions• Survey responses to sensitive (privacy-related) questions are
affected by different survey modes.– Online survey in which respondents are less concerned
about their privacies is more likely to elicit franker (more ac-curate) responses.
• Paper-pencil (group administration), telephone, interview modes lead respondents to privacy concern & misreport.
• Suggestion: Dual-mode surveyTelephone Sampling Online Questionnaire
Any idea or sug-gestion
would be wel-
comed.