cell phones for data collection: costs and challenges michael link 1, michael battaglia 2, martin...
TRANSCRIPT
Cell Phones for Data Collection: Costs and Challenges
Michael Link1, Michael Battaglia2, Martin Frankel3,
Larry Osborn4, and Ali Mokdad5
1 Nielsen Media Research2 Abt Associates Inc.3 Baruch College, City University of New York
4 Knowledge Networks
5 Centers for Disease Control and Prevention
The Plague of Cell Phones!!!
Cell phones and telephone surveys Reliance on cell phones increasing (July-
December NHIS): 57.1% of households have a working cell phone 11.6% of households (11.8% of adults) are cell
phone only 25.4% for age 18-24, 29.1% for age 25-29, 54.0%
of unrelated adult households w/o children, 26.4% for renters
Result: increased potential for noncoverage bias
Cell Phone Summit 2005, TSM II 2006, AAPOR 2007 – special issue of POQ
Cell phones and telephone surveys
Conducting surveys via cell phones can be operationally challenging Cell phone frame may not be that efficient Geographic specificity is a problem Cannot use autodialers/predictive dialers Charges for incoming calls/minutes used Safety concerns Potential mode effects / measurement errors Level of cognitive engagement
Behavioral Risk Factor Surveillance System (BRFSS)
Monthly state-based landline RDD survey of health issues and related risk factors
50 states, District of Columbia, Puerto Rico, Guam, and Virgin Islands
350,000+ adult interviews conducted in 2006
Significant declines in participation overall, particularly among younger adults and males
2007 BRFSS cell phone pilot
Conducted in Georgia, New Mexico, & Pennsylvania
Target: 200 cell & landline / 200 cell-only adults (per state) 1,200 total interviews
Abbreviated BRFSS core interview: 66 questions 15-17 minutes (on average)
Incentives: $10 post-paid incentive for completing the
detailed interview $1 for completing the screener
Sample design Marketing Systems Group (MSG):
All designated cellular 1,000 banks Implicit stratification by area code and exchange Equal probability sample of telephone numbers
Survey Sampling Inc. (SSI): All 100 series banks designated as cellular “Mixed use” (landline / cell phone) banks
containing zero residential directory-listed numbers
Implicit stratification by FIPS, carrier, & 100-block Systematic random sample of 100 blocks Randomly generate last 2 digits of number
Screening questions Introduction
Confirmed telephone number
Is this a cellular telephone?
Are you 18 years of age or older?
Are you a resident of (state)?
“Do you also have a landline telephone that is used to make and receive calls?” Yes – took subsample of respondents No – took all respondents
Survey Participation Rates
Calculation of rates
Used detailed disposition codes modeled after Callegaro et al (2007) with some modifications/additions
Included “ring, no answer” and “voice mail” as working residential numbers Only cases confirmed by company message as
being not in service were excluded
Used AAPOR response rate guidelines
Calculated separate rates for: Screening for eligible respondent Completion of interview
Participation rates
GA NM PA
Starting sample size: 9,000 4,400 9,997
Completed interviews: 405 413 346
% Working cell numbers: 64.7 63.2 72.5
Screener rate: 40.2 47.5 34.3
Interview rate: 60.8 65.8 67.6
Response rate: 24.4 31.3 23.2
Interview rate by landline access
0102030405060708090
100
Total GA NM PA
Cell only Cell & landline
64.364.9
57.766.1
66.764.1 70.2
64.8
Percent working cell numberby sample vendor
0102030405060708090
100
GA NM PA
MSG SSI
63.366.1
61.265.3
70.574.5
Comparison of respondent demographics
Landline and Cell phonepopulations and frames
CELL PHONELANDLINE A B C
0
10
20
30
40
50
60
70
80
90
100
Landline (only) Cell & landline Cell & landline Cell (only)
46.651.1
38.237.9
Landline survey Cell phone survey
Percent male
State equalized design weight applied
0
10
20
30
40
50
60
70
80
90
100
Landline (only) Cell & landline Cell & landline Cell (only)
24.0
51.4
19.614.5
Landline survey Cell phone survey
Percent 18-34 years
State equalized design weight applied
0
10
20
30
40
50
60
70
80
90
100
Landline (only) Cell & landline Cell & landline Cell (only)
15.2 21.412.216.8
Landline survey Cell phone survey
Percent Hispanic
State equalized design weight applied
0
10
20
30
40
50
60
70
80
90
100
Landline (only) Cell & landline Cell & landline Cell (only)
15.0 15.87.59.3Landline survey Cell phone survey
Percent black
State equalized design weight applied
0
10
20
30
40
50
60
70
80
90
100
Landline (only) Cell & landline Cell & landline Cell (only)
39.848.5
33.6
60.3
Landline survey Cell phone survey
Percent high school or less education
State equalized design weight applied
0
10
20
30
40
50
60
70
80
90
100
Landline (only) Cell & landline Cell & landline Cell (only)
62.0
32.0
69.8
49.5
Landline survey Cell phone survey
Percent married
State equalized design weight applied
Summary of significant differencesacross demographic subgroups
Cell only v.s. cell & landline adults (from cell phone survey): Significant differences for 12 of 24 subgroups
examined Particularly age, employment status &
marital status
Cell & landline adults (cell phone survey v.s. landline survey): Significant differences for 11 of 24 subgroups
examined Particularly sex, race, marital status, and
children in household
Comparison of key survey estimates
0
10
20
30
40
50
60
70
80
90
100
Landline (only) Cell & landline Cell & landline Cell (only)
86.0
70.1
89.0
78.7
Landline survey Cell phone survey
Percent any kind of health care coverage
State equalized design weight applied
0
10
20
30
40
50
60
70
80
90
100
Landline (only) Cell & landline Cell & landline Cell (only)
16.324.9
10.220.4
Landline survey Cell phone survey
Percent not received care due to cost barrier
State equalized design weight applied
0
10
20
30
40
50
60
70
80
90
100
Landline (only) Cell & landline Cell & landline Cell (only)
19.731.1
17.324.8
Landline survey Cell phone survey
Percent currently smoke cigarettes
State equalized design weight applied
0
10
20
30
40
50
60
70
80
90
100
Landline (only) Cell & landline Cell & landline Cell (only)
43.6
54.2
36.637.5
Landline survey Cell phone survey
Percent ever tested for HIV
State equalized design weight applied
0
10
20
30
40
50
60
70
80
90
100
Landline (only) Cell & landline Cell & landline Cell (only)
13.023.521.1
11.0Landline survey Cell phone survey
Percent binge drink past 30 days
State equalized design weight applied
Comparison of Survey Costs
Cost per Interview
Data collection costs only
Level of effort: RDD = 7.4 calls/case Cell = 3.2 calls / case
Response rate: RDD = 38% Cell = 26%
Interview length: RDD = 25 minutes Cell = 12 minutes
$0
$50
$100
$150
$200
Landline survey
Cell phone survey
Cell only (via screening)
$60 $74
$196
What have we learned?
Group with both landline & cell phone differ across landline and cell phone surveys Mode effect? Response/nonresponse
effect? Frame effect? This is an important issue when we try to
combine landline & cell phone surveys
Cell phone only group differs significantly from landline group on some health variables, but not others Risk behaviors seem most problematic
What have we learned? Cell phone & landline usage varies
significantly across states Makes use of national estimates from the NHIS
for post survey adjustment problematic
Compared to landline surveys, cell phone surveys: Have lower rates of response at the screener
stage But similar rates at the interview stage
Working residential rates lower, but not as bad as expected
Are considerably more expensive, especially if we decide to screen for cell-only adults
Future directions Problem is not going away, but will continue to
worsen
Focus on combining estimates from cell and landline frames (Frankel and Battaglia are working on this for the BRFSS) Cell only households not the only problem Need more focus on primary cell users
Best to reach by landline or cell frame?
Mode effects – what measurement issues are raised by interviewing via cell phones?