survey research and cell phones: is there a problem? · · 2013-01-07survey research and cell...
TRANSCRIPT
![Page 1: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/1.jpg)
Survey Research and Cell Phones:
Is There a Problem?Prepared for Harvard University conference on
“New Technologies and Survey Research”May 9, 2008
Scott KeeterDirector of Survey Research
Pew Research CenterWashington, DC
![Page 2: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/2.jpg)
Growth in the Cell-only Population
4.4%
12.6%
18.0%
29.1%
0%
5%
10%
15%
20%
25%
30%
35%
2004 2005 2006 2007
All Hispanic Ages 18-29
Source: National Health Interview Survey
![Page 3: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/3.jpg)
The Cell Phone Problem for RDD Surveys
• One-in-eight U.S. adults is cell-only according to June 2007 NCHS data
• The cell-only population is demographically different from the landline population
• As a result, landline surveys have experienced a sharp decline in the percentage of younger respondents interviewed in their samples
![Page 4: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/4.jpg)
Percent Ages 18-34
15
20
25
30
35
40
45
1990 1992 1994 1997 1999 2000 2002 2003 2004 2006
Parameter
Source: Pew Research Center surveys
![Page 5: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/5.jpg)
Percent Ages 18-34
15
20
25
30
35
40
45
1990 1992 1994 1997 1999 2000 2002 2003 2004 2006
Parameter
Unweighted
survey statistic
Source: Pew Research Center surveys
![Page 6: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/6.jpg)
Practical ConsiderationsIs it Feasible to Survey People on
Their Cell Phones?
![Page 7: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/7.jpg)
Yes, But It’s Expensive
• Cell interviews roughly 3x more expensive– Manual dialing– Reimbursement ($10 vs. $20)– Lower eligibility rate
• Many (42%) under age 18• Higher incidence of non-English speakers
• Cell-only interviews roughly 4-5x more expensive than landline– About 35% in cell sample are cell-only
![Page 8: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/8.jpg)
Interviewing Rates
Landline sample
Cell phone sample
Contact rate (2) 84% 83%
Eligibility rate 86% 45%
Cooperation rate (3) 27% 28%
Response rate (3) 23% 23%
Break-off rate 12% 10%
Source: October 2007 survey (results for three other surveys were highly similar)
![Page 9: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/9.jpg)
What We Did• 4 dual-frame surveys in 2006, 4 in 2007, 1 in 2008• Two surveys on politics and the campaign• One on gadgets/internet• One on economics• One of the Hispanic population
Oct 07GP
Dec 07 GP
Oct-Nov 07
Hispanic
Oct-Dec07GP
Jan-Feb 08GP
Landline N 1,507 1,089 1101 1554 1659Cell phone N 500 341 899 500 754Total 2,007 1,430 2,000 2,054 2,413
![Page 10: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/10.jpg)
Respondent Characteristics
Landline sample versus Cell-only adults
![Page 11: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/11.jpg)
Characteristics of Landline Sample and Cell-only Respondents*
12%
0% 20% 40% 60% 80%
Income <$30K
Age 18-29
Standard landline sampleCell-only
*Figures based on unweighted data
![Page 12: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/12.jpg)
Characteristics of Landline Sample and Cell-only Respondents*
12%46%
0% 20% 40% 60% 80%
Income <$30K
Age 18-29
Standard landline sampleCell-only
*Figures based on unweighted data
![Page 13: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/13.jpg)
Characteristics of Landline Sample and Cell-only Respondents*
41%
26%
13%
19%
61%
46%
21%
57%
6%
11%
48%
12%
0% 20% 40% 60% 80%
Income <$30K
Married
Hispanic
Black
Male
Age 18-29
Standard landline sampleCell-only
*Figures based on unweighted data
![Page 14: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/14.jpg)
How are estimates affected?
![Page 15: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/15.jpg)
Impact on Estimates from Including Cell Phones
37%
0% 10% 20% 30% 40%
Conservative
Republican/LeanRepublican
Approve ofPresident Bush
Republican/LeanRepublican
Standard landline sampleCombined landline + cell sample
![Page 16: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/16.jpg)
Impact on Estimates from Including Cell Phones
37%36%
0% 10% 20% 30% 40%
Conservative
Republican/LeanRepublican
Approve ofPresident Bush
Republican/LeanRepublican
Standard landline sampleCombined landline + cell sample
![Page 17: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/17.jpg)
Impact on Estimates from Including Cell Phones
28%
30%
35%
36%
27%
30%
35%
37%
0% 10% 20% 30% 40%
Satisfied withstate of nation
Approve ofPresident Bush
Conservative
Republican/LeanRepublican
Standard landline sampleCombined landline + cell sample
![Page 18: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/18.jpg)
Impact on Estimates from Including Cell Phones
1%
4%
9%
12%
17%
20%
22%
1%
4%
8%
11%
18%
19%
21%
0% 10% 20% 30% 40% 50%
Hunter
Paul
Thompson
Romney
Huckabee
Giuliani
McCain
Standard landline sampleCombined landline + cell sample
Based on Republican & Rep-leaning RVs(December 2007)
![Page 19: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/19.jpg)
Impact on Estimates from Including Cell Phones
0%
2%
3%
3%
14%
26%
46%
0%
2%
4%
3%
14%
27%
44%
0% 10% 20% 30% 40% 50%
Dodd
Biden
Richardson
Kucinich
Edwards
Obama
Clinton
Standard landline sampleCombined landline + cell sample
Based on Democrat& Dem-leaning RVs
(December 2007)
![Page 20: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/20.jpg)
Media Consumption Estimates
8%
8%
5%
28%
54%
40%
0% 20% 40% 60% 80%
Web: Natl newspaper
Web: local newspaper
Web: newsp yesterday
Network news
Local TV news
Newspaper (yesterday)
Standard landline sampleCell-only
![Page 21: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/21.jpg)
Media Consumption Estimates, 2006
11%
12%
9%
21%
43%
27%
8%
8%
5%
28%
54%
40%
0% 20% 40% 60% 80%
Web: Natl newspaper
Web: local newspaper
Web: newsp yesterday
Network news
Local TV news
Newspaper (yesterday)
Standard landline sampleCell-only
![Page 22: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/22.jpg)
Media Consumption Estimates, 2006
8%
9%
6%
27%
53%
40%
8%
8%
5%
28%
54%
40%
0% 20% 40% 60% 80%
Web: Natl newspaper
Web: local newspaper
Web: newsp yesterday
Network news
Local TV news
Newspaper (yesterday)
Standard landline sampleCombined landline + cell-only
![Page 23: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/23.jpg)
Under Age 30: Landline Sample and Cell-only Respondents*
32%
32%
12%
16%
48%
55%
0% 20% 40% 60% 80% 100%
Income <$30K
Married
Hispanic
Black
Male
Age 18-24
Landline sampleCell-only
*Figures based on unweighted data
![Page 24: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/24.jpg)
Under Age 30: Landline Sample and Cell-only Respondents*
48%
15%
15%
15%
62%
70%
32%
32%
12%
16%
48%
55%
0% 20% 40% 60% 80% 100%
Income <$30K
Married
Hispanic
Black
Male
Age 18-24
Landline sampleCell-only
*Figures based on unweighted data
![Page 25: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/25.jpg)
Under Age 30: Impact on Estimates from Including Cell Phones
27%
34%
28%
38%
25%
35%
27%
35%
0% 10% 20% 30% 40%
Conservative
Republican/LeanRepublican
Approve ofPresident Bush
Iraq: keeptroops in
Standard landline sample estimate for ages 18-29Combined landline + cell sample estimate for ages 18-29
![Page 26: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/26.jpg)
What’s the Potential for Bias?
• Example: % Favoring Iraq Withdrawal
• Is the combined estimate less biased?Under current conditionsIf cell-only adults uniformly favored withdrawalIf few cell-only adults favored withdrawal
![Page 27: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/27.jpg)
What’s the Potential for Bias?
![Page 28: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/28.jpg)
What’s the Potential for Bias?
![Page 29: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/29.jpg)
What’s the Potential for Bias?
![Page 30: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/30.jpg)
What’s the Potential for Bias?
![Page 31: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/31.jpg)
Quality of Responseslandline versus cell phone interviews
![Page 32: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/32.jpg)
Quality of Responses
• Are cell phone respondents more distracted?
Landline sample
Cell phone sample
Very distracted 1 2Somewhat 11 11Not too 18 14Not at all 70 74
100% 100%(n=1,507) (n=500)
Source: October survey
![Page 33: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/33.jpg)
Quality of Responses
• Are cell phone respondents less cooperative?
Respondent’s cooperation was…
Landline sample
Cell phone sample
Very good 78 81Good 15 13Fair 6 5Poor/Very poor 1 1
100% 100%(n=1,507) (n=500)
Source: October survey
![Page 34: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/34.jpg)
Sample Sizes:Will a combined design yield more
interviews with groups relying mostly on cell phones?
![Page 35: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/35.jpg)
Not if the cost is fixedExpected n
landline sampleExpected ncell sample
Standard Design ($100,000)Total sample 2,000 0 2,00018-29 yr olds 246 0 246
Blacks 212 0 212
![Page 36: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/36.jpg)
Not if the cost is fixedExpected n
landline sampleExpected ncell sample
Standard Design ($100,000)Total sample 2,000 0 2,00018-29 yr olds 246 0 246
Blacks 212 0 212
Combined Design ($100,000)Total sample 1,100 300 1,40018-29 yr olds 135 83 218
Blacks 117 45 162
![Page 37: Survey Research and Cell Phones: Is There a Problem? · · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research](https://reader030.vdocuments.us/reader030/viewer/2022021512/5b18e4877f8b9a37258c3e69/html5/thumbnails/37.jpg)
Conclusions• The risk of error from exclusion of cell-only
adults is increasing• No evidence of error yet for overall estimates• Comparable response rates for cell / landline• Cell interviewing is about 3x more expensive• Some evidence for gains in estimates for
groups relying mostly on cell phones• Budget permitting, a cell sample may be
prudent & boost credibility of findings