2/22/2011

6975 Union Park Center Salt Lake City, UT 84047

800-678-3748 www.DiscoveryResearchGroup.com

Comparison of Mobile Online, Landline, Cell Phone, & Online Access Panel

By David Johnson, Director of Project Analytics

2

Contents

Executive Summary ........................... 2

Introduction ......................................... 3

Secondary Research .......................... 4

Research Methods .............................. 6

Data Analysis ...................................... 9

Findings ............................................. 10

Limitations ......................................... 17

Conclusions & Recommendations . 18

References ........................................ 20

Appendices ....................................... 23

Executive Summary

The landline, cell phone, and online access panel are well known and frequently used channel modes for data collection. Given the relatively new frontier of mobile online using smartphones as a collection tool it is important to understand this mode and the characteristics of respondents who do respond. The present study focuses on four modes (traditional landlines, cell phones, PC-based online access panel, and mobile-based online) for collecting data from the general population in a national survey of the contiguous United States. A total of 852 individuals responded to the survey, answering between 7 and 12 questions. Question wording and response options were identical across modes. There are three categorical question types, namely demographic questions, cell phone preferences and usage, and one potentially sensitive question. Most findings are unsurprising and to be as expected, such as landline respondents are older, they use fewer data functions on their cell phones, and most are female respondents. An interesting finding did come out of this research. In this sample, online access panel users were found to be more like landline respondents than cell phone or mobile online respondents. Online access panel respondents, compared with cell phone and mobile online respondents, are on average older, more likely to be landline only or make calls with a landline instead of a cell phone, and use fewer data functions on their cell phones in general.

3

Each mode is better suited for reaching a different age demographic: Landlines are best suited for reaching respondents age 55 and older; online access panels appear to represent 45-55 year olds most accurately; cell phone respondents fell out in the 35-44 age category; while the mobile online was the youngest at 18-34 years old. A combination of the four modes reflects the least amount of difference to US Census data than each mode individually. In order to adjust for effects derived from each mode, weight schemes are created and briefly discussed in this paper. As a result of adjusting for modal differences to Census data a small percent of response comparisons gained significance where not previously found, while a large percentage of the categorical differences lost significance.

Introduction

Discovery Research Group (DRG) has been a key market research industry data provider for over 20 years, and with the advent of new modes of data collection seeks to not only provide the capability but have become an industry leader concerning these modes and methods. Landline telephone surveys are becoming increasingly more difficult to administer due to rising costs and declining response rates. Cell phone dominance over landline use is continuing to grow, thus affecting landline response rates and representativeness. The internet as a tool for data collection is fast becoming a viable and acceptable alternative. The big question mark in data collection channels regards the mobile online mode. Mobile online on smartphones is relatively new and thus has many unanswered questions concerning its appropriateness as a channel mode for data collection. Demographic characteristics of respondents, the instances in which it is a feasible option, and how the cost of administration compares are a few of the presently unknown issues about mobile online. DRG, in an effort to better understand some of the unknowns concerning this mode, has commissioned this study on mobile online survey administration. Results of mobile online responses are compared against landline, cell phone, and online access panel sample. A combined sample file of the four modes is also compared against each individual mode. Mobile online, much like the online access panel, has some enticing realistic advantages, such as a consistent question delivery, no interviewer bias, less pressure to respond in a socially acceptable fashion, ability to easily present visual and/or audible materials, respondents have spatial and time flexibility to complete, and an ability for the respondent to transfer to a different mode (e.g. self-directed mobile online to an interviewer).

4

Secondary Research Using an internet-based research approach has many benefits over other methods like landline or paper and pencil. The number of individuals who have web access continues to increase, thus potentially providing access to large samples of individuals that have been unreachable by traditional means (Michalak 1998 & Pasveer & Ellard 1998). As reported by The Nielsen Company from the results of their mobile insights survey in March of 2010, the market share of mobile online smartphones is projected to surpass the “feature” cell phone by Christmas of 2011, thus increasing the accessibility of a traditionally difficult-to-reach young subpopulation. A mobile online approach, like the traditional online access panel, benefits from a shorter time period needed for fielding, ability to include visual and auditory materials, requires no data entry by an interviewer, and still allows for complex skip patterns (Blyth 2008; Bronner & Kuijlen 2007; Roberts 2007; and Webb, Zimet, Fortenberry, and Blythe 1999). There is a concern that the quality of the data from a mobile online sample will have higher sampling error. Sampling error occurs when the whole population is not surveyed. Like most online access panel sampling, mobile online sampling is not created from probability but from a convenience approach, and may result in systematic bias, and the confidence interval of the associated variability cannot be reliably described (Nagelhout et al 2010). When using a mixed-mode administration, weighting was found to reduce bias and enhance post-stratification adjustments (Baines, Partin, Davern, and Rockwood 2007). Population weighting schemes typically include geography, race, age, gender, and ethnicity to improve the representation of these groups. . Baines, Partin, Davern, and Rockwood (2007) report that post-stratification weighting adjustments are enhanced and non-response bias is decreased with the use of a mixed mode survey administration. Gallagher et al (2005), de Leeuw (2005), Sala and Lynn (2007) found that mixed-mode surveys encouraged responses, reduced respondent burden, and were an effective means to increase response rates. When the response rate of a survey is less than 100%, there needs to be some amount of attention spent addressing how non-responders differ from responders. With an understanding of how these two differ, more appropriate assumptions and conclusions concerning the target population can be made from the collected data. Generally, it is assumed that the data are “missing at random” (MAR). MAR is built on the premise that those subgroups that are not well represented do not differ qualitatively in their responses within the survey from the members of that subgroup that did respond (Groves 1989). Baines, Partin, Davern, and Rockwood (2007) discovered that phone non-responders who were followed up with a mail survey were different. Through the use of a mixed-mode design they were allowed to reach segments that would

5

have otherwise been missed if using only one mode. Their findings are not universally supported. When mixed-modes have been researched, the results do not clearly provide evidence for equivalent data or different results (O‟Toole, Battistutta, Long, & Crouch, 1986; Wu et al 1997; Weinberger et al 1994; McHorney, Kosinski, & Ware 1994; & Galobardes et al 1998). Differences in findings have been noted in mode of administration (interviewer-directed versus self-directed; visual versus auditory), variable question responses, characteristics of the sample, or social desirability bias (Herzog & Rodgers 1988; Sala & Lynn 2007, Brambilla & McKinlay 1987; Groves 1989; & Hox, de Leeuw 1994). The interviewer presence, in-person or over the phone, can be a distraction to respondents. Respondents have been shown to provide a large amount of positive or socially desirable responses because of their desire to not reveal beliefs they feel will not likely be endorsed by the interviewer (Bowling 2001). Interviewers vary in their ability to be neutral. Probing, listening, intonations, appearance, and recording of responses are all possible areas where interviewers can bias responses (Bowling 2001). Monitoring and training of interviewers can minimize this source of bias, but self-administered modes, such as online access panel and mobile online, avoid this source of bias. Bowling continues by stating that when respondents interact with an interviewer, the interview becomes a social interaction, thus increasing the likelihood that respondents will take into consideration social norms and the interviewer‟s perception. Self-administration methods, like online access panels and mobile online, have a greater amount of impersonality than interviewer-directed methods, like landline and cell phone. As a result, findings may be confounded with respondents under-reporting undesirable behaviors and over-reporting desirable behaviors. Relationships between questions found during analysis will be suspect. There is much support for Bowling‟s finding in other studies. Lyons et al (1999), Bowling, Bond, Jenkinson, & Lamping (1999), Presser & Stinson (1998), Tourangeau et al (1997), and Tourangeau & Smith (1996) all found that respondents did give more positive and socially acceptable responses in interviewer-directed (phone and face-to-face) surveys than in self-administered (postal) surveys even when taking into account the order and contextual effects. There is also clear support from many studies that there is a positive exaggeration of estimates concerning positive health status, health related quality of life, and desirable behavior and activity engagement, when based on face-to-face or telephone interviews (Lyons et al 1999; Vuillemin et al 2000; Brambilla & McKinlay 1987; McHorney, Kosinski, & Ware 1994; Perkins & Sanson-Fisher 1998; and Weinberger, Oddone, Samsa, & Landsman 1996). The use of self-administered modes, as compared to interviewer-directed, are likely to increase respondent willingness in disclosing sensitive information. Anonymity in mail surveys has been shown to have higher item responses and more accurate reporting on sensitive topics of health and behavior because of its weak social presence (Tourangeau, Rips, & Rasinski 2000; Pruchno & Hayden

6

2000; Parker & Dewey 2000; Davis 1999; Pasveer & Ellard 1998; Smith & Leigh 1997; and Siemiatycki 1979). Research Methods Though there is some similarity, each data collection mode had its own methodology, which will be provided shortly. All modes utilized the same questions and logic within the questionnaire. Respondents answered 12 questions if they were both a landline and cell phone owner, if not they answered 7. The questionnaire was written and prepared by DRG staff. A copy of the questionnaire is provided in Appendix A along with question wording source, if applicable. Questions asked pertain to demographics (5), type of phone service (1), cell phone preferences (5), and potentially sensitive nature (1). Demographic questions allow for an understanding of who respondents are within each mode and for creating a weighting scheme using Census data. Type of phone service provides a glimpse into what phone types are being used and how large each group is. This question will be important for tracking when it is asked in subsequent questionnaires. Cell phone preference questions allow for a gauging of which features and products are important to cell phone owners. The potentially sensitive question is used for determining the impact of mobile online as a data collection tool, and to verify its feasibility in studies that contain questions respondents may find difficult to answer to an interviewer. Administration of the survey was done in English with no other languages offered. Fielding of the survey resulted in 200 landline interviews, 202 cell phone interviews, 200 interviews using online access panel, and 252 mobile accessed online access panel interviews. Each mode was collected independently of the other modes, so that each mode could stand alone as a small contiguous national sample of that mode. The determination of 200 completes was based on the margin of error. Individually, each mode has a margin of error of 7% and if combined 2%. DRG coordinated and carried out all fielding, which took place between September 29, 2010 and November 14, 2010. DRG utilized their own call centers in Logan, Utah, and Twin Falls, Idaho – 365 and 59 hours respectively for both landline and cell phone dialing. Survey Sampling International provided panel access for the 200 interviews via online access panel as well as 52 of the mobile online interviews, while DMS Insights provided access for the remaining 200 mobile accessed online interviews. There were 2758 potential respondents that started the survey. Of the total that began, 1906 did not complete it (69%), while the remaining 852 completed the survey (31%). The most common reason for not completing the survey is due to sample sources being filled (76%). Only 2% of non-completes were the result of respondent choice. The remaining 22% were screened out before reaching the survey because they were under 18-years-of-age, operating a vehicle, or not in a place to speak freely (11%, 3%, 8% respectively).

7

Self-administration in online access panels and mobile online modes allows respondents to proceed at a pace comfortable to them. Conversely, interviewers are trained to dictate the interaction between respondent and the questionnaire. The control interviewers place on the process inherently adds length to the overall time of completing the survey. The median time to complete the survey across all modes is 2 minutes 8 seconds. Self-directed respondents median time to finish is 47 seconds quicker than those responding via interviewer (1 minute 41 seconds and 2 minute 28 seconds, respectively). The difference between self-directed and interviewer-directed averages are significantly different (1 minute 59 seconds and 2 minutes 50 seconds). Online access panel respondents have the lowest average completion time while cell phone respondents have the highest (1 minute 38 seconds and 3 minutes 14 seconds accordingly). Prior to dialing on both landline and cell phone sample for the project, 35 interviewers from the two phone centers, were required to attend and participate in a briefing that was approximately 30 minutes in length. During the landline and cell phone interviewing, there were one to six interviewers dialing at any given time. Fielding of the project occurred September 29 – November 14, 2010. In accordance with federal law, all interviewers calling a known cell phone were hand-dialed. For both landline and cell phone dialing, all calls were placed between 11am MST and 9pm MST. Each time zone would be dialed between local times of 1pm and 8pm. The first time zone to be dialed was the East Coast with the last being the West Coast. All dialings were for the contiguous United States only.

Table 1. Diagnostics of landline and cell phone dialing

Diagnostics: Landline Cell Phone

Sample Diagnostics

Dialing Yield 26% 66% Contact Rate

46% 64%

Working Phones Rate

36% 50%

Incidence Diagnostics Net Effective Incidence (MRA)

79% 78%

Coop Rate (MRA)

81% 85% Term Rate

19% 15%

Net Incidence (MRA)

79% 78%

Refusal Diagnostics Refusal Rate (1)

69% 77%

Refusal Rate (CMOR)

63% 52% Refusal Rate (YOC)*

76% 80%

Response Rate (CASRO)

-54% -26%

8

LANDLINE - A Random Digit Dialing (RDD) national sample of landline phone numbers was obtained from Survey Sampling International (SSI) using their Random B sample selection method. Random B is a sample of random numbers distributed across all blocks in proportion to their density of listed telephone households. SSI did not screen numbers from the list (i.e., business, protected, etc.). A more detailed explanation of SSI‟s landline sample methodology is in Appendix F. Before loading the SSI RDD sample into DRG‟s Computer-Assisted Telephone Interviewing (CATI) program, the sample was scrubbed against an in-house Do Not Call list. During the dialing of phone numbers, if the interviewer was unable to reach the respondent and complete the survey or attach a final disposition to the phone number, sample records would be dialed 6 times before they were considered exhausted.

CELL PHONE – SSI also generated the cell phone number sample. Currently, there are no sources for listed or assigned cell phone numbers available. Using Telcordia‟s monthly TPM (Terminating Point Master) Data files, SSI selects phone records based on FIPS codes, NPA-NXXs codes, and 100- and 1000-blocks with carrier names that are identified as Cellular. A more detailed explanation of SSI‟s wireless sample methodology can also be found in Appendix G. ONLINE ACCESS PANEL AND MOBILE ONLINE – Online and mobile online access panels have a standard sampling frame that they recommend for ensuring representativeness. DRG intentionally choose an open form of sampling with no quota stops. To best evaluate online and mobile online access panels to RDD landline and cell phone it was determined that a natural fall-out of the sample was necessary. The project began with one sample source for both mobile online and online access panel also provided by SSI. SSI‟s Panel Management methodology recruits panel members among a varied number of sources (i.e., thousands of websites, single ISP, portal, or membership in a loyalty program) to optimize the probability that panelists reflect the overall composition of the online population. A more detailed explanation of SSI‟s Panel Management methodology is located in Appendix H.

SSI sent email invitations to panel members to participate in the online survey. The survey was created and hosted by DRG. To ensure that each mode, online access panel and mobile online, filled correctly, a question was added to the survey regarding the device being used to complete the interview. When the online access panel quota filled, the question was moved to the beginning of the survey to act as a screener. The screener question became costly as DRG found that few respondents in the SSI panel completed interviews via their smartphone.

9

To combat the online access panel‟s difficulty in targeting smartphones, a new sample was obtained from DMS Insights. DMS Insights‟ River was able to effectively identify mobile respondents by filtering for the type of browser being used to access the online survey. DMS Insights‟ River increased access to smartphone users through a constantly changing stream of participants who were identified as using smartphone based browsers. While browsing the internet, if the potential respondent clicks on a particular banner they are asked screening questions. After completing the initial questions and identifying that the browser is installed on a smartphone, they were then directed to the survey. See Appendix I Respondents participating in the survey through an online access panel or smartphone mobile online browser were not required to download an application (“app”). Access to the survey was made available at a static web address. The choice to provide a web address instead of an app was made because of the creation and maintenance costs associated with an app, decreased traffic from needing to download an app, and having one data storage location for all four modes simplified the data collection process and amount of programming. Data Analysis

Three types of analysis are performed when examining the findings. At the first level, frequencies of each question, irrespective of channel mode, are examined for face validity. For each question‟s frequency distribution it was asked “Do the responses appear as would be expected?” This test provides confidence in the data and a simple understanding of the data but little else. To understand more of the inter-question and question-channel mode associations, cross-tabulations are performed. Significance tests, Chi-square, Z-tests for percentages, and T-tests for means are used to determine if the findings can be attributed to more than chance.

The final analysis performed is a dichotomous logistic regression (DLR) also referred to as binary logistic regression. DLR estimates the likelihood and probability of a relationship between the dependent and covariates. The questions in the survey are categorical and are more suited for DLR. A dummy variable of each mode and the two numerical ranking questions was created. DLR is used to help determine likelihood of reaching subgroups (age, gender, education, etc.) within each mode.

Weighting. In order to account for channel mode bias in demographic differences and non-response, a weighting scheme was created. Population weights increase the representation of select groups (geography, race, age, gender, and ethnicity) that are less likely to respond in a survey (Baines, Partin, Davern, & Rockwood 2007). Unless specified differently, the tables are

10

unweighted. The final piece of part 1 included in the findings covers the difference between unweighted and weighted data. Findings The four distinct modes are compared in the following ways.

1. Modal differences

2. Differences associated with mode of collection

3. Odds ratios

1. Modal differences The differences to each question by mode are explored here. No two modes are identical on every question, though some strong similarities do emerge. The full table of marginal and cross-tabulations of each question run against mode is provided in the appendix. Channel mode differences highlight the oversampling and undersampling bias. Sex. During fielding, individuals over 18 were the only ones screened to participate. No criteria for selection by sex or most recent birthday was instituted. All but one sample source had a greater percentage of female participants. More males than females were reached on cell phones. RDD landline and the mobile online oversampled females more than males.

Age. The average age of landline respondents is significantly higher than the other sample sources. Highest median age is among landline respondents and youngest is mobile online (60 years-old and 37 years-old, respectively). As anticipated, very few RDD landline sample respondents are in the heavily sought-after 18-24 year-old range (4.0%). There are no statistically significant differences between cell phone, online access panel, or mobile online mode respondents between the ages of 18-24 years. Mobile online has the highest

40.5 33.0 53.5 46.5

31.2 48.7

59.5 67.0 46.5 53.5

68.8 51.3

0%

20%

40%

60%

80%

100%

Combined Landline Cell Phone OnlineAccess Panel

MobileOnline

Census

Sex of respondent by mode with Census data for comparison (percentage)

Female

Male

11

percentage of adult respondents 18-34 (57.2%). Just slightly under 10% of respondents in mobile online mode were 55 years-of-age or older (9.6%).

Education. In each mode, the highest level of education for roughly 1 of 3 respondents is a high school degree or less (34.0%). No sample source is significantly different statistically than any other source at each categorical level of education. This symmetry across modes shows no obvious bias by one mode for over- or under-sampling for a particular level of education.

Phones. Among all sample sources, the proliferation of cell phones is evident with 9 of 10 respondents having a cell phone (89.9%). Once the primary communication for homes, the landline is now reported to be used by only two-thirds of respondents. An interesting note: only half of cell phone and mobile online sample respondents have a landline. The landline as the only telephone

50.4

59.6

48.1 54.6

39.9 36.8

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

Combined Landline Cell Phone Online AccessPanel

Mobile Online Census

Mean age of respondent by mode with Census data for comparison

34.0 37.0 36.6 31.0 32.0 43.3

32.3 27.5 32.7 35.0 33.6

28.8

21.1 22.5 18.8 21.0 22.0 17.6

12.6 13.0 11.9 13.0 12.4 10.3

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Education level by mode with Census data for comparison (percentage)

Post graduate degree

College degree (4-year)

Some college

High school degree or less

12

for the respondent represents only 9.9% while cell phone-only households are 32.2%. Nearly two-thirds of cell phone respondents are cell phone dominant (82.2%) using only a cell phone or making more calls with their cell phone than landline. As expected, this is most prevalent among the cell phone sample and mobile online panel. As a whole, online access panel respondents evenly divide the amount of use between cell phone and landline.

Individuals with both landline and cell phones in landline and online access panel samples are not likely to switch to cell phone only (64.4% and 55.6%, respectively). One in five online access panel respondents are neither likely nor unlikely to switch (18.8%). Nearly half of mobile online and cell phone responders state they will likely switch to being cell phone only (48.8% and 49.1% accordingly). Among cell phone owners, mobile online users perform significantly more activities with their phones than the other modes. On average, mobile online respondents use 4.1 of the 6 listed data functions on their cell phones. Significantly fewer landline and online access panel respondents take advantage of the multiple data functions on their cell phones, averaging 0.8 and 2.1 functions, respectively. Age is strongly correlated with cell phone data function use. As the age of respondents increase, there is a sharp decrease in the level of use. Regardless of mode, significantly more respondents 45 years and older reported not using any of the cell phone functions listed. Social Desirability: To compare self- and interviewer-directed questionnaire administration methods, the final question asked of each respondent was likely seen as of a sensitive nature to some. The goal in this comparison is to

9.9

27.0 15.0

14.6

22.0

5.0

29.0

4.8

10.6

14.0

10.4

9.5

8.8

30.8

32.0

33.7

20.0

36.0

32.2

48.5

25.5

50.0

2.1 5.0 2.5 1.0

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Combined Landline Cell Phone OnlineAccess Panel

MobileOnline

Respondent preference for phone line by mode (percentage)

Other only/refused

Cell phone only

More with cell phone

Equal

More with landline

Landline only

13

determine if one data collection strategy is more fitting for these types of questions. Fewer respondents providing their answers through an interviewer instead of an electronic medium reported the use of illicit drugs. Mobile and internet survey taking removes some of the pressure of responding in a way others in society would find acceptable. This finding holds true when controlling for education level and gender, three of the four geographical Census regions, and younger and older respondents (18-24 and 55 years and older).

Census. Four of the fifteen possible survey questions are comparable to Census data. There is a small difference between the amalgamation of each mode and Census data. Even though some response categories in each mode are similar to the Census, none are consistently closest to the Census on all questions. Overall, cell phone and online access panel sample modes are the most similar to Census data. Across all sample sources, the region makeup was most similar to Census data because of quotas that were set up during the fielding process. A combining of all four modes has less variance to Census data than each mode by itself.

2. Differences associated with mode of collection

20.9

10.5

15.3

25.5

30

0

5

10

15

20

25

30

35

Combined Landline Cell Phone Online AccessPanel

Mobile Online

Affirmation of illicit drug use by mode (percent)

3.6 7.1

4.0 3.7 7.8

0.0

5.0

10.0

Combined Landline Cell Phone Online AccessPanel

Mobile Online

Average percentage difference between Census data and demographic questions by mode

14

Inter-channel mode differences. Differences are explored through examination of the number of statistically significant differences and the number of responses that differ by 4 percentage points or more from a control group. Among the four modes, landline has been the most acceptable form of data collection for representative samples; the other three modes (online access panel, cell phone, and mobile online) are relatively new. Landline is compared against the other modes collectively and individually. Weighting. Channel mode differences highlight the oversampling and undersampling bias. As a means to account for and adjust for these biases, a weighting scheme was created. This proprietary weighting scheme is, in short, derived from current Census data and respondent bias. When applied, comparisons of weighted and unweighted data reveal significant changes. One of every three comparisons (31.4%) remained significantly different from landline when weighted. A very small percentage of responses gained significance (1.4%), and two-thirds (67.1%) are no longer significantly different after weighting. Statistically significant intermodal differences (See Table 2). Using the T-test for means and Z-test, for percentages to compare the landline sample against the other three modes (cell phone, online access panel, and mobile online panel), there are many observed significant differences in the unweighted data. The mean percentage of statistically significant differences is slightly more than one of every three comparisons (37.6%). Among the mobile online mode, every other comparison is significantly different (50.0%). When the results are weighted, there is sharp drop in the number of significant differences observed. All but the online access panel mode has a 20-percentage point drop or more in the number of observed significant differences. When the findings are weighted, the cell phone mode replaces mobile online as the mode with the greatest percentage of observed differences.

Table 2. Significance test results for landline to each mode and combined

Landline versus

Cell Phone, Online Access Panel & Mobile Online

(%)

Cell Phone

(%)

Online Access Panel (%)

Mobile Online

(%)

Comparisons 186 (100) 62 (100) 62 (100) 62 (100) Observed significant differences Unweighted 70 (37.6) 24 (38.7) 15 (24.2) 31 (50.0)

Weighted 28 (15.1) 7 (11.3) 6 (9.7) 15 (24.2)

15

Difference 42 (22.6) 17 (27.4) 9 (14.5) 16 (25.8) Response differences within 4-percentage points (see Table 3). Each mode is markedly different from landline sample in unweighted and weighted results. The percentage of response choices in unweighted findings that are within 4 percentage points is between 40.3% and 50.0% with an average of 46.2% for all modes combined. The mean percentage point difference of each mode to landline is 9.2%. Online access panel respondent answers are the most similar to landline with an average difference of 6.0%. Weighting of the data resulted in all modes and the three modes combined to become more similar to landline sample. Cell phone sample shows the largest percentage-point change in the percentage of responses within 4 percentage points from unweighted to weighted.

3. Odds ratios In order to better understand the complexities and disentangle the roles played by each variable, four distinct models were run based on sample source: landline, cell phone, online access panel, and mobile online. The results of the regression models are summarized in Table 5 and 6 in Appendix A. Each sample source has unique characteristics that are similar to other sources but none that follow the same trend.

Table 3. Summary of the magnitude of differences to landline by mode and combined mode

Landline versus

Cell Phone, Online Access

Panel & Mobile Online

Cell Phone

Online Access Panel

Mobile Online

Number of response choices with a difference of 4 percentage points or less Unweighted

86 (46.2)

31 (50.0)

30 (48.4)

25 (40.3)

Weighted 114 (61.3) 43 (69.4) 40 (64.5) 31 (50.0)

Difference

28 (15.1)

12 (19.4)

10 (16.1)

6 (9.7)

Mean percentage-point difference Unweigted 9.2 8.9 6.0 14.8 Weighted 6.3 4.6 4.2 10.1

16

In support of the previously mentioned bivariate findings of age characteristics by mode and other research, landline sample respondents tend to be older (Pew Research, 2006). The odds of reaching respondents 55 years-of-age and older in a landline project as opposed to ones that are 18-24 years-old are high (Odds Ratio (OR) = 3.6, 3.6, and 17.4). Of online access panel respondents, the odds of 25-44 year-olds and those over 75 being represented in the online access panel sample are 40% of 18-24 year-olds (OR = 0.4 for each). In mobile online sample, the odds of being 55 years-of-age or older is much less likely than having those that are between the ages of 18 and 24 (OR = 0.4 and 0.4 – no odds are provided for those 75 years or older).

Of interest to note, landline and cell phone respondents were less likely to have some college education than a high school graduate or less education. The opposite is true in online access panel and mobile online samples, but the differences are not significant enough to assert that these findings are more than found by chance. Cell phone and mobile online sample respondents with both a cell phone and landline are less likely to be in the sample than cell phone-only individuals (OR = 0.4 and 0.6, respectively). Online access panel sample respondents with only a landline have 2.4 times greater odds of being in the sample than cell phone-only respondents. In the landline sample, landline-only respondents have 2.8 times greater odds of being in the sample than those with both a cell phone and a landline. As asserted in the bivariate findings that respondents would respond differently to sensitive questions because of the presence of the interviewer and which is supported in the logistic regression, respondents that spoke with an interviewer in the landline and cell phone sample had greater odds of saying “no” than “yes” (OR = 1.6 and 2.1, respectively). The opposite is found of respondents that were self-directed through the survey, online access panel and mobile online (OR = 0.7 and 0.6).

1.0 1.1 3.3 2.3

3.6 3.6

17.4

0.0

4.0

8.0

12.0

16.0

20.0

18-24(Baseline)

25-34 35-44* 45-54 55-64** 65-74** 75+***

(Od

ds

Rat

io)

Note: *P≤0.05, ** P≤0.01, ***P≤0.001;

Likelihood of age groups to be included in landline sample

17

Individuals with both a landline and cell phone vary in characteristics depending on the type of sample they are reached with. Cell phone and mobile online respondents greatly prefer making calls with their mobile device than with their landline. The odds of them making more calls with their landline instead of more with their cell phone is small (OR = 0.3). Online access panel respondents are just the opposite, being 4.3 times more likely to use their landline than their cell phone. Cell phone and landline sample respondents are also markedly different in their odds of becoming cell phone only. The odds ratio for landline respondents of being not likely to be cell phone only is only 0.6. Cell phone respondents are 1.6 times greater the odds of being not likely to be cell phone only than likely.

Limitations

Interpretation of study findings should be done while taking into account several limitations. Online access panel and mobile online sample are based on convenience sampling. Generalizability of these two samples is suspect due to their self-selection to participate in the survey, but this is more true with mobile online than the online access panel. The findings confirm the general knowledge that the proliferation of mobile online use on smartphones is not as prevalent as

1.6

2.1

0.7 0.6

0.0

0.5

1.0

1.5

2.0

2.5

Landline Cell phone** Online Access Panel Mobile Online

(Od

ds

Rat

io)

Note: *P≤0.05, ** P≤0.01, ***P≤0.001;

Effects of sample type on respondents to say 'No'

0.6

1.6

0.8

1.3

-

0.5

1.0

1.5

2.0

Landline* Cell phone* Online Access Panel Mobile Online

(Od

ds

Rat

io)

Note: *P≤0.05, ** P≤0.01, ***P≤0.001;

Sample type respondent likelihood to be cell phone only

18

online access panel use in American households. This study begins the process, but more research will need to be performed relating to mobile online as a source for data collection. During fielding, the sample provider for the mobile online portion was changed. As described previously, the low level of incidence and the growing limitations with the sample required the acquisition of a new mobile online sample provider. There were slight differences between the original and new providers. SSI and DMS Insights were no different by region and age. Differences by sex were significant with more males in DMS sample, but both are very different from Census data so the point is of little importance. Differences in education between samples showed SSI to be more likely to report some college or less and DMS as fairly balanced in each of the four education levels. More research will need to be done to compare sample source and determine the impact of the sample provider‟s methodology. The findings in this paper are limited due to the relatively small sample sizes for a national study. As a whole, the margin of error is 2% and each individual mode‟s margin of error is 7%. Future studies will benefit by using larger sample sizes and as a result, will have smaller margins of error. Due to the short and focused survey, many ideal questions were not asked. Future research would do well to examine mobile online as a source of data collection for the influence of location when responding, length of time between noticing invitation and starting survey, and evaluate whether certain approaches work better than others (web advertisement, email invitation, app invite (FourSquare, Facebook Places, etc.), text message, etc.). Conclusions & Recommendations As a provider of data through many modes of administration (paper, IVR, IDI, landline, cell phone, online access panel, and now mobile online), DRG has a responsibility to learn the intricacies of emerging methods. This knowledge is gleaned from existing research as well as studies like this, which are done in-house. Through DRG‟s own experience, the administration of landline telephone survey work is becoming increasingly more difficult due to rising costs and declining response rates. More and more households and individuals are becoming cell phone-only or cell phone-dominant households. The capability to sample desired populations needs to continue to adapt to new technologies as it has in the past. The internet has become a more economical, viable, and acceptable data collection tool; and with the smartphone mobile online market opening up to researchers, this web trend is likely to continue to grow. The differences between landline and each of the other modes is evident in all questions. As is shown in the findings, each mode was more biased toward particular subgroups. Cell

19

phone respondents were more likely to be male and landline female. Landline respondents are more likely to be older and mobile online younger. From oldest to youngest, the average age of respondent by mode are: landline, online access panel, cell phone, and lastly, mobile online. The oversampling bias presented in each mode lends support that mixed-mode approaches do reach different types of respondents. One mode is not sufficient to reach a representative sample; when all modes are combined together, the amount of variance from Census data was less than each of the modes individually. Even though using a mobile online as the only mode of data collection is fraught with challenges due to samples being generated from non-random, self-picked, convenience methods and shown to be markedly different from Census data, mobile online interviewing is well suited for point of transaction and customer provided sample lists. The collection of data through a mobile online device has the potential to decrease the amount of needed time for nearly all phases of the research process. The traditional restrictions for responding to a survey imposed on time or space by static location devices (desktop internet connection or landline) is not true with the mobile online. Data that is captured can be closer to the point of interest and data can be instantly rendered in dashboards without someone behind the scenes to upload the data.

20

References Baines, A. D., Partin, M. R., Davern, M., & Rockwood, T. H. 2007. Mixed-mode administration reduced bias and enhanced poststratification adjustments in a health behavior survey. Journal of Clinical Epidemiology, 60, 1246-1255. Bowling A. (2001). Research methods in health. Investigating health and health services. Buckinghamshire: Open University Press.

Bowling A., Bond M., Jenkinson, C., & Lamping, D. (1999). Short Form-36 (SF-36) Health Survey Questionnaire: Which normative data should be used? Comparisons between the norms provided by the Omnibus Survey in Britain, The Health Survey for England and the Oxford Health and Lifestyle Survey. Journal of Public Health Medicine, 21, 255–270.

Brambilla, D. J., McKinlay, S.M. (1987). A comparison of responses to mailed questionnaires and telephone interviews in a mixed mode health survey. American Journal of Epidemiology, 126, 962-971.

Davis, R. (1999). Web-based administration of a personality questionnaire: Comparison with traditional methods. Behavior Research Methods, Instruments, and Computers, 31(4), 572–577. Galobardes, B., Sunyer, J., Anto, J. M., Castellsague, J., Soriano, J. B., Tobias, A. (1998). Effect of the method of administration, mail or telephone, on the validity and reliability of a respiratory health questionnaire. The Spanish centers of the European Asthma Study. Journal of Clinical Epidemiology, 51, 875-881.

Groves, R. (1989). Response effects of the mode of data collection. Groves R (ed), Survey Errors and Survey Costs, New York: John Wiley & Sons, 501-579. Hox, J. J., & De Leeuw, E. D. (1994). A comparison of nonresponse in mail, telephone, and face-to-face surveys: Applying multilevel modeling to meta-analysis. Quality & Quantity, 28, 329-344. Lyons, R. A., Wareham, K., Lucas, M., Price, D., Williams, J., & Hutchings, H. A. (1999). SF-36 scores vary by method of administration: Implications for study design. Journal of Public Health, 21, 41–45.

McHorney, C. A., Kosinski, M., & Ware, J. E. (1994). Comparisons of the costs and quality of norms for the SF-36 Health Survey collected by mail versus telephone interview: Results from a national survey. Medical Care, 32, 551–567. Michalak, E. (1998). The use of the Internet as a research tool: The nature and characteristics of seasonal affective disorder (SAD) amongst a population of users. Interacting with Computers, 9, 349–365.

21

Nagelhout, G. E., Willemsen, M. C., Thompson, M. E., Fong, G. T., van den Putte, B., & de Vries, H. (2010). Is web interviewing a good alternative to telephone interviewing? Findings from the International Tobacco Control (ITC) Netherlands Survey. BMC Public Health, 10, 351-361. O‟Toole, B. l., Battistutta, D., Long, A., & Crouch, K. (1986). A comparison of costs and data quality of three health survey methods: Mail, telephone and personal home interview. American Journal of Epidemiology, 124, 317-328. Parker, C., & Dewey, M. (2000). Assessing research outcomes by postal questionnaire with telephone follow-up. TOTAL Study Group. Trial of occupation therapy and leisure. International Journal of Epidemiology, 29, 1065–1069.

Pasveer, K., & Ellard, J. (1998). The making of a personality inventory: Help from the WWW. Behavior Research Methods, Instruments, and Computers, 30(2), 309–313.

Perkins, J. J., & Sanson-Fisher, R. W. (1998). An examination of self- and telephone- administered modes of administration for the Australian SF-36. Journal of Clinical Epidemiology, 51, 969–973.

Presser, S., & Stinson, L. (1998). Data collection mode and social desirability bias in self-reported religious attendance. American Sociological Review, 63, 137–145.

Pruchno, R.A., & Hayden, J. M. (2000). Interview modality: Effects of costs and data quality in a sample of older women. Journal of Aging and Health, 12, 3–24. Siemiatycki, J. (1979). A comparison of mail, telephone and home interview strategies for household health surveys. American Journal of Public Health, 69, 238–245. Smith, M., & Leigh, B. (1997). Virtual subjects: Using the Internet as an alternative source of subjects and research environment. Behavior Research methods, Instruments, and Computers, 29(4), 496–505. The Nielsen Company. (2011). Mobile Insights Survey results. Retrieved January 18, 2011, from http://blog.nielsen.com/nielsenwire/consumer/smartphones-to-overtake-feature-phones-in-u-s-by-2011/. The Pew Research Center for the People & the Press. (2006). The Cell Phone Challenge to Survey Research: National Polls Not Undermined by Growing Cell-Only Population. Retrieved January 27, 2011, from http://people-press.org/report/276/ Tourangeau, R., Rasinski, K., Jobe, J. B., Smith, T. W., & Pratt, W. F. (1997). Sources of error in a survey of sexual behavior. Journal of Official Statistics, 13, 341–365.

22

Tourangeau R, Rips LJ, Rasinski K. (2000). The psychology of survey response. Chapter 10: Mode of data collection. Cambridge: Cambridge University Press. Tourangeau, R., & Smith, T. W. (1996). Asking sensitive questions: The impact of data collection mode, question format, and question context. Public Opinion Quarterly, 60, 275–304. Vuillemin, A., Oppert, J. M., Guillemin, F., Essermeant, L., Fontvieille, A. M., Galan, P., Kriska, A. M. , & Hercberg, S. (2000). Self-administered questionnaire compared with interview to past-year physical activity. Medicine and Science Sports and Exercise, 32, 1119–1124. Weinberger, M., Nagle, B., Hanlon, J.T., Samsa, G. P., Schmader, K., Landsman, P. B., Uttech, K. m., Cowper, P. A., Cohen, H. J., & Feussner, J. R. (1994). Assessing health-related quality of life in elderly outpatients: Telephone versus face-to-face administration. Journal of the American Geriatrics Society, 42, 1295-1299. Weinberger, M., Oddone, E. Z., Samsa, G. P., & Landsman, P. B. (1996). Are health-related quality of life measures affected by mode of administration? Journal of Clinical Epidemiology, 49, 135–140. Wu, A. W., Jacobson, D. L., Berzon, R. A., Revicki, D. A., van der Horst, C., Fichtenbaum, C. J., Saag, M. S., Lynn, L., Hardy, D., & Feinberg, J. (1997). The effect of mode of administration on medical outcomes study health ratings and EuroQol scores in AIDS. Quality of Life Research, 6, 3-10.

23

Appendix A

Thank you for taking part in our short survey. Your answers will help the research industry better understand the use of mobile and landline based telephone use. QS1. Are you driving, operating heavy machinery, or in a place where it is unsafe for you to complete this interview?

1. Yes [THANK AND TERMINATE] 2. No

QS2. Are you at least 18 years old?

1. Yes 2. No [THANK AND TERMINATE]

QS3. Are you in a place where you can speak freely?

1. Yes 2. No [THANK AND TERMINATE]

Q1. Are you…? (Phone – RECORD BY OBSERVATION)

1. Male 2. Female

Q2. In what year were you born? (OPEN END NUMERIC RESPONSE) Q3. What state do you live in? Q4. What is your highest level of education? (DO NOT READ LIST)

1. High school degree or less 2. Associates (2-or 3- year) degree / Some college 3. College degree (4-year) 4. Post-graduate degree (for example, masters or post-doctoral) 5. Trade 6. Other

Q5. Do you have a…? (Multiple Select)

1. Landline telephone number at home for making calls. 2. Mobile/Cell phone 3. Another type of phone service like VoIP or Skype

Q5a. [IF Q5 = 1 and 2] Thinking about all the phone calls you make, do you make more calls with your cell phone or more calls with your regular home phone? (READ LIST)

1. More with cell phone 2. More with regular home phone 3. About equal, use regular phone and cell phone about the same

24

Q5b. [IF Q5 = 1 and 2] How likely are you to stop using your regular landline phone and switch instead to using only a cell phone? (READ LIST)

1. Very likely 2. Somewhat likely 3. Neither Likely nor unlikely 4. Not that likely 5. Not at all likely

Q6. [IF Q5 = 2] For the cell phone you use most often, who is your current cell

phone service provider? (READ LIST) 1. T-mobile 2. Verizon 3. AT&T (Cingular) 4. Sprint 5. Tracfone 6. Other

Q7. [IF Q5 = 2] Overall, how satisfied are you with this cell phone provider? (READ LIST)

1. Very satisfied 2. Somewhat satisfied 3. Neither satisfied nor dissatisfied 4. Somewhat dissatisfied 5. Very Dissatisfied

Q8. [IF Q5 = 2] Last month, which of the following items did you use on your cell phone? (Multiple select)

1. Send text message 2. Browse the internet 3. Download apps 4. Take picture, record audio or video 5. Access social networking site or blog 6. Listened to music or played games 7. Did not do any of these

Q9. Have you EVER used illicit substances or illegal drugs, your response will remain completely anonymous?

1. Yes 2. No 3. Prefer not to answer (DK/RF)

Q10. What type of technology did you use to complete this interview?

1. Landline telephone 2. Telephone capabilities on a cell phone 3. Mobile based internet 4. Computer based internet

25

Appendix B

Table 4. Marginals by sample source (Unweighted)

Sample Source

Interviewer-Directed Self-Directed

Total Landline Cell Phone

Online Access Panel

Mobile Online

Q1. Sex % % % % %

Male 40.5 33.0 53.5 46.5 31.2

Female 59.5 67.0 46.5 53.5 68.8

Q2. Age (years)

18-24 14.2 4.0 16.3 17.5 18.0

25-34 22.2 6.0 24.3 15.0 39.2

35-44 14.0 13.5 13.4 10.0 18.0

45-54 16.0 14.0 16.8 18.0 15.2

55-64 16.2 21.0 17.8 19.5 8.4

65-74 8.7 15.0 5.4 16.0 0.4

75+ 5.3 17.5 1.0 4.0 0.0

Refused 3.5 9.0 5.0 0.0 0.8

Mean 1960 (50) 1950 (60) 1962 (48) 1955 (55) 1970 (40)

SD 15.28 15.71 12.94 13.80 10.12

Median 1960 1950 1962.5 1953 1973

Q3. Region

South 37.3 34.5 40.6 37.0 37.2

West 23.1 24.5 25.2 20.0 22.8

Midwest 21.7 21.0 18.8 24.0 22.8

Northeast 17.8 20.0 15.3 19.0 17.2

Q4. Education

High school degree or less 34.0 37.0 36.6 31.0 32.0

Some college 32.3 27.5 32.7 35.0 33.6

College degree (4-year) 21.1 22.5 18.8 21.0 22.0

Post graduate degree 12.6 13.0 11.9 13.0 12.4

Q5. Phone type(s) in home

Landline 67.6 100.0 51.5 73.5 50.0

Cell Phone 89.9 73.0 100.0 84.0 100.0

Other 10.9 8.0 15.8 6.5 12.8

Q5a. Phone type(s) in home

More with cell phone 53.3 43.8 65.4 34.2 72.0

More with regular home phone 25.2 30.1 9.6 49.6 9.6

About equal with both 18.3 19.2 20.2 16.2 17.6

Refused 3.3 6.8 4.8 0.0 0.8

Q5b. Stop using landline

Very likely 16.1 11.0 23.1 7.7 24.0

Somewhat likely 20.1 13.7 26.0 17.9 24.8

Neither likely nor unlikely 10.4 4.1 4.8 18.8 14.4

Not that likely 19.7 23.3 12.5 19.7 21.6

Not at all likely 30.5 41.1 28.8 35.9 14.4

Refused 3.3 6.8 4.8 0.0 0.8

Q6. Current cell phone service provider

AT&T 33.2 25.3 30.2 28.6 43.2

Verizon 28.7 33.6 33.2 25.6 24.4

T-Mobile 10.6 6.8 9.4 13.7 11.6

Sprint 10.1 8.9 8.4 7.7 13.6

Tracfone 3.1 4.1 2.0 7.7 0.4

26

Other 13.6 21.2 14.4 16.7 6.4

Refused 0.8 0.0 2.5 0.0 0.4

Q7. Satisfaction with cell phone service provider

Very satisfied 52.9 55.5 49.0 48.2 57.6

Somewhat satisfied 33.0 30.8 36.6 32.1 32.0

Neither satisfied nor dissatisfied 6.5 7.5 5.0 11.3 4.0

Somewhat dissatisfied 4.7 4.8 3.0 6.0 5.2

Very dissatisfied 2.1 1.4 4.0 2.4 0.8

Refused 0.8 0.0 2.5 0.0 0.4

Q8. Cell phone uses

Send text messages 71.4 50.7 68.8 57.1 95.2

Take picture, record audio or video 65.4 43.2 62.9 49.4 91.2

Browse internet 50.8 22.6 38.6 29.8 91.2

Listen to music/play games 44.0 18.5 34.7 21.4 81.6

Download apps 38.6 11.0 28.2 15.5 78.8

Access social networking 36.8 13.0 27.7 20.2 69.2

Did not do any of these 21.3 41.1 18.3 36.3 2.0

Mean number of cell activities 2.8 0.8 2.1 3.5 4.1

18-34 4.0

35-54 2.9

55+ 1.1

Q9. Illicit drug use

Yes 20.9 10.5 15.3 25.5 30.0

No 76.3 86.5 82.7 73.5 65.2

2.8 3.0 2.0 1.0 4.8

27

Appendix C

Table 5. Marginals by sample source (weighted)

Sample Source

Interviewer-Directed Self-Directed

Total Landline Cell Phone

Online Access Panel

Mobile Online

Q1. Sex % % % % %

Male 48.4 49.4 47.3 49.1 50.4

Female 51.6 50.6 52.7 50.9 49.6

[Base] [852] [200] [202] [200] [250]

Q2. Age (years)

18-24 10.1 9.8 10.2 10.0 7.6

25-34 19.8 18.1 19.7 19.0 20.2

35-44 18.4 17.9 19.3 19.0 25.8

45-54 18.9 19.3 18.4 19.5 23.5

55-64 14.9 15.1 14.9 15.3 13.2

65-74 9.0 10.4 8.3 8.5 9.6

75+ 8.8 9.3 9.0 8.7 0.0

Refused 0.0 0.0 0.0 0.0 0.0

Mean 1963 (47) 1962 (48) 1964 (46) 1964 (46) 1967 (43)

SD 18.08 17.99 17.93 17.77 13.20

Median 1965 1963 1965 1964 1968

Q3. Region

South 36.5 34.1 38.1 35.1 38.3

West 23.5 23.0 21.9 24.5 23.0

Midwest 22.8 23.3 21.9 23.6 24.0

Northeast 17.3 19.5 18.1 16.8 14.7

Q4. Education

High school degree or less 45.1 46.8 45.5 44.9 53.0

Some college 28.8 27.7 29.8 28.3 24.5

College degree (4-year) 16.3 16.0 15.3 16.6 13.8

Post graduate degree 9.8 9.5 9.4 10.1 8.6

Q5. Phone type(s) in home

Landline 62.3 100.0 43.0 65.9 45.9

Cell Phone 84.4 66.0 100.0 76.1 100.0

Other 5.7 4.4 8.4 5.0 4.9

Q5a. Phone type(s) in home

More with cell phone 51.8 57.4 68.8 31.8 80.8

More with regular home phone 27.0 20.1 10.1 48.2 8.8

About equal with both 18.0 17.3 15.8 20.0 10.3

Refused 3.2 5.2 5.3 0.0 0.1

Q5b. Stop using landline

Very likely 35.0 31.2 51.0 24.6 38.0

Somewhat likely 15.1 14.0 27.9 5.8 16.9

Neither likely nor unlikely 19.9 17.1 23.0 18.8 21.1

Not that likely 9.4 6.9 5.6 19.4 10.4

Not at all likely 19.7 22.9 10.7 22.0 16.7

Refused 32.7 33.7 27.5 34.0 34.9

Q6. Current cell phone service provider

AT&T 32.1 25.4 26.9 31.1 38.3

Verizon 27.8 33.7 38.1 21.5 27.4

T-Mobile 10.5 8.7 8.9 14.0 10.0

Sprint 10.0 8.7 6.9 8.8 15.9

28

Tracfone 3.3 2.5 1.5 7.6 0.2

Other 15.5 21.0 15.4 17.1 8.3

Refused 0.8 0.0 2.3 0.0 0.0

Q7. Satisfaction with cell phone service provider

Very satisfied 51.9 51.8 49.7 48.3 56.1

Somewhat satisfied 34.4 34.0 36.3 31.6 32.8

Neither satisfied nor dissatisfied 5.9 7.4 3.8 10.5 2.8

Somewhat dissatisfied 4.5 4.1 1.8 5.2 7.0

Very dissatisfied 2.5 2.7 6.1 4.4 1.3

Refused 0.8 0.0 2.3 0.0 0.0

Q8. Cell phone uses

Send text messages 70.7 67.4 61.0 59.9 95.0

Take picture, record audio or video 63.2 53.6 55.3 50.5 93.4

Browse internet 50.0 32.4 34.1 31.3 92.9

Listen to music/play games 42.5 24.5 29.5 19.9 73.5

Download apps 37.2 13.3 25.7 14.2 68.0

Access social networking 35.2 14.1 26.8 19.8 57.4

Did not do any of these 20.9 27.7 21.3 32.7 2.2

Q9. Illicit drug use

Yes 21.9 16.5 13.0 24.5 30.1

No 74.9 80.8 84.7 73.3 64.4

Prefer not to answer 3.2 2.7 2.3 2.2 5.5

29

Appendix D Table 6. Odds ratios from logistic regression by sample source

Sample Source

Landline

Cell Phone

Online Access Panel

Mobile Online

Gender

Male 1

1

1

1

Female 1.57 * 0.43 *** 0.84

1.89 *** Age

18-24 1

1

1

1

25-34 1.11

0.95

0.42 ** 1.85

35-44 3.26 * 0.92

0.43 * 1.20

45-54 2.29

1.26

0.72

0.80

55-64 3.64 ** 1.44

0.79

0.35 **

65-74 3.58 ** 1.00

1.34

0.41 **

75+ 17.36 *** 0.29

0.38 *

Region

West 1

1

1

1

Midwest 0.77

0.78

1.42

1.00

South 0.84

0.90

1.36

0.95

Northeast 0.76

0.80

1.40

1.14 Education

HS degree or less 1

1

1

1

Some college or more 0.81

0.73

1.27

1.15

Type of Phone R has

Cell phone only -

1

1

1

Both cell phone & landline 1

0.41 *** 1.35

0.62 *

Landline only 2.76 ** -

2.35 * -

Other and cell phone or landline 0.62

1.18

0.54

0.71

Use of illicit drugs

Yes 1

1

1

1

No 1.59

2.05 ** 0.69

0.56 ** Constant -8.77 -9.37 3.39 -11.83 Percent Overall Correct 81.60

76.80

75.60

72.70

Model Chi-Squarea 247.15 *** 104.57 *** 56.27 *** 192.25 ***

Nagelkerke R Square 0.41

0.19

0.10

0.30 Note: *P≤0.05, ** P≤0.01, ***P≤0.001; The first listed item is the baseline and is represented in each model with a “1” with no

digits. An exception to this is in the landline model. Due to no landline sample having only a cell phone the baseline was changed to the second response item

30

Appendix E Table 7. Odds ratios from logistic regression for cell phone and landline respondents by sample source

Sample Source

Landline

Cell Phone

Online Access Panel

Mobile Online

Phone use

More with cell phone 1

1

1

1

More with landline 1.92

0.30 ** 4.27 *** 0.30 **

About equal, use both the same 1.18

1.02

1.39

0.69

Becoming cell phone only

Likely 1

1

1

1

Not likely 0.57 * 1.64 * 0.84

1.26 Service provider

T-Mobile 1

1

1

1

Verizon 1.91

3.23 * 0.34 ** 0.67

AT&T 1.13

2.15

0.36 ** 1.55

Sprint 1.31

2.56

0.25 * 1.43

Tracfone 1.18

2.14

0.86

0.30

Other 3.33 ** 2.91

0.27 ** 0.33 *

Satisfaction with provider

Satisfied 1

1

1

1

Dissatisfied 1.16

1.01

0.57 * 1.55 Constant -1.09 -1.60 -0.78 -1.68

Percent Overall Correct 71.8

79.2

77.7

73.9 Model Chi-Square

a 26.52 ** 29.44 ** 59.79 *** 53.409 ***

Nagelkerke R Square 0.08

0.09

0.18

0.16 n=447

Note: *P≤0.05, ** P≤0.01, ***P≤0.001; The first listed item is the baseline and is represented in each model with a “1” with no digits. An exception to this is in the landline model. Due to no landline sample having only a cell phone the baseline was changed to the second response item

31

Appendix F SSI - RDD Landline Sample Methodology I. Creation of the Random Digit Database SSI starts with a database of all directory-listed households in the USA. Using area code and exchange data regularly obtained from Telcordia and additional databases, this file of directory-listed telephone numbers is subjected to an extensive cleaning and validation process to ensure that all exchanges are currently valid, assigned to the correct area code, and fall within an appropriate set of ZIP Codes. Most SSI samples are generated using a database of “working blocks.” A block (also known as a 100-bank or a bank) is a set of 100 contiguous numbers identified by the first two digits of the last four digits of a telephone number. For example, in the telephone number 203-567-7200, “72” is the block. A block is termed to be working if one or more listed telephone numbers are found in that block. Each exchange is assigned to a single county. Nationally, about 72% of all exchanges appear to fall totally within single county boundaries. For those overlapping county and/or state lines, the exchanges are assigned to the county of plurality, or the county with the highest number of listed residents within the exchange. This assignment ensures known probabilities of selection for all telephone numbers. II. Sample Stratification All SSI samples are generated using stratified sampling procedures. Stratified sampling divides the population of sampling units into subpopulations called strata. A separate sample is then selected from the sampling units in each stratum. SSI‟s database has been stratified by county. Measure of Size (MOS) Weights Prior to sample selection, the sample is allocated proportionally across all strata in the defined geography using one of several frame adjustment options. The sampling frame determines the way a sample is distributed across geography at the county level. SSI offers five different measure of size (MOS) stratification frames for its random digit samples: • Estimated Number of Telephone Households Estimates for telephone households are updated annually. The estimates are calculated by subtracting Census non-telephone household counts from current household estimates. Sample units will be allocated to each county in proportion to its share of telephone households. Estimated telephone households is the most commonly

32

used sampling frame for Random B samples. These estimates do not take into account those households that have only wireless telephone service since estimates of wireless-only households are currently not available for states or counties. • Number of Households Estimates for households are updated annually. Sample units will be allocated to each county in proportion to its share of households in the defined geography. • Total Population Estimates for population are updated annually. Sample units will be allocated to each county in proportion to its share of population in the defined geography. • Total Active Blocks Sample will be distributed by county in proportion to the total eligible blocks in the exchanges assigned to that county. Rather than being an estimate of target population, all frame units are represented with equal probability across counties. The number of eligible blocks in an exchange is multiplied by 100 (the number of possible 10-digit telephone numbers in a block) to calculate the total number of possible phone numbers. Sample will be allocated to each county in proportion to its share of these possible 10-digit telephone numbers. This is the recommended frame for apportioning Random A samples. • Other Sample allocation may also be based on special frames which may or may not result in equal probability samples. Such frames may be user-defined or based on incidence estimates and may be used singly or in combination with these or other sampling frames. Samples are first systematically stratified to each county in the survey area in proportion to the sampling frame selected. After a geographic area has been defined as a combination of counties, the sum of the estimated telephone households or requested frame value is calculated and divided by the desired sample size to produce a sampling interval and determine the amount of sample to be allocated to each county in the sample. The counties are ordered by alphabetical state and county within state. A random number between zero and one is generated and multiplied by the sampling interval to calculate a random starting point between one and the sampling interval. A cumulative count of elements is calculated. At the point at which the accumulation reaches the random starting point, a specific county is selected and the next sampling point is one interval away. Accumulation continues in this fashion until the entire sample has been apportioned. Sampling Frame Adjustments • Minimum Acceptable Block Size

33

Approximately 2.8 million 100-blocks are identified as working (having one or more listed numbers). By raising the minimum acceptable block size from 1 to 3 (SSI‟s default) or more, further gains in efficiency can be achieved with only minimal reduction in coverage. Blocks with 1-2 listed numbers represent 7.4% of all working blocks but only 0.4% of all listed telephone numbers. These listed numbers are far more likely to be keying errors or White Page business listings than the only listed number in a given block. SSI uses a default minimum block size of 3 listed numbers, but this minimum may be adjusted up or down based on the user‟s specifications. Users can even sample from blocks with zero listed numbers, but efficiency may fall as low as 10%. III. Sample Selection After the sample has been allocated, three methods of systematic sample selection are available. Random B is an SSI term denoting samples of random numbers distributed across all eligible blocks in proportion to their density of listed telephone households. All blocks within a county are organized in ascending order by area code, exchange, and block number. Once the quota has been allocated to all counties in the frame, a sampling interval is calculated by summing the number of listed residential numbers in eligible blocks within the county and dividing that sum by the number of sampling points assigned to the county. From a random start between zero and the sampling interval, blocks are systematically selected in proportion to their density of listed households. Once a block has been selected, a two-digit number is systematically selected in the range 00-99 and is appended to the exchange and block to form a 10-digit telephone number. Random A is an SSI term denoting samples of random numbers systematically selected with equal probability across all eligible blocks. All blocks within a county are organized in ascending order by area code, exchange, and block number. Once the quota has been allocated to all the counties in the frame, a sampling interval is calculated for each county by summing all the eligible blocks in the county and dividing that sum by the number of sampling points assigned to the county. From a random start between zero and the sampling interval, blocks are systematically selected from each county. Once a block has been selected, a two-digit number is systematically selected in the range 00-99 and is appended to the exchange and block to form a 10-digit telephone number. EPSEM Samples (Equal Probability of Selection Method) are single stage, equal probability samples of all possible 10-digit telephone numbers in blocks with one or more listed telephone numbers. The Working Phones Rate (WPR) for an EPSEM sample is on average 32%, but can range from 20-60% depending on the size and nature of the geographic area and local telephone number assignment practices.

34

EPSEM sampling uses a total active blocks frame and Random A sampling methodology. A sample of random numbers is systematically selected with equal probability across all blocks containing one or more listed numbers, which distributes the sample across counties in proportion to their share of total active blocks. EPSEM samples have the following characteristics: • Minimum block size is 1. • Business numbers cannot be replaced, but can be flagged. • Number protection is unavailable. IV. Selection Options SSI‟s database and sampling software support a variety of other EPSEM and non-EPSEM sampling options designed to accommodate different sample specifications or study objectives: Random B ... provides the most efficient random digit sample available. Each exchange and working block will have a probability of selection equal to its share of listed telephone households in eligible blocks. Numbers are protected against reuse for a period of six months. Business numbers are eliminated. Random A ... provides an extremely representative random digit sample. Each exchange and working block will have a probability of selection equal to its share of eligible working blocks. As an option, numbers selected in the past six months may be eliminated. (See Number Protection for further details.) Also as an option, business numbers may be eliminated. EPSEM (Equal Probability Selection Method) ... every possible telephone number (including business and protected numbers) in a working block with at least one directory-listed telephone number has an equal probability of selection. Telephone numbers are not protected. Business numbers may be identified, but not removed. Business Number Purge To improve efficiency, SSI maintains a database of over 11 million business telephone numbers, compiled from Yellow Page directories and special directories (Standard & Poor‟s and industry specific directories). Once a 10-digit telephone number has been selected for a sample, the status of the number generated may be compared to SSI‟s list of known business numbers. If the RDD number matches a known business listing, two options are available: Replace the number with the next number that is not a known business number. This is a non-EPSEM procedure but ensures that the requested sample size is met. In order to prevent introducing additional sampling bias, this procedure operates within strict limits. During either Random A or B sample selection, the search will not go beyond the boundaries of the selected block.

35

Select the number but flag it as a business number. This option preserves EPSEM sampling. Business numbers selected and flagged may be included as part of the sample or removed. If these numbers are retained in the final sample file, they may be sorted to the bottom of the file or the bottom of each replicate. If these numbers are removed, the final sample file will fall short of the requested sample size. Number Protection Virtually every SSI Random Digit Sample is marked on the database to protect against reuse for a period of six months. The SSI Protection System was designed to reduce the chance of selecting the same number for multiple projects or multiple waves of a single project conducted by a single research firm or by competing research firms. Incorporating number protection during sample selection is only an option. Once a 10-digit telephone number has been selected for a sample, the “protected” status of the number selected is checked. If the number has not been selected for a sample in the previous six months, the selected number is marked as “protected” and sampling continues. If the number is identified as having been selected for a sample in the previous six months, two sampling options are available: Replace the number with the next number that is not a “protected” number. In order to prevent introducing sampling bias in areas which have been frequently sampled, this procedure operates within strict limits. In either Random A or B sample, the search for an eligible replacement will not go beyond the boundaries of the selected block. In the event that an eligible replacement cannot be found within these limits, the originally selected “protected” number will be taken. Select the number any way, preserving EPSEM sampling. Screen for Disconnected Numbers The SSI Sample Screening Service is a stand-alone, post-production process that identifies non-working or unassigned numbers, as well as modem and fax numbers in random digit telephone samples. It employs a new and proprietary technology that recognizes almost half of these numbers, thereby improving the effective working phones rate of random digit telephone samples by an average of 10-15%. Once these numbers have been identified, two options are available: Remove disconnects from the sample. If these numbers are removed, the final sample file will fall short of the requested sample size. Number removal may be exercised either before replication (sample will have equal sized replicates) or after replication (sample will have unequal sized replicates but each replicate will contain exactly the same “good” telephone numbers as it would have if the sample had not been screened).

36

Include disconnects as part of the sample. If these numbers are retained in the final sample file, they are flagged and may be sorted to the bottom of the file or the bottom of each replicate and printed on separate sample pages. (Refer to the SSI Sample Screening Service information sheet for more detail.)

37

Appendix G SSI Wireless Sample Methodology There are no sources for listed or assigned cell phone numbers. Therefore Survey Sampling‟s Wireless frame is not a list-assisted frame. Without directory-listed information, there is no way to truncate the frame to „active‟ prefixes and 100-blocks to improve efficiency. A. FRAME CONSTRUCTION SSI starts with the most recent monthly Telcordia TPM (Terminating Point Master) Data file. This is Telcordia‟s master file of NPA-NXX and Block-ID records for the North American Number Plan. It contains at least one record per NPA-NXX. For prefixes (NPA-NXX‟s) where 1000-block number pooling is in effect, this file also provides information for individual 1000-blocks. This allows users to identify those 1000-blocks that have either not been assigned for service or that have been allocated to different service providers. 1. Selection of wireless prefixes and blocks: From this file we select all records (NPA-NXXs and/or 1000-blocks) where the NXXTYPE is: 04 Dedicated to Cellular 55,60 Special/Selective Billing Option – Cellular 65 Miscellaneous Service (non-500 PCS) 67,68 Special/Selective Billing Options – PCS 58,63 Special/Selective Billing Options - Cell, paging,mobile 50 Shared 3 or more POTS, Cellular, Paging, etc. 54 Shared POTS and Cellular 66 Shared POTS and PCS (non-500) and Dialable Indicator is 1 (yes, dialable by subscriber) Point ID is 48 non-protected, Alaska and Hawaii Change Code is not Z (deleted but preserved for billing) or D (Deleted) • New exchanges/1000-blocks are included in the frame. Analysis suggests that almost 80% of these records have an activation date on or before the release date associated with the release date of the TPM file. • Dedicated US Mobile (IMTS) and Paging services will not be included. However numbers belonging to paging or IMTS services may be encountered in „mixed‟ or „shared‟ blocks selected above. 2. File expansion: Exchange records for which there were no 1000-block records are expanded to ten (10) 1000-block records. This file is then further expanded to a file of 100-

38

blocks, 10 100-block records per 1000-block containing the same information as the 1000-block record. “Mixed” or “shared” 100-blocks (NXXTYPES 50, 54, 66) are then compared to Survey Sampling‟s list-assisted RDD database. 100-blocks with no listed numbers are retained in the wireless frame and 100-blocks containing listed numbers on the RDD frame are removed. The result is a frame of 100-blocks that is mutually exclusive of our list-assisted RDD frame while allowing coverage in prefixes and 1000-blocks that potentially provide both wireline and wireless service. 3. Telcordia Information: • OCN (Operating Company Number) for the 1000-block is retained on each record. • The Carrier Name has been attached to each record based on a Telcordia table of OCN codes and names. Carrier Names and codes (OCN) frequently do not reflect mergers and acquisitions. These acquisitions may be national in scope or state specific. For example, Southwestern Bell Wireless and parts of Pacific Bell Wireless were bought by Cingular but still reflect the original carrier name and OCN. Carrier Names and codes are frequently different by state, particularly for large Wireless companies offering national or almost national service. These companies also frequently distinguish (different corporate entities) between their wireline, cellular and paging divisions. Therefore a unique OCN will not always represent a unique carrier and visa versa. The carrier identification is for the 1000-block and will be inaccurate for subscribers who have ported their cell phone number to a different cellular provider. • Exchange Type (NXXTYPE). • Billing Center (Exchange Place Name) associated with the prefix. • Billing State for Billing Center of the prefix or exchange. • Change Code. As described above most New “N” prefixes are likely to be populated at the time a sample is selected. However, clients may optionally exclude 100-blocks with a Change Code of “N” in the interests of improved efficiency or receive the Change Code in the keyline of the sample for informational/analytical purposes. 4. SSI Information: • FIPS state and county codes are appended to each 100-block based on the billing coordinates of the prefix. Since wireless exchange boundaries are significantly larger than most wireline exchanges, many counties will not be represented in this file or may under-represented. • Additional FIPS based geography codes (Census Region, MSA, etc.) may be added to sample records based on this FIPS code. • Alpha Time Zone • State Abbreviation (based on FIPS Code and may be different from Billing State.

39

• Shared Flag easily identifies 100-blocks that have NXXTYPE = 50, 54 or 66. Clients may optionally include or exclude 100-blocks with a Shared Flag of “Y” or receive the Shared Flag in the keyline of their sample for informational or analytic purposes. o “Y” = Block is in a „shared‟ prefix/1000-block o “N” = Block is not in a „shared prefix/1000 block B. SAMPLE SELECTION The file of 100-blocks is sorted by FIPS code, Carrier name and 100-block. The intent is to provide a stratification that will yield a sample that is representative both geographically and by large and small carrier. A sampling interval is determined by dividing the universe of eligible 100-blocks by the desired sample size. From a random start within the first sampling interval, a systematic nth selection of 100-blocks is performed and a 2-digit random number between 00 and 99 is appended to each selected 100-block stem.

Replication is available as Number of Replicates or Size of Replicate. De-duplication against prior samples is available. Duplicates can be detected and substituted for during the random number generation process (nonprobability method) or may be removed after the sample has been selected (epsem method). In the latter instance, the sample delivered may be short. Screening for disconnects is not available for wireless sample since it would violate TCPA rules prohibiting dialing wireless numbers using “automated” telephone equipment. The sample is delivered in standard sort: Replicate, TZ, Phone number. A variety of count reports for the sample are available. Universe information is available upon request. Wireline numbers that have been ported to wireless service will not be included since they are on the RDD frame. However, they can be flagged and included in a landline sample. Wireless numbers that have been ported to landline service will be included and can‟t be identified.

40

Appendix H

SSI’s Panel Management Survey Sampling International (SSI) understands the importance of representation and proper sampling techniques in the research process. Sampling is our core competency – SSI‟s sole focus. SSI‟s panel recruitment, management, and usage practices are designed to deliver the highest quality sample for research.

SSI‟s processes for recruiting, screening, and accepting prospective panelists are designed to insure the best possible representation and the highest measure of panelist integrity. Panelists are recruited through thousands of web sites to optimize the probability that the panel reflects the overall composition of specific segments of the online population (close to 70%). SSI does not concentrate panelist recruitment among a limited number of sources (i.e., a single ISP, portal, or membership in a loyalty program).

SSI‟s recruitment policy is broad in scope but is not an “open door” methodology. SSI‟s recruitment methodology combines this broad scope, which leads to the best representation of the online population, with multiple quality checks at the join phase, a rigorous panel management practice, and programmatic and other panel hygiene steps. SSI‟s quality approach to panel management insures that the panel includes only “fresh” engaged respondents while at the same time purges hyperactive respondents, and panelists determined to be cheaters, frauds, or duplicates.

SSI takes many proprietary steps to ensure the quality of our online panel. SSI‟s panel cleaning processes include the following steps: • Panelists unable to provide a valid postal address based on the most recent US Postal Service data base are removed. • Duplicate panelists based on e-mail address matching and name and address matching are removed. • Potentially fraudulent or duplicate panelists are identified via the use of proprietary techniques such as systematic checks of passwords and other data items. • Suspicious panelists based on constant input/ feedback from clients where panelists may be satisficing, fatiguing, or delivering inconsistent answers are removed. • Data consistency checks are used to identify panelists who have given incorrect or contradic tory information. Where possible, for cases of genuine error, the data is corrected; in other cases the panelist is removed. • Panelists who don‟t respond to surveys for a predetermined period are removed.

41

• Hyperactive panelists are rested. Independent measures of panelist integrity consistently demonstrate that SSI outscores alternatives in the quality of panelist responses.

42

Appendix I

DMS River Sample Methodology -Information concerning DMS is from a a PowerPoint they provided

• The Original River Methodology • A constant stream of non-professional survey participants is

generated through real-time online promotions • Our “RDD for the Web” approach engages respondents who would

not otherwise participate in online research • The result is a “freshness” of general online users that minimizes

the potential for bias from exposure to prior surveys • Representative Sample

• Sample is drawn from the AOL and Platform-A networks, visited by 90% of U.S. HH Internet users with a reach of 170MM unduplicated monthly UVs

• Site demographics are comprehensive, and ideal for consumer markets

• Sample Consistency • Methods and processes produce repeatable, reliable, high-quality

samples • Proven year over year stability – validated by clients since 1996

• Key results: • On a monthly basis, approx. HALF of all visitors to the area are

NEW • The area is continually promoted, providing fresh sample

unseen in panels and purportedly similar river methods

43

• 69% of the visitors to Opinion Place do not belong to another panel (and never have)

• 59% of respondents report they don‟t participate in other online surveys

• Broad sample combined with strong systems • Survey completion rates of 85%+ deliver the highest quality data

• Quality is achieved by minimizing non-response bias and maximizing completion within your sampling frame

While the source of sample is critical, how that sample is managed is just as important. Our approach:

• Identity Locks • Verify the identity of all respondents through public and proprietary

databases • Employ “digital-fingerprinting” for identifying the computers people

use • Usage Locks • Lockout to Opinion Place if survey taken: 7 Days • Lockout to Opinion Place if no survey taken: 3 Days • Lockout to screening for a research category: 90 Days Minimum • Screening questions are randomly served to only those qualified,

limiting exposure • Lockout to survey: No repeat. Some trackers choose to „unlock‟

after 12 months • Data Locks • Upon validation and site entry, data traps are set throughout

screening and surveys ensuring undesirable respondents are locked out of your data