Phone Surveys for Health: Methods and Opportunities

Abigail Greenleaf, PhD, MPHMay 5, 2020

Overview

1. Cell Phone Ownership in Sub-Saharan Africa (SSA)2. Modes of Data Collection3. Data Collection Goal4. Total Survey Error Framework Applied to Cell Phone Surveys in SSA

• Frame • Sampling• Response • Post- Adjustment

5. Wrap-up6. Questions

• The first cell phone call in SSA was in 1987 in DRC (formerly Zaire)

• Mobile phone penetration rate: % of unique users within a population

• In 2018, SSA had 44% (or 456 million) unique subscribers

• Looking at penetration by region, growth will be equal but West Africa has highest current penetration (48%) and Central Africa has lowest (40%)Source: GSMA 2019

Cell Phone Ownership in SSA: Past & Present

• SSA continues to be the fastest growing region for cell phone ownership, but the Compound Annual Growth Rate (CAGR) is slowing

• Mobile phone penetration: 50% by 2025

• Next 5 years: additional 167 million subscribers with half of new subscribers coming from 5 countries:

1. Nigeria (31 mil) 2. Ethiopia (18 mil)3. DRC (15 mil)4. Tanzania (10 mil)5. Kenya (9 mil)6. Others (84 mil)

Mobile Phone Penetration in SSA, 2018

Source: Greenleaf 2018

Cell Phone Ownership in SSA: Future

Evolution of Cell Phone Data Collection Studies

• Likely 1st reported cell phone survey in SSA (Dillon, 2010):• Tanzania: Data collection (Sept 2009-Jul 2010) among 195 cotton farmers in 15 villages

sampled from village registry• Face-to-Face Enrollment, Live Interviewer Follow-up (World Bank, 2012-2015):

• Tanzania, Mali, South Sudan, Listening to Africa (Face-to-face [FTF] enrollment, Computer Assisted Telephone Interview [CATI] follow-up)

• Random Digit Dial: • Cote D’Ivoire, 2013: CATI with call center in France (Larmarange, 2016) • Zimbabwe, Afghanistan, Ethiopia, Mozambique: Interactive Voice Response [IVR] (Leo,

2015)• Literature Reviews: Articles from low and middle-income countries

• Population-based cell phone surveys: 11 articles, 9 used CATI (Gibson, 2017)• Mode comparison: 10 articles, (1 article overlap with Gibson) (Greenleaf, 2017)

SMS: Short Message Service

Strengths Weaknesses• Software automates data collection so human oversight of data collection is not needed• Respondent can answer when convenient• Message sends when respondent is back in network coverage

• Requires literacy • Texting not a ubiquitous

activity• Keypad-based response

menu can be difficult to navigate or impossible to use if low literacy• Inboxes get full quickly on

phones with limited memory• Limited to 160 characters

• Difficult to judge data quality

• Generational differences in texting: In Nigeria, 89% of cell phone owners, ages 18-34, use SMS compared to 67% of owners, ages 35+ (Pew, 2015)

• SMS Random Digital Dial in Ghana, Kenya, Uganda, and Nigeria: response rate < 1% and young men overrepresented (Lau, 2019)

• Emerging literature: South Africa, Kenya

Receives more SMSOpt-in

Selects preferred language

Introduces survey

IVR: Interactive Voice Response

• IVR: Respondents listen to a pre-recorded question, then answer by pressing the number corresponding to their answer on their phone keypad

• Hybrid IVR: begins with live interviewer

Strengths Weaknesses• Recorded questions reduce interviewer bias • Software automates data

collection so human oversight of data collection is not needed• Rapid implementation

• Respondent may not be familiar with pre-recorded calls which decreases response rate• Keypad-based response menu can be difficult to navigate or impossible to use if low literacy • Difficult to judge data

quality (i.e. respondents understanding, or if respondents was distracted)

Source: Abigail Greenleaf 2017 (Top)Source: Abdoul Djalil, 2020 (Right)

CATI: Computer Assisted Telephone Interview

Strengths Weaknesses• Live operator, familiar

interaction• Good for low literacy

settings• Operators can clarify

questions• Interviewer builds

relationships with respondents thus reducing refusals and attrition

• Resource intensive (interviewers, supervisors)• Interviewer bias

Source: Greenleaf 2017

Comparison of Modes for Health Surveys

• Representation: • Tanzania (Pariyo, 2019): Female respondents, those with lower levels of schooling and

who are rural residents, have higher participation rates for CATI compared to IVR• Also found in Nigeria (Lau et al.) and among women in Burkina Faso (Greenleaf et al.),

CATI produced more representative sample than IVR • Cost:

• Nigeria (Lau, 2019): For a survey of 3,000 completes, IVR is 43% the cost of CATI, and SMS is 24% the cost of CATI

• Burkina Faso (Greenleaf, 2019.):

CATI Hybid IVRInterview Length 6 minutes 13 minutesCost per completed survey $52 $144

Follow-up to FTF Survey among Women, Burkina Faso, 2017

Source: Greenleaf et al., PMA2020, 2019

Data Collection Goal

Surveillance• Health care workers: e.g. automated calls with

community health care workers in Niger• Populations: Ebola in Liberia, Sierra Leone; chronic

health conditions in Tanzania, Uganda; ‘the cholera phone’ in Bangladesh

Monitoring and Evaluation

Surveys• Sampling: RDD or FTF follow-up

Source: ICAP 2014

Total Survey Error

Research Question: Have tigers in Thailand started wearing face masks to reduce

transmission of COVID-19?

Source: Groves, 2009 Source: WWF

Measurement Representation

Survey Statistic

Construct

Measurement

Response

Edited Response

Validity

Measurement Error

Processing Error

Target Population

Sampling Frame

Sample

Respondents

Postsurvey Adjustments

Coverage Error

Sampling Error

Nonresponse Error

Adjustment Error

Coverage Error

• Target Population • The group about whom you want to make an

inference

• Sample Frame• List of everyone in your target population• If you do not have a cell phone, you cannot be on

the sample frame

• Coverage error• Those included in the target population and

sample frame are different

• Types of Coverage Error • 4 types but undercoverage concerning for cell phone

surveys• Undercoverage: units that should be in the sample

frame are not included

Sample Frame

1

2

3

4

5

6

7

8

9

10

Target Population

Frame

Source: WWF

Coverage Error: Who Owns Cell Phones?

Broadly speaking, owners are:Cell phone ownership data sources:• Surveys:

• PHIA (HH and individual); • Demographic and Health Surveys (HH and female

ownership; • PMA2020 (HH and female ownership);• Afrobarometer (HH ownership)

• Private Sector and Governmental: • GSMA• International Telecommunications Union

Young (young women less likely than men)

Men Educated

Urban

Peer-Reviewed Literature on Phone Ownership

• Mozambique (Velthausz, 2016): • Of the population in four Provinces that used a phone, 68% owned a phone, 32% had

access only.• 90% of follow-up respondents were phone owners

• Among female phone owners in Burkina Faso (Greenleaf, 2018):• 83% reported only one phone number• 14% had two numbers• 3% had three or more phone numbers• The mean cell phone numbers:1.2

• Countries with higher mobile penetration (Ethiopia, Zimbabwe) resulted in less sample distortion and more comparable estimates to recent FTF surveys compared to the countries with lower phone penetration (Afghanistan, Mozambique) (Leo, 2015)

Women and Cell Phone Ownership

• 2 districts in Southern Tanzania (Vasudevan, 2016)• 6% of women reported used a phone daily • 80% of women reported using a phone less than once a week• Urban women more likely (70%) to be sole owner compared to rural women (44%)

• Burkina Faso (Greenleaf, 2018) • 86% of households owned a cell phone and 47% of women reported personal cell phone ownership • Women with secondary education or higher had 4 times the odds of cell phone ownership compared

to women with no formal education • 5 States in Nigeria (Jennings, 2015)

• Women who did not have phone access had half the odds of reporting modern contraceptive use compared to women that had phone access

• GSMA 2020 Gender Gap Report• Barriers to mobile phone ownership for women in SSA

• Affordability• Literacy and Skills • Family does not approve

Coverage Error: Conclusions

• Compare the profile of phone owners and non-owners to see if your sample frame reflects your target population

• Look at ownership by gender• Cell phone owners are demographically distinct

from non-owners

• As cell phone ownership increases, so will representativeness of your study

• US started phone surveys in 1970s when ~80% of HH had landlines

• Reaching non-phone owners• FTF enrollment: give a cell phone • Pass the phone to non-phone owner• Half-interval open

Source: ICAP 2014

Sampling

Sampling: the process of selecting a subset of observations from an entire population of interest so that characteristics from the subset (sample) can be used to draw conclusion or making inference about the entire population

Sampled1

2

3

4

5

6

7

8

9

10

Target Population

Frame

Sample

Source: WWF

Random Digit Dial

• Creating or obtaining phone numbers to call • Use prefixes to randomly generate numbers (Gibson, 2017) • Possibly get active numbers from mobile network operator (MNO)• List of respondents who previously completed a survey

• Take into account market share each MNO has and sample accordingly

• MNO matters: having a premium cell phone provider, was significant predictor of participation in Tanzania (Croke, 2012)

• Quotas to ensure representative of hard-to-reach pops • Most RDD studies in SSA to date used IVR • RDD study using CATI to contact women in Burkina Faso

(Greenleaf, 2020)• Of the calls that were picked up 72% were men • To reach 1 rural respondent ages 15-19, ~ 30 picked-up calls Source: Aliou Gadiaga, 2018

Follow-up to Face-to-Face Survey

• World Bank gave phones and solar charges to respondents and achieved high response rates during a panel study (monthly contact)

• FTF survey serves as your sample frame

• Often see everyone in frame included

• Differential between those who are eligible (have cell phone number) and consent?

Source: Abigail Greenleaf, 2017

Sampling: Conclusions

RDD • “RDD is not suitable for context in which the target population has low coverage rates and the

aim is to create a representative sample” (Dabalen, 2016)• Quotas extend data collection significantly

• 7 additional days to fill 15-19 rural quota in Burkina Faso

FTF Enrollment • Follow-up as soon as possible after FTF enrollment• Over-sample groups that are harder to contact • Over-sample groups that are less likely to complete questionnaire

Non-Response

• Non-response error: Those who respond to the survey are different from those who do not respond

• Non-response in phone surveys generally stems from 3 causes: • Failure to contact sampled

respondents • Refusal to participate • Health or language limitations

Respondents

2

4

6

8

10

Target Population

Frame

Sample

Respondents

Source: WWF

Non-Contact

• Causes of non-contact in cell phone survey in SSA• Removing SIM from phone

• Mozambique: 39% of respondents for 4 provinces who were enrolled FTF and followed up with CATI swapped their SIM Card within the 4-month period, often to achieve better reception (Velthausz, 2016)

• Dead battery• Out of range of network service

• Quicker the follow-up post-contact the higher the retention

• Mozambique: 80% non-contact of random list (Velthausz, 2016)

• Burkina Faso: 1 year, 43% non-contact (Greenleaf, 2019)

Source: Wikimedia, Jorge Barrios

Refusal

• Trying to encourage participation among those that do own a phone (Gibson, 2017)

• Incentive• Amount, timing, structure

• Introduction • Phrasing, voice (gender), modality (CATI or IVR for

IVR)• Modality

• IVR or CATI

• Refusals – usually happens before consent• Burkina Faso follow-up CATI & Hybrid IVR (women only):

8% (Greenleaf, 2019) • Burkina Faso RDD CATI (women only): 3.1% (Greenleaf,

2020 • Ghana RDD IVR: 7% (L’Engle, 2018)• Tanzania CATI: 1.6%; IVR 0.3% (Pariyo, 2019)

Participant Concerns in Burkina Faso RDD

Where are you?

I don’t want to expose my life via telephone

Why is it *me* that you’re calling?

Why do you want to know my age?

Language Fractionalization

• Order of language options matters: • IVR Pilot in Burkina Faso (Greenleaf, 2019). When

least spoken language (Fulfulde) was listed first, 3 times as many people chose that language

• Results from 2015 IVR study • Zimbabwe: ~40% of the population speaks Shona,

while 76% of the sample selected Shona• Ethiopia: 85% of the sample selected Amharic, but

primary language for only an estimated 29%

Source: African Languages Interpreters

Response Wrap-up

• Challenge of distinguishing coverage error from non-response error • Frame unknown so don’t know if not included in sample or not responding

• Approaches to comparing respondent & non-respondents• DHS, Census, administrative data, other • Early versus late respondents

• Burkina Faso follow-up: survey respondents different from non-respondents for both CATI & Hybrid IVR, with greater sample distortion for Hybrid IVR (Greenleaf, 2019)

• Response Rates• Response rates are low for cell phone surveys

• U.S.: 9% • Response rates are difficult to standardize because the number of non-working phone numbers

is hard to discern (Lau, 2018)• Mode: CATI higher than IVR, Sampling: FTF higher than RDD

Non-Response: Conclusions

• A high refusal rate does not necessarily translate to biased estimates if the propensity to respond is not correlated with the outcome of interest

• Refusal should be randomly distributed within your sample for your estimate to be un-biased

• Try to collect or identify data about how your population interacts with their phone

• Ownership vs. access • Ask respondents for back-up phone numbers to

contact them • Frequency of phone use

Source: ICAP 2014

Post-adjustment Error

When post-survey adjustments to account for representation errors do not improve the accuracy of

the survey estimate

2

6

8

10

Target Population

Frame

Sample

Respondents

Post-adjustmentSource: WWF

Post-adjustment Error in SSA Cell Phone Surveys

• Nigeria: RDD comparing IVR, SMS with DHS• Outcome: voting estimate• Weighting did not improve estimate of voting

• Burkina Faso: CATI RDD among women compared with concurrent in-person survey

• Weights: age, education, residence• Did not improve modern contraceptive estimates

FTF Full sample

FTF Phone owners

RDD Unweighted

RDDWeighted

% using modern contraception

26.0 (22.7 – 29.6)

31.7 (30.0 – 35.6)

40.2(38.2 – 42.2)

38.7 (36.7- 40.8)

Source: Charles Lau, 2019

Source: Abigail Greenleaf, 2020

Representation in cell phone surveys

Cell phone surveys exclude people without cell phones, but often draw inferences to the general population1. Phone owners are male, educated, urban and

young and new ownership growth is slowing2. Phone owners are different from non-owners to

point that weighting hasn’t improved estimates. Due to social connections? Female empowerment? Other?

3. RDD sampling is challenging 4. Data on non-respondents is hard to come by but

refusals can be reduced by following good cell phone survey practice

Source: WWF

Practice Safe Cell Phone Surveys

• Be explicit about who and was not included in your survey! • Try to assess sources of representation error • Stratify results by gender

• Use AAPOR [call] Disposition Codes (AAPOR, 2016)

• CATI, when possible, is preferred to increase representation

• FTF follow-up preferred over RDD

• New frontiers while acknowledging current limitations • Mixed modes • WhatsApp

Recent Resources

• Survey CTOhttps://www.surveycto.com/blog/surveycto-cati/• JPAL https://www.povertyactionlab.org/blog/3-20-20/best-practices-conducting-phone-surveys• World Bankhttps://blogs.worldbank.org/impactevaluations/mobile-phone-surveys-understanding-covid-19-impacts-part-i-sampling-and-mode

Questions

Contact: arg2177@cumc.columbia.edu