tablets and smartphones - csdiworkshop.org€¦ · •mcapi survey in 15 cities using 7-inch...
TRANSCRIPT
![Page 1: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/1.jpg)
Sarah Hughes, Senior Survey Director, NORC
Sam Haddaway, Senior Research Analyst, NORC
Tablets and Smartphones:
adopting new technologies for
household surveys in Kenya
![Page 2: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/2.jpg)
2
Using Paper and Pencil
• Low tech
• Low cost
• High error rate
• No ‘checks’
• Delayed data
review
![Page 3: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/3.jpg)
3
mCAPI: Revolutionizing Data Collection
• Programmed
‘checks’
• Rapid data
review
• Interviewer
monitoring
• Easy
programming
• Small size, long
battery life
• Durable and
secure
![Page 4: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/4.jpg)
4 4
Using Tablets for Surveys
• 5 countries
• 150,000+
households
• 30,000
interviews
International Projects: Evaluations, Data Collections
![Page 5: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/5.jpg)
5
PROS CONS
• Much less lag between data
collection and analysis
• Fewer data entry errors, cleaner
data
• Rapid quex changes/fixes
• Rapid feedback to interviewers
• Cheaper (maybe)
• Less dependent on
subcontractors
• No case management (yet)
• Difficult to reconcile errors after
the fact - no paper trail
• Higher risk of stolen data and
harm to enumerators
• Difficult for supervisors to
review enumerators’ work
• Changes in work
processes/training
• Format limitations (tables, text)
• Pros outweigh cons for some, but not all projects
• When choosing mCAPI, does device affect data
quality?
mCAPI Pros/Cons Over PAPI
![Page 6: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/6.jpg)
6
• World Bank’s Kenya State of the Cities Survey
• mCAPI survey in 15 cities using 7-inch Android tablets
• Listed 194,000 households and administered 14,600
thirty-minute surveys (July 2012-March 2013)
• Selected two interviewers to administer 200
interviews using smartphones instead of tablets
• 50 interviews each in two cities (Nairobi and Thika)
• Compared quality of data collected using tablets
to data collected using smartphones
• Exact same UI on phones and tablets (designed for
tablets)
Assessing data quality by mobile device*
*Funded by a grant from the Center for Excellence in Survey Research
![Page 7: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/7.jpg)
7
• Interview Duration
• Item Missingness (Response data)
• Item Missingness (GPS Data)
• Valid Phone Numbers
• Other Findings
We found that our two interviewers differed greatly from
one another, so we present their results separately.
Tools for measuring quality
![Page 8: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/8.jpg)
8
• Mean interview durations:
(Jane showed no significant results tablets vs. phones, but
she showed significantly shorter durations overall)
Interview Duration
John (Nairobi) John (Thika)
Phones 25.8 40.2
Tablets 37.5 31.8
Difference NUMBERS NOT MAKING SENSE
![Page 9: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/9.jpg)
9
• Average number of missing responses (DK/R):
Jane performed worse on phones
Item Missingness (Responses)
John Jane
Phones 1.5 2.8
Tablets 1.4 1.7
Difference -0.12 -1.17***
![Page 10: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/10.jpg)
10
• Percent of interviews with GPS coordinates:
Difference only in Thika; probably due to mobile network
availability, GPS chip quality, software…
Item Missingness (GPS)
John Jane
Phones 99.0 97.7
Tablets 88.4 97.7
Difference -10.33*** 0.13
![Page 11: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/11.jpg)
11
• Average number of valid phone numbers:
Phones were worse everywhere: Due to typos? Physical
difficulty typing on phones? Respondent reluctance?
Valid Phone Numbers
John Jane
Phones 0.9 0.5
Tablets 1.1 0.7
Difference 0.14** 0.18***
![Page 12: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/12.jpg)
12
• Confidence and comfort in typing dependent on
past experience with device and touchscreens
• Interviewers felt more likely to accidentally select
options on phones
• Interviewers admitted to not scrolling completely
through questions/responses on phones
• Felt more professional with tablets
• Felt safer with phones
• Tablets attracted unwanted attention
Other Findings
![Page 13: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/13.jpg)
13
• Phones associated with more typing error
• Long open-ended questions, long numeric strings are
difficult
• Otherwise, phones and tablets affect different
interviewers differently
• Some observed differences probably due to
respondents’ perceptions rather than hardware
• Phones probably are not worse for all surveys,
but much more research is needed
Conclusions
![Page 14: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/14.jpg)
14
• Bigger study exploring more dimensions with
more interviewers, introduce randomization
• Review results by respondent characteristics,
sample characteristics, other?
• Observe human-computer interaction (HCI)
among different enumerators administering the
same survey
• Further research questions:
• Is there an “optimal” screen size? How is it determined
– age, gender, technical capacity, prior experience?
• What is the role of software?
Next Steps
![Page 15: Tablets and Smartphones - csdiworkshop.org€¦ · •mCAPI survey in 15 cities using 7-inch Android tablets •Listed 194,000 households and administered 14,600 thirty-minute surveys](https://reader033.vdocuments.us/reader033/viewer/2022050418/5f8e528f330adf6da83b972d/html5/thumbnails/15.jpg)
Thank You!