Somani Patnaik1, Emma Brunskill1, William Thies2

1 Massachusetts Institute of Technology2 Microsoft Research India

Accuracy of Data Collection on Mobile Phones: A Study of Forms, SMS, and Voice

Mobile Data Collection • Broad applications in

––––

Mobile bankingMicrofinanceHealthcareEnvironmental monitoring

• Benefits Immediate digitization Fast and cheap Environment friendly Flexible questionnaires

Study of data collection interfaces• Comparison of three interfaces for health data collection

1. Electronic Forms Interface

1. Electronic Forms Interface

• General Strengths Easy patient identification Ongoing cost is low (SMS or data plan) Can store visits when connectivity is poor

• General Weaknesses Requires programmable phones Requires basic literacy skills Hard to alter survey questions Hard to enter in free-form notes Application can be deleted by user

• Consist of numeric fields and multiple-choice menus.• Can be implemented in Java or a native phone platform.

2. SMS Interface

2. SMS Interface• Sending a structured SMS messages to a server• Logical fields separated by delimiters in the message• General Strengths

Can be used with any phone Ongoing cost is low (SMS or data plan) Many workers familiar with SMS

• General Weaknesses Requires basic literacy skills Changing survey requires new cue card Quite easy to fake visits (copy old SMS) Hard to enter in free-form notes

3. Live Operator Interface

3. Live Operator Interface• A normal telephone call• Live human operator that enters the data into a centralized spreadsheet• General Strengths

No literacy required of workers Can be used with any phone Hard to fake a visit: operator can ask new questions

• General Weaknesses Ongoing cost of operator salary Voice plans often higher cost than SMS Awkward 3-way social interaction

Operations Setup time - Electronic forms require application,

which requires either an Internet-enabled phone or an external computer

Training time – Voice interface requires least amount of education and background

System coverage and reliability - voice most reliable but require sufficient number of operators.

Flexibility - Ability to modify the data collection interface, fix an error, improve usability etc

Effectiveness•Characterized by two factors-

Data should not be intentionally faked by the user The data should be accurate(not intentionally faked)

• Unfortunately, quite easy to fake data in SMS systems, just requires copying and pasting prior messages • Faking data on electronic systems slightly harder and most difficult in voice.

• Voice also allows correcting previous visits

Cost Comparison• Electronic phone requires a programmable phone(such as a Java enable phone or windows phone)

• Voice has the ongoing cost of the operator

• Cost of live-operator in India proves to be cost-effective

• Decreased cost of voice-only handsets, training time and literacy requirements of health workers compensate the cost

Cost Analysis

..Cost Analysis

Context: Rural Tuberculosis Treatment• With local partners, working to improve tuberculosis treatment in rural Bihar.

• Monitoring the patients symptoms remotely by collecting data remotely.

Study Participants• 13 health workers and hospital staff (Gujarat, India)

Age(Median)

Education Cell PhoneExperience

• Training Health workers: big groups, 6-8 hours Hospital staff: small groups, 1-2 hours Every user made two error-free reports on each interface

Testing and Results•Testing

Tested in pairs, one patient on data entry, and the other being the fake patient

Two complete patient–worker interactions for electronic forms and SMS interfaces

•Results

Results

• Error rate higher in health workers as compared to hospital staff

• Two possible reasons: Hospital staff more educated and old Difference in training

Sources of ErrorTypes of errors

• Misplacement of decimal point in the temperature entry • Non-revision of cue cards for the SMS interface• Putting wrong patient identity when using SMS

Usability barriers

• small keys • scrolling/selection• SMS encoding

Conclusions•Accuracy of mobile data collection demands attention

5% error rates for those lacking experience

• There exist cases where a live operator makes sense Can be cost effective, esp. for short calls

• Study has limitations Small sample size Varied education, phone experience, training of participants

• Future work Distinguish factors responsible for error rates Compare to paper forms, Interactive Voice Response