1

Innovations in LSMS data & implementation

Kathleen BeegleWorkshop 17, Session 3Designing and Implementing Household SurveysMarch 31, 2009

2

Overview

Testing: measuring ability through tests Computer Assisted Personal Interviewing

(CAPI) New health measures Impact evaluation Other new areas….

3

Testing (D. Filmer and N. Schady work in Cambodia) Students

Mathematics test Peabody Picture Vocabulary Test

Early Child Development Test de Vocabulario en Imagenes Peabody (TVIP) Woodcock-Johnson III Memory for Names Ages & Stages test for child development

Control measures of caregiver cognitive ability Progressive Matrices Test

4

Testing, cont. Appropriateness vs. Benchmarking

For example: Developing math test (need to ensure compatibility with

expectations/curriculum) Adapting vocabulary test (at what point is it no longer comparable?)

Translation Even more important to be precise

Training Familiarity with / mastery of materials Rapport with interviewees (esp. children for child development tests)

Implementation Quiet space

Cost Internationally comparable copyrighted tests expensive (TVIP/WJ)

5

Implementing tests

6

What is CAPI? Interviewing and data entry are brought together. No

paper questionnaires PDAs, mini-PCs (ultra mobile PCs) Software

Off-the-shelf products designed for linear questionnaires On-going development of software for complicated

questionnaire structures (like LSMS) Hardware can come with built in cameras, sound

recording, CPS, external keyboard, finger scanner……etc…..

Models to allow internet access through cell phone network

7

Why CAPI? Interviewing and data entry are brought

together Faster data: data are available as soon as

transferred (same day) More accurate data: quality improved through

validation checks during/immediately after interview Other advantages

Flexible questionnaire design Can embed listing, sampling, past data (panels), multi-

languages, instructions from manual

An example: UMPCs in Tanzania

• Ultra Mobile Personal Computer• Large touch Screen• Runs standard desktop operating systems• Multiple ports just like a desktop

Large screen allows you to see the whole picture

Whole sections are displayed and flow

maintained

Whole sections are displayed and flow

maintained

10

Rapid click navigation to other sections

You are about to finish an interview when a child walks in that the respondent had not mentioned before. With a UMPC layout you can get to the

census/ roster section in one click to add the details and continue your interview

You are about to finish an interview when a child walks in that the respondent had not mentioned before. With a UMPC layout you can get to the

census/ roster section in one click to add the details and continue your interview

Display of picture options for greater accuracy

Respondent says “a bunch of bananas”. With a large UMPC screen and storage you can display different sizes to choose from, calculate the correct energy value and therefore obtain accurate data (food,

medication, amount of production, size of asset...)

Respondent says “a bunch of bananas”. With a large UMPC screen and storage you can display different sizes to choose from, calculate the correct energy value and therefore obtain accurate data (food,

medication, amount of production, size of asset...)

12

CAPI Experience Adapting DE software is complicated. Through 3 surveys, gaining experience on

practicalities of this technology: security, remote support, battery life, etc…

Challenges of developing questionnaire UMPCs have some advantages over PDA:

Maintaining the best features of a well-laid out questionnaire: screen size and segueing to the next screen.

13

Collecting New Health Measures in LSMS/IS Surveys

Anthropometrics BP/pulse Observed physical ability Arm circumference

Hemoglobin (anemia) HIV/AIDS Malaria Avian flu STDs

Health measures collected multi-topic surveys to measure outcomes that are not reliably self-reported.

In the latter group, testing in a number of ways: laboratory testing of blood samples in-home testing (small blood sample from finger pin-prick in case of Hemoglobin, saliva for HIV) physical examination.

14

Collecting New Health Measures: Challenges Staffing. May require a trained nurse (VCT) to be

added to the field team. Cost/Management implications

Human Subjects Protocols. Internal Review Board (IRB) process: both within country and with other institutions Consent processes: verbal consent replaced with signed

forms. Data security: both within field teams and once electronic. Informing respondents of the test result. Obligations for

follow-up (ability to provide, implications for research).

15

Impact evaluation

LSMS is (usually) not designed for specific evaluation work, but can potentially be used for this

Usually requires coordination of the evaluation team with the national statistics office either before or after, since additional may be needed

16

Impact evaluation, cont. Nicaragua FISE evaluation (Social Fund)

Booster sample of FISE communities. 1998 LSMS provided adequate non-FISE communites (piggy-backing)

Oversample FISE households and use the LSMS survey data (i) as a backdrop for comparison for assessing poverty targeting and (ii) to construct a control group to assess impact

Malawi Fertilizer Voucher Evaluation (2006/07) Constructed a panel from the Malawi LSMS in 2004/05.

Re-interviewed households after the implementation of a new fertilizer voucher scheme.

17

Pros matching method can allow for

use of data collected during one time period to conduct an evaluation -- more rapid than method with baseline

good measures of targeting and impact (consumption-based poverty estimates; variety of outcomes including anthropometric measures, etc.)

more cost-effective than separate, in-depth survey

Cons somewhat lengthy

evaluation process use of LSMS

questionnaire limits ability to tailor questions to FISE Evaluation

cross-sectional surveys may not be intended to become panels (availability of re-contact information, names, etc..)

Pros and Cons of “Piggy Backing” on an LSMS

18

Extending an LSMS to other new areas New methods to extend the range of topics in an

LSMS and improve the quality of data.

Expectations of the future. Using stones/beans to illicit expected probabilities of

respondents E.g. Pastoralists with little formal education asked about

expectations of rainfall in the coming season, allocating stones across three different piles for “above normal”, “normal” and “below normal”

Validity discussed in “Measuring Subjective Expectations in Developing Countries.” 2009. Delavande et al. World Bank Policy Research Working Paper 4824.

19

Extending an LSMS to other new areas Subjective reporting of health, welfare,

satisfaction of life, etc… Problem of different scales across people. A

person’s standard of good health may be different from another person’s standard.

Hypothetical vignettes: Respondents score a set of vignettes describing different scenarios related to the topic (such as health status) for a hypothetical person. These vignettes can help reveal the respondents own scale.

King at al (2004) and King and Wand (2007).

20

Extending an LSMS to other new areas These new areas imply new costs:

Piloting to ensure they work in different context Time to administer questionnaire Training interviewers appropriately