Experiences with Lot Quality Assurance Sampling Cairo – April 2009

29th-2nd/03/2009 Conference AfrEA - NONIE - 3ie Cairo, Egypt

Olivia Nakayiza & Luc Vanhoorickx – Save the Children in Uganda

Programme Overview Save the Children in Uganda (SCiUG) is a unification of Save the Children

Norway (Managing Member), Save the Children Denmark, Save the Children USA, Save the Children UK, Save the Children Sweden and Save the Children Italy

In 2008:• SCiUG programmes in over 26 districts

in 4 regions of Uganda• SCiUG works with 59 partners including

local NGOs, CBOs and local government.

• Total 313 staff as per December 2008• Reached 213,051 children

(116,137 girls & 96,915 boys)

Strategic Choices

In the year of unification, Save the Children in Uganda (SCiUG) developed a strategic framework that builds both on the strengths of participating members as well as on emerging analyses of the child rights situation that calls for more integrated programmingSCiUG has also continued to scale up its emergency response capacity to deal with emergency contexts, focusing primarily on emergency education and child protection

Community Based Family Planning Program Background

Increased voluntary use of key family planning services and RH behaviors among women and men of RA

A 4 year USAID/FLEXIBLE Fund Community Based FP/RH- Luwero, Nakaseke and Nakasongola Districts in Uganda

Women of Reproductive age (15-49) 68,826, Married WRA 33,036, Men 68,423. Total # of beneficiaries 170,285

Partnerships: District, MoH, etc

FP/RH Strategies and Intermediate results

Partnership Defined Quality (PDQ) process……

Behavior Change Communication, …

community-based services to increase access to FP methods ,...

Build capacity of district staff, …

strategic objective : “Increased and sustainable use of key family planning services and RH behaviors among women and men of reproductive age”.

INTERMEDIATE RESULTSIR-1: Increased knowledge, interest and use of FP/RH services

IR-2: Improved quality of FP service delivery by providers at the facility and community level

IR-3: Increased access of communities to FP services and information

IR-4: Improved social and policy environment for FP/RH services and behaviors

Results FrameworksWhere is results monitoring?

Goal

Strategic Objective

IR 1 IR 2 IR 3 IR 4

Strategies

Activities

Impacts

Outcomes

Inputs & Outputs

Time

Longer Term

Medium Term

Shorter Term

Check change in knowledge, behavior, status

RESULTS MONITORING:

Monitoring & Evaluation => LQAS

Higher level monitoring

Measure outcome indicators : compare with baseline & final targets

=> information needed = population/beneficiary based

=> sampling needed !

Effects of Behavior Change campaign: … behaviors

LQAS = Lot Quality Assurance Sampling

“determination of the quality of a lot by sampling”

comparison between areas (lots) possible small sample size, typically 19 per area

From the Monitoring we knew that we were generally carrying out the activities as planned, but wanted to Monitor&Evaluate whether it all had the desired effect/result/outcome/impact

Lot = sub county from a district

What is LQAS?

•A sampling method that: Can be used locally, at the level of a “supervision area,” to identify priority areas or indicators that are not reaching average coverage or an established benchmark

•Can provide an accurate measure of coverage or health system quality at a more aggregate level (e.g. program area)

For example, what LQAS can: just by sampling 19 women in a targeted population, at least 92% of the time LQAS will determine correctly whether yes or no these women have adopted the family planning method.

LQAS: dichotomic - 19Lot Quality Assurance Sampling in general small sample of 19 provides an acceptable

level of error for making management decisions; at least 92% of the time, it identifies whether a coverage benchmark has been reached or whether a supervision is substantially below the average coverage of a program area.

Samples larger than 19 have practically the same statistical precision as 19. They do not result in better information, and they cost more

Only if it’s above/below target

Decision Rule

LQAS cannot be used for coverage estimates in lot / district!

Some indicators that were measured

• Contraceptive Use (CU) or Contraceptive Prevalence Rate (also known as Met Need)Percentage of women of reproductive age (WRA) 15-49 who are married or in union using (or whose partner is using) a modern method of family planning

• Percentage of women of reproductive age (WRA) currently married or in union who are fecund (not pregnant or unsure if they are pregnant and not sterilized) who desire to have no more children or postpone childbearing, and who are not currently using a method of family planning (Unmet Need for Family Planning)

• Percent of Demand (Met need + Unmet need) satisfied• Knowledge about family planning methods:

FP METHODS THAT WERE MEASURED• Depo-Provera• Pills• Condoms• Norplant and Tubal Ligation

LQAS in Family Planning project

Effects/outcomes of project interventions:… behaviors

% of women of reproductive age who report having access to FP services% of women of reproductive age who were counseled about the birth spacing

Sampling: Systematic Random Sampling from …• 5 lots were chosen for different districts• 2-step random sampling procedure. Communities were selected using

systematic random sampling. HH selected using classical random sampling

Data collection tools: Adapted from FlexFund survey guidelinesData analysis: Standard LQAS tables in Excel worksheets for analysis & graphs

As the mid-term evaluation of the Family Planning project

Process

5 “supervision areas” Random samples of 19

from monitoring records of women of reproductive age (WRA)

Interchanged extension workers for data collection

Minimal data-entry where PDA were used

Instant results Fast feedback

LQAS Summary Tabulation

Indicators

Priorities where Target ( T ) and/or Coverage ( C ) is not achieved

TargetLQAS Coverage

95% CI

Lot 1 Lot 2 Lot 3 Lot 4 Lot 5

% of women 15-49 years who are not pregnant or are unsure, who are using a modern family planning method (Contraceptive Prevalence Rate or Met Need) OK C OK OK OK 50% 68.9% 10.8%% Women of reprod. age know about at least three methods of family planning (Knowledge about Family Planning Methods) OK OK OK OK OK 70% 100.0% 0.0%% of women of reproductive age that receive counseling about birth spacing C OK C OK OK 60% 68.8% 8.7%% of women of reproductive age who report having access to FP services OK OK OK OK OK 70% 96.0% 3.3%% of women of reproductive age who report discussing FP with a health worker or family planning worker/promoter T & C OK T OK OK 70% 67.4% 9.5%% of WRA (15-49) currently married or in union who are fecund (not pregnant or unsure if they are pregnant and not sterilized) who desire to have no more children or postpone childbearing, and who are not currently using a method of family planning (Unmet Need for Family Planning) N/A N/A N/A N/A N/A 30% 34.2% 13.6%Total Demand for Family Planning   93.1% 4.9%Percent of Demand Satisfied       72.6%  % of sexually active respondents who report discussing FP issues with their spouse or (cohabitating) sexual partner in the past 12 months OK T T & C OK OK 60% 54.3% 10.2%% of respondents that lives within 5 km of a family planning service delivery point (SDP), [among women who know where to obtain a method] (Proximity to Family Planning Service Delivery Point) C C OK OK OK 30% 73.0% 8.7%                 

FP LQAS OUTCOMES LQAS Family Planning '07

68.9%

100.0%

68.8%

96.0%

67.4%

34.2%

93.1%

72.6%

54.3%

73.0%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

ContraceptivePrevalence Rate

Knowledge aboutFamily Planning

methods

Counseling aboutbirsth spacing

Access to FamilyPlanning services

Discussing FamilyPlanning withhealth worker

Unmet Need forFamily Planning

Total Demand forFamily Planning

Percent ofDemand Satisfied

Discussing FamilyPlanning with

spouse of sexualpartner

Proximity toFamily Planning

Service

LQAS Coverage

Target

Average Coverage

LQAS Health outcomes

48.0

%

96.9

%

96.0

%

36.5

%

84.9

%

79.9

%

78.4

%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Daily water use percapita above 20l

Appropriate handwashing behavior

Bring their childrento appropriate

provider with fever

% of womenreporting currentuse of moderncontraceptives

Givingcomplementaryfoods at 4 to 6

months

Give more fluidsduring diarrhea

episode

Know 2 or morecauses for growth

faltering

Baseline

MTE

FY07

Final Target

After the quantitative survey…

• LQAS gives the numbers (it’s quantitative) Identifies the problem How big the problem is Where the problem is

• BUT NOT why we have the problem… Hence the need for qualitative research

/discussions to o Explain quantitative resultso Identify the reasons whyo Provide recommendations on program design

and strategies

Lessons Learned (1)

• Internal interviewers were used but swapped to reduce biasQuantitative LQAS survey replicated yearly

• Although some statistical analysis is not possible with LQAS, indicators can be analyzed further to reveal additional informatione.g. analyze which ‘part’ of knowledge / behavior adopted => focusing messages

Lessons Learned (2)• Results discussed w/ program staff during

participatory feedback meetings: discussions & decision making w/officers + review meeting w/extension staff

• Guided by principal areas for improvement identified in LQAS surveys: provide recommendations to guide the team in second half of the project and inform planning

PDA Experience/Lessons learnt (3)

• PDA to be easy to carry in the pocket of the riding gear

• PDA contained automatic skip patterns

• Of completeness of the data, all required questions were responded to

• a reminder in most required questions in the PDA which could beep

• Once an electronic questionnaire was filled in, data is instantly stored and possibilities were built-in to review the data

• data is clear in terms of spellings and uniformity

• Data entry time is obviously reduced to zero

Experiences with Lot Quality Assurance Sampling Cairo – April 2009

Thank You !

LQAS & the MTE