harmonizedmethodologyof conducting …€¦ · 1 harmonizedmethodologyof conducting baseline/impact...
TRANSCRIPT
1
HARMONIZEDMETHODOLOGYOF CONDUCTING BASELINE/IMPACT STUDIES
HARMONIZED METHODOLOGY
March 2014
By:
KOUADJO Yao Jean Marc Lecturer- researcher at ENSEA [email protected]
Dr NIAMBELE YOUSSOUF SAMBA [email protected]
Dr FE Doukoure Charles Lecturer- researcher at ENSEA [email protected]
KANGA Désiré Lecturer- researcher at ENSEA [email protected] NAYO AnkouviMawoudoudji Lecturer- researcher at ENSEA [email protected]
Gbahiluc Agricultural statistics lecturer [email protected]
2
Table of content
HARMONISED METHODOLOGY ...............................................................................................1
I. Country Typology ................................................................................ Erreur ! Signet non défini.
A. Coastal countries ............................................................................. Erreur ! Signet non défini.
B. Sahelian countries ........................................................................... Erreur ! Signet non défini.
C. Intermediate countries ........................................................................ Erreur ! Signet non défini.
II. Establishing the baseline ..................................................................... Erreur ! Signet non défini.
A. Overall framework ..................................................................................................................7
1.Preliminaries to the establishment of a baseline .....................................................................7
2. Methodological approach ........................................................... Erreur ! Signet non défini.
B. Scope of the study............................................................................ Erreur ! Signet non défini.
1. Criterion for identification............................................................ Erreur ! Signet non défini.
2.Delimitation of the scope of thestudy ............................................... Erreur ! Signet non défini.
C. Statiscal units .........................................................................................................................9
D. CensusJsampling .............................................................................. Erreur ! Signet non défini.
1. Importance of sampling plan ....................................................... Erreur ! Signet non défini.
2. Enumeration areas approach vs. Village approach ...................... Erreur ! Signet non défini.
3. Identification of areas within project start period....................... Erreur ! Signet non défini.
4. Statistical units selection ............................................................. Erreur ! Signet non défini.
E. Development of data collection tools ............................................ Erreur ! Signet non défini.
F. Frequency of collection .................................................................... Erreur ! Signet non défini.
G. Case of livestock farming ................................................................. Erreur ! Signet non défini.
III. ESTABLISHING THE PANEL ............................................................... Erreur ! Signet non défini.
A. Importance of the implementation of the panel .............................. Erreur ! Signet non défini.
B. Proposal for a methodological approach to the panel establishment ........ Erreur ! Signet non défini.
1. Defining the scope of the study .................................................... Erreur ! Signet non défini.
2. Statistical units and data to collect .............................................. Erreur ! Signet non défini.
3. How to collect information ? ........................................................ Erreur ! Signet non défini.
4. Working out the survey plan ........................................................ Erreur ! Signet non défini.
5. Survey group ................................................................................ Erreur ! Signet non défini.
6. Selecting mode of the survey group ............................................. Erreur ! Signet non défini.
7. Establishing the size of the sample group .................................... Erreur ! Signet non défini.
3
C. Tools for collecting information ....................................................... Erreur ! Signet non défini.
IV. Impact assessment .......................................................................... Erreur ! Signet non défini.
A. General framework............................................................................................................... 19
1. Reason for assessing ......................................................................................................... 19
2. The Issue of the impact assessment .................................................................................. 19
B.Types of impact assessment ...................................................................................................... 19
C. Methods of impact assessment ............................................................................................ 19
D. The types of biased selection ................................................................................................ 20
E. Internal and external validity ............................................................................................... 20
F. The different types of errors ............................................................ Erreur ! Signet non défini.
G. Some impact assessment method .................................................. Erreur ! Signet non défini.1
1. Modèle de discontinuité de la régression ................................... Erreur ! Signet non défini.1
2. Comparison before/after and during the program .................... Erreur ! Signet non défini.3
3. Comparison with the program and withoutthe program ........... Erreur ! Signet non défini.3
4. Double difference (DD) ............................................................... Erreur ! Signet non défini.3
5.Matching ........................................................................................ Erreur ! Signet non défini.4
6. Combination of methods ............................................................ Erreur ! Signet non défini.5
7. Qualitative Assessment ............................................................. Erreur ! Signet non défini.5
H. RECOMMENDATIONS .................................................................... Erreur ! Signet non défini.5
V.List and contact of consultants who participated inthe writingàf the harmonized methodology. 286
4
I. COUNTRY TYPOLOGY
The Economic Community of African States (ECOWAS) is a group of fifteen countries,
including Benin, Burkina Faso, Cape Verde, Côte d'Ivoire, Gambia, Ghana, Guinea the
Guinea Bissau, Liberia, Mali, Niger, Nigeria, Senegal, Sierra Leone, and Togo. Its mission is
to promote economic integration in all fields of economic activity, particularly industry,
transport, telecommunications, energy, agriculture, natural resources, commerce, monetary
and financial matters , social and cultural issues. Because of the extent of the economic zone
and geographical location of the countries (coastal countries,sahelian countries, etc..),climatic
variations are observed from acountry to another as well as inside the same country
(according to the size of the country and the diversity of the topology).
As far as agriculture is concerned, several speculations among which agricultural calendars
and crop cycles that depend on factors mentioned above are observed.
Under the project PPAAO a number of products have been identified in each country, to be
supported in terms of technological innovation, management, technology transfer, etc.
Globally, the countries of ECOWAS can be grouped into three major types.
A. Coastal countries
Most of these countries are located on the Gulf of Guinea and are characterized by two major
climatic zones which stand alongside each other: the equatorial climate and tropical climate.
The equatorial climate is characterized by low amplitude temperatures, a high rate of
humidity and abundant rainfalls. This area experiences two dry seasons and two rainy
seasons. In the tropical climate,daily and annual changes in temperature range are relatively
important. In this climate, the number of rainy seasons and dry seasons vary between one and
two according to countries.
The agricultural sector in this area plays a key role in terms of national economies and rural
households income. The agricultural activity in these countries is characterized by a
significant production of cash crops (cocoa, coffee, rubber, oil palm, cashew, etc.) earmarked
to be exported and annual crops for local consumption (cereals, tubers, plantain, etc).
5
Inthese same countries, most WAAPP food cropsareshortcycles, but several cyclesare
observedduringthe yearbecause of the differentrainy anddry seasonfor off season crops.
Moreover,the number of cyclesdecreasesas we approachthenorth of thesecountries in general.In order
to havea good estimate ofthe area planted, it is necessary to take into accountallproduction cycles.
The key countriesare:Benin, Côte d'Ivoire, Ghana,Guinea, Liberia, Nigeria,SierraLeoneandTogo.
B. The sahelian countries
Generally locatednorth of thecoastal countries,they are characterized bya semi-aridtropical climate
witha longdry seasonand a wet seasonrelativelyshort andthese countries havepotentialfor
breeding,butalso practice ahigh activity ofFood crop production.
This area has a herd largely consisting of cattle, sheep, goats, pigs and poultry. Thus there is a
strong meat production and a significant production of eggs and milk. Large livestock systems
practiced are: pastoral systems (animal feed based on the exploitation of natural resources
such as grassand shrubs), off-ground systems (characterizing urban production) and agro-
pastoral systems (which combine grazing on uncultivated land, consumption of cultivated
fodder, agro-industrial by-products etc ...). The period of crop in these countries is
approximately three months.
C. Intermediate countries
These countries are characterized by climates that range from wet and semi-arid tropical.They
straddle the coastal and Sahelian countries. Agriculture and animal husbandry are practiced
according to potential. This area is the production belt of coarse grains such as millet and
sorghum, maize under food crops, oilseeds (sesame, shea, peanut). Crop period lasts 6
months.
Map1: Location ofmain food crops production areas
6
II. Establishing the baseline
Abaselineisan originalphotograph ofinterestindicatorsin the absence ofthe
project.Thisoriginalphotograph must bedonebearing inmind theimpact assessment of the
project. The reference period of such asituational analysisis the year.
A. Overall framework
1. Preliminaries to the establishment of a baseline
Establishing abaseline, supposes:
• The existence of a project result framework
The results framework shows the variables to be monitored in this study.
These are the variables that will be our collection medium.
However, other variables could be added for a national and international need.
• The existence of a TDR which further specifies indicators to mesure
7
The TDR is a command. It is the formulation of an order specifying constraints (expected
results, due time, required skills ...)
2. Methodological approach
To better carry out a baseline study, two approaches can be combined: a literature review and
/ or further investigations. Indeed, the establishment of the position of references does not
necessarily involve the collection of data in the classical statistical sense of surveys (as
opposed to administrative data, or estimates).
a) Littérature review :
This is to review existing documents. The literature review can be based on secondary data,
reports, studies carried out, etc.. Two cases are possible:
• If the indicators to be completed are recorded in reports and can properly respond to
information needs, they must be used.
• If the indicators to be completed have already been collected and databases are
available, secondary data must be resorted to/used to carry out a baseline.
b) Further surveys
Further surveys are required to collect information for non completed indicators in the
literature or for those theinformation of which are available but would be obsolete. The
implementation of the survey for the collection of additional data should follow a coherent
approach respecting a set of statistical principles. The different steps to follow are described
later in this section.
B. Scope of the study
It is aboutidentifying theparts of the countryor areas of thecountry where the studywillbe
conducted.Severalcriteria must be consideredto identify thescope of the study.
1. Criterion for identification
8
The scope of the study is defined using the criteria below:
• speculation retained in the project;
• the level of detail of indicators geographically;
• Specificity of the country;
• the presence of speculation.
2. Delimitation of the scope of the study
It is often recommended to ensure national coverage of the study as the aim of WAAPP /
PPAAO being eventually to cover all the producing areas of selected crops/speculations.
Nevertheless, the national coverage refers to the main production areas of crops/
speculations.Indeed, if a speculation is produced only in a specific part of the country,
focusing on this part is enough to have a national coverage.
C. Statistical units
Statistical Units
The types of statistical units to be considered when collecting are:
• Households;
• Individual producers within households;
• Production cooperatives;
• Modern and artisanal processors (semi-finished products, finished products);
• Those involved in marketing (buyers / traders);
• Research Centers and seed centers;
• The structures of support and extension.
The information contained in the table will be collected with these people
Target population Collected information
Households (head of
household, operator
and sub-operator)
- Living condition of their households
- Cultivation practices
• Change in cultural techniques used
• Varieties of agricultural crops used, their perception (adoption)
9
and the source ;
• Types of support they receive
- Monitoring of the campaign (rainy seasons and against
seasonal)
• Speculations practiced
• Cultivated areas, yield (production)
- Operating account
• Manpower cost
• Access to inputs
• Care of plots (phyto….)
• Other expenses related to the practice of the activity
• Marketing
- Identification of difficulties and constraints (flow line,
pricing….)
- Perception (quantitative and qualitative) of the project effects by
recipients
Processors
- Intermediate food crop consumption
- Finished/end products of food production
- Management of the
People involved in
marketing
- Flow line
- Marketing channel
- Product prices
- Difficulties and constraints
- Management of the activity
Production
coopératives
- Acquisition of agricultural and plant equipment for producers
- Rural training/extension
- Support counselling
Support structures
and research and
seed centers
- Types of support provide to producers
- Strategies for cultural techniques extension
- Technology transfer
- Research and innovation development through target channels
10
D. Census/Sampling survey
In sofar that national agricultural censuses (RNA) in this sub-region are generally surveys and
taking into account budgetary constraints and also for reasons of convenience, the use of
sampling technique is the most suitable.
1. Importance of a sampling method
The sampling plan adopted aims at two main objectives:
• Extrapolate the results at the national level;
• Help build up a control sample to facilitate the estimation of the counterfactual
(impact assessment).
To meet these objectives, a double sampling will be set up and broken down as follows:
• Sampling at the national level in areas out of project (this sample will be our control
sample);
• A second sampling exclusively in the project zones (This sample will be our target
sample)
2. Enumeration Area approach (EA) vs. Village approach
The villageis usually abasicadministrative unitcomposed ofneighborhoods,
themselvesformedby the gathering ofmembers of one orseveral families
andpossiblyencampmentsattached to it. It is headedby a chief who, to be recognized by the
State,must be freelychosen by thevillagersaccording to customaryrules,by consensus orbyany
other means.The village is,therefore,integrated into theadministrative division.
The Enumeration Area (EA) is also called, depending on the country, enumeration zone
(Benin...) and Enumeration Section (Mali ...).
The EAs were set up to facilitate census operations in the field and survey. EAs generally
speaking correspond to the workload of an enumerator: about 800 inhabitants in rural area
and 1500 inhabitants in urban area. Given the relative homogeneity of the SE compared to the
village from the standpoint of size, identification, etc, the aggregate better meets the
characteristics of a sampling frame.
11
The Enumeration Area is most oftenused asPrimarySampling Unit(PSU) in a multi-stage
sampling.
The village approach is to consider the village as Primary Sampling Unit (within which the
secondary statistics units will be selected ) . It can be effective when located in a state where
the census is recent. However, this approach has numerous disadvantages. The number and
size of villages vary frequently. The village approach leads to a bias sampling insofar as:
Only producers who are in the core villages during the survey are included in the sample
while those camping in Hamlets attached to the village, are excluded. Of course, this happens
if the enumerationunits in the village have not been exhaustive.This risk is high due to the
high dispersion of crop areas and also, often the big size of the villages;
In addition, producers who are not resident in the villages (peri-urban farmers) are excluded to
the extent that the villages do not constitute a partition of the country.
The approach is to consider the EAas primary unitfrom which other statistical units will be
selected. This approach has advantages and disadvantages. The major drawback of this
approach is that in a situation where the population census dates several years back, some
agricultural EAs may become nonfarm EAs leading to the reduction of agricultural EAs (due
to urbanization). Moreover,some community issues are often treated at the village level - the
village representing an administrative body whose head is formally appointed by the regional
administration.
It should be noted that in general, WAAPP interventions occur at village level. So
naturally,thesamplingstend tousevillageapproaches.
However, this approach has many advantages compared to the village approach. The main
advantages of the EA approach are:
• The EA approach allows extrapolation of results at the national level because of the
homogeneity of the EAs;
• Enumeration of statistical units within a relatively easy due to the harmonization of
EAssizes in terms of households;
• The EA approach allows the inclusion of all entities within the EAs (Villages and
hamlets);
• The number of EAis constant between twoPopulation Census.
• The EA constitutes a partition of the territory
12
3. Identification of areas within project start period
When measuring the baseline, it may be that the intervention areas are not yet known. It is
therefore difficult to identify target areas. Onecan resort to speculation on the likely areas
ofintervention in consultation with WAAPP local supervisors/staff. The existence of projects
can promote or even disadvantage the implementation of new projects
4. Statistical units selection
The selection of the Statistical unitswill bemade based ontwo methodsaccording to the
criterion of thepresence of culturepracticed.
1st case: widespread crop presence
In the case ofcommonculture,the methodology adoptedwill consist in making random
sampling ofstatisticalunits within theEAs.This approach is basedon the assumption thatin
anEA,the probability of findinga producerof cultureis not zero.
2ndcase: presence of uncommon crop
In uncommoncrops, the first method has certain disadvantages. To address these drawbacks, a
second methodology will be carried out in the EAs when one is in the presence of unusual
culture. This approach may be made of two approaches:
• The first approach is to list the EAs where the culture is practiced. It is from this list
that the statistical units will be drawn
• The second approach is to establish a list of the producers of the uncommon cultures
within the EAs culture. This list will be the sampling frame for the selection of
statistical units.
The secondmethodhas the advantage of selecting in EAsproducers inan area where theculture
is verylittle practiced.
E. Development of data collection tools
The questionnaires usedfordata collectionare:
13
• Households /producers/speculations surveyquestionnaire
• Thehouseholdquestionnaire
• The plot per speculation/crop survey questionnaire;
• The square yieldquestionnaire;
• The producerquestionnaire
• The processorquestionnaire
• The traderquestionnaire
• The supportstructure andresearch center questionnaire.
F. Frequency of collection
The collection of datais accordingto the agricultural calendarof theselected crops. It will bea
survey which the number ofpassageswill depend on:
• Thenumber of cyclesforeach crop;
• Off crop season if there are some;
• According toTDR
It is notnecessary to keepagentsthroughout the yearunlessthey are civil servants.
Thefirstpassage/visitis used tocapture thestructure variablesand demographicvariables.During
this passage, the areas of land under crop aremeasuredand squaredensity /yield laidfor
futuredata collection on crops.
Another passage/visitis necessaryto capture for crop cutting and areas measurement.This
passage isnecessaryto monitorcropduringthe entire agriculturalcycle.
G. Livestock farming The methodology for capturing indicators of livestock is multidimensional:
• We have the National Livestock Census (NLC) to collect data on structures
• Annual surveys to capture cyclical indicators
• Information systems on livestock
Among other information systems on livestock, we have:
14
• The national markets information system of cattle (NMISC)
• Medical monitoring of livestock (veterinary and vaccination campaigns)
• The monitoring of slaughterhouses
• Data file from the customs services (exports / imports)
All thesemethodsapply tobothtraditional(sedentary traditional livestock andtranshumant)
sector andmodern sector(operationkeeping accountsbeyondthe criterion ofnumber ofhead)
For indicators monitored by the WAAPP, it is recommended that:
1 - Make a literature review on the existing data from the livestock Census, annual
surveys and various information on livestock systems.
2 - Make a supplementary national survey based on non-completed indicators after the
literature review.
Collection methodsare differentfor different types oflivestock andindicatorsto complete.
For thesedentarylivestock, household surveys are the best way for collectinginformation.
As fortranshumant livestockregardingruminants,data are collectedat water points(which are
theplaces ofherding) forindicators such asthenumber of livestock, livestockcharacteristic etc.
III. ESTABLISHING THE PANEL
A. Importance of the implementation of the panel
The main objective of this program is to improve the productivity in food crops in ECOWAS
countries. To do this, it is advisable to conduct adequate monitoring of all the actors involved
in the production of food crops in the different countries, taking into account their
geographical, ecological, climate etc. The construction and implementation of a panel is a
very appropriate approach to meet this objective insofar as it will allow better understanding
of the different changes that may appear over time, to make comparisons and conduct impact
studies. To adopt a panel is useful for measuring the evolution of different indicators,
particularly marginal variations from the baseline. Monitoring panel can identify early
distortions, and implement corrective/remedial actions.
B. Proposal for a methodological approach to the panel
establishment
15
To establish thepanel, each country should define thescope of the study, the statistical units,
the sampling methodology and the datacollection tools.
1. Defining the scope of the study
For each country, the study will focus onthe set of alltheWAAPPfarmsinfoodproductspresent
on thenational territory.Asample isalready establishedin recentstudiesin eachcountry;thenit
remains to examinesome of thesereference samplesfor each typeof product.
2. Statistical units and data to collect
The target population forthis studyconsistsof farm households andcooperativesspecializedin
the cultivation offood crops, processors and distributors offood products, research
centersandseedand supportstructures.Theinformation to be collectedrelate tocultural
practices,monitoring campaigns, operating account, identifying difficulties, the process of
transformation andthe distributionof food productsand supportprogramsto farmers.
3. How to collect information?
The process of collecting information will be done in three stages:
- First stage: at this stage, the information collected will be about living conditions of
households, the characteristics of the farm and the farming methods and the size of the
exploited areas.
- Second stage: will begin straight after the first stage and will be about harvests and
the yield of the farm. It will also make it possible to follow the activities of traders,
processors, research centers, seed companies and support organizations on a weekly
basis. Collected information will be about the quantities purchased, sold or processed
and the added costs.
- Last stage:it will take place during supervising missions to capture new areas.
Follow-up/Monitoring of the panel must be done on a regular basis according to an
established periodicity. Survey clerk (enumerators) must be working on the field with
producers in their activities in order to avoid loss of statistic data. These three stages
will be performed in the areas benefiting or not benefiting by the program, in
conformity with the agricultural calendar.
16
4. Working out the survey plan.
The survey plan selected to conduct the study must specify the following aspects.
- The size of the representative sample of producers.
- Creation of producers survey frame.
- The selection method of the groups to be interviewed.
5. Survey group.
The survey group selected must be made up of the lists of producers initially sampled from a
recent study carried out at a national level.
6. Selecting mode of the survey groups.
The way of selecting the statistic survey groups depends on the availability of the exhaustive
list of all the people who benefit from the project without any omission, without double
count/counting.
In this case one can proceed to a drawing at random of the statistic groups. This method is
relevant to extrapolate the results at national level. Its implementation can for example consist
in a survey drawing stratified at two levels. The levels can be set up according to
administrative divisions (Regions, districts, sous-préfectures….). The choice of the
administrative division will allow taking into account constraints related to the maximum
realization of the panel. In the first position, the farming SD (Survey Districts) can be selected
in conformity with the stratum weight. In the second position, farmers will be selected
following a simple random survey within the Survey Districts.
If the list of those who benefit by the project is not available, one can decide on a reasoned
selection.
7. Establishing the size of the sample.
One of the sensitive questions any statistician has to face before sampling is the choice of the
size of the sample: how many survey groups do we need to have an estimate of the interest
17
parameter of ‘good’ precision? The answer to this question is not simple. It’s clear that to
have a good estimation, one must choose sampling size big enough. But how far can we go in
the choice. In the absolute, there is no straight answer as far as the « optimal » size of the
sample to work on is concerned. But in practice, we may have two constraints
- budgetary constraint
- constraint of precision
The budgetary constraint will consist in deciding on the size of the sample taking into account
the whole budget of the survey. Thus, if C represents the total budget and c the unit cost for a
single individual, to be surveyed (bonus, allowance, questionnaire etc.), therefore, we have:
n =��
However, this size does not guarantee ‘’good” results.
If budgetary constraint is weak, one will have to decide on a precision a liability level to reach
and calculate the size of the sample that will enable us to guarantee this precision. The
calculations methods mostly used are
222 )1(96,1
pkppn −
=
Where p represents a proportion for a particular indicator and k the margin of error on p
or 22
2296,1
xkn σ
=
whereσ is standard deviation and k (%)percentage of error onan averagefor examplearea?
Some methods also estimate the minimum size that enables to observe a future impact with
the desired precision. However the range of methods is wide and the method to be used
depends on the context.
C. The tools for collecting information.
The tools are mainly about
• a counting form
• a surveyor’s guide
18
• questionnaires
Indeed, a questionnaire must be set up for each statistic group. This is inherent to the fact that
the information collected from the different statistic groups can be of different nature
therefore, heterogeneous.
We can set up the following questionnaire:
• household questionnaire
• land counting questionnaire
• producer questionnaire
• processor questionnaire
• trader questionnaire
• questionnaire for research centers, seed companies and support structures
IV. Impact assessment.
A. General framework.
1. Reason for assessing.
The PPAAO/WAAPP impact assessment must eventually enable us to:
• First of all identify the changes in welfare of the individuals that could be due to their
participation to the program.
• Next, to assess the efficiency of the program by showing what works and what does
not work. This will permit to fill the weaknesses spotted out during the
implementation of the program.
Then to reinforce transparency by giving an account of the results of the program to the
public. Finally, to prove that the program gives results that are enough to justify granting
financial resources needed to extend and complete the program.
2. The issue of the impact assessment
19
The assessor usually faces three challenges. The first challenge is to prove that the changes
(results obtained) are essentially due to the program. Indeed, the changes can be the results
ofother projects and originated from other conditions outside the program. The task for the
assessor will be to push aside any other factor that could have influenced the results obtained.
The second challenge will consist in establishing the counterfactual, that is to say any result
that could have been obtained by a group of individuals directly if the program had not
existed. The third challenge will be to eliminate any biased selection. The way of selecting the
beneficiary indirectly implies a selection bias if the choice is not done at random.
B. Types of assessment.
There are two (02) types of assessment: prospective and retrospective assessments.
Prospective evaluation (ex-ante) is characterized by two elements. On the one hand, it is
included in the program right from the start and on the other hand the analysis of impact is
done before setting up the program. The retrospective assessment (ex-post) also has two (02)
characteristics. It is carried out after the program has started and the data are (baseline) are
collected after the program got started. Concerning the WAAPP impact assessment of the
program, the retrospective method is recommended.
C. Methods of impact assessment.
There is a lot of literature on assessment methods. But remember that there is no perfect
impact assessment method. The ways you select beneficiaries determine the appropriate
assessment method.
We have three (03) assessmentmethods: the experimental, half-experimental and not
experimental methods.
The experimental method is based on a random choice of the beneficiaries of the program. It
is the perfect method in determining the sample group. Indeed, this method allows us to find a
non-beneficiary group having observable and non-observable characteristics similar to those
of a beneficiary group by the program. It permits to eliminate any biased selection. In
practice, its implementation has to face political reluctances.
The half-experimental method is a mixed method. It combines an experimental method
(random) and a non-experimental method (empirical). Its advantage is that it is used in a
situation of reference. Moreover, it uses econometric and statistic instruments. It costs less
20
and requires a shorter time. Yet, its use does not permit to eliminate the issue of biased
selection. It is the most recommended method in the WAAPP program assessment.
The non-experimental method is essentially empirical (no random selection). It has the
advantage of being easy to carry out. Its implementation requires econometric patterns.
However, results are not reliable in terms of statistics. Selection bias is a disadvantage for the
liability of the results of this method.
D. The types of biased selection.
There are two(02) types of selectionbias. The first type puts together methods which have
observable differences or others in the information (geographic location). While the second
type is characterized by methods which have non-observable differences (not in the
information) such as motivation, intelligence, family health history and pre-existing
conditions…)
These two types of selection bias can give inaccurate results which can lead to
underestimation or overestimation of the real impact of the program. Thus, we might
deducethat the impacts are negative when the real impacts of the program are positive (and
vice versa) and statistically insignificant when they are actually significant.
E. Internal and external validity.
A quality impact assessment must fulfill 2 main criteria:internal and external validities.
Internal validity is when the sample group chosen has the observable and non-observable
characteristics similar to those of the beneficiary group. Otherwise, any declared impact
would be wrong.
External validity is when the sample chosen statistically represents the population surveyed. If
the sample cannot be extrapolated to the level of the population, therefore any declared impact
would be wrong.
F. The different types of error.
21
Beyond the overestimation or underestimation of the assessment, two types of error can
appear during the assessment of the impact of the program:
Type 1 error: consist in saying that the program has no positive impact whatsoever on
people’s welfare whereas the program has actually contributed positively in improving living
conditions of the targeted population.
Type 2 error: consist in saying that the program had a positive impact on population welfare
whereas the program has not contributed in improving living conditions of the targeted
populations.
These two types of error appear when internal and external validity hypothesises are not
verified.
G. Some impact assessment methods.
There are different impact assessment methods. In this report, we will show some assessment
methods according to the type of allotment to the beneficiary by the program.
1. Model of discontinuity in the regression.
Just like the experimental method, this pattern uses a pre-defined and clear rule for
theselection on the beneficiaries of the program. An index is used to select people eligible for
the program. The model of discontinuity of regression which better fits the program for which
an index of eligibility is established and a limit is clearly defined to differentiate beneficiaries
from non-beneficiaries.
Ex: age limit for retirement program, poverty index, scores for school grants program …
An example of application: household expenses and poverty level. Before the intervention
Poverty index 50 is used as criteria to be eligible for the program. Thus, any household with
an index superior to 50 is considered as « non-poor » and cannot get any subsidy from the
program. Only households with a poverty index below 50 can benefit by the program.
Besides, homes with a 49.9 index are similar in many points to those with a 50.1 index, but
contrary to the second, the first mentioned benefit by the program.
22
After the intervention. Given the fact that the two groups were similar before the program and are exposed to the
same factors (such as prices fluctuations, regional and national farming policies, etc), the
program is the only reason that can explain the results after the intervention.
However, the method does not permit to assess the impact of small households (with a 20
poverty index) because they do not compare with any other group.
23
2. Comparison before/after and during the program. This method consists in determining the impact of a program by studying the evolution of the
results of the participants to the program as time goes by.
The hypothesis is that if the program had not existed, the result for the participants to the
program would have been exactly the same as before the program. Unless one is capable of
controlling statistically other factors like climate (droughts, important pluviometry) that can
influence the result of the study. Therefore, it is not possible to determine with certainty, the
real impact of the program, making comparison before-after/during the program.
3. Comparison with the program and without the program. With the method of comparison with or without, the assessment of the impact of the program
is likely to be biased. This generally happens when the comparison group is not eligible for
the program or when they decide not to participate.
There are characteristics non-observable between the two groups of people leading to a
selection bias which consequently also leads to biased results of the impact of the program.
4. Double difference (DD)
24
Random allotment and the model of discontinuity of regression allow the estimation of the
counterfactual if the basic rules of allocation of the program are known and included with
less hypothesises and conditions.
The Diff-diff and matching provide the assessor with tools when the rules of allotmentare
less clear.
DD compares differences in results between a beneficiary group and a sampled group as time
goes by. The difference in results before-after for the group participating (the first difference)
controls for time-invariable factors which influence the group, for the simple reason that we
compare the group to itself.
The difference before-after does not take into account variable external factors as time goes
by. A way of taking into account these external time-variable factors is to measure the
difference before-after for a group which is not part of the program.
a. Quality check for DD
The quality of the sampled group determines the quality of the assessment.
There are many methods to check the quality of DD. However, we will show the « placebo »
method. This method consists in using a fake group of beneficiaries clearly knowing that they
are not part of the program.
Ex : for the coming years (e.g -2, -1)
If the DD estimate is different from 0, the trends are not parallel and our original DD will
probably be biased. Therefore, one must use a different sampled group. On the other hand, if
the DD estimate is nil, the results are good.
b. DD hypothesizes.
The use of the DD method is subject to hypothesizes.
The DD method enables us to take into account the differences between the group of
beneficiaries and the sampled group if they are invariable as time goes by. But it does not
permit to eliminate the differences between the two groups when they change as time goes by.
Thus, one has to go from the principle that if the program didn’t exist, the changes between
the given treatment group and the comparison group would evolve in parallel i.e that there
isn’t any variable difference between the two groups as time goes by.
On this point, at least three observations as time goes by are necessary to check this
hypothesis. Indeed, two observations are required to estimate the treatment before the
25
intervention which permits to check the hypothesis at equal trends. And a post- intervention to
assess the impact of the intervention.
5. Matching.
This method consists in using the characteristics observed within the beneficiary and non-
group to generate a sample group. Practically, it requires the use of an important database
and complex statistic techniques to generate the best artificial comparison group possible for
a given survey group.
But it does not take into account the characteristics not observed likely to affect the results.
6. Combination of methods.
The combination of the two methods is called the paired double difference (PDD). For the
implementation, the matching is to be done on the basis of characteristics observed in the
database. Then apply the double difference in order to assess a counterfactual for the change
of the result for each sub-group of paired units. And finally, calculate the average of these
double differences for all the sub-groups.
7. Qualitative Assessment. Qualitative assessment is in favour of techniques and instruments such as semi-structured
interviews in groups and thematic debates with the beneficiaries (beneficiary assessment).
H. RECOMMENDATIONS.
In this WAAPP program, we suggest to the different countries, to implement the Paired
Double Difference method combined with the qualitative assessment. This qualitative study
will help understand certain informative qualitative aspects such as school grants, financed
trainings, study trips…
More than the technical aspects presented, the appropriate elements of recommendations are
to be taken into account for the good realization of the assessment of the WAAPP program.
First of all, questionnaires meant for farmers must cover the development indicators and
indicators related to the results (23). Besides, we must not lose sight of the gender and
environmental issues.
Moreover the assessor must be involved in the implementation of the situation of reference.
This will help avoid omitting pieces of information at the moment of assessment.
26
Then, it is useful to create a consolidated trading account for beneficiaries and witnesses, for
crops which become priorities as time goes by. The present gross value (PGV) and theinternal
investment rate (IIR) can be used as criteria of investment in the program in every country.
Then a panel must be set. Follow-up of the panel permits a better visibility of the real impact
of the WAAPP.
Finally, we must guarantee the internal and external validity, that is to say, the quality of the
sample group through the « placebo » method or the matching method.
V. CONCLUSION
The development of a harmonized methodology for baseline and impact measurement may
seems utopian. First of all the countries are geographically and climatically different and in
other hand, the selected crop are not the same from one country to the other. hence a
harmonized approach is indicative broadly and particularly focuses on best practices to be
implemented for data collection, survey design…
The scope of the studies should be national insofar as WAAPP project aspires to expand its
hedging activities throughout the whole country (it should be clarified that the national
coverage is mainly refers to all of the agro-ecological zone where these crops are practiced).
The statistical units to be considered must be households, individual holders within
households, cooperatives of production, modern and artisan processors, marketing actors
(buyers / traders), research centres and seed centers and finally the support organization and
outreach organization. In addition, within the household, should identify all members and
their involvement in agricultural activities. This approach allows addressing gender issues and
understanding the participation of women, men, young people, etc.
For getting data for the baseline, two approaches can be distinguished: the literature review
and further data collection operation.
Further data collections are needed are needed only if the indicators are not documented in the
literature or if the available information is outdated. In this case, it is necessary to do a double
sampling. Sampling at the national level in out of project area and sampling in the project
area. This in order to measure the impact of the project. Enumeration Areas units (EA) should
be preferred to villages approach to facilitate extrapolations, although effective interventions
PPAAO cover the villages.
27
In the case that the major crops are rampant, the EAs are randomly selected and household
within EA are entirely listed, while in the case of scarce crop, a sampling frame of producers
and other actors can be made directly by snowball methods.
The entire production cycles must be taken into account in estimating the sown area.
The methodology for capturing indicators of livestock is generally multidimensional
depending on the type of livestock considered.
To track the effect or impact of the project, it is necessary to establish a panel of holder that is
followed regularly before measuring the final impact. This approach is essential
It is necessary to carry out the same survey similar to the baseline in order to measure the
impact of the project. The method of impact assessment is based on the diff in diff method.
28
VI. List and contact of experts who participated in the editing of the harmonized methodology.
NAMES EXPERTISE E-MAIL ADRESS
KOUADJO YAO Jean Marc :
Lecturer-researcher at ENSEA, responsible for surveys and survey practice.
[email protected] [email protected]
Dr NIAMBELE YOUSSOUF SAMBA
Lecturer at Atlanta Technical college, responsible for impact assessment [email protected]
Dr FE Doukoure Charles :
Lecturer-researcher at ENSEA, responsible for economical matters
KANGA Désiré : Lecturer-researcher at ENSEA, responsible for survey practice and the processing of statistical data
NAYO AnkouviMawoudoudji :
Lecturer-researcher at ENSEA, responsible for the processing of statistical data, impact assessment and the new technologies of data collection.
[email protected] [email protected]
Gbahiluc Lecturer at ENSEA, Agricultural statistician . [email protected]