Introduction to Impact Assessments in Microinsurance

I

01

In a nutshell: impact assessments in

microinsuranceRalf Radermacher and Katja Roth

16

1.1. Introduction

Everyone is confronted with various risks in life – but the consequences of such risks are often more severe for the poor than for better-off house-holds. When confronted with the finan-cial consequences of calamities like illness, accidents, death, or loss of agricultural or other produce, poor households often have to take actions that harm their productivity and pros-perity prospects. These actions could include: borrowing at high interest rates, selling productive assets, taking children out of school or sending them to labour, compromising on a balanced diet, or leaving illnesses untreated. Such responses to risk impede house-holds on their way out of poverty or may even force them deeper into poverty.

Over the past 15 years, microinsurance has increasingly been seen as an alter-native and potentially more efficient way for the poor to manage their risks. Stated simply, microinsurance is insur-ance targeting the poor. It is meant to be offered at an affordable price, and so comes with some limitations in risk coverage. Microinsurance can be offered for all kinds of risks and by all kinds of providers. Often, communi-ties or non-government organisations (NGOs) arrange these programmes themselves, but commercial insurers are increasingly engaging in the provi-sion of microinsurance (see chapter 2

“What is microinsurance?”). They recognise that the largest untapped insurance market in terms of clients is amongst the poor, with an estimated market size of four billion people (Swiss Re 2010). Governments can also be providers of microinsurance – or take an interest in the promotion and regu-lation of the market, either as part of a financial sector development or as part of a social protection approach.

It is essential to learn more about the effects of insurance on the insured and the communities in which they live. This is particularly true for many gov-ernments and funders of development, for whom the provision of insurance is not an end in itself, but rather a tool to achieve certain development objectives like, for example, reduction of poverty, improved access to health care and, through this, fewer maternal and infant deaths.

The effect microinsurance has on the lives of the poor or the larger society is what we call its impact. These changes can be positive or negative, direct or indirect, and intended or unintended. Furthermore, they can happen before (i.e., ex-ante) or after (i.e., ex-post) the occurrence of insured events and

17In a nutshell: impact assessments in microinsurance

can affect not only the insured, but also their household members, their communities, or other populations (Radermacher et al. 2012; OECD 2002). Hence, microinsurance can change the life of people even if they never make a claim or – even more – the lives of people that are not even insured.

Regarding these effects, one needs to differentiate between impacts and outcomes. Impacts are an end in itself, whilst outcomes are the bridge expected to lead to the desired impacts. Impacts are usually expected to be long-term effects of an intervention, whilst outcomes are normally achievable in a short term (OECD 2002).

Measuring the impact of a single microinsurance scheme can provide information about its absolute impact or its effectiveness, i.e., whether the insurance makes a difference for those covered and whether these differences meet the objectives one was aiming at with the insurance. It is, however, also of key interest to know whether one approach to microinsurance has a larger or different kind of impact than another approach – or how

microinsurance compares to other risk management techniques. Such comparative information, called relative impact or efficiency, can be immensely helpful for the design of future microinsurance programmes and investment decisions of donors and governments. Furthermore, collecting information on the costs of the implementation can provide insights into the cost-efficiency of programmes with similar goals.

However, to use the information to com-pare cost-efficiency between different schemes or microinsurance and other risk management approaches, it is im-portant that the programmes are at a similar stage of implementation. On one hand, pilot projects may be compara-tively more expensive than larger roll-outs, as economies of scale and learn-ing effects are likely to reduce costs per insured. On the other hand, the quality of implementation is usually higher in the pilot, potentially leading to higher impacts per insured compared to pro-grammes taken to scale. Generally, ef-fects of scale and scope need to be con-sidered when comparing programmes with regard to their efficiency.

18

When examining the effects of microinsurance, it is useful to understand the effect on the insured. Sometimes information on the whole community offered the insurance, no matter whether they chose to enrol or not, is also helpful. Moreover, effects can differ for different subgroups of a sample, such as women versus men, children versus adults, or extremely poor versus better-off households. Understanding these differences can provide insight on equity and equality effects, which might also be the desired effects of an insurance programme. Donor organisations might particularly aim to improve the well-being of specific vulnerable subgroups. Hence, their prosperity compared to other subgroups might be in their focus. For example, it might be of interest to know how a health microinsurance programme helps to improve the health of girls compared to that of boys, since boys often get better access to care.

Understanding the impact of microin-surance is useful and necessary. The big question, however, is how to mea-sure this impact.

1.2. Measuring impact

Microinsurance can have various effects on its clients and their community. Defining the aspects of interest is the first step in measuring impact. A core set of topics will be derived from the initial

objectives for which a microinsurance scheme was set up. This set of topics should also include potential effects that are unintended or negative. A causal chain is established to describe how the insurance is expected to unfold the impact in focus. Such a causal chain is called theory of change and is described in more detail in chapter 3 “Why and how does impact happen?” Formulating such a theory should be based on a detailed understanding of the context and functioning of the insurance scheme. Not only do potential – intended and unintended – outcomes and impacts need to be included, but also the inputs and outputs of the scheme, specifically on aspects of insurance processes and the insurance product, as these are likely to influence the kind and magnitude of the impact seen.

Take, for example, a health microin-surance scheme aimed at reducing maternal and infant deaths by promot-ing institutional deliveries. A theory of change for this scheme should take into account as its inputs: all scheme design decisions with respect to benefits cov-ered, educational activities, and how the scheme is delivered (and to whom). The output of this is then the particu-lar insurance scheme with a benefit

19In a nutshell: impact assessments in microinsurance

package taken up by a certain number of women in a particular region. It is assumed that women understand the scheme. As direct financial barriers for accessing these services are removed or lowered by the scheme, the utilisa-tion of these services is expected to go up (outcome), which assumes that health services are available and ser-vice providers work with the insurance agency as required. This higher utilisa-tion is expected to lead to a reduction in maternal and infant deaths (impact), which assumes the health services are provided in a timely manner and are of sufficient quality. Differences in the in-put, i.e., scheme design and execution, might lead to different output, outcome, and impact. For example, including the women’s spouses into an information campaign on the importance of ob-stetric care might lead to much bet-ter outputs, outcomes, and impacts as compared to targeting the informa-tion campaign on the women alone. Processes thus matter a lot for under-standing impact and must be consid-ered in a good impact assessment.

Some outputs and outcomes measur-able on the day-to-day level are captured in social performance indicators (Sandmark 2013), which

complement the financially oriented key performance indicators (Wipf and Garand 2010). With their detailed and comparable information on scheme design and results, such indicators can be very helpful in impact research design and interpretation of results. Particularly in comparative studies they can provide a set of structured criteria for comparing observed differences in outcomes and impact. When it has been decided what to mea-sure, a careful research design needs to ensure that the impact of insurance is actually captured, separating it from impact of other interventions or external events. This cannot be done with simple before and after comparisons. Instead, the situation (of the insured households, the communities, etc.) with microinsurance needs to be compared to the counterfactual, i.e., how the situation would be for the same persons (or households, communities, etc.) without microinsurance. Since the same household cannot be simultaneously insured and uninsured, a comparison group, as similar to the treatment group as possible, needs to be established. Chapter 4 “What can we learn from impact assessments?” elaborates these research design aspects in more depth.

20

Several approaches are possible for obtaining a reliable comparison group. Ideally, impact evaluations of programmes should be planned before the programme implementation starts. This can allow for the use of randomised controlled trials, described in detail in chapter 5 “Experimental designs”. In such a design, the insurance is offered to a randomly selected set of households or individuals and not offered to a control group of comparative nature. Based on the assumption that observable and unobservable differences between individuals in these groups are equally distributed, the effects observed after the insurance implementation can be at-tributed to the insurance itself. However, such a randomised trial, considered the most robust evaluation approach, requires not only planning the evaluation before the rollout of the insurance, but also requires close cooperation between evaluator and implementer – and often more management and coordination effort for the implementer. This might not be possible in every case, and frequently evaluations are also commissioned when a project is already running for several years. Hence, a randomised trial is not possible in all situations, or may not be the solution of choice. Other statistical methods can also be used to enhance comparability of data on the insured and uninsured. Chapter 6 “Non-experimental design methodologies for quantitative analysis” explains these approaches.

In their core nature, the techniques described above are mainly built on quantitative data. Such data are crucial for making any statement about the magnitude of effects. However, quantitative approaches alone frequently fall short of explaining some of the phenomena observed and confirming the causal chain described in the theory of change or in exploring aspects which might have escaped the attention of the researchers. Qualitative research approaches, described in chapter 7 “Qualitative designs”, can fill these gaps and contribute to answering questions on why (explanatory) and what (explorative). An ideal impact evaluation would combine both quantitative and qualitative techniques and is thus referred to as “mixed methods” (see chapter 8 “The case of mixed methods for impact evaluation in microinsurance”).

With the research focus and method decided, the evaluation can be further operationalised. A key question here is how to measure the research aspects of interest, i.e., defining indicators that adequately describe the aspects of interest. All indicators need to be precisely described and understand-able to respondents to ensure that the collected information can be useful. Chapter 9 “Defining indicators for im-pact assessments” explains aspects linked to the choice of indicators and pro-vides examples of indicators commonly

21In a nutshell: impact assessments in microinsurance

used. As the number of evaluations of microinsurance schemes is still limited, the indicators used to date may not necessarily correspond to the main areas of interest in the microinsurance space. Yet, including these indicators in the evaluation makes it easier to compare studies. Agreeing on core in-dicators for the microinsurance space is thus of key importance, so that fu-ture studies can include these indica-tors and enhance the body of evidence on topics considered of key importance. Chapter 10 “Core outcomes, impacts and indicators for microinsurance” suggests a number of core indicators, based on a Delphi study amongst mi-croinsurance experts.

The selected indicators constitute the basis for the questions to be included in data collection instruments (also chapter 10). Transforming indicators into questions includes careful consideration of the targeted respondents, the precise formulation of the question, and a balancing of what eventually can be included in the research. The design of the research tool and selection of the respondents has direct influence on the planning of field work. For certain respondents, seasonality will affect availability, e.g., due to harvest or temporal migration. Further, social and cultural norms must be taken into account when selecting the research team and planning a smooth entry into the target area.

The amount of training and capacity building will need to be decided to ensure the research tools are properly administered – and we recommend planning for close supervision, enabling the evaluator to react to any problems encountered. Such close involvement and supervision can help increase data quality, which is essential for obtaining reliable answers to the research aspects in focus. Chapter 11 “Operationalising impact evaluation: from theory to practice” develops these aspects further.

The primary answer provided by any given study is whether the specific insurance scheme under scrutiny has resulted in a certain impact. As chapter 12 “Drawing conclusions” describes, the data needs to be checked for validity and related back to the theory of change. Also, conclusions should include prac-tical recommendations for the specific scheme. However, generalisation of the insights or their transferability to other settings, like other regions or target

22

groups, is often intended as well. High external validity is required for this. It is also desirable to make the insights of the evaluation available publicly, to allow others to learn from a scheme’s results—regardless of whether the results are desirable or undesirable. To allow such learning, information on the context in which the scheme operates and how the evaluation was conducted is essential. A common reporting standard can help in achiev-ing this, as described in chapter 13 “Reporting and disseminating find-ings”, which also provides an overview about dissemination options.

A growing body of evidence increases the reliability of insights gained by different studies, providing a basis for systematic reviews. When these studies use common indicators and provide a comparable set of background information, conclusions on what works and why can be derived from an analysis across studies. Such analyses are called systematic reviews, which may include a statistical synthesis using meta-analyses (see chapter 14 “Systematic reviews”). Reviews provide answers on specific questions by drawing on the entire body of evidence available. They can be a key tool for providing reliable guidance for the development of the microinsurance sector, but need robust studies as a foundation. This book is written to support the creation of more and better

quality evidence on the impact of microinsurance – evidence which in turn should then be used to create an even more impactful microinsurance sector.

References

Organisation for Economic Co-operation and Development (OECD). 2002. Glossary of key terms in evaluation and results based management. Paris: OECD Publications. http://www.oecd.org/dac/evaluation/2754804.pdf.

Radermacher, R., H. McGowan, and S. Dercon. 2012. What is the impact of microinsurance? In: Protecting the poor: A microinsurance compendium. Volume II, ed. C. Churchill and M. Matul. 59-82. Geneva: International Labour Organization.

Sandmark, T. 2013. Social performance indicators for microinsurance – A handbook for microinsurance practitioners. Luxembourg: Appui au Développement Autonome (ADA). http://microinsurancenetwork.org/publication/fichier/Social_Performance_Indicators_web_2.pdf.

Swiss Re. 2010. Microinsurance—risk protection for 4 billion people. Sigma 6. http://media.swissre.com/documents/sigma6_2010_en.pdf.

Wipf, J. and D. Garand. 2010. Performance indicators for microinsurance – A handbook for microinsurance practitioners. 2nd ed. Luxembourg:

Appui au Développement Autonome (ADA).http://microinsurancenetwork.org/publication/fichier/KPI_MI_Handbook_v2_EN.pdf.