ciff.org

Fostering research for policy and practitioners: Lessons and Opportunities

Symposium on Cohort and Longitudinal Studies

October 14, 2014

Urbanisation

Education Nutrition Climate Change

Energy transformation

HFCs

School-based Deworming

Pre-primary Education

Stunting and wasting

Strategic Priority Areas

(SPA)

Land Use

Evidence, Measurement,

Evaluation

Strategy and Partnerships

Finance and Operations

Health

Perinatal mortality

PMTCT

Adolescent Reproductive Health

SECTORS

BUSINESS SUPPORT

Our mission: To demonstrably improve the lives of children in developing countries by achieving large-scale, sustainable impactOur mission: To demonstrably improve the lives of children in developing countries by achieving large-scale, sustainable impact

CIFF seeks to achieve transformational impact at the SPA/ Sector level. We are currently developing a framework that will allow us to assess collective impact at the organisational level.

All elements of the value chain will need to be covered but not necessarily by CIFF

Changing the paradigm @ global level? @ regional level? @ national level?

TIME

IMPACT

Transformation involves a systemic approach looking across the value chainTransformation involves a systemic approach looking across the value chain

ResearchResearch

Programme evaluationProgramme evaluation

• Creating an enabling environment via policy change, advocacy for evidence, resources political commitment

Policy and advocacy evaluation

Policy and advocacy evaluation

Capacity building assessment

Capacity building assessment

1. We embed measurement throughout the lifecycle of an investment, with a clear (but flexible) plan on when and how data will be generated to inform key decisions

2. We have a fit for purpose approach to evaluation such that the purpose determines design and methods used

3. We ensure complementarity of monitoring and evaluation data to increase relevance and efficiency in data generation, using (and often strengthening) national data systems

4. We ensure independence of our evaluations, to increase objectivity and credibility of our evaluation findings both internally and with the broader development community

5. We engage with data throughout implementation so that we not only invest where there is greatest potential for impact but that the data is used to optimise our ability to deliver maximum impact

Simple but critical questions that guide our measurement•What do we want to know? (prioritisation of inquiry)•How will we know? (methodology, source)•When will we know? (timelines for decisions)•For whom is the information? (audience/s)

Simple but critical questions that guide our measurement•What do we want to know? (prioritisation of inquiry)•How will we know? (methodology, source)•When will we know? (timelines for decisions)•For whom is the information? (audience/s)

CIFF’s approach to evaluationsCIFF’s approach to evaluations

Portfolio of 57 investments totalling $560mln

Portfolio spread by value chain:

Innovation/ design/ evidence: 8 investments

Pilot /model development: 8 investments

Delivery at scale: 21 investments

Systems change: 20 investments

Our portfolio indicates that we directly support a vast number of independent evaluations where data is/ will be used to achieve implementation at scale or achieve systems change, often through influencing policy change, advocacy for more financial resources and political commitment.

Current portfolio spread: By value chainCurrent portfolio spread: By value chain

$78.214%

$411.273%

$48.39%

$22.44%

Systems change Scale Pilots Innovation

Factors that have enhanced uptake of research/ evaluation findings into the policy space, and which have led to implementation at scale

Factors that have enhanced uptake of research/ evaluation findings into the policy space, and which have led to implementation at scale

A quick review of 71 investments (current and closed) indicate that we have achieved some form of policy change or scale up in a few investments

Where successful:•Political commitment existed to a large extent•Financial resources were already planned for or CIFF provided ‘seed’ funding and/ or co-ordinated funding from other partners•Independent evaluation data provided further impetus•Implementing partners had credibility to influence change and communicate evidence

Where we have failed:•Political commitment was uncertain (topic was not of sufficient priority to stakeholders)•Data was not robust enough to influence discussions•Context shifted during implementation (burden of disease, political environment or newer evidence)•No upfront thinking on how to achieve scale up, should intervention prove impactful

Living Goods – Reducing under 5 mortality in UgandaLiving Goods – Reducing under 5 mortality in Uganda

Proof-of-concept investment designed to reduce under-5 mortality by 15% and build a financially viable model that can be sustained over time. Livings Goods and BRAC hired, trained and deployed over 1000 Community Health Promoters (CHPs) to go door-to-door to over 100,000 households to diagnose, refer and treat under-5 cases of malaria, diarrhoea and pneumonia as well as offer pre-natal and post-natal counselling, care and referrals to pregnant women.

The investment was impactful and delivered on the transformational proposition. A cluster-based RCT showed that mortality went down by 25% in intervention areas compared to the control. We are finalising the cost-effectiveness calculations. Data shows that branches are able to recover 70% of their costs, paving a promising path towards sustainability. CHPs’ income has averaged $10-20 per month. Significant interest in CIFF and other Foundations to scale up model. Model is already being scaled up by Living Goods in parts of Kenya,

Factors contributing to scale-up potential:

•Robust evaluation data that tested both impact and financial viability•Investment provided a possible solution to increase health access - sustainable community health delivery•Implementing partner (LG) has the capacity, networks and financial resources to catalyse scale up

Catching up on basic skills: The remedial education experience in GhanaCatching up on basic skills: The remedial education experience in Ghana

Three government agencies, with support from a technical team, tested different approaches to remedial education using assistants to assess whether education impacts can be achieved with government led implementation. Project was evaluated using a RCT.

Results show that (1) teaching assistants make a positive impact on learning outcomes; (2) impact is enhanced when using a differentiated approach; (3) effects accrue to higher grades; (4) differences in impact come from variability in implementation more than from programme design; and (5) Where implementation overcame systemic challenges, effects sizes were higher. Cost-effectiveness analyses are being finalised.Programme has not been scaled up yet, despite significant findings using robust evaluation methodology. Possible reasons:

•Shift in political context •No financial stake by government – does this reduce incentive?•No clear-cut, scale-able solution – probably needs some more testing to determine which regions it is most suited for•Contextual factors undermine solution (e.g., assistant absenteeism, lack of school infrastructure, low teacher compliance, insufficient monitoring and support)

Above lessons are being applied in Kenya, where we are doing implementation research to understand how to scale up a remedial programme, with Government putting in significant financial resources

• How do we define scale? How much attention to ‘quality’? What should come first?

• Evidence is necessary but not sufficient to influence policy (evidence-based policy or policy-based evidence?)

• Policy influence does not necessarily imply implementation, let alone with quality standards – what influences each of these target audiences (policy makers and practitioners) varies

• There exists a tendency for research to pay insufficient attention to costs or scalability in their inquiry or analyses

• Fundamentally, there is lack of ‘demand’ for research and evaluations in many of the contexts we work in

([Evaluation] is not a field where Say’s Law obtains: evaluation supply does not generate its own demand

This might include many facto rs, - o fte n the y include : the time line ss o f the re se arch finding s; the urg e ncy o f the issue ; po litical mo me nts; re so urce s and re adine ss fo r impleme ntatio n; co mmunicatio ns - the way the re se arch is pre se nte d (and to who ), and re latio nships with ke y acto rs

ChallengesChallenges

• Be purposeful about research and evaluations we support – have a clear ‘theory of change’ from data through to policy and/ or implementation that can be tracked routinely

“The value of evaluation (and research?) must ultimately be judged by its usefulness in helping to improve outcomes for target beneficiaries”

• Include an objective in our research and evaluations that will evaluate costs (this may require expertise of a different organisation)

• Twin research proposals with ongoing/ planned investments that can apply the evidence

• Learning from other disciplines/ sectors on achieving policy change or implementation through evidence

• Investing more in communicating evidence from our own research/ evaluations to wider audiences – throughout an investment lifecycle

• Partner with others as we certainly cannot achieve transformation alone“Partnerships are critical, because it will take the combined efforts of the many to successfully tackle big challenges: such as preventing child and maternal deaths and removing climate changing carbon from the atmosphere.”

How are we approaching these challenges at CIFF?How are we approaching these challenges at CIFF?