RESULTS  MEASUREMENT  

Proving and Improving Results

Proving  Results   Improving  Results  

M&E  system  capable  of  credibly  demonstra5ng  programme  

results  

M&E  system  supports  improved  

design  and  implementa5on  of  

interven5ons  

System  Requirements  

ü Well  defined  indicators  ü Robust  measurement  tools  ü Robust  monitoring  plan  ü Baselines  ü Credible  a>ribu5on  strategy  ü Credible  aggrega5on  strategy  

ü M&E  system  generates  5mely  and  relevant  informa5on  for  management  ü Knowledge  management  

M&E in a Market Systems Context

•  Market  development  programmes  explicitly  acknowledge  that  systems  are  complex  

•  M&E  needs  to  integrate  complexity  into  the  way  it  funcEons,  including  accommodaEng  for:  

•  FormulaEng  and  tesEng  assumpEons,  piloEng  intervenEons,  proceeding  by  trial  and  error  

•  Scaling-­‐up  intervenEons  that  work,  modifying  or  dropping  intervenEons  that  do  not  work  

•  Providing  credible  evidence  on  sustainability,  systemic  change  and  poverty  reducEon  

Implications

•  TradiEonal  approaches  to  M&E  are  insufficient  in  market  systems  context:  

•  Measure  results  only  at  the  middle  and  end  of  the  programme  

•  M&E  divorced  from  management  funcEon  •  M&E  system  uses  LogFrame  only  •  “EvaluaEon”  seen  as  an  external  acEvity  only  •  Use  of  Randomised  Control  Trials  

Impact Profiles High-­‐le

vel  Impa

ct  

Time  Project  LifeEme  

Direct  Delivery  Market  Development  

SCALE   SUSTAINABILITY  

• Market  development  programmes  will  have  different  impact  profiles  • Can  take  1-­‐3  years  to  deliver  high-­‐level  impact  • Progress  can  be  bumpy!  

RESULTS MANAGEMENT FRAMEWORK

    • Theory  of  Change  

    • Results  Chains  

    • Monitoring  Plans  

WHAT IS A THEORY OF CHANGE?

 Theories  of  Change  posit  the  underlying  causes  that  are  driving  poor  performance  and  resul5ng  limited  benefits  accruing  to  the  poor.      Theories  of  Change  then  provide  an  overview  of  the  systemic  change  needed  to  improve  performance,  with  a  specific  focus  on  how  the  poor  will  benefit  from  improved  performance.  

Monitoring & Results Measurement Cycle

1.  Draw  Interven5on  Results  Chain  

2.  Define  Indicators  

3.  Establish  Baseline  

4.  Predict  Results  

5.  Data  Collec5on  

6.  Data  Analysis  

8.  Knowledge  Management  

7.  Report  Results  

Theory  of  Change  

Results Chains

•  Results  Chains  map  the  systemic  change  process  of  an  intervenEon    

•  RCs  are  more  granular  and  flexible  than  LogFrames  •  RCs  are  a  key  design  and  monitoring  tool    

Best  PracEce:  • RCs  developed  jointly  by  the  M&E  team  and  intervenEon  team  

• Every  RC  includes  boxes  to  capture  systemic  change:  sustainability,  copying  /  crowding-­‐in  

Results Chains: Basic Structure

System-­‐level  change  

Ac5vi5es  

Enterprise  performance  

Impact  on  poor  men  and  women  

• Programme  facilita5on  ac5vi5es  

• Improved  linkages  between  system  actors  • Improved  market  access  • Changes  in  formal/informal  rules  • Introduc5on  of  new/improved  interconnected  systems  (e.g.  policy,  inputs,  finance,  ICT)  

• Improved  produc5vity/compe55veness  • E.g.  improved  yields  for  small-­‐holder  farmers  

• Increased  incomes;  more  secure  livelihoods  • Job  crea5on  • Economic  empowerment  

Project  iden5fies  Millers  willing  to  adopt  new  supply  

chain  strategy  

Project  facilitates  strategic  planning  process  on  new  supply  chain  prac5ces  

Miller  adopts  new  supply  chain  prac5ces  (incl  performance  

incen5ves  and  transparent  pricing)  

Miller  delivers  embedded  services  to  small-­‐holders  

Small-­‐holders  access  new  services  

Small-­‐holders  apply  new  prac5ces  

Targeted  small-­‐holders  achieve  higher  yields  &  sales  

Targeted  small-­‐holders  increase  incomes  

Millers  increase  incomes  

SYSTEMIC  CHANGE  

Copying  by  non-­‐project  supported  Millers  

Copying  by  non-­‐targeted  Small-­‐holders  

ACTIVITIES  

SYSTEM

 CHA

NGE

 EN

TERP

RISE  

PERFORM

ANCE

 IM

PACT

 

Establishing Baseline

•  To  assess  change,  you  need  to  establish  a  baseline  for  each  indicator  

•  Common  mispercepEon  is  to  conduct  one  big  baseline  study  of  the  target  populaEon  at  the  beginning  of  the  programme  

•  “Treatment”  group  is  hard  to  define  at  start    •  AUribuEon  consideraEons  can  be  difficult  to  understand  at  start  

•  SomeEmes  necessary  to  conduct  retrospecEve  baselines  

•  Best  pracEce:  targeted  baselines  conducted  for  each  intervenEon,  based  on  clear  understanding  of  change  process  

Predicting Results

•  Predict  results  for  each  box  in  the  Results  Chain:  •  What  change  do  you  expect?  •  By  when?  

•  PredicEng  results  helps  with:  •  Planning  data  collecEon  •  TesEng  the  underlying  logic  and  making    •    assumpEons  explicit  •  Monitoring  progress    

•  Best  pracEce:  predicted  impact  is  used  to  decide  whether  to  proceed  with  an  intervenEon  or  not  –  intervenEons  that  offer  low  predicted  Value  for  Money  (VFM)  are  dropped  or  modified  

Predicting Results: Rice Example

Project  supports  2  Millers  

1  Miller  adopts  new  model  

Miller  offers  services  to  200  Smallholders  during  first  season  

50  Smallholders  accept  service  condi5ons    

Average  yields  increase  by  20%  for  50  Smallholders  during  first  season  

Average  income  increase  by  20%  for  50  Smallholders  during  first  season  

Applying Degrees of Evidence

MSE/Household  level:  May  be  able  to  aUribute  changes  to  project  intervenEon  through  staEsEcal  and  qualitaEve  research  

Market  system  level:  Use  qualitaEve  techniques  to  determine  preponderance  of  evidence  on  effecEveness  at  market  level  

Triangulate  evidence  from  all  sources  

 

Summary Check-list

•  M&E  system  is  capable  of  “proving”  and  “improving”  results  •  Every  intervenEon  has  a  clear  results  chain  •  Specific  and  measureable  indicators  are  defined  for  each  box  

in  the  results  chain  •  Baselines  are  established  for  each  indicator  •  Predicted  results  produced  for  each  intervenEon  •  Every  intervenEon  has  a  measurement  plan,  with  

appropriate  aUribuEon  strategy  •  Data  analysis  is  Emely  and  accessible  •  Management  decisions  are  made  on  the  basis  of  evidence/

results  •  Results  are  aggregated  (controlling  for  overlap)  and  reported  

in  an  accessible  way