routes to resilience...routes to resilience: lessons from monitoring braced 2 reflections in brief...
TRANSCRIPT
ROUTES TO RESILIENCELESSONS FROM MONITORING BRACEDPaula Silva Villanueva and Catherine Gould
Reflection paper
CONTACT THE AUTHORS
Paula Silva Villanueva is the director of ResilienceMonitor
and leads the BRACED Knowledge Manager Monitoring and
Results Reporting team. Over the past 10 years she has developed
a number of monitoring and evaluation frameworks and systems
in the field of climate change adaptation, disaster risk reduction
and sustainable development.
@ResilienceMonit
Catherine Gould is a senior programme manager working in the
BRACED Knowledge Manager Monitoring and Results Reporting
team. She has over 10 years’ experience designing programme
monitoring and evaluation frameworks and developing practical
tools, templates, processes and guidance for development,
emergency response and resilience building programmes.
ACKNOWLEDGEMENTS
This reflection paper was written by Paula Silva Villanueva and Catherine Gould,
based on our experiences of monitoring change in the BRACED programme.
We wish to acknowledge critical contributions from Florence Pichon, as well as
Emily Wilkinson, Blane Harvey, Katie Peters, Fran Walker and Dave Wilson from
the BRACED Knowledge Manager. The authors are also grateful to the M&E
and project leads of the BRACED Implementing Partners and to Annie Bonnin
Roncerel and Jim Djontu of the BRACED Fund Manager, for openly sharing their
experiences and reflections. The paper has benefited from critical review from
Robbie Gregorowski and Katie Peters of the Knowledge Manager and Derek
Poate (external). The donor DFID have also provided their feedback and discussed
how to ensure the lessons are taken up and applied, both within BRACED and
other similar programmes. Finally, we thank Charlotte Rye and Clare Shaw of
the Knowledge Manager for their support in the publication process.
Contents
Reflectionsinbrief 2
1. Introduction 7
1.1 Whatisthispaperabout?� 7
1.2 Whoisthispaperfor?� 8
1.3 Howhavewereflected?� 9
1.4 Howisthispaperstructured?� 10
2. MonitoringandevaluationinBRACED 11
2.1 TheBRACEDM&Eframework� 11
2.2 Theconcepts� 12
2.3 ReportingprogressinBRACED� 15
3. Whathavewelearntsofar? 17
3.1 Translatingconceptsintopractice� 18
3.2 RollingoutM&Eframeworks� 21
3.3 Reportingonresilience� 23
3.4 Aggregatingandsynthesisingdataatscale� 25
4. HowcanBRACEDbuildonthislearning? 28
4.1 Questionforfurtherreflection� 31
Annex1:MonitoringandevaluationinBRACED 33
Annex2:BRACEDtheoryofchange 34
Annex3:Project-to-programmesynthesismethodology 35
Annex4:FurtherKnowledgeManagerandImplementingPartnerreflections
onexperiencesofreportingagainsttheBRACEDM&Eframework 36
Annex5:BRACEDM&E‘Infrastructure’ 44
Annex6:RollingouttheBRACEDM&Eframework 45
2ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� 2ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED�
Reflections�in�brief
The�basis�for�reflection
Thispapersharesinsights,reflectionsandlessonslearntfromdesigning,
implementingandreportingagainsttheBuildingResilienceandAdaptation
toClimateExtremesandDisasters(BRACED)programme’sMonitoringand
Evaluation(M&E)framework.TheBRACEDprogrammeaimstobuildthe
resilienceofupto5millionpeoplevulnerabletoclimateextremesanddisasters
andsupportsinternational,regionalandlocalorganisations,workingin15
consortiaacross13countriesinEastAfrica,theSahelandSouth-EastAsia.
Tounderstandwhatworksandwhatdoesnotinbuildingclimateanddisaster
resilience,theBRACEDKnowledgeManagerisdevelopingandtestingavariety
ofresiliencemeasurementandmonitoringapproachesandframeworks.The
BRACEDM&Eframeworkisdesignedtoenabledatacollectionandevidence
generationtotrack,measureandunderstandtheprocessesofchangethatlead
toclimateanddisasterresilience.
Eachyear,theBRACEDprojectImplementingPartnersandtheKnowledge
Manager’sMonitoringandResultsReportingteamaddressthecriticalquestion:
‘How are BRACED projects contributing to building resilience?’Theanswerhas
beencapturedinourcompanionsynthesisreport–‘Routes to resilience: insights
from BRACED year 1’.�Thisreflectionpaperwaswrittenfollowingthecompletion
ofthefirstannualsynthesisreportandprovidestheMonitoringandResults
Reportingteam’sreflectionson‘What lessons have we learnt from the
monitoring and results reporting efforts to date in BRACED?’
DuringthefirstyearofBRACED,wehaveaddressedthefollowingM&Echallenges:
• movingfromconceptstopractice
• rollingoutaprogramme-levelM&Eframeworkto15projectsworking
across13countries
• triallingqualitativereportingapproachesatproject-andprogramme-level
• aggregatingandsynthesisinghighlycontextuallyspecificdata.
Theseexperienceshavegeneratednewinsightsintohowtoapproachthe
monitoringandresultsreportingofaresilience-buildingprogrammeatthescale
ofBRACED.
Developingprogramme-levelM&Eframeworksforresilience-building
programmesisarelativelynewareaofwork,withlimitedexperiencetodraw
on.ReflectionabouttheBRACEDM&Eframeworkisthereforeacriticallearning
stepforBRACEDitselftoimproveM&Epracticeandevidencegenerationwithin
theprogramme.Italsoprovidesanexcitingopportunitytocontributetobuilding
theknowledgebaseonresiliencemonitoringandmeasurementforthewider
community.Wehopethatthereflectionssharedinthispaperwillcontribute
toongoingandfutureresilience-buildingprogrammes.
3ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED�
Lessons�learnt�so�far
Inyear1,BRACEDprojectImplementingPartnershaveembracedanewway
ofmonitoringandreportingchange.Wehavelearntagreatdealasaresult
oftakingaprogramme-levelviewofhowresilienceisbeingbuiltinBRACED.
ThekeylessonsemergingfromourBRACEDexperiencetodateinclude:
1. Translating�concepts�into�practice:�Measuring�progress�on�resilience�
cannot�be�done�with�one�‘simple’�indicator.�It�requires�qualitative�and�
explanatory�frameworks�that�contextualise�results�against�shocks�and�
stresses,�as�well�as�the�wider�context�projects�operate�within.�Thereisa
riskoflosingandobscuringcriticallearningaboutresiliencebuildingifwe
measureresilienceusingjustoneindicator.�Understandingthedeterminants
ofclimateanddisasterresilienceiscomplexandtherearen’tanyready‘yes’
or‘no’answers.
Moredetailedlessonsontranslatingconceptsintopracticecanbefound
insection3.1ofthispaper.
2. Rolling�out�M&E�frameworks:�There�are�different�options�for�rolling�
out�programme-level�M&E�frameworks�and�systems,�but�each�comes�
with�its�own�trade-off.�Optionsandtrade-offsincludedecisionsabout
thetypeandlevelofsupporttoprovidetoprojectpartners.Therolling
outofprogramme-levelM&Eframeworksandsystemsmustfindabalance
betweenlight-touchandresource-intensiveoptions.Theyalsoneedto
allowforcontinualadjustmentsbasedontheemergingbodyofknowledge
andexperienceregardingthemonitoringandmeasuringofresilience.In
BRACED,theKnowledgeManagerwassetupaftertheprojectlogframes,
theoriesofchangeandM&Eplansweredefined.EstablishingtheBRACED
programme-levelM&Eframeworkwouldhavebeeneasierifithadbeen
developedatthesametimeasthe15BRACEDprojects’M&E.
MoredetailedlessonsonrollingoutM&Eframeworkscanbefound
insection3.2.
3. Reporting�on�resilience:�Qualitative�and�explanatory�frameworks�offer�
an�opportunity�to�complement�resilience�indicators.�However,�if�we�are�
totruly�engage�with�these�frameworks,�we�need�to�shift�mindsets�from�
accountability�to�learning-oriented�M&E.�Engagingwithqualitativeand
explanatoryframeworksrequiresM&Epracticestogobeyond‘business
asusual’andaccountability-drivenexercises.M&Eexpertsandproject
managersalsoneedtoengageinmorerefinedandcomplexdatacollection
andanalysisprocessesthaninatraditionalprogramme.
Moredetailedlessonsonreportingonresiliencecanbefoundinsection3.3.
4ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED�
4. Aggregating�and�synthesising�data�at�scale:�Synthesising�and�
aggregating�data�while�retaining�context�specificity�requires�time,�
resources�and�thorough�synthesis�methodologies.Qualitativeand
explanatoryframeworkscallforexhaustivesynthesisprocessesthatare
abletodealwithcomplexdataanalysis,varyinglevelsofdataquality
andself-reportingbias.Thislengthenstheleadtimebetweenproject-level
annualreportingandprogramme-levellearning,whichmaylimitthefindings’
potentialimpactonprogrammeandprojectdecision-making.
Moredetailedlessonsonaggregatingandsynthesisingdataatscalecan
befoundinsection3.4.
How�can�BRACED�build�on�this�learning?
BRACEDisnearlytwoyearsintoitsthree-yearimplementationtimeframe
andproject-andprogramme-levelM&Eisalreadysetupandestablished.
Therearethereforesomelimitationstowhatcanbeadaptedandachieved
intheremainderoftheprogramme.Inthiscontext,oursuggestionsbelow
areforboththeBRACEDprogrammeandothersimilarinitiatives.
“Genuinelyunderstandingresilienceinpracticemeansmovingawayfromalogframe-drivenand‘accountability’-focusedM&Eculture”
Monitoringresilience-buildingeffortsandreportingontheirprogressis
challenging.M&EforresilienceprogrammingisstillnascentandBRACEDis
learning-by-doing.Akeymessageemergingfromthispaper,togetherwithits
companionprogramme-levelsynthesisreport,isthatgenuinelyunderstanding
resilienceinpracticemeansmovingawayfromalogframe-drivenand
‘accountability’-focusedM&Eculture.Movingforward:
• Project�Implementing�Partners�should�enhance�their�ongoing�monitoring�
and�results�reporting�efforts�by�taking�a�more�reflective�and�critical�
approach.Thiscouldchallengeprojectassumptionsandwillbuildabetter
understandingofhowtobuildclimateanddisasterresilienceinfragileand
vulnerablecontexts.
• The�programme-level�Monitoring�and�Results�Reporting�team�should�
consider�how�to�encourage�this�critical�Implementing�Partner�reflection�
and�dialogue.Therearelimitstowhatreportingtemplatesalonecanachieve
inthisregard.WethereforeplantoprovidefurthertrainingtoImplementing
Partners,alongwithlight-touchhelpdesksupport.
5ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED�
• Programmes�like�BRACED�need�to�find�and�resource�efficient�ways�of�
achieving�a�sufficient�level�of�reflection�and�learning�for�the�benefit�of�
both�project-�and�programme-level�evidence�generation.Ideally,the
programme-levelM&Eframeworkshouldbedesignedinconjunctionwith
theproject-levelframeworks.
“Outcome-levelindicatorsneedtobecomplementedbysystematicmonitoringandevaluationofresilienceinthecontext
ofactualshocks”
Tobetterunderstandthestabilityofoutcome-levelchangesovertimeandhow
communitieslearnand‘bouncebackbetter’fromdisasterevents,outcome-level
indicatorsneedtobecomplementedbysystematicmonitoringandevaluation
ofresilienceinthecontextofactualshocks.Movingforward:
• Implementing�Partners�are�in�a�unique�position�to�contribute�to�
knowledge�about�how�to�quantify�the�number�of�people�whose�resilience�
has�been�built�(KPI�4)�at�the�project level.TheMonitoringandResults
Reportingteam,togetherwithwidermembersoftheKnowledgeManager
andtheBRACEDFundManager,shouldfurtherexploreoutcome-level
resilienceindicatorsindifferentcontexts:theadvantagesanddisadvantages,
aswellasopportunitiesandtrade-offs.
• When�designing�and�funding�similar�programmes�in�the�future,�the�
Department�for�International�Development�(DFID)�should�adopt�a�
pragmatic�and�realistic�view�on�the�feasible�level�of�outcome-level�
data�and�evidence�generation�in�a�three-year�programme�like�BRACED.
Resilience-buildingeffortsarenotonlycomplex,butalsoinvolveprocesses
ofchangethattaketimetomaterialise.Prioritisingannualdatacollection
effortsagainstquantitativeindicatorsmaycomeatthecostoflosing
criticalevidenceaboutwhatworksandwhatdoesnotinbuilding
resiliencetoclimateextremesanddisasters.
• Programmes�like�BRACED�should�consider�having�a�diverse�set�of�
methodologies�and�analysis�in�place�for�interrogating�quantitative�
outcome-level�resilience�indicators.Theyshouldbepragmaticabout
whatsortofoutcome-leveldataandinformationcanbeexpectedin
athree-yearperiod.
6ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED�
Whilemuchattentionhasbeengiventoproject-levelapproachesto
monitoringandmeasuringresilience,programme-leveleffortsfaceaunique
setofchallenges.Todate,thereisbothlimitedliteratureandexamplesfrom
otherprogrammesaddressingthesechallenges.InBRACED,wehavebeen
learning-by-doingonanongoingbasis.Movingforward:
• The�Monitoring�and�Results�Reporting�team,�together�with�Implementing�
Partners,�should�consider�ways�to�further�capture�their�monitoring�and�
results�reporting�experiences�within�BRACED.�Thiswouldbenefitboth
BRACEDandotherexistingandfutureresilience-buildingprogrammes.
• Programmes�like�BRACED�should�also�share�experiences�and�contribute�
to�building�knowledge�in�this�relatively�new�area�of�work.
Question�for�further�reflection
Thispaperaimstocontributetoongoinglearningaboutresilienceprogramming
andtoinitiatediscussion.WewishtoengageBRACEDImplementingPartners,
thebroaderBRACEDKnowledgeManager,theBRACEDFundManager,DFID
andwideraudiencesinthefollowingcriticalquestion,whichemergesasaresult
ofourcollectivereflectionsandlearningsofar.Together,weshouldcontinue
toansweritthroughouttheBRACEDprogramme.
How�complex�does�M&E�for�resilience�need�to�be?TheBRACEDM&E
frameworkbringstogetherthreedifferentlensesintotheanalysisofproject-
andprogramme-leveldata.Experiencetodate,bothfromImplementingPartners
andtheBRACEDKnowledgeManager’sMonitoringandResultsReportingteam,
indicatesthefollowing:
• Whiletheframeworkprovidesamultidimensionalviewintothevarious
levelsofcomplexity,itrequiressignificanttime.
• Italsonecessitatesdifferent,non-traditionalwaysofengagingwithdata.
• Itdoesnot–andcannot–providesimple‘yes’or‘no’answersabout
whetherresiliencehasbeenbuiltand,ifso,how.
Asoutlinedinourcompanionreport,‘Routes to resilience: insights from
BRACED year 1’,theessenceof‘resilience’isthatchangeandprogressare
notlinear–soresultsreportingshouldnotbelineareither.TheBRACEDM&E
frameworkiscomplexenoughtobeabletounderstandBRACEDresilience-
buildingefforts,butcoulditbemadetobemoreuser-friendly,whilestill
retainingthecomplexityandnuancesofresilience-buildingprojects?
7ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� IntroductIon
1.1�What�is�this�paper�about?TheDFID-fundedBRACEDprogrammecomprises15field-basedresilience-
buildingprojectsacross13countriesintheSahel,EastAfricaandSouth-East
Asia.1Theseprojectsareimplementedby15ImplementingPartners,whose
performanceanddeliveryisoverseenbytheBRACEDFundManager.The
programmealsohasadedicatedKnowledgeManager2taskedwithgenerating
andconsolidatinglearningabouttheactionsthatworkbesttostrengthen
communityresiliencetoclimateextremesanddisastersinavarietyofcontexts.
M&EactivitiesinBRACEDareundertakenatboththeprojectlevelandthe
programmelevel.(SeeAnnex1forfurtherdetails.)
AkeyareaoftheKnowledgeManager’sworkisgeneratingknowledgeabout
monitoringandevaluationpracticeinacomplexresilience-buildingprogramme.
Tothisend,itisdevelopingandtestingavarietyofresiliencemeasurement
approachesandframeworksthroughasetofmonitoringandresultsreporting,
1 www.BrAcEd.org
2 BrAcEdKnowledgeManager(2016)‘LearningaboutresiliencethroughtheBrAcEdprogramme:AnintroductiontotheroleoftheBrAcEdKnowledgeManager’.BrAcEdKnowledgeManagerinformationleaflet.London:odI.
Image:neilPalmer,cIAt
1.INTRODUCTION
8ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� IntroductIon
evaluationandresearchactivities.Asacorepartofthiswork,theKnowledge
ManagerMonitoringandResultsReportingteamhasdevelopedtheBRACED
programmetheoryofchange(seeAnnex2)andrelatedM&Eframework.The
BRACEDM&Eframeworkwasdevelopedtoestablishprogramme-levelevidence
andlearningabouthowBRACEDprojectsarebuildingresilience.
ThecomponentsoftheM&Eframeworkwerenewanduntestedwaysof
monitoring,measuringandunderstandingresilience-buildingefforts.Duringthe
firstyearofBRACED,theframeworkhasbeenadoptedbyprojectImplementing
PartnersandappliedtotheirM&E.Attheendofyear1,ImplementingPartners
providedsystematicqualitativeandexplanatoryreportingagainsttheBRACED
M&Eframeworkforthefirsttime.TheMonitoringandResultsReportingteam
havesinceundertakenaprogramme-levelsynthesisofallproject-levelyear1
annualreports.
ThispaperpresentstheMonitoringandResultsReportingteam’sreflections
sofar,fromapplyingtheBRACEDM&Eframework,overseeingyear1project-
levelreportingandcompletingtheprogramme-levelsynthesisofthisdata.The
paperaddressesthequestion:‘Whatlessonshavewelearntfromthemonitoring
andresultsreportingeffortstodateinBRACED?’Thefindingsoftheprogramme-
levelsynthesisitselfhavebeencapturedinaseparatereport,‘Routes to resilience:
insights from BRACED year 1’.3Thecompanionsynthesisreportanswersthe
questionof‘HowareBRACEDprojectscontributingtobuildingresilience?’
andexploresBRACEDprogrammeprogresstodateagainsttheBRACED
M&Eframework.
1.2�Who�is�this�paper�for?Designingandimplementingprogramme-levelM&Eframeworksforresilience-
buildingprogrammesisarelativelynewarea,wherelimitedexperienceexists.
ReflectionabouttheBRACEDM&Eframeworkisthereforeacriticallearning
stepfortheBRACEDprogramme.Itprovidesanexcitingopportunity,notonly
toimproveM&EpracticewithinBRACED,butalsotocontributetobuilding
theknowledgebaseonresiliencemonitoringandmeasurementforthewider
community.Thispaperisaimedat:
• Stakeholders�internal�to�BRACED:�project�Implementing�Partners,�
the�Knowledge�Manager,�Fund�Manager�and�the�donor�DFID.Forthis
audience,weprovideasetoflessonsonhowchangecanbemonitored,
measuredandunderstoodintheBRACEDprogramme.Thesereflections
shouldbeusedtoenhancetheBRACEDM&Esystemforyears2and3of
theprogramme.Theywillinformongoingmonitoringandresultsreporting
atboththeprojectandprogrammelevels,aswellaswiderBRACED
KnowledgeManagerworkontrackingandmeasuringresilienceoutcomes.
3 SilvaVillanueva,P.,Gould,c.andPichon,F.(2016)‘Routes to resilience: insights from BRACED year 1’.BrAcEdKnowledgeManager.Synthesisreport.Brighton:Itad.
9ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� IntroductIon
• Outside�BRACED:�M&E�practitioners,�donors�and�programme�
staff�with�an�interest�in�designing,�implementing,�monitoring�and�
measuring�resilience-building�efforts.�Forthisaudience,weoutline
thepracticalexperiencesoftheBRACEDprogrammesofarintermsof
conceptualising,rollingoutandreportingonhowresilienceisbeingbuilt.
Wealsoprovidelessonsandreflectionsthatarerelevantandapplicable
toothersimilarprogrammes.
Thispapershouldbereadalongsideourcompanionsynthesisreport,‘Routes
to resilience: insights from BRACED year 1’.
1.3�How�have�we�reflected?InordertogatherlessonsaboutBRACEDexperiencestodateinrelationto
theM&Eframeworkanditsassociatedreportingtemplates,theMonitoring
andResultsReportingteamundertookthefollowing:
• AsystematicreviewandsynthesisofeachprojectImplementingPartner’s
annualreportagainsttheM&Eframeworkfortheprogramme-levelsynthesis
report(seeAnnex3).Here,we:
• identifiedthedatagapsandinconsistenciesintheuseofdefinitions
andindicators
• lookedatthequalityofthedataprovided,identifyingcommon
practicesandchallengesacrossproject-levelreports.
• AconsultationwiththeImplementingPartnersthemselves,gathering
informalfeedbackontheirexperienceofmonitoringandreportingagainst
theBRACEDM&Eframework(seeAnnex4).WealsoaskedtheFund
Managerfortheirexperiencesofgatheringandaggregatingquantitativedata
alongsidetheKnowledgeManager’squalitativework.
• Areflectiononwhatworkedwellandnotsowellinthedesign,rolling
outandimplementingtheM&Eframework.Wealsoheldaone-dayinternal
reflectionworkshopontheM&EframeworkwithbroaderKnowledgeManager
teammembers.Thiswasbasedontheinitialfindingsinrelationtothe
questionaddressedintheprogramme-levelsynthesis:‘How are BRACED
projects contributing to building resilience to climate extremes and disasters?’.
Thispaperisnotanin-depthtechnicalassessmentoftheBRACEDM&E
framework.Rather,itisareflectionpiecetoshareemergingreflectionsand
lessonstodate.TheKnowledgeManager’smonitoringandresultsreporting
effortssitwithinalargerBRACEDM&Einfrastructure(seeAnnex5).Detailed
analysisoftheframework,alongwithmonitoringandmeasuringresilience
inBRACED,ispartofawidercollectiveeffort.Thelessonsinthispaperare
limitedtotheMonitoringandResultsReportingteam’sareaofwork.
10ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� IntroductIon
1.4�How�is�this�paper�structured?Insection2,wedescribetheBRACEDM&Eframeworkandprovidean
overviewoftheBRACEDmonitoringandresultsreportingapproachand
system.Insection3,wepresentourreflectionsandlessonsidentifiedfrom
theBRACEDmonitoringandresultsreportingeffortsundertakenduring
year1.Finally,insection4,weprovideconclusionsandrecommendations
forBRACEDstakeholdersandsuggestareasforfurtherconsideration.
11ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� M&EInBrAcEd
2.1�The�BRACED�M&E�frameworkMonitoringandevaluationactivitiesareundertakenatboththeprojectand
programmelevel.FurtherinformationonhowM&Eissetupandmanagedacross
theBRACEDprogrammeisinAnnex1.
TheBRACEDlogframe4andtheoryofchangearethetwocornerstone
documentsoftheBRACEDM&Eframework.Twomandatoryprogramme-wide
keyperformanceindicators(KPIs)oftheInternationalClimateFundwerein
placefromthebeginningoftheprogrammeaspartofproject-levelM&E:
• ‘The number of people supported by BRACED to cope with the effects of
climate change’ (KPI1–anoutputlevelindicatorofBRACEDlogframe).
• ‘The number of people whose resilience has improved as a result of BRACED
support’(KPI4–anoutcome-levelindicatoroftheBRACEDlogframe).
Theprogrammetheoryofchange(seeAnnex2)andBRACEDM&Eframework
weresubsequentlydevelopedbytheKnowledgeManagertofurtherunpackthe
assumptionsandprocessesbehindtheprogrammelogframe.Theyareintended
4 https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/328271/revised-BrAcEd-Interim-Logframe-23June14.xlsx
Image:neilPalmer,cIAt
2.MONITORING�AND�EVALUATION�IN�BRACED
12ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� M&EInBrAcEd
toenablecomplementaryqualitativedatagenerationinordertoexplainand
contextualisethenumbersandtounderstandhowresilienceisbeingimproved.
ThetheoryofchangeandM&EframeworkdothisbysituatingBRACEDefforts
inthebiggerpictureofchangeandencouragingcriticalthinkingandreflection
aboutchangepathways.Theyalsosurfaceandarticulateassumptionsregarding
howchangehappens.
“Thechallengewastodevelopacoherentprogramme-levelframeworkacross
interventions,flexibleenoughtoberelevantacrossdifferentsocio-political,geographical
andclimaticcontexts,whileretainingrobustnessandcoherence”
TheoriginalpurposeoftheBRACEDM&Eframeworkwastoprovidea
programme-levelvisionofchangeandensureacommonlanguageandminimum
alignmentofmonitoring,resultsreportingandevaluationeffortsacrossBRACED,
whileacknowledgingandaccommodatingproject-specificM&Eapproachesand
plansindifferentcontexts.EachprojecthaditsowntheoryofchangeandM&E
frameworkandapproach.Thechallengewastodevelopacoherentprogramme-
levelframeworkacrossinterventions.Theframeworkneededtobeflexible
enoughtoberelevantacrossanumberofdifferentsocio-political,geographical
andclimaticcontexts,whileretainingrobustnessandcoherence.
TheframeworkwasrolledoutacrossBRACEDprojectsduringthefirst
monthsoftheprogramme,throughtheprovisionofguidance,5feedbackand
one-to-onesupport.(Formoreinformationonhowwerolledouttheframework,
seeAnnex6.)Inadditiontothetwoprogramme-widequantitativemeasuresof
resilience,BRACEDImplementingPartnersmonitorandreportprojectresults
againstthethreemaincomponentsoftheBRACEDtheoryofchangeonan
annualbasis:‘AreasofChange’,the‘3As’and‘EvaluativeMonitoring’.
2.2�The�concepts
Tracking�resilience�pathways�through�‘Areas�of�Change’
ThroughreviewingtheassumptionsandcausalchainsunderpinningBRACED
projecttheoriesofchange,weidentifiedfourareaswherechangehadto
happenforBRACEDtoachieveitsoutcomes.Thesewereandcontinuetobe:
(a)Knowledgeandattitudes;(b)Capacitiesandskills;(c)Qualityofpartnerships;
and(d)Decision-makingprocesses.The‘AreasofChange’representwhatisoften
5 WrittenguidancewasprovidedbytheKnowledgeManagertoallprojectImplementingPartnersintheBrAcEdM&EGuidancenotes.
13ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� M&EInBrAcEd
referredtoasthe‘missingmiddle’inprojectlogframes.Theyenableustobetter
understandthesetofprocesseslinkingprojectoutputstoresilienceoutcomes
and,ultimately,impactsonhumanwell-being.Theyalsoprovidetheframework
forassessmentoftheBRACEDprogramme’strajectorytowardsimpact.
ThefourAreasofChangearedefinedas:
• Changes�in�knowledge�and�attitudesinrelationtoresilience-building,
inordertofurtherstrengthenpoliciesandpractices.
• Changes�in�the�capacities�and�skillsofnationalandlocalgovernment,
civilsocietyandtheprivatesectortomanagetherisksofclimateextremes
anddisasters.
• Changes�in�the�quality�of�partnershipstodeliverinterventions.
• Changes�in�decision-making�processesthroughinclusiveparticipation,
asonekeyaspectofaresilientsystem.
How do we track and report change?TheAreasofChangepayparticular
attentiontothestakeholdersandactorsinvolvedinBRACEDbyaskingtwo
simplequestionsacrossthem:Who is changing? And how? Theyuseanadapted
outcomemappingapproachtomeasurechange.Thisutilisesgraduated
progressmarkersfrom‘expecttosee’to‘liketosee’to‘lovetosee’changes.
MonitoringandresultsreportingagainstthefourAreasofChangebuilds
knowledgeandunderstandingaboutthe‘pathwaystoresilience’inBRACED.
MoredetailsontheBRACEDAreasofChangeareavailableintheBRACEDM&E
GuidanceNotes6(Note3).
Understanding�resilience�outcomes�through�the�‘3As’�and�transformation
InBRACED,resilienceisunderstoodasthecapacityofasystemtochangeand
adaptinthecontextofmultipleandinteractingshocksandstresses.Ananalysis
ofnearly50existingresilienceframeworksidentifiedthreeinterlinked‘capacities’
astheoutcomesoftheAreasofChangeprocesses.Theseare:thecapacityto
Anticipate,AbsorbandAdapttoshocksandstresses(the3As).The3Asaimto
measureandunderstandchangesinresilienceoutcomesatdifferentlevelsand
withregardtodifferentkindsofshocksandstresses.Insteadofspecifyingaset
ofindicatorstomeasure‘resilience’,the3AsframeworkenablesImplementing
Partnerstodevelopcontext-specificindicatorswiththeirrespectivestakeholders.
ItisanorganisingtooltoanalysetheoutcomesthatBRACEDprojectsmaybe
achieving.Monitoringandresultsreportingagainstthe3Asbuildsunderstanding
6 SilvaVillanueva,P.,Gould,c.,Gregorowski,r.,Bahadur,A.(2015)‘BrAcEdprogrammemonitoringandevaluationguidancenotes’.BrAcEdKnowledgeManager.BrightonItad.
14ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� M&EInBrAcEd
of‘resilienceoutcomes’inBRACED.MoredetailsontheBRACED3As
areavailableintheBRACEDM&EGuidanceNotes(Note4)andthepaper
‘The3As:TrackingresilienceacrossBRACED’.7
Transformation,ontheotherhand,isnotregardedasatypeofcapacitythat
contributestoresilienceinthesamewayasthe3As.Rather,itrepresentsan
outcomerelatedtotheholisticandfundamentalwaysinwhichpeople’scapacity
toanticipate,absorbandadapttoshockscanbebuilt,reshapedandenhanced.
TheBRACEDtheoryofchangehypothesisesthatBRACEDislikelytobemore
transformationalinitsfocuscountriesifitachieveschangesthatarecatalytic,
atscaleandsustainable.
How do we track and report change?BRACEDprojectsreportagainsttwo
InternationalClimateFundkeyperformanceindicators(KPIs)attheoutcomelevel:
• KPI4is‘the number of people whose resilience has been improved’.In
BRACED,thisnumberisderivedfromcollatingproject-levelreportingat
theoutcomelevel,whereImplementingPartnershaveidentifiedtheproject-
specificoutcomeindicatorsthatwilldemonstratechangesinresilience.
Theseindicatorshavebeentaggedinrelationtotheresiliencecapacities–
anticipatory,absorptiveandadaptive.Insomecases,Implementing
Partnershaveidentifiedadditionalindicatorsfortransformativechange.
ImplementingPartnersdecidehowtoweighttheindicatorsaccordingto
theirprojecttheoriesofchangeandhowtheyexpecttoseeprogressin
buildingresilience.
• Thesecondoutcome-levelindicatorisKPI15:‘the extent to which interventions
are likely to have a transformational impact’.InBRACED,transformation
isaself-assessedqualitativeindicator.Trackingtransformationisdifficult
–suchchangescanbedeliberatelyengineered,butareoftenbeyond
thescopeofasingleintervention.TheKnowledgeManagerdevelopeda
scorecardtosupportImplementingPartnersinmonitoringthelikelihoodof
transformationalimpactattheprojectlevelinacomparableway.Through
thisscorecard,theprogrammeM&Esystemdoesnotdefinetransformational
outcomesexante,butinsteadtracksthelikelihoodoftransformationagainst
threepillarsidentifiedintheliterature:policy,empowermentandinnovation.
Itincludescertaincharacteristicsoftransformation,including‘catalytic’,‘at
scale’and‘sustainableoutcomes’.ImplementingPartnersareencouragedto
reportchangestheyinterprettoberepresentingthesepillarsorcharacteristics
oftransformation.
7 Bahadur,A.V.,Peters,K.,Wilkinson,E.,Pichon,F.,Gray,K.andtanner,t.(2015)‘the3As:trackingresilienceacrossBrAcEd’.BrAcEdKnowledgeManagerWorkingPaper.London:odI.
15ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� M&EInBrAcEd
Contextualising�results�through�‘Evaluative�Monitoring’
BRACEDoperatesinsomeofthemostfragileandchallengingcountriesin
theworld.8Whiletheprogrammeisnotexplicitlyconflictorsecurityfocused,
manyoftheprojectsarebeingimplementedinacontextaffectedbyconflictor
instability.However,thereisanassumptionwithintheprogrammetheory
ofchangethatBRACEDpathwaystoresilienceandresilienceoutcomesare
achievedwithinanenablingenvironmentatthelocal,sub-nationaland
nationallevels.ThroughEvaluativeMonitoring,ImplementingPartnersexplore
thenatureoftheprevailingcontext–specifically,thegovernancestructure,
decision-makingprocesses,incentivesandrelationshipsbetweendifferentgroups
andindividuals–andtheextenttowhichthiscontextsupportsorconstrains
change.Thisbuildsknowledgeandunderstandingabout‘resilienceincontext’
inBRACED.MoredetailsonBRACEDEvaluativeMonitoringareavailablein
theBRACEDM&EGuidanceNotes(Note5).
How do we track and report change?BRACEDreportingincludesEvaluative
Monitoringasacriticalpartofthereflectionprocess.EvaluativeMonitoring
bringsanevaluationlenstothereportingexercises.Itdoesthisbysituatingthe
datacollectedwithinanunderstandingoftheprevailingcontext.Theaimisto
shedsomelightonprojects’risksandassumptionsandbeexplicitaboutthe
factthatchangeoccursasaresultofmanyactorsandfactors.Monitoringand
reportingquestionsinclude:What are the key contextual factors (at local, sub-
national and national) that may enable or constrain change in the project? How
are these contextual factors enabling or constraining change from the project?
Have they contributed to any unexpected outputs or outcomes?
2.3�Reporting�progress�in�BRACEDCombined,theAreasofChange,3AsandEvaluativeMonitoringenablethe
BRACEDprogrammetotrack,measureandunderstandtheprocessesofchange
thatleadtoclimateanddisasterresiliencetospecificshocksandstresses,in
specificcontexts.Underpinningthisapproachistheneedtocriticallyreflecton
projectandprogrammetheoriesofchangeand,inturn,questionthem.Thisis
intendedtofosterinternallearningandbuildarobustevidencebaseregarding
howandwhyinterventionsaresuccessfullycontributingtoimprovingclimate
resilience(ornot).
TheKnowledgeManagerMonitoringandResultsReportingteamdeveloped
asetofreportingtemplatesduringyear1toenableBRACEDImplementing
Partnerstoreflectonandreportannuallyagainst:
• theresiliencecapacitiesbeingbuilt
• thechangeprocessesunderway
8 Wilkinson,E.andPeters,K.(Eds.)(2015)‘climateextremesandresiliencepovertyreduction:developmentdesignedwithuncertaintyinmind’.BrAcEdKnowledgeManagerresearchPaper.London:odI.
16ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� M&EInBrAcEd
• ifandhowthecontextisaffectingthesechanges
• whattheimplicationsare,ifany,fortheprojecttheoryofchangedesign.
Attheirownrequest,ImplementingPartnerssubmitasingleannualprogress
reporttoboththeFundManager(againsttheprojectlogframe,foronward
reportingtoDFID)andtheKnowledgeManager(againsttheprojecttheory
ofchange,forevidencegeneration).
Figure1summarisesthestructureforreportingagainsttheM&E
framework.ImplementingPartnersalsoreporttheiruseofclimateand
weatherinformationandthelikelihoodoftransformativechange,aspart
ofthelogframereporting.ForthespecificquestionsaskedofImplementing
Partners,seethereportingtemplates.9
Figure�1:�BRACED�Knowledge�Manager�M&E�Framework
9 theM&EframeworkreportingtemplatesformthesecondoftwopartsoftheBrAcEdprojectannualreports.Inthefirstpart,ImplementingPartnersreportprogressagainsttheirlogframeindicatorstotheBrAcEdFundManager.Inthesecondpart,theyexplore,explainandcontextualisetheseresults.
How we monitor and assess change
Key questions
What is the long term change and development impact the project is seeking to support?
Theo
ry o
f cha
nge
What are the underlying assumptions about how this change would happen and did they hold?
How will change happen? Who are the key actors?
What are the contextual drivers that may enable or constrain change?
3As (Anticipatory, Absorptive & Adaptive)
Areas of Change
Evaluative Monitoring
Reflection and learning – testing theory of change
17ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� WhAthAVEWELEArntSoFAr?
ThissectionpresentstheMonitoringandResultsReportingteam’sreflectionsand
lessonsidentifiedtodate,fromdevelopingandimplementingtheBRACEDM&E
frameworkandconductingthefirstprogramme-levelsynthesisofBRACEDproject
resultsreporting.Inparticular,wereflectonthefourmainchallengesfacedwhen
developingaprogramme-levelM&EframeworkforaprogrammelikeBRACED,
whichcomprises15uniqueprojectsworkingacross13differentcountries:
1. Moving�from�concepts�to�practice:translatingnovelconceptsintopractical
monitoringframeworksthatareapplicableandrelevantacrossprojects.
2. Rolling�out�programme�M&E�frameworks�and�systems�to�the�project�
level:designingacoherentprogramme-levelframeworkthatisflexible
enoughtoberelevantacrossanumberofdifferentsocio-political,
geographicalandclimaticcontexts,whileatthesametimeretaining
itsrobustnessandcoherence.
3. Trialling�qualitative�reporting�approaches�at�project-�and�programme-
level:developingasetofreportingtemplatestogatherdataagainsteach
oftheM&Eframeworkcomponentstoenablestandardisedandcomparable
reportingbyallImplementingPartners.
Image:uSAId/nepal
3.WHAT�HAVE�WE�LEARNT�SO�FAR?
18ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� WhAthAVEWELEArntSoFAr?
4. Data�aggregation�and�synthesis�at�the�programme�level:aggregating,
synthesisingandanalysingself-reportedqualitativeandexplanatorydata
from15projectsinawaythatgeneratesevidenceandlessonsabouthow
resilienceisbeingimprovedattheprogrammelevel.
3.1�Translating�concepts�into�practice1.�Encapsulating�resilience�concepts�and�thinking�into�quantitative�measures�
limits�what�outcome-level�indicators�may�tell�us.Tobetterunderstandthe
outcome-levelfiguresreportedagainstKPI4,BRACEDappliesthe3Asframework
toanalysethenatureofresilienceoutcomes.Thishasenabledtheprogramme
tounderstandhowBRACEDprojectshavebeenaddressinganticipatoryand
absorptivecapacitiestodate,butitremainsunclearhowtheywillcontribute
tolongertermadaptivecapacities.OthercomponentsoftheBRACEDM&E
frameworkhavealsorevealedhowtheinterplayofsocio-culturalandgovernance
dynamicsshapesresiliencepathways.Suchcriticaldetailswouldhavebeen
missedthroughtheuseoftheKPI4indicatoralone.
Tracking�progress�against�KPI�4�–�Emerging�insights�from�BRACED�
Each�BRACED�project�adopts�a�different�approach�to�measuring�resilience.�
This�is�context�specificandbasedontheirconceptualisationofresilience,
althoughalluseacompositeindexwithconstituentindicators‘fed’through
theuseoflargesamplehouseholdsurveys.ImplementingPartnersuseKPI4
andtheguidanceprovided,10whichencouragesthemtocontextualisetheir
approachtomeasurement.Thismeansresultsarenoteasytocompare–
alimitationofboththemethodandtheapproachtomeasurement.The
varietyinmethodologieshasbeenaparticularchallengefortheBRACEDFund
Manager,whohaveneededtoengagewithandunderstandeachproject’s
methodologyindetailinordertounderstandandaggregatethedata.
There�is�significant�risk�of�obscuring�potentially�rich�detail�in�the�data�by�
reporting�only�a�number.ThereisalsoscopeforreportingerrorswhenKPI4
numbersaregeneratedacrosshouseholds,communities,regionsandprojects,
whicharethenaggregatedtotheprogrammelevel.Suchsimpleaggregation
acrossprojectsmaynotstanduptoscrutiny.Thisisduetomultiple
interpretationsoftheKPI4guidance(WilsonandYaron,2016).11Quantitative
approachestoKPI4measurementshouldthereforebecomplementedby
qualitativedatagatheringfortriangulationandasanexplanatorytoolfor
morein-depthandnuancedunderstanding.
10 KPI4guidance:https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/328254/BrAcEd-KPI4-methodology-June2014.pdf
11 Wilson,d.andYaron,G.(2016)‘Layingthefoundationsformeasuringresilience’.BrAcEdKnowledgeManagerWorkingPaper.Brighton:Itad.
19ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� WhAthAVEWELEArntSoFAr?
The3Asframeworkhasenabledalevelofconsistencyandcomparability,
sheddingsomelightonthenatureofresilienceoutcomes.However,experience
todatealsotellsusthatthe3Ashavebeenfoundtobeausefulanalyticaltool
ratherthanoneformeasuringchangesinresilience.
Similarinsightshaveemergedfromearlyexperiencesintrackingtransformation:
Tracking�transformation�(KPI�15)�–�Emerging�insights�from�BRACED
Over-reporting�on�transformation�is�common.�Intheiryear1reporting,
ImplementingPartnersprovidedevidenceofsmallpositivechangesas
potentiallytransformational.Incrementalchangesmaybebestunderstood
withincontext,butitisimportantnottodilutetheconceptoftransformation
byreportingall changeastransformative.Adheringtothepillarsof
transformation(strategicplanningandpolicy;leadership,empowermentand
decision-makingprocesses;andinnovativeapproaches)canclarifythetypesof
transformationchangethattheBRACEDprogrammeaimstoachieve.
Capturing�transformation�that�is�‘catalytic’,�‘at�scale’,�‘and�sustainable’,�
was�challenging�in�year�1�project-level�M&E�reporting.�Thescorecardfor
transformationattemptedtocapturethedifferent,oftencountry-specific,
dimensionsoftransformationalchange,whileremainingsufficientlysimple
tobeunambiguous.However,theseconceptsdidnotappeartobeuseful
inprobingformoredescriptiveinformationinprojectreporting.Often,
theevidenceprovidedunderonecharacteristiccouldeasilyfitunder
another.Thesedefinitionsmaynotbehelpfulforexplainingthenatureof
transformationalchangeattheprojectlevel.Ultimately,transformational
changesrequireacritical masstoovercomepolitical,marketandothersocio-
economicandpoliticalfactorsthatcannotbeaddressedbyasingleproject.
A�qualitative�outcome�indicator�for�tracking�transformation�enables�the�
monitoring�(not�measuring)�of�the�likelihood�of�transformation,�relative�to�
expected�change.�Reportingtransformationalchangecannotbeaggregatedat
theoverallprogrammelevelinthesamewayasKPI4.Attheprogrammelevel,
resultscanbesynthesised.Theycanalsoenabletheidentificationofpatterns
andtrendsasameanstoassessoverallprogress,andtoteaseoutlessons,
ratherthanformaviewontheexpectedtransformationalimpactofBRACED.
Thereisagrowinginterestinunderstandingandcreatingtransformational
changesthroughprogrammesthatbuildresiliencetoclimateextremesand
disasters.TheBRACEDprogrammeprovidesacollectiveopportunitytolearn
abouttransformationalchangeinordertoimproveresilienceanddevelopment
outcomesbothwithinBRACEDprojectsandexternally.However,thedriveto
quantifyandqualifytransformationalimpactplacesunrealisticexpectations
attheprojectlevel.
20ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� WhAthAVEWELEArntSoFAr?
BeyondthechallengesandlimitationsofKPI4andKPI15,thisyear’s
programme-levelsynthesisofBRACEDprojectreportsalsohighlightsthe
unaddressedchallengeofsystematicmonitoringofprogressagainstsuch
indicatorsinthecontextofshocksandstresses.Todate,itremainsunclear
inBRACEDhowwecanbestcontextualiseresultsagainsttheclimaticcontext
withinwhichprojectsoperate.Moreworkisrequiredatboththeprojectand
programmelevelstofullyunderstandthepracticeofmonitoringandreporting
progressinBRACEDresilience-buildingeffortsinthefaceofshocksandstresses.
“MoreworkisrequiredatboththeprojectandprogrammelevelstofullyunderstandthepracticeofmonitoringandreportingprogressinBRACEDresilience-buildingeffortsinthe
faceofshocksandstresses”
2.�Assessing�resilience�and�transformation�involves�tracking�progress�against�
multifaceted�processes.�Concepts�and�definitions�are�critical.Conceptualising
resilienceintermsofcapacitiesputshumanagencyatthecentreofresilience
building.Effortstoquantifyresilienceneedtotakecarewhendescribingchanges
inresiliencecapacities,asthesewillultimatelybecontingentonpeople’s
attitudesandchoices.TheAreasofChangeframeworkattemptstounpack
thelesstangiblebutvitalprocessesthroughwhichBRACEDprojectsinfluence
changesinattitudesandpractice.
Findingsfromtheyear1programme-levelsynthesishighlightthatImplementing
PartnerstendedtoreportagainsttheAreasofChange‘Knowledgeandattitudes’
and‘Capacitiesandskills’withadegreeofoverlap.Thissuggestsalackof
understandingofthedifferencebetweenthetwoandaneedformoreclarity.
Capacity,inparticular,isadifficultconcepttoframe,monitorandevaluate.
Whenreportingagainst‘Partnerships’,someImplementingPartnersreferto
formalpartnershipswithamemorandumofunderstanding,whileothersrefer
moretorelationshipsofcollaborationandcoordination.InBRACED,project-
levelpartnershipsdifferenormouslyinbothscaleandscope,soitisimportant
toclarifythetypeofpartnershipbeingreferredto.Meanwhile,information
providedon‘Decision-making’variedsignificantlyfromoneImplementing
Partnertoanotherafteryear1,withmostfocusingontherepresentation
ofwomenindecision-makingstructuresratherthanthebroaderissuesof
inclusionandparticipation.
Differinginterpretationsanddefinitionsacrossprojectshavenotjustbeen
anissueforqualitativedatareportinginBRACED.Theyhavealsopresenteda
challengefortheFundManagerwhenaggregatingprojects’quantitativedatato
theprogramme-levelBRACEDlogframe.Whilemuchattentionhasbeengiven
todefinitionsandM&Emethodologiesfortrackingchangesinresiliencelevels,
ourexperiencetodaterevealsthatasimilaremphasisshouldbeplacedonclearly
definingconceptssuchascapacity,knowledgeandinclusivedecision-making.
21ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� WhAthAVEWELEArntSoFAr?
3.�Understanding�the�processes,�outcomes�and�context�of�resilience�is�
complex�and�there�are�no�straightforward�‘yes’�or�‘no’�answers.�Oneofthe�
BRACEDprogramme’sambitionsistoconstructanevidencebaseofwhatworks
andwhatdoesnotinbuildingclimateanddisasterresilience.Thisrequires
intensiveM&Eeffortscomparedtoothermoretraditionalprogrammes,bothin
termsofquantifyingchangesinresilienceandunderstandinghowthesechanges
haveoccurred.ImplementingPartners,whorelyheavilyonlocalpartnersinthe
field,arefacingcapacityconstraintsinapplyingtheBRACEDM&Eframework
andinmeetinglogframereportingrequirements.Thisisduetothenoveltyofthe
conceptsandframeworksbeingtested,thelevelofrigourandreflectionrequired
toengagewiththese,andtheM&Eframeworkbeingintroducedafterthedesign
ofproject-levelM&E.
Theintegratednatureofresilience-buildingprojectsrequiresintegratedanalytical
frameworks.Inordertobetterunderstandthecausalpathwayslinkingoutputsto
outcomes,aswellasthefactorscontributingtoresiliencebuilding,theBRACED
M&Eframeworkinvestigateschangeprocessesfromdifferentangles.Though
theywereoriginallyintendedtobeusedasacoherentwhole,thecomponent
conceptswithinthisframeworkweredevelopedseparately.Eachaspectwas
designedtounderstandsomethingslightlydifferentabouthowBRACEDprojects
andtheprogrammeasawholearebuildingresilience.Whileeach‘lens’provides
avaluableinsightintohowresilienceisbeingbuilt,theydonotprovidesimple
answerstotheresilience-buildingquestionevenwhentakentogether.
“Understandingtheprocesses,outcomesandcontextofresilienceiscomplexandthereare
nostraightforward‘yes’or‘no’answers”
Ourexperiencetodateindicatesthat‘zoomingin’tospecificAreasof
Changeordifferentresiliencecapacitiescanmeanlosingthe‘biggerpicture’.
Forexample,improvementsinthecapacityofspecificstakeholderstomanage
aspecificshockneedtobeassessedalongsidefeaturessuchastheroleofa
project’spartnersinthatparticularprocessandthesocio-culturalnormsinthat
specificcontext.Whenundertakingtheprogramme-levelsynthesis,wefound
thatunderstandingthecontributingfactorstoresilience-buildingrequiresdata
collectionandananalysisofwhyImplementingPartnersareengagingincertain
activitiesandhowthesearecontributingtobuildingresilience.Thisisdifficult
whendatafromdifferentframeworksisreportedinisolation.Whiledetailed
analysisiscertainlyrequired,theintegrationandtriangulationoffindings,
alongwithreflectiononhowtheyrelatetoeachother,isequallyimportant.
3.2�Rolling�out�M&E�frameworksAnnex6providesdetailsregardinghowtheMonitoringandResultsReporting
teamrolledouttheM&Eframework.Thereareanumberoflessonstoconsider
fortheremainderofBRACED,aswellforothersimilarprogrammes:
22ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� WhAthAVEWELEArntSoFAr?
1.�M&E�designs�at�the�project�and�programme�level�should�be�in�place�before�
implementation�begins.Acommonunderstandingandvisionoftheprogramme
itselfanditsM&EiscriticalforthesubsequentsuccessofanyM&Esystem.The
startingpointfortheKnowledgeManagerMonitoringandResultsReportingteam
whendesigningandrollingoutanappropriateM&Eframeworkwastodevelop
aprogramme-widetheoryofchangetoarticulateBRACEDobjectives,assumptions
andpathwaysofchange.Thisstepwasparticularlyimportant,aseachBRACED
projecthasdifferentdefinitionsandresiliencemeasurementapproaches.
“Learningaboutwhatworksandwhatdoesn’tinbuildingresiliencerequiresownershipandcommitmenttointerrogatingtheprogramme-
leveltheoryofchange”
Learningaboutwhatworksandwhatdoesn’tinbuildingresiliencerequires
ownershipandcommitmenttointerrogatingtheprogramme-leveltheoryof
change.Inthisregard,engagingimplementingpartners,researchers,thedonor
andthefundmanagerindevelopingacommonvisionisrequiredrightfromthe
beginning.However,theBRACEDKnowledgeManagerwassetuponceBRACED
projectshadalreadybeendesignedandapproved,meaningtheprogramme-level
theoryofchangeandM&Eframeworkhadtoberetrofittedtotheexistingproject-
levelM&E.Toaddressthischallenge,theMonitoringandResultsReportingteam
followedaconsultativeprocess,combiningabottom-upandtop-downapproach,
todeveloptheBRACEDprogrammetheoryofchange.Thisincludedareviewof
all15project-leveltheoriesofchangeandlogframes,alongwithaconsultation
withtheprojectImplementingPartners.Attheprogrammelevel,frameworkswere
developedtoenable,asmuchaspossible,thestandardisationofconcepts,analysis
andreportingagainsttheprogramme-leveltheoryofchange.
Combiningabottom-upandtop-downapproachmeantthattheprogramme
couldretainproject-levelcontextspecificity.However,mostImplementing
Partnershadnotplannedorresourcedfordataanalysisandreflectionfor
programme-levelM&Ebeyondthemandatorylogframereportingagainstthe
relevantInternationalClimateFundKPIs.Havingtheprogramme-levelKnowledge
ManagerM&Eteaminplacefromthestartwouldhavehelpedavoidthissituation
andmayhavealsoensuredthatinformationreportingwouldnotbeseenasatop-
downrequirement.Instead,thiswouldpresentanopportunityforcriticalanalysis
andorganisationallearning,informingdecision-makingandimpactassessment
atboththeprojectandprogrammelevels.
2.�There�are�different�options�for�rolling�out�programme-level�M&E�
frameworks�and�systems,�but�each�comes�with�its�own�trade-off.Rolling
outprogramme-levelM&Eframeworksandsystemstoensureaggregationand
comparabilityrequiresagreementonkeyconceptsanddefinitionsaswellas
standardreportingtemplates.Otherwise,project-leveldifferencesinconceptual
framingsandterminologycanmakeitdifficulttomonitorandassessprogress
23ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� WhAthAVEWELEArntSoFAr?
attheprogrammelevel,asfoundbyboththeKnowledgeManagerMonitoring
andResultsReportingteamandtheFundManager.TheBRACEDM&EGuidance
Notessetouttheprogramme-leveldefinitionstoImplementingPartners.
Follow-uptrainingandsupportthenenhancedtheirunderstandingofthese.
However,thissupportwasprovidedinthecontextofproject-levelM&Ealready
beingdesigned.(TheBRACEDKnowledgeManagerwassetupaftertheproject
logframes,theoriesofchangeandM&Eplansweredefined.)
“Rollingoutprogramme-levelM&Eframeworksandsystemstoensureaggregationand
comparabilityrequiresagreementonkeyconceptsanddefinitions”
Optionsandtrade-offsforrollingoutprogrammeM&Eframeworksinclude
decisionsaboutthetypeandlevelofsupportprovidedtoprojectpartners.
WhencomparabilityandaggregationareakeypurposeoftheM&Esystem,
moreresource-intensiveoptionsforproject-levelsupportmaybebetter–onesuch
examplewouldbeM&EtrainingforprojectImplementingPartners,complemented
byongoingone-to-oneinteractionswiththem.Thisisunderlinedfurtherin
thecontextofaresilience-buildingprogramme,asknowledge,capacityand
experiencesarestillemerging.Wehavetakenamoreresource-intensiveapproach
atkeymoments:inthedesignoftheM&Eframeworkanditsapplicationby
ImplementingPartnersto(a)project-levelM&Eand(b)year1reporting.However,
wehavealsousedlessresource-intensiveoptions,suchaswrittenM&Eguidance
andone-offengagementswithgroupsofImplementingPartners.Rollingout
programme-levelM&Eframeworksandsystemsmustfindtherightbalance
betweenbothapproachesandallowforcontinualadjustmenttothegrowingbody
ofknowledgeandexperience.EstablishingtheBRACEDprogramme-levelM&E
frameworkwouldhavebeeneasierifithadbeendevelopedatthesametime
asthe15BRACEDprojects’M&E.
3.3�Reporting�on�resilienceThiswasthefirstyearwhereImplementingPartnerscollecteddataand
reportedaboutchangesinresilienceandtheKnowledgeManagerMonitoring
andResultsReportingteamsynthesisedandanalysedthedata.FromaKnowledge
Managerperspective,thefirstyear’sreportingprocesswentwelloverall.The
M&EframeworkitselfhasbeenfoundtoberelevanttotheworkoftheBRACED
programmeandtheBRACEDprojectsthemselves.ImplementingPartnerreports
wereparticularlyimpressive,giventhat:
• Project-levelM&Esystemswerenotsetuptoreportconsistentlyagainst
theBRACEDM&Eframework.
• Thereportingwasearly,comparedtowhenresultsintermsofresilience
buildingcouldrealisticallybeexpected.
24ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� WhAthAVEWELEArntSoFAr?
• ImplementingPartnerswerecompletingthetemplatesforthefirsttime.
(SeeAnnex4forourfurtherreflectionsonyear1reporting.)
FromanImplementingPartnerperspective,theexperienceofcompleting
thetemplateswashelpfulforinternalreflectionforsomeandaddedvalueto
theirownprojectM&E.Therewere,however,anumberofchallengescited
incompletingthetemplatesagainsttheM&Eframework.Thiswasoftendue
toImplementingPartnersnothavingrespondedtoouroriginalfeedback
toalignproject-levelM&Etotheprogrammelevel.Ithasbeenanongoing
tensionandchallengeforImplementingPartnerstomonitorandreport
againsttheprogrammelevelM&Eframeworkwhenthiswasintroducedafter
projectimplementationbegan.(SeeAnnex4forfurtherImplementingPartner
reflectionsonthefirstyear’sreportsandreportingprocess.)
Thereareanumberoflessonstoconsiderforthereportingandanalysisfor
years2and3:
1.�The�level�of�data,�analysis�and�reflection�required�from�Implementing�
Partners�in�BRACED�is�high.Asalreadynoted,thenatureofthequantitative
andqualitativedatainBRACEDischallengingandgoesbeyondtheM&E
requirementsofmosttraditionalprogrammes.Thisongoingmonitoringwork
andannualreportingisinthecontextofImplementingPartnersalsoundertaking
projectmid-termreviews(mid-year2),wheretheyareencouragedtofurther
understandhowchangeishappeninginmoredetail,buildingontheirfirst
annualreportanalysis.Thisreportingworkloadissignificantandthelevelof
informationandanalysisthathasbeenprovidedbyImplementingPartnersin
theiryear1reportontheM&Eframeworkisadmirable.
Intermsofrecommendationsforthefuture,severalImplementingPartners
highlightedtheutilityofthetrainingworkshopheldonthereportingtemplates
andrequestedthatsimilarsupportbeprovidedagain.Itwasrecognisedthat,
thoughthetemplatesthemselvesarecomplex,thesecondannualreportwill
beeasiertocompletenowthatImplementingPartnersaremorefamiliarwith
these.Nextyear’sreportingwillseektobuildoncollectivelearningfromtheyear
1reportingprocess.Itwillaimtoachieveanoptimumlevelofdatainannual
reportsintermsofquantity,relevanceandqualitytosupportbothprogramme-
levelsynthesisandproject-leveladaptiveprogrammingintheremainder
ofBRACED.
2.�It�is�too�early�in�the�programme�to�provide�evidence�about�substantive�
change.Whengivingfeedbackonthefirstdraftsofyear1reports,werepeatedly
requestedthatImplementingPartnersprovidemoreinformationonthemain
changestheirprojecthascontributedtoandtheproject’scontributiontosuch
results,withconcreteexamplesandevidenceofhowtheyknew,wherepossible.
MostImplementingPartnerswereabletoprovidefurtherinformation,butthis
wasoftenlimitedbytheavailabilityofevidence.Reasonsforthisincluded:
theprojectM&Esystemnotbeingsetuptocollectthedata;thetimeneeded
forthesetypesofchangetohappenbeingmorethanoneyear;anddelaysin
projectstart-upandimplementation.TheFundManagerreflectedthat,although
resilience-buildingresultsmaynothavebeendemonstrablebytheendofyear1,
25ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� WhAthAVEWELEArntSoFAr?
theImplementingPartners’workinsettinguptheprojectinfrastructuresnow
meansthatresiliencebenefitswillbegintobeseenfrommid-year2onwards.
3.�Truly�engaging�with�qualitative�and�explanatory�frameworks�requires�
shifting�mindsets�from�accountability�to�learning-oriented�M&E.�TheBRACED
M&Eframeworkaimstochallengeproject-leveltheoryofchangeassumptions
andfosterdiscussionandunderstandingaroundhowprojectsmovefromoutputs
tooutcomesandwhathappensinbetween.However,itmaybetooearlyinthe
programmetoexploretheprocessesbehindtheprogressmadetowardsoutcomes
andhowcontextualfactorsareenabling/constrainingchange.Implementing
Partnershavespentalargeproportionofyear1settingupstructuresand
partnerships,andtestingactivities.ItmayalsobethecasethatImplementing
Partnersarenotaccustomed(orincentivised)tomonitor,reflectandreporton
suchthingsfromalearningperspectiveratherthananaccountabilityone.The
FundManagerhasexperiencedasimilarchallenge.ImplementingPartnershave
spentconsiderabletimeandeffortinaddingupcomplexfigurestocalculate
thenumbersofpeoplesupportedandhowmanyhavehadtheirresiliencebuilt.
However,reflecting,analysingandreportingaboutwhatthesefiguresmeanhas
receivedlimitedattentioninyear1.TheKnowledgeManagerandFundManager
agreeontheneedtomovebeyondthenumbersandwillworktogetherinthe
remainderofBRACEDtoencouragefurtherreflectionbyImplementingPartners
onthecombinationoftheirquantitativeandqualitativedata.
3.4�Aggregating�and�synthesising�data�at�scaleTheprocessofbringingtogetherandsynthesisingevidencefromImplementing
Partners’year1projectannualreportsattheprogrammelevelhastaughtusthat:
1.�It�is�important�to�achieve�an�optimal�level�of�comparability,�while�also�
retaining�project-specific�visions�and�understanding.�Akeyobjectiveofthe
BRACEDM&Eframeworkwastoensureenoughcomparabilitybetweenproject-
levelM&Etomonitor,measureandunderstandtheresilience-buildingefforts
ofBRACEDprojectsattheprogrammelevel.However,aspreviouslymentioned,
ImplementingPartnershavestruggledtoadheretotheoverarchingprogramme-
widedefinitionsoftheM&Eframework,particularlywhenoutliningthedifferent
levelsofchange(‘expect’,‘like’and‘lovetosee’)thatwerebothanticipated
andrealisedacrossthefourAreasofChange.ItispositivethatImplementing
Partnershaveengagedwithandtakenownershipoftheseandinterpretedthem
fortheirprojectcontext.Broaddefinitionsareintendedtofacilitateproject-to-
programmedatasynthesis.However,wheretheoverarchingdefinitionshave
notbeenfollowed,comparableanalysishasbeenmademoredifficult.�We
soughttoovercomethischallengebysynthesisingproject-leveldataagainstthe
programme-leveldefinitionsassetoutintheoriginalBRACEDM&EGuidance
NotesandusingtheMonitoringandResultsReportingteam’sexpertjudgement
wheredifferencesarose.
26ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� WhAthAVEWELEArntSoFAr?
2.�Synthesising�and�aggregating�data�while�retaining�context�specificity�not�
only�takes�time.�It�also�requires�carefully�developed�synthesis�methodologies.�
Ashighlightedinthecompanionprogramme-levelsynthesisreport,�BRACED
projectscoverawiderangeofissuesandoperateinverydifferentcontexts.
Asanexample,theseinclude:
• securing,servicingandpromotingtrans-borderlivestockmobilityacross
theSahel
• sharingskillsandtechnologytoimprovetheuptakeofclimateinformation
inEthiopia
• supportingsmallholderfarmersinNepaltotakeadvantageofeconomic
opportunitiesandinvestmentsinclimate-smarttechnologies.
Thisprojectandcontextspecificityhasproventobeachallengeforthe
programme-levelsynthesisandaggregationofdiversedata.Wesoughtto
overcomethisbyundertakingathematicsynthesisanalysis,enablingthe
identificationofcommonpatternsandthemesacrossthesetofprojects.
Thisapproachhasrequiredconsiderabletimeandresources.
3.�Dealing�with�self-reporting�bias�requires�triangulation�with�other�sources�
of�information.Intheiryear1reports,ImplementingPartnersvariedbetween
over-reportingchangesseensofaranddownplayingorunder-reportingchanges
thattheMonitoringandResultsReportingteamalreadyknewaboutfromother
sources.Thissuggeststhatthereportingprocessdidnotencourageadequate
reflectionorconsistencyacrossprojects.Toensurethattheprogramme-level
synthesiswasrepresentativeofwhatwasactuallyhappeningontheground,
wecomplementedtheproject-leveldatawithourexistingknowledgeofeach
project.Wealsocross-checkedandgap-filledbyconsultingKnowledgeManager
staffwhohadworkedwithImplementingPartnersinthefieldandreferringto
otherprojectdocumentation.TheFundManageralsohadtotriangulatethe
quantitativeinformationprovidedbyImplementingPartnersinordertocomeup
withsensible,realisticnumbers,withsomeImplementingPartnersover-reporting
onprogress.Infutureyears,wecoulddrawmoreontheFundManager’s
knowledgeandperspectivebasedontheirongoingmonitoringandinteraction
withImplementingPartners.
4.�The�lead�time�between�project-level�annual�reporting�and�programme-level�
annual�synthesis�is�long�and�limits�programme-level�real-time�learning�and�
flexibility.�Duetothescaleandlevelofanalysisrequiredforaprogrammelike
BRACED,thereportingtaskforbothImplementingPartnersandtheKnowledge
ManagerMonitoringandResultsReportingteamissignificant.InJune2016,
ImplementingPartnerssubmittedtheirreportsonprogressmadeuptotheend
ofMarch2016.Thiswasthenfollowedbyarelativelylongprogramme-level
27ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� WhAthAVEWELEArntSoFAr?
analysisandsynthesisexercise,lastingfivemonths(JulytoNovember2016),12
todrawcross-project,programme-levelfindings,lessonsandrecommendations.
Thelengthofthisprocesswasduetothepreviouslymentionedchallenges
thatthenuancesoftheprocessesandoutcomesofBRACEDpresented,namely
thedifficultiesinprovidingclear‘yes’or‘no’answerstohowresiliencehas
beenbuilt,alongwiththeneedtorigorouslyaggregateandsynthesisedata
whileretainingcontextspecificity.Thesubsequentprogramme-levelsynthesis
findingsandrecommendationsidentifiedinNovember2016shouldthereforebe
consideredasareflectionofthesituationuptoMarch2016.Inthecontextof
evolvingchange,thisleadtimemaylimitthefindings’relevanceandpotential
impactonprogrammeandprojectdecision-making.
12 thetimegapbetweenreceivingImplementingPartnerreportsandstartingtheprogramme-levelsynthesisispartlyexplainedbythetimerequiredfromtheKnowledgeManagerMonitoringandresultsreportingteamtocontributetothedFIdBrAcEdAnnualreview(July-August2016).
28ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� BuILdInGonthISLEArnInG
M&EforresilienceprogrammingisinitsearlydaysandBRACEDislearning-by-
doingintermsofwhatkindsofprogresscanbemeasuredandreported,andhow
besttodothis.TheBRACEDprogrammeasawholeistestingasetofframeworks
andapproachesthatneedtobefine-tunedasevidenceemergesandexperience
isbuilt.Inyear1,BRACEDImplementingPartnershaveembracedanewway
ofmonitoringandreportingchange.Wehavelearntagreatdealasaresultof
takingaprogramme-levelviewofhowresilienceisbeingbuiltinBRACED.
“TheBRACEDprogrammeasawholeistestingasetofframeworksandapproachesthatneed
tobefine-tunedasevidenceemergesandexperienceisbuilt”
Inthispaper,�wehavesharedourexperiencesinBRACEDsofarandhopethatthis
willgoontocontributetoongoingandfutureeffortsindesigning,implementing
andreportingagainstM&Eframeworksofresilience-buildingprogrammes.BRACED
monitoringandresultsreportingeffortswillcontinueduringyears2and3andwe
willcarryonreflectingonexperiencesandlessonslearnt.
Image:GeorginaSmith/Worldfish
4.HOW�CAN�BRACED�BUILD�ON�THIS�LEARNING?
29ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� BuILdInGonthISLEArnInG
ThelessonsandexperiencetodatewillbeusedtoimprovetheBRACED
M&Eframeworkandreportingforyears2and3.TheMonitoringandResults
Reportingteamwillexploreoptionstoimproveproject-toprogramme-level
reporting,including:
• simplifyingreportingtemplates,wherepossible
• reducingthereportingleadtimebetweenproject-andprogramme-
levelreporting
• fosteringpeer-to-peerlearning(inconjunctionwiththebroaderKnowledge
Manager’songoingwork)
• creatingsystematicopportunitiesfordatatriangulationacrossprojects.
BRACEDisnearlytwoyearsintoitsthree-yearimplementationtimeframe
andproject-andprogramme-levelM&Eisalreadysetupandestablished.
Therearethereforesomelimitationstowhatcanbeadaptedandachieved
intheremainderoftheprogramme.Inthiscontext,oursuggestionsbelow
areforboththeBRACEDprogrammeandothersimilarprogrammes.
“M&Eexpertsneedtoengageinmorerefinedandcomplexdatacollectionandanalysis
processesthanistypicalthroughexplanatoryandqualitativeindicatorframeworks”
EstablishingM&Eframeworksandreportingsystemsforresilience-building
programmesandprojectsremainsanareawheremoreknowledge,experience
andlearningarerequired.LessonstodatehighlightthatM&Eforresilience
buildingrequiresapproachesandpracticesthatgobeyond‘businessas
usual’.ThisinvolvesM&Eexpertsengaginginmorerefinedandcomplexdata
collectionandanalysisprocessesthanistypical.Italsorequiresthesetting
upofexplanatoryandqualitativeframeworkswithstronglinkagestoresearch
efforts.Wehavefound,throughundertakingtheprogramme-levelsynthesis
ofBRACEDproject-leveldataforthefirsttime,thatgenuinelytryingtomeasure
resiliencemeansweneedtomoveawayfroman‘accountability’M&Eculture.
Goingforward:
• Implementing�Partners�should�enhance�their�ongoing�monitoring�
and�results�reporting�efforts�by�taking�a�more�reflective�and�critical�
approach.Thiscouldchallengeprojectassumptionsandwillbuildabetter
understandingofhowtobuildclimateanddisasterresilienceinfragileand
vulnerablecontexts.
• The�Monitoring�and�Results�Reporting�team�should�consider�how�to�
encourage�this�critical�reflection�and�dialogue,�as�well�as�innovative�
ways�of�capturing�rich�and�reflective�data�from�BRACED�projects.�There
arelimitstowhatreportingtemplatesalonecanachieveinthisregard.We
30ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� BuILdInGonthISLEArnInG
thereforeplantoprovidefurthertrainingtoImplementingPartners,along
withlight-touchhelpdesksupport.
• Programmes�like�BRACED�need�to�find�and�resource�efficient�ways�of�
achieving�a�sufficient�level�of�reflection�and�learning�for�the�benefit�
of�both�project-�and�programme-level�evidence�generation.Ideallythe
programme-levelM&Eframeworkshouldbedesignedinconjunctionwith
theproject-levelframeworks.
“QuantitativeindicatorslikeKPI4enabledataaggregationandsynthesisatthe
programmelevel,butmisscriticalinformationforongoingevidencegenerationandlearning
aboutwhatworksandwhatdoesnotinbuildingresilienceinBRACED”
Indicatorsenabledataaggregationandsynthesisattheprogrammelevel,but
misscriticalinformationforongoingevidencegenerationandlearningabout
whatworksandwhatdoesnotinbuildingresilienceinBRACED.�Thedesire
tounderstandifandhowresilienceisbeingbuiltisatthecoreoftheBRACED
M&Eframework.ThoughKPI4mightbenecessaryforDFIDtotrackoutcome-
levelchangesacrossawiderportfolioofprogrammes,itlimitsgeneralisable
lessonsandlosesimportantdetailaboutcontext,particularlywhenindicators
requirethequantificationofacomplexconceptlikeresilience.KPI4dataneeds
tobecomplementedbythesystematicM&Eofresilienceinthecontextofactual
shocks.Thiswillenableustobetterunderstandthestabilityofoutcome-level
changesovertimeandhowcommunitieslearnand‘bouncebackbetter’from
disasterevents.Goingforward:
• Implementing�Partners�are�in�a�unique�position�to�contribute�to�
knowledge�generation�about�how�to�quantify�the�number�of�people�
whose�resilience�has�been�built�(KPI�4)�at�the�project level.The
MonitoringandResultsReportingteam,togetherwithwidermembersofthe
KnowledgeManagerteamandtheFundManager,shouldfurtherexplorethe
advantagesanddisadvantagesofKPI4attheprogrammelevel,aswellasthe
opportunitiesandtrade-offsofresilienceindicatorsindifferentcontexts.
• When�designing�and�funding�similar�programmes�in�the�future,�DFID�
should�adopt�a�pragmatic�and�realistic�view�on�the�feasible�level�of�
outcome-level�data�and�evidence�generation�in�a�three-year�programme�
like�BRACED.�Resilience-buildingeffortsarenotonlycomplex,butalso
involveprocessesofchangethattaketimetomaterialise.Prioritisingannual
datacollectioneffortsagainstquantitativeindicatorsmaycomeatthecost
oflosingcriticalevidenceaboutwhatworksandwhatdoesnotinbuilding
climateanddisasterresilience.
31ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� BuILdInGonthISLEArnInG
• Programmes�like�BRACED�should�consider�having�a�diverse�set�of�
methodologies�and�analysis�in�place�for�interrogating�quantitative�
outcome-level�resilience�indicators.�Theyshouldbepragmaticabout
whatsortofoutcome-leveldataandinformationtheycanexpectina
three-yearperiod.
“Whilemuchattentionhasbeengiventoproject-levelapproachestomonitoringand
measuringresilience,programme-leveleffortsfaceauniquesetofchallenges”
Whilemuchattentionhasbeengiventoproject-levelapproachesto
monitoringandmeasuringresilience,programme-leveleffortsfaceaunique
setofchallenges.Theseinclude:theharmonisationofM&Eframeworksacross
awideportfolioofprojects,real-timelearningfromprojecttoprogrammeand
back,andtheaggregationofhighlycontextualquantitativeandqualitativedata
throughflexibleyetstandardframeworks.Todate,therearelimitedexamples
(andsubsequentliterature)fromotherprogrammesaddressingsuchchallenges.
BRACEDhasbeenlearning-by-doingonanongoingbasis.Goingforward:
• The�Monitoring�and�Results�Reporting�team,�together�with�Implementing�
Partners,�should�consider�ways�to�further�capture�their�monitoring�and�
results�reporting�experiences�within�BRACED.Thiswouldbenefitboth
BRACEDandotherexistingandfutureresilience-buildingprogrammes.
• More�transparent�and�reflective�discussions�are�required�to�address�
the�challenges�and�lessons�learnt�in�establishing�M&E�frameworks�for�
resilience-building�programmes.ProgrammeslikeBRACEDshouldalso
shareandcontributetobuildingknowledgeandexperienceinthisrelatively
newareaofwork.
4.1�Question�for�further�reflectionWiththeaimofcontributingtoongoinglearningaboutresilienceprogramming,
wewishtoengageBRACEDImplementingPartners,thebroaderBRACED
KnowledgeManager,theBRACEDFundManager,DFIDandwideraudiencesin
thefollowingcriticalquestionthatemergesasaresultofreflectionsandlearning
againsttheM&Eframeworksofar.Emerginginsightsfromthispapershedsome
lightforinitiatingdiscussion.However,theBRACEDprogrammeshouldcontinue
toanswerthisquestionthroughoutitslifetime:
How�complex�does�M&E�for�resilience�need�to�be?TheBRACEDM&E
frameworkbringstogetherthreedifferentlensesintotheanalysisofproject-
andprogramme-leveldata.Experiencetodate,bothfromImplementingPartners
andtheBRACEDKnowledgeManager’sMonitoringandResultsReportingteam,
indicatesthat:
32ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� BuILdInGonthISLEArnInG
• Whiletheframeworkprovidesamultidimensionalviewintothevarious
levelsofcomplexity,itrequiressignificanttime.
• Italsonecessitatesdifferent,non-traditionalwaysofengagingwithdata.
• Itdoesnotandcannotprovidesimple‘yes’or‘no’answersaboutwhether
resiliencehasbeenbuiltand,ifso,how.
“Theessenceof‘resilience’isthatchangeandprogressarenotlinear–soresultsreporting
shouldnotbelineareither”
Asoutlinedinourcompanionreport,‘Routes to resilience: insights from BRACED
year 1’,theessenceof‘resilience’isthatchangeandprogressarenotlinear–so
resultsreportingshouldnotbelineareither.TheBRACEDM&Eframeworkis
complexenoughtobeabletounderstandBRACEDresilience-buildingefforts,
butcoulditbemadetobemoreuser-friendly,whilestillretainingthecomplexity
andnuancesofresilience-buildingprojects?
33ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
Annex�1:�Monitoring�and�evaluation�in�BRACEDM&Eactivitiesareundertakenatboththeprojectandprogrammelevel
withinBRACED.
Eachofthe15BRACEDprojectshasitsowntheoryofchange,logframe,M&E
planandM&Esystem.EveryImplementingPartnerreportsprogressandlearning
againstitslogframeandtheoryofchangeonanannualbasis.Eachprojectisalso
carryingoutamid-termreviewandafinalevaluation.
Progressagainsttheproject-levellogframesisreportedtotheBRACEDFund
Manager,whomanagestheprojectgrantsonbehalfofDFID.TheFundManager
thenaggregatesandreportsBRACEDprojectresultsagainsttheBRACED
programme-levellogframe.TheFundManageralsoundertakesongoing
projectperformancemonitoring.
Attheprogrammelevel,thereisanoverarchingtheoryofchange(seeAnnex2)
andasetofMonitoringandResultsReportingandEvaluationactivities,ledby
theKnowledgeManager.TheKnowledgeManagerhasprovidedsupportand
guidancetoprojectImplementingPartnerstoensurethealignmentofproject-
andprogramme-levelM&EthroughtheM&Eframework.Progressandlearning
againstproject-leveltheoriesofchangeisreportedbyImplementingPartners
totheBRACEDKnowledgeManagerusingtheM&Eframeworkinorderto
understandhow resilience is being builtat the programme level.Thispaperis
basedonthoseinputs.
WithintheBRACEDM&Esystem,theFundManagerisresponsibleforoverseeing
thedeliveryoftheBRACEDprojects.Theydothisby(a)collectingfinancial
accountabilityinformation,(b)annuallyreportingattheoutput/activitylevel,
and(c)collatingrelevantdataagainsttheBRACEDmandatorykeyperformance
indicatorsoftheInternationalClimateFund,includingKPIs1,4&15.
TheKnowledgeManagerM&Eteamfocusesonbuildingandsharingevidence
andknowledgethroughatheoryofchangeapproach.Alongsidesubstantive
monitoringandresultsreportingactivities,theBRACEDKnowledgeManager
isalsoundertakingasetofevaluationactivitiestounderstandtheextentto
whichBRACEDprojectinterventionswork(seeAnnex5).
34ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
Annex�2:�BRACED�theory�of�change
SeeNote2oftheBRACEDM&EGuidanceNotesforafullnarrativeoftheTheoryofChange.
BRACED invests in projects directly targeting:
Working with a whole variety of stakeholders:
Assumptions:effectiveness of the BRACED fund
To support changes in 7 thematic areas, which will strengthen 4 areas of change:
Assumptions:BRACED outputs
Which will directly deliver a set of 4 OUTPUTS at different scales leading to the BRACED OUTCOME:
From which BRACED will derive lessons to deliver a set of ‘amplified’ results by influencing policy making and development planning from the international to the local level:
And, in the long term will bring about:
Assumptions:BRACED amplified effect
Impact:Improved well-being of poor people, despite exposure to climate extremes and disasters
Households and community level
Components A&B
Regional/ international organisations
National government
Sub-local government
Research institutions
NGOs CSOs
Communities
Thematic areasClimate & weather information
Technology & innovation
Gender & social equality
Markets & local economic empowerment
Delivery of basic services
Governance & natural resource management
Resilience concepts
Areas of changeKnowledge & attitudes
Capacity & skills
Partnerships
Decision-making
National and local government capacity
Component D
Knowledge, learning and evidence
Component C
Output 4:Improved policies in
targeted areas
Output 2:Increased capacity of local
government, CSOs and private sector to respond to climate-related
shocks and stresses
Output 1:Poor people receive support to reduce their
vulnerability to climate-related shocks and stresses
Assumptions:BRACED outcomes
Outcome:Poor people in developing countries have improved their levels of resilience to climate-related shocks and stresses.
Measuring the three dimensions of resilience:Anticipatory, Absorptive and Adaptive capacity.
Output 3: Better understanding of w
hat works in
building resilience to climate extrem
es and disasters
BRACED
amplifie
d
results
35ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
Annex�3:�Project-to-programme�synthesis�methodology
Project-level analysis and
synthesis
Step 1: Systematic review
and screening against project-level
grid
Step 2: Characterising project-level
(organising data – key word search)
Step 3: Project-level
synthesis against analytical
framework
Step 5: Thematic analysis
Step 4: Comparative analysis and identification
of themes
Project-to-programme
Programme-level
synthesis
Framework synthesis
Thematic synthesis
Consultation with ongoing research
streams to deepen analysis
of findings
36ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
Annex�4:�Further�Knowledge�Manager�and�Implementing�Partner�reflections�on�experiences�of�reporting�against�the�BRACED�M&E�framework
Implementing�Partners’�informal�feedback�on�the�reporting�process
Basedontheyear1reports,thereseemstobebuy-in�across�all�
Implementing�Partners�to�the�key�concepts�of�the�M&E�framework
andreportingagainstthese.Onthewhole,ImplementingPartnerstold
theMonitoringandResultsReportingteamthattheyfoundtheframework�
a�relevant�way�of�telling�their�project’s�story.NotonlydidImplementing
Partnersreportinformationagainsteachpartoftheframework,many
alsoreferredtotheconceptsthroughoutthereport,suchastalkingabout
resilienceoutcomesintermsofcapacitiestoanticipate,adapttoandabsorb
shocksandstresses.ThisisasignificantachievementgiventhattheM&E
frameworkwasdevelopedafterproject-levelM&Esystemshadlargelybeen
designed,thereforemeaningithadtobe‘retrofitted’.Overall,thereisa
good�understanding�of�the�various�parts�of�the�M&E�framework.
ImplementingPartnerswereinformallyasked:Did you find the BRACED M&E
framework a relevant way to frame/structure the ‘story’ of your project? Are there
any gaps?
Overall,Implementing�Partners�seem�to�find�the�M&E�framework�a�relevant�
way�to�tell�the�story�of�the�project,withonesayingit‘has guided us to tell
our stories in a realistic way’.AnotherImplementingPartnertoldusthat‘it was
helpful in drawing lessons across the consortium’.However,membersofthesame
team‘felt it did not add much to [their] existing framework for evaluating progress
and impact, and so was more useful to the KM than to IPs themselves’.
IntermsofthespecificaspectsoftheM&Eframework,
OneImplementingPartner‘felt the request for an analysis of “changes in
capacity” in the Areas of Change [template] overlapped significantly with the
3As [template]. As a result, we struggled to parse out which unique insights the
KM was looking for in one section versus another’.
Anotherfoundit‘difficult to fit [sub-indicators] under the 3As, though
evidently they all fold up under KPI 4’asthey‘were not at the time developed
around the 3A capacities’.
Anothersuggestedthat‘there could be more consideration of “depleted
capacities”. That is, while the work of BRACED projects [is] building the
adaptive, anticipatory and absorptive capacity of the communities, there are
very often ongoing shocks and stresses that require participants to draw on
those sources of capacity in the midst of building them up. The [template]
talks about the potential trade-offs between the 3As, but I believe a common
37ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
part of many projects’ stories will be this cyclical pattern of growth in capacity
development, then a shock or stressor that calls them to draw upon them, and
then potentially another cycle of growth, etc. It is a non-linear projection that
may not clearly be evident in the annual milestone trackers’.
ImplementingPartnerswereinformallyasked:How appropriate were the Part
2 templates for capturing your project’s story? What would you change about
the M&E framework or the report template to tell your project’s story better
(remembering that, next year you will have more results to share)?
SomeImplementingPartnersreportedthattheexperience�of�completing�the�
templates�was�helpful�for�internal�reflection�and�added�value�to�their�own�
project�M&E:
‘We appreciated all the KM’s efforts to use the template to help us tell the
real story of what is happening on the ground in a factually supported way.’
Thereportingexercisewas ‘definitely a great opportunity for us to look into
our own achievements more critically and systematically – asking ourselves
various questions about the project performances/challenges, which we would
have otherwise not even thought of. In the beginning, although it appeared
highly time consuming and exhausting, with the completion of the report,
we felt very satisfied with greater insights and deeper understandings about
the project status and future needs.’
‘We felt the reporting process was useful in reducing the gap between our
project’s theory of change and BRACED programme’s theory or change,
and plan to adapt our own [theory of change] and logframe, based on key
takeaways gleaned from this exercise during the mid-term review process.’
However,oneImplementingPartnerreflectedthat:‘Some of the changes that
we were asked to report on cannot be objectively evaluated within the project
lifetime, and so felt like less of a useful exercise. For example, select aspects
of our theory of change cannot be objectively measured within the lifetime
of the project, because we are looking at long-term changes in resilience
(+5 years).’ Alongwiththis,anotherImplementingPartnerfelttherewasnot
spacetosharetheiralreadydocumentedcasestudiesandsuccessstories.
Therewerea�number�of�challenges�cited�in�completing�the�templates�against�
the�M&E�framework.ThesewereoftenduetoImplementingPartnersnot
respondingtooriginalKnowledgeManagerfeedbacktoalignproject-levelM&E
totheprogrammelevel:
Inrelationtotheidentificationofkeystakeholders/partners, Implementing
Partners asserted the following:
‘Though we, to our best of our knowledge, identified the stakeholders relevant
to Areas of Change, it was a bit confusing how to categorise them. It was also
difficult to ascertain if we have missed any important stakeholders which we
could have elaborated.’
38ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
‘Ours was one of three to four consortia who did not include markers
(‘expect’, ‘like’, ‘love to see’) in our baseline, as we were told these were not
mandatory. As a result, our project’s ‘story’ was less suited to the template
than those who set benchmarks at the start of the project.’
‘We continued to struggle to define and apply the concepts ‘scale of impact’
and ‘catalytic effect’ embedded in [the template] on transformative impact
to our analysis.’
Intermsofrecommendationsforthefuture,several�Implementing�Partners�
highlighted�the�utility�of�the�training�workshop�held�on�the�reporting�
templatesandrequestedthat,shouldonebeheldforthenextreport,they
wouldlikethistobeorganisedfurtherinadvance,toenableparticipationand
attendancebyfield-basedteammembers:
‘We’d like to request that, to the maximum extent possible, future sessions
like the one held in May either be held in Africa, perhaps as part of the annual
learning event, and/or communicated as far in advance as possible, so that
we can better plan for attendance, adequate coverage, as well as conduct
the logistics necessary for travel (visas, etc.) We recognise the limitations
surrounding this round, but just wanted to emphasise that we might have
been able to better leverage the learning for our staff and/or contribute
more efficiently had we been able to send more and/or more appropriate
staff for the workshop.’
‘Thanks again for all of your terrific work in pulling together the workshop
session. It really was quite helpful and insightful. If there are considerations
of holding a similar event next year, I would also suggest ample notification
so country staff who know the programmes best would be able to attend
and get visas… [The in-country] team were very understanding and terrific
at interpreting our insights second-hand, but I’m sure [they] could contribute
to the conversation in a richer way than I was able by attending in-person
[themselves].’
AnotherImplementingPartnerreflectedthat: ‘it can be challenging to think in
terms of strategic, observable change, rather than, say, outputs; it is difficult to
bring rigour to a largely qualitative, narrative-based section.” Inordertoimprove
thequalityoftheirreportnextyear,theyaskedfortheMonitoringandResults
Reportingteamtoshare“‘best in class’ examples for some of the answers and how
the IP substantiated it, something we could emulate for the next round, including
whether the IP used their normal M&E, conducted special surveys and/or collect
testimonials? We feel a concrete example would be extremely helpful for several
“templates”’.
Itwasrecognisedthatalthoughthetemplateitselfiscomplex,thesecondannual
reportwillbeeasiertocompletenowthatImplementingPartnersaremore
familiarwiththetemplate.
39ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
Appropriateness�of�M&E�reporting
Duetothenumberoftemplates,withImplementingPartnerscompletingthese
inafairlycomprehensivemanner,itwasahugetasktogothroughall14project
annualreports,whichoftentotalled60–80pageseach.13Anumberoffactors
increasedthesizeofthis:
• Sometimestherelevant�information�was�not�always�provided�in�the�
template�intended�for�that�data.�Forexample,changesreportedinthe
‘likelihoodoftransformation’templatewereoftenrelevantforinclusion
undertheAreasofChangebutweremissedornotreferenced.Therewas
thereforeaneedtoconsidereachreportasawhole,ratherthanlooking
tospecifictemplatesforspecificinformation,whensummarisingand
synthesisingattheprojectlevel.However,thisdidenabletheteamto
makesenseofdataprovidedbytriangulatingwithinformationelsewhere
inthereport.
• Sometimesnot�enough�detail�was�provided�by�the�Implementing�
Partner�for�the�reported�change�to�be�fully�understood�and�included
inthesynthesis.Forexample,theymayhavereportedthataparticular
stakeholderhadappliedtheirknowledgebutnotwhatknowledgeand
how.Goingforward,exampleswouldbeusefultoensurethatthechanges
reportedarenottoogeneric/vague,especiallygiventheanecdotalnature
ofevidenceinyear1.SomeImplementingPartnersweregoodatproviding
these.Also,some�were�more�self-critical�than�others,�and�downplayed�
or�under-reported�thingsthattheMonitoringandResultsReporting
teamknewtheywereactuallydoingverywell,suggestingthatthe
reportingprocessdidnotencourageadequatereflection.Toensure
thispaperisreflectiveofwhatisactuallyhappeningontheground,the
MonitoringandResultsReportingteamcomplementedthedatawith
theirexistingknowledgeoftheprojectandalsocross-checked/gap-filled
withthelogframeactivityreportingandotherprojectdocumentation
(e.g.theoriesofchange,websitesandpublications).Therewasn’tcapacity
torefertoallprojectbaselinesindetail,thoughthiswouldhavebeenuseful
forunderstandingwhatchangeshadbeenachievedduringthelifetime
oftheproject.
• Sometimes,not�all�of�the�reported�information�was�relevant�or�needed
fortheprogramme-levelsynthesis.Forexample,intheAreasofChange
template,ImplementingPartnersincludedinformationaboutactivitiesthey
haddonewithassociatednumbers(e.g.thenumberoftrainingscompleted).
However,itisacknowledgedthatthechangebeingdescribedwassoclosely
associatedwiththeactivitybeingdonetoachieveit,insomecases,itwas
difficulttodisassociateanddistinguishbetweenthetwo.IntheUse of
climate and weather informationtemplate,ImplementingPartnersoften
includedinformationonwhattheyexpectedtohappeninthecomingyear.
TheMonitoringandResultsReportingteamconsideredonlythatinformation
13 thisincludesPart1ofthereport,whichwasmostlyaimedattheFundManager.
40ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
relevantfortheprogramme-levelsynthesis.However,itmaybeusefulto
referbacktothiskindofdatawhenproducingnextyear’sversionofthe
synthesis,asthiswillenableustoseethelevelofactivity/changeover
thecourseofayear.
Thesefactorsmadeitmoredifficulttounderstandprojectstories/pathways
andthendrawcross-project,programme-levellessons.Itwillcontinuetobe
achallengetoachievethebalancebetweenImplementingPartnersproviding
enoughoftherightinformationtounderstandtheprojectanditschanges,and
toomuchdetail.TheMonitoringandResultsReportingteamwillworkalongside
ImplementingPartnerstofurtherimprovethisfornextyear’sreporting.
Thereportingtemplatesandprocessalsoseemedtoaffectthedataprovided
toacertainextent:
• TheAreasofChange,3AsandEvaluativeMonitoringtemplateseach
includedasectionforreportingprogressandanotherforproviding
reflectionandanalysis(e.g.‘Towhatextentarethesechangeprocesses
contributingtotheprojectoutcome?’)These‘bigger’,more�reflective�
questions�received�limited�answers�from�Implementing�Partners.Most
projectshaveaknowledge,learningandresearchcomponentthatcould
supportthesereflections;however,thesedidnotcomeout.Themost
effectivetemplateswerethosethatwalkedImplementingPartnersthrough
aprocessofanalysisandreflection,suchas‘Useofclimateandweather
information’.ApplyingthisapproachtotheEvaluativeMonitoringtemplate,
inparticular,wouldbehelpfulfor(a)reducingrepetitionofcontextual
factorsatdifferentlevelsandtimes,and(b)improvingspecificityaboutif
andhowthesehaveenabledorconstrainedchange.Itmayhavealsoaided
furtherreflectionbyImplementingPartnerswhenconsideringthecontinued
validityoftheirprojecttheoriesofchange,thoughprematuretimingin
relationtomid-termreviewslimitedinputshere.
• Duringthetemplatedesign,mucheffortwentintounpackingthe
BRACEDtheoryofchangeandunderstandingthechangesbeingbrought
aboutbyprojectsfromdifferentperspectives.Attimes,‘zooming�in’�to�
capacity,�decision-making�and�the�3As,�means�the�story�of�the�project�can�
be�lost(e.g.whyImplementingPartnersareengagingincertainactivitiesand
howthesearecontributingtobuildingresilience).PerhapstheKnowledge
Managerwasaskingtoomuchofprojectsinthe‘likelihoodoftransformation’
templateasclaimsheretendedtobeoverlyambitious.Inaddition,the3As
templateisnotcurrentlydesignedtothinkaboutsequencing:ifaparticular
activityhasmorefocusinyear1comparedwithyears1or3,oriftheproject
isfocusingonbuildingonecapacitymorethantheothers.
• The�factthe�report�to�the�Knowledge�Manager�was�combined�with�
reporting�to�the�Fund�Manager�(for�accountability)�may�have�affected�
what�information�Implementing�Partners�provided�for�the�M&E�
framework�sections(forlearning)andhowtheyframedandpresented
thisinformation.Reportstendedtofocusondeliveryratherthanlearning
andchangepathways,andthecontextwasconsideredintermsofrisksto
41ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
activitiesratherthanasanenablerorconstrainerofchange.Thereporting
wascombinedattherequestofImplementingPartnersatthestartofthe
programme.Whileresultsmayhavebeenaffected,thecombinednature
ofthereportmayhavealsosupportedImplementingPartnersbuyinginto
reportingagainsttheM&Eframework.
Overall,fromaKnowledgeManagerperspective,thefirstyear’sreportingprocess
wentwell.ImplementingPartnerreportswereparticularlyimpressive,giventhat
(a)project-levelM&Esystemswerenotsetuptoreportconsistentlyagainstthe
BRACEDM&Eframework,(b)thereportingwasearly,comparedtowhenresults
couldrealisticallybeexpected,and(c)ImplementingPartnerswerecompleting
thetemplatesforthefirsttime.
ItwasusefulfortheMonitoringandResultsReportingteamtobeableto
commentonthefirstdraftofeachprojectreport,highlightgapsandreceive
updatedreportsintrackedchangeswithanexplanation/responsefromthe
ImplementingPartneragainsteachgeneralpointoffeedback.Itseemedto
beconfusingforsomeImplementingPartnersthatthisKnowledgeManager
feedbackwasprovidedthroughtheFundManager.Thiswasdonebecause
thoseweretheofficialreportinglines.However,someImplementingPartners
atthatpointthoughtthatthefeedbackwascomingfromtheFundManager.
Adifferentprocesscouldbeconsiderednexttime.
Asetofreportswithboththesameconceptualisationandoptimumlevel
ofdetailandrelevanceofinformationwillgreatlyfacilitatetheprocessfor
movingfromproject-levelreportstotheprogramme-levelsynthesisreport.
TheMonitoringandResultsReportingteamwillworkwithImplementing
Partnerstoconsiderhowtoimproveboththetemplatesandreporting
process,tomaketheexercisebothasusefulandlightaspossibleforboth
theImplementingPartnersandtheKnowledgeManager.TheMonitoring
andResultsReportingsynthesisprocesscouldpossiblybefurtherrefined
bycombiningthetaskofsummarisingprojectreportsandsynthesisingthem.
TheMonitoringandResultsReportingteamwillconsiderhowtoachieve
aprogramme-levelviewthatisbothcomprehensiveandyetspecificby
referringbacktoindividualprojects.
Feedback�given�to�Implementing�Partners�on�their�initial�reporting
Therewassome�confusion�over�the�term�‘baseline’�in�the�Areas�of�Change�
template.ThisreferredtothechangestheImplementingPartnerwould
‘expect’,‘like’and‘love’toseeduringthelifetimeoftheproject,ratherthan
thesituationatthetimeofthebaselinestudy.Consequently,forasmallnumber
ofImplementingPartners,thedata‘baseline’informationreceivedwasa
mixtureofwhatthesituationactuallywasatthestartoftheprojectandwhat
theImplementingPartnercould‘expect’,’like’or‘love’toseeatitsinception.
Whiletheformerwasinterestingandusefulinformation,particularlyinterms
ofunderstandingwhathadbeenachievedwithintheprojectlifetime,itwasthe
latterthatwasbeingsought.
42ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
Insomeinstances,ImplementingPartnersreportedchangesatthe‘liketosee’
or‘lovetosee’levelswithoutdetailingtheearlierchangesintheprocess(i.e.the
‘expecttosee’aspects).Itisnotyetclearifthisisanissuethatchallengesthe
AreasofChangeasmulti-stepprocesses.ImplementingPartnersalsotendedto
reportchangesbeyond‘expecttosee’,eventhoughtheiranalysisstatedthat
mostchangesseensofarwereatthe‘expecttosee’level.Uponrequestfor
clarification,theyexplainedthatchangesseenbeyondthe‘expecttosee’
levelweremoretentative.
TherewasalsosomeoverlapinprojectreportingonAreaofChangechallenges
andEvaluativeMonitoringconstrainingfactors.Reportingprocessesofchange
inisolationfromanunderstandingofthecontextwithinwhichprojectsoperate
isnotuseful.Furtherthinkingisrequiredonhowbesttointegratethesetwo
reportingtemplates.Underpinningthegapsandchallengesisthedifficultyin
translatingkeyconceptsinprogramme-leveltheoryofchangeintomonitoring
templatesattheprojectlevel.
ItisalsothecasethatanumberofImplementingPartnersmissedthe
opportunitytoprovidetheoverallproblembeingaddressedforeachAreaof
Change.Itisthoughtthiswasduetoalackofclarityonwhetherthisinformation
wasrequired;ImplementingPartnerswereabletoprovideitonrequest.
Underthe3Astemplate,alack�of�quantitative�‘KPI�4’�data�on�how�many�
people’s�resilience�had�been�built�in�the�first�year�of�the�projectheightened
theneedforanexplanationofhow/whytheprojectindicatorswouldcontribute
tobeneficiaries’capacitytoanticipate,absorbandadapttoshocksandstresses.
SomeImplementingPartnersleftthisblankinthefirstversionoftheirreports.
However,mostwereabletofillthisonrequest.Ofallthetemplates,the3Aswas
theonemostlikelytobeadaptedorcompletedincorrectly.Itseemsthereisa
needtofurtherexplainsub-indicatorsandtheexactinformationbeingsought.
Theyear1reportsrevealedthatImplementing�Partners�are�either�not�
taking�a�continual�‘Evaluative�Monitoring’�approach�to�understanding�and�
responding�to�the�changing�context�of�their�projects,�or�not�reporting�on�it.
AllImplementingPartnerswereabletodetailthecontextualfactorsaffectingthe
projectatlocal,sub-nationalandnationallevels,bothatthestartoftheproject
andtheendofyear1.Most,however,struggled�to�provide�in-depth�analysis�
and�specific�examples�showing�how�these�factors�were�constraining�and/or�
enabling�change�in�the�project(particularlythelatter).Theywerealsounclear
onwhetherthiswasexpectedornot.Inafewcases,shocksandstresseswere
missingfromtheEvaluativeMonitoringreportingonhowtheprojectcontext
hadchangedduringtheyear.
Anotherissuewheredatawaslimitedwasinrelationtohowtheuse
ofclimateinformationintheBRACEDprojectimprovestheresilienceof
beneficiaries.OneImplementingPartnerreflectedthat,‘there is not a lot to
mention about how these processes were linking or contributing to beneficiary
resilience’.Sometimes,itwasdifficult�to�distinguish�between�changes�or�
results�that�had�actually�happened�(encompassing�whether�these�were�
during�the�lifetime�of�the�project�or�before�it�began)�and�those�that�were�
43ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
anticipated�on�the�basis�of�year�1�progress.ThiswasclarifiedbyImplementing
Partnersintheirsecondversionofthereport.
Finally,thelasttemplateofthereport–whichwasfocusedonthecontinued
validityoftheprojecttheoryofchangebasedonlearningfromreportingagainst
theM&Eframework–wasonlylightlyfilledinbymostImplementingPartners.
Themainreasongivenforthiswasthatitwastoo�early�in�the�project�cycle�to�
review�the�theory�of�changeandthatchangestoitwouldbeconsideredafter
theprojectmid-termreviewhadbeencompleted(mid-waythroughyear2).
44ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
Annex�5:�BRACED�M&E�‘Infrastructure’
BRACED Fund
Manager
FM Results Team
Quarterly & Monthly
Reporting
Direct engagement
with IPs
Annual Reporting Synthesis
Evaluation
Quarterly Performance
Reporting
Monitoring visit reports
BRACED Knowledge Manager
Project to programme evidence &
learning
Monitoring & Results Reporting
(MRR)
Consistent project results
reporting (Outcome level)
Evaluative Monitoring
(context analysis)
Areas of Change (Outcome Mapping)
3As – Resilience outcomes
Contribution Analysis
(Country Case Studies)
Realist Evaluation
Case based analysis
Quasi-Experimental
Impact Evaluation
Contribution Analysis
EA1: BRACED Programme
ToC
EA2: BRACED interventions
EA5: PHASE
EA3: BRACED Projects
EA4: Adaptive Social Protection
(System level)
Activity Method
How is BRACED performing?
How are BRACED projects building resilience?
How effectively are activities being delivered?
What results has BRACED delivered?
Does the BRACED model work? For whom?
What does this mean for future resilience programming?
What does this mean for resilience strengthening more broadly?
What have we learned about monitoring and measurement of resilience programming?
* EA: Evaluation Activity* ToC: Theory of Change
45ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
Annex�6:�Rolling�out�the�BRACED�M&E�frameworkTheBRACEDM&Eframeworkwasdevelopedandrolledoutasfollows:
• Theprogramme-levelMonitoringandResultsReportingteamcombined
abottom-upandtop-downapproachtodeveloptheBRACEDprogramme
theoryofchange.Thisincludedareviewofall15project-leveltheories
ofchangeandlogframesandconsultationwiththeprojectImplementing
Partners.Itsetoutkeyelementsoftheprogramme-levelM&Eframeworkto
whichallprojectswouldcontributebothresultsanddata.Attheprogramme
level,frameworksweredevelopedtoenablestandardisation(totheextent
possible)ofconcepts,analysisandreportingagainsttheprogramme-level
theoryofchange.
• Theprogramme-levelMonitoringandResultsReportingteamthenprovided
arangeofsupporttoprojectImplementingPartners:
• DetailedwrittenguidancewassharedwithallImplementingPartnerson
theM&Eframeworkandhowtooperationaliseit(intheBRACEDM&E
GuidanceNotes).
• One-to-oneconversationswereheldwitheachImplementingPartner
toansweranyquestionsinrelationtotheframeworkandhowtoapply
ittoproject-levelM&E.
• Writtenfeedbackwasprovidedonupdatedversionsofproject-level
M&Edocumentstocheckalignmenttotheprogrammelevel.
• Groupsupportwasprovided,bothface-to-faceandremotelyonagroup
basis,totrainImplementingPartnersinthereportingtemplatesand
showthemhowtoreportagainsttheM&Eframework.
DuringthefirstyearofBRACEDimplementation,theKnowledgeManager
MonitoringandResultsReportingteamhasworkedalongsideImplementing
PartnersandtheFundManagerto(a)ensurethealignmentofproject-levelM&E
withtheprogramme-levelM&Eframework,and(b)enhancetheoverallquality
andcomparabilityofBRACEDM&E.
Afterface-to-faceconsultationwithallImplementingPartnersduringtheBRACED
inceptionworkshopinSenegal,February2015,focusingonthedraftframework,
theKnowledgeManagerdevelopedwrittenguidanceforallImplementingPartners
tooperationalisetheframework.FollowingthesharingoftheBRACEDM&E
GuidanceNotes(March2015),theKnowledgeManagerprovidedaprogrammeof
1-2-1supporttoImplementingPartnerstohelpthemapplytheframeworkintheir
finalisationofproject-leveltheoriesofchange,logframesandM&Eplans.14
14 thisincludeda1-2-1SkypecallwitheachImplementingPartnertodiscusstheimplicationsofdatacollectionagainsttheM&Eframeworkatproject-level(April–May2015)andawrittenreviewofupdatedproject-levelM&Edocuments(June–July2015),beforetheyweresignedoffbytheFundManager.
46ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX
TheMonitoringandResultsReportingteamthendevelopedthesetof
reportingtemplatestogatherdataagainsteachoftheM&Eframework
components.This,inturn,enabledallImplementingPartnerstoundertake
standardisedandcomparablereportingofyear1projectresults.TheKnowledge
Manageralsodevelopedtemplatesfortwoaspectsofprogrammelogframe
reporting.Thesewerecentredonthe‘useofclimateandweatherinformation’
andthe‘likelihoodoftransformation’.Draftversionsofthesetemplateswere
sharedwithImplementingPartnersfortheirfeedbackattheBRACEDAnnual
LearningEventinSenegal,February2016andatawebinarinMarch2016.
TheKnowledgeManagerthenworkedcollaborativelywiththeFundManager
tobothfinalisetheoverallannualreporttemplate(March2016)andtrain
ImplementingPartnersinitscompletion(May2016).15
ImplementingPartnerssubmittedtheiryear1reportsattheendofMay2016.
RevisedversionswerethenproducedduringJune2016.Theserespondedto
KnowledgeManagerfeedbackinordertoenhancethecompletenessandutility
ofthedataprovided.Allofthe14ImplementingPartnersthatsubmittedreports
completedalltemplates.
15 Atwo-dayface-to-faceparticipatorytrainingeventwasheldwithrepresentativesofallBrAcEdprojects,organisedjointlybytheFundManagerandKnowledgeManager.ImplementingPartnershadtheopportunitytoshareexamplesoftheirdraftreportswiththegroupforfeedbackandcontinuedraftingthem,whileaskingquestionsandreceivingreal-timefeedbackfromtheKnowledgeManagerandFundManager.
BRACED aims to build the resilience of up to 5 million vulnerable people against
climate extremes and disasters. It does so through a three year, UK Government
funded programme, which supports over 120 organisations, working in 15
consortiums, across 13 countries in East Africa, the Sahel and Southeast Asia.
Uniquely, BRACED also has a Knowledge Manager consortium.
The Knowledge Manager consortium is led by the Overseas Development Institute
and includes the Red Cross Red Crescent Climate Centre, the Asian Disaster
Preparedness Centre, ENDA Energie, ITAD and Thomson Reuters Foundation.
The views presented in this paper are those of the author(s) and do not necessarily represent
the views of BRACED, its partners or donor.
Readers are encouraged to reproduce material from BRACED Knowledge Manager Reports for
their own publications, as long as they are not being sold commercially. As copyright holder, the
BRACED programme requests due acknowledgement and a copy of the publication. For online
use, we ask readers to link to the original resource on the BRACED website.
The BRACED Knowledge Manager generates evidence and learning on
resilience and adaptation in partnership with the BRACED projects and
the wider resilience community. It gathers robust evidence of what works
to strengthen resilience to climate extremes and disasters, and initiates
and supports processes to ensure that evidence is put into use in policy
and programmes. The Knowledge Manager also fosters partnerships to
amplify the impact of new evidence and learning, in order to significantly
improve levels of resilience in poor and vulnerable countries and
communities around the world.
Published December 2016
Website: www.braced.org Twitter: @bebraced Facebook: www.facebook.com/bracedforclimatechange
Cover image: Mikkel Ostergaard
Designed and typeset by Soapbox, www.soapbox.co.uk