ICF Hectares Indicator Methodology Testing FINAL REPORT | July 2015 This report describes the results of an eight month project to test a methodology for estimating the area of forest loss avoided due to ICF supported activities ­ The Hectares Indicator. It was funded by the European Space Agency (ESRIN Contract No.4000112345/14/I­NB: Earth Observation Support for Assessing the Performance of UK government’s ICF Forest Projects), with additional support from NERC (Innovation Voucher Scheme).

Authors Karin Viergever 1, Veronique Morel 1, Richard Tipper 1, Edward Mitchard 2. 1. Ecometrica; 2. University of Edinburgh, School of GeoSciences

ICF Hectares Indicator Methodology Testing | July 2015 | Ecometrica | FINAL REPORT | Page 2

Acknowledgements Ecometrica would like to thank DFID and DEFRA staff for their assistance in organising field visits, for guidance and help with data acquisition. We also thank project teams and stakeholders in Brazil, including Francisco Oliveira, Carla Leal and Patricia Abreu (MMA), Edson Sano (IBAMA), Dalton Valeriano (INPE), Bernadette Lange and Daniella Arruda (World Bank), Ana Paula Gutierrez (Defra), in Ghana including Winston Asante (Kwame Nkrumah University of Science and Technology), Bright Obeng Kankam (CSIR­FORIG), Edward Obiaw (GFC­RMSC), Julia Falconer (DFID), and in Nepal Ramu Subedi, Bishwas Rana and Dharam Raj Uprety (MSFP), Basanta Shrestha, Hammad Gilani (ICIMOD), Yam Prasad Pokharel and Anish Joshi (FRA Nepal), Sabitha Thapa (DFID Nepal) Johan Oldekop and Anil Bhargava (IFRI). This project was funded by the European Space Agency (ESA), with additional support from NERC (Innovation Voucher Scheme). 1

Glossary

GIS Geographic Information Systems

GFW Global Forest Watch

ICIMOD International Centre for Integrated Mountain Development

UMD University of Maryland

1 ESRIN Contract No.4000112345/14/I­NB: Earth Observation Support for Assessing the Performance of UK government’s ICF Forest Projects

ICF Hectares Indicator Methodology Testing | July 2015 | Ecometrica | FINAL REPORT | Page 3

Background The UK’s International Climate Fund (ICF) is a combined, co­ordinated ODA commitment from the UK (DFID, DECC and Defra) of £3.84 billion from 2011 to 2016 towards international action on climate change mitigation and adaptation. Approximately 20% of the ICF activity is directed to forests, in support of the UK’s goal of halving deforestation in developing countries by 2020. ICF resources are applied through various bilateral and multilateral channels, including some pre­existing programmes. Throughout the ICF there is a strong emphasis on monitoring, evaluation and learning with the aim of producing knowledge and evidence about how low carbon, climate resilient development, including reduced forest loss, supports growth and reduces poverty. One of the challenges for effective evaluation and learning is the availability of robust evidence of outcomes, and the ICF has agreed a number of Key Performance Indicators (KPIs) that should provide good quality evidence for the monitoring, evaluation and learning process, and for assessing value for money. KPI #8, the Hectares Indicator, is one of 14 KPIs covering a range of social, economic and environmental impacts expected from the ICF.

The Hectares Indicator is a measure of the avoided loss of forests, and increase in forest area, resulting from ICF programmes. It is an important outcome measure for all forest related ICF activities and will underpin other KPIs on ecosystem services and biodiversity. In 2013 DFID asked Ecometrica, the University of Edinburgh and LTSI to review and recommend improvements to an initial methodology for the Hectares Indicator. This review included an extensive literature review of methods used in projects, programmes and related scientific research. The report , published in 2

2014, recommended a risk based approach using data from Earth Observation (EO) by satellites to provide much of the evidence on forest cover and forest loss. In October 2014 Ecometrica, in partnership with the University of Edinburgh, began an ESA funded project to test the application of a risk based methodology in three countries selected by the ICF ­ Nepal, Brazil and Ghana.

2 Tipper et al. (2014) The ICF Hectares Indicator: a review and suggested improvements to the indicator methodology (Download)

ICF Hectares Indicator Methodology Testing | July 2015 | Ecometrica | FINAL REPORT | Page 4

Project Activities The four stage risk based method, summarised in the diagram below, was tested on ICF programmes in Nepal (Terai and Churia forests) , Brazil (Cerrado biome)3

and Ghana . Test implementation, from November 2014 to April 2015, involved field visits to programme offices and collaborating organisations in Nepal and 4 5

Brazil, whereas the Ghana case was carried out as a desk study with expert input via telephone. Key questions addressed were the availability of suitable data to complete the method, skills and resources required, and the comparability of results from different countries. Web mapping platforms were set up to collect and analyse the spatial data for each country. The project also assessed the accuracy of the University of Maryland’s (UMD) global forest cover loss data , distributed by Global Forest Watch in selected areas 6

in Ghana and the Brazilian Cerrado. High resolution multi­date optical satellite data were secured for the test areas and relevant time intervals. The project did not examine the question of attribution of results to different initiatives or efforts occurring within or influencing an area. The process of attributing portions of an outcome to different contributors to a programme may have adverse consequences and may not yield useful information about what works, where and how.

3 DFID Multi­stakeholder forest programme Nepal (link) 4 DEFRA Cerrado programme (link) 5 Forest Governance, Markets and Climate. DFID website (link) 6 Mitchard et al. (2015). Assessment of the accuracy of University of Maryland (Hansen et al.) Forest Loss Data in 2 ICF project areas – component of a project that tested an ICF indicator methodology (download report, download appendices to report )

ICF Hectares Indicator Methodology Testing | July 2015 | Ecometrica | FINAL REPORT | Page 5

Why A Risk Based Method? Ecometrica’s 2014 review of methods applicable for setting credible reference levels for programmes to avoid deforestation and forest degradation, examined a 7

wide range of approaches and recent applications in practical projects, the scientific literature and theoretical papers. Most methods can be grouped into one of three classes: (1) extrapolation from historic rates; (2) economic models and specific case arguments; (3) risk mapping.

Extrapolation of % Deforestation Economic models and case arguments Risk Mapping

For: Simple to apply to a fixed area, e.g. country

Against:

Evidence shows historic loss rates are not a good predictor of future losses

Deforestation patterns are uneven so % loss rates cannot be applied to sub­jurisdictional areas

Provides no information about which forests are at risk of being lost (% is applied evenly to all forest within the area)

For: Can capture many national, regional and local

details Good for exploring causality and scenarios

Against:

Difficult to apply consistently, generic models tend to fail in specific locations

Demanding and costly in time and data Difficult to verify, since they are often complex

(few examples of models that have been verified)

Accuracy of such models not demonstrated in the scientific literature

For: Can apply consistently at any scale from local

to international Shows specific areas of forest at risk ­

demonstrating need for intervention Risk classification can be verified and

calibrated using forest loss data Useful tool for discussions on programme

needs and effectiveness Against:

More complicated than an extrapolation of historical deforestation rates

Complex scenario analysis is more limited than economic models

Risk mapping was recommended because it is consistently applicable to projects and programmes of different scales, it reflects the need for intervention in specific areas and it has potential to generate good quality evidence for evaluation purposes.

7 Tipper et al. (2014) The ICF Hectares Indicator: a review and suggested improvements to the indicator methodology (Download)

ICF Hectares Indicator Methodology Testing | July 2015 | Ecometrica | FINAL REPORT | Page 6

Results All stages of the method were completed in all three test countries. Sufficient data was found for forest mapping and risk mapping in all cases. National stakeholders in Brazil and Nepal provided useful inputs on the best forest cover maps to use, and factors to use in risk mapping. Links to detailed results and processes are given in the table below. The most challenging application was Ghana, where the definition of relevant forest areas was complicated by ICF’s broad policy­level intervention, which potentially affects many areas of the country. This was addressed by considering all high forests in southern Ghana as relevant (an assumption that could be reviewed in future). Secondly, the definition of forest in Ghana is such that a large proportion of the national territory is classified as forest, and losses of trees from a landscape infrequently pass the threshold to be defined as deforestation . Third, much of the forest change occurs in small, irregular patches that are part 8

of smallholder activities, making them difficult to detect. Finally, high levels of cloud cover in Ghana limited the availability of good quality optical satellite images for assessing the UMD forest loss data.

Methodology Step App (password required)

Brazil icf­cerrado.ourecosystem.com

Nepal icf­nepal.ourecosystem.com

Ghana icf­ghana.ourecosystem.com

1. Map relevant areas completed: used UMD forest cover for test purposes but PROBIO data expected to replace this. Effect of different forest definitions can be seen on App.

completed: used ICIMOD forest cover map since national forest map not available. Good agreement with UMD forest map. Specific areas provided by DFID.

completed: used UMD 2000 forest cover updated to 2012.

2. Map risk completed: risk map, with inputs from national experts in country (Cerrado risk methodology report )

completed: risk map, with inputs from national experts in country (Terai & Churia risk methodology report)

completed: risk map, with inputs from UK experts ­ desk study (southern Ghana risk methodology report)

3. Monitor deforestation completed: UMD forest loss was incorporated into Cerrado App. Accuracy of UMD detection of Cerrado loss was assessed

completed: UMD forest loss was incorporated into Nepal App.

completed: UMD forest loss was incorporated into Ghana App. Accuracy of UMD detection of high forest loss was assessed

4. Calculate avoided loss completed: App runs automated calculations for relevant areas

completed: App runs automated calculations for relevant areas

completed: App runs automated calculations for relevant areas

8 Deforestation is defined as conversion of land from forest to non­forest

ICF Hectares Indicator Methodology Testing | July 2015 | Ecometrica | FINAL REPORT | Page 7

Draft Reporting Format The following template shows how the results of the Hectares Indicator analysis for a given area can be presented in a single slide format: The template was completed in under 1 hour, taking results from the mapping platform.

ICF Hectares Indicator Methodology Testing | July 2015 | Ecometrica | FINAL REPORT | Page 8

Forest Loss Detection Accuracy Assessment Calculations of avoided forest loss require estimates of actual loss to be compared with a reference level. Accurate measurement of forest loss is therefore required irrespective of the method used to determine a reference level. The most practical way to detect forest loss across large areas and over multiple years is through earth observation by satellites. An ideal forest loss detection data product for this purpose would have: (a) global coverage, (b) annual repeat cycle, (c) high accuracy, and (d) low cost. The University of Maryland (UMD) global forest loss data product is an innovative forest monitoring product that has been widely disseminated by Global Forest 9

Watch . The UMD product meets the criteria of coverage, repeat cycle and cost. Furthermore it is produced at high resolution (approx. 25 m pixels), and is 10

generally compatible with the Hansen et al (2013) tree canopy cover. However, there is limited understanding of the accuracy of the UMD data products in the context of particular regions, forest types and change types. The following table shows quantitative and qualitative aspects of accuracy in change detection.

Quantitative measures of accuracy Qualitative aspects of accuracy

Errors of omission­ forest loss that is missed by the data product Errors of commission­ false detections of forest loss Mis­allocation between years­ forest loss correctly detected but

assigned to wrong year

Information about the limits of the detection product, types of change that may be confused or missed. For example small irregular patches of forest loss are typically more difficult to detect than large regular patches.

To determine the suitability of of the UMD forest loss product for detecting forest loss within relevant areas of ICF forestry programmes we conducted accuracy assessments in two contrasting areas: (1) a sample area within the Brazil cerrado project region where the main land use change is conversion of cerrado to medium and large scale mechanised agriculture; (2) a sample area of high forest in Ghana, where forest disturbance occurs at small scales driven by subsistence farming. We obtained high resolution images (SPOT and RapidEye) for these areas for multiple dates and carried out accuracy assessments in two ways(both processes were carried out independently but were subject to a common quality assurance process):

Comparison with results of a semi­automated, manually supervised classification of sample area Comparison with fully manual classification of sample area

9 Hansen, et al. (2013) “High­Resolution Global Maps of 21st­Century Forest Cover Change.” Science 342 (15 November): 850–53. 10 Global Forest Watch data access: (link)

ICF Hectares Indicator Methodology Testing | July 2015 | Ecometrica | FINAL REPORT | Page 9

Results of Forest Loss Detection Accuracy Assessment Brazil Cerrado Assessment Results Ghana High Forest Assessment Results

Quantitative Accuracy Low to moderate errors of commission (~5­25% of detection = false positive) Moderate errors of omission (10­20% loss missed) Moderate mis­allocation (2­4% loss mis­allocated)

Quantitative Accuracy Low errors of commission

(<5 % of detection = false positive) Very high errors of omission

(>80 % loss missed) Mis­allocation ­ not quantified, as not applicable

Qualitative Assessment UMD forest loss works well in this area. Large regular patches of cerrado woodland conversion to agriculture are reliably detected with few errors of commission. There is some mis­allocation, due to late detection.

Qualitative Assessment UMD forest loss only picked up a small fraction of forest loss occurring in small patches of Ghana’s high forest, apparently driven by smallholder agriculture and extraction.

Details of the assessment and results are given in Mitchard et al (2015) (download report, download appendices to report)

ICF Hectares Indicator Methodology Testing | July 2015 | Ecometrica | FINAL REPORT | Page 10

Consistency and Comparability We found that the Hectares Indicator methodology under test provided a consistent way of quantifying and visualising the forest at risk, and measuring detected losses relative to estimated probability of loss. The amount of forest at different levels of risk can be compared between countries / regions and within project areas. The rates of loss within different areas, risk categories and other forest situations can also be compared to provide evidence on the relative effectiveness of different interventions, targeting of programme activities and value for money. Relevant analytical questions might include:

how much high risk forest does our intervention address? how forest loss is avoided in area X versus area Y? is our intervention succeeding in protecting high / medium risk forests? are we using resources to protect forest at low risk of loss?

Risk maps should be updated periodically to take account of emerging risks or changing national circumstances.

Measures of risk and maps of forest at risk should be comparable between countries and regions:

ICF Hectares Indicator Methodology Testing | July 2015 | Ecometrica | FINAL REPORT | Page 11

Practical Requirements and Transferability The local skills and technical requirements for implementing the methodology depend to a large extent on the amount of central support available. If mapping platforms and earth observation advice are available centrally then the main requirements for local teams are:

Good understanding of the areas where the programme is operating (ability to point out the relevant districts, municipalities or forest areas) Good understanding of the threats / risks facing the forests in the relevant areas, and a broader level of understanding of forest risks and the nature of

forest changes is essential Analytical capabilities that allow interpretation of the results of the hectares indicator calculations, understanding how the calculations work and what

underlying drivers will change the results over time. If earth observation and spatial platform services are not available then medium to high levels of GIS skills would be needed within each country or reporting entity, along with supporting equipment and software. In the absence of a centralised geospatial platform service there would be a staffing requirement to ensure integration and maintenance of the spatial assets required and generated by the indicator work. The methodology is transferable to other countries and could be extended to support the analysis of other Key Performance Indicators, notably KPI 6 “Change in greenhouse gas emissions”, and KPI 10 “Value of ecosystem services preserved or generated”. This could be achieved by incorporating carbon values and ecosystem service values into the mapping platform. While programmes that have clear target areas are relatively straightforward to assess using this methodology, broader policy or market related programmes, such as FGMC that have an indirect effect on land use change are harder to assess, given that they may be one of several influencing initiatives. 11

11 Forest Governance, Markets and Climate. DFID website (link)

ICF Hectares Indicator Methodology Testing | July 2015 | Ecometrica | FINAL REPORT | Page 12

Conclusions

1. The method proposed for the Hectares Indicator is feasible, practical and cost­effective to implement (with appropriate tools) and can be applied to a range of projects. Decisions are required regarding the selection of forest definitions to use. Where suitable local or national forest maps are not available the University of Maryland Global Forest Cover 2000 product can provide a good starting point.

2. Coordination efforts are required to ensure the availability of relevant data sets within ICF countries. Many countries and institutions are involved in

mapping forests and forest change related issues. However, data assets are often fragmented, poorly documented or unavailable for use by donors and stakeholders. More effort is needed to ensure that effective data curation and dissemination practices are adopted within relevant institutions. Brazil is becoming an exemplar of good practice.

3. Risk mapping has advantages over alternative reference level setting methods (extrapolation of historic rates and complex modelling) in providing

information relevant to the allocation of resources and understanding of whether drivers of forest loss are being addressed. Risk mapping methods can be improved over time and updated periodically to reflect new circumstances and data availability.

4. The global forest loss mapping from the University of Maryland (UMD), distributed in partnership with Global Forest Watch, is a useful product, however,

its suitability for different regions and forest types varies considerably and should be assessed case by case before application. Collaboration with UMD and Global Forest Watch to improve the accuracy of the product in specific regions of interest and to provide more information about its capabilities is recommended. Global Forest Watch should consider how information about the accuracy of the product is reflected in communications.

5. Monitoring forest change in fragmented forests and agro­forest landscapes, particularly in regions of high cloud cover, is a technical challenge that

requires further research into appropriate techniques. Combined use of radar and optical methods should be examined, with airborne and spaceborne LiDAR also offering potential for sampling­based higher­fidelity estimates. We see great potential in the future for products combining data from the EU’s Sentinel 1 & 2 satellite series: by later this decade two of each of these satellites will be operational, offering high resolution (~10 m) radar and optical data respectively, with very frequent image capture ­ as high as one every 8 days for Sentinel 2. ESA’s planned BIOMASS mission in 2020 should also mark a substantial step forward in terms of data availability, though it should be noted that it only has an expected mission life of 4 years and no successor is planned.

6. The Hectares Indicator analysis can provide a wealth of insights into the effectiveness of donor programmes aiming to conserve or manage forests,

however most users will require training in the interpretation of the results. It will be important to integrate these interpretation skills within ICF’s monitoring, evaluation and learning process.

ICF Hectares Indicator Methodology Testing | July 2015 | Ecometrica | FINAL REPORT | Page 13

References and Related Documents Project reports: Mitchard, et al. (2015). Assessment of the accuracy of University of Maryland (Hansen et al.) Forest Loss Data in 2 ICF project areas – component of a project that tested an ICF indicator methodology. Mitchard, et al. (2015). Appendices to report: Assessment of the accuracy of University of Maryland (Hansen et al.) Forest Loss Data in 2 ICF project areas – component of a project that tested an ICF indicator methodology. Morel, et al. (2015). Risk based methodology for assessing avoided deforestation with application in ICF forest programmes in Brazilian Cerrado. Morel, et al. (2015). Risk based methodology for assessing avoided deforestation with application in ICF forest programmes in Southern Ghana. Morel, et al. (2015). Risk based methodology for assessing avoided deforestation with application in ICF forest programmes in Terai and Churia in Nepal. Background report: Tipper, et al. (2014). The ICF Hectares Indicator: A review and suggested improvements to the indicator methodology.