Measuring the Impact of the RDP

Issues being addressed at an EU level with regards to measuring the impact of the

Rural Development programmesB. Schuh

1www.nrn.ie

Content

• Assessment of impacts in the RDPs – the basics

• Process of assessing RDP impacts• Evaluation architecture/ challenges to

overcome• The seven fields of impacts (econ. growth,

employment creation, productivity, biodiversity, HNV, water quality, climate change)

www.nrn.ie 2

Assessment of impacts in the RDPs – the basics:

• CMEF – the „bible“• Intervention logic • Additional

programme specific indicators

• Evaluation questions

www.nrn.ie 3

Context baseline

SWOT

StrategyObjectives Impact

Result

Outputs

InputsObjectiverelated baseline

Hierarchyo

f Objectives

Defi

nition o

f measu

reIdentifyingstrengthsand weaknesses

supporting and counterproductive trends

reference for impact

Process of assessing RDP impacts:

• Gauging the evidence of change

• Identifying the drivers of change

• Understanding change and concluding on future interventions

www.nrn.ie 4

Otherinformation(statistics,

surveys/inquiries…)

Additional Programme specific

indicators

Trends of baselineindicators

Common Impact Indicators

Programme-specificEvaluation Questions

Conclusions and Recommendations

EvaluationResults

Information gatheringand analysis

Interpretation Judgement

Body ofevidence

Common Evaluation Questions

(per Axis and horizontal)

Output & ResultIndicators

Reflectionspace

Evaluation architecture/ challenges to overcome:

Factors determinig evaluation architecture:• Dealing with uncertainties• Reducing complexity through a

consistent approach• Constraints in utilization of the

evaluation results – the evaluation and policy cycle

Methodological challenges to overcome:• The counterfactual assessment of impacts

quasi-experimental design, non-experimental design; DiD method

• Taking into account and cross-relating impacts at micro and macro level e.g. econometric modeling, CGE models, system dynamics modeling

• Netting out the programme effects by reducing deadweight, leverage, displacement, substitution and multiplier effects

• data collection and processing qualitative & quantitative data, FADN & Co

• Bridging the gap between measuring impact indicators and providing answers on programme impacts qualitative methods as add-ons – interviews, CS

www.nrn.ie 5

The socio-economic impact indicators:

• Economic Growth:– Calculation via DiD methods, quantification possible, micro-macro link through

modeling approaches

• Employment creation– Calculation via Propensity Score Matching, Standard regression model,

Assessing employment effects at macro level - modelling– Time lag, Missing critical mass – CS, Welfare effects

• Labour productivity- Calculation via DiD methods, quantification possible, micro-macro link through

modeling approaches- Limitations concerning measuring labour productivity Competitive

Performance, Revealed Comparative Advantage, Growth Competitiveness Indicator, Domestic Resource Cost

www.nrn.ie 6

The environmental impact indicators – special issues:

• Data availability • Systemic borders

– Environmental impacts do evolve also from those measures, which do not deem that such impacts will occur

– Conception of the environment within the evaluation (ecosystem functions vs. ecosystem services)

• Difficult to depict the full range of rather vast, complex fields of environmental phenomena like “climate change” or “Biodiversity loss”

• “evaluation” vs. assessment – aggregation methods not easily applicable

• Cumulative impacts – crossing effects between environmental impacts www.nrn.ie 7

The environmental impact indicators:

• Biodiversity– Measured by Farmland Bird Index (FBI) – bottom-up aggregation of micro level

observations– Crucial issue of regional/ national specifics, bottom-up assessment, additional

information (‘control’ other influences), more than birds

• High Nature Value farming/ forestryMeasured through:

• Land cover characteristics, especially farmland with a high proportion of semi-natural vegetation and in some cases a diversity of land cover types.

• Farming practices, especially a low use of inputs (including live stock density) and specific practices such as shepherding, late hay-cutting, orchard grazing and arable fallowing.

– Strong dependence on baseline data comparison of baseline conditions, main challenge indicator as „work in progress“

www.nrn.ie 8

The environmental impact indicators 2:

• Water quality– Measured as changes in gross nutrient balance (GNB), should be interpreted

as a potential risk indicator for water quality only; assessment by bottom up approach – aggregation as methodological challenge (modelling – e.g. RAUMIS)

– Many uncertainties remain different land cover, land use and farming types & atmospheric N fixation and deposition - measuring water quality in agricultural catchment, net nitrogen balance as additional indicators

• Climate change– Measured as net greenhouse gas emissions reduction and production of

renewable energy– Limitations Additionality/net effects, Displacement of energy & production,

Boundary issues, issue of the temporal attribution of longer-term impacts to the policy period of the spending

www.nrn.ie 9