1

Joint meeting of ESF Evaluation Partnership and DG REGIO Evaluation Network in Gdańsk (Poland)

on 8 July 2011

The Use of Counterfactual Impact Evaluation Methods in Cohesion Policy

Bernhard BoockmannInstitute for Applied Economic Research (IAW), Tübingen (Germany)

2

Structure of the Presentation Motivation and key questions

Which Cohesion policies provide suitable treatments for counterfactual evaluation?

Isolating the impact of Cohesion policy from the effect of other interventions

Strategies for reliable and robust identification of treatment effects

Meeting data requirements

Conclusions

3

Motivation and Key Questions Counterfactual evaluation is an established

methodology for deriving policy impacts and provides a basis for cost-benefit analysis.

The key questions for the paper are the following:- Which particular difficulties arise in the

application of counterfactual evaluation in the context of Cohesion policy?

- What limits the applicability of counterfactual evaluations in Cohesion policy?

- How can these difficulties and limitations be overcome?

4

Structure of the Discussion

5

Which Cohesion Policies Provide Suitable Treatments for Counterfactual Evaluation?

6

Conditions for Suitable Treatments

(1)The treatment must be clearly defined and separable from other treatments; it must be homogenous, so that all treated individuals in the population receive the same treatment.

(2)The treatment must be unambiguously directed to a group of individuals, while there are other individuals who are not treated (i.e., treatment is not universal).

(3)Feedback, spillover and other indirect effects from the treated to the non-treated must be excluded.

(4)The effect of the treatment must be measurable and non-negligible in magnitude.

7

Example: OP for Saxony (Germany), 2007-2013 The OP centres on four priority axes. There is a large range of interventions :

- Measures for unemployed individuals (such as classroom and on-the-job training programmes).

- Measures directed at particular target groups (programmes for the illiterate, persons starting their own companies, young people).

- Measures at the level of projects and activities (such as support for mentoring and programmes for the strengthening of local actors).

8

Example: OP for Saxony (Germany), 2007-2013 Some interventions do not provide a suitable

treatmentCondition violated

Intervention

1 Qualification measures specific to particular occupations (e.g. health care)

2 Financial support given to initiatives and networks

3 Displacement effects relevant for programmes directed at the unemployed with particular disadvantages

4 Programmes for the development of soft skills and intercultural competencies

9

Example: OP for Saxony (Germany), 2007-2013 Interventions are not suited for counterfactuals

impact assessment and must be examined with other methods, such as simulation, macro-evaluation or qualitative approaches.

There remain important interventions from the operational programme that could quite successfully be evaluated through counterfactual methods, such as general qualification measures for the unemployed, hiring subsidies or support to start-ups by the unemployed.

10

Isolating the Impact of Cohesion Policy from the Effects of Other Interventions ESF interventions are closely connected with the

policies of the member states, filling gaps or complementing existing programmes.

Can the additional impact of the ESF be ascertained using counterfactual evaluation problems?

Answer: yes, by multiple treatment approaches. However, applying them is costly, as several

measures must be analysed jointly. Costs may be reduced by conducting studies

jointly for national and Cohesion policy interventions.

11

Strategies for Reliable and Robust Identification of Treatment Effects All non-experimental methods in counterfactual

require identifying assumptions. It is not sufficient simply to have “a control group”.

E.g., matching requires that outcome and treatment must not be correlated once the covariates have been taken into account.

12

Example: Identifying assumption for the matching estimatorLanguage-training programme for non-native parts of the population (e.g. Dmitrijeva 2008).

If participation depends on language skills before the start of the programme (e.g., the programme is directed towards those with the lowest skills), and initial language skills are unobserved, results may be biased.

This is because participants may consist of those most in need of training (who possess systematically lower initial language skills than non-participants).

Hence, initial language skills should be in the data.

13

Meeting Data Requirements

Advantages of administrative data over survey data:

- Large number of observations.- No unit non-response or panel attrition.- Longitudinal dimension.- Exactness, especially regarding past events

and histories.- Less costly in comparison with survey data.

14

Member States Making Administrative Data Available for Evaluation Research

Country Studies (examples)

Austria Lechner and Wiehler (2007), Winter-Ebmer (2006)

Belgium Cockx et al. (2006)

Czech Republic Terell (1999)

Denmark Rosholm and Swarer (2004)

Estonia Leetmaa and Vörk (2004)

France Crépon et al. (2007)

Germany Biewen et al. (2007), Fitzenberger et al. (2007), Lechner et al. (2007)

Italy Paggiaro et al. (2005)

Latvia Dmitrijeva (2008)

Netherlands Abbring et al. (2005)

Slovak Republic Lubyova and van Ours (1999)

Spain García Pérez and Rebollo Sanz (2009)

Sweden Albrecht et al. (2004), Forslund et al. (2004), Frederiksson and Johannson (2008), Richardson and van den Berg (2002)

United Kingdom Blundell et al. (2004)

15

Meeting Data Requirements

Problems with administrative data:- Survey data is easier to handle.- There are confidentiality problems and lack

of information for certain groups.- There is too little individual information

available (see language-training example above).

- Wunsch and Lechner (2011): Inclusion of information on individual health and on the last employer is particularly important.

16

Meeting Data Requirements

A possible (but costly) way out:- Merge survey data with administrative

data.- Example for an ESF programme evaluation:

Deeke and Kruppe 2006.- Multiple-purpose linked administrative and

survey data such as the IZA evaluation dataset (Caliendo et al. 2010) reduce some of the costs of conducting reliable counterfactual evaluations.

17

Conclusions (1) A choice must be made regarding which

interventions to evaluate using counterfactual impact evaluation. Alternative approaches must be used when causal impact evaluation is not feasible.

The methodology is apt in principle to identify the added value of Cohesion policy. However, this requires taking into account national or regional programmes. These should be evaluated in conjunction with Cohesion policy interventions.

18

Conclusions (2) Provision of administrative data is of key

importance. Survey data is insufficient for state-of-the-art evaluations.

ESF funding should be used to create an appropriate data base if no data base is available.

Until administrative data with sufficient quality of information are available, surveys provide a useful (but costly) alternative to complement administrative data.

19

Conclusions (3) It is important to establish quality standards

for causal impact evaluation All data should be made available for

replication purposes. Include cost-benefit analysis based on estimated impacts.

Ensure that the results are discussed in scientific debate.