counterfactual impact evaluation: what it can (and cannot) do for cohesion policy alberto martini...
TRANSCRIPT
![Page 1: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/1.jpg)
Counterfactual Counterfactual impact evaluation: impact evaluation:
what it can (and cannot) what it can (and cannot) do for cohesion policy do for cohesion policy
Alberto MartiniProgetto Valutazione
Torino, Italy
![Page 2: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/2.jpg)
ALL I REALLY NEED TO KNOW I ALL I REALLY NEED TO KNOW I LEARNED IN KINDERGARTENLEARNED IN KINDERGARTEN
by Robert Fulghumby Robert FulghumShare. Play fair. Don't hit people. Clean up your own mess. Wash your hands before you eat. Flush.
![Page 3: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/3.jpg)
ALL IT REALLY MATTERS IN ALL IT REALLY MATTERS IN IMPACT EVALUATION COMES IMPACT EVALUATION COMES FROM COMMON SENSEFROM COMMON SENSE
It’s nice to have an impact. Not all we obtain is due to our actions.Some things happen without our help.To improve things we must understand’emWe must separate what we caused from
what would happen anyway.Flush.
![Page 4: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/4.jpg)
Do we need counterfactuals?Do we need counterfactuals?The answer is simple: it depends on what
we need (can, want) to know and for which purpose
I’ll follow the COSCE approach (Common Sensical Counterfactual Evaluation)
[COSCE = Conference On Security and Cooperation in Europe]
What would have happened anyway What would have happened anyway = counterfactual= counterfactual
![Page 5: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/5.jpg)
COSCE rule n.1COSCE rule n.1If your purpose is to be accountable, don’t
worry too much about counterfactuals◦Your main worry is to show that the money was
spent ◦Maybe you want to show how well it was spent◦Maybe you want to show for whom it was spent◦You might go further by showing your
contribution to objectives; e.g. to the Lisbon strategy
◦To impress DG-Regio, use a macro-model
![Page 6: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/6.jpg)
COSCE rule n.2COSCE rule n.2If your purpose is to improve policy,
macro models will not doIf your purpose is to improve policy,
probably indicators will not doIf your purpose is to improve policy, you
need to learn:
◦What works and, if it does, why it works
◦What does not work and, if it doesn’t, why it doesn’t work
![Page 7: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/7.jpg)
COSCE rule n.3COSCE rule n.3Learning “what works” precedes logically
learning “why it works”Otherwise we do not know what to explain
Learning why it works (or doesn’t) is:◦More important◦More interesting◦More difficult
than learning what worksThis is why it should be done later
![Page 8: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/8.jpg)
COSCE rule n.4COSCE rule n.4Counterfactul Impact Evaluation tries to learn
something about “what works” on average (not very interesting) and for whom it works (data permitting)
It produces numbersIt requires good data and large samplesIt imposes non-testable assumptionsIts results are NOT the truth, are NOT universal
laws, are NOT scientificIt is (should be) a fallible, improvable,
intellectually honest human enterprise
![Page 9: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/9.jpg)
COSCE rule n. 5COSCE rule n. 5Theory-based Impact Evaluation tries to learn
something about “why it works” indentifying the mechanisms that make a policy produce its effects (or fail to do so)
It produces narratives and insightsIt collects its data through qualitative methods
and doesn’t need large samplesIt develops a theory of change and then
observes policies as they are implemented, to learn which elements of the theory are verified
![Page 10: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/10.jpg)
COSCE rule n.6COSCE rule n.6To learn something about “what works” one
needs to clarify Effects (impacts) on what?
Which outcomes Y Effects (impacts) of what?
Which treatment T
COSCE curse n. 1COSCE curse n. 1Effects and impacts are the same thing, the best example of distinction without a difference
![Page 11: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/11.jpg)
COSCE rule n. 7COSCE rule n. 7The heart of CIE is to answer the question:what is the direction, size and significance of
the effect of treatment T of outcome Y?
AN EXAMPLE
A program providing subsidies to increase R&D expenditures among small and
medium enterprises
“subsidizing SME to do more R&D”
![Page 12: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/12.jpg)
A MULTIPLE CHOICE TESTA MULTIPLE CHOICE TEST
What is the effect of the subsidies?What is the effect of the subsidies? the number of R&D projects funded and completed the take-up rate of the subsidy among eligible SME the increase in R&D expenditures
among subsidized SME the difference in R&D expenditure among subsidized and non subsidized SME none of the above
![Page 13: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/13.jpg)
the number of R&D projects funded and completed the take-up rate of the subsidy among eligible SME
The number can be very high, the take-up rate can be 100 %, the effect
can be zero
COSCE curse n. 2COSCE curse n. 2the number of R&D projects is not a “gross impact”. It is not an impact. It’s a measure of activity. There is no such thing as a gross impact
![Page 14: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/14.jpg)
the increase in R&D expenditures among subsidized SME
COSCE curse n. 3COSCE curse n. 3The deadweight (DW) is nothing else than the counterfactual. The only special thing about it is that is used when money is clearly wasted. Demonstrable Waste (DW) is a better name for it
The increase in not an effect, the subsidies might have gone to firms
with growing R&D expenditures
![Page 15: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/15.jpg)
the difference in R&D expenditure among subsidized and non subsidized SME
The post-treatment difference in outcomes does not identify any effect, the difference
might be all due to initial differences (selection bias)
COSCE curse n. 3COSCE curse n. 3The Commission is stuck on the decomposition
“gross impact=net effect + deadweight”
The world literature focuses on the decomposition “observed difference=effect + selection bias”
![Page 16: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/16.jpg)
![Page 17: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/17.jpg)
What does COSCE have to say about What does COSCE have to say about the limitations of counterfactual the limitations of counterfactual
impact evaluation?impact evaluation?
In some quarters, CIE is seen as a universal approach, able to solve all the inferential problems through use of
ever more sophisticated methods.
COSCE disagree and views the CIE as an important contribution, with important limitations in their
applicability to Structural Funds, both in terms or relevance and compatibility.
![Page 18: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/18.jpg)
Support for R&D projects
Transport infra-
structure
Human capital
investment
Urban renewal
Renewable energy
Investment support
Behavioral (vs. redistributive) motive
Replicable nature (vs. idiosyncratic)
Homogenous treatment (vs. composite)
Large numbers of eligible units
++ ++ ++ + + +
++ ++ + - - ++
++ + + - - - +
+ ++ + - - - ++
Different types of cohesion policies
Relevance and com
patibility
![Page 19: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/19.jpg)
What timing for counterfactual What timing for counterfactual impact evaluation?impact evaluation?
When it is prospective, i.e. it is designed together with the intervention, impact evaluation can have a strong disciplinary effect.
First, it can help focus the attention of both policy-makers and beneficiaries on objectives.
Secondly, it creates an incentive to assemble the information necessary to assess results.
Thirdly, it brings to light the criteria by which beneficiaries are selected
BARCA DIXITBARCA DIXIT
![Page 20: Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy amartini@prova.org](https://reader035.vdocuments.us/reader035/viewer/2022081516/55152c0d55034673228b5576/html5/thumbnails/20.jpg)
Above timing, Above timing, above relevance, above relevance,
above compatibility, above compatibility,
the most important determinant of the diffusion of the most important determinant of the diffusion of counterfactual impact evaluation is counterfactual impact evaluation is
the interest and willingness, on the part of some the interest and willingness, on the part of some influential stakeholder, of truly influential stakeholder, of truly
learning about what workslearning about what works
and whyand why..