evaluation culture and evaluation impact patries boekholt technopolis group vienna, research club...
TRANSCRIPT
![Page 1: Evaluation culture and evaluation impact Patries Boekholt Technopolis Group Vienna, Research Club March 23, 2009](https://reader035.vdocuments.us/reader035/viewer/2022072014/56649eb35503460f94bbaa15/html5/thumbnails/1.jpg)
Evaluation culture and evaluation impact
Patries BoekholtTechnopolis GroupVienna, Research Club March 23, 2009
![Page 2: Evaluation culture and evaluation impact Patries Boekholt Technopolis Group Vienna, Research Club March 23, 2009](https://reader035.vdocuments.us/reader035/viewer/2022072014/56649eb35503460f94bbaa15/html5/thumbnails/2.jpg)
2
This presentation
• What do we mean with an ‘evaluation culture’
• When do evaluations have an impact?
• What can we learn from European and global ‘good -practices’
![Page 3: Evaluation culture and evaluation impact Patries Boekholt Technopolis Group Vienna, Research Club March 23, 2009](https://reader035.vdocuments.us/reader035/viewer/2022072014/56649eb35503460f94bbaa15/html5/thumbnails/3.jpg)
3
A definition
• The systematic application of social research procedures for assessing the...
- conceptualisation- design- implementation- effects
…of public actions
![Page 4: Evaluation culture and evaluation impact Patries Boekholt Technopolis Group Vienna, Research Club March 23, 2009](https://reader035.vdocuments.us/reader035/viewer/2022072014/56649eb35503460f94bbaa15/html5/thumbnails/4.jpg)
4
Fulfilling a need for information about public action
Citizens
Government
Intermediaries
Addressees
Wider society
Financialresources
Advantages
Informationfor managingpublic interventions
Informationfor accountabilityand policy-making
Parliament
![Page 5: Evaluation culture and evaluation impact Patries Boekholt Technopolis Group Vienna, Research Club March 23, 2009](https://reader035.vdocuments.us/reader035/viewer/2022072014/56649eb35503460f94bbaa15/html5/thumbnails/5.jpg)
5
The European evaluation culture
• Evaluation culture is growing in Europe• Positive influence European Commission funding
(particularly Structural Funds)• Anglo-Saxon and Nordic countries by far the most
‘evaluation’ oriented• Strong culture of New Public Management (accountability,
steering on performances)• Separate evaluation units to oversee evaluation activities
• While a legal obligation for evaluations helps, countries with very strong culture (UK, Sweden) do not have a legal basis for evaluation
![Page 6: Evaluation culture and evaluation impact Patries Boekholt Technopolis Group Vienna, Research Club March 23, 2009](https://reader035.vdocuments.us/reader035/viewer/2022072014/56649eb35503460f94bbaa15/html5/thumbnails/6.jpg)
6
Impact assessment in various countries Impact
assessmentemphasis onŌsoft effectsÕ
Outputmeasurement
quantitativeanalysis
Outcomemeasurement
quantitativeanalysis
Quantitativeassessment
widereconomic
impact
Cost-benefitanalysis
(C-BA)
Comparativeanalysis on
basis ofperformance
indicators
ComparativeC-BA
analysis
Australia
Canada
Finland
France
Germany
New Zealand
Norway
UK
USA
EU
![Page 7: Evaluation culture and evaluation impact Patries Boekholt Technopolis Group Vienna, Research Club March 23, 2009](https://reader035.vdocuments.us/reader035/viewer/2022072014/56649eb35503460f94bbaa15/html5/thumbnails/7.jpg)
7
Evaluation in the Policy Cycle
ImplementationImplementation
Agenda setting
Agenda setting
Policy Formulation
Policy Formulation
AdoptionAdoptionEvaluationEvaluation
Politics!
Stakeholders
Who’s listening?
BureaucracyGood practice?
Clear goals?
EvaluationCapacity?
![Page 8: Evaluation culture and evaluation impact Patries Boekholt Technopolis Group Vienna, Research Club March 23, 2009](https://reader035.vdocuments.us/reader035/viewer/2022072014/56649eb35503460f94bbaa15/html5/thumbnails/8.jpg)
8
What improves the chances for a larger impact?
• High-level policy (and political) interest in evaluation activities and their results• E.g. USA Government Performance and Results Act
• Links between evaluations and budget allocation• E.g. Canada and the influence of the Treasury Board
• The existence of Evaluation Frameworks: while designing programmes their future evaluation is already considered
• Timeliness: the evaluation results are in time to influence decision making for the continuation (or termination)• Provides a particular challenge for RTDI evaluations
• A ‘policy learning’ culture: readiness to improve • Distance between those commissioning the evaluation and the
‘evaluated’ (programme) managers• Transparency: evaluation results are in the public domain• Systematic evaluations that also consider the wider policy
context
![Page 9: Evaluation culture and evaluation impact Patries Boekholt Technopolis Group Vienna, Research Club March 23, 2009](https://reader035.vdocuments.us/reader035/viewer/2022072014/56649eb35503460f94bbaa15/html5/thumbnails/9.jpg)
9
Some EU examples
• UK• ROAME
• Finland• System evaluation by Council for Science and Technology Policy• TEKES: separate evaluation unit• Long term monitoring of business data
• The Netherlands:• By law evaluations at least every 5 years and discussed in
Parliament• Now every new major programme needs a baseline study, a
monitoring framework, mid-term evaluations and final after 5 years• But does not necessarily lead to change in policy making
• Germany• BMWI: evaluation of whole range of SME policies
![Page 10: Evaluation culture and evaluation impact Patries Boekholt Technopolis Group Vienna, Research Club March 23, 2009](https://reader035.vdocuments.us/reader035/viewer/2022072014/56649eb35503460f94bbaa15/html5/thumbnails/10.jpg)
10
What can we learn from EU examples?
• Do not expect a simplistic mechanical use of evaluations • Radical changes in policy making rarely happen: needs a
‘window of opportunity’• Evaluation studies alone are not the only piece of strategic
intelligence that make policy makers change policies• Build in user focus in the execution of evaluations• Create a culture of accountability and performance based
steering throughout government policy• Develop evaluation frameworks as integral part of policy making• Make performance based indicators smart and meaningful
• Disentangle evaluations from day-to-day programme management as well as from short term politics
![Page 11: Evaluation culture and evaluation impact Patries Boekholt Technopolis Group Vienna, Research Club March 23, 2009](https://reader035.vdocuments.us/reader035/viewer/2022072014/56649eb35503460f94bbaa15/html5/thumbnails/11.jpg)
11
Thank you
Further information: [email protected]
Technopolis Group has offices in Amsterdam, Ankara, Brighton, Brussels, Paris, Stockholm, Tallinn and Vienna.