science &technology policy in italy: the role of evaluation
DESCRIPTION
Slides by Giorgio Sirilli (CERIS-CNR, ROARS) CONACYT Seminar on evaluation of STI policies Mexico City June 19, 2014TRANSCRIPT
![Page 1: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/1.jpg)
CONACYT
Seminar on evaluation of STI policies
Mexico City
June 19, 2014
S&T policy in Italy: The role of evaluation
Giorgio SirilliCERIS – CNR
ROARS
![Page 2: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/2.jpg)
2
Outline of the presentation
Science policy in Italy and bejondThe most relevant evaluation exercises:
lessons learnedThe use of evaluation in S&T policy makingConcluding remarks
![Page 3: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/3.jpg)
3
Italy
60 million inhabitants
![Page 4: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/4.jpg)
4
Italy, a founding member of he European Union
Italy’s policies, including R&D policy, are heavily affected by the the European Union
![Page 5: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/5.jpg)
5
Some data on Italy
R&D expenditure = 19,834 million euro
Researchers = 106,151 (full-time equivalent)
R&D/GDP = 1.25%
90 universities
40 Public Research Agencies (PRAs)
![Page 6: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/6.jpg)
6
SectorR&D
expenditurePercentage
Government 2.723 13.8
Privare non profit 621 3.1
Business enperprises 10.813 54.5
University 5.677 28.6
Total 19.834 100,0
Source: ISTAT
R&D expenditure by performing sector - Italy 2012 (million euro)
The Lisbon target
66.6%
![Page 7: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/7.jpg)
7
Sector Number Percentage
Government 18,780 17.7
Privare non profit 3,735 3.5
Business enperprises 39,808 37.5
University 43,828 41.3
Total 106,151 100.0
Source: ISTAT
Researchers by performing sector - Italy - 2011 (number in equivalent full time)
![Page 8: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/8.jpg)
8
Science and technology policy in Italy: main actors
Junior minister for Research (1963)
Ministry of Universities and Research (1989)
Ministry of industry (Economic development)
Junior minister for Innovation (2000)
Ministry of Economy
![Page 9: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/9.jpg)
9
Ministry of universities and research
Committee for Research Evaluation (CIVR)
National Committee for the Evaluation of the University System (CNUSV)
National Agency for the Evaluation of Universities and Research (ANVUR)
Universities
National Research Council
Other research agencies
Ministry of economic development
The evaluation system in Italy: main actors
![Page 10: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/10.jpg)
Start: 2011
Members of the Editorial board: 14
Collaborators: 217
Contacts: 8 million (November 2011 – June 2014)
Average daily contacts: 8,000 November 2011; 13,000 in 2014)
Articles published: 1,627
Comments by readers: 24,880
ROARS is ranked 8° among the top cultural national blogs
ROARS, a genuine expression of democracy and participation, has become a very important player in the policy debate and in policy making
A new player in S&T policy
![Page 11: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/11.jpg)
11
The neo-conservative wave of the 1980s
Ronald Reagan Margaret Thatcher
![Page 12: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/12.jpg)
12
The new catchwords
New public management
Value for money
Accountability
Relevance
![Page 13: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/13.jpg)
13
The UK
“All areas of public expenditure should demonstrate ‘value for money’”
Thatcher’s three Es:
economy
efficiency
effectiveness
![Page 14: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/14.jpg)
14
The UK
Research Assessment Exercise (RAE)
Research Excellence Framework (REF) (impact)
“The REF will over time doubtless become more sophisticated and burdensome. In short we are creating a Frankenstein monster” (Ben Martin)
![Page 15: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/15.jpg)
15
“Why shoud we payresearchers if we make the best shoes in the world?”Silvio Berlusconi
“Culture does not provide food”(Con la cultura non si mangia)Giulio Tremonti
The neo-conservative wave in Italy in the 1990s
![Page 16: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/16.jpg)
16
The neo-conservative wave in Italy in the 1990s
Letizia Moratti
Italian minister of education and research
“You first show that you use efficiently and effectively the public money, then we will open the strings of the purse” - Never happened
![Page 17: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/17.jpg)
17
Science policy in Italy
The neo-liberal approach after Reagan and ThatcherThe legitimation of R&D and higher education
![Page 18: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/18.jpg)
18
The Entetreneurial State
The “visible hand” of the state
Innovationsmicro hard disksiliconmulti touchinteernetgpssirilcd screenlitium batteries
![Page 19: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/19.jpg)
19
Science policy in Italy
The neo-liberal approach after Reagan and ThatcherThe legitimation of R&D and higher educationThe autonomy of University and PRAs
- the (mis)use of autonomy- the impact on the scientific community
The reduction of resourcesThe role of evaluation as a S&T policy instrument
![Page 20: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/20.jpg)
20
Is the Italian scientific system efficient and effective?
The answer is YES
![Page 21: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/21.jpg)
21
Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 6.1)
(dati ISI Web of Knowledge, Thomson-Reuters)http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf
# p
ap
ers
/million
US
D (
PP
P)
Number of publications (2004)/unit of R&D expenditure
![Page 22: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/22.jpg)
22
Average annual growth of publications (2004-2010) (%)
Fonte: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 3.2)•(dati ISI Web of Knowledge, Thomson-Reuters)
http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf
![Page 23: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/23.jpg)
Number of citations to the Web of Knowledge articles (2004-2010)
Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 5.3)(dati ISI Web of Knowledge, Thomson-Reuters)
•http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf
![Page 24: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/24.jpg)
24
Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 5.3)(dati ISI Web of Knowledge, Thomson-Reuters)
•http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf
# c
ites/m
illion
US
D (
PP
P)
Number of citations (articles 2004)/unit of R&D expenditure
![Page 25: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/25.jpg)
25
The reduction of public resources for R&D
Data on government budget appropriations for R&D in Italy show that from 2009 to 2012 there has been a 9.8% decrease in absolute values (from €9,778 million to €8,822 million) and a 12.7% decrease in real terms
In particular, research funds granted to universities by the ministry of education and research have declined from €366 to €96 million; the Prin shrunk from €170 million in 2010 and 2011 to €38 million in 2012, and may drop to zero in 2014, while the Ordinary fund for higher education, which provides block funding for universities, shrunk from €7.5 billion in 2009 to €6.6 billion in 2013
![Page 26: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/26.jpg)
26
The difficult situation of the National Research Council
In the period 2005 – 2014 the overall financing from the ministry of education decreased in real terms of 5.3% – the overall budget of the agency is about 1 billion euro.
The financing has witnessed a dramatic structural change: the block grant which is managed autonomously by CNR has decreased by 20.4% while the financing linked to specific projects has rocketed by 1,675%.
In 2014 the block grant of €500 million will not even be enough to cover the obligatory expenses (salaries and fixed costs) of €620 million. This means that CNR has lost its autonomy and that, in order to survive, the agency will have to activate contracts linked to research and technical services both from public and private sources
![Page 27: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/27.jpg)
27
Established in 2011
A government agency, not an authority
The relationship with MIUR (Ministry of education, universities and research)
ANVUR activities:
1. Evaluation of the Quality of Research (EQR)
2. National Scientific Qualification (NSQ)
3. Accreditation of universities (AVA)
Evaluation
![Page 28: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/28.jpg)
28
Model: Research Assessment Exercise (RAE)Objective: Evaluation of Areas, Research structures and
Departments (not of researchers) Reference period: 2004-2010Start: 2011Actors:- ANVUR (National Agency for the Evaluation of Universities and
Research Institutes)- GEV (Evaluation Groups) (#14) (450 experts involved plus
referees)- Research structures (universities, research agencies)- Departments- Subjects evaluated: researchers (university teachers and PRA
researchers)
ANVUR Evaluation of the Quality of Research
![Page 29: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/29.jpg)
29
Researchers’ products to be evaluated- journal articles- books and book chapters- patents- designs, exhibitions, software, manufactured items, prototypes, etc.
University teachers: 3 “products” over the period 2004-2010
Public Research Agencies researchers: 6 “products” over the period 2004-2010
Scores: from 1 (excellent) to -1 (missing)
ANVUR Evaluation of the Quality of Research
![Page 30: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/30.jpg)
30
Outcome:- mobilisation of the scientific community- heavily criticised (methodologies, too fast)- a measure of effectiveness- considered a first exercise to be improved- used to assign resources by ministry
ANVUR Evaluation of the Quality of Research
![Page 31: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/31.jpg)
31
Bibliometrics vs peer review
NSE, SSH
Journals: A, B, C
Median
Examiners
National Scientific Qualification
![Page 32: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/32.jpg)
32
Was it worth?
69,000 applications
46,000 habilitated
1,100 legal actions
cost per habilitation (5,000 euro per candidate habilitated; 25, 000 euro per candidate hired)
the link between VQR and ASN
National Scientific Qualification
![Page 33: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/33.jpg)
33
Under way
Strong opposition from universities
Accreditation of universities
![Page 34: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/34.jpg)
34
Too powerful?
Too bureaucratic?
Playing a political game?
The role of ANVUR
![Page 35: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/35.jpg)
35
The relationship between ANVUR and ROARS
A “frank”, and sometimes tough, dialogue
ROARS’s methodological suggestions often accepted
![Page 36: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/36.jpg)
Some estimates
VQR 300 million euro (180,000 “products”)
NSQ: 1126 million euro (46,000 candidates)
RAE: 126 million euro (180.000 “products”)
ANVUR budget: 9 million euro
Rule of thumb: less than 1% of R&D budget devoted to its evaluation – this is not often the case
Evaluation is expensive
![Page 37: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/37.jpg)
37
Knowledge: the bundle
![Page 38: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/38.jpg)
University
teaching
research
“third mission”(60 indicators)
The learned society
Research agencies
research
problem solving
management
![Page 39: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/39.jpg)
The Frankenstein monster (Martin)
League tables
Competition vs cooperation of scientists
Opportunistic behaviour
The split of the academic community (the good and the bad guys)
Peer review vs bibliometrics
NSE vs SSH
The threat of the equilibrium amongst the components of the bundle (research at the expense of other activities)
The other side of the coin of evaluation
![Page 40: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/40.jpg)
Performance based research funding systems are national systems of ex-post university research output evaluation used to inform distribution of funding
Performance based research funding systems
![Page 41: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/41.jpg)
41
“The rationale of performance funding is that funds should flow to institutions where performance is manifest: ‘performing’ institutions should receive more income than lesser performing institutions, which would provide performers with a competitive edge and would stimulate less performing institutions to perform. Output should be rewarded, not input.”
Herbst, 2007
Performance based research funding systems
The rationale
![Page 42: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/42.jpg)
42
Share of university funding dependent on Performance-Based Research Funding Systems
Country Share (%) of what
Australia 6 Total revenue
Italy 2 Block grant
New Zeland 10 Block grant
Norway 2 Total funding
Slovak Republic 15 Total funding
UK 25 Research support
Hicks D., Reseach Policy (2012)
![Page 43: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/43.jpg)
43
Share of university funding dependent on Performance-Based Research Funding Systems
Hicks D., Reseach Policy (2012)
“The distribution of university research funding is something of an illusion”
“It is the competition for prestige that creates powerful incentives within university systems”
“Performance-based research funding systems aim at excellence: they may compromise other important values such as equity and diversity”
“Control by professional elites”
![Page 44: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/44.jpg)
44
Few evaluations carried out in the last years
Some analyses of innovation policies using innovation data (CIS)
Some analyses of the impact of a specific measure for the financing of R&S in firms through a counterfactual method
Additionality difficult to evaluate
Impact difficult to evaluate
No major pressure (or interest) from policy makers to evaluate support to industry
Evaluation of public policies and programs in Italy
![Page 45: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/45.jpg)
45
Main methodological results of the Italian experience:
- impact difficult to measure
- very few exercises carried out
- indicators need improvement
Evaluation of the “Plans for development of scientific and technological networks”
![Page 46: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/46.jpg)
46
Concluding remarks
The Italian S&T system is under pressureLegitimation of S&T and higher educationEvaluation as a key policy instrumentEvaluation was introduced lately but it is expected to stayEvaluation was used in an ideological wayEvaluation exercises heavily criticised from a
methodological point of viewImpact on the scientific community and on researchers’
behaviorToo much evaluation is harmfulMisuse of the results of evaluationEvaluation is expensiveConcentration in “excellent” institutions or spreading?Paradox: the S&T Italian system is good
![Page 47: Science &Technology policy in Italy: The role of evaluation](https://reader034.vdocuments.us/reader034/viewer/2022051609/547d6a6eb37959652b8b525c/html5/thumbnails/47.jpg)
Thank you for attention