research assessment for 2010-2014 at the academy of sciences of the czech republic eva zažímalová
DESCRIPTION
The Academy of Sciences of the Czech Republic. Research assessment for 2010-2014 at the Academy of Sciences of the Czech Republic Eva Zažímalová. The Academy of Sciences of the Czech Republic. Aims Principles Organizational structure, disciplines - PowerPoint PPT PresentationTRANSCRIPT
Research assessment for 2010-2014 at the Academy of Sciences of the Czech Republic
Eva Zažímalová
The Academy of Sciences of the Czech Republic
Aims
Principles
Organizational structure, disciplines
Qualitative profile of teams and Institutes (Phase I)
Evaluation of teams and Institutes – other aspects (Phase II)
Time schedule
The Academy of Sciences of the Czech Republic
Research Evaluation Seminar 2014, Prague, April 17, 2014 2/26
Aims
Principles
Organizational structure, disciplines
Qualitative profile of teams and Institutes (Phase I)
Evaluation of teams and Institutes – other aspects (Phase II)
Time schedule
The Academy of Sciences of the Czech Republic
3/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Aims
To obtain qualitative and quantitative information about science in the Academy of Sciences of the Czech Republic, in national, European and worldwide context, including differentiation of teams and Institutes according to quality.
To obtain complex information for strategy management of the Academy as a whole, including financing of the Institutes.
To mediate independent and comparable evaluation and feed-back for management of individual Institutes and research teams.
To meet the demands of responsibility in using public resources, including publishing the evaluation results.
The Academy of Sciences of the Czech Republic
4/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Aims
Principles
Organizational structure, disciplines
Qualitative profile of teams and Institutes (Phase I)
Evaluation of teams and Institutes – other aspects (Phase II)
Time schedule
The Academy of Sciences of the Czech Republic
5/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Previous evalutation (2005-2009), performed in 2010
6th Evaluation exercise at the Academy
Increasing quality of evaluation exercises, including impact(s) on the Institutes
First evaluation on the level of research teams, but evaluation of Institutes as a whole was also included.
Evaluation on-site, based on peer review, international Evaluation Committees organized according to Sections within the Research Areas of the Academy (9 sections – 9 Committees)
6/26Research Evaluation Seminar 2014, Prague, April 17, 2014
The Academy of Sciences of the Czech Republic
Previous evalutation (2005-2009) - drawbacks and imperfections:
Definitions of research teams – too different in some casesSteering Committee – unclear competencesEstablishment of Evaluation Committees (conflict of interest, insufficient knowledge re bibliometry etc.)Establishment of Evaluation Committees according to Sections (not according to disciplines)Uneaqual rigidness between Evaluation Committees (comparison between Sections not possible)Insufficient knowledge about interpretation of scientometric data in Institutes, teams, and Evaluation Committees (summ of IF per capita, IF as the main indicator of quality, underestimation of trends, etc.)It was not obligatory for foreing evaluators to evaluate all teams within the Institute (free choice of only some teams)Short evaluation on-site (one day)Vague definition of rules for appeal/rebuttal
7/26Research Evaluation Seminar 2014, Prague, April 17, 2014
The Academy of Sciences of the Czech Republic
5 Principles:
1. Evaluation within research disciplines
2. Informed peer review
3. Two-phase evaluation
4. Evaluation of scientific quality separated from financing
5. Transparentness
The Academy of Sciences of the Czech Republic
8/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Principle 1 - Evaluation within research disciplines
Quality of outputs and results of teams will be evaluated within disciplines (associated into groups of disciplines) by Panels (phase I) and Review Committees (phase II) consisting of foreign evaluators.
Evaluation within disciplines will respect specific features of particular disciplines and will enable comparison with worldwide and European level of the disciplines.
The Academy of Sciences of the Czech Republic
9/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Principle 2 – Informed peer review
Based on evaluation of quality of particular outputs and results submitted by individual researchers and teams (publications, citations, bibliometric data), and evaluation on-site.
Bibliometric evaluation: standard for natural sciences, not for humanities and social sciences.
The Academy of Sciences of the Czech Republic
10/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Principle 3 - Two-phase evaluation
Phase I: Peer review of research outputs and results – foreign evaluators organized into Panels (remote, web-based). Criterion – quality in international context, regardless the Institutes.
Phase II: Peer review of Institutes – international Review Committees, evaluation results from the Phase I will be summarized at Institutes and integrated into the context of evaluation of other aspects of Intitutes’ activities. Evaluation on-site.
The Academy of Sciences of the Czech Republic
11/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Principle 4 – Evaluation of scientific quality separated from financing
Evaluation of scientific quality without direct relationship to financing.
Final report about the evaluation exercise 2010-2014 will not contain any financial advice and recommendations.
However: Information about utilization of financial resources in relation to financial demands of particular discipline and to scientific outputs and results will be one of the outputs of evaluation exercise.
The Academy of Sciences of the Czech Republic
12/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Principle 5 – Transparentness
Preparation and arrangement: Institutes will be informed regularly. Trial registration to disciplines.
Publishing: Results of the evaluation exercise will be published in the frame of the Academy, and presented on the web-site of the Academy. Co-operation with PR-department.
Enhancement of quality of evaluation and operational communication with Institutes (in Phase II).
The Academy of Sciences of the Czech Republic
13/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Aims
Principles
Organizational structure, disciplines
Qualitative profile of teams and Institutes (Phase I)
Evaluation of teams and Institutes – other aspects (Phase II)
Time schedule
The Academy of Sciences of the Czech Republic
14/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Organizational structureWithin the Institutes: Assessment units
(each usually composed of one research team)
INSTITUTETeams
Team member – Researcher (categories 3 to 5 according to Career Structure of the Academy
Team member
The Academy of Sciences of the Czech Republic
15/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Disciplines
Based on the official international lists (ERC, OECD, REF/RAE) in context to JCR (WoS) categories.
OECD („Revised field of science and technology (FOS) classification in the Frascati manual”): 6 groups of disciplines / 42 disciplines
for the Academy 5 groups of disciplines / 27 disciplines (based on trial registration):
1. Natural sciences (including Agricultural sciences)
2. Engineering and technology
3. Medical and health sciences
4. Social sciences
5. Humanities
The Academy of Sciences of the Czech Republic
16/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Disciplines – Natural sciences (example)
Discipline Group of disciplines Panel
1 Natural sciences
1.1 Mathematics 1
1.2 Computer and information sciences 2
1.3 Physical sciences 3
1.4 Chemical sciences 4
1.5 Earth and related environmental sciences 5
1.6B Biochemistry and molecular cell biology, biophysics, …
6
1.6O Biological sciences including biotechnology and agricultural sciences
7
The Academy of Sciences of the Czech Republic
17/26Research Evaluation Seminar 2014, Prague, April 17, 2014
OECD List of disciplines and JCR categoriesExample: Discipline 1.4 Chemical sciences – sub-disciplines
OECD classification WoS (JCR categories)
1 Organic chemistry CHEMISTRY, ORGANIC 292 Inorganic and nuclear chemistry CHEMISTRY, INORGANIC &
NUCL.26
3 Physical chemistry CHEMISTRY, PHYSICAL 304 Polymer science POLYMER SCIENCE 14
85 Electrochemistry (dry cells,
batteries, fuel cells, corrosion metals, electrolysis)
ELECTROCHEMISTRY 47
6 Colloidochemistry 7 Analytical chemistry CHEMISTRY, ANALYTICAL 248 CHEMISTRY, APPLIED 259 CHEMISTRY,
MULTIDISCIPLINARY28
The Academy of Sciences of the Czech Republic
18/26Research Evaluation Seminar 2014, Prague, April 17, 2014
The Academy: Disciplines and outputs
Počty publikací WoS v oborech 1 – 27 (2010-2013)
Natural sciences
Engineering & technologyMedical and
health sciences
Social sciences Humanities
Multidisciplinary sciences
Number of papers 2010-2013 (WoS)
The Academy of Sciences of the Czech Republic
19/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Aims
Principles
Organizational structure, disciplines
Qualitative profile of teams and Institutes (Phase I)
Evaluation of teams and Institutes – other aspects (Phase II)
Time schedule
The Academy of Sciences of the Czech Republic
20/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Phase I: Qualitative profile of teams and Institutes
Each researcher: Maximum 4 scientific outputs
Team: The best result(s) (consisting of outputs) for a team (in relation to number of researchers in a team).
Remote evaluation
Qualitative profile will be generated (outputs and results assorted into 5 classes)
The Academy of Sciences of the Czech Republic
21/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Evaluation: Data and requirements for evaluators
Bibliometry for all submitted outputs
All outputs at the disposal (web links), for team results all relevant outputs
Requirement: Evaluation of content in context with the discipline
The Academy of Sciences of the Czech Republic
22/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Aims
Principles
Organizational structure, disciplines
Qualitative profile of teams and Institutes (Phase I)
Evaluation of teams and Institutes – other aspects (Phase II)
Time schedule
The Academy of Sciences of the Czech Republic
23/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Phase II: Evaluation of teams and Institutes – other aspects
o In Phase II, other aspects of teams and Institutes will be evaluated, such as:
- grants, projects, collaborations- management (human resources, career policy)- education (teaching, supervision of students) - other scientific activities (scientific infrastructures,
popularization etc.)
o The Institute and teams will submit self-evaluation report
o Evaluation on-site: International Review Committees
The Academy of Sciences of the Czech Republic
24/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Aims
Principles
Organizational structure, disciplines
Qualitative profile of teams and Institutes (Phase I)
Evaluation of teams and Institutes – other aspects (Phase II)
Time schedule
The Academy of Sciences of the Czech Republic
25/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Time schedule - tentative
Key dates
Beginning End
Period for evaluation 01. 01. 2010 31. 12. 2014
Documentation finished 31. 06. 2014
Functionality of information system
31. 06. 2014
Registration 01. 01. 2015 31. 03. 2015
Impact on financing 01. 01. 2017
The Academy of Sciences of the Czech Republic
26/26Research Evaluation Seminar 2014, Prague, April 17, 2014
Acknowledgements:
prof. Jana Musilová, prof. Jiří Chýla, prof. Petr Ráb
Members of Working Group of the Council for Sciences
Members of Working Group of the Academy Council
The Academy of Sciences of the Czech Republic
Research Evaluation Seminar 2014, Prague, April 17, 2014