research assessment for 2010-2014 at the academy of sciences of the czech republic eva zažímalová...

Research assessment for 2010-2014 at the Academy of Sciences of the Czech Republic

Eva Zažímalová

The Academy of Sciences of the Czech Republic

Aims

Principles

Organizational structure, disciplines

Qualitative profile of teams and Institutes (Phase I)

Evaluation of teams and Institutes – other aspects (Phase II)

Time schedule


Research Evaluation Seminar 2014, Prague, April 17, 2014 2/26

Aims

Principles




Time schedule


3/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Aims

To obtain qualitative and quantitative information about science in the Academy of Sciences of the Czech Republic, in national, European and worldwide context, including differentiation of teams and Institutes according to quality.

To obtain complex information for strategy management of the Academy as a whole, including financing of the Institutes.

To mediate independent and comparable evaluation and feed-back for management of individual Institutes and research teams.

To meet the demands of responsibility in using public resources, including publishing the evaluation results.



Aims

Principles




Time schedule



Previous evalutation (2005-2009), performed in 2010

6th Evaluation exercise at the Academy

Increasing quality of evaluation exercises, including impact(s) on the Institutes

First evaluation on the level of research teams, but evaluation of Institutes as a whole was also included.

Evaluation on-site, based on peer review, international Evaluation Committees organized according to Sections within the Research Areas of the Academy (9 sections – 9 Committees)



Previous evalutation (2005-2009) - drawbacks and imperfections:

Definitions of research teams – too different in some casesSteering Committee – unclear competencesEstablishment of Evaluation Committees (conflict of interest, insufficient knowledge re bibliometry etc.)Establishment of Evaluation Committees according to Sections (not according to disciplines)Uneaqual rigidness between Evaluation Committees (comparison between Sections not possible)Insufficient knowledge about interpretation of scientometric data in Institutes, teams, and Evaluation Committees (summ of IF per capita, IF as the main indicator of quality, underestimation of trends, etc.)It was not obligatory for foreing evaluators to evaluate all teams within the Institute (free choice of only some teams)Short evaluation on-site (one day)Vague definition of rules for appeal/rebuttal



5 Principles:

1. Evaluation within research disciplines

2. Informed peer review

3. Two-phase evaluation

4. Evaluation of scientific quality separated from financing

5. Transparentness



Principle 1 - Evaluation within research disciplines

Quality of outputs and results of teams will be evaluated within disciplines (associated into groups of disciplines) by Panels (phase I) and Review Committees (phase II) consisting of foreign evaluators.

Evaluation within disciplines will respect specific features of particular disciplines and will enable comparison with worldwide and European level of the disciplines.



Principle 2 – Informed peer review

Based on evaluation of quality of particular outputs and results submitted by individual researchers and teams (publications, citations, bibliometric data), and evaluation on-site.

Bibliometric evaluation: standard for natural sciences, not for humanities and social sciences.



Principle 3 - Two-phase evaluation

Phase I: Peer review of research outputs and results – foreign evaluators organized into Panels (remote, web-based). Criterion – quality in international context, regardless the Institutes.

Phase II: Peer review of Institutes – international Review Committees, evaluation results from the Phase I will be summarized at Institutes and integrated into the context of evaluation of other aspects of Intitutes’ activities. Evaluation on-site.



Principle 4 – Evaluation of scientific quality separated from financing

Evaluation of scientific quality without direct relationship to financing.

Final report about the evaluation exercise 2010-2014 will not contain any financial advice and recommendations.

However: Information about utilization of financial resources in relation to financial demands of particular discipline and to scientific outputs and results will be one of the outputs of evaluation exercise.



Principle 5 – Transparentness

Preparation and arrangement: Institutes will be informed regularly. Trial registration to disciplines.

Publishing: Results of the evaluation exercise will be published in the frame of the Academy, and presented on the web-site of the Academy. Co-operation with PR-department.

Enhancement of quality of evaluation and operational communication with Institutes (in Phase II).



Aims

Principles




Time schedule



Organizational structureWithin the Institutes: Assessment units

(each usually composed of one research team)

INSTITUTETeams

Team member – Researcher (categories 3 to 5 according to Career Structure of the Academy

Team member



Disciplines

Based on the official international lists (ERC, OECD, REF/RAE) in context to JCR (WoS) categories.

OECD („Revised field of science and technology (FOS) classification in the Frascati manual”): 6 groups of disciplines / 42 disciplines

for the Academy 5 groups of disciplines / 27 disciplines (based on trial registration):

1. Natural sciences (including Agricultural sciences)

2. Engineering and technology

3. Medical and health sciences

4. Social sciences

5. Humanities



Disciplines – Natural sciences (example)

Discipline Group of disciplines Panel

1 Natural sciences

1.1 Mathematics 1

1.2 Computer and information sciences 2

1.3 Physical sciences 3

1.4 Chemical sciences 4

1.5 Earth and related environmental sciences 5

1.6B Biochemistry and molecular cell biology, biophysics, …

6

1.6O Biological sciences including biotechnology and agricultural sciences

7



OECD List of disciplines and JCR categoriesExample: Discipline 1.4 Chemical sciences – sub-disciplines

OECD classification WoS (JCR categories)

1 Organic chemistry CHEMISTRY, ORGANIC 292 Inorganic and nuclear chemistry CHEMISTRY, INORGANIC &

NUCL.26

3 Physical chemistry CHEMISTRY, PHYSICAL 304 Polymer science POLYMER SCIENCE 14

85 Electrochemistry (dry cells,

batteries, fuel cells, corrosion metals, electrolysis)

ELECTROCHEMISTRY 47

6 Colloidochemistry 7 Analytical chemistry CHEMISTRY, ANALYTICAL 248 CHEMISTRY, APPLIED 259 CHEMISTRY,

MULTIDISCIPLINARY28



The Academy: Disciplines and outputs

Počty publikací WoS v oborech 1 – 27 (2010-2013)

Natural sciences

Engineering & technologyMedical and

health sciences

Social sciences Humanities

Multidisciplinary sciences

Number of papers 2010-2013 (WoS)



Aims

Principles




Time schedule



Phase I: Qualitative profile of teams and Institutes

Each researcher: Maximum 4 scientific outputs

Team: The best result(s) (consisting of outputs) for a team (in relation to number of researchers in a team).

Remote evaluation

Qualitative profile will be generated (outputs and results assorted into 5 classes)



Evaluation: Data and requirements for evaluators

Bibliometry for all submitted outputs

All outputs at the disposal (web links), for team results all relevant outputs

Requirement: Evaluation of content in context with the discipline



Aims

Principles




Time schedule



Phase II: Evaluation of teams and Institutes – other aspects

o In Phase II, other aspects of teams and Institutes will be evaluated, such as:

- grants, projects, collaborations- management (human resources, career policy)- education (teaching, supervision of students) - other scientific activities (scientific infrastructures,

popularization etc.)

o The Institute and teams will submit self-evaluation report

o Evaluation on-site: International Review Committees



Aims

Principles




Time schedule



Time schedule - tentative

Key dates

Beginning End

Period for evaluation 01. 01. 2010 31. 12. 2014

Documentation finished 31. 06. 2014

Functionality of information system

31. 06. 2014

Registration 01. 01. 2015 31. 03. 2015

Impact on financing 01. 01. 2017



Acknowledgements:

prof. Jana Musilová, prof. Jiří Chýla, prof. Petr Ráb

Members of Working Group of the Council for Sciences

Members of Working Group of the Academy Council


Research Evaluation Seminar 2014, Prague, April 17, 2014

research assessment for 2010-2014 at the academy of sciences of the czech republic eva zažímalová...

Documents

evaluation of institutes

teamsshort evaluation

institutesfirst evaluation

comparable evaluation

evaluation results

evaluation committees

definitions of research

level of research teams