The impact of evaluations on developments in academic

careersFrank van der Most

DANS and the e-Humanities Group !

ACUMEN meeting, 9 - 11 January 2013

�1

ACUMENAcademic Careers Understood through

Measurement and Norms

www.research-acumen.eu

!

“Currently, there is a discrepancy between the criteria used in performance assessment and the broader social and economic function of scientific and scholarly research.” (proposal, p. 3)

!2

ACUMEN

✦ “... broader social functions of the scientific system are often not included in its quality control mechanisms.” (proposal, p. 3)

✦ so, ACUMEN will develop

•a portfolio for researchers, and

•criteria and guidelines for Good Evaluation Practices

!3

Research Question

✦ How do evaluations influence the development of individuals’ careers in academia? (What is the impact of evaluations?)

•Use perspective for ACUMEN

✦ Concepts

•Wide notion of evaluations

•Careers + ‘work’

!4

Project partner Country People

Universiteit Leiden Netherlands Paul Wouters, Clifford Tatum, Laura de Ruiter

Bar Ilan University Israel Judit Bar-Ilan, Hadas Shema

Consejo Superior de Investigaciones Cientificas (CSIC) Spain Isidro F. Aguillo, Mario Fernández Pérez,

Amalia Mas Bleda

University of Wolverhampton United Kingdom Mike Thelwall, Kayvan Kousha

Sihtasutus Archimedes Estonia Ülle Must, Arto Mustajoki, Marcus Denton,Taavi Tiirik, Oskar Otsus

Humboldt-Universität zu Berlin Germany Stefan Gradmann, Frank Havemann

Technische Hochschule Wildau Germany Rainer Voss, Hildrun Kretschmer, Ulla Große, Martina Brandt, Frank Hartman, Dana Mietzner

Danmarks Biblioteksskole Denmark Birger Larsen, Jesper W. Schneider,Lorna Wildgaard

Koninklijke Nederlandse Akademie van Wetenschappen (KNAW) Netherlands Andrea Scharnhorst, Sally Wyatt,

Frank van der Most

Who does what?

!6

Work Package Who

1 Evaluation impact KNAW, Archimedes

2 Institutional Web Presence University of Wolverhampton, CSIC

3 Researchers’ Web image Bar Ilan University

4 Gender effects of evaluation Technische Hochschule Wildau

5 New bibliometric indicators Humboldt-Universität ,Danmarks Biblioteksskole

6 ACUMEN portfolio All

7 Management Universiteit Leiden

Impact

!7

A change in a career

Evaluatio

F.V. van der Most, 2012

Impact

!7

A change in a career

Event A

Event B

Event C

Evaluatio

Event U

Event V

Event WEvent D

F.V. van der Most, 2012

Micro story

Method✦ Quite structured, in depth interviews

✦ 4 Countries : UK, Germany, the Netherlands, Poland

✦ 4 Disciplines : Astronomy/astrophysics, Environmental engineering, Philosophy (+), Public Health

✦ 3 levels of seniority: PhD + 5 to 10 years, + 11 to 25 years, + 26 years or more (or equivalents)

!8

Method: interview protocol

✦ What are the three most important developments in your career?

✦ What are the three most influential evaluations in your career?

!9

Work in progress

!10

Number of interviews done AA EE P+ PH Total

UK 4 (3+1) 4 (3+1) 3 (2+1) 4 (3+1) 15 (11+4)

Germany 2 (2+0) 3 (3+0) 3 (2+1) 3 (3+0) 11 (10+1)

The Netherlands (ongoing) 1 (1+0) 2 (1+1) 1 (1+0) 3 (2+1) 7 (5+2)

Poland(feb 2013) - - - -

Numbers in brackets: individuals + deans/HR managers

Indicator, what is the overlap?

!11

Don’t know exactly, yet.

Basic answers: 3 most influential evaluations

!12

Coding group Total

Job applications 21Grants 14

Job-related evaluations (not appl.s) 11

Diplomas 6

Other 5

Paper reviews 5

Informal evaluations (other) 3

Evaluations leading to invitations 3

Feedback 3

Evaluation of teaching 1

Total 72

Basic answers: 3 most influential evaluations

!13

Coding group Total

Basic answers: 3 most influential evaluations

!13

Coding group Total

Job applications 16

Basic answers: 3 most influential evaluations

!13

Coding group Total

Job applications 16Grants 14

Basic answers: 3 most influential evaluations

!13

Coding group Total

Job applications 16Grants 14

Informal evaluations 12

Basic answers: 3 most influential evaluations

!13

Coding group Total

Job applications 16Grants 14

Informal evaluations 12

Diplomas 6

Other 5

Paper review 4

Job-related evaluations (not appl.s) 3

Evaluations leading to invitations 1

Evaluation of teaching 1

Total 62

Examples: informal job applications

!14

Examples: informal job applications

!14

✦ A MA student’s supervisor offers a job on a project leading to a PhD (DE AA, DE PH)

Examples: informal job applications

!14

✦ A MA student’s supervisor offers a job on a project leading to a PhD (DE AA, DE PH)

✦ An informal conversation leads from a temporary to a permanent position as lab manager (UK EE)

Examples: informal job applications

!14

✦ A MA student’s supervisor offers a job on a project leading to a PhD (DE AA, DE PH)

✦ An informal conversation leads from a temporary to a permanent position as lab manager (UK EE)

✦ An earlier conversation later on appears to be an important element in a job application (UK AA)

Examples: evaluations leading to invitations

!15

Examples: evaluations leading to invitations

✦ A HoD asks the interviewee to lead a project, a great opportunity for a relatively junior researcher (UK PH)

!15

Examples: evaluations leading to invitations

✦ A HoD asks the interviewee to lead a project, a great opportunity for a relatively junior researcher (UK PH)

✦ An editor asks the interviewee to become the successor (UK PH)

!15

Examples: other

!16

✦ Course applications (BA to PhD) (2)

✦ ‘Still to be determined’ (1)

✦ Winning a prize (1)

✦ Examinations during a study (1)

Relevance for governance of science✦ Evaluations seem to influence careers (but further

analysis needed)

✦ Share of informal evaluations may be more influential than expected

•The ‘system’ is not so ‘systematic’ ornot so transparent

•Sometimes systematically opaque (Academic freedom)

!17

Thank you for your attention!

!

!

Questions, suggestions?

!

!

!

www.researchaffairs.nl