![Page 1: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/1.jpg)
The impact of evaluations on developments in academic
careersFrank van der Most
DANS and the e-Humanities Group !
ACUMEN meeting, 9 - 11 January 2013
�1
![Page 2: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/2.jpg)
ACUMENAcademic Careers Understood through
Measurement and Norms
www.research-acumen.eu
!
“Currently, there is a discrepancy between the criteria used in performance assessment and the broader social and economic function of scientific and scholarly research.” (proposal, p. 3)
!2
![Page 3: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/3.jpg)
ACUMEN
✦ “... broader social functions of the scientific system are often not included in its quality control mechanisms.” (proposal, p. 3)
✦ so, ACUMEN will develop
•a portfolio for researchers, and
•criteria and guidelines for Good Evaluation Practices
!3
![Page 4: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/4.jpg)
Research Question
✦ How do evaluations influence the development of individuals’ careers in academia? (What is the impact of evaluations?)
•Use perspective for ACUMEN
✦ Concepts
•Wide notion of evaluations
•Careers + ‘work’
!4
![Page 5: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/5.jpg)
Project partner Country People
Universiteit Leiden Netherlands Paul Wouters, Clifford Tatum, Laura de Ruiter
Bar Ilan University Israel Judit Bar-Ilan, Hadas Shema
Consejo Superior de Investigaciones Cientificas (CSIC) Spain Isidro F. Aguillo, Mario Fernández Pérez,
Amalia Mas Bleda
University of Wolverhampton United Kingdom Mike Thelwall, Kayvan Kousha
Sihtasutus Archimedes Estonia Ülle Must, Arto Mustajoki, Marcus Denton,Taavi Tiirik, Oskar Otsus
Humboldt-Universität zu Berlin Germany Stefan Gradmann, Frank Havemann
Technische Hochschule Wildau Germany Rainer Voss, Hildrun Kretschmer, Ulla Große, Martina Brandt, Frank Hartman, Dana Mietzner
Danmarks Biblioteksskole Denmark Birger Larsen, Jesper W. Schneider,Lorna Wildgaard
Koninklijke Nederlandse Akademie van Wetenschappen (KNAW) Netherlands Andrea Scharnhorst, Sally Wyatt,
Frank van der Most
![Page 6: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/6.jpg)
Who does what?
!6
Work Package Who
1 Evaluation impact KNAW, Archimedes
2 Institutional Web Presence University of Wolverhampton, CSIC
3 Researchers’ Web image Bar Ilan University
4 Gender effects of evaluation Technische Hochschule Wildau
5 New bibliometric indicators Humboldt-Universität ,Danmarks Biblioteksskole
6 ACUMEN portfolio All
7 Management Universiteit Leiden
![Page 7: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/7.jpg)
Impact
!7
A change in a career
Evaluatio
F.V. van der Most, 2012
![Page 8: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/8.jpg)
Impact
!7
A change in a career
Event A
Event B
Event C
Evaluatio
Event U
Event V
Event WEvent D
F.V. van der Most, 2012
Micro story
![Page 9: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/9.jpg)
Method✦ Quite structured, in depth interviews
✦ 4 Countries : UK, Germany, the Netherlands, Poland
✦ 4 Disciplines : Astronomy/astrophysics, Environmental engineering, Philosophy (+), Public Health
✦ 3 levels of seniority: PhD + 5 to 10 years, + 11 to 25 years, + 26 years or more (or equivalents)
!8
![Page 10: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/10.jpg)
Method: interview protocol
✦ What are the three most important developments in your career?
✦ What are the three most influential evaluations in your career?
!9
![Page 11: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/11.jpg)
Work in progress
!10
Number of interviews done AA EE P+ PH Total
UK 4 (3+1) 4 (3+1) 3 (2+1) 4 (3+1) 15 (11+4)
Germany 2 (2+0) 3 (3+0) 3 (2+1) 3 (3+0) 11 (10+1)
The Netherlands (ongoing) 1 (1+0) 2 (1+1) 1 (1+0) 3 (2+1) 7 (5+2)
Poland(feb 2013) - - - -
Numbers in brackets: individuals + deans/HR managers
![Page 12: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/12.jpg)
Indicator, what is the overlap?
!11
Don’t know exactly, yet.
![Page 13: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/13.jpg)
Basic answers: 3 most influential evaluations
!12
Coding group Total
Job applications 21Grants 14
Job-related evaluations (not appl.s) 11
Diplomas 6
Other 5
Paper reviews 5
Informal evaluations (other) 3
Evaluations leading to invitations 3
Feedback 3
Evaluation of teaching 1
Total 72
![Page 14: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/14.jpg)
Basic answers: 3 most influential evaluations
!13
Coding group Total
![Page 15: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/15.jpg)
Basic answers: 3 most influential evaluations
!13
Coding group Total
Job applications 16
![Page 16: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/16.jpg)
Basic answers: 3 most influential evaluations
!13
Coding group Total
Job applications 16Grants 14
![Page 17: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/17.jpg)
Basic answers: 3 most influential evaluations
!13
Coding group Total
Job applications 16Grants 14
Informal evaluations 12
![Page 18: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/18.jpg)
Basic answers: 3 most influential evaluations
!13
Coding group Total
Job applications 16Grants 14
Informal evaluations 12
Diplomas 6
Other 5
Paper review 4
Job-related evaluations (not appl.s) 3
Evaluations leading to invitations 1
Evaluation of teaching 1
Total 62
![Page 19: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/19.jpg)
Examples: informal job applications
!14
![Page 20: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/20.jpg)
Examples: informal job applications
!14
✦ A MA student’s supervisor offers a job on a project leading to a PhD (DE AA, DE PH)
![Page 21: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/21.jpg)
Examples: informal job applications
!14
✦ A MA student’s supervisor offers a job on a project leading to a PhD (DE AA, DE PH)
✦ An informal conversation leads from a temporary to a permanent position as lab manager (UK EE)
![Page 22: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/22.jpg)
Examples: informal job applications
!14
✦ A MA student’s supervisor offers a job on a project leading to a PhD (DE AA, DE PH)
✦ An informal conversation leads from a temporary to a permanent position as lab manager (UK EE)
✦ An earlier conversation later on appears to be an important element in a job application (UK AA)
![Page 23: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/23.jpg)
Examples: evaluations leading to invitations
!15
![Page 24: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/24.jpg)
Examples: evaluations leading to invitations
✦ A HoD asks the interviewee to lead a project, a great opportunity for a relatively junior researcher (UK PH)
!15
![Page 25: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/25.jpg)
Examples: evaluations leading to invitations
✦ A HoD asks the interviewee to lead a project, a great opportunity for a relatively junior researcher (UK PH)
✦ An editor asks the interviewee to become the successor (UK PH)
!15
![Page 26: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/26.jpg)
Examples: other
!16
✦ Course applications (BA to PhD) (2)
✦ ‘Still to be determined’ (1)
✦ Winning a prize (1)
✦ Examinations during a study (1)
![Page 27: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/27.jpg)
Relevance for governance of science✦ Evaluations seem to influence careers (but further
analysis needed)
✦ Share of informal evaluations may be more influential than expected
•The ‘system’ is not so ‘systematic’ ornot so transparent
•Sometimes systematically opaque (Academic freedom)
!17
![Page 28: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/28.jpg)
![Page 29: The impact of evaluations on developments in academic careers](https://reader033.vdocuments.us/reader033/viewer/2022060108/554e865db4c90526358b46c6/html5/thumbnails/29.jpg)
Thank you for your attention!
!
!
Questions, suggestions?
!
!
!
www.researchaffairs.nl