relevance and impact of humanities research and how eric can help jack spaapen royal netherlands...
TRANSCRIPT
Relevance and Impact of
Humanities Research
and how ERiC can help
Jack SpaapenRoyal Netherlands Academy of Arts and Sciences
Humanities Conference Vienna 15 and 16 December 2008
Relevance, impact and evaluation
Various functions of evaluationVarious functions of evaluation Judgment about Quality, both scientific and societal Accountability to university, to government, to society Improvement of research quality and impact
Various problemsVarious problems
Functions get mixed in evaluations, but with different weight Some seem to be more obvious than others Criteria and indicators slanted towards scientific quality and to practices of natural and biomedical sciences Different actors have different expectations
Peer review under pressure, evaluation procedures tooPeer review under pressure, evaluation procedures too
Evaluating Research in Context (www.eric-project.nl)
endorsed byendorsed by * Dutch Academy, * National research council (NWO), * Association of Dutch universities, * Council of the professional schools,* Rathenau Institute (Science Systems assessment) * QANU, * Ministry of Education and Sciences
ERiC: Development of alternative evaluation methods
WHY ERiCWHY ERiC
to inform boards / government about societal relevance of research (accountability and Lisbon ambitions)
to help research areas that are not happy with dominant evaluation approaches : humanities, social sciences, technical sciences, health research, MIT, etc.
Core problem
The world is changing
Science (academic) is changing into research (MIT) (Latour)
Evaluation is not yet changing, or at least still dominated by criteria and indicators that used to work in the old situation
methodological development
RMW: the societal impact of health research [2002] SWR / RGW: judging research on its merits [2005] ERiC: evaluating research in context [2005, 2007] AWT: alfa stralen (humanities shine) disciplinary developments: health research (RGO /
UMCs), development research (CERES); law (disciplinary committee)
pay back, UK research councils, Scandinavian countries, etc.
examples from outside Europe: USA, Australia, Canada
“Judging research on its merits”[KNAW/SWR/RGW 2005]
Collaboration of two Academy Councils, social sciences and humanities
Dissatisfied with current evaluation systems mainly focusing on SCI (high impact journals)
Two problems: 1. doesn’t necessarily fit research communication patterns in soc sc and hum; 2. doesn’t value societal output which is important for a lot of fields
Search for more fitting evaluation schemes
Necessary conditions for better evaluations
Self evaluation focus on both scientific reputation
and communication, and also on interaction with other audiences
List of target groups: peers, students, professionals, policy makers, business, broader public
List of indicators per target group: publications, citations, but also text books, reviews, grants from policy, collaborations with business, professionals, awards, popular publications, etc.
Benchmarking is critical process
Goals of ERiC
stimulate debate about alternative methods of evaluation, fit for all disciplines regarding the value of research for society
develop methods to do such evaluations
website www.eric-project.nl
how does ERiC work?
Context group as a central unit publications guidelines workshops, national and international pilot studies : agricultural research,
pharmaceutical research, architecture, law, engineering
European project : SIAMPI website : www.eric-project.nl
Results so far
Working definition of Societal value:o As a broad concept : societal relevanceo As a target-oriented concept : societal impacto As a mainly economic concept : valorization
these three have different consequences for evaluation
Assessment approach fit for all disciplines: based on an analysis of the mission orientation of a
research group o scientific communityo professional sectoro Industryo policy-orientedo society at large
Test method
Test model: 4 steps
1. Reflection on mission orientation, self evaluation report
2. Empirical stage, find criteria and indicators for quality and relevance
3. Consultation with researchers and stakeholders, presentation of findings
4. Feed back and forward look
Step 2 : productive interactions
Identify social domains in context (policy, industry, society at large)
Analyse the 4 main interaction channels between research and context: texts, people, artifacts, money
Feed back to research community and stakeholders [workshops]
example of evaluation of societal quality – radar graph
0
0,5
1
1,51
2
3
4
5
6
789
10
11
12
13
14
15
example3
Science
Market
Societa
example REPP – table graph
Science, certified knowledge
relative citation impact --
productivity scientific publications+
+**(1
international visibility and collaborations =
representation in editorial boards ++
invited lectures ++
Industry, market
non-academic/commercial citing environment ++
productivity professional publications+
+*(1
involvement in industry/market -
advisory and expert roles in commercial domain --
editorships professional journal ++**
Policy, societal
involvement in policy domain +
memberships and expert roles in governmental bodies ++
memberships of societal organisations: advisory/ education ++*
production of public goods +
additional grants from policy +
example of evaluation of societal quality – radar graph
Crop and Grassland Science
% co-publ. Neth. (not KCW)% Neth. citing (not KCW)
% co-pub. Internat.
% Internat. citing
% cited journ. art.
2 cit. / journ. art.
2 journ. art. / fte total
1,5 member sci journal/ fte WP
orientation on science ( in %)
second fte (20 % of fte total)
% coop/fin proj ->res. grps (incl.KCW)
% mobility to research (incl. KCW)
diss. (cat 1)/ 4 fte AIO/OIOJunior, AIO,OIO students (1 Jun. / 2 Sen. -
1997))# diss (cat 1) / # junior staff% mobility to company
third budget fte (45% of fte total)orientation on professionals ( in %)
coop/fin proj innovating inst
1 patent / 3 WP
2 prof art/ fte total
1,5 member advisory board/WP
% coop/fin proj gov/spec. prog.
1,5 members gov. or spec. prog./WP
% mobility to gov./policy arena
involvement NGO's (scored in %)
% mobility to KCW
KCW coop/fin proj
KCW co-pubKCW citing
Research Embeddement & Performance Profile
Science & Certified Knowledge
Collaboration & Visibility
Public Policy
Innovation & Professional Education
& Training
KCW
2,1 memb / fte senior2,9 journ art / fte tot
0,54 f te junior / 1 fte senior
extra 15,25 dissertations (cat 1)
ERiC and the humanities
Express the relations of humanities research with society in a meaningful way
Help find criteria and indicators that might help to evaluate that
Bring together the expertise of both stakeholders and researchers
Stimulate them to find consensus about how to evaluate
What can humanities do?
don’t try to emulate other fields try and find consensus as a field about
indicators convince policy makers and managers
that you are capable of defining indicators that are robust and fit for your field
NetherlandsFrance, NL, EUNL, UKSpain, UK
HealthNanotechnologyICTSocial sciences and humanities
ERiCSIAMPI
www.ERiC-project.nl
focus in human sciences
1. Describe the content, (societal) objectives, organization and evaluation practices of the program/projects involved in the pilot.
2. Identify research strategies and policies aiming at measuring the scientific and social impact of the program/projects in the various relevant dimensions.
3. Discuss the findings with the stakeholders related to this case, evaluating the validity, reliability and relevance of the results.
4. Advise on further development of methods for the assessment of social impact. Analysis of strength, weaknesses and possible improvements for the method to be developed.
ERiC and SIAMPI and the Humanities
1. ERiC : cases in the Netherlands, but perhaps abroad
2. SIAMPI : dissemination of results through comments, conferences, joint workshops