how to measure the impact of research ?
TRANSCRIPT
How to measure the impact of research?
Anne-Catherine Rota, Research Intelligence, Elsevier
7 December, 2015
Elsevier, Partner for Research
Access to full text of 2,500 journals and 11,000 eBooks revues
The largest bibliographic database , 22,000 active journals
| 3 | 3 | 3
Analyze research: what for?
Monitor the trends in my own research area
Position myself / my team / my institution
Evaluate
Get more funding
Being more visible
| 4 | 4 | 4
The French research landscape: EU funding
H2020: Funding allocated per country – 4,320 contracts signed - July 15, 2015
FP7: Funding allocated per country (2007-2013)
Contribution to the EU budget (2011)
| 5 | 5 | 5
Scopus, the output database
22 000 journals from 5,000 publishers
o A well-balanced coverage in terms of research fields
o A well-balanced coverage for locations
o IndependantContent Selection Adivsory Board
o 56 million articles
o 6,9 million conference papers
o + 90 000 eBooks
o “Articles in Press” from 5,000 titles
o +3,473 Open Access journals
Elsevier journals:
only 10% of
indexed journals
in Scopus
88%
12% Scopus
Non-Scopus
Top 100 institutions
| 6 | 6 | 6
Among Scopus users
2015 Best Chinese University Ranking report (évaluation par le Shanghai Ranking Consultancy’s)
International rankings
Public policies, research assessments and funding agencies
Canadian Institutes of Health Research
U.S. National Science Foundation (NSF) Selects Elsevier Data to Support its Science and Engineering Indicators http://www.elsevier.com/about/press-releases/science-and-technology/u.s.-national-science-foundation-nsf-selects -elsevier-data-to-support-its-science-and-engineering-indicators
Research metrics: best practice in their use in merit systems
7
Research metrics can be used inappropriately
- One metric used in isolation often leads to undesirable outcomes
Research metrics can be manipulated
- Normal research practice can be abused if you really try
Also true of peer review - How can research metrics be used with
similar trust and benefit?
All merit systems should be based on multiple types of measurement: • Peer review
• Expert opinion
• Narrative
• Research Metrics
Research Metrics input should always rely on at least 2 metrics
- Reduces opportunities to game and can drive desirable outcomes
Research metrics indicate many different ways of being good
8
Research Metrics input should always rely on at least 2 metrics that ideally reflect distinct ways of performing excellently
Within Research Area of “Sitting”: Output, citations and views show different ways of being excellent
Most
productive
Most cited
Most viewed
Viewing activity is complementary to being cited
Recent publication viewed, even though not cited yet
Erratum not (expected to be) cited but actively looked at
This publication is of global interest
Snowball Metrics in a nutshell
12
• Bottom-up initiative: universities endorse metrics valuable to
them to generate a strategic dashboard
• Aim: enable benchmarking by driving quality and efficiency
across higher education’s research and enterprise activities
• Draw on all data available: university, commercial, public
• Build on existing definitions and standards where sensible
• Output: free recipes (methods) that can be used by anyone
• www.snowballmetrics.com for more info and Recipe Book
UK group
Societal-Economic Impact
13
Recipes: partner with businesses
Contract Research
Spin-Off-Related Finances IP Volume / Licenses
Academic-Corporate Collaboration
Altmetrics: esteem and impact
15
Scholarly Activity: posts in
scholarly tools
Social Activity: social
media posts
Scholarly Commentary:
comments in scholarly tools
Mass Media: references
from newspapers etc.
Altmetrics - Esteem Altmetrics - Impact
Altmetrics in Scopus
16
| 17 | | 17
Societal Economic Impact in SciVal - Patent Article Citations
| 18 | | 18
Societal Economic Impact in Scival - Patent Article Citations
Societal-economic impact
19
Partner with businesses
• Academic-Corporate Collaboration
• Business Consultancy Activities
• Volume & financial benefits of IP &
spin-outs
Esteem
• Scholarly Activity
• Scholarly Commentary
Impact
• Social Activity
• Mass Media
• Public Engagement
Brain and Neuroscience research is highly collaborative
20
Inte
rna
tio
na
l c
oll
ab
ora
tio
n (
% o
f to
tal o
utp
ut)
Brain and Neuroscience Researchers: cross-disciplinary mobility
| 22
Measure interdisciplinarity
| 23
Four ingredients to be gatherer in order to analyze research
1. Data
3. Metrics
4. Tools
2. IT system
Quality & reliability
Transparency
Granularity
as hoc services & APIs
Immediate access
| 24
Documents under initial ENPC profile: 2,007
Scopus optimized identification of French affiliations
| 25 | 25 | 25
| 26 | 26 | 26
• Offer a basket of metrics
• That can be generated in an automated and scalable way, so they are available for all peers
• Do not make decisions on behalf of users
• Provide options and enable users to decide approach
• Absolute transparency so you can make your own judgments about whether something is appropriate, or has been manipulated
• No methodological black boxes – no exceptions
• Transparency about underlying data contents and structure
• Constructed in a reasoned way
Research metrics: best practice in their supply
27
Absolute transparency from your metrics supplier is essential
28
http://bit.ly/scivalmetricsguidebook
http://bit.ly/usageguidebook
http://bit.ly/hefceresponse
Presented By
Date
Thank you for your time and attention