Download - ROI and Beyond - King
The library’s (potential) role in creating research metrics for their
organization
June 15, 2011
Special Libraries Association Conference
Philadelphia, PA
James King, NIH Library
Office of Research Services
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
http://echosphere.net/star_trek_insp/star_trek_insp.html
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
What does this mean?
• Snap out of it!• We live in a time when:
• Change is constant• Flexibility matters• Attitude is everything
• We need to be:• Positive• Interesting• Inventive• Persistent
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
http://collections.nlm.nih.gov/
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
http://www.morganstanley.com/views/perspectives/
tablets_demand.pdf
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
http://techcrunch.com/2010/01/05/quantcast-mobile-web-apple-android/
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
http://becker.wustl.edu/impact/assessment/index.html
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
Trends in scholarly research Competition for government research funding increasing
Available funding decreasing
Competition for top research faculty is on the rise
Accountability:– Research spending – Demonstrating return on investment (ROI)
Proving the institution’s quality of research to: – Prospective students – Prospective faculty members/research staff– Investors/donors
Result: Institutions seek objective data on research performance, for data-based decision making
Thanks to Thomson Reuters for this and the next couple slides
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
Different stakeholders in research evaluation
Government agencies/funding organizations /accreditors/state government
External Entities
IndividualsFaculty,
staff
Institutional Depts/DivisionsDeans; Department Heads
Institutional assessment, academic affairs
Institutional Management
Management, including committees, provost, vice
provosts, rectors, chancellors
Library
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
Relative Impact/Bench-marking
Journal actual/expected
1.14
Category actual/expected
1.48
Mean percentile 33.91
% papers in top 10% of their field
19%
% papers in top 1% of their field
5%
Special-ization
Disciplinarity index 0.97
Productivity # papers 149
Total influence
# citations 1,136
Self-citing papers
57
H-index 22
Efficiency Avg. citation rate
7.62
Percent of papers cited
43
What is a good citation count?
15
Journal and category performance ratios compare this researcher’s citation counts to the norms or “expected” rates in his discipline. Boonen is performing above average on both the journal and category levels (1.14 ,1.48).
Mean percentile reflects the percentile performance of the researcher’s work, on average. e.g. in top 33.91 percent
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
Capturing corporate knowledge
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
People
Opportunities
AwardedGrants
Publications ANALYTICS
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
Performance &
Evaluation
Performance &
Evaluation
Demonstration of
Capabilities
Demonstration of
Capabilities Research EnhancementResearch Enhancement
Institution/CountryInstitution/Country Researcher/GroupResearcher/Group InstitutionInstitution ResearcherResearcher
“Showcas
e”
Q4 2011
“Showcas
e”
Q4 2011
SciVerse
Scopus
SciVerse
Scopus
Office of Research
DeanDept. Head
ResearchersStudents
ResearchersResearchersAdministrato
rs
Researchers
What are my interdisciplinar
y strengths?
Who are the high
performers in my strategic priorities?
How can I act upon my
competitive advantage?
How are my researchers performing
relative to their peers?
I want to evaluate
researchers with
the criteria that
are important to me.
I want to demonstrate
my institution’s research
expertise in the areas that are
of strategic importance to
us.
I want to find researchers
within my institution with
whom I can collaborate.
I want to understand ‘who knows what’ in my institution.
I want to find funding
opportunities
I need to determine the right funding
strategy
I want to increase
success rates
What are the most highly-cited articles
and authors within
a research area?
What is the h-index of an
author?
Overview of all of an author’s or institution’s publications
Overview of SciVal Solutions
Aggregation
Primary User
Customer
Challenges
The Solution
Addresses
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
Who can use VIVO?Who can use VIVO?
…and many more!
http://www.vivoweb.orghttp://www.vivoweb.org
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
Electronic Scientific Portfolio Assistant
Multi-Dimensional Science Management Tool
eSPA supports the NIH mission by providing a comprehensive solution for program planning and evaluation staff and NIH
leadership to search, build, and collaboratively review and analyze portfolios of NIH
research projects.
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
Information Wants to Be Free
“On the one hand information wants to be expensive, because it's so valuable. The right information in the right place just changes your life. On the other hand, information wants to be free, because the cost of getting it out is getting lower and lower all the time. So you have these
two fighting against each other.”
Stewart Brand
First Hackers' Conference in 1984
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION
Types of citation metrics and what they measure
24
Metric Calculation Evaluator questions
Productivity -# papers-share of papers in field
-# papers-# papers in field/ papers in field
What is the research output of X? (a country, institution, researcher, etc.)
Total influence
# citations # citations What is the overall impact of a body of work?
H-index Number of papers (N) with at least N citations each.
What is the impact and productivity of a body of work?
Efficiency
Average citation rate Total citations/Total papers
What is the rate at which a body of work is cited?
Percent of papers cited
# papers with at least one citation/ Total # papers in population
How many papers get cited? Never get cited?
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION25
Metric Calculation Evaluator questions
Relative Impact/Benchmarking
Journal performance ratio (actual/expected)
Sum of citations/sum of journal expected citation rates
Expected citation rate calculated for a journal, for each year and document type combination (e.g. JAMA, 2001, review)
Has this body of work performed better than average vis-à-vis the journals represented?
Category performance ratio(actual/expected)
Sum of citations/sum of category expected citation rates
Expected citation rate calculated for a journal category, for each year and document type combination (e.g. Physics, 1995, article)
Has this body of work performed better than average vis-à-vis the specific disciplines represented?
Types of citation metrics and what they measure
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION26
Metric Calculation Evaluator questions
Relative Impact/Benchmarking
Aggregate Performance Indicator
Impact of an institution or country relative to an expected citation rate for the entity. The indicator is normalized for field differences in citation rates as well as size differencesamong entities and time periods.
Where ci is the total citations for institution (or country) i, Pift is the number of publications from institution i in field f and year t, cft/pft is the average citation rate of papers in field f and year t. Sums are over all years and all fields.
What is the overall impact of my institution’s or country’s research amongst the fields we publish in?
Types of citation metrics and what they measure
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION27
Metric Calculation Evaluator questions
Relative Impact/Benchmarking
% papers in top x% of their field
e.g. 10% of Dr. Lopez’ papers are in the top 1% of their fields
What proportion of a body of work achieves a specific level of performance?
Percentile in category and mean percentile
Percentile placement of article within a journal category (e.g. oncology, 2002)
How has this body of work performed compared to the disciplines represented?
Types of citation metrics and what they measure
THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION28
Metric Calculation Evaluator questions
Specialization Disciplinarity index
where s is the share of papers in category i and n is the number of categories
How multidisciplinary is a body of work?
Interdisciplinarity index
where p is the share of papers in category i and n is the number of categories
How dispersed is a body of work across disciplines?
Emerging areas of research
Research Fronts Clusters of highly cited papers identified via co-citation analysis
What are the emerging areas of research in chemistry?
Indirect impact Second generation citations
Citations received by a paper’s citing papers
Did the papers citing a body of work go on to have impact?
Types of citation metrics and what they measure