A month in the life of a university bibliometrician

Dr Ian RowlandsUniversity of Leicester, UK

THE University of Leicestermy rolea typical month

Part 1: CONTEXT

University of LeicesterCollegesArts, Humanities & LawEngineering & ScienceMedicine & Life SciencesSocial Sciences

People21,800 students (7,700 distance)1,300 academic staff

Research incomeResearch grants and contracts£ 55 million (2013)REF-related QR income£ 24 million (2013)Famous forDNA fingerprintingSpace scienceRichard III

David Wilson Library

Library Research Services

Research analytics

Research data management

Researcher training and development

Institutional repository

Gold open access funds

Interface with University systems

Briefings on information issues

PhD training (library tools)

Project support

Bibliometrics support

Internal consultancy

Management reports

My diary March 2014

Staff training and development•Briefing on journal impact factors•Briefing on researcher identifiers•Live webinar on h-index•Develop `citation tips’ leaflet

Reports•Background report on QS World University Rankings for Vice-Chancellor• Analysis of collaboration between universities in the English Midlands• Internal review of Department of Chemistry

Major enquiries•Can we use citation data to help thin out the print journal collection?•How can citation data be used in a systematic health review?

Quick enquiries•Handling Spanish surnames•Finding someone’s h-index

Routine maintenance•Quarterly update of InCites address database

Citations 66.9

Citations 56.3

RESEARCH EXCELLENCE FRAMEWORKUSE OF CITATION DATA IN REF2014

Part 2: REF2014

REF2014

REF2014 is a UK-wide framework for assessing research in all disciplines. Its purpose is:

• to inform research funding allocations (approximately £2 billion per year)

• provide accountability for public funding of research and demonstrate its benefits

• to provide benchmarks

Assessment framework

65% 20% 15%

Assessment criteria

The criteria for assessing the quality of outputs are originality, significance and rigour*

Four star Quality that is world-leading in terms of originality, significance and rigour

Three starQuality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence

Two star Quality that is recognised internationally in terms of originality, significance and rigour

One star Quality that is recognised nationally in terms of originality, significance and rigour

UnclassifiedQuality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of this assessment

Highest possible grade

… or highest possible volume?

Assessment criteria

The criteria for assessing the quality of outputs are originality, significance and rigour*

Four star Quality that is world-leading in terms of originality, significance and rigour

Three starQuality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence

Two star Quality that is recognised internationally in terms of originality, significance and rigour

One star Quality that is recognised nationally in terms of originality, significance and rigour

UnclassifiedQuality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of this assessment

NOT FUNDED

NOT FUNDED

NOT FUNDED

REF2014 and use of citation data

• The following subpanels will make use of citation data:

- Main Panel A: Subpanels 1-6 [life sciences]

- Main Panel B: Subpanels 7-11 [physical sciences]

- Main Panel C: Subpanel 18 [economics]

• REF2014 will use only use Scopus citation data

• None of the sub-panels will use journal impact factors, journal rankings, or other forms of bibliometric analysis

How we used citation data

• Each paper was graded by two assessors, at least one external to the University

• We counted Scopus citations and used published REF2014 calibration tables to locate papers in the top 1%, 5%, 10% or 25% of world impact

• We repeated the exercise using Web of Science InCites, where we have continuous citation percentiles and other metrics such as journal impact factors

How we used citation data

• Citation data showed a very good fit with human judgments, so were a helpful source of additional information

• The data were particularly useful in guiding decisions at the critical 3*/2* boundary

• The data helped us to model the whole submission and (hopefully) optimise the trade-off between grade and volume

?