Research Information and Analytics at CambridgeInsight over measurement

Research Information Office, Academic Division

Dr Owen RobersonResearch Information Analystowen.roberson@admin.cam.ac.uk@orobersonORCiD: 0000-0001-6938-5162

Introduction – my environment

Research Information

team

Research Office

Grant reporting(e.g. Researchfish)

HESA (return on statistics)

REF return

HRCRISGrantsDSpace

Academics

Administrators/Co-ordinators

University Library

Senior Academic Committees

Research Strategy Office

Research Information Landscape

• What is Research Information?

• Publications

• Grants

• Conferences

• Patents, software, compositions, performances, exhibitions…amongst others.

Not BIG DATA but WIDE DATA.

What it’s like being a researcher

I like doing research!

Are you doing anything interdisciplinary?

What were your recent publications?

Where did you publish them?

Is the data open too?

How did you pay the APC?

Where is your Research heading?

Are they Open Access?

Was it well received?

Did it have any impact?

How many grants have you applied for so far?

How many grant applications were successful?

How are you going to sustain your grant income profile?

Where is your grant money?

What did you do with the grant income you received?How did you cost your overheads?

Does it have an international dimension?

Did it have any Impact?

How many students are you teaching?How many students are you supervising?

Did it make a spinoff?

Does it have any IMPACT?

What’s your next project?

WILL IT HAVE ANY IMPACT?

So what makes a good analytics approach?

You can generate statistics on all of these things, but be cautious of the tail wagging the dog.

• What is your goal as an analytics team?

• Once you have an analysis, what can you actually do? (How many carrots and sticks do you have?)

• What are the implications of using the sticks?

What can we do, What should we do?

What Research Information is NOT, and the perils of metricide

• What is a KPI in the context of Research?

• How sure are you that you’re measuring what you think you’re measuring?

• Are you presenting it fairly?

• If a measure is biased, are there ways of creating measures that counterbalance the bias, and if not, are you asking the right question?

What Research Information is NOT, and the perils of metricide

• What is a KPI in the context of Research?

• How sure are you that you’re measuring what you think you’re measuring?

• Are you presenting it fairly?

• If a measure is biased, are there ways of creating measures that counterbalance the bias, and if not, are you asking the right question?

What is a KPI in the context of research?

There are none that work.

This is a game where the rules and goals change continuously.

•Donald Schon’s Reith Lecture: “The Loss of the Stable State”, as part of the series Change and Industrial Society, 1970. http://www.bbc.co.uk/programmes/p00h65rn#play

What Research Information is NOT, and the perils of metricide

• What is a KPI in the context of Research?

• How sure are you that you’re measuring what you think you’re measuring?

• Are you presenting it fairly?

• If a measure is biased, are there ways of creating measures that counterbalance the bias, and if not, are you asking the right question?

How sure are you that you’re measuring what you think you’re measuring?

• Impactful

• REFability

• EPSRC CompliancyHTTP://XKCD.COM/552/

What Research Information is NOT, and the perils of metricide

• What is a KPI in the context of Research?

• How sure are you that you’re measuring what you think you’re measuring?

• Are you presenting it fairly?

• If a measure is biased, are there ways of creating measures that counterbalance the bias, and if not, are you asking the right question?

What Research Information is NOT, and the perils of metricide

http://figshare.com/articles/What_aren_t_you_seeing_Advance_in_visualisation_reserach_/1464961

Once you’ve got a metric, are you presenting it fairly?

What Research Information is NOT, and the perils of metricide

• What is a KPI in the context of Research?

• How does calculating something inappropriate affect the conversation?

• How sure are you that you’re measuring what you think you’re measuring?

• Are you presenting it fairly?

• If a measure is biased, are there ways of creating measures that counterbalance the bias, and if not, are you asking the right question?

Scientocentricity, alienation, and bad incentives

HTTP://XKCD.COM/435/

Also…

NEVER USE JOURNAL IMPACT FACTORS

If you’re going to do it, do it well

Research Information must:•Provide insight•Be explained in its context•Be aware of its limitations

•Good places to start:1

• Have a statement about your position re DORA• Be aware that the watchmen are being watched

too..

____1. Thanks to Jane Tinkler, whose original presentation can be found here.

Making it work: enter once, use multiple times

HTTP://XKCD.COM/927/

Research Information Systems @ Cambridge

• With apologies to Harry Beck….

Core I: Symplectic meets institutional data

Core II: Further mapping of internal data to Symplectic

Core III: REF – Research Assessment

Post REF: Altmetrics and ORCID

Post REF: Reporting tools, CV & biosketch generation

Closing the gap: Open Access

Bridging the gap: Dimensions & rich grant meta data

From Integration to Innovation – VIVO and network analysis

And what all this means…

Social Media Impact and Comms Strategy

Publication Strategy

HEFCE ComplianceGrant Capture, RCUK

Compliance

Academic Workload

Genuinely new and useful information

researchers want to hear

Network Analysis: finding connections

This shows collaborations between REF2014 UoAs; indicating that interdisciplinary research is already a key

element of research practice at Cambridge.

It is possible to explore the connections between members of a defined group and consider not only the direct connections

between them (through collaboration or publication), but who their collaborators collaborate with, and what other connections

they may have made.

Colours indicate University departments. Points or nodes represent researchers. Connections or edges

represent an instance of co-authorship.

Roberson, Owen; Wastl, Juergen (2015): Beyond Serendipity: Networks of Research at Cambridge. figshare.http://dx.doi.org/10.6084/m9.figshare.1431829

Glossary

Altmetric.com – Provider of article-level metrics focussing mainly on social media platforms, reliant on DOIs in the main, to track articles through internet sources. Also covers Wikipedia entries and some policy documents. Digital Science portfolio.

CASRAI – Not-for-profit seeking to reduce the administrative burden on academics, by supporting single entry, multiple use of data.

Dimensions – Provider of publically available grants data, including abstracts and semantic searches. Medical and US/UK focussed at the moment, but growing constantly. Digital Science portfolio.

DOI – Digital Object Identifier – unique string used to identify an online object. Operates like a URL, and is public. Mainly applied to articles in the research space, use will probably grow in the near future.

DORA – San Francisco Declaration on Research Assessment: an open declaration from researchers and institutions that commits its signatories to the improvement of research assessment metrics and the avoidance of Journal Level Metrics.

ORCiD – a unique identifier for a researcher that operates like a DOI for an individual.

ResearchFish – the Company and software used by a number of funders, but crucially by RCUK, to track research outcomes for the grants its participants have funded. Mandatory annual returns. Database uses the grant as its principal component.

RIMS – Research Information Management System – a database containing metadata about publications. Key offerings in the field are Symplectic Elements, PURE and ePrints, as well as a host of institution-specific, home-grown solutions. Most solutions use the academic as the principal component.

Symplectic – Company that produces the Elements system, the RIMS that we use at Cambridge. Digital Science portfolio.

VIVO – Open Source ontology of researchers and their research outputs originally developed at Cornell. Can be driven by a RIMS, used by Melbourne for “Find the Expert”.

Further Reading and Links:

https://responsiblemetrics.org/about/

http://orcid.org/

http://www.vivoweb.org/

http://casrai.org/

http://figshare.com/articles/What_aren_t_you_seeing_Advance_in_visualisation_reserach_/1464961

http://www.research-information.admin.cam.ac.uk/

http://figshare.com/articles/Beyond_Serendipity_Networks_of_Research_at_Cambridge/1431829

Donald Schon’s Reith Lecture: “The Loss of the Stable State”, as part of the series Change and Industrial Society, 1970. http://www.bbc.co.uk/programmes/p00h65rn#play