niamh brennan (trinity college dublin) – cerify

http://cerify.ukoln.ac.uk

Mahendra Mahey (UKOLN)Stephanie Taylor (UKOLN)Niamh Brennan (TCD)Kevin Kiely (TCD)

Eva

luat

ion

of

CE

RIF

an

d C

RIS

Imp

ort

an

d E

xpo

rt o

f D

ata

and

Fa

cili

tati

on

of

dat

a ex

chan

ge

Insti

tutio

ns s

ee v

alue

of C

ERIF

and

CRI

S, T

ell t

heir

regi

onal

insti

tutio

ns a

t reg

iona

l an

d na

tiona

l eve

nts.

CERI

F an

d CR

IS E

ngag

emen

t Inc

reas

es

Incr

easi

ng le

vel o

f Eng

agem

ent w

ith a

CRI

S /

CERI

F

Research Information ManagementProcess Mapping and Gap Analysis

Current and past CRIS/CERIF/RIM projects

InCites

Research in View

TCD CRIS TCD CRIS

Thomson CRIS

Research in View

and InCites

Thomson CRIS

Research in View

and InCites

Other, e.g.

PURE?

Other, e.g.

PURE?

Data Surgery

CERIFTCDXML2002

CERIFTCDXML2002

CERIFTCDXML2008

CERIFTCDXML2008Mappings

and Cross Walks

relevant CERIF Schema

Mappings and Cross

Walks relevant

CERIF Schema

PriorityProcess Areas –

‘as is’ and ‘to be’

PriorityProcess Areas –

‘as is’ and ‘to be’

CERIF HealthCheck

CERIF HealthCheck

CERIFy ApproachPeople Centered Socio Cognitive

Reducing duplication

TCD – CERIF People and Publications

User requirements elicitationPrioritization of RIM Processes / Identification,

Mapping, Gap and Stakeholder Analysis

Questionnaires Site Visits

Business Process Analysis‘as is’ and ‘to be’

4 Priority Processes

Data from at least 2

partners

Data from at least 2

partners


Mahendra MaheyStephanie TaylorRosemary RussellMichael DayTalat Chaudhri

Niamh BrennanKevin KielyJimmy TangRoisin Croker

InCites Exchange of Data


How Is This Used?


• RAE requirement• Comparison with other universities (e.g.

Russell Group)• Global comparisons (updated annually)• Citation information generated from this tool

provides a lot of key information

Collection of Data


• Two-way process between Thomson Reuters & individual institution

• Requires a lot of work by institution• Institutional data collected in local database of staff

research information

User Issues


• “There is a lot of effort involved in understanding the data - there would be much value in standardising the data.”

• E.g. - problems with author ID - duplicates & null fields

User Requirements


• “If the data was in a standardised format it would make life much easier!”

• Nightly updates• Internationalised data sets• Relational database with mapping

research output onto staff ID number for returned data

• Better mapping to institutional schools structure

Measures of Esteem


How Is This Used?


• RAE requirement• Personal review for promotion process• Assessment & benchmarking - internal & external• As key information in the assessment in judging the

overall quality of research outputs• Drafting REF documentation• Part of institution-wide annual planning cycle

Collection of Data


• From individual researchers on adhoc basis• Systematic for RAE• Continuous tracking for REF• Ask individual schools to provide this on regular basis

User Issues


• Process is ‘woolly’ - difficult to provide meaningful data• Perception varies depending on given point in a career path• Citations alone are not accurate enough• Context is needed to make sense of raw data• Defining this process is very difficult• Researchers can undersell themselves and don’t make

information available

User Requirements


• Systematic way of capturing this is required• Bringing a bibliometrics perspective to esteem-related

information • Take into account that one measurement doesn’t fit all• Acknowledge differences in different subjects• When one size doesn’t fit all, find where to plug the gaps in

information • Personalised audit tool to be built into RIM system

Thomson Reuters data exchange – what’s InCites?

Why CERIFy it?• The community expressed interest in this process• Current data is unsatisfactory ; the dataset is a specific view

– Institutional Citation Report is ‘historic’: based on affiliation at the time of authorship. – Does not show Schools, Research Centres etc. (WoS API will not provide this either).– Author, institution name variants (no automatic DAI in WoS/InCites)– Missing papers (aside from general coverage issues)– But.. institutions want the metrics + institutional structure in InCites + to embed data in local CRIS/IR:

Sample

• Current exchange process is unsatisfactory: non-standard schema, heavy demands on institution, disappointing results

• Opportunity to create a CERIF data model for exchange of people, institutional affiliations, publication data & metrics

• Opportunity to demonstrate a 360⁰ data exchange using CERIF• If successful, it should help to show value and help to engage institutions with CERIF• Model can be re-used for other similar exchange processes

Process 1: InCites ‘as is’ – Based on Queen’s University Belfast

Process 1: InCites ‘to be’ – Generic (based on CERIFy Data Surgery)

Thomson Reuters

MIS/CRIS developers

Research Office / Library

DATA EXCHANGE PARTNERS

Business Intelligence Officers/Institutional Analysts

1. Online export of CERIF-based data to TR from RIM/CRIS via web services (nightly updates)

5. Reporting on Research 8.1 Internal Reporting 8.2 External Reporting

[6] 5 TR send report to institution on results of matching process (including matched Author ID)

3 Institutional Data automatically matched with TR records for authors & publications

2. TR import and update deltas

4. InCites 'Institutional Profile' established and updated automatically

[7] TR data integrated with CRIS data (if required) Note: may use TR API instead or alongside

EXTE

RNAL

EXTE

RNAL

Data drilldown

Creation of CERIF Data Model for ‘InCites’ Exchange

Over to Kevin…

Where we are – results• CERIF Data model• Data exchange next week!• Includes: extended ‘publication types’ list• Multiple identifiers (Researcher ID; Institutional ID, HESA ID,

Publication ID etc• Full metrics = standardisation

Next stepsQUB

Thomson Reuters are working on automating the process using Web Services…

Notes on Esteem & REF

“‘Esteem’ is no longer included as a distinct element in the assessment” 67. All information provided by HEIs in submissions to the 2014 REF must be capable of verification [note: can CERIF help

provide the reference or source of data?] Under ‘Definition of Impact for the REF’: “a. Impacts on research or the advancement of academic knowledge within the higher education sector (whether in the UK

or internationally) are excluded. (The submitted unit’s contribution to academic research and knowledge is assessed within the ‘outputs’ and ‘environment’ elements of REF.)”

Part 3 Section 5: Environment template (REF5) 183. Information is required about the research environment for each submitting unit relating to the period 1 January 2008

to 31 July 2013. Each submission must include a single completed REF5 form, consisting of the following sections: • Overview. • Research strategy. • People, including: – Staffing strategy and staff development. – Research students. • Income, infrastructure and facilities. • Collaboration and contribution to the discipline.

From: Assessment framework and guidance on submissions, HEFCE, July 2011 http://www.hefce.ac.uk/research/ref/pubs/2011/02_11/02_11.pdf

REF5 ‘Environment’:•Maximum score reduced from 30%(RAE 2008) to 15% (REF 2014)•Indicators of Esteem = Collaboration & Contribution to the Discipline

http://www.hefce.ac.uk/research/ref/pubs/2011/02_11/02_11.pdf

Research Monitoring Team

HoD/DDR

DATA EXCHANGE PARTNERS

Deans

Academic

2. Details of form transferred to RAD

6. Access electronic copy deposited within RAD SharePoint site Departmental folder

3. Pdf report produced

4. Hardcopies printed for Research Monitoring meetings

1 Completes ‘Other Activities’ annual form on Research Activity Database (RAD) SharePoint interface

5. Electronic copy deposited within RAD SharePoint site Departmental document folder

7. Hardcopies printed for Research Monitoring meetings

Process 1: Esteem ‘as is’ – Based on Aberystwyth University

Research Monitoring Team

HoD/DDR

Deans

4. Reports generated by HoDs, Deans etc.

3. CRIS / RIM system data is queried and reported by Research Monitoring Team

6. Data Exported to external agencies as required

Academic

2. Academic updates CRIS/RIM with other elements of Measures of Esteem

MIS/Library

1. CRIS/RIMacademic profile is automatically populated with elements of 'Measures of Esteem'

External agencies/Funders

5. Reports generated by HoDs, Deans etc.

Process 1: Esteem ‘to be’ – CERIFy Data Surgery

Indicators of Esteem:based on CERIFy DataSurgery, current practiceIn partner institutions + a number of external sources

Indicators of Esteem - the extended version:based on original CERIFy IE, With additions to publications & outputs

Please see spreadsheet for latest list CERIFy Indicators ofEsteem

OrgUnitProjectPersonPubl.Etc.

Link tables

cfOrgUnitId etc.cfEsteemIndicatorId

cfClassIDcfClassSchemeId

cfStartDatecfEndDate

Link tables




Link tables




Link tables




OrgUnitProjectPersonPubl.Etc.

Esteem Indicatorlanguage table

Link table

cfEsteemMeasureId cfEsteemIndicatorId


Esteem Measure table

cfEsteemMeasureId cfEsteemMeasureKind

cfCountIntegercfValueFloatingPoint

cfValueJudgemental NumericcfCountIntegerChange

cfValueFloatingPointChangecfValueJudgementalChange

cfValueJudgementalText

Esteem Indicator table

CERIFy Esteem Indicators Model – based on MICE (Impact) Note: current focus for Esteem is more on Indicators rather than on Measures

NB: this adaptation has yet to be finally validated by Brigitte

Esteem Indicators mapped to UK RAE 2008, Australian ERA

Esteem Indicators for REF5: Contribution to Discipline/Collaborations

The Issue with Esteem Indicators [as with Impact]• The processes associated with the capture and reporting of Esteem Indicators

can benefit more if there is an effort to standardise at least some of the data required.

• Most of the data is not currently available from sources other than narratives or reports supplied by members of academic staff

• To reduce time and effort & assist with verification:- Where data can be imported from external sources it should.- ‘Authority-controlled’, taxonomies and definitions lists should be agreed and

built into the systemNOTE: the RAE 2008 submissions under ‘Research Environment, Measures of Esteem’ provide an extremely valuable source of information on the type of information provided by institutions in this area. This body of knowledge is not available for other REF-relevant areas such as ‘Impact’. The RAE 2008 ‘Measures of Esteem’ is a particularly rich source of data from the disciplinary viewpoint. The analysis of these ‘texts’ has great potential to assist with developing the type of ‘authority lists’/taxonomies described above. The following slides show the results from analysing the RAE 2008 University of Bath ‘Measures of Esteem’ submission in the field of Pharmacy. This text was analysed using Many Eyes and Open Calais. The results of using Open Calais for semantic analysis and entity extraction even for this small sample are very positive.

Acknowledgements• The CERIFY Team & Partners• Brigitte Jorge, EuroCRIS• Thomson Reuters InCites CERIFy team, Philadelphia• Bo Alroe, Atira /UK Pure User Group (Esteem Indicators)• http://cerify.ukoln.ac.uk

http://cerify.ukoln.ac.uk/

niamh brennan (trinity college dublin) – cerify

Technology