November 2005 Anne AssersonResearch Administration Department,UiB
How can you assess your organisation through a
CRIS?
The Norwegian national CRIS, FRIDA, and evaluation by means of research
output
November 2005 Anne AssersonResearch Administration Department,UiB
Structure of Presentation
• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion
November 2005 Anne AssersonResearch Administration Department,UiB
Aim of Ministry for Assessment
Model for assessment based on indicators to Stimulate research and Allocate resources
• The best researchers (by allocating money based on publications)
• The best ideas (allocating money to the competition-based financing of projects)
• Allocate money to special thematic areas
November 2005 Anne AssersonResearch Administration Department,UiB
New efforts: to use result indicators for academic/scientific publishing as a basis to budget the research at Universities and State Colleges. The Publications from 2004 will be included in the budgets for 2006
Main issues: 1. Provide better data on Academic Publishing2. Take in consideration different practices – thus avoiding
unfortunate effect on the publication pattern.3. Divide in Quality levels to avoid reduction in quality of the
publications pattern and avoid volume of low publications4. Develop indicators for activities within dissemination
St.prp.nr.1, UFD, pg. 115: ”Financing of Research” 2003
November 2005 Anne AssersonResearch Administration Department,UiB
• Ministry of Education and Research wish to improve research results in the universities
• Measure through indicators that say something of quality of results
• Results will trigger resource allocation to the universities
• 14% of total budget allocated based on research output indicators
November 2005 Anne AssersonResearch Administration Department,UiB
• Result Indicators that have been chosen – Doctoral theses– EU funding– National Research Council Funding– Publications
• These are quality indicators; all subject to peer review
November 2005 Anne AssersonResearch Administration Department,UiB
• Institution receives an allocation from the ministry
• The model is NOT recommended for use internally
• The ministry stresses the need for creating internal models to reward / stimulate
• Necessary to think complementarily in order to support particular subject areas
November 2005 Anne AssersonResearch Administration Department,UiB
Structure of Presentation
• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion
November 2005 Anne AssersonResearch Administration Department,UiB
Distribution of published research by academic field and type of publication. Percentage of overall scientific production, University
of Bergen 1997-1998*
0,0
10,0
20,0
30,0
40,0
50,0
60,0
70,0
Biology Biomedicine Medicine Chemistry Physics Earthsciences
Mathematicsand
informatics
Socialsciences
Humanities
%
Int. jour. art. Scand./Norw. jour. art. Proceedings
*) Classification by institute assignments. Unpublished works, abstracts and posters are not included in the countings
Source: FORSKDOK, University of Bergen
November 2005 Anne AssersonResearch Administration Department,UiB
Definition of an academic Publication
The Publication must:• Present new insight/knowledge• Be in a form that make it possible to validate or
useful in new research • Be in a language and have a distribution that
can make the publication available for the researchers of whom it may be of interest
• Be in a publication channel ( journal, series, publisher, website) with routines for Peer review.
November 2005 Anne AssersonResearch Administration Department,UiB
• Publications– Peer reviewed articles– Academic monographs– Chapters / parts of anthologies– (Proceedings)
• Only a few classes of publications used in assessment
November 2005 Anne AssersonResearch Administration Department,UiB
Structure of Presentation
• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion
November 2005 Anne AssersonResearch Administration Department,UiB
Functionality Required
• Must be able to express :– Person belonging to one or more
organisational unit– The output produced at one or more
organisational units– from one year or a temporal interval– From one or several projects
• And these in any combination
November 2005 Anne AssersonResearch Administration Department,UiB
Functionality Required 2
– Person must be linked to publication, organisational unit and project
– Reference format
• Vancouver, APA etc
– Person must be linkable to two or more organisational units
November 2005 Anne AssersonResearch Administration Department,UiB
Structure of Presentation
• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion
November 2005 Anne AssersonResearch Administration Department,UiB
Structure Required
• relational model as basis with dynamic role-based temporal links between base entities
• Extended with elements of publication– Journal name, ISSN, shortname, acronym,
pagefrom, pageto, volume, issue
November 2005 Anne AssersonResearch Administration Department,UiB
PROJECT
ORGUNITPERSON
Skills
CV
GeneralFacility
ParticularEquipment
Contact
ResultsPublication
ResultsPatent
ResultsProduct
Service
FundingProgramme
Event
Classification
The Whole Thing
Prize/Award
November 2005 Anne AssersonResearch Administration Department,UiB
PROJECT
ORGUNITPERSON
Result_Publication
Can Express: (where DT-date/time)Person A (DT1 - DT2) (is author of) Publication XOrgunit O (DT1 - DT2) (is owner of IPR in) Publication XPerson A (DT1 - DT2) (is employee of ) Orgunit OPerson A (DT1 - DT2) (is project leader of) Project PPerson A (DT1-DT2) (is member of) Orgunit MPerson A (DT1-DT2) (is member of) Orgunit NOrgunit M (DT1-DT2) (is part of) Orgunit OOrgunit N (DT1-DT2) (is part of) Orgunit O
Secondary Base Entities: example: RESULT_PUBLICATION
November 2005 Anne AssersonResearch Administration Department,UiB
• National system for the four universities
• Funded and owned by the universities
• Central database
• Based on CERIF (with extensions)
• 300,000 records in 2006
• In production use for 1 year
November 2005 Anne AssersonResearch Administration Department,UiB
Structure of Presentation
• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion
November 2005 Anne AssersonResearch Administration Department,UiB
A publications is divided in author and institutional parts within co-authoring
Publication
Author 1 Author 2 Author 3
Institution 1 Institution 1 Institution 2 Institution 3
November 2005 Anne AssersonResearch Administration Department,UiB
Channel
Form
Eexample : scientific article in Nature or scientific monografi at Aschehoug forlag
We find Academic Publications in the intersection between Publications Channel and Publication
Form.
November 2005 Anne AssersonResearch Administration Department,UiB
Level 1 and Level 2
• At level 2 there are specified certain publication channels within all subjects.Level 2 are expressing the desirable direction for the publication pattern.
• Level 1 is the ”normal” and includes all publishing according to the definition.
November 2005 Anne AssersonResearch Administration Department,UiB
Level 2: ca. 1/5 of the publications
Level 1: ca. 4/5 of the publications
November 2005 Anne AssersonResearch Administration Department,UiB
Guidelines for nomination at level 2
In general the publication channel must be
• Considered as leading within broader scientific community
• Publish the most important publications for researchers from different countries
November 2005 Anne AssersonResearch Administration Department,UiB
Structure of Presentation
• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion
November 2005 Anne AssersonResearch Administration Department,UiB
Quality
• Import of ISI records– Problem: person and organisational unit not
connected
• Routines and responsibility for quality assurance– Researchers– Organisational unit
• Researcher as end-user validates records
November 2005 Anne AssersonResearch Administration Department,UiB
Bibliographical sources
• ISI-indexes will cover 62 % of all journal articles.
• Norart can cover further 32% of the articles by adjusting the application and by Nordic cooperation.
• Bibsys could possible cover 70-80% of book publishing
NORART: Norske tidsskriftartikler
November 2005 Anne AssersonResearch Administration Department,UiB
November 2005 Anne AssersonResearch Administration Department,UiB
Structure of Presentation
• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion
November 2005 Anne AssersonResearch Administration Department,UiB
• Quality– Using selected publication channels, not only
publication volume, gives quality
• Requirements of System– Cannot be done by a library system– Requires a flexible structured CRIS