bibliometric indicators, research evaluation and funding
TRANSCRIPT
Bibliometric indicators, research evaluation
and funding parameters
HEFCE Consultation Events January 2008
Henk F. Moed
Centre for Science and Technology Studies (CWTS) Leiden University, the Netherlands
Contents
1. Short introduction to citation analysis
3. Earlier studies and their outcomes
5. Effects of performance assessments upon performers’ behavior
7. Metrics, research evaluation and funding parameters
Citation Analysis in Research Evaluation
Springer, 2005, 350 pp.
Henk F. Moed
CWTS, Leiden University, the Netherlands
1.
A concise introduction to citation analysis
Topics
• Differences in citation practices among fields
• Skewness of citation distributions
• Field-normalised citation impact indicators
• Coverage of the Web of Science
• Types of bibliometric studies
• What do citations measure
Scope: Science vs. Technology
Contribution of technologies to technological progress
The influence of technology upon scientific development
Technology
The science base of technology
Contribution of science groups to scientific progress (this lecture)
Science
TechnologyScience
Influenced/CitingInfluencing/cited
Major differences in citation levels and half-lives (CHL) among research fields
0.1
1
10
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
AGE
CIT
ES
/PA
PE
R
BIOCHEM &MOL BIOL
CLINNEUROLOGY
ORTHOPEDICS
PHARMACOL& PHARMACY
CHL=5.4 ± 0.1
CHL=5.0 ± 0.1
CHL=12.1 ± 0.4
CHL=8.8 ± 0.3
Geology; MineralogyOrthopedics12
Mathematics; Ecology11
Acoustics; Appl Math10
Agriculture; GeosciencesClin Neurol; Emerg Med9
Mech Eng; At Mo Ch PhysDentistry; Otorhinolaryn8
Cond Mat Phys;Chem EngAnat; Surgery; Nursing7
Astron; Org Chem;Med Chem; Neurosci6
Nanoscience & Technol; Appl Physics;
Pharmacol & Pharmacy Bioch & Mol Biol; Oncol
5
Immunol; Transplant4
Natural SciencesMedical-Biological SciCited Half Life
Normalised citation impact (1.0 = at world average)
The average citation rate of a unit’s papers
÷world citation average in the subfields
in which the unit is active
Corrects for differences in citation practices among fields,
publication years and type of article
Normalized journal impact factors
0.01
0.1
1
10
0.01 0.1 1 10 100
JCR-like IF
No
rmal
ized
IF
Mathematics
Biochem & Mol Biol
Profile of a group in Medical Pharmacology
0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0
NEUROSCIENCES (1.42)
ENDOCRIN & METAB (1.89)
BIOCH & MOL BIOL (0.92)
PHARMACOL & PHAR (1.66)
MULTIDISCIPL SC (0.52)
BEHAVIORAL SC (0.85)
PHYSIOLOGY (1.20)
GENETICS & HERED (3.74)
REPROD BIOLOGY (0.51)
CELL BIOLOGY (1.49)
OBSTETRICS & GYN (0.90)
RHEUMATOLOGY (2.20)
CLIN NEUROLOGY (1.87)
DEVELOPMENT BIOL (0.69)
UROLOGY & NEPHRO (2.61)
ZOOLOGY (0.74)
% Articles
Impact:Blue=High
Orange=AverageWhite=Low
Normalised citation impact
159 NL Academic Chemistry Departments
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
0 200 400 600 800
Total Publications
Nor
mal
ised
Cita
tion
Impa
ct
World Average
Skewed citation distribution of 2 journals
0.01
0.1
1
10
100
1 10 100
Nr Cites
% P
aper
s
ANALYT CHEMANALYT CHIM ACTA
N=1,932Mean=4.5
%Uncited=12
N=1,466Mean=1.9
%Uncited=28
A skewed citation distribution of a research groups’s papers is a normal phenomenon
0
2
4
6
8
10
12
14
16
18
20
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40
Nr Cites
% P
aper
s
N=150Mean=38.5Skewn=2.6
N=145Mean=4.8
Skewn=1.6
N=174Mean=5.5
Skewn=4.6
72280
Skewed citation distributions: ‘Oeuvre-hypothesis’
• Articles are elements from publication oeuvres of groups carrying out a research programme
• Authors citing an oeuvre/programme tend to cite ‘key’ or ‘flag’ papers from that oeuvre
• Key papers may snatch away citations from the oeuvre’s other articles
WoSNon-WoS
Non-WoS WoS
Citing/Source
Cited/Target
?%?%
Measurement of internal WoS Coverage
Non-Wos Journals
Books
Conference proceedings
Reports
Etc.
ZUCKERMAN H, SCIENTOMETRICS, v 12, p 329, 1987
ROUSSEAU R, SCIENTOMETRICS, v 43, p 63, 1998
MERTON RK, ISIS, v 79, p 606, 1988
GILBERT GN, SOC STUDIES SCI, v 7, p 113, 1977
GARFIELD E, ESSAYS INFORMATION S, v 8, p 403, 1985
GARFIELD, E. CITATION INDEXING, 1979 (BOOK!)
ABT HA, J AM SOC INF SCI T, v 53, p 1106, 2004
SCIENTOMETRICS 60 (3): 295-303, 2004
In basic science the percentage of 'authoritative' references decreases as bibliographies become shorter
Moed, HF; Garfield, E.
Y
Y
Y
Y
N
N
Y
Y
in WO
S
RF
SO
TI
AU
WoS Coverage = 5/7 = 71%
Not in WoS
Overall WoS coverage by main field
Humanities & Arts
Other Soc SciSoc Sci ~ MedicinePhys & Astron
MODERATE (<40 %)GeosciencesClin Medicine
EngineeringPsychol & PsychiatChemistry
EconomicsBiol Sci – Anim & Plants
Biol Sci – Humans
MathematicsAppl Phys & ChemBiochem & Mol Biol
GOOD(40-60%)VERY GOOD (60-80%)EXCELLENT (> 80%)
WoSNon-WoS
Non-WoS WoS
Citing/Source
Cited/Target
Three types of citation analysis
2. Target Expanded
3. Source Expanded
1. Pure WoS
4 Types of bibliometric studies
ModerateNo citation analysis at all4
Good – Moderate
WoS+non WoSWoS+non WoSSource expanded
3
Very Good – Good
WoSWoS+non WoSTarget expanded
2
Excellent – Very Good
WoSWoS‘Pure’ WoS1
WoS coverage
Citing/SourceCited/TargetType
What do citations measure?
• Many studies showed positive correlations between citations and qualitative judgments
• In principle it is valid to interpret citations in terms of intellectual influence
• But the concepts of citation impact and intellectual influence do not coincide
2.
Some earlier studies and their outcomes
16 broad disciplines
Social Sci ~ MedicineSOC-MEDEngineeringENG
Social Sci(SOC)Economics & BusinessECON
Psychol & PsychiatPSYClinical MedicineCLM
Physics & AstronPHYSChemistryCHEM
Multi-disciplinary(MULTI)Biol Sci ~ HumansBIOL-HU
Mol Biol & BiochemMOLBBiol Sci ~ Anim & Plants
BIOL-A&P
MathematicsMATHAppl Phys & ChemAPC
GeosciencesGEOArts & Humanities(A&H)
General European Univ
TOP 25%BOTTOM 25%
TOP 25%
BOTTOM 25%
Impact
Publications
(A&H)
APC
BIOL-AP
BIOL-HU
CHEM
CLM ECON
ENG
GEO
MATH
MOLB
(MULTI)
PHYS
PSY
SOC-MED
(SOC)
0
25
50
75
100
0255075100PUBLICATION RANK PTCL
CIT
AT
ION
I P
AC
T R
AN
K P
CT
L
Among top 25 % in publication output and citation impact
‘Top’ US/UK research university
(SOC)
SOC-MEDPSY
PHYS (MULTI) MOLB
MATH
GEO
ENG ECON
CLM
CHEM
BIOL-HUBIOL-AP
APC
(A&H)
0
25
50
75
100
0255075100PUBLICATION RANK PTCL
CIT
AT
ION
I P
AC
T R
AN
K P
CT
L
University has a top position
in each discipline
Univ Milano Univ Minnesota – Minnea.15
Univ Michigan - Ann Arbor Osaka Univ 14
Erasmus Univ Rotterdam Univ Penn 13
Univ Wien Univ Oxford 12
Univ Pittsburgh Stanford Univ 11
Univ Calif Los Angeles Univ Coll London 10
Univ Washington - Seattle Johns Hopkins Univ 9
Univ Tokyo Univ Cambridge 8
Univ Penn Kyoto Univ 7
Univ Calif San Francisco Univ Michigan - Ann Arbor 6
Karolinska Inst Stockholm Univ Washington - Seattle 5
Univ Toronto Univ Calif Los Angeles 4
Johns Hopkins Univ Univ Toronto 3
Harvard Univ Univ Tokyo 2
Univ Texas - Houston Harvard Univ 1
OncologyAll fieldsRank
1.74
1.91
1.35
1.22
1.26
1.01
1.52
0.90
1.15
1.49
1.22
1.43
1.61
1.69
0.90
1.26
1.26
0.72
1.47
1.50
1.13
1.31
1.07
2.10
1.43
1.00
1.21
1.08
1.68
2.02
1.85
2.65
1.63
1.33
10% 8% 6% 4% 2% 2% 4% 6% 8% 10%
BIOCHEM&MOL BIOL
NEUROSCIENCES
VETERINARY SC
RAD,NUCL MED IM
MEDICINE,GEN&INT
CLIN NEUROLOGY
IMMUNOLOGY
SURGERY
ONCOLOGY
GENETICS&HEREDIT
CARD&CARDIOV SYS
CHEM,MULTIDISC
CELL BIOLOGY
PSYCHIATRY
MULTIDISCIPL SC
PHARMACOL&PHARMA
PLANT SCIENCES
UNIV ZURICH UNIV COLL LONDON
% Articles
Impact:Black=High
Dashed=AverageBlank=Low
Normalised citation impact
ZURICH
BASEL
BERN
EPFLETHZ
GENEVE
LAUSANNE
AMSTERDAM
CAMBRIDGE
EDINBURGH
FREIBURG
HEIDELBERG
HELSINKIKAROLINSKA
LEIDENLEUVEN
LUND
MILANO
MUNCHEN
OXFORD
PARIS VI
PARIS XISTRASBOURG I UTRECHT
BERKELEY
FU BERLIN
HU BERLINKCL
KOBENHAVNLYON I
MELBOURNE
MICHIGAN
OSLO
PENN
TORONTO
UPPSALAWIEN
WURZBURG
0.8
1.0
1.2
1.4
1.6
1.8
2.0
0 50 100 150 200 250
NR ARTICLES IN WORLD TOP 5 %
AC
TU
AL
/EX
PE
CT
ED
NR
AR
TS
IN W
RL
D T
OP
5 %
2.5Biol Sci ~ Humans
3.
Effects of performance assessments upon
performers’ behavior
Topics
3. The UK Research Assessment Exercises (RAE)
Timing effects and shifts in criteria in UK Research Assessment Exercises (RAE)
-4.0
-3.0
-2.0
-1.0
0.0
1.0
2.0
3.0
4.0
86 87 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03
Years
An
nu
al G
row
th R
ate
% UK Articles% UK Authors
1995/61992 2000
Total Publication
Counts
Shift from Quantity to
Quality
ResearchActiveStaff
% UK authors
% UK Articles
Can one increase actual citation impact by…..
• Increasing author self citation?
• Publishing in high impact journals?
• Collaborate more intensively?
• Publishing with US authors because they overcite their own papers?
• Publishing less, only the very best papers?
• Making citation arrangements?
At the level of research groups, actual citation impact and journal prestige tend to show only weak correlations
[Set of 2,150 UK authors with > 10 articles per year]
23 %Normalised journal impact / prestige
11 %Average journal impact factor
0 %No published articles
Explained variance in actual citation impact
Indicator
More collaboration M higher impact?
• Some studies report positive correlations between a paper’s number of authors and its citation impact
• They ignore differences among fields
• It all depends upon who collaborates with whom
• Causality issue: ‘Good’’ research may lead to collaboration
Do US scientists overcite papers from their own country?
• The crucial issue at stake is the adequacy of the norm against which referencing practices of US scientists is evaluated
• A first study found no conclusive evidence that US scientists in science fields excessively cite papers originating from their own country (Moed, 2005)
Publishing less P Higher impact?
• One would expect a higher citation impact per paper (crown indicator)
• But what are the longer tem effects?
• PhD students need papers in their CV’s
• Relationship between a research group’s ‘bricks’ and ‘flag’ papers is complex
Mutual citation arrangements?
• A high impact group receives its citations from dozens if not hundreds of institutions
• The distribution of citations amongst citing institutions is very skewed
• The contribution of the tail is very large
• Making arrangements with a few institutions will not help much
Why use sophisticated citation analysis in RAE?
• Shift in focus from quantity to quality
• Reduction unintended effects of using less sophisticated indicators
• Use of absolute rather than relative standards
• Formal rather than informal use (transparency)
4.
Metrics, research evaluation and funding parameters
Metrics and funding parameters
• Policy level: Central vs. institutional (e.g., national vs. university)
• Elaboration of indicator data: Statistical vs. evaluative
Data ElaborationFunding
parameters
Central
Institution
Indicatorsof groups / individuals
Aggregate per
institution
Combine with peer
review
Acrossinstitutions
Within aninstitution
Metrics and funding parameters
Use of metrics in allocation of research funds
• At a central level: To distribute funds across institutions based on aggregate statistics
• At an institutional level: Combined with peer review to evaluate groups and individuals; outcomes are used to distribute funds within an institution
Aggregate statistics
• Random errors to some extent cancel out
• Identify and neutralize systematic errors
END