—

•

•

•

•

•

AGENDA

— BACKGROUND

•

•

—

TRENDS IN SCHOLARLY RESEARCH

—

—

—

—

—

WHY DO INSTITUTIONS EVALUATE RESEARCH?

—

—

—

—

—

WHY DO INSTITUTIONS EVALUATE RESEARCH?

WHY USE BIBLIOMETRICS?

STAKEHOLDERS IN RESEARCH EVALUATION

WHAT DO CITATION COUNTS REALLY MEASURE?

WHAT DO CITATION COUNTS REALLY MEASURE?

—

BIBLIOMETRIC PERSPECTIVES ON SELF-CITATION

•

•

•

•

—

USEFULNESS ≠ QUALITY

CITATION FREQUENCY CORRELATES WITH

OTHER MEASURES OF PEER ESTEEM

BEST PRACTICE: INFORMED PEER REVIEW

CITATION METRICS ARE ONE PIECE OF

THE RESEARCH PERFORMANCE PUZZLE.

—

—

—

•

•

•

•

•

—

•

•

— WEB OF SCIENCE

CITATION METRICS

1

8

•

•

CITATION METRICS ARE ONLY AS GOOD AS THEIR SOURCE

THE WEB OF SCIENCE CORE COLLECTION

SOURCE & FOUNDATION

— APPROPRIATE USE

•

•

SAN FRANCISCO DECLARATION ON

RESEARCH ASSESSMENT (DORA)

—

THE LEIDEN MANIFESTO FOR RESEARCH METRICS

—

EXERCISES USING INCITES

•

•

•

USE A VARIETY OF INDICATORS

ADDITIONAL REFERENCE MATERIAL

— JOURNAL ANALYSIS

•

•

—

JOURNAL IMPACT FACTOR REFLECTS

A JOURNAL’S OVERALL PERFORMANCE

—

Eigenfactor score reflects a journal’s footprint in the overall journal-citation

network, measuring its influence in the entire network. It is based on the

Google PageRank method.

Journal C has a

higher PageRank or

“weight” than Journal

E, even though it has

fewer links citing it;

the one cite it does

have is of a much

higher value.

From Wikipedia

– “PageRank”

EIGENFACTOR SCORE

— NORMALIZED EIGENFACTOR SCORE

Normalized Eigenfactor

Score: a value of 1 indicates

average influence. A higher

value indicates above average

influence

— INSTITUTIONAL ANALYSIS

3

4

•

WHY NORMALIZE?

0

500

1000

1500

2000

0 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54 57 60 63 66 69 72 75 78 81 84 87

0

500

1000

1500

2000

0 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54 57 60 63 66 69 72 75 78 81 84 87

10

7.5

36

ACCOUNT FOR FIELD, AGE, AND DOCUMENT TYPE

IS 20 CITATIONS GOOD OR BAD?

ACCOUNT FOR FIELD, AGE, AND DOCUMENT TYPE

Percentile in subject area

smaller is better — in this example 0.04% of the papers in the

category (plant sciences) in 2014 had more citations

ACCOUNT FOR FIELD, AGE, AND DOCUMENT TYPE

Average percentile

for a group of papers, we average all of the documents’

percentiles

— AUTHOR ANALYSIS

3

9

•

AUTHOR IDENTIFICATION

40

Author clustering -applied to Web of Science

Core Collection regularly

6.2 million Web

of Science Core

Collection records

•722,000

ResearcherIDs

•7.7 million Web

of Science Core

Collection records

AUTOMATED AUTHOR VERIFIED

THE H-INDEX

DETERMINING H-INDEX

—

—

•

•

—

•

—

USING H-INDEX IN RESEARCH EVALUATION

THESE THREE AUTHORS HAVE THE SAME H-INDEX (4)

THANK YOU

— APPENDIX

4

6

—

WHY CITATION METRICS?

—

—

—

—

Bornmann, L., & Marx, W. (2015). Methods for the generation of normalized citation impact scores in bibliometrics: Which method best reflects the judgements of experts? Journal of

Informetrics, 9(2), 408-418. doi:10.1016/j.joi.2015.01.006

Clark, K.E. (1957). The APA study of psychologists. American Psychologist, 9, 117–120.

Cole, S., and Cole, J.R. (1967). Scientific output and recognition: A study in the operation of the reward system in science. American Sociological Review, 32, 377–390.

Derrick, G. E., Haynes, A., Chapman, S., & Hall, W. D. (2011). The Association between Four Citation Metrics and Peer Rankings of Research Influence of Australian Researchers in Six

Fields of Public Health. Plos One, 6(4). doi:10.1371/journal.pone.0018521

Garfield, E., and Welljams-Dorof, A. (1992a). Of Nobel class: A citation perspective on high impact research authors. Theoretical Medicine, 13, 118–126.

Lovegrove, B. G., & Johnson, S. D. (2008). Assessment of research performance in biology: How well do peer review and bibliometry correlate? Bioscience, 58(2), 160-164.

doi:10.1641/b580210

Mryglod, O., Kenna, R., Holovatch, Y., & Berche, B. (2013). Absolute and specific measures of research group excellence. Scientometrics, 95(1), 115-127. doi:10.1007/s11192-012-

0874-7

Norris, M., & Oppenheim, C. (2010). Peer review and the h-index: Two studies. Journal of Informetrics, 4(3), 221-232. doi:10.1016/j.joi.2009.11.001

Oppenheim, C. (1997). The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology. Journal of

Documentation, 53(5), 477-487. doi:10.1108/eum0000000007207

Small, H.G. (1977). Co-citation model of a scientific specialty: – a longitudinal study of collagen research. Social Studies of Science, 7 (2), 139–166.

Smith, A.T., and Eysenck, M. (2002). The correlation between RAE rankings and citation counts in psychology. Technical Report, Psychology, University of London, Royal Holloway.

http://cogprints.ecs.soton.ac.uk/archive/00002749/01/citations.pdf

Van Raan, A. F. J. (2006). Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics, 67(3), 491-

502. doi:10.1556/Scient.67.2006.3.10

Vieira, E. S., Cabral, J. A. S., & Gomes, J. (2014). How good is a model based on bibliometric indicators in predicting the final decisions made by peers? Journal of Informetrics, 8(2),

390-405. doi:10.1016/j.joi.2014.01.012

Virgo, J. A. (1977). A Statistical Procedure for Evaluating the Importance of Scientific Papers. Library Quarterly, 47 (4), 415-430.

VALIDATION STUDY BIBLIOGRAPHY

THOMSON REUTERS – THE AUTHORITY ON CITATION DATA

KEEP DATA COLLECTION AND ANALYTICAL PROCESSES

OPEN, TRANSPARENT, AND SIMPLE

ACCOUNT FOR VARIATION BY FIELD IN PUBLICATION AND

CITATION PRACTICES

Understand the subject areas you are using

InCites subject area schemes Source Type

Web of Science Thomson Reuters Journal to category

Essential Science Indicators Thomson Reuters Journal to category

Global Institutional Profiles Project (GIPP) Thomson Reuters Category to category

ANVUR Italy Category to category

Australia ERA Australia Journal to category

China SCACD China Category to category

FAPESP Brazil Category to category

OECD (Frascati) OECD Category to category

UK RAE and REF UK Category to category

KAKEN Japan Category to category

BIBLIOMETRIC PERSPECTIVES ON “NEGATIVE” CITATION

citations citation stacking

article-type manipulation

self-citation false publications

mentions Purchased likes mention bots

Purchased

retweets social media promotion tools

& Bots

Usage Bots

Harvesters & scrapers