ranking journals using - amazon s3€¦ · • shown how to extract journal ranking from mentions...

Ranking

Journals

Using

Altmetrics

© Imperial College London

15th International Society of Scientometrics and Informetrics

Conference, Istanbul

30th June 2015

Tim Evans

Centre for

Complexity Science

Work done with:

Tamar Loach


Ranking Journals Using Altmetrics

• Traditional Journal Metrics

• The Altmetric Data

• Journal Ranking from Altmetric Data

• Results

• Visualisation

• Conclusion

Paper Citation Network OR Journal Citation Network


Paper

Journal A Journal B

A

B

3

2

3

2

Journal or Paper Citation Network?

• Measure for each paper then aggregate

over papers in journal

– e.g. Impact Factor

• Aggregate papers in journal, then analyse

citations between journals e.g.

– e.g. EigenFactor

Order makes a difference when distributions

are fat-tailed and methods are non-linear.


Traditional Journal Metrics Used Here

We used

• Impact Factor 2013

– Number of citations in 2013 to articles

published in journal in 2011 & 2012,

divided by number of articles published in

‘11 and ‘12.

• EigenFactor 2013 [Bergstrom 2007]

– PageRank on Journal citation network


Note both based on the same

Web of Science database

Traditional Journal Metrics

• Aim to measure attention received by a

journal

• Based on citations

• Simple Citation Counts per paper

• Count citations weighted by the

importance of source (self-consistent,

PageRank style)






• Results

• Visualisation

• Conclusion

Altmetric Data

• Data from altmetric.com

• 20 months ending June 2013

• Sources include

– Facebook

– Twitter

– BBC

– Blogs

– …


Information in our altmetric.com data

• An account can be one or many people

• One person may contribute to several accounts


A1

P1

J1 Journal identifier

Paper identifier

Author (account and source)





• Results

• Visualisation

• Conclusion

Simple Altmetric Counts

Rate Journals by counting altmetric mentions

Example: CA = Combined altmetric.com

Counts number of mentions of articles

published in a journal weighted by source

– Blog or News item worth more than Twitter mention


>,

Pairwise Comparison by each Author

• No statement

about a journal if

author does not

read its articles


A1

P1

J1

P2 P3

J2

P4

J3

Authors Define their own Field

Pairwise comparison

by each Author

means comparisons

are within their field


A1

P1

J1

P2 P3

J2

P4

J3 J4

J5

Count Mentions


A1

J1 J2 J3

2

1

1

0

0

Jaj=

j

a-th column of matrix J

counts mentions by author a

a=A1,

j=J1,J2,J3,J4,J5

J4

J5

Combine Counts using Author Comparison


Score for journal j

when compared to

journal l

number of mentions

of journal j by

authors who mention

BOTH j and l,

divided by number of

such authors

𝑆𝑗𝑙 =1

𝐴𝑗𝑙

𝑎∈𝐴𝑗𝑙

𝑆𝑎𝑗

Sjl =

Sjl =

, Ajl = set of authors who

mention papers in

both journal j and l

1

j l

32

Combine Counts using Author Comparison: Example


• Author 1 reads journal j

twice but journal l once.

• Author 2 reads both

journals once.

• Author 3 does not read

journal j.

𝑆𝑗𝑙 =1

𝐴𝑗𝑙

𝑎∈𝐴𝑗𝑙

𝐽𝑎𝑗

a=2 a=3

a=1

j l

Sjl = ½(2+1) = 1.5

Slj = ½(1+1) = 1.0

Combine Counts using Author Comparison: Other

© Imperial College London𝑃𝑗𝑙 =

1

𝐴𝑗𝑙

𝑎∈𝐴𝑗𝑙

Θ 𝐽𝑎𝑗 − 𝐽𝑎𝑙

a=2 a=3

a=1

j l

Pjl = ½(1+0.5) =0.75

Plj = ½(0+1) =0.25

Many, many alternatives[Langville & Meyer, 2012]

e.g. 1 point for a ‘win’,

0 for a ‘loss’,

½ for a ‘draw’

Produce Journal Ranking and Rating

Once you have a journal-journal

score matrix, Sjl, Pjl, etc etc

apply use centrality measures

• PageRank

• HITS

• PSR = Points Spread Rating

• etc, etc …


[Langville & Meyer, 2012]





• Results

• Visualisation

• Conclusion

Overall Comparisons

• Basic Multivariate Statistics for overview of

behaviour of different ratings

– Hierarchical Clustering

– Principle Component Analysis

• Look at top forty or so journals by eye



EF=EigenFactor

IF=Impact

Factor

CA=Combined

altmetric.com

Most rating

measures

similar

except

PSR

(Points

Spread

Rating)

One source, several rating variations

[Spearman Correlation Matrix]

Different Rating Methods

Most

sources

reasonably

similar

but they do

show

interesting

variations



Different Sources

Blogs


EF=EigenFactor

IF=Impact

Factor


Blogs & Other Sources

Blogs are

closest to

traditional

standard

rankings:

Impact

Factor

&

Eigen-

Factor

Blogs


EF=EigenFactor

IF=Impact

Factor


CA = Combined altmetric.com

Blogs

CA(combined

altmetric.com)

weighted

altmetric

count

balances

source

differences

nicely,

much

closer to

IF & EF

CA=Combined

altmetric.com


EF=EigenFactor

IF=Impact

Factor


Simple Alt Media Counts

CA=Combined

altmetric.com

Simple

counts of

alt. media

are often

similar to IF

& EF:

CA (2),

Blind Count

(1),

Paper

Count (7),

Social

Media

Count (14)

BC (1) &

SMC (14)

Paper

Count (7)


Q, H, news Q, H,Blogs S,PR,blogs IF 20131 Nature Nature Nature N.E.J.Medicine2 PNAS PNAS Science Rev.Mod.Phys.3 PLoS ONE Science PNAS Chemical Rev.4 Science PLoS ONE PLoS ONE Nat.Rev.Genetics

5 N.E.J.Medicine N.E.J.Medicine N.E.J.Medicine The Lancet6 B.M.J. B.M.J. B.M.J. Nature7 Nature Comm.s The Lancet The Lancet Nature Rev.Mol.Cell Bio.

8 JAMA JAMA JAMA Ann.Rev.Immunology

9 The Lancet Proc.R.Soc.B: Bio.Sci PLoS Biology Nature Materials

10 Pediatrics Current Biology Current BiologyNature Genetics11 Psychological Sci. Pediatrics PLoS Medicine Nat.Rev.Cancer12 Current Biology Psychological Sci. Proc.R.Soc.B: Bio.Sci Adv. in Physics13 Sci. Reports J.Neuroscience Psychological Sci. Nat.Rev.Immunology

14 Cell Am.J.Clinical Nutrition Pediatrics Nat.Rev.Drug Discovery

15 Nature Medicine PLoS Medicine Cell Nat.Biotechnology





• Results

• Visualisation

• Conclusion

Cube Box Space

• Each journal has D

rankings

• Connect

from journal j to

journal l

iff

𝑗𝑟 < 𝑙𝑟 ∀ 𝑟 = 1,2,⋯ , 𝐷


D=2

[Bollobás & Brightwell 1991]

Ranking 1

Rankin

g 2

Journal Top Trumps: Nature vs. PRL


Nature

Nature is a weekly, international, interdisciplinary journal of science, published by MacMillan Publishing.

Physical Review

Letters

The American Physical Society publishes Physical Review Letters with short article

on fundamental and interdisciplinary physics research.

Connect as Nature higher than PRL on all three ratings

Journal Top Trumps: Nature vs. NEJM


The New England

Journal of Medicine

The mission of NEJM has been to bring physicians the best research at

the intersection of biomedical science and clinical practice.

Nature

Nature is a weekly, international, interdisciplinary journal of science, published by MacMillan Publishing.

No connection as Nature beats (loses) NEJM on EF & CA (IF)

Box Space Representation of

Journal Rankings


• Node = Journal

• Directed Edge

if journal has

better IF, EF

& CA

• Height

=

minimum

longest

path

distance

to root

• Top 1000

journals

used

• Top 40

shown

wors

e

Hasse diagram technique[Brüggeman et al. 1994]





• Results

• Visualisation

• Conclusion

Conclusions

• Shown how to extract journal ranking from

mentions of papers in alternative media

sources

• Shown that most ranking schemes are

reasonable

– But not Point Ranking Scheme

• Some interesting differences

– To understand why need some bibliometric input

• Which scheme is best?

– Axiomatic, Social science criteria?


Tim Evans, Imperial College London

http://netplexity.org

Work done with: Tamar Loach

http://netplexity.org/

Thanks

Work done in

collaboration with Tamar Loach

• Euan Adie (altmetric.com)

– provided and explained the altmetric data

• Daniel Hook (Digital Science)

– provided financial support and useful discussions.

• Jonathan Adams (Digital Science)

– for useful discussions


Bibliography

• Archambault, É., & Lariviére, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635–649.

• Bergstrom, C. (2007). Eigenfactor: Measuring the value and prestige of scholarly journals. College and Research Libraries News, 68, 314-316.

• Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of

Informetrics, 8, 895-903.

• Brin, S. & Page, L. (1998). The anatomy of a large-scale hypertextual Web search engine. Computer networks and ISDN systems, 30, 107-117.

• Bruggemann, R.; Halfon, E.; Welzl, G.; Voigt, K. & Steinberg, C. Applying the Concept of Partially Ordered Sets on the Ranking of Near-Shore

Sediments by a Battery of Tests J. Chem. Inf. Model., American Chemical Society (ACS), 2001, 41, 918-925.

• Bruggemann, R., M¨unzer, B. and Halfon, E., An algebraic/graphical tool to compare ecosystems with respect to their pollution – the German

river “Elbe” as an example - I: Hasse-diagrams, Chemosphere, 28 (1994) 863–872.

• Garfield, E. (1955). Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas, Science, 122 (3159), 108-

111.

• Garfield, E. (1990). Journal citation studies 52. The multifaceted structure of crystallography research. Part 1. Core journals, high impact

papers, and current research fronts. Essays of an Information Scientist: Journalology, KeyWords Plus, and other Essays, 13, 327-336

• Kleinberg, J. M. (1999). Authoritative sources in a hyperlinked environment. JACM, 46, 604-632.

• Langville, A. N. & Meyer, C. D. (2012) Who’s #1?: The science of rating and ranking, Princeton: Princeton University Press.

• Clough, J. R.; Gollings, J.; Loach, T. V. & Evans, T. S., Transitive reduction of citation networks, J.Complex Networks, 2015, 3, 189-203

[10.1093/comnet/cnu039 arXiv1310.8224]

• Clough, J.R. & Evans, T.S. What is the dimension of citation space? 2014 arXiv:1408.1274

• Evans, T.S., Complex Networks, Contemporary Physics, 2004, 45, 455-474 [10.1080/00107510412331283531 arXiv:cond-mat/0405123]

• Evans, T.S., Clique Graphs and Overlapping Communities, J.Stat.Mech, 2010, P12037 [10.1088/1742-5468/2010/12/P12037 arXiv:1009.0638]

• Evans, T.S. & Lambiotte, R., Line Graphs, Link Partitions and Overlapping Communities, Phys.Rev.E, 2009, 80, 016105

[10.1103/PhysRevE.80.016105 arXiv:0903.2181]

• Evans, T.S. & Lambiotte, R., Line Graphs of Weighted Networks for Overlapping Communities, Eur.Phys.J. B 2010, 77, 265–272

[10.1140/epjb/e2010-00261-8 arXiv:0912.4389]

• Expert, P.; Evans, T. S.; Blondel, V. D. & Lambiotte, R. Uncovering space-independent communities in spatial networks, PNAS, 2011, 108,

7663-7668 10.1073/pnas.1018962108 [10.1073/pnas.1018962108 arXiv:1012.3409]

• Garfield, E.; Sher, I. H. & Torpie, R. J., The use of citation data in writing the history of science, DTIC Document, 1964.

• Holme, P. & Saramäki, J., Temporal Networks, Physics Reports, 2012, 519, 97-125.

• Hummon, N. P. & Dereian, P., Connectivity in a citation network: The development of DNA theory, Social Networks, 1989, 11, 39-63.


Tim Evans

Centre for Complexity Science

http://netplexity.org

http://dx.doi.org/10.1093/comnet/cnu039

http://arxiv.org/abs/1310.8224


http://dx.doi.org/10.1080/00107510412331283531

http://uk.arxiv.org/abs/cond-mat/0405123

http://dx.doi.org/10.1088/1742-5468/2010/12/P12037


http://dx.doi.org/10.1103/PhysRevE.80.016105


http://dx.doi.org/10.1140/epjb/e2010-00261-8


http://dx.doi.org/10.1073/pnas.1018962108

http://dx.doi.org/10.1073/pnas.1018962108


http://netplexity.org/

ranking journals using - amazon s3€¦ · • shown how to extract journal ranking from mentions...

Documents