what’s all the h about? a summary of performance metrics for academics and journals
Post on 05-Dec-2014
636 Views
Preview:
DESCRIPTION
TRANSCRIPT
context quantities merit elite
metrics timeline h analysis
significancecontrasts options application
data are the currency of science quantities
data are the currency of science
+
quantities
the greater good = feedback, filtering, change in behaviors.
quality versus quantity
volume of scientific literature immense
volume of scientific literature immense
constellation of ideas
merit
performancetime
loss
filter
rank
sort
group
legacy
quality
impact forms
+ -
unfortunately, much of the filtering not based on reading but numbers & tendencies
merit
critical assumption
0
5
10
15
20
25
30
-5 -4 -3 -2 -1 0 1 2 3 4 5
Cite
s/yr
/pap
er a
nd IF
Effect size
cites/yr/paper
IF
reject accept0
0.1
0.2
0.3
0.4
0.5
0.6E
ffect
siz
e
Support hypotheses
0
1
2
3
4 5 6 7
LOG
cita
tions
per
pub
licat
ion
LOG total funding
Why is this merit concept important?
Fig. a)
Number of Articles
Num
ber o
f HC
0
5
10
15
20
25
100 200 300 400
Fig. b)
Number of Citations
Num
ber o
f HC
0
5
10
15
20
25
30
0 5000 10000 15000 20000
Fig. c)
Proportion Citations to Most Cited Article
Num
ber o
f HC
0
10
20
30
40
50
60
70
0.0 0.1 0.2 0.3 0.4 0.5
Fig. d)
Number of Journals
Num
ber o
f HC
0
10
20
30
40
0 20 40 60 80
Box 1
elite
Fig. a)
log(Articles in Nature or Science)
Num
ber o
f HC
0
20
40
60
80
0 1 2 3 4 5
Fig. b)
Proportion Articles in Nature and Science
Prop
ortio
n C
itatio
ns
0.0
0.2
0.4
0.6
0.8
●●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●● ●●
●●
●●
●
● ●●●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●● ●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●● ●
●
●
●
●●
●
●
●●
●
●
● ●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●●
0.1 0.2 0.3 0.4 0.5 0.6
Fig. c)
Proportion Articles in Favorite Journal
Num
ber o
f HC
0
10
20
30
0.2 0.4 0.6 0.8
Fig. d)
Proportion Articles in Favorite Journal
Prop
ortio
n C
itatio
ns
0.2
0.4
0.6
0.8
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
Box 2
Fig. a)
Proportion Single Authored Articles
Num
ber o
f HC
0
20
40
60
80
100
120
0.0 0.2 0.4 0.6
Fig. b)
Proportion Single Authored Articles
Prop
ortio
n C
itatio
ns
0.0
0.2
0.4
0.6
0.8
●
●
●●
●
●● ●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●
●●
● ●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●●
●●
●
●●
●
●●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
●
●
●●●●
●
●
●
●
●
● ●●
●●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.1 0.2 0.3 0.4 0.5 0.6
Fig. c)
Proportion First Authored Articles
Num
ber o
f HC
0
5
10
15
20
25
30
0.0 0.2 0.4 0.6 0.8
Fig. d)
Proportion First Authored Articles
Prop
ortio
n C
itatio
ns
0.0
0.2
0.4
0.6
0.8
1.0
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8
Box 3
the metric elite misses not only diversity of people but ideas
# publications# citationsh-index g-index w-index a-index r-index e-index m-index hg-index q2-index i10-index
20052006
2011
1905
mcpp
timeline
1902
JIF
Why isn’t impact factor or citations sufficient?
h
h relies on citations to papers not journalsnot skewed by singletons
not influenced by large body of uncited papersminimizes politics of publication
useful for similar stage comparisonsapplied to any group
counts citations regardless of whydoes not account for variations in average numbers of pubs
ignores number & position of authorslimited by total number so juniors disadvantagedincreasing h at high levels difficult so compression
data looks backwards not forward
+-
analysis
psychology
contrasts
economics
finance
marketing
quality quantity
h solution
however at larger scales, N still important
overvalued scientists do publish significantly more
Matthew effect
m
options
i10
h derivations such as h-coreor g index
the solution is composite.
still numbers, just a different box
volume of scientific literature immense application
volume of scientific literature immense
curation connections
application
application
journals individuals
departmentscollections
curation
crowdsource
crowdsource
reputation economynot based on citation capital
journals/collections
editorssubject editors
refereesreaders
writers
dissemination pyramid
curation by connection - big data
not by merit or citations - by relationships
individuals ideas data place utility
science
merit
ideas
metrics
metrics
filters
publications
however, mini-manuscripts, figshare, slideshare, pre-print servers, and data publications are transforming the publication process in science and providing new opportunities for discovery.
metrics that illuminate & provide insight will be critical
ultimately, publications in all forms are extensions of learning
citations to datasets/figures in talk
Costas, R. and Borodons, M. 2007. The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. - The Journal of Informetrics 1: 193-203.
Harzing, A. W. and van der Wal, R. 2008. Comparing the Google Scholar h-index with the ISI Journal Impact Factor. - Resarch in International Management Products & Services for Academics Report.: 1-25.
Lortie, C. J., Aarssen, L. W., Budden, A. E., Koricheva, J., Leimu, R. and Tregenza, T. 2007. Publication bias and merit in ecology. - Oikos 116: 1247-1253.
Lortie, C. J., Aarssen, L. W., Parker, J. N. and Allesina, S. 2012. Good news for the people who love bad news: an analysis of the funding of the top 1% most highly cited ecologists. - Oikos 121: 1005-1008.
Lortie, C. J., Aarssen, L. W., Budden, A. E. and Leimu, R. 2012. Do citations and impact factors relate to the real numbers in publications? A case study of citation rates, impact, and effects sizes in ecology and evolutionary biology. - Scientometrics DOI: 10.1007/s11192-012-0822-6.
Marnett, A. 2013. H-Index: What It Is and How to Find Yours. - Benchfly blog.
Priem, J., Piwowar, H. and Hemminger, B. M. 2012. Altemtrics in the wild: using scoial media to explore scholarly impact. - arXiv 1203.4745v1.
Wardle, D. A. 2010. Do ‘Faculty of 1000’ (F1000) ratings of ecological publications serve as reasonable predictors of their future impact? . - Ideas in Ecology and Evolution 3: 11-15.
top related