impact vitality – a measure for excellent scientists

17/05/2006 1 Herhaling titel van presentatie

Impact Vitality – A Measure for Excellent Scientists

Nadine Rons (1,*), Lucy Amez (1,2) (1) Vrije Universiteit Brussel (VUB); (2) Policy Research Centre for R&D Indicators VUB (SOOI-VUB) (*) Corresponding Author

STI 10, Vienna, Austria, 17-20 September 2008 Version 12.09.2008

September 2008 - Slide 2 STI 10

A MEASURE FOR EXCELLENT SCIENTISTS

o  Context & Research Question

o  Approach

o  Proposed Indicator

o  Evaluation

o  Conclusion


CONTEXT & RESEARCH QUESTION

o  Context: Funding programmes aimed at researchers performing at excellence level

o  Need: Measures in support of peer review

o  Focus: Individual excellence in terms of publication activity

o  Approach: Construction of an indicator reflecting a description of excellence


Approach

  =>

Test Case

Test Results

Proposed Indicator

  =>

A definition of excellence: Funding Programmes =>Excellence in terms of publications

Indicator requirements: Discussions on Bibliometric Indicators =>Principles & Requirements verification of

requirements


Towards a DEFINITION of EXCELLENCE

Descriptions of excellence from programme outlines that can be related to publications

o  "at the forefront of the field", o  "prominence in the field", o  "substantial contribution to the development of the field", o  "continued performance at top level", o  "increasing productivity", o  "lead author of papers with significant impact in the field".

e.g. Spinoza prize (NL), Leibniz prize (DE), Research professor recruitment award (IR), Methusalem programme (BE), Odysseus programme (BE)


EXCELLENCE

Individual performance at excellence level, as manifested in publication activity.

  An excellent researcher is prominently present in the field, continuously publishing new knowledge and ideas over a longer period of time.

  As an established reference in the field, his/her contributions are eagerly followed by colleagues and his/her ideas are picked up fast in their further research.

  As such, he or she is a central figure in a strong research dynamic, at the level of the researcher's own research team as well as for the research area as a whole, increasing both volume and impact of research in the field.


INDICATOR REQUIREMENTS / CHALLENGES

principles   a reflection of relevant capacities   sufficiently well correlated with peer review

acknowledging characteristics of the individual career   independent of career length   a balanced appreciation of collaborative output

fit for common use   up to date   easily calculated

acknowledging the nature of science / the discipline   outlyer proof   avoiding bias

resisting human error / interventions   error proof   manipulation proof


PROPOSED INDICATOR: Impact Vitality

Inspiration: Klavans & Boyack, STI9, concerning a vitality measure for national & institutional comparison V = (∑j=1→n 1 / ( age(j) + 1 )) / n, where n is the number of references j in a set of papers

Impact Vitality

Weighted sum of the numbers of citing publications P(y) over the years y, Weights: Lower weight for higher age i in years, Time window: n years, Normalization: 1 = P(y) constant over time; 0 = no longer cited.

IV(y1, n) = [n (∑i=1→n P(yi) / i) / ∑i=1→n P(yi) - 1] / [∑i=1→n 1/i - 1] with n > 1, yi+1 = yi - 1 and ∑i=1→n P(yi) > 0

or, as a sum over all m citing publications j in the time window: IV = [ (∑j=1→m 1 / age(j)) / (m/n) - 1] / [∑i=2→n 1/i]

Profile over time Options: moving with fixed length; growing with career from fixed start


WHAT IT REPRESENTS

A combination of publication and citation data: counting publications that cite a body of work.

A value > 1 : the number of publications citing the work increases over time, increasing "impact area" of research (impact vitality).

An Impact Vitality profile over time above 1 : a continuously growing uptake of the work in recent developments in

the field.


CHARACTERISTICS

Five fictive examples of numbers of citing publications per year

Year A B C D E F

2006 5 5 1 10 1 3

2005 5 4 2 8 2 2

2004 5 3 3 6 3 1

2003 5 2 4 4 2 2

2002 5 1 5 2 1 3

IV(2006, 5) 1,0 1,5 0,5 1,5 0,8 1,1

Constant amount of citing documents over time (case A): value = 1 Increasing amount of citing documents over time (case B): value > 1 Decreasing amount of citing documents over time (case C): value < 1

Same proportional increase of citing documents over time (cases B and D): same value

Fluctuating amount of citing documents over time: value according to the final trend, < 1 for a final negative trend (case E) and > 1 for a final positive trend (case F).


SIMILAR STRUCTURE, DIFFERENT CONTENT

AR-index AR = (∑j=1→h cit (j) / age(j))^(1/2)

where age(j) is the age of article j, cit(j) is the number of citations to it and h is the number of elements in the h-core (Jin et al., 2007).

Impact Vitality IV = [(∑j=1→m 1 / age(j))/(m/n) - 1] / [∑i=2→n 1/i]

where age(j) is the age of citing documents j and m is the number of citing documents in the time window of n years.

BASIS numbers of citations numbers of citing publications

WEIGHTS lower weight for older cited publications

lower weight for older citing publications

LIMITS citations to the h-core citing publications in a chosen time frame

UNLIMITED age of the citations age of the publications cited

CONTENT accumulated impact of the most highly cited publications, with an emphasis on the impact of the most recent highly cited publications

impact of the work in a specified period, with an emphasis on the impact in the most recent years


INDICATOR EVALUATION

principles   a reflection of relevant capacities: cfr. definition of excellence   sufficiently well correlated with peer review

acknowledging characteristics of the individual career   independent of career length: emphasis on recent citing publications   a balanced appreciation of collaborative output: trend calculation (not absolute value)

fit for common use   up to date: emphasis on recent citing publications   easily calculated: select set of documents citing the right author

acknowledging the nature of science / the discipline   outlyer proof: modulation of the impact vitality profile   avoiding bias: trend calculation (not absolute value); target expanded analysis; continued

impact of older work while impact in a new topic is built up

resisting human error / interventions   error proof: count of citing documents with at least one correctly cited name   manipulation proof: artificially raising publications, self citations, impact factor does not work

GENERAL ROBUSTNESS: errors, construction, data source


OUTCOMES: FIRST TEST RESULTS

Peer review based selection vs. Impact Vitality: first test results

IVPhD ≥ 1 for all years IVPhD < 1 for one or more years

Selected 5 — Not selected 4 4 Source: VUB Research Fellowship calls 2000-2006, applications in predefined research themes, excluding themes in Social Sciences and Humanities. Impact Vitality calculated using the Web of Science.

Senior Career Impact Vitality:

IVPhD(y1) = IV(y1, nPhD), where nPhD = y1 - yPhD + 1, the age in years of the PhD

Profile: growing window starting at PhD-year

Applicants selected by peers can be characterized by:

Senior Career Impact Vitality continuously above 1: prominence of recent impact during the whole career

Applicants not selected (for quality of budgetary reasons) also correspond to:

Start of career or Senior Career Impact Vitality with periods below 1


PROFILE OF SELECTED APPLICANTS


PROFILES OF NOT-SELECTED APPLICANTS


CONCLUSIONS

A novel indicator is proposed that reflects what is expected from excellent scientists.

Advantages of the indicator: o  relatively easy to calculate, o  hard to manipulate, o  limited sensitivity to outlyers in citation counts and to errors in references, o  a scope broader than the scientist's indexed publications, o  independence regarding size and citation culture of the research community.

First test results for a limited sample look promising.


SOME FUTURE WORK

Studies on robustness and validation.

Studies on larger samples of applicants and the selective power of different indicator variants

-> recommendation of a particular variant for the assessment of individual scientists? (within a set of indicators)

Further exploration of application possibilities.


THANK YOU!

Nadine Rons   Vrije Universiteit Brussel (VUB)   VUB R&D dept., Research Coordination Unit   Pleinlaan 2, B-1050 Brussels, Belgium   E-mail: [email protected]

Lucy Amez   Vrije Universiteit Brussel (VUB) and Policy Research Centre for R&D Indicators (SOOI)   VUB R&D dept., Research Coordination Unit   Pleinlaan 2, B-1050 Brussels, Belgium   E-mail: [email protected]

impact vitality – a measure for excellent scientists

Education

publications o

impact of research

publications j

time window of n years

o prominence

number of citing documents

age of citing documents

time case b