relative indicators and relational charts for comparative assessment of publication output and...

Scientometrics, Vol. 9, Nos 5 - 6 (1986) 281-291

R E L A T I V E I N D I C A T O R S A N D R E L A T I O N A L C H A R T S F O R C O M P A R A T I V E ASSESSMENT

OF P U B L I C A T I O N O U T P U T A N D C I T A T I O N IMPACT

A. SCHUBERT, T. BRAUN*

Information Science and Scientometrics Research Unit, Library o f the Hungarian Academy o f Sciences,

H-1361, P.O. Box 7, Budapest (Hungary)

(Received May 27, 1985)

Cross-field comparison of scientometric indicators I is severely hindered by the differences in publication and citation habits of science fields. However, relating publication and citation indicators to proper field-specific reference standards, relative indicators can be built, which may prove rather useful in the comparative assessment of scientists, groups, institutions or countries. The use of relational charts in displaying the indicators broadens the scope of such assessments. Relative indicators of chemistry research in 25 countries are presented as an illustrative example.

Introduction

The pros and cons of publication and citation measures of scientific performance

and impact are thoroughly discussed in several recent publications (see GarfieM, 2

Griffith, 3 and Edge, a among others). Although there are only a few authors who

completely reject the use of such tools in assessing scientific research, that certain

reservations are reasonable cannot be denied. The inadequacy of publication and

citation rates in crossfield comparisons is one of the most frequently claimed caveats.

Without doubt, it is illusory to match bare publication and citation counts of, say, a

pathologist and a topologist and, even within the same field or discipline, there might

be substantial differences among publication and citation habits of various subfields.

A reliable way to overcome this problem seems to be the assessment of each single paper against its own standards, thus building some kind of relative indicators of

performance or impact. The matching of these measures may then reveal some real

differences between the contributions under study.

*Also with the Institute of Inorganic and Analytical Chemistry, L. E6tv6s University, P.O. Box 123, 1443, Budapest, Hungary.

Scientometrics 9 (1986) Elsevier, Amsterdam-Oxford-New York Akaddmlai Kiad6, Budapest

A. SCHUBERT, T. BRAUN: RELATIVE INDICATORS

In the present paper, a series of such relative indicators of publication performance and citation impact is dealt with.

The basic methodological principle used consistently throughout this paper is that science journals are considered the fundamental units of assessment. Although excep- tions are known, science journals, as a rule, encompass definite research areas (frequently a single "paradigm") and also a certain standard of quality is guaranteed by the editorial "gatekeeping" process. Therefore, it seems justified to assign a set of papers to subject fields on the basis of the field classification of journals. Moreover, the average citation rate of the journal in which a paper was published is a valid standard to which its citation rate can be matched.

Indicators of publication output and citation impact might be needed to assess any kind of scientific communities: research institutes, departments, scientific societies, or whole countries. The only requirement is that the community under study should have produced a large enough number of publications to allow statistically reliable conclusions. (Statistical reliability of the indicators will be treated later in detail). The use of relative indicators will be exemplified here by the assessment of chemistry research in 25 countries. It should, however, be stressed that countries can be substituted - mutatis mutandis - by any other kind of scientific communities as

well.

Definition and properties of relative indicators

The Activity lndex

The Activity Index ( A I ) was first proposed by Frame s and was used, among others, by Braun et al.6 -7 It characterizes the relative research effort a country

devotes to a given subject field. Its definition is

the country's share in world's publication output in the given field A I =

the country's share in world's publication output in all science fields

or, equivalently,

A I = the given field's share in the country's publication output

the given field's share in world's publication output

A I = 1 indicates that the country's research effort in the given field corresponds

282 Scientometrics 9 (1986)


precisely to the world average; A I > 1 reflects higher-than-average, A I < 1 lower-than- average effort dedicated to the field under study.

It should be kept in mind that no.country can possess high A/'s in all science fields. The definition makes c!ear that the average, in a certain sense, of the A/'s over the different fields must be equal to one for each single country.

The Attractivity lndex

The Attractivity l ndex ( AA1 )7 characterizes the relative impact of a country's publications in a given subject field as reflected in the citations they attract. Its

definition is

the country's share in citations attracted by publications in the given field AA1 =

the country's share in citations attracted by publications in all science fields

or, equivalently,

the given fields's share in citations attracted by the country's publications AAI=

the given field's share in citations attracted by all publications of the world

A A I = 1 indicates that the country's citation impact in the given field corresponds precisely to the world average; AAI > 1 reflects higher-than-average, AA1 < 1 lower- than-average impact.

It follows from the def'mition that any country can have high AAFs in some fields only at an expense of having lower values in others.

Although both A I and A A I may be used by themselves their real appeal can be revealed by displaying them on a two-dimensional relational chart, explained later on.

The Relative O'tation Rate

In case of the Relative 63tation Rate ( RCR ), the reference standard to which the citation count of each paper is matched is the mean citation rate per publication of the journal in which the paper in question was published) ,7,a If the actual citation rate of a paper is called its observed citation rate, and the average citation rate of the journal in which the paper has been published is referred to as the expected citation rate, then RCR can be defined as

RCR = Z observed citation rate

x expected citation rate

$cientometrics 9 (1986) 283


where X denotes summation over all papers published by the given country in the given field.

RCR = 1 indicates that the set of papers under study are cited exactly at an average rate (as if all papers were an average paper of the corresponding journal); RCR > 1 suggests that the citation rate of the assessed papers, in average, is beyond the reference standard, RCR < 1 indicates that the papers are, in average, less cited than the

reference standard. In principle, a country may have RCR values higher than 1 in each science field;

for this indicator, the weighted average over all countries within a single field must

be equal to 1. Under proper conditions, RCR makes possible direct comparison and even linear

ranking of citation impact of publications in different fields. Since, however, even within a single subfield, various countries may use publication channels (journals) of very different quality, it may be misleading to consider only the RCR values. A two- dimensional relational chart displaying both observed and expected citation rates is

usually much more instructive.

Relational charts

Relational charts are simple two-dimensional orthogonal diagrams with identically scaled axes displaying quantities such that the "main diagonal" (the straight line x = y) represents some kind of "balanced" situation. Points above this diagonal are to be considered "higher class", those below "lower class" in a sense depending on the actual content of the chart.

Two kinds of relational charts may contribute much to grasping the relative

indicators defined above.

Relational chart displaying the attractivity vs. activity indices

The frame of this diagram is shown in Fig. 1. The vertical and horizontal dashed lines represent the unit level (the world average) of relative research effort and impact, respectively; points right to the vertical or above the horizontal dashed line reflect higher-than-average performances. However, for evaluative purposes, the most relevant question is "cost-effectiveness", i.e., whether the effort devoted to a research field (namely, the publication effort) has sufficient return in terms of its impact (viz., in terms of citations). This is revealed in the relational chart by the position of the point relative to the main diagonal (solid line). For example, point A in Fig. 1 represents a research field whose share in a country's citation "income" is much higher than its

284


share in publication "investments". Therefore, in spite of the fact that both its

relative effort and impact are below the world average, this research field deserves special attention and support. On the other hand the relative citation impact of the research field represented by point B, though higher than the world average, does not compensate for the even higher relative publication effort devoted to it.

A

x 2-- ,m.~~~m ~ "(3 C_

>

<~

1 / / 2

Activity index

Fig. i . Frame for re|atJona] chart dJsp]ay~_g the attractJvJty vs, activity radices

Relational chart displaying the observed vs. expected mean citation rates

The frame of this chart, shown in Fig. 2, is similar to that of the A / - AAI diagram. The interpretation of the different regions of this chart requires, however, somewhat more sophistication. In the case of citation rates, there is no boundary such as the value 1 was for A I and AA/. Which values of the citation rate are to be considered high or low depend primarily on the length of the source and citation periods concerned. A rather fortunate choice of these periods was made by Garfield 9

for journal evaluation studies: a source period of two years and citations in the third year are considered. Mean citation rates per publication ("impact factors") calculated on this basis for thousands of science journals are listed in the Journal Citation

Reports volumes. 10 Not without an element of arbitrariness, the value 1 can be regarded as a boundary between "high" and "low" impact factors, as indicated in Fig. 2. For any valid assessment based on citation rates it is, however; absolutely necessary to consider also the subject field of the publications in question. If, for example, points A and B in Fig. 2 reflect the citation impact of two distant fields of the same country (say, A represents algebra, B biochemistry), then it may well be

8cientometrics 9 (1986) 285


1

i I I ) i .

A �9 # y

0 .U~

O ( I I . I 2

low impact high impact journals journats

Expected mean citation rate

Fig. 2. Frame for relational chart displaying the observed vs. expected mean citation rates

possible that the great difference in the expected citation rate simply reflects the different citation habits of the two fields. In this case, field A was clearly proved to be highly effective (it scored much higher observed citation rate than expected), whereas field B fell below the schedule. If, however, the same two points reflect citation rates of two countries in the same discipline, scientists of country A should definitely be condemned for using gratuitously low quality publication channels. The publication strategy of country B would seem overly respectable, though there is room enough for improvement in performance.

Statistical reliability and error estimation of relative indicators

A necessary prerequisite of any indicator to be used for comparative assessment purposes is the possibility of complementing it with reliable error estimates. An attempt is made here to apply some unsophisticated but mathematically well-established methods of error estimation and reliability tests to the relative indicators introduced above.

Activity and Attractivity lndices

Let us consider the second of the definitions of A I and AAI, respectively. Of the two proportions, that in the numerator is the main source of statistical error. The

286 Seientometrics 9 (1986)


denominator, based on world aggregate data, is obviously loaded with a much smaller

error than the contribution of a single country. That is why the relative errors of A I

and A A 1 are taken as equal to that of their numerator. What is then needed are the error bounds within which the observed share of a given field in a country's publica-

tions (respectively citations) can be considered the estimator of the probabil i ty that a randomly drawn item will fall into the field in question. Using the error formula of the binomial distribution the following relations can be obtained:

z2u4I = A I x / 1 / N - 1/S

zkAAI = A A I ~ / 1 / M - l i T

where N and M are respectively the number of the country's publications and citations in the given field, and S and T are respectively the number of the country's publica-

tions and citations in all science fields. I f the field in question represents only a small fraction of the scientific endeavour

of the country (1 IN >~ l /S ) , then the above relations reduce to

~ A I ~- A I / x / ' N -

z kAAI ~- AAI/~/-M ---

and one can rely upon the rule of thumb that staying within 10% error bounds requires a sample of publications (respectively citations) of at least 100 items in the

given field. A simple test statistics to decide whether an A I or AAI value differs significantly

from 1 can be defined as

t A l = ( A I - 1)/AAI

~AAI = ( A A I - 1 ) /AAAI .

This statistics is a random variable of Student's t-distribution, which, provided that

the indicators are based on a sample of any reasonable size, can be approximated by a standard normal distribution. Thus, e.g., if t < 2, then the indicator does not differ

significantly from 1 at a significance level of 0.95, which is the most commonly used level in such kind of assessments.

Scientometrics 9 (1986) 287

' A. SCHUBERT, T. BRAUN: RELATIVE INDICATORS

Relative Oration Rate

The theoretical backgrounds of statistical reliability calculations of citation rates were published in a previous paper) 1 The reasoning was based on the postulate of a negative binomial distribution of citations and resulted in the folowing error formula:

ARCR = ~/RCR " Q]N

Here N is the number of the country's publications in the given field, and Q the solution of the equation

log Q/(Q - 1) = - log f /X,

where f is the fraction of uncited publications and X the mean (observed) citation rate per publication.

The test statistic as to whether an RCR value differs significantly from 1 is deffmed

as

tR CR = (RCR - 1)/ARCR

and has to be interpreted identically to the test statistics of A I and AA1.

An illustrative example

Relative indicators of chemistry research in 25 countries are presented as an illustration. In building the indicators, tapes of the Science Oration lndex (SC1) data base of the Institute for Scientific Information (ISI, Philadelphia, PA, USA) were used as data source. Relevant papers (articles, notes, letters, and reviews) published in the 1978-79 issues of 243 chemistry journals and citations to them recorded in the 1980 Citation tapes of the SC1 data base were counted. Publications were assigned to different countries according to the mail address of their first authors as indicated in the byline of the publications.

The results are shown in two relational charts (Figs 3 and 4). In addition, a list of the countries ranked by their relative citation rate is presented in Table 1. Error estimates of RCR and expected citation rates are also indicated in this table. The latter emphasizes the two-dimensionality of our method: RCR measures the relative contribution of the countries to the citation rate of the journals used for publication,

288 8cientometrics 9 (1986)


i I 0~

21-- IRL 3PN j"$UN

I " ,,r l l DEU

I rAtJ BEL#/

=NZL ~ AI'T | tsR.~J

&rtivitu index Fig. 3. Relational chart displaying the attractivity vs. activity indices in chemistry (Papers publishext

in 1978-1979, cited in 1980). Symbols: o - only A ! differs significantly from 1, �9 - both AI and AAI differ significantly from 1 (at a 95% confidence level). For country codes see Table 1

i c.. / sw~. 0

ONKoAU ~ N,Z~.

/ "IPN lO'o FRA

,I- ,4Z ~ csK/_ ?,,FJN

I ~#2_.N / 7 a ~P . i,,4o.--

Expected mean citation rate

Fig. 4. Relational chart displaying the observed vs. expected citation rates in chemistry (Papers published in 1978-1979, cited in 1980). Symbols: o - RCR does not differ significantly from 1, �9 - RCR differs significantly from 1 (at a 95% confidence level). For country codes see Table 1

Seientometrics 9 (1986) 289

A. SCHUBERT, T.BRAUN: RELATIVE INDICATORS

but the real merits of this contribution can be judged only by considering also the

absolute quality level of the publication channels (journals), particularly as all indicators

are here related to a single science field: chemistry.

No specific comments to the scores of the single countries are made here, first,

because our present aim was not to give a comparative evaluation of countries but

to introduce the methodology, second, because the results are self-explanatory and

any commentary seems to be redundant.

Table 1 List of countries ranked by relative citation rate

Citation rate Country Code relative expected

Sweden SWE 1.25 • 0.062 1.36 Switzerland CHE 1.14 • 0.058 1.82 The Netherlands NLD 1.11 +- 0.047 1.62 Denmark DNK 1.10 • 0.079 1.36 United Kingdom GBR 1.06 • 0.022 1.65 USA USA 1.05 • 0.014 2.16 FR Germany DEU 1.03 • 0.022 1.67 Canada CAN 1.03 • 0.030 1.78 Belgium BEL 1.01 • 0.050 1.42 Norway NOR 1.01 -+ 0.086 1.31 Japan JPN 0.99 • 0.016 1.25 Australia AUS 0.97 • 0.039 1.57 France FRA 0.97 • 0.021 1.39 USSR SUN 0.96 • 0.011 0.30 German DR DDR 0.96 • 0.030 0.68 Austria AUT 0.94 • 0.067 1.10 Ireland IRL 0.94 • 0.124 1.70 New-Zealand NZL 0.92 +- 0.099 1.70 Italy ITA 0.90 • 0.027 1.34 Finland FIN 0.89 • 0.070 0.95 Hungary HUN 0.89 • 0.044 0.83 Czechoslovakia CSK 0.87 • 0.029 0.79 Poland POL 0.83 • 0.026 0.69 Spain ESP 0.83 • 0.039 0.83 Israel ISR 0.77 • 0.058 1.95

References

1. T. BRAUN, W. GL~NZEL, A. SCHUBERT, Scientometric Indicators. A 32 Country Comparison o f Publication Producffvity and Citation Impact, World Scientific Publishing Co., Singapore, 1985.

290 Scientometrics 9 (1986]


2. E. GARFIELD, Is citation analysis a legitimate evaluation tool? Scientometrics, 1 (1979) 359. 3. B. C2 GRIFFITH, Science literature: How faulty a mirror of science? Aslib Proceedings, 31

(1979) 381. 4. D. EDGE, Quantitative measures of communication in science: A critical review, History of

Science, 27 (1979) 102. 5. J. D. FRAME, Mainstream research in Latin America and the Caribbean, lnterciencia, 2 (1977)

143. 6. E. BUJDOSO, Y. BRAUN,publication indicators of relative research efforts in physics subfields,

Journal o f the American Society for lnformation Science, 34 (1983) 150. 7. T. BRAUN, E. BUJDOSO, A. SCHUBERT, Research and Publication Trends in Analytical

Chemistry to be published in 1986. 8. A. SCHUBERT, W. GLXNZEL, T. BRAUN, Relative citation rate: A new indicator for

measuring the impact of publications, In: D. TOMOV, L. DIMITROVA (Eds), Proceedings of the First National Conference with International Participation on Scientometrics and Linguistics of Scientific Text, Varna, 1983, p. 80-81.

9. E. GARFIELD, Citation analysis as a tool in journal evaluation, Science, 178 (1972) 471. 10. E. GARFIELD, (Ed.), Science Otation lndex, Journal Citation Reports. A Bibliometric

Analysis of Science Journals in the ISI Database. Institute for Scientific Information, Philadelphia, annually from 1975.

11. A. SCHUBERT, W. GL)[NZEL, Statistical reliability of comparisons based on the citation impact of scientific publications, Scientometrics, 5 (1983) 59-74

Scientometries 9 (1986) 291

7

relative indicators and relational charts for comparative assessment of publication output and...

Documents