quantitative analysis of a visible tip of the peer review iceberg: book reviews in chemistry

Scientomerrics VoL 6. No. 6. (1984) 433-443

QUANTITATIVE ANALYSIS OF A VISIBLE TIP OF THE PEER REVIEW ICEBERG:

BOOK REVIEWS IN CHEMISTRY

A. SCHUBERT, S. ZSINDELY, A. TELCS, T. BRAUN*

Library o f the Hungarian Academy o f Sciences, Department for Informatics and Science Analysis,

Budapest, P. 0 . Box 7, H,1361 (Hungary)

(Received October 6, 1983 in revised form February 20, 1984)

Book reviews are practically unique in being public, "visfble" manifestations of the peer review process. Two hundred reviews of 39 books on chemical topics were subjected to statistical context analysis. Dominance of attitudes, consensus among reviewers, correlation between the reviewers' evaluations and the subsequent citation rate of the reviewed book were analysed,

Introduction

Peer review serves gatekeeping functions and interweaves all forms of scientific activity. Scientists are subject to the continuous scrutiny of their peers during their whole career, through judgements made during editorial refereeing, the evaluation of grant applications, awards, book reviewing, promotions, hiring, etc. In most of these cases the peers keep their anonimity. In the case of book reviews published in scientific journals, however, the reviewer usually signs his/her comments. The custom of disclosing the identity of book reviewers has its orion in the past t . However, there are leading English journals which do not conform to this practice 2 . We consider that signed book reviews represent the visible tip of the peer review iceberg. According to Bry and Afflerbach 3, book reviewing is a process of scholarly interaction that is as old as the scientific community itself. The earliest journals launched in the later part of the 17th century, consisted primarily of book notices.

A good review has been defined by Borchers 4 as "one prepared by an expert in the particular field; it contains essential specific informations; it evaluates the author with his work; it classifies and summarizes the book; it places it in its proper frame of reference; and it presents an accurate, trustworthy, critical examination in an accep- table style".

*Also with the Institute of Inorganic and Analytical Chemistry, L. E6tv6s University, Budapest, P. O. Box 123, H-1443 (Hungary).

Sctentometrics 6 (1984) 433

A. SCHUBERT et aL: THE PEER REVIEW ICEBERG

According to Sarton 2 , a review should describe the book at hand, but also evaluate it in terms of the subject with which it is concerned. The author's qualifications, purpose in writing the book, research ability, and success in achieving his purpose should be assessed. The reviewer should offer a considered judgement of the book's overall merit. Wavering, voidly "impartial" reviews should be avoided.

Although book reviews have existed for as long as journals, they never attracted the same degree of critical attention as journal articles. For while scholarly book reviewing is viewed by some as a structured substantive form of academic communication, it is seen by others, as an idiosyncratic second-class citizen among academic publications s .

Snizek and Fuhrman 6 have pointed out that the book review in sociology, as an important mode of professional communication, is particularly well suited to the exigences of the knowledge explosion. In addition, the book review represents an important source of feedback for the author. It can also help decision making about hiring, promotions or salary increases and equally serves the publisher, the author or editor of the book, the reviewer and the journal (and its readers) 1,7,s.

A special feature of book reviews, in contrast to primary communications, is that a number of comments can be written about the same subject (book for reviewing) by various authors in diverse journals, and, of course, the reviews may disagree. The book review appeared first - although theoretically it could influence the next reviewers' comments-doesn't enjoy the advantageous position which is so typical in scientific publication in general.

Hirsch et al.9 delineating the gatekeeping process in scientific communication, investigated the importance of the relative professional status of the author and the reviewer in the book review process, in philosophy, political sciences and economics. They found that the higher the reviewer's status, the more favourable their review of a given book; and the higher the status of the reviewer and the lower the status of the author, the more favourable the review.

Snizek and Fuhrmann ~ found both the age and experience of authors to be significantly associated with the degree of favourableness of the reviews given their books. It is quite plausible that older, professionally more experienced authors are likely to receive more favourable reviews to their work than their younger, less experienced collegues~ however, surprisingly, experienced authors write more favourable reviews as well.

Champion and Morris t o analysed a number of book reviews published in sociology journals. They found that nearly 70 percent of the reviews under investigation expressed positive attitudes toward the books, less than 20 percent were negative and the remainder contained no criticism. In more exact areas, viz. in the natural sciences, one would expect more realistic and objective judgements. However, according to Landauer ~

434 $cientometrics 6 (1984)

A. SCHUBERT et al.: THE PEER REVIEW ICEBERG

"the evaluation of scientific work has become much like the stock market: influence of the opinions of others provides a coupling leading of positive feedback and to a strong dominance by fads and fashions".

The aim of this study was to scrutinize some aspects of book review practices in a single science field, chemistry. Dominance of attitudes, consensus among reviewers, correlation between the reviewers' evaluation and citation rate were quantitatively analysed in various dimensions of the reviews.

Data sources and methods of data analysis

The Index to Book Reviews in the Sciences OBRS) published by the Institute for Scientific Information (ISI, Philadelphia, USA) 1: was used as a primary source of data. In the IBRS, bibliographic data of book reviews published in the source journals of the Science Citation Index (SCI) database are listed under the data of the books reviewed. Books on chemical topics (according to the title of the book) getting at least five reviews in the January-June, 1981 volume of IBRS were selected for the study. The review file was then completed by reviews of the same books referred in the July-December, 1981 volume of IBRS. Since several of the reviews were published in journals not readily available, we decided to include in the study all books for which at least three reviews were at hand. The Final database comprised 200 reviews on 39 books. Citation data were compiled from the I981 and 1982 Citation Index volumes of the SCI 1 s

The text of each review related to the respective book was evaluated according to seven dimensions:

I. Scope of the book, compilation of the material; 2. Professional (theoretical and/or experimental) standard; 3. Style; 4. Bibliography, references; 5. Typographical level; 6. Usefulness; 7. Price.

The opinion of the reviewers in each dimension was rated as positive (+), negative ( - ) , ambivalent (• or nil (0). Positive ratings were given only for definitely positve assertions on some concrete feature of the book, such as " . . . there is lucid treat- ment of equilibrium modelling of chemical speciation, a well integrated and well documented discussion of suspended particles and their interactions, and a welcome account of interstitial waters of sediments". Obligatory phrases of politeness (e.g. "the whole book is a valuable collection of information" or "this is a worthwile volume

$cientometrics 6 (1984) 435

A. SCHUBERT et as THE PEER REVIEW ICEBERG

containing valuable information for all who are interested in the field") were left unnoticed. On the other hand, any indication of disapproval resulted in a negative, or in case of simultaneous assertion, in an abvivalent rating. Being such hard-hearted, the general tendency of saying "nil nisi bene" could somewhat be counterbalanced.

Thus for any given book k number of ratings of value +, - , +, or 0 in each of the seven dimensions were obtained, k being the number of reviews received by the book. Based on the k ratings, two indices were built:

- Interreviewer Agreement Index (IAI), measuring the degree of consensus among the reviewers concerning a certain dimension of a given book;

- Average Rating Index (ARI), measuring the mean evaluative judgement of the reviewers about a book either in a given dimension or in total.

Two methods for measuring interrater agreement have been so far commonly used in the literature: the classical interclass correlation (see e.g. Ref. 14) and the more recent technique of kappa and weighted kappa statistics I s ,16. Interclass correlation

assumes an ordinal scale of ratings, while kappa statistics - though also applicable to nominal scales - is designed for measuring agreement between two raters with regard to a large number of cases. Therefore, neither of them appeared to be particularly suited for our purposes. Moreover, the ambivalent (_+) rating was required to possess the special property of exhibiting partial agreement both with positive and negative ratings; this requirement seems to be alien from any of the statistical techniques in use. We were thus lead to develop a novel statistical procedure, which is based on a two-dimensional rating scale (it can readily extended to higher dimensions), and its use can be suggested in the traditional fields of agreement studies as well.

Theory

Let k+, k_, k• and ko be the number of positive, negative, ambivalent and nil ratings, respectively, of a given book in a single dimension. Obviously, k+ + k_ + k• + ko = k, the total number of reviews of the book. Indices IAI and ARI are then defined as follows:

• / ( k + - k _ ) ~ + k ~ 2 :

IAI --

a n d

436 Scientometrics 6 (1984)


k+ - k_ ARI=

k

IAI can easily be interpreted geometrically as k "1 times the length of a vector with components [(k+ - k_); k• while ARI is k -1 times the x-component of the same vector.

By definition, the indices have the following properties: - IAI ranges between O to 1. Value 0 shows complete disagreement (the same

number of positive and negative ratings, no ambivalent ratings, arbitrary number of nil ratings); value 1 indicates complete agreement (no nil ratings, all ratings are of the same value).

- ARI ranges between -1 to +1; limiting values correspond to all negative and positive ratings respectively. Obviously, assigning n number of ratings to a book quite randomly, in general, results in an IAI greater than zero. The question now arises, whether the observed IAI values differ significantly from those gained by mere chance. Therefore, relative frequencies p+, p_, p• and Po of positive, negative, ambivalent and nil ratings were calculated in each dimension, and by regarding these values as estimators of probabilities for assigning the respective ratings, expected values (EXPAI) and standard errors (STEAI) of IAI were calculated for any given k (see Appendix). If the ratings were independent random events (there would be no consensus among the reviewers) then the Corrected Agreement Indices (CORAI) defined as

CORAl -

IAI - EXPAI

STEAl

could be considered independent, identically distributed random variables of unit variance, therefore the statistics

SUMA = Z CORA /, -

(n is the number of books) is a standard normal random variable. If SUMAI is within the bounds of the suitably chosen percentile points of the standard normal distribution, then it can be said that no significant deviation from random agreement could be found at the given confidence level.

Sclenrometrics 6 (1984) 437


Results and discussion

Raw counts of positive, negative and ambivalent ratings in each dimension as well

as the total number o f reviews for each book are presented in Table 1.

Table 2 contains the percentage o f the different ratings in each dimension and in

total. It can be seen that the majority o f the reviewers express some kind o f opinion

concerning the professional standards, usefulness and scope o f books, while comparatively

few references were made to the price and the quality of tipography. Positive ratings

Table 1 Number of reviews and distribution of ratings

No. of Book

reviews

Dimension

1 2 3 4 5 6 7 + - + + - + + -- + + - + + - + + - • + - •

1 5 2 8 3 9 4 6 5 5 6 5 7 3 8 4 9 4

10 4 11 8 12 4 13 7 14 3 15 4 16 3 17 4 18 8 19 3 20 5

21 13 22 7 23 6 24 3 25 4 26 3 27 4 28 7 29 3

1 2 0 5 0 0 2 0 0 1 0 0 1 1 0 1 0 1 0 1 0 2 0 0 1 0 3 2 0 1 5 1 0 0 0 1 4 0 0 0 0 0 1 7 0 4 0 4 1 0 0 2 1 0 4 0 0 3 2 0 0 0 0 3 1 0 1 1 0 2 1 0 0 0 0 0 0 0 5 1 0 1 0 0 0 0 0 2 0 1 2 0 0 3 1 0 2 0 0 3 0 0 0 0 0 0 4 0 1 1 2 0 1 0 0 1 0 1 0 0 1 0 1 0 0 0 0 0 0 3 0 0 1 0 0 2 0 1 1 0 0 1 0 0 0 0 0 0 2 0 1 1 0 0 0 0 0 0 0 2 0 0 1 1 1 0 1 1 2 0 0 0 2 0 2 0 0 1 1 1 2 0 0 2 0 0 0 0 0 0 1 0 3 0 1 0 0 0 4 0 0 0 1 0 0 1 2 0 0 0 2 2 1 4 1 3 6 0 0 0 0 0 5 0 0 7 0 0 0 1 0 1 2 0 3 0 1 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 1 3 1 1 3 1 1 0 0 2 2 0 0 0 0 5 1 0 0 0 0 1 1 0 1 1 1 1 0 0 1 1 0 1 1 0 2 0 0 0 0 0 0 0 1 0 2 1 2 0 0 1 0 0 0 0 0 3 0 0 0 0 0 1 0 0 1 1 1 1 0 0 3 0 0 0 2 0 2 0 1 0 0 0 1 1 0 3 0 1 3 0 0 1 O 0 0 0 0 4 0 0 0 1 0 3 1 0 5 1 1 0 2 0 1 2 0 0 0 0 8 0 0 0 0 0 1 0 0 1 1 1 0 0 0 2 0 0 0 0 0 3 0 0 0 0 0 3 2 0 3 0 1 f r O 0 3 0 0 0 0 0 3 0 0 0 3 0

4 3 0 2 6 3 4 1 1 6 5 1 1 0 0 1 0 1 1 1 O 0 1 3 0 4 2 1 5 0 0 0 2 0 3 2 0 5 2 0 0 0 1 1 2 0 2 2 0 2 1 0 0 1 0 0 0 0 4 2 0 0 0 0 1 1 0 2 0 1 0 0 0 1 1 0 0 0 1 2 0 0 0 2 0 0 ~ 2 0 2 1 1 I 1 0 1 1 0 0 1 0 2 1 1 0 1 0 0 1 1 3 0 0 0 0 0 1 0 0 1 0 0 1 0 2 0 0 0 2 0 0 3 0 0 1 0 0 0 0 1 0 0 0 4 0 0 0 0 0 2 4 0 2 3 2 1 2 0 2 1 2 0 0 2 6 0 1 2 0 0 0 0 1 2 0 1 2 0 0 3 0 0 0 0 0 2 0 0 0 0 0

438 $ctentometrics 6 (1984)

Table 1 (cont.)


No. of Book

reviews

Dimension

1 2 3 4 5 6 7 + - - + + - - • + - - • + - - • + - - • + - - + + - - +

30 5 3 1 0 3 2 0 3 0 0 0 1 0 0 0 0 4 1 0 0 0 0 31 11 3 3 3 2 3 4 7 0 0 6 1 2 0 3 0 9 0 2 "1 1 0 32 5 2 1 0 2 2 0 0 0 0 4 0 0 1 0 0 3 0 2 0 0 0 33 4 1 2 0 2 0 2 1 0 0 1 2 0 0 0 0 3 0 1 0 0 0 34 5 1 2 1 3 0 2 1 1 0 0 0 0 0 0 0 3 1 0 2 0 0 35 3 1 2 0 1 1 1 1 0 0 1 0 0 0 0 0 1 0 0 0 0 0 36 5 1 3 0 3 1 1 2 0 0 2 1 0 1 0 0 4 1 0 0 0 0 37 3 0 1 0 1 0 1 2 1 0 1 1 0 1 0 0 1 0 1 0 1 0 38 4 0 2 0 1 0 3 1 0 0 2 0 1 0 0 0 4 0 0 0 0 1 39 3 2 1 0 2 0 1 2 0 0 2 0 0 0 0 0 3 0 0 0 0 0

Table 2 Percentage of positive, negative, ambivalent and nil ratings

Relative Dimension Total

frequencies 1 2 3 4 5 6 7

p+ % 20.0 42.5 31.0 32.5 13.5 66.0 4.0 29.9

p__ % 31.5 19.0 6.0 13.5 5.5 7.5 6.0 12.7

p~ % 4.5 24.0 0.5 5.0 2.0 8.5 1.5 6.6

PO % 24.0 14.5 62.5 49.0 79.0 18.0 88.5 50.8

are most predominant concerning the style and the usefulness of the books; there are more negative than positive ratings with regard to the scope and price of the books; the percentage of ambivalent ratings is the highest in the judgement of professional standards.

Values o f average IAI and SUMAI are presented in Table 3. I t is striking that

while average IAI (which ranges between 0 to 1) might reach rather high values

(as it does in the dimensions o f professional standards and usefulness), all these values

can essentially be at t r ibuted to mere chance. The highest SUMAI value (0.6749, in

the dimension o f professional standards) differs significantly from zero only at a

confidence level o f 50%, which means practically that the question o f tendentious

agreement vs. randomness remained completely open. Intercorrelat ion among ARI

$clentometries 6 (1984) 439


values in different dimensions is characterized by matrix of correlation coefficients

in Table 4. Low correlation coefficients suggest that ratings in the seven dimensions

chosen are practically independent (i.e., there is no redundancy in the dimensions

used in the study).

Correlation of ARI values with the number of citations received by the book in

1981 and 1982 was also determined. By the evidence of the correlation coefficients

of Table 5, reviewers' ratings and citation rates proved to be completely uncorrelated.

Table 3 Agreement indices

Dimension

1 2 3 4 5 6 7

MeanIAI 0.2981 0.5053 0.2847 0.3152 0.1660 0.6297 0.I116

SUMAI 0 .0768 0.6749 -0.0955 -0.1234 0.4314 0.1197 0.6234

Table 4 Intercorrelation among Average Rating Index values

2 0.085 3 0.281 0.002 4 0.250 0 . 1 7 3 0.096 5 -0.198 0 . 0 4 2 0.078 -0.315 6 0.526 -0.089 0 . 2 6 1 0.018 7 0.004 --0.205 -0.045 -0.028

-0.180 -0.077

1 2 3 4 5

Dimension

0.107

6

Table 5 Correlation between Average Rating Index

and citation rate

Dimension Total

1 2 3 4 5 6 7

-0.076 0 .259 -0.049 0 .026 -0.026 -0.068 0.208 0.101



However, the regression plot on Fig. 1 uncovers a peculiar pattern: the majority of points gathers in a triangular region. This can be interpreted in the fo/lowing way: an unfavourably reviewed book is unlikely to receive many citations, while a favourably reviewed book may be highly cited, but not necessarily. In other words, low opinion seems to be better reinforced by the early citation behaviour than higher one.

t30 --

5o!

40

t

30 -- �9

/

�9 26 �9

2o / / ti

�9 14ee~ 38 32

~25 37. 21 "2e e

,, ,0 3, ,o do I

I [ �9 s I 15 9

e I �9 ~2 ~.,6 .,9 �9 e6 8 e I / 13 28 4 30

. , S I Ol" 35"]' .22 ~ " l o., o o., 0.2 03

/,, /

I

r /

I I

I I

� 9

17

27

39

7 *1 0.4 0.5

Average rating index

Fig. 1. Regression plot of citation rate vs. total Average Rating Index

Condmiom

Quantitative studies on either the "visible tip" or the "invisible" part of the peer review iceberg are rather scarce. Orr and Kassab 17 investigating the editorial refereeing in some biomedical journals, have found that the agreement between referees concem-

Sdentomelrlcs 6 (1984) 441


ing the acceptance or rejection of the papers exceeded only slightly the value corres- ponding to mere chance. Quite similar results were reported by Smigel and Ross ~ s

for sociology journals. Analysing National Science Foundation's grant decisions, Cole et al.t 9 concluded "that the fate of a particular grant application is roughly half

determined by the characteristics of the proposal and the principal investigator, and

about half by apparently random elements which may be characterized as the luck of

the reviewer draw". In concordance with the results, the measure of consensus among authors of book

reviews did not differ significantly from its random value in our study either. Another line of investigations deals with the possible sources of bias in the peer

review process 2~ It was found that biasses can mainly be attributed to the

relations between the prestige of the author and the reviewer and/or their institu- tions. Although we didn't investigate systematically this point, a few additional

sources of bias in book reviewing can be mentioned: the nationality of the author and the reviewer (as a typical example the English translation of a German book has

received far more favourable reviews in German than in English journals), and the prestige of the publisher.

The predominant positive attitude of the book reviewers seems to be due to the "visibility" (i.e. non-anonymity) of their opinion. We found this most pertinently with

regard to the "usefulness" of the books. However, maybe, it was only the manifesta-

tion of the ancient wisdom (enounced by Pliny, the Elder two thousand years ago):

"No book is so bad that one cannot find something good to glean from it".

References

1. T. LINCOLN, The Book Review Business, Nature, 302 (1983) 757. 2. G. SARTON. Notes on the Reviewing of Learned Books, Science, 131 (1960) 1182-1187. 3. I. BRY, L. AFFLERBACH, Book Reviewin~ in the Science of Human Behaviour as a

Contr~ution to Scholarship by the Scientific Community, Mental Health Book Review index, 6 (1961) i.

4. G. BORCHERS, What We Expect of a Book Review, QuarterlyJ. Speech, 37 (1951.) 81-86. 5. A.P. YOUNG, Scholarly Book Reviewing in America, Libri, 25 (3) (1975) 174-182. 6. W. G, SNIZEK, E. R. FUHRMAN, Some Factors Affecti~ the Evaluative Content of Book

Reviews in Sociology, Amer. Sociologist, 14 (1979) 108-114. 7. N.D. GLENN, On the Misuse of Book Reviews, Contemp. Sociologist, 7 (1978) 254-255. 8. A . I . DALLOS, Book Reviews as Aids to Book Selection, J. Res. Commun. Stud., 2

(1979/1980) 189-192. 9. W. HIRSCH, A. M. KULLEY, R. T. EFRON, The Gatekeeping Process in Scientific

Communication; Norms, Practices, and Content of Book Reviews in Professional Journals, Working Paper No. 83, Institute of the Study of Social Change, Purdue University, 1974.

10. D.J. CHAMPION, M. F. MORRIS, A Content Analysis of Book Reviews in the AJS, ASR and Social Forces, Amer. J. Sociology, 78 (1973) 1256-1265.



11. M. LANDAUER, Nonlinearity, Multistabflity and Fluctuations: Reviewing the Reviewers, Amer. J. Physics, 241 (1981) R107-Rl13.

12. Index to Book Reviews in the Sciences (IBRS), Volumes January-June and July-December 1981, Institute for Scientific Information, Philadelphia, 1982.

13. Science Citation Index, Volumes 198I and 1982, Institute for Scientific Information, Philadelphia 1982/1983.

14. G.U. YULE, M. G. KENDALL, An Introduction to the Theory o/Statistics, 14th Edition, Hafner, New York 1950.

15. J. COHEN, A Coefficient of Agreement for Nominal Scales, Educ. Psychol. Measurement 20 (1960) 37-46.

16: J. COHEN, Weighted Kappa' Nominal Scale Agreement with Provision for Scaled Disagreement or Partial Credit, Psychol. Bult, 70 (4) (1968) 213-220.

17. R.H. ORR, J. KASSAB, Peer Group Judgements of Scientific Merit: Editorial Refereeing, Congress of the Intern. Federation for DocumentatiOn, Washington D. C., October 15, 1965.

18. E . O . SM1GEL, H. L. ROSS, Factors in the Editorial Decision, Amer. Sociologist, 5 (February 1970) 19-21.

19. S. COLE, J. R. COLE, G. A. SIMON, Chance and Consensus in Peer Review, Science, 214 (1981) 881-886.

20. H. ZUCKERMAN, R. K. MERTON, Patterns of Evaluation in Science: Institutionalisation, Structure and Functions of the Referee System, Minerva, 9 (1) (1971) 66-100.

21. S. COLE, L. RUBIN, J. R. COLE, Peer Review and the Support of Science, Scientific American, 237 (4) (1977) 34-41,

221 D.P. PETERS, S. J. CECI, Peer-review Practices .of/Psychological Journals: The Fate of Published Articles Submitted Again, Behav. Brain Sciences, 5 (1982) 187-255.

23. M.D. GORDON, The Role of Referees in Scientific Communication, in: J. HARTLEY (Ed.), The Psychology o f Written Communication, Kogan-Page, 1980.

Appendix

The m-th moment o f IAI is

E m = 1~ {[(k+ - k _ _ ) 2 4- l ~ ] / k I m/2 (t.,/t, i k 'k 'k i~-k. k k• k o ~ , , . / s + . . • o.jt,4- P2_'P+ Po '

where summation runs over all non-negative integers k+ , k _ , k+, k o, for which

k + + k _ + 1r + k ~ = k. Thus, the expected value and the standard error o f IAI

can be expressed as

EXPAI = E l ,

and

STEAl - (E 2 - E2) t/2

respectively.

Sdentometrics 6 (1984] 5

443

quantitative analysis of a visible tip of the peer review iceberg: book reviews in chemistry

Documents