A Matter of Standards. II. Grantsand academic positions

As with the matter of standards for the individual scientist

(see previous editorial), the question of the appropriate

standards for awarding research grants and academic

positions seems straightforward initially. For the awarding of

grants, the aim is to fund the best research, as assessed in

terms of the goals of the granting agency. For new positions,

the aim is to promote the best scientists. At this level of Platonic

ideality, the matter of standards is indeed simple and non-

controversial.

The problems begin when one starts to weigh the specific

working criteria that are used to judge the ‘‘best’’, whether of

research proposals or of scientists. Once one enters this

realm, the topic becomes vast, given the diversity of criteria

employed by different institutions and bodies for judging

research projects and individuals. For research grants, a

granting agency with a fairly narrowly defined mission in

applied research, for instance one with a medical or

agricultural mission, will have a different set of criteria than

the National Science Foundation, with its broader remit. With

respect to hiring, a small university whose academic staff have

heavy teaching loads will have different criteria for appoint-

ments than one of the top research universities, though often

the smaller institutions place a premium on research ability as

well as teaching skill. In effect, each institution will, and indeed

should, set its own evaluative system and criteria.

Nevertheless, certain general features of both evaluative

processes—features that tend to work against the general

goal of selecting the ‘‘best’’—have become apparent in recent

decades. The focus here will be on the US and major European

systems and the emphasis in this brief comment will be

on basic research (research that does not have a specific

developmental application as its goal) and on the criteria

for hiring the academics who do such research. (One major

problem, of course, is that ‘‘basic’’ research in nearly all the big

funding systems is increasingly under pressure to justify its

existence in terms of potential applications(1) but that aspect,

worthy of separate treatment at length, will not be dealt with in

this piece).

The awarding of every grant and position involves a set of

comparative judgments in order to make a bet on the future,

using past performance as a guide. Since future outcomes

necessarily have a measure of uncertainty, there will always be

a degree of doubt attached to the outcome. It is human nature,

in such situations, to try to raise the chances of making the bet

a successful one. Since ‘‘success’’ in research is often equated

with the number of publications produced and since data-rich

papers are virtually guaranteed publication, this fully under-

standable impulse leads to the preferential awarding of grants

to projects that have the highest probability of yielding data.

Yet, while new data can, and often do, lead eventually to new

insights, they are not the equivalent of ‘‘discovery’’ in the

classic sense, in which truly new insights into difficult problems

are directly obtained. The consequence is that the largest

granting systems (e.g. the NIH in the US, most of the EU

‘‘framework’’ programmes in Europe) tend to reward the most

conservative strategies, in which significant discovery is a

hoped-for by-product but where the real, immediate pay-off is

in data, numbers of papers, etc.

There is a rich paradox here. Scientific discovery is about

exploring the unknown. The most probing research, designed

to do just that, is inherently uncertain. Yet, today, the primary

funding mechanisms are highly risk-averse and projects that

are seen as having unpredictable outcomes are at a relative

disadvantage in the judging process. It would be going too far

to say that the major funding mechanisms today positively

inhibit scientific discovery (in the classic sense) but their mode

of operation hardly fosters it.(2)

There are two further undesirable consequences of this

general tendency. First, it favours larger and larger enterprises

that, in effect, transform scientific research into an industrial

enterprise: big laboratories or consortia of co-operating

laboratories, with top-down management structures and lots

of junior scientists as workers who frequently have minimal

intellectual input into the whole enterprise. Indeed, such

systems, in which a small minority get to think and the great

majority of scientists serve as data-generating hands have

been hailed by some as the wave of the future.(3)

Yet what a waste of talent such systems involve! Most

individualswho attain Ph.D.s have far more to offer than simply

their efforts as worker-drones in data-gathering enterprises. If

the great majority of research jobs in the biological sciences

come to be of this sort, those talents will be thrown away.(2)

Creative insights always arise in the minds of individual human

beings, though often as the result of conversations with one or

a small number of other individuals. Scaling up the number of

individuals working on a project, where there is a strong

division of labour, does not scale up the number or frequency

of those insight-generating interactions. In fact, it almost

certainly does the reverse. To say all this is to restate the

familiar argument that ‘‘big’’ science is not the best structure for

the biological sciences, that is, if the aim of science is to obtain

new insights into the workings of the natural world. Even if it

is true that the nature of scientific discovery in biology

has largely changed to an incrementalist mode, favoured by

BioEssays 30:923–925, � 2008 Wiley Periodicals, Inc. BioEssays 30.10 923

Editorial

‘‘data mining’’,(4) funding mechanisms should surely still

allow ample scope for adventurous and conceptually creative

risk-takers.

The other way in which the conservatism of the main

granting bodies works against scientific discovery is that the

system tends to reward those who are already established and

discriminates against younger scientists who have not yet fully

proved themselves and who are struggling to establish

labs and careers. There is a lot to be said for rewarding

older, experienced scientists, those who have already proven

themselves but to the extent that the system works against,

and discourages, younger scientists, it works against the

future health of science. Older (50þ) biologists often have a

breadth and depth of understanding that younger scientists

cannot match but it is usually the younger members of the

profession who make the most important experimental

discoveries. For example, virtually all of the major discoveries

of the golden era of molecular genetics and molecular biology

(roughly the late 1940s to the mid-1960s) were men and

women in their early twenties (e.g., Joshua Lederberg, Jim

Watson) to their late 30s. Fortunately, the tendency of the

present systems to discriminate against young scientists is

increasingly recognized as a serious problem and special

programmes to fund young investigators now exist both in the

US and Europe. In addition, the recently launched European

Research Council(5) initiative, to pick truly imaginative projects,

is a particularly hopeful development that should serve as a

(partial) corrective of the tendency of the granting systems to

favour conservative strategies. Yet, these programmes are

only a start. Nor, it has to be said, are the funding mechanisms

alone in stifling initiative and working against the chances and

hopes of young scientists. Those countries with sclerotic

university systems and in which much of the research is carried

out in universities—Italy is a prime example—are in serious

danger of sacrificing their scientific futures through inertia and

the strength of vested interests. In those countries, the reform

of the funding system has to go hand-in-hand with larger

changes in the higher educational systems as a whole.

There are no simple or easy solutions to these problems

with the major funding mechanisms. Yet, they only get worse

the greater the imbalance between resources and the

numbers of people competing for them, as the present woeful

grant awardee rates in the N.I.H. extramural system (<10%)

attest. Indeed, when the competition is sufficiently severe, the

relative disadvantage of young scientists is diminished: the

system becomes more like a lottery for everyone, with

the great majority ending up losing, regardless of experience

or merit. These problems are widely recognized but they have

not yet generated the sense of urgency, on the part of the

various scientific establishments, to begin to correct them. A

first step would be acknowledging their existence and serious-

ness. More money for basic research would certainly help but

the fundamentals of the system, and the hidden premises,

need searching re-examination. There is no question that peer

review should continue to be a central element of any grant-

awarding system. The question is how to make the various

peer-review systems fairer and more conducive to the

fostering of scientific creativity and discovery.

In principle, judging the relative abilities of individuals for

new positions should be far easier than judging the likely

success of work that has not yet been done. In the former, one

is working with a lot of information, the complete publication

records for all the competing researchers. Yet, it turns out

that the question of criteria of who is the ‘‘best’’ is just as

problematic. On short time scales, it is not always apparent

who has done the most creative and important research.

Indeed, sometimes work that is regarded as unsound, or even

mad, proves to be brilliant and ground-breaking. Howard

Temin’s experiments in the mid-1960s that indicated that there

were some RNA viruses that could ‘‘hide’’ in DNA genomes is

an example; the concept of retrotransposition did not exist

outside his experiments and his early work was largely

ridiculed at the time.

In general, the temptation on the part of the bodies charged

with deciding academic promotions is always to look for a

simple, single metric that will measure ‘‘quality’’. The attraction

of a such a metric is its ostensible objectivity, such that the

losers in these competitions cannot claim that they lost

because of personal bias on the part of the selectors. Thus,

in litigious times, the use of a single metric helps provide legal

protection for the governing bodies that make these decisions.

In the 1960s and 1970s, the single metric was often the

number of publications. Over time, the focus came to be on the

number of publications in the agreed upon ‘‘good’’ journals.

And in the past 10 years or so, the definition of what makes for a

‘‘good’’ journal has been deemed to be that of high impact

factor. The problems with using journal impact factors as a

measure of quality are, by now, well understood.(6) To state the

most obvious, someone doing brilliant research on a non-

fashionable subject is far less likely to be cited often, let alone

get a chance to publish in one of the top impact factor journals,

than someone doing fairly routine research on a major disease

condition, who turns up a new fact about that condition. The

relative difference in citation frequency between two such

individuals would hardly reflect the difference in their scientific

abilities. Furthermore, an excessive reliance on the criterion of

publication in high impact factor journals as a guide to hiring is

tantamount to outsourcing such decisions to the editors of

those journals. Given all the hazards of chance and subjectivity

that the publication process entails (see next month’s edito-

rial), such outsourcing amounts to passing the buck. Fur-

thermore, where an individual’s name is one of several authors

in such publications, it is not always clear how much he/she

can take credit for each individual paper.

But the problem of evaluating accurately scientific ability

goes well beyond the difficulties involved in the use of impact

Editorial

924 BioEssays 30.10

factors. Most generally, there is a fundamental flaw in looking

for a unitary metric to measure the degree of quality of

scientists. A well-established theorem in mathematics is that

one cannot rank members of a multidimensional set in a

unique way that maintains ‘‘order’’, namely preserves ‘‘neigh-

bourhoods’’ of these elements. (More formally, there is no

unique—one-to-one—order-conserving mapping of a multi-

dimensional set onto non-negative real numbers.) You can, of

course, do rankings of the values of each component

dimension for the different elements and then do some kind

of averaging for the rankings of the different components. But

in cases where the measurement of each property has a

subjective or hard-to-measure element, much information will

inevitably be lost and the final result will be a spurious

objectivity. Furthermore, using metrics of output as assays of

intrinsic qualities is always problematical. An example is the

use of IQ tests to measure the highly multi-dimensionsal

property of intelligence.(7) Comparably, the desirable abilities

of research scientists—which determine the quality of the

work produced—includes such disparate qualities as creative

insight, good analytical ability (often including but not restricted

to good mathematical ability), a sense of what constitutes a

really good control or a falsifiable hypothesis, good organiza-

tional skills for carrying out research programmes, team

management abilities, and the capacity to motivate and

encourage students and post-docs. Finally, different styles of

work can generate different kinds of valuable work. Some

scientists show their worth through in-depth and exhaustive

explorations of areas, revealing whole new landscapes of

particular subjects, while others publish less but show an

imaginative spark that helps put old problems in entirely new

perspectives. Still others have a knack for recruiting and

training gifted young investigators.

Given the complexity of evaluating the worth of individuals,

there can never be either a one-size fits all system of

evaluation or, for that matter, a set of perfect systems. There

will always be inequities and injustices in evaluation. Never-

theless, a move away from reliance on simple metrics of worth

is imperative. Academic committees that make decisions on

appointments need to be braver in admitting that the process is

inherently complex and, to a degree, subjective. Yet, when

the decisions are truly collective and based on thorough

discussion of all the factors being weighed, they can be far

sounder than decisions based on simple (and inadequate)

metrics of worth. Those who lose out in hiring decisions will

need to accept that such bodies have the right to make

decisions in this manner.

Nevertheless, it will certainly remain the case that scientists

will be—and should be—evaluated in large part on their

publication records. This consideration, however, immediately

brings one to the problems inherent in the editorial judgments

of scientific journals. We will turn to that topic in the final

editorial in this short series, which will appear in our next

and last issue for 2008, the November–December double

issue.

Adam S. Wilkins is finishing his term as Editor of

BioEssays. He can now be reached c/o Clare Hall, the

University of Cambridge, Herschel Road, Cambridge CB3

9AL, UK.

References1. Lavelle S, D’Ari D. 1998. The new scientific spirit. BioEssays 18:603–605.

2. Medina MA. 2006. The pursuit of creativity in biology. BioEssays 28:1151–

1152.

3. Brent R. 2000. Genomic biology. Cell 100:169–183.

4. Bassett DE, Eisen MB, Boguski MS. 1999. Gene expression informatics–

it’s all in your mine. Nature Genetics Supplement 21:51–55.

5. Editorial. 2008. Starting well in Europe. Nature Genetics 40:485.

6. Lawrence PA. 2003. The politics of publication. Nature 422:259–261.

7. Wilkins A. 2008. Dr Watson’s woeful words– and two missed oppor-

tunities. BioEssays 30:99–101.

DOI 10.1002/bies.20835

Published online in Wiley InterScience (www.interscience.wiley.com).

Editorial

BioEssays 30.10 925