Citation counts and theResearch Assessment

Exercise VArchaeology and the 2001 RAE

Michael Norris and Charles OppenheimDepartment of Information Science Loughborough University

Loughborough UK

Keywords Archaeology Research Libraries United Kingdom

Abstract A citation study of the 692 staff that makes up unit of assessment 58 (archaeology) inthe 2001 UK Research Assessment Exercise (RAE) was undertaken Unlike earlier studies whichwere obliged to make assumptions on who and what had been submitted for assessment thesewere for the regrst time available from the RAE Web site This study therefore used the speciregcsubmission details of authors and their publications Using the Spearman rank-order correlationcoefregcient all results showed high statistically signiregcant correlation between the RAE result andcitation counts The results were signiregcant at 001 per cent The regndings conregrm earlier studiesGiven the comparative cost and ease of citation analysis it is recommended that correctly appliedit should be the initial tool of assessment for the RAE Panel members would then exercise theirjudgement and skill to conregrm regnal rankings

IntroductionIn this paper we describe research on the correlation between citation countsand an ofregcial 2001 assessment of research excellence in UK universityarchaeology departments To assess the research performance of UKuniversities the UK Higher Education Funding Councils carry out a periodicResearch Assessment Exercise (RAE) The RAE is concerned with making aqualitative judgement of the research output of those university departmentswho submit themselves to the procedure As part of this assessment academic(and in some cases industrial) peers carry out a review of the published work ofsubmitting departments over a given period of time The departments are alsoassessed by other criteria including the number of higher degree students theyhave the amount of research income they have received and a generalcommentary on their current and future research programmes Universitydepartments are then ranked and consequently funded by central governmentfor their research activities based upon the score they achieve

The RAE is carried out every regve years or so and is viewed by many as anexpensive and contentious process Alternatives are sometimes suggested that

The Emerald Research Register for this journal is available at The current issue and full text archive of this journal is available at

httpwwwemeraldinsightcomresearchregister httpwwwemeraldinsightcom0022-0418htm

q Michael Norris and Charles OppenheimPart IV in this series can be found in Holmes and Oppenheim (2001)

Archaeology andthe 2001 RAE

709

Journal of DocumentationVol 59 No 6 2003

pp 709-730MCB UP Limited

0022-0418

are claimed to be both cheaper and less divisive One such alternative is citationanalysis

Generally the more citations an author receives the more important thatauthorrsquos work is assumed to be (Baird and Oppenheim 1994) Previousresearch (see below) has shown that there is a strong correlation between theranked scores obtained by the RAE and the ranked scores found by countingthe citations received by authors in their research writings Given theconsiderable amounts of money involved in funding research in highereducation the process of allocating funds to any research institution is likely tobe contentious The RAE has been the mechanism since 1986 by which theresearch funding apportionment has been resolved The RAE process itself hasgenerated a large amount of interest It has itself become ironically a subject ofresearch and a signiregcant amount of material has been published on its effectsand the possible alternatives that may be used in its place Henkel (1999)provides a good overview of the background and effect of the RAE onuniversities Martin (1996) gives an assessment of multiple performanceindicators that are available to measure research output Among the alternativeperformance indicators that can be used to measure research output is citationanalysis This could be used to assess an academic department in terms ofcounting the number of citations that its staff members have received for theirpublished work Such a set of statistics from each department could then beused as an alternative funding hierarchy (Holmes and Oppenheim 2001)

The RAE affects not only UK higher education institutions but also thepeople involved in its management and those academics whose research outputis critically judged The effect of the RAE extends well beyond the boundariesof academia and government The scholarly publishing sector is alsonoticeably affected by it Similarly those deciding where to place researchfunding and those simply looking to regnd the best place to do research will beinmacruenced by RAE results At a time of growing international competition thecost of basic research rising and the continuing need to demonstrate theusefulness of university research the regrst RAE was instituted in 1986Including the 2001 RAE there have been regve such assessments

The 2001 RAEThe Guide to the 2001 Research Assessment Exercise (Publications 2002) givesan appreciation of the scale history and importance of the 2001 RAE TheGuide states

The purpose of the RAE is not just to enable funding to be allocated selectively but also topromote high quality research in higher education institutions conducting the best researchreceive the largest proportion of the grant

Funding decisions are about promoting high quality research and improvingthe productivity and effectiveness of that research A document from HEFCE

JD596

710

reports the regndings of a consultation process on research (HEFCE 2001a) Itclaimed an overall satisfaction level of 98 per cent with the question

Should there continue to be a research assessment process based on peer review building onthe foundations of the RAE

The respondents were made up of 102 English universities and colleges 32subject associations and 30 other organisations All were in broad agreementwith the overall thrust of the funding strategy The closing date forconsultation was however December 2000 a year prior to the announcementof the regnal RAE results

Rogers (at the time responsible for the RAE) disagrees with the manycriticisms of the RAE He argues for example that there is no wholesalemovement of staff nor is interdisciplinary research discriminated against andthat the diversity of published forms that are acceptable as submissions is wideand all are viewed as being equally valid (Rogers 2000)

For the individual the effects of the RAE may be extreme Individualsdesignated as ordfresearch inactiveordm may regnd that they have a drastic loss ofstatus and are relegated to higher administrative and teaching loads Theindividuals may be viewed as liabilities ordfResearch activeordm staff are likely toregnd themselves being managed and directed in an atmosphere of ordfcollectiveinterest rather than individual ambition and structures and policies have beendeveloped accordinglyordm (Henkel 1999)

Scholarly publication and the RAESeveral surveys and much anecdotal evidence suggest that the RAE and itsdemands on academics have affected the scholarly publishing industry TheRAE ofregcially does not favour any one particular form of material submitted toit for evaluation yet clearly the high impact factor journal article will have thestamp of peer approval and a general recognition of quality already attached toit (Naylor 2001) Campbell (1999) examined the affect that the RAE has had onpublishing with particular regard to academic lawyers The clear view emergesthat journal articles are important and the actual journal in which articlesappear will lend authority to the work published It is widely (but probablyincorrectly) believed that the RAE assessment panel will view such work morefavourably than more obscure publications This conclusion is not shared byBence and Oppenheim (2001) who examined the regeld of business andmanagement studies The subject area attracted in the 1996 RAE 5494 articlesto 1275 different journal titles They concluded that the academic author in thisregeld attaches importance to the journal in which their work appears but thatthere is a much more diverse range of journals and this makes the task ofderegning a core set of high impact factor journals much more difregcult

In the humanities there is a noticeable preference for publishing academicresearch in monograph form unlike scientiregc research Meadows suggests thisdifference may be seen as the ordfhardordm quantitative nature of scientiregc subjects

Archaeology andthe 2001 RAE

711

being suited to journal publication and the more discursive ordfsoftordm humanitiesresearch lending itself to monographic publication (Meadows 1998) In thepublishing industry itself Mynott of the Cambridge University Press clearlyidentireges the pressures placed upon him Some of the normal editing andrevision processes were foreshortened and there was a ordfcompetitionordm with someworks which would only be offered if they could be published within the giventime-scale (Mynott 1999) He noted that his Press published 122 more titles inthe regrst six months of 1996 compared to the previous year

Walford (2000) also found a sharp increase in the number of contributions toscholarly journals in the run up to a RAE He suggested that whilst someordfsalami-slicingordm took place there was also a proliferation of run-of-the-millpapers which though weeded out by the peer review process put strain on therefereeing process

A new discovery in medicine confers status to the published work and theauthor therefore making it excellent RAE material The humanities scholarrsquoswork on the other hand has to be judged much more regnely on the contributionof its scholarship to the body of existing knowledge and this Bernard arguesmakes judging the quality of the work much more difregcult (Bernard 2000)Complexities start to arise in the humanities when scholars begin to interprettheir research material in controversial ways Whilst scientiregc discoveries canbe tested and experiments replicated the opinion of one historian againstanother cannot be tested to verify its truth Thus the quality of some particularresearch as well as its suitability for RAE submission could be questioned Itcould be argued that archaeology is in a similar position to historical researchsince although it is based on clear evidence (archaeological dig regndingshistorical records) much of the work is speculative and based on the judgementand interpretation of the researcher Such possible ambiguities in assessing thequality of humanities research clearly allow for the possibility of inaccuraciesin awarding RAE ratings

Citation analysisCitation analysis is based on counts of the number of citations that an article ormonograph has received in a particular period Citation analysis was not easilyundertaken until the launch of Science Citation Index Social Sciences CitationIndex and Art and Humanities Citation Index Garregeldrsquos seminal work oncitation analysis although 25 years old provides an authoritative historicaland conceptual understanding of the topic and its many applications (Garregeld1979) A more up to date overview can be obtained from (Cronin and BarskyAtkins 2000)

The use of citation studies in place of or as a supplement to the RAEassessment has been considered extensively by Oppenheim and his co-workersThey have published a number of articles demonstrating that the correlationbetween the overall RAE ranking of departments and the collective scores

JD596

712

found by counting the citations received by the authors from thosedepartments is statistically signiregcant (Holmes and Oppenheim 2001Oppenheim 1995 1996 1997) He argues that citation analysis costs afraction of the RAE and is much less intrusive than the RAE This has beenfollowed up with related suggestions for improving the RAE in the future(Harnad et al 2003)

Other researchers have found similar strong correlations between citationcounts and RAE scores Seng and Willett (1995) examined the correlationbetween the publications of UK library schools between the years 1989-1990and the 1992 RAE result More recent work to consider citation analysis as analternative or supplement to the RAE was carried out by Smith and Eysenckwho examined the citation record of psychologists for the year 1998 citationswere checked for their probity to eliminate self-citations and correct anymisnamed individuals The results were compared to the 1996 and 2001 RAEsA 09 and 085 correlation was found for all-author citations respectively (Smithand Eysenck 2002) These statistically signiregcant results they argue areample supporting evidence for the use of citation analysis as an effectiveevaluation tool of research quality Sarwar studied UK civil engineeringdepartments to establish if there was a correlation between the 1996 RAE and acitation analysis of those departments who had achieved a rating of 4 or aboveComparing the citation record of the authors in those departments for the years1995-1997 and the departmentrsquos 1996 RAE rating he (Sarwar 2000) conregrmeda signiregcant correlation between the two sets of results

In one of his papers Oppenheim (1997) examined the subject area ofarchaeology He demonstrated that archaeology produced the highestcorrelation of the three subject areas studied

Citation analysis has many critics and some have focussed their criticism onthe idea that citation analysis could be used as an input into the RAE process(Johnson 2001 Warner 2000) or on the bias introduced by self-citation (Seglen1992) The response of proponents of the method is the inherent robustness ofcitation analysis as a valid tool of measurement since all the experiments so farhave shown strong statistically signiregcant correlations with RAE results Thequestion of self-citation is dealt with satisfactorily by (Snyder and Bonzi 1998)

While Oppenheim is a strong proponent of the use of citation analysis toestablish a funding hierarchy he has concluded (Baird and Oppenheim 1994)that

[T]here is not and never can be one single measure of the value of information that will beuniversally acceptable However there are a number of measures that might in combinationlead to some sort of index of the value of a piece of information an individualrsquos researchcontribution or a collection of information

This collection view of ordfmeasuresordm which together could be used to provide amore balanced and objective indicator of research quality is one shared byMartin (1996) who analysed the possible measures that could be used Martin

Archaeology andthe 2001 RAE

713

crucially notes that citation counts are an indicator of impact of the authortheir past reputation and the organisation in which they work He concludesthat high citation counts may not always indicate quality or importance butmay be for controversy fraud or a simply incorrect work Studies underway inthis department[1] are examining citation counts for controversial articlesMartin also makes the important observation that the more indicators ofresearch you have the more difregcult it is for individuals to manipulate them totheir advantage

Van Raan (1996) describes an assessment exercise undertaken in theNetherlands where peer review is used jointly with advanced bibliometrictechniques to evaluate research The majority of the reviewers come fromoutside the Netherlands and they undertake a quality assessment of theresearch output from the universities In addition a scrupulous bibliometricassessment process designed to remove the many anomalies found withincitation counting such as self-citing incorrect counts journal coverage etcwas carried out The result is a ordfcleanedordm citation count Van Raan (1996)concluded that

We showed that the resulting indicators are useful they address signiregcant concepts in theframework of evaluation and therefore can be considered an indispensable element next topeer review in research performance assessment procedures

This approach validates the use of citation analysis as a sound measure ofresearch performance when used along side peer review

The research described in this paper was aimed at establishing whether thewell-established correlations between RAE performance and citation countscontinue to apply to the 2001 RAE and also whether the ability to moreprecisely identify who had been returned for the 2001 RAE leads to a bettercorrelation than simply carrying out an analysis based on all staff in adepartment Archaeology was chosen as a subject for study for two reasonsregrstly it had been studied before so comparisons with earlier results could bemade and secondly it is not such a large discipline so the research could becompleted in a reasonable time frame

Methods employedA citation analysis was undertaken which counted the citations received bythose academics in archaeology departments submitted for peer review in the2001 RAE In earlier assessments it was not possible to identify the speciregcacademics returned by any particular department nor the publications theysubmitted for assessment This is now however possible for the 2001 RAEThus the methods adopted differ slightly from previous similar studies It isworth stressing that each academicrsquos submission is recorded on the RAE Website and is therefore a public document Therefore no permission was neededfrom this individuals to include them in our analysis or to present resultsrelating to them (see Table I)

JD596

714

Archaeology was chosen following an assessment carried out to regnd a subjectthat was both sufregciently discrete to allow its careful measurement and was ofthe right size and scale that it could be comfortably managed Archaeology isdenoted in the 2001 RAE as unit of assessment (UoA) 58 A total of 26university departments returned submissions into UoA 58 They were rankedin the 2001 RAE with scores from 3b to 5 for their research excellence Asnoted above the report of the 2001 RAE (HEFCE 2001b) for the regrst time madeavailable details of those academics that had been submitted for peer review bydepartments All the names of the academics who were submitted for UoA 58were thus obtained Each academic was allowed to submit up to four itemswhich they had produced during the assessment period We noted details of all682 staff including name institution and the details of the items submitted forthe RAE ISI Web of Science was used to carry out the citation counting Thesearches were conregned to the Art and Humanities Citation Index (AHCI) sinceit covers the regeld of archaeology The cited reference searches were carried outbetween August and September 2002 The searches were conregned to the RAEassessment period ie 1994 to 2000 and looked for citations to material thatwas published during this interval by the authors concerned noting thenumber of citations this material had received Named authors were normallyentered into the search as given In the case of hyphenated or compound namesstandard ISI practice of contraction was used For example the name Van DerLeeum was entered as Vanderleeum Some difregculty was encountered withauthors who had several initials Where the search returned a signiregcantnumber of hits for all the initials it was considered that the author routinelyused their fully initialled name in authorship Where a nil or very low returnwas encountered for a multiple initialled author further searches were carriedout by progressively reducing the number of initials until an appropriate resultwas yielded The cited references were regrst scrutinised to ensure that they fellwithin the RAE assessment period ie 1994 to 2000 Each cited reference wasthen examined to ensure it was the right subject and that it could be attributedto the author concerned Use was made of the authorrsquos submission to the RAE

Author Citation count Afregliation

PJ Reimer 565 BelfastP Forster 471 CambridgeREM Hedges 268 OxfordEJ Rhodes 127 OxfordMGL Baillie 120 BelfastMB Roberts 94 UCLJ Elsner 94 OxfordJ Pollard 92 BradfordFG McCormac 81 BelfastIR Hodder 81 Cambridge

Table IThe top ten most cited

authors and theirafregliation

Archaeology andthe 2001 RAE

715

which indicated his or her subject area Where ambiguity was apparent inauthorship records were marked and the abstract and cited referencesexamined to verify that the correct author and subject had been identiregedSimilarly ambiguous records which had direct electronic links to an abstractand cited references were also scrutinised

A small number of cited works were ignored ie those listed withoutpublication dates unpublished works and those ordfin pressordm Where ambiguitywas still apparent in attributing authorship a search was made at the authorrsquosdepartmental web site and access to their publication record clariregedresponsibility for the cited work

Each authorrsquos citation count was then calculated Anomalous records wererechecked as necessary The 682 named authors were transcribed onto aspreadsheet into their respective departments and citation counts attributableto them were likewise entered The authors were differentiated to show whichof them had been submitted for peer review Of the 682 authors listed 73 hadnot been submitted for review From this general data a number of speciregccitation statistics by department including averages were calculated to allowsubsequent correlation calculations to be made

In order to carry out a Spearman rank-order correlation calculation (themethod used in all these studies) we ranked the archaeology departments bytheir RAE scores and their different citation statistics The 26 archaeologydepartments were listed in ascending order of their 2001 RAE ratings Eachdepartment was then assigned a separate ascending numerical rank Wherethere were two or more equal values in the departmental rating an average ofthe numerical ranking was applied For example Reading Oxford andCambridge each had a 5 (the highest possible) rating and so each were given a2 ranking

The citations from the authors were collated into their respective universitydepartments and totalled From these collations a set of four different tableswere produced each containing the 26 departments their RAE rating and thetotal number of citations Two of the tables were differentiated by a calculatedaverage citation rate dependent on the status of the cited author The other twotables were differentiated by the total citations received by each departmentbut again dependent on the status of the cited authors In both sets of tables thecriteria for differentiation were submitted and non-submitted authors and theircitation scores To again obtain a Spearman rank-order correlation eachdepartment in each table was then assigned a numerical rank and if necessaryan average numerical rank applied

Four sets of calculations were then performed

(1) a ranking for all citations received by each department irrespective ofwhether the author had been submitted or not

(2) a ranking by the total citations received by each department countingsubmitted authors only

JD596

716

(3) a ranking which shows the average citation rate for each departmentusing all departmental staff irrespective of whether the author had beensubmitted or not and

(4) a ranking which shows the average citation rate by department usingsubmitted departmental staff only

Issues arising out of the method adoptedCitation counts can however well carried out lead to erroneous counts Themethods adopted in this study go a long way to minimise such erroneouscounts A number of commonly voiced criticisms of citation counting (Liu1993 MacRoberts and MacRoberts 1989) were potentially present in this studyand are worthy of comment The general problem of mistakenly countingauthors with the same surname and initials but in different disciplines wasunderstood and careful inspection was exercised to limit such miscounting

This process was considerably helped by the RAE submissions whichidentireged the authorsrsquo interests and some of their publications Examination ofthe bibliographical record of the citing article for its subject and its fellowcitations helped minimise potential error Unquestionably however judgementwas required Where it was very difregcult to assess the validity of a particularcitation the record was included Such cases were infrequent and are not likelyto distort the total counts or their relative rankings In any case it is thecomparative ranking of the results which is being sought rather than anabsolutely accurate citation count

Where formerly only regrst named authors were cited now second andsubsequent authors are also cited in the ISIrsquos indexes thus the comment thatco-authors were uncited within the index and hence uncounted no longerapplies This has improved the citation count achieved and removed asigniregcant objection However Lange suggests that some caution needs to beexercised in assuming that a regrst and subsequent cited author search can bedeemed to be absolutely exact (Lange 2001) Self-citation has long beencriticised in citation analysis such practice it has been suggested leads todistortion and bias in conclusions drawn from counts The evidence andstudies conducted so far however show that this practice has little effect on theresults of citation analysis Snyder and Bonzi (1998) in a study whichexamined the patterns of self-citation in six disciplines clearly showed itsprevalence but were able to demonstrate that the patterns of self-citation arevery similar between disciplines and that the lowest self-citation rate of 3 percent was found in the humanities including presumably archaeology

An author making a submission to the RAE may choose material that hasbeen published at any time during the qualifying period This may make theitems submitted potentially ordfoldordm or very ordfnewordm material Counting thecitations to a population of ordfnewordm articles against ordfoldordm could easily give askewed result in favour of the old articles given that more citations are likely to

Archaeology andthe 2001 RAE

717

accrue to the older material We did not examine this in detail but a simpleoverview indicated that the pattern of age of items submitted did not differmuch between the departments examined

Humanities scholars tend to publish more monographs than those in otherdisciplines A random sample of 40 submitted authors examined here credited atleast 32 of them with either having published a monograph or chapter in onesubmitted for peer review In many cases they had published more than onemonograph or chapter in the assessment period Cronin et al (1997) have examinedthe differing author citation patterns when comparing journals and monographs inthe regeld of sociology They suggest that journals and monographs attract twoquite distinct populations of authors amongst the very highly cited this is certainlya concern and it can be assumed that insufregcient citations totals will result fromthis However we had no evidence that this affected one department more thananother and it is worth reiterating that it is the comparative ranking of citationcounts that is studied here rather than the precise count

A criticism may be levelled at the scope and coverage of the AHCI in termsof its bias towards USAEuropean publications as a distorting feature whenmaking citation counts (Braun et al 2000) Therefore where an author hasmade contributions to non-USAEuropean literature there is less likelihood ofhis or her work being noted V Nanda of University College London did notreceive any citations for his work which has been published in Asia LikewisePM Carroll of the University of Shefregeld did not receive any citations to hisworks when published in German While this is not conclusive evidence as toany particular bias the work of Braun et al (2000) clearly shows a strongpreference for a USAEuropean dominance of journal titles and publisherswithin Web of Science Examining the archaeology journal list in the AHCIconregrms a preference for USAEuropean journals Table II shows thedistribution of journal title by place of publication

This shows there is a deregnite bias toward USAEuropean journals so thelikelihood of regnding the cited work of UK archaeologists is considerable plusmnunless they publish elsewhere This bias in practice adds credibility to thisparticular study This it is contended will help support the results obtainedand the conclusions drawn

Place of publication Number of journals

Denmark 1France 4Germany 3Israel 1South Africa 1UK 10USA 17Total 37

Table IIDistribution of journalsindexed by ISI Web ofScience for Archaeology

JD596

718

ResultsThe results have been summarised in a number of Tables These Tables giveseveral views of the data notably by a ranking of the RAE score obtained byeach archaeology department and several different citation statistics Allcorrelations found were signiregcant thus conregrming earlier studies We alsofound that correlations for just staff returned for assessment is even higherthereby arguing that this is an even better predictor of RAE results

Table III gives an overall view of the RAE score obtained by eachdepartment and a comparative ranking by the four basic citation statistics thathave been calculated for each of them Table IV describes the core statisticscollected and of the range of values obtained for each of the departments

University name

RAEresult2001

Ranking bysubmitted

staffcitations

Rankingby all

citations

Ranking byaverage

citations forsubmitted staff

only

Ranking byaverage

citations for allstaff members

Trinity College Carmarthen 3b 25 23 20 24King Alfredrsquos College

Winchester 3a 24 24 26 25University of Wales College

Newport 3a 23 25 25 26University of Edinburgh 3a 21 21 22 21Bournemouth University 3a 18 18 21 20University of Newcastle 3a 17 17 17 17University of York 3a 15 15 18 18Birkbeck College (one staff

member only) 4 26 26 12 13University of Bristol 4 22 22 24 23University of Nottingham 4 19 19 15 15University of Birmingham 4 16 16 23 22University of Glasgow 4 14 14 19 19University of Wales Lampeter 4 13 11 13 9University of Exeter 5 20 20 16 16University of Liverpool 5 12 13 11 11Cardiff University 5 11 12 7 8University of Leicester 5 9 9 14 14University of Southampton 5 8 8 8 7University of Bradford 5 7 6 5 4University of Durham 5 6 7 9 10University of Shefregeld 5 5 5 4 3University College London 5 4 4 10 12The Queens University of

Belfast 5 2 2 1 1University of Reading 5 10 10 6 6University of Cambridge 5 3 3 3 5University of Oxford 5 1 1 1 2

Table IIIComparative rankingwith the RAE rating

result 2001 staffcitation counts and

citation count averagesfor UoA 58

(archaeology)

Archaeology andthe 2001 RAE

719

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Bir

kbec

kC

olle

ge

60

61

06

000

006

00U

niv

ersi

tyof

Bir

min

gham

660

6630

22

200

002

06B

ourn

em

outh

Univ

ersi

ty47

047

160

294

000

294

Univ

ersi

tyof

Bra

dfo

rd23

878

316

193

125

326

00

143

6U

niv

ersi

tyof

Bri

stol

370

3718

02

060

002

06U

niv

ersi

tyof

Cam

bri

dge

811

8889

943

2318

86

383

136

2U

niv

ersi

tyof

Durh

am25

22

254

363

700

067

651

Univ

ersi

tyof

Exet

er41

041

110

373

000

373

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

115

167

21

572

501

78U

niv

ersi

tyof

Lei

cest

er16

63

169

352

474

150

457

Univ

ersi

tyof

Liv

erpoo

l11

10

111

180

617

000

617

Univ

ersi

tyof

New

cast

le51

657

152

340

300

335

Univ

ersi

tyof

Not

tingham

450

4512

03

750

003

75U

niv

ersi

tyof

Oxfo

rd1

192

761

268

5911

202

06

9118

11

Univ

ersi

tyof

Rea

din

g16

40

164

200

820

000

820

Univ

ersi

tyof

Shef

regel

d31

462

376

202

157

031

00

170

9U

niv

ersi

tyof

Sou

tham

pto

n22

423

247

303

747

767

748

Univ

ersi

tyC

olle

ge

Lon

don

550

955

988

46

252

256

08U

niv

ersi

tyof

Yor

k77

077

230

335

000

335

Univ

ersi

tyof

Edin

burg

h38

038

140

271

000

271

Univ

ersi

tyof

Gla

sgow

103

010

331

03

320

003

32

(continued

)

Table IVSummary results ofcitation counts staffnumbers and citationcount averages for UoA58 (archaeology)

JD596

720

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Tri

nit

yC

olle

ge

Car

mar

then

612

182

73

001

712

00C

ardif

fU

niv

ersi

ty14

30

143

191

753

000

715

Univ

ersi

tyof

Wal

esL

ampet

er10

653

159

195

558

106

06

63U

niv

ersi

tyof

Wal

esC

olle

ge

New

por

t14

014

80

175

000

175

The

Quee

ns

Univ

ersi

tyof

Bel

fast

977

698

315

365

13

200

546

1G

rand

tota

ls5

790

423

621

360

973

951

579

911

Table IV

Archaeology andthe 2001 RAE

721

Calculations using the core data and the RAE score give the four measures ofcorrelation required Two of the measures deal with total citation counts andthe other two are concerned with average citations per member of staff The2001 RAE rating achieved by each archaeology department and theirrespective citation statistics are shown in Table V Table V has been sortedwithin each RAE rating by the number of submitted staff citations receivedgiving a snapshot view of the range of citations received and also how thiscompares between ratings To compare the 1996 and 2001 RAE ratingsTable VI shows the scores obtained in both assessments and the movementbetween the two periods

Table VII summarises the results obtained for the Spearman rank-ordercorrelation analyses for the four differing citation measures that were usedThe correlation scores have been rounded to two signiregcant reggures Each ofthese correlation scores is highly signiregcant ( p 001)

The results obtained by Oppenheim in his earlier study on archaeology areshown in Table VIII for comparison They are consistent with the currentresults

Analysis of results682 staff members were checked for their citation counts Between them theyaccumulated 6213 citations although 262 of the authors did not receive anycitations at all in the assessment period The individual citation count rangedfrom 0 to 565 with an overall average of 911 Of the cited authors just over 50per cent of them have a citation level of between one and six citations each Theexceptional score of 565 citations obtained by PJ Reimer can be attributed tojust three articles These were articles dealing with radiocarbon dating linkedto dendrochronolgy and were as follows

Quaternary Science Reviews 1996 15(7) 655 (10 citations)

Radiocarbon 1998 40(3) 1041 (491 citations)

Radiocarbon 1998 40(3) 1127 (64 citations)

Reimerrsquos nearest rival was P Forster from Cambridge University with 471citations from four articles They deal with DNA studies tracing the migrationof humans from Africa

American Journal of Human Genetics 1996 59(4) 935 (108 citations)

American Journal of Human Genetics 1996 61(3) 691 (73 citations)

American Journal of Human Genetics 1996 59(1) 185 (164 citations)

Genetics 1995 141(2) 743 (126 citations)

These two authors account for one-sixth of all the citations counted and theirarticles appear in just four journals all of which are available in both print andelectronic form

JD596

722

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

reports the regndings of a consultation process on research (HEFCE 2001a) Itclaimed an overall satisfaction level of 98 per cent with the question

Should there continue to be a research assessment process based on peer review building onthe foundations of the RAE

The respondents were made up of 102 English universities and colleges 32subject associations and 30 other organisations All were in broad agreementwith the overall thrust of the funding strategy The closing date forconsultation was however December 2000 a year prior to the announcementof the regnal RAE results

Rogers (at the time responsible for the RAE) disagrees with the manycriticisms of the RAE He argues for example that there is no wholesalemovement of staff nor is interdisciplinary research discriminated against andthat the diversity of published forms that are acceptable as submissions is wideand all are viewed as being equally valid (Rogers 2000)

For the individual the effects of the RAE may be extreme Individualsdesignated as ordfresearch inactiveordm may regnd that they have a drastic loss ofstatus and are relegated to higher administrative and teaching loads Theindividuals may be viewed as liabilities ordfResearch activeordm staff are likely toregnd themselves being managed and directed in an atmosphere of ordfcollectiveinterest rather than individual ambition and structures and policies have beendeveloped accordinglyordm (Henkel 1999)

Scholarly publication and the RAESeveral surveys and much anecdotal evidence suggest that the RAE and itsdemands on academics have affected the scholarly publishing industry TheRAE ofregcially does not favour any one particular form of material submitted toit for evaluation yet clearly the high impact factor journal article will have thestamp of peer approval and a general recognition of quality already attached toit (Naylor 2001) Campbell (1999) examined the affect that the RAE has had onpublishing with particular regard to academic lawyers The clear view emergesthat journal articles are important and the actual journal in which articlesappear will lend authority to the work published It is widely (but probablyincorrectly) believed that the RAE assessment panel will view such work morefavourably than more obscure publications This conclusion is not shared byBence and Oppenheim (2001) who examined the regeld of business andmanagement studies The subject area attracted in the 1996 RAE 5494 articlesto 1275 different journal titles They concluded that the academic author in thisregeld attaches importance to the journal in which their work appears but thatthere is a much more diverse range of journals and this makes the task ofderegning a core set of high impact factor journals much more difregcult

In the humanities there is a noticeable preference for publishing academicresearch in monograph form unlike scientiregc research Meadows suggests thisdifference may be seen as the ordfhardordm quantitative nature of scientiregc subjects

Archaeology andthe 2001 RAE

711

being suited to journal publication and the more discursive ordfsoftordm humanitiesresearch lending itself to monographic publication (Meadows 1998) In thepublishing industry itself Mynott of the Cambridge University Press clearlyidentireges the pressures placed upon him Some of the normal editing andrevision processes were foreshortened and there was a ordfcompetitionordm with someworks which would only be offered if they could be published within the giventime-scale (Mynott 1999) He noted that his Press published 122 more titles inthe regrst six months of 1996 compared to the previous year

Walford (2000) also found a sharp increase in the number of contributions toscholarly journals in the run up to a RAE He suggested that whilst someordfsalami-slicingordm took place there was also a proliferation of run-of-the-millpapers which though weeded out by the peer review process put strain on therefereeing process

A new discovery in medicine confers status to the published work and theauthor therefore making it excellent RAE material The humanities scholarrsquoswork on the other hand has to be judged much more regnely on the contributionof its scholarship to the body of existing knowledge and this Bernard arguesmakes judging the quality of the work much more difregcult (Bernard 2000)Complexities start to arise in the humanities when scholars begin to interprettheir research material in controversial ways Whilst scientiregc discoveries canbe tested and experiments replicated the opinion of one historian againstanother cannot be tested to verify its truth Thus the quality of some particularresearch as well as its suitability for RAE submission could be questioned Itcould be argued that archaeology is in a similar position to historical researchsince although it is based on clear evidence (archaeological dig regndingshistorical records) much of the work is speculative and based on the judgementand interpretation of the researcher Such possible ambiguities in assessing thequality of humanities research clearly allow for the possibility of inaccuraciesin awarding RAE ratings

Citation analysisCitation analysis is based on counts of the number of citations that an article ormonograph has received in a particular period Citation analysis was not easilyundertaken until the launch of Science Citation Index Social Sciences CitationIndex and Art and Humanities Citation Index Garregeldrsquos seminal work oncitation analysis although 25 years old provides an authoritative historicaland conceptual understanding of the topic and its many applications (Garregeld1979) A more up to date overview can be obtained from (Cronin and BarskyAtkins 2000)

The use of citation studies in place of or as a supplement to the RAEassessment has been considered extensively by Oppenheim and his co-workersThey have published a number of articles demonstrating that the correlationbetween the overall RAE ranking of departments and the collective scores

JD596

712

found by counting the citations received by the authors from thosedepartments is statistically signiregcant (Holmes and Oppenheim 2001Oppenheim 1995 1996 1997) He argues that citation analysis costs afraction of the RAE and is much less intrusive than the RAE This has beenfollowed up with related suggestions for improving the RAE in the future(Harnad et al 2003)

Other researchers have found similar strong correlations between citationcounts and RAE scores Seng and Willett (1995) examined the correlationbetween the publications of UK library schools between the years 1989-1990and the 1992 RAE result More recent work to consider citation analysis as analternative or supplement to the RAE was carried out by Smith and Eysenckwho examined the citation record of psychologists for the year 1998 citationswere checked for their probity to eliminate self-citations and correct anymisnamed individuals The results were compared to the 1996 and 2001 RAEsA 09 and 085 correlation was found for all-author citations respectively (Smithand Eysenck 2002) These statistically signiregcant results they argue areample supporting evidence for the use of citation analysis as an effectiveevaluation tool of research quality Sarwar studied UK civil engineeringdepartments to establish if there was a correlation between the 1996 RAE and acitation analysis of those departments who had achieved a rating of 4 or aboveComparing the citation record of the authors in those departments for the years1995-1997 and the departmentrsquos 1996 RAE rating he (Sarwar 2000) conregrmeda signiregcant correlation between the two sets of results

In one of his papers Oppenheim (1997) examined the subject area ofarchaeology He demonstrated that archaeology produced the highestcorrelation of the three subject areas studied

Citation analysis has many critics and some have focussed their criticism onthe idea that citation analysis could be used as an input into the RAE process(Johnson 2001 Warner 2000) or on the bias introduced by self-citation (Seglen1992) The response of proponents of the method is the inherent robustness ofcitation analysis as a valid tool of measurement since all the experiments so farhave shown strong statistically signiregcant correlations with RAE results Thequestion of self-citation is dealt with satisfactorily by (Snyder and Bonzi 1998)

While Oppenheim is a strong proponent of the use of citation analysis toestablish a funding hierarchy he has concluded (Baird and Oppenheim 1994)that

[T]here is not and never can be one single measure of the value of information that will beuniversally acceptable However there are a number of measures that might in combinationlead to some sort of index of the value of a piece of information an individualrsquos researchcontribution or a collection of information

This collection view of ordfmeasuresordm which together could be used to provide amore balanced and objective indicator of research quality is one shared byMartin (1996) who analysed the possible measures that could be used Martin

Archaeology andthe 2001 RAE

713

crucially notes that citation counts are an indicator of impact of the authortheir past reputation and the organisation in which they work He concludesthat high citation counts may not always indicate quality or importance butmay be for controversy fraud or a simply incorrect work Studies underway inthis department[1] are examining citation counts for controversial articlesMartin also makes the important observation that the more indicators ofresearch you have the more difregcult it is for individuals to manipulate them totheir advantage

Van Raan (1996) describes an assessment exercise undertaken in theNetherlands where peer review is used jointly with advanced bibliometrictechniques to evaluate research The majority of the reviewers come fromoutside the Netherlands and they undertake a quality assessment of theresearch output from the universities In addition a scrupulous bibliometricassessment process designed to remove the many anomalies found withincitation counting such as self-citing incorrect counts journal coverage etcwas carried out The result is a ordfcleanedordm citation count Van Raan (1996)concluded that

We showed that the resulting indicators are useful they address signiregcant concepts in theframework of evaluation and therefore can be considered an indispensable element next topeer review in research performance assessment procedures

This approach validates the use of citation analysis as a sound measure ofresearch performance when used along side peer review

The research described in this paper was aimed at establishing whether thewell-established correlations between RAE performance and citation countscontinue to apply to the 2001 RAE and also whether the ability to moreprecisely identify who had been returned for the 2001 RAE leads to a bettercorrelation than simply carrying out an analysis based on all staff in adepartment Archaeology was chosen as a subject for study for two reasonsregrstly it had been studied before so comparisons with earlier results could bemade and secondly it is not such a large discipline so the research could becompleted in a reasonable time frame

Methods employedA citation analysis was undertaken which counted the citations received bythose academics in archaeology departments submitted for peer review in the2001 RAE In earlier assessments it was not possible to identify the speciregcacademics returned by any particular department nor the publications theysubmitted for assessment This is now however possible for the 2001 RAEThus the methods adopted differ slightly from previous similar studies It isworth stressing that each academicrsquos submission is recorded on the RAE Website and is therefore a public document Therefore no permission was neededfrom this individuals to include them in our analysis or to present resultsrelating to them (see Table I)

JD596

714

Archaeology was chosen following an assessment carried out to regnd a subjectthat was both sufregciently discrete to allow its careful measurement and was ofthe right size and scale that it could be comfortably managed Archaeology isdenoted in the 2001 RAE as unit of assessment (UoA) 58 A total of 26university departments returned submissions into UoA 58 They were rankedin the 2001 RAE with scores from 3b to 5 for their research excellence Asnoted above the report of the 2001 RAE (HEFCE 2001b) for the regrst time madeavailable details of those academics that had been submitted for peer review bydepartments All the names of the academics who were submitted for UoA 58were thus obtained Each academic was allowed to submit up to four itemswhich they had produced during the assessment period We noted details of all682 staff including name institution and the details of the items submitted forthe RAE ISI Web of Science was used to carry out the citation counting Thesearches were conregned to the Art and Humanities Citation Index (AHCI) sinceit covers the regeld of archaeology The cited reference searches were carried outbetween August and September 2002 The searches were conregned to the RAEassessment period ie 1994 to 2000 and looked for citations to material thatwas published during this interval by the authors concerned noting thenumber of citations this material had received Named authors were normallyentered into the search as given In the case of hyphenated or compound namesstandard ISI practice of contraction was used For example the name Van DerLeeum was entered as Vanderleeum Some difregculty was encountered withauthors who had several initials Where the search returned a signiregcantnumber of hits for all the initials it was considered that the author routinelyused their fully initialled name in authorship Where a nil or very low returnwas encountered for a multiple initialled author further searches were carriedout by progressively reducing the number of initials until an appropriate resultwas yielded The cited references were regrst scrutinised to ensure that they fellwithin the RAE assessment period ie 1994 to 2000 Each cited reference wasthen examined to ensure it was the right subject and that it could be attributedto the author concerned Use was made of the authorrsquos submission to the RAE

Author Citation count Afregliation

PJ Reimer 565 BelfastP Forster 471 CambridgeREM Hedges 268 OxfordEJ Rhodes 127 OxfordMGL Baillie 120 BelfastMB Roberts 94 UCLJ Elsner 94 OxfordJ Pollard 92 BradfordFG McCormac 81 BelfastIR Hodder 81 Cambridge

Table IThe top ten most cited

authors and theirafregliation

Archaeology andthe 2001 RAE

715

which indicated his or her subject area Where ambiguity was apparent inauthorship records were marked and the abstract and cited referencesexamined to verify that the correct author and subject had been identiregedSimilarly ambiguous records which had direct electronic links to an abstractand cited references were also scrutinised

A small number of cited works were ignored ie those listed withoutpublication dates unpublished works and those ordfin pressordm Where ambiguitywas still apparent in attributing authorship a search was made at the authorrsquosdepartmental web site and access to their publication record clariregedresponsibility for the cited work

Each authorrsquos citation count was then calculated Anomalous records wererechecked as necessary The 682 named authors were transcribed onto aspreadsheet into their respective departments and citation counts attributableto them were likewise entered The authors were differentiated to show whichof them had been submitted for peer review Of the 682 authors listed 73 hadnot been submitted for review From this general data a number of speciregccitation statistics by department including averages were calculated to allowsubsequent correlation calculations to be made

In order to carry out a Spearman rank-order correlation calculation (themethod used in all these studies) we ranked the archaeology departments bytheir RAE scores and their different citation statistics The 26 archaeologydepartments were listed in ascending order of their 2001 RAE ratings Eachdepartment was then assigned a separate ascending numerical rank Wherethere were two or more equal values in the departmental rating an average ofthe numerical ranking was applied For example Reading Oxford andCambridge each had a 5 (the highest possible) rating and so each were given a2 ranking

The citations from the authors were collated into their respective universitydepartments and totalled From these collations a set of four different tableswere produced each containing the 26 departments their RAE rating and thetotal number of citations Two of the tables were differentiated by a calculatedaverage citation rate dependent on the status of the cited author The other twotables were differentiated by the total citations received by each departmentbut again dependent on the status of the cited authors In both sets of tables thecriteria for differentiation were submitted and non-submitted authors and theircitation scores To again obtain a Spearman rank-order correlation eachdepartment in each table was then assigned a numerical rank and if necessaryan average numerical rank applied

Four sets of calculations were then performed

(1) a ranking for all citations received by each department irrespective ofwhether the author had been submitted or not

(2) a ranking by the total citations received by each department countingsubmitted authors only

JD596

716

(3) a ranking which shows the average citation rate for each departmentusing all departmental staff irrespective of whether the author had beensubmitted or not and

(4) a ranking which shows the average citation rate by department usingsubmitted departmental staff only

Issues arising out of the method adoptedCitation counts can however well carried out lead to erroneous counts Themethods adopted in this study go a long way to minimise such erroneouscounts A number of commonly voiced criticisms of citation counting (Liu1993 MacRoberts and MacRoberts 1989) were potentially present in this studyand are worthy of comment The general problem of mistakenly countingauthors with the same surname and initials but in different disciplines wasunderstood and careful inspection was exercised to limit such miscounting

This process was considerably helped by the RAE submissions whichidentireged the authorsrsquo interests and some of their publications Examination ofthe bibliographical record of the citing article for its subject and its fellowcitations helped minimise potential error Unquestionably however judgementwas required Where it was very difregcult to assess the validity of a particularcitation the record was included Such cases were infrequent and are not likelyto distort the total counts or their relative rankings In any case it is thecomparative ranking of the results which is being sought rather than anabsolutely accurate citation count

Where formerly only regrst named authors were cited now second andsubsequent authors are also cited in the ISIrsquos indexes thus the comment thatco-authors were uncited within the index and hence uncounted no longerapplies This has improved the citation count achieved and removed asigniregcant objection However Lange suggests that some caution needs to beexercised in assuming that a regrst and subsequent cited author search can bedeemed to be absolutely exact (Lange 2001) Self-citation has long beencriticised in citation analysis such practice it has been suggested leads todistortion and bias in conclusions drawn from counts The evidence andstudies conducted so far however show that this practice has little effect on theresults of citation analysis Snyder and Bonzi (1998) in a study whichexamined the patterns of self-citation in six disciplines clearly showed itsprevalence but were able to demonstrate that the patterns of self-citation arevery similar between disciplines and that the lowest self-citation rate of 3 percent was found in the humanities including presumably archaeology

An author making a submission to the RAE may choose material that hasbeen published at any time during the qualifying period This may make theitems submitted potentially ordfoldordm or very ordfnewordm material Counting thecitations to a population of ordfnewordm articles against ordfoldordm could easily give askewed result in favour of the old articles given that more citations are likely to

Archaeology andthe 2001 RAE

717

accrue to the older material We did not examine this in detail but a simpleoverview indicated that the pattern of age of items submitted did not differmuch between the departments examined

Humanities scholars tend to publish more monographs than those in otherdisciplines A random sample of 40 submitted authors examined here credited atleast 32 of them with either having published a monograph or chapter in onesubmitted for peer review In many cases they had published more than onemonograph or chapter in the assessment period Cronin et al (1997) have examinedthe differing author citation patterns when comparing journals and monographs inthe regeld of sociology They suggest that journals and monographs attract twoquite distinct populations of authors amongst the very highly cited this is certainlya concern and it can be assumed that insufregcient citations totals will result fromthis However we had no evidence that this affected one department more thananother and it is worth reiterating that it is the comparative ranking of citationcounts that is studied here rather than the precise count

A criticism may be levelled at the scope and coverage of the AHCI in termsof its bias towards USAEuropean publications as a distorting feature whenmaking citation counts (Braun et al 2000) Therefore where an author hasmade contributions to non-USAEuropean literature there is less likelihood ofhis or her work being noted V Nanda of University College London did notreceive any citations for his work which has been published in Asia LikewisePM Carroll of the University of Shefregeld did not receive any citations to hisworks when published in German While this is not conclusive evidence as toany particular bias the work of Braun et al (2000) clearly shows a strongpreference for a USAEuropean dominance of journal titles and publisherswithin Web of Science Examining the archaeology journal list in the AHCIconregrms a preference for USAEuropean journals Table II shows thedistribution of journal title by place of publication

This shows there is a deregnite bias toward USAEuropean journals so thelikelihood of regnding the cited work of UK archaeologists is considerable plusmnunless they publish elsewhere This bias in practice adds credibility to thisparticular study This it is contended will help support the results obtainedand the conclusions drawn

Place of publication Number of journals

Denmark 1France 4Germany 3Israel 1South Africa 1UK 10USA 17Total 37

Table IIDistribution of journalsindexed by ISI Web ofScience for Archaeology

JD596

718

ResultsThe results have been summarised in a number of Tables These Tables giveseveral views of the data notably by a ranking of the RAE score obtained byeach archaeology department and several different citation statistics Allcorrelations found were signiregcant thus conregrming earlier studies We alsofound that correlations for just staff returned for assessment is even higherthereby arguing that this is an even better predictor of RAE results

Table III gives an overall view of the RAE score obtained by eachdepartment and a comparative ranking by the four basic citation statistics thathave been calculated for each of them Table IV describes the core statisticscollected and of the range of values obtained for each of the departments

University name

RAEresult2001

Ranking bysubmitted

staffcitations

Rankingby all

citations

Ranking byaverage

citations forsubmitted staff

only

Ranking byaverage

citations for allstaff members

Trinity College Carmarthen 3b 25 23 20 24King Alfredrsquos College

Winchester 3a 24 24 26 25University of Wales College

Newport 3a 23 25 25 26University of Edinburgh 3a 21 21 22 21Bournemouth University 3a 18 18 21 20University of Newcastle 3a 17 17 17 17University of York 3a 15 15 18 18Birkbeck College (one staff

member only) 4 26 26 12 13University of Bristol 4 22 22 24 23University of Nottingham 4 19 19 15 15University of Birmingham 4 16 16 23 22University of Glasgow 4 14 14 19 19University of Wales Lampeter 4 13 11 13 9University of Exeter 5 20 20 16 16University of Liverpool 5 12 13 11 11Cardiff University 5 11 12 7 8University of Leicester 5 9 9 14 14University of Southampton 5 8 8 8 7University of Bradford 5 7 6 5 4University of Durham 5 6 7 9 10University of Shefregeld 5 5 5 4 3University College London 5 4 4 10 12The Queens University of

Belfast 5 2 2 1 1University of Reading 5 10 10 6 6University of Cambridge 5 3 3 3 5University of Oxford 5 1 1 1 2

Table IIIComparative rankingwith the RAE rating

result 2001 staffcitation counts and

citation count averagesfor UoA 58

(archaeology)

Archaeology andthe 2001 RAE

719

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Bir

kbec

kC

olle

ge

60

61

06

000

006

00U

niv

ersi

tyof

Bir

min

gham

660

6630

22

200

002

06B

ourn

em

outh

Univ

ersi

ty47

047

160

294

000

294

Univ

ersi

tyof

Bra

dfo

rd23

878

316

193

125

326

00

143

6U

niv

ersi

tyof

Bri

stol

370

3718

02

060

002

06U

niv

ersi

tyof

Cam

bri

dge

811

8889

943

2318

86

383

136

2U

niv

ersi

tyof

Durh

am25

22

254

363

700

067

651

Univ

ersi

tyof

Exet

er41

041

110

373

000

373

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

115

167

21

572

501

78U

niv

ersi

tyof

Lei

cest

er16

63

169

352

474

150

457

Univ

ersi

tyof

Liv

erpoo

l11

10

111

180

617

000

617

Univ

ersi

tyof

New

cast

le51

657

152

340

300

335

Univ

ersi

tyof

Not

tingham

450

4512

03

750

003

75U

niv

ersi

tyof

Oxfo

rd1

192

761

268

5911

202

06

9118

11

Univ

ersi

tyof

Rea

din

g16

40

164

200

820

000

820

Univ

ersi

tyof

Shef

regel

d31

462

376

202

157

031

00

170

9U

niv

ersi

tyof

Sou

tham

pto

n22

423

247

303

747

767

748

Univ

ersi

tyC

olle

ge

Lon

don

550

955

988

46

252

256

08U

niv

ersi

tyof

Yor

k77

077

230

335

000

335

Univ

ersi

tyof

Edin

burg

h38

038

140

271

000

271

Univ

ersi

tyof

Gla

sgow

103

010

331

03

320

003

32

(continued

)

Table IVSummary results ofcitation counts staffnumbers and citationcount averages for UoA58 (archaeology)

JD596

720

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Tri

nit

yC

olle

ge

Car

mar

then

612

182

73

001

712

00C

ardif

fU

niv

ersi

ty14

30

143

191

753

000

715

Univ

ersi

tyof

Wal

esL

ampet

er10

653

159

195

558

106

06

63U

niv

ersi

tyof

Wal

esC

olle

ge

New

por

t14

014

80

175

000

175

The

Quee

ns

Univ

ersi

tyof

Bel

fast

977

698

315

365

13

200

546

1G

rand

tota

ls5

790

423

621

360

973

951

579

911

Table IV

Archaeology andthe 2001 RAE

721

Calculations using the core data and the RAE score give the four measures ofcorrelation required Two of the measures deal with total citation counts andthe other two are concerned with average citations per member of staff The2001 RAE rating achieved by each archaeology department and theirrespective citation statistics are shown in Table V Table V has been sortedwithin each RAE rating by the number of submitted staff citations receivedgiving a snapshot view of the range of citations received and also how thiscompares between ratings To compare the 1996 and 2001 RAE ratingsTable VI shows the scores obtained in both assessments and the movementbetween the two periods

Table VII summarises the results obtained for the Spearman rank-ordercorrelation analyses for the four differing citation measures that were usedThe correlation scores have been rounded to two signiregcant reggures Each ofthese correlation scores is highly signiregcant ( p 001)

The results obtained by Oppenheim in his earlier study on archaeology areshown in Table VIII for comparison They are consistent with the currentresults

Analysis of results682 staff members were checked for their citation counts Between them theyaccumulated 6213 citations although 262 of the authors did not receive anycitations at all in the assessment period The individual citation count rangedfrom 0 to 565 with an overall average of 911 Of the cited authors just over 50per cent of them have a citation level of between one and six citations each Theexceptional score of 565 citations obtained by PJ Reimer can be attributed tojust three articles These were articles dealing with radiocarbon dating linkedto dendrochronolgy and were as follows

Quaternary Science Reviews 1996 15(7) 655 (10 citations)

Radiocarbon 1998 40(3) 1041 (491 citations)

Radiocarbon 1998 40(3) 1127 (64 citations)

Reimerrsquos nearest rival was P Forster from Cambridge University with 471citations from four articles They deal with DNA studies tracing the migrationof humans from Africa

American Journal of Human Genetics 1996 59(4) 935 (108 citations)

American Journal of Human Genetics 1996 61(3) 691 (73 citations)

American Journal of Human Genetics 1996 59(1) 185 (164 citations)

Genetics 1995 141(2) 743 (126 citations)

These two authors account for one-sixth of all the citations counted and theirarticles appear in just four journals all of which are available in both print andelectronic form

JD596

722

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

being suited to journal publication and the more discursive ordfsoftordm humanitiesresearch lending itself to monographic publication (Meadows 1998) In thepublishing industry itself Mynott of the Cambridge University Press clearlyidentireges the pressures placed upon him Some of the normal editing andrevision processes were foreshortened and there was a ordfcompetitionordm with someworks which would only be offered if they could be published within the giventime-scale (Mynott 1999) He noted that his Press published 122 more titles inthe regrst six months of 1996 compared to the previous year

Walford (2000) also found a sharp increase in the number of contributions toscholarly journals in the run up to a RAE He suggested that whilst someordfsalami-slicingordm took place there was also a proliferation of run-of-the-millpapers which though weeded out by the peer review process put strain on therefereeing process

A new discovery in medicine confers status to the published work and theauthor therefore making it excellent RAE material The humanities scholarrsquoswork on the other hand has to be judged much more regnely on the contributionof its scholarship to the body of existing knowledge and this Bernard arguesmakes judging the quality of the work much more difregcult (Bernard 2000)Complexities start to arise in the humanities when scholars begin to interprettheir research material in controversial ways Whilst scientiregc discoveries canbe tested and experiments replicated the opinion of one historian againstanother cannot be tested to verify its truth Thus the quality of some particularresearch as well as its suitability for RAE submission could be questioned Itcould be argued that archaeology is in a similar position to historical researchsince although it is based on clear evidence (archaeological dig regndingshistorical records) much of the work is speculative and based on the judgementand interpretation of the researcher Such possible ambiguities in assessing thequality of humanities research clearly allow for the possibility of inaccuraciesin awarding RAE ratings

Citation analysisCitation analysis is based on counts of the number of citations that an article ormonograph has received in a particular period Citation analysis was not easilyundertaken until the launch of Science Citation Index Social Sciences CitationIndex and Art and Humanities Citation Index Garregeldrsquos seminal work oncitation analysis although 25 years old provides an authoritative historicaland conceptual understanding of the topic and its many applications (Garregeld1979) A more up to date overview can be obtained from (Cronin and BarskyAtkins 2000)

The use of citation studies in place of or as a supplement to the RAEassessment has been considered extensively by Oppenheim and his co-workersThey have published a number of articles demonstrating that the correlationbetween the overall RAE ranking of departments and the collective scores

JD596

712

found by counting the citations received by the authors from thosedepartments is statistically signiregcant (Holmes and Oppenheim 2001Oppenheim 1995 1996 1997) He argues that citation analysis costs afraction of the RAE and is much less intrusive than the RAE This has beenfollowed up with related suggestions for improving the RAE in the future(Harnad et al 2003)

Other researchers have found similar strong correlations between citationcounts and RAE scores Seng and Willett (1995) examined the correlationbetween the publications of UK library schools between the years 1989-1990and the 1992 RAE result More recent work to consider citation analysis as analternative or supplement to the RAE was carried out by Smith and Eysenckwho examined the citation record of psychologists for the year 1998 citationswere checked for their probity to eliminate self-citations and correct anymisnamed individuals The results were compared to the 1996 and 2001 RAEsA 09 and 085 correlation was found for all-author citations respectively (Smithand Eysenck 2002) These statistically signiregcant results they argue areample supporting evidence for the use of citation analysis as an effectiveevaluation tool of research quality Sarwar studied UK civil engineeringdepartments to establish if there was a correlation between the 1996 RAE and acitation analysis of those departments who had achieved a rating of 4 or aboveComparing the citation record of the authors in those departments for the years1995-1997 and the departmentrsquos 1996 RAE rating he (Sarwar 2000) conregrmeda signiregcant correlation between the two sets of results

In one of his papers Oppenheim (1997) examined the subject area ofarchaeology He demonstrated that archaeology produced the highestcorrelation of the three subject areas studied

Citation analysis has many critics and some have focussed their criticism onthe idea that citation analysis could be used as an input into the RAE process(Johnson 2001 Warner 2000) or on the bias introduced by self-citation (Seglen1992) The response of proponents of the method is the inherent robustness ofcitation analysis as a valid tool of measurement since all the experiments so farhave shown strong statistically signiregcant correlations with RAE results Thequestion of self-citation is dealt with satisfactorily by (Snyder and Bonzi 1998)

While Oppenheim is a strong proponent of the use of citation analysis toestablish a funding hierarchy he has concluded (Baird and Oppenheim 1994)that

[T]here is not and never can be one single measure of the value of information that will beuniversally acceptable However there are a number of measures that might in combinationlead to some sort of index of the value of a piece of information an individualrsquos researchcontribution or a collection of information

This collection view of ordfmeasuresordm which together could be used to provide amore balanced and objective indicator of research quality is one shared byMartin (1996) who analysed the possible measures that could be used Martin

Archaeology andthe 2001 RAE

713

crucially notes that citation counts are an indicator of impact of the authortheir past reputation and the organisation in which they work He concludesthat high citation counts may not always indicate quality or importance butmay be for controversy fraud or a simply incorrect work Studies underway inthis department[1] are examining citation counts for controversial articlesMartin also makes the important observation that the more indicators ofresearch you have the more difregcult it is for individuals to manipulate them totheir advantage

Van Raan (1996) describes an assessment exercise undertaken in theNetherlands where peer review is used jointly with advanced bibliometrictechniques to evaluate research The majority of the reviewers come fromoutside the Netherlands and they undertake a quality assessment of theresearch output from the universities In addition a scrupulous bibliometricassessment process designed to remove the many anomalies found withincitation counting such as self-citing incorrect counts journal coverage etcwas carried out The result is a ordfcleanedordm citation count Van Raan (1996)concluded that

We showed that the resulting indicators are useful they address signiregcant concepts in theframework of evaluation and therefore can be considered an indispensable element next topeer review in research performance assessment procedures

This approach validates the use of citation analysis as a sound measure ofresearch performance when used along side peer review

The research described in this paper was aimed at establishing whether thewell-established correlations between RAE performance and citation countscontinue to apply to the 2001 RAE and also whether the ability to moreprecisely identify who had been returned for the 2001 RAE leads to a bettercorrelation than simply carrying out an analysis based on all staff in adepartment Archaeology was chosen as a subject for study for two reasonsregrstly it had been studied before so comparisons with earlier results could bemade and secondly it is not such a large discipline so the research could becompleted in a reasonable time frame

Methods employedA citation analysis was undertaken which counted the citations received bythose academics in archaeology departments submitted for peer review in the2001 RAE In earlier assessments it was not possible to identify the speciregcacademics returned by any particular department nor the publications theysubmitted for assessment This is now however possible for the 2001 RAEThus the methods adopted differ slightly from previous similar studies It isworth stressing that each academicrsquos submission is recorded on the RAE Website and is therefore a public document Therefore no permission was neededfrom this individuals to include them in our analysis or to present resultsrelating to them (see Table I)

JD596

714

Archaeology was chosen following an assessment carried out to regnd a subjectthat was both sufregciently discrete to allow its careful measurement and was ofthe right size and scale that it could be comfortably managed Archaeology isdenoted in the 2001 RAE as unit of assessment (UoA) 58 A total of 26university departments returned submissions into UoA 58 They were rankedin the 2001 RAE with scores from 3b to 5 for their research excellence Asnoted above the report of the 2001 RAE (HEFCE 2001b) for the regrst time madeavailable details of those academics that had been submitted for peer review bydepartments All the names of the academics who were submitted for UoA 58were thus obtained Each academic was allowed to submit up to four itemswhich they had produced during the assessment period We noted details of all682 staff including name institution and the details of the items submitted forthe RAE ISI Web of Science was used to carry out the citation counting Thesearches were conregned to the Art and Humanities Citation Index (AHCI) sinceit covers the regeld of archaeology The cited reference searches were carried outbetween August and September 2002 The searches were conregned to the RAEassessment period ie 1994 to 2000 and looked for citations to material thatwas published during this interval by the authors concerned noting thenumber of citations this material had received Named authors were normallyentered into the search as given In the case of hyphenated or compound namesstandard ISI practice of contraction was used For example the name Van DerLeeum was entered as Vanderleeum Some difregculty was encountered withauthors who had several initials Where the search returned a signiregcantnumber of hits for all the initials it was considered that the author routinelyused their fully initialled name in authorship Where a nil or very low returnwas encountered for a multiple initialled author further searches were carriedout by progressively reducing the number of initials until an appropriate resultwas yielded The cited references were regrst scrutinised to ensure that they fellwithin the RAE assessment period ie 1994 to 2000 Each cited reference wasthen examined to ensure it was the right subject and that it could be attributedto the author concerned Use was made of the authorrsquos submission to the RAE

Author Citation count Afregliation

PJ Reimer 565 BelfastP Forster 471 CambridgeREM Hedges 268 OxfordEJ Rhodes 127 OxfordMGL Baillie 120 BelfastMB Roberts 94 UCLJ Elsner 94 OxfordJ Pollard 92 BradfordFG McCormac 81 BelfastIR Hodder 81 Cambridge

Table IThe top ten most cited

authors and theirafregliation

Archaeology andthe 2001 RAE

715

which indicated his or her subject area Where ambiguity was apparent inauthorship records were marked and the abstract and cited referencesexamined to verify that the correct author and subject had been identiregedSimilarly ambiguous records which had direct electronic links to an abstractand cited references were also scrutinised

A small number of cited works were ignored ie those listed withoutpublication dates unpublished works and those ordfin pressordm Where ambiguitywas still apparent in attributing authorship a search was made at the authorrsquosdepartmental web site and access to their publication record clariregedresponsibility for the cited work

Each authorrsquos citation count was then calculated Anomalous records wererechecked as necessary The 682 named authors were transcribed onto aspreadsheet into their respective departments and citation counts attributableto them were likewise entered The authors were differentiated to show whichof them had been submitted for peer review Of the 682 authors listed 73 hadnot been submitted for review From this general data a number of speciregccitation statistics by department including averages were calculated to allowsubsequent correlation calculations to be made

In order to carry out a Spearman rank-order correlation calculation (themethod used in all these studies) we ranked the archaeology departments bytheir RAE scores and their different citation statistics The 26 archaeologydepartments were listed in ascending order of their 2001 RAE ratings Eachdepartment was then assigned a separate ascending numerical rank Wherethere were two or more equal values in the departmental rating an average ofthe numerical ranking was applied For example Reading Oxford andCambridge each had a 5 (the highest possible) rating and so each were given a2 ranking

The citations from the authors were collated into their respective universitydepartments and totalled From these collations a set of four different tableswere produced each containing the 26 departments their RAE rating and thetotal number of citations Two of the tables were differentiated by a calculatedaverage citation rate dependent on the status of the cited author The other twotables were differentiated by the total citations received by each departmentbut again dependent on the status of the cited authors In both sets of tables thecriteria for differentiation were submitted and non-submitted authors and theircitation scores To again obtain a Spearman rank-order correlation eachdepartment in each table was then assigned a numerical rank and if necessaryan average numerical rank applied

Four sets of calculations were then performed

(1) a ranking for all citations received by each department irrespective ofwhether the author had been submitted or not

(2) a ranking by the total citations received by each department countingsubmitted authors only

JD596

716

(3) a ranking which shows the average citation rate for each departmentusing all departmental staff irrespective of whether the author had beensubmitted or not and

(4) a ranking which shows the average citation rate by department usingsubmitted departmental staff only

Issues arising out of the method adoptedCitation counts can however well carried out lead to erroneous counts Themethods adopted in this study go a long way to minimise such erroneouscounts A number of commonly voiced criticisms of citation counting (Liu1993 MacRoberts and MacRoberts 1989) were potentially present in this studyand are worthy of comment The general problem of mistakenly countingauthors with the same surname and initials but in different disciplines wasunderstood and careful inspection was exercised to limit such miscounting

This process was considerably helped by the RAE submissions whichidentireged the authorsrsquo interests and some of their publications Examination ofthe bibliographical record of the citing article for its subject and its fellowcitations helped minimise potential error Unquestionably however judgementwas required Where it was very difregcult to assess the validity of a particularcitation the record was included Such cases were infrequent and are not likelyto distort the total counts or their relative rankings In any case it is thecomparative ranking of the results which is being sought rather than anabsolutely accurate citation count

Where formerly only regrst named authors were cited now second andsubsequent authors are also cited in the ISIrsquos indexes thus the comment thatco-authors were uncited within the index and hence uncounted no longerapplies This has improved the citation count achieved and removed asigniregcant objection However Lange suggests that some caution needs to beexercised in assuming that a regrst and subsequent cited author search can bedeemed to be absolutely exact (Lange 2001) Self-citation has long beencriticised in citation analysis such practice it has been suggested leads todistortion and bias in conclusions drawn from counts The evidence andstudies conducted so far however show that this practice has little effect on theresults of citation analysis Snyder and Bonzi (1998) in a study whichexamined the patterns of self-citation in six disciplines clearly showed itsprevalence but were able to demonstrate that the patterns of self-citation arevery similar between disciplines and that the lowest self-citation rate of 3 percent was found in the humanities including presumably archaeology

An author making a submission to the RAE may choose material that hasbeen published at any time during the qualifying period This may make theitems submitted potentially ordfoldordm or very ordfnewordm material Counting thecitations to a population of ordfnewordm articles against ordfoldordm could easily give askewed result in favour of the old articles given that more citations are likely to

Archaeology andthe 2001 RAE

717

accrue to the older material We did not examine this in detail but a simpleoverview indicated that the pattern of age of items submitted did not differmuch between the departments examined

Humanities scholars tend to publish more monographs than those in otherdisciplines A random sample of 40 submitted authors examined here credited atleast 32 of them with either having published a monograph or chapter in onesubmitted for peer review In many cases they had published more than onemonograph or chapter in the assessment period Cronin et al (1997) have examinedthe differing author citation patterns when comparing journals and monographs inthe regeld of sociology They suggest that journals and monographs attract twoquite distinct populations of authors amongst the very highly cited this is certainlya concern and it can be assumed that insufregcient citations totals will result fromthis However we had no evidence that this affected one department more thananother and it is worth reiterating that it is the comparative ranking of citationcounts that is studied here rather than the precise count

A criticism may be levelled at the scope and coverage of the AHCI in termsof its bias towards USAEuropean publications as a distorting feature whenmaking citation counts (Braun et al 2000) Therefore where an author hasmade contributions to non-USAEuropean literature there is less likelihood ofhis or her work being noted V Nanda of University College London did notreceive any citations for his work which has been published in Asia LikewisePM Carroll of the University of Shefregeld did not receive any citations to hisworks when published in German While this is not conclusive evidence as toany particular bias the work of Braun et al (2000) clearly shows a strongpreference for a USAEuropean dominance of journal titles and publisherswithin Web of Science Examining the archaeology journal list in the AHCIconregrms a preference for USAEuropean journals Table II shows thedistribution of journal title by place of publication

This shows there is a deregnite bias toward USAEuropean journals so thelikelihood of regnding the cited work of UK archaeologists is considerable plusmnunless they publish elsewhere This bias in practice adds credibility to thisparticular study This it is contended will help support the results obtainedand the conclusions drawn

Place of publication Number of journals

Denmark 1France 4Germany 3Israel 1South Africa 1UK 10USA 17Total 37

Table IIDistribution of journalsindexed by ISI Web ofScience for Archaeology

JD596

718

ResultsThe results have been summarised in a number of Tables These Tables giveseveral views of the data notably by a ranking of the RAE score obtained byeach archaeology department and several different citation statistics Allcorrelations found were signiregcant thus conregrming earlier studies We alsofound that correlations for just staff returned for assessment is even higherthereby arguing that this is an even better predictor of RAE results

Table III gives an overall view of the RAE score obtained by eachdepartment and a comparative ranking by the four basic citation statistics thathave been calculated for each of them Table IV describes the core statisticscollected and of the range of values obtained for each of the departments

University name

RAEresult2001

Ranking bysubmitted

staffcitations

Rankingby all

citations

Ranking byaverage

citations forsubmitted staff

only

Ranking byaverage

citations for allstaff members

Trinity College Carmarthen 3b 25 23 20 24King Alfredrsquos College

Winchester 3a 24 24 26 25University of Wales College

Newport 3a 23 25 25 26University of Edinburgh 3a 21 21 22 21Bournemouth University 3a 18 18 21 20University of Newcastle 3a 17 17 17 17University of York 3a 15 15 18 18Birkbeck College (one staff

member only) 4 26 26 12 13University of Bristol 4 22 22 24 23University of Nottingham 4 19 19 15 15University of Birmingham 4 16 16 23 22University of Glasgow 4 14 14 19 19University of Wales Lampeter 4 13 11 13 9University of Exeter 5 20 20 16 16University of Liverpool 5 12 13 11 11Cardiff University 5 11 12 7 8University of Leicester 5 9 9 14 14University of Southampton 5 8 8 8 7University of Bradford 5 7 6 5 4University of Durham 5 6 7 9 10University of Shefregeld 5 5 5 4 3University College London 5 4 4 10 12The Queens University of

Belfast 5 2 2 1 1University of Reading 5 10 10 6 6University of Cambridge 5 3 3 3 5University of Oxford 5 1 1 1 2

Table IIIComparative rankingwith the RAE rating

result 2001 staffcitation counts and

citation count averagesfor UoA 58

(archaeology)

Archaeology andthe 2001 RAE

719

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Bir

kbec

kC

olle

ge

60

61

06

000

006

00U

niv

ersi

tyof

Bir

min

gham

660

6630

22

200

002

06B

ourn

em

outh

Univ

ersi

ty47

047

160

294

000

294

Univ

ersi

tyof

Bra

dfo

rd23

878

316

193

125

326

00

143

6U

niv

ersi

tyof

Bri

stol

370

3718

02

060

002

06U

niv

ersi

tyof

Cam

bri

dge

811

8889

943

2318

86

383

136

2U

niv

ersi

tyof

Durh

am25

22

254

363

700

067

651

Univ

ersi

tyof

Exet

er41

041

110

373

000

373

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

115

167

21

572

501

78U

niv

ersi

tyof

Lei

cest

er16

63

169

352

474

150

457

Univ

ersi

tyof

Liv

erpoo

l11

10

111

180

617

000

617

Univ

ersi

tyof

New

cast

le51

657

152

340

300

335

Univ

ersi

tyof

Not

tingham

450

4512

03

750

003

75U

niv

ersi

tyof

Oxfo

rd1

192

761

268

5911

202

06

9118

11

Univ

ersi

tyof

Rea

din

g16

40

164

200

820

000

820

Univ

ersi

tyof

Shef

regel

d31

462

376

202

157

031

00

170

9U

niv

ersi

tyof

Sou

tham

pto

n22

423

247

303

747

767

748

Univ

ersi

tyC

olle

ge

Lon

don

550

955

988

46

252

256

08U

niv

ersi

tyof

Yor

k77

077

230

335

000

335

Univ

ersi

tyof

Edin

burg

h38

038

140

271

000

271

Univ

ersi

tyof

Gla

sgow

103

010

331

03

320

003

32

(continued

)

Table IVSummary results ofcitation counts staffnumbers and citationcount averages for UoA58 (archaeology)

JD596

720

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Tri

nit

yC

olle

ge

Car

mar

then

612

182

73

001

712

00C

ardif

fU

niv

ersi

ty14

30

143

191

753

000

715

Univ

ersi

tyof

Wal

esL

ampet

er10

653

159

195

558

106

06

63U

niv

ersi

tyof

Wal

esC

olle

ge

New

por

t14

014

80

175

000

175

The

Quee

ns

Univ

ersi

tyof

Bel

fast

977

698

315

365

13

200

546

1G

rand

tota

ls5

790

423

621

360

973

951

579

911

Table IV

Archaeology andthe 2001 RAE

721

Calculations using the core data and the RAE score give the four measures ofcorrelation required Two of the measures deal with total citation counts andthe other two are concerned with average citations per member of staff The2001 RAE rating achieved by each archaeology department and theirrespective citation statistics are shown in Table V Table V has been sortedwithin each RAE rating by the number of submitted staff citations receivedgiving a snapshot view of the range of citations received and also how thiscompares between ratings To compare the 1996 and 2001 RAE ratingsTable VI shows the scores obtained in both assessments and the movementbetween the two periods

Table VII summarises the results obtained for the Spearman rank-ordercorrelation analyses for the four differing citation measures that were usedThe correlation scores have been rounded to two signiregcant reggures Each ofthese correlation scores is highly signiregcant ( p 001)

The results obtained by Oppenheim in his earlier study on archaeology areshown in Table VIII for comparison They are consistent with the currentresults

Analysis of results682 staff members were checked for their citation counts Between them theyaccumulated 6213 citations although 262 of the authors did not receive anycitations at all in the assessment period The individual citation count rangedfrom 0 to 565 with an overall average of 911 Of the cited authors just over 50per cent of them have a citation level of between one and six citations each Theexceptional score of 565 citations obtained by PJ Reimer can be attributed tojust three articles These were articles dealing with radiocarbon dating linkedto dendrochronolgy and were as follows

Quaternary Science Reviews 1996 15(7) 655 (10 citations)

Radiocarbon 1998 40(3) 1041 (491 citations)

Radiocarbon 1998 40(3) 1127 (64 citations)

Reimerrsquos nearest rival was P Forster from Cambridge University with 471citations from four articles They deal with DNA studies tracing the migrationof humans from Africa

American Journal of Human Genetics 1996 59(4) 935 (108 citations)

American Journal of Human Genetics 1996 61(3) 691 (73 citations)

American Journal of Human Genetics 1996 59(1) 185 (164 citations)

Genetics 1995 141(2) 743 (126 citations)

These two authors account for one-sixth of all the citations counted and theirarticles appear in just four journals all of which are available in both print andelectronic form

JD596

722

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

found by counting the citations received by the authors from thosedepartments is statistically signiregcant (Holmes and Oppenheim 2001Oppenheim 1995 1996 1997) He argues that citation analysis costs afraction of the RAE and is much less intrusive than the RAE This has beenfollowed up with related suggestions for improving the RAE in the future(Harnad et al 2003)

Other researchers have found similar strong correlations between citationcounts and RAE scores Seng and Willett (1995) examined the correlationbetween the publications of UK library schools between the years 1989-1990and the 1992 RAE result More recent work to consider citation analysis as analternative or supplement to the RAE was carried out by Smith and Eysenckwho examined the citation record of psychologists for the year 1998 citationswere checked for their probity to eliminate self-citations and correct anymisnamed individuals The results were compared to the 1996 and 2001 RAEsA 09 and 085 correlation was found for all-author citations respectively (Smithand Eysenck 2002) These statistically signiregcant results they argue areample supporting evidence for the use of citation analysis as an effectiveevaluation tool of research quality Sarwar studied UK civil engineeringdepartments to establish if there was a correlation between the 1996 RAE and acitation analysis of those departments who had achieved a rating of 4 or aboveComparing the citation record of the authors in those departments for the years1995-1997 and the departmentrsquos 1996 RAE rating he (Sarwar 2000) conregrmeda signiregcant correlation between the two sets of results

In one of his papers Oppenheim (1997) examined the subject area ofarchaeology He demonstrated that archaeology produced the highestcorrelation of the three subject areas studied

Citation analysis has many critics and some have focussed their criticism onthe idea that citation analysis could be used as an input into the RAE process(Johnson 2001 Warner 2000) or on the bias introduced by self-citation (Seglen1992) The response of proponents of the method is the inherent robustness ofcitation analysis as a valid tool of measurement since all the experiments so farhave shown strong statistically signiregcant correlations with RAE results Thequestion of self-citation is dealt with satisfactorily by (Snyder and Bonzi 1998)

While Oppenheim is a strong proponent of the use of citation analysis toestablish a funding hierarchy he has concluded (Baird and Oppenheim 1994)that

[T]here is not and never can be one single measure of the value of information that will beuniversally acceptable However there are a number of measures that might in combinationlead to some sort of index of the value of a piece of information an individualrsquos researchcontribution or a collection of information

This collection view of ordfmeasuresordm which together could be used to provide amore balanced and objective indicator of research quality is one shared byMartin (1996) who analysed the possible measures that could be used Martin

Archaeology andthe 2001 RAE

713

crucially notes that citation counts are an indicator of impact of the authortheir past reputation and the organisation in which they work He concludesthat high citation counts may not always indicate quality or importance butmay be for controversy fraud or a simply incorrect work Studies underway inthis department[1] are examining citation counts for controversial articlesMartin also makes the important observation that the more indicators ofresearch you have the more difregcult it is for individuals to manipulate them totheir advantage

Van Raan (1996) describes an assessment exercise undertaken in theNetherlands where peer review is used jointly with advanced bibliometrictechniques to evaluate research The majority of the reviewers come fromoutside the Netherlands and they undertake a quality assessment of theresearch output from the universities In addition a scrupulous bibliometricassessment process designed to remove the many anomalies found withincitation counting such as self-citing incorrect counts journal coverage etcwas carried out The result is a ordfcleanedordm citation count Van Raan (1996)concluded that

We showed that the resulting indicators are useful they address signiregcant concepts in theframework of evaluation and therefore can be considered an indispensable element next topeer review in research performance assessment procedures

This approach validates the use of citation analysis as a sound measure ofresearch performance when used along side peer review

The research described in this paper was aimed at establishing whether thewell-established correlations between RAE performance and citation countscontinue to apply to the 2001 RAE and also whether the ability to moreprecisely identify who had been returned for the 2001 RAE leads to a bettercorrelation than simply carrying out an analysis based on all staff in adepartment Archaeology was chosen as a subject for study for two reasonsregrstly it had been studied before so comparisons with earlier results could bemade and secondly it is not such a large discipline so the research could becompleted in a reasonable time frame

Methods employedA citation analysis was undertaken which counted the citations received bythose academics in archaeology departments submitted for peer review in the2001 RAE In earlier assessments it was not possible to identify the speciregcacademics returned by any particular department nor the publications theysubmitted for assessment This is now however possible for the 2001 RAEThus the methods adopted differ slightly from previous similar studies It isworth stressing that each academicrsquos submission is recorded on the RAE Website and is therefore a public document Therefore no permission was neededfrom this individuals to include them in our analysis or to present resultsrelating to them (see Table I)

JD596

714

Archaeology was chosen following an assessment carried out to regnd a subjectthat was both sufregciently discrete to allow its careful measurement and was ofthe right size and scale that it could be comfortably managed Archaeology isdenoted in the 2001 RAE as unit of assessment (UoA) 58 A total of 26university departments returned submissions into UoA 58 They were rankedin the 2001 RAE with scores from 3b to 5 for their research excellence Asnoted above the report of the 2001 RAE (HEFCE 2001b) for the regrst time madeavailable details of those academics that had been submitted for peer review bydepartments All the names of the academics who were submitted for UoA 58were thus obtained Each academic was allowed to submit up to four itemswhich they had produced during the assessment period We noted details of all682 staff including name institution and the details of the items submitted forthe RAE ISI Web of Science was used to carry out the citation counting Thesearches were conregned to the Art and Humanities Citation Index (AHCI) sinceit covers the regeld of archaeology The cited reference searches were carried outbetween August and September 2002 The searches were conregned to the RAEassessment period ie 1994 to 2000 and looked for citations to material thatwas published during this interval by the authors concerned noting thenumber of citations this material had received Named authors were normallyentered into the search as given In the case of hyphenated or compound namesstandard ISI practice of contraction was used For example the name Van DerLeeum was entered as Vanderleeum Some difregculty was encountered withauthors who had several initials Where the search returned a signiregcantnumber of hits for all the initials it was considered that the author routinelyused their fully initialled name in authorship Where a nil or very low returnwas encountered for a multiple initialled author further searches were carriedout by progressively reducing the number of initials until an appropriate resultwas yielded The cited references were regrst scrutinised to ensure that they fellwithin the RAE assessment period ie 1994 to 2000 Each cited reference wasthen examined to ensure it was the right subject and that it could be attributedto the author concerned Use was made of the authorrsquos submission to the RAE

Author Citation count Afregliation

PJ Reimer 565 BelfastP Forster 471 CambridgeREM Hedges 268 OxfordEJ Rhodes 127 OxfordMGL Baillie 120 BelfastMB Roberts 94 UCLJ Elsner 94 OxfordJ Pollard 92 BradfordFG McCormac 81 BelfastIR Hodder 81 Cambridge

Table IThe top ten most cited

authors and theirafregliation

Archaeology andthe 2001 RAE

715

which indicated his or her subject area Where ambiguity was apparent inauthorship records were marked and the abstract and cited referencesexamined to verify that the correct author and subject had been identiregedSimilarly ambiguous records which had direct electronic links to an abstractand cited references were also scrutinised

A small number of cited works were ignored ie those listed withoutpublication dates unpublished works and those ordfin pressordm Where ambiguitywas still apparent in attributing authorship a search was made at the authorrsquosdepartmental web site and access to their publication record clariregedresponsibility for the cited work

Each authorrsquos citation count was then calculated Anomalous records wererechecked as necessary The 682 named authors were transcribed onto aspreadsheet into their respective departments and citation counts attributableto them were likewise entered The authors were differentiated to show whichof them had been submitted for peer review Of the 682 authors listed 73 hadnot been submitted for review From this general data a number of speciregccitation statistics by department including averages were calculated to allowsubsequent correlation calculations to be made

In order to carry out a Spearman rank-order correlation calculation (themethod used in all these studies) we ranked the archaeology departments bytheir RAE scores and their different citation statistics The 26 archaeologydepartments were listed in ascending order of their 2001 RAE ratings Eachdepartment was then assigned a separate ascending numerical rank Wherethere were two or more equal values in the departmental rating an average ofthe numerical ranking was applied For example Reading Oxford andCambridge each had a 5 (the highest possible) rating and so each were given a2 ranking

The citations from the authors were collated into their respective universitydepartments and totalled From these collations a set of four different tableswere produced each containing the 26 departments their RAE rating and thetotal number of citations Two of the tables were differentiated by a calculatedaverage citation rate dependent on the status of the cited author The other twotables were differentiated by the total citations received by each departmentbut again dependent on the status of the cited authors In both sets of tables thecriteria for differentiation were submitted and non-submitted authors and theircitation scores To again obtain a Spearman rank-order correlation eachdepartment in each table was then assigned a numerical rank and if necessaryan average numerical rank applied

Four sets of calculations were then performed

(1) a ranking for all citations received by each department irrespective ofwhether the author had been submitted or not

(2) a ranking by the total citations received by each department countingsubmitted authors only

JD596

716

(3) a ranking which shows the average citation rate for each departmentusing all departmental staff irrespective of whether the author had beensubmitted or not and

(4) a ranking which shows the average citation rate by department usingsubmitted departmental staff only

Issues arising out of the method adoptedCitation counts can however well carried out lead to erroneous counts Themethods adopted in this study go a long way to minimise such erroneouscounts A number of commonly voiced criticisms of citation counting (Liu1993 MacRoberts and MacRoberts 1989) were potentially present in this studyand are worthy of comment The general problem of mistakenly countingauthors with the same surname and initials but in different disciplines wasunderstood and careful inspection was exercised to limit such miscounting

This process was considerably helped by the RAE submissions whichidentireged the authorsrsquo interests and some of their publications Examination ofthe bibliographical record of the citing article for its subject and its fellowcitations helped minimise potential error Unquestionably however judgementwas required Where it was very difregcult to assess the validity of a particularcitation the record was included Such cases were infrequent and are not likelyto distort the total counts or their relative rankings In any case it is thecomparative ranking of the results which is being sought rather than anabsolutely accurate citation count

Where formerly only regrst named authors were cited now second andsubsequent authors are also cited in the ISIrsquos indexes thus the comment thatco-authors were uncited within the index and hence uncounted no longerapplies This has improved the citation count achieved and removed asigniregcant objection However Lange suggests that some caution needs to beexercised in assuming that a regrst and subsequent cited author search can bedeemed to be absolutely exact (Lange 2001) Self-citation has long beencriticised in citation analysis such practice it has been suggested leads todistortion and bias in conclusions drawn from counts The evidence andstudies conducted so far however show that this practice has little effect on theresults of citation analysis Snyder and Bonzi (1998) in a study whichexamined the patterns of self-citation in six disciplines clearly showed itsprevalence but were able to demonstrate that the patterns of self-citation arevery similar between disciplines and that the lowest self-citation rate of 3 percent was found in the humanities including presumably archaeology

An author making a submission to the RAE may choose material that hasbeen published at any time during the qualifying period This may make theitems submitted potentially ordfoldordm or very ordfnewordm material Counting thecitations to a population of ordfnewordm articles against ordfoldordm could easily give askewed result in favour of the old articles given that more citations are likely to

Archaeology andthe 2001 RAE

717

accrue to the older material We did not examine this in detail but a simpleoverview indicated that the pattern of age of items submitted did not differmuch between the departments examined

Humanities scholars tend to publish more monographs than those in otherdisciplines A random sample of 40 submitted authors examined here credited atleast 32 of them with either having published a monograph or chapter in onesubmitted for peer review In many cases they had published more than onemonograph or chapter in the assessment period Cronin et al (1997) have examinedthe differing author citation patterns when comparing journals and monographs inthe regeld of sociology They suggest that journals and monographs attract twoquite distinct populations of authors amongst the very highly cited this is certainlya concern and it can be assumed that insufregcient citations totals will result fromthis However we had no evidence that this affected one department more thananother and it is worth reiterating that it is the comparative ranking of citationcounts that is studied here rather than the precise count

A criticism may be levelled at the scope and coverage of the AHCI in termsof its bias towards USAEuropean publications as a distorting feature whenmaking citation counts (Braun et al 2000) Therefore where an author hasmade contributions to non-USAEuropean literature there is less likelihood ofhis or her work being noted V Nanda of University College London did notreceive any citations for his work which has been published in Asia LikewisePM Carroll of the University of Shefregeld did not receive any citations to hisworks when published in German While this is not conclusive evidence as toany particular bias the work of Braun et al (2000) clearly shows a strongpreference for a USAEuropean dominance of journal titles and publisherswithin Web of Science Examining the archaeology journal list in the AHCIconregrms a preference for USAEuropean journals Table II shows thedistribution of journal title by place of publication

This shows there is a deregnite bias toward USAEuropean journals so thelikelihood of regnding the cited work of UK archaeologists is considerable plusmnunless they publish elsewhere This bias in practice adds credibility to thisparticular study This it is contended will help support the results obtainedand the conclusions drawn

Place of publication Number of journals

Denmark 1France 4Germany 3Israel 1South Africa 1UK 10USA 17Total 37

Table IIDistribution of journalsindexed by ISI Web ofScience for Archaeology

JD596

718

ResultsThe results have been summarised in a number of Tables These Tables giveseveral views of the data notably by a ranking of the RAE score obtained byeach archaeology department and several different citation statistics Allcorrelations found were signiregcant thus conregrming earlier studies We alsofound that correlations for just staff returned for assessment is even higherthereby arguing that this is an even better predictor of RAE results

Table III gives an overall view of the RAE score obtained by eachdepartment and a comparative ranking by the four basic citation statistics thathave been calculated for each of them Table IV describes the core statisticscollected and of the range of values obtained for each of the departments

University name

RAEresult2001

Ranking bysubmitted

staffcitations

Rankingby all

citations

Ranking byaverage

citations forsubmitted staff

only

Ranking byaverage

citations for allstaff members

Trinity College Carmarthen 3b 25 23 20 24King Alfredrsquos College

Winchester 3a 24 24 26 25University of Wales College

Newport 3a 23 25 25 26University of Edinburgh 3a 21 21 22 21Bournemouth University 3a 18 18 21 20University of Newcastle 3a 17 17 17 17University of York 3a 15 15 18 18Birkbeck College (one staff

member only) 4 26 26 12 13University of Bristol 4 22 22 24 23University of Nottingham 4 19 19 15 15University of Birmingham 4 16 16 23 22University of Glasgow 4 14 14 19 19University of Wales Lampeter 4 13 11 13 9University of Exeter 5 20 20 16 16University of Liverpool 5 12 13 11 11Cardiff University 5 11 12 7 8University of Leicester 5 9 9 14 14University of Southampton 5 8 8 8 7University of Bradford 5 7 6 5 4University of Durham 5 6 7 9 10University of Shefregeld 5 5 5 4 3University College London 5 4 4 10 12The Queens University of

Belfast 5 2 2 1 1University of Reading 5 10 10 6 6University of Cambridge 5 3 3 3 5University of Oxford 5 1 1 1 2

Table IIIComparative rankingwith the RAE rating

result 2001 staffcitation counts and

citation count averagesfor UoA 58

(archaeology)

Archaeology andthe 2001 RAE

719

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Bir

kbec

kC

olle

ge

60

61

06

000

006

00U

niv

ersi

tyof

Bir

min

gham

660

6630

22

200

002

06B

ourn

em

outh

Univ

ersi

ty47

047

160

294

000

294

Univ

ersi

tyof

Bra

dfo

rd23

878

316

193

125

326

00

143

6U

niv

ersi

tyof

Bri

stol

370

3718

02

060

002

06U

niv

ersi

tyof

Cam

bri

dge

811

8889

943

2318

86

383

136

2U

niv

ersi

tyof

Durh

am25

22

254

363

700

067

651

Univ

ersi

tyof

Exet

er41

041

110

373

000

373

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

115

167

21

572

501

78U

niv

ersi

tyof

Lei

cest

er16

63

169

352

474

150

457

Univ

ersi

tyof

Liv

erpoo

l11

10

111

180

617

000

617

Univ

ersi

tyof

New

cast

le51

657

152

340

300

335

Univ

ersi

tyof

Not

tingham

450

4512

03

750

003

75U

niv

ersi

tyof

Oxfo

rd1

192

761

268

5911

202

06

9118

11

Univ

ersi

tyof

Rea

din

g16

40

164

200

820

000

820

Univ

ersi

tyof

Shef

regel

d31

462

376

202

157

031

00

170

9U

niv

ersi

tyof

Sou

tham

pto

n22

423

247

303

747

767

748

Univ

ersi

tyC

olle

ge

Lon

don

550

955

988

46

252

256

08U

niv

ersi

tyof

Yor

k77

077

230

335

000

335

Univ

ersi

tyof

Edin

burg

h38

038

140

271

000

271

Univ

ersi

tyof

Gla

sgow

103

010

331

03

320

003

32

(continued

)

Table IVSummary results ofcitation counts staffnumbers and citationcount averages for UoA58 (archaeology)

JD596

720

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Tri

nit

yC

olle

ge

Car

mar

then

612

182

73

001

712

00C

ardif

fU

niv

ersi

ty14

30

143

191

753

000

715

Univ

ersi

tyof

Wal

esL

ampet

er10

653

159

195

558

106

06

63U

niv

ersi

tyof

Wal

esC

olle

ge

New

por

t14

014

80

175

000

175

The

Quee

ns

Univ

ersi

tyof

Bel

fast

977

698

315

365

13

200

546

1G

rand

tota

ls5

790

423

621

360

973

951

579

911

Table IV

Archaeology andthe 2001 RAE

721

Calculations using the core data and the RAE score give the four measures ofcorrelation required Two of the measures deal with total citation counts andthe other two are concerned with average citations per member of staff The2001 RAE rating achieved by each archaeology department and theirrespective citation statistics are shown in Table V Table V has been sortedwithin each RAE rating by the number of submitted staff citations receivedgiving a snapshot view of the range of citations received and also how thiscompares between ratings To compare the 1996 and 2001 RAE ratingsTable VI shows the scores obtained in both assessments and the movementbetween the two periods

Table VII summarises the results obtained for the Spearman rank-ordercorrelation analyses for the four differing citation measures that were usedThe correlation scores have been rounded to two signiregcant reggures Each ofthese correlation scores is highly signiregcant ( p 001)

The results obtained by Oppenheim in his earlier study on archaeology areshown in Table VIII for comparison They are consistent with the currentresults

Analysis of results682 staff members were checked for their citation counts Between them theyaccumulated 6213 citations although 262 of the authors did not receive anycitations at all in the assessment period The individual citation count rangedfrom 0 to 565 with an overall average of 911 Of the cited authors just over 50per cent of them have a citation level of between one and six citations each Theexceptional score of 565 citations obtained by PJ Reimer can be attributed tojust three articles These were articles dealing with radiocarbon dating linkedto dendrochronolgy and were as follows

Quaternary Science Reviews 1996 15(7) 655 (10 citations)

Radiocarbon 1998 40(3) 1041 (491 citations)

Radiocarbon 1998 40(3) 1127 (64 citations)

Reimerrsquos nearest rival was P Forster from Cambridge University with 471citations from four articles They deal with DNA studies tracing the migrationof humans from Africa

American Journal of Human Genetics 1996 59(4) 935 (108 citations)

American Journal of Human Genetics 1996 61(3) 691 (73 citations)

American Journal of Human Genetics 1996 59(1) 185 (164 citations)

Genetics 1995 141(2) 743 (126 citations)

These two authors account for one-sixth of all the citations counted and theirarticles appear in just four journals all of which are available in both print andelectronic form

JD596

722

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

crucially notes that citation counts are an indicator of impact of the authortheir past reputation and the organisation in which they work He concludesthat high citation counts may not always indicate quality or importance butmay be for controversy fraud or a simply incorrect work Studies underway inthis department[1] are examining citation counts for controversial articlesMartin also makes the important observation that the more indicators ofresearch you have the more difregcult it is for individuals to manipulate them totheir advantage

Van Raan (1996) describes an assessment exercise undertaken in theNetherlands where peer review is used jointly with advanced bibliometrictechniques to evaluate research The majority of the reviewers come fromoutside the Netherlands and they undertake a quality assessment of theresearch output from the universities In addition a scrupulous bibliometricassessment process designed to remove the many anomalies found withincitation counting such as self-citing incorrect counts journal coverage etcwas carried out The result is a ordfcleanedordm citation count Van Raan (1996)concluded that

We showed that the resulting indicators are useful they address signiregcant concepts in theframework of evaluation and therefore can be considered an indispensable element next topeer review in research performance assessment procedures

This approach validates the use of citation analysis as a sound measure ofresearch performance when used along side peer review

The research described in this paper was aimed at establishing whether thewell-established correlations between RAE performance and citation countscontinue to apply to the 2001 RAE and also whether the ability to moreprecisely identify who had been returned for the 2001 RAE leads to a bettercorrelation than simply carrying out an analysis based on all staff in adepartment Archaeology was chosen as a subject for study for two reasonsregrstly it had been studied before so comparisons with earlier results could bemade and secondly it is not such a large discipline so the research could becompleted in a reasonable time frame

Methods employedA citation analysis was undertaken which counted the citations received bythose academics in archaeology departments submitted for peer review in the2001 RAE In earlier assessments it was not possible to identify the speciregcacademics returned by any particular department nor the publications theysubmitted for assessment This is now however possible for the 2001 RAEThus the methods adopted differ slightly from previous similar studies It isworth stressing that each academicrsquos submission is recorded on the RAE Website and is therefore a public document Therefore no permission was neededfrom this individuals to include them in our analysis or to present resultsrelating to them (see Table I)

JD596

714

Archaeology was chosen following an assessment carried out to regnd a subjectthat was both sufregciently discrete to allow its careful measurement and was ofthe right size and scale that it could be comfortably managed Archaeology isdenoted in the 2001 RAE as unit of assessment (UoA) 58 A total of 26university departments returned submissions into UoA 58 They were rankedin the 2001 RAE with scores from 3b to 5 for their research excellence Asnoted above the report of the 2001 RAE (HEFCE 2001b) for the regrst time madeavailable details of those academics that had been submitted for peer review bydepartments All the names of the academics who were submitted for UoA 58were thus obtained Each academic was allowed to submit up to four itemswhich they had produced during the assessment period We noted details of all682 staff including name institution and the details of the items submitted forthe RAE ISI Web of Science was used to carry out the citation counting Thesearches were conregned to the Art and Humanities Citation Index (AHCI) sinceit covers the regeld of archaeology The cited reference searches were carried outbetween August and September 2002 The searches were conregned to the RAEassessment period ie 1994 to 2000 and looked for citations to material thatwas published during this interval by the authors concerned noting thenumber of citations this material had received Named authors were normallyentered into the search as given In the case of hyphenated or compound namesstandard ISI practice of contraction was used For example the name Van DerLeeum was entered as Vanderleeum Some difregculty was encountered withauthors who had several initials Where the search returned a signiregcantnumber of hits for all the initials it was considered that the author routinelyused their fully initialled name in authorship Where a nil or very low returnwas encountered for a multiple initialled author further searches were carriedout by progressively reducing the number of initials until an appropriate resultwas yielded The cited references were regrst scrutinised to ensure that they fellwithin the RAE assessment period ie 1994 to 2000 Each cited reference wasthen examined to ensure it was the right subject and that it could be attributedto the author concerned Use was made of the authorrsquos submission to the RAE

Author Citation count Afregliation

PJ Reimer 565 BelfastP Forster 471 CambridgeREM Hedges 268 OxfordEJ Rhodes 127 OxfordMGL Baillie 120 BelfastMB Roberts 94 UCLJ Elsner 94 OxfordJ Pollard 92 BradfordFG McCormac 81 BelfastIR Hodder 81 Cambridge

Table IThe top ten most cited

authors and theirafregliation

Archaeology andthe 2001 RAE

715

which indicated his or her subject area Where ambiguity was apparent inauthorship records were marked and the abstract and cited referencesexamined to verify that the correct author and subject had been identiregedSimilarly ambiguous records which had direct electronic links to an abstractand cited references were also scrutinised

A small number of cited works were ignored ie those listed withoutpublication dates unpublished works and those ordfin pressordm Where ambiguitywas still apparent in attributing authorship a search was made at the authorrsquosdepartmental web site and access to their publication record clariregedresponsibility for the cited work

Each authorrsquos citation count was then calculated Anomalous records wererechecked as necessary The 682 named authors were transcribed onto aspreadsheet into their respective departments and citation counts attributableto them were likewise entered The authors were differentiated to show whichof them had been submitted for peer review Of the 682 authors listed 73 hadnot been submitted for review From this general data a number of speciregccitation statistics by department including averages were calculated to allowsubsequent correlation calculations to be made

In order to carry out a Spearman rank-order correlation calculation (themethod used in all these studies) we ranked the archaeology departments bytheir RAE scores and their different citation statistics The 26 archaeologydepartments were listed in ascending order of their 2001 RAE ratings Eachdepartment was then assigned a separate ascending numerical rank Wherethere were two or more equal values in the departmental rating an average ofthe numerical ranking was applied For example Reading Oxford andCambridge each had a 5 (the highest possible) rating and so each were given a2 ranking

The citations from the authors were collated into their respective universitydepartments and totalled From these collations a set of four different tableswere produced each containing the 26 departments their RAE rating and thetotal number of citations Two of the tables were differentiated by a calculatedaverage citation rate dependent on the status of the cited author The other twotables were differentiated by the total citations received by each departmentbut again dependent on the status of the cited authors In both sets of tables thecriteria for differentiation were submitted and non-submitted authors and theircitation scores To again obtain a Spearman rank-order correlation eachdepartment in each table was then assigned a numerical rank and if necessaryan average numerical rank applied

Four sets of calculations were then performed

(1) a ranking for all citations received by each department irrespective ofwhether the author had been submitted or not

(2) a ranking by the total citations received by each department countingsubmitted authors only

JD596

716

(3) a ranking which shows the average citation rate for each departmentusing all departmental staff irrespective of whether the author had beensubmitted or not and

(4) a ranking which shows the average citation rate by department usingsubmitted departmental staff only

Issues arising out of the method adoptedCitation counts can however well carried out lead to erroneous counts Themethods adopted in this study go a long way to minimise such erroneouscounts A number of commonly voiced criticisms of citation counting (Liu1993 MacRoberts and MacRoberts 1989) were potentially present in this studyand are worthy of comment The general problem of mistakenly countingauthors with the same surname and initials but in different disciplines wasunderstood and careful inspection was exercised to limit such miscounting

This process was considerably helped by the RAE submissions whichidentireged the authorsrsquo interests and some of their publications Examination ofthe bibliographical record of the citing article for its subject and its fellowcitations helped minimise potential error Unquestionably however judgementwas required Where it was very difregcult to assess the validity of a particularcitation the record was included Such cases were infrequent and are not likelyto distort the total counts or their relative rankings In any case it is thecomparative ranking of the results which is being sought rather than anabsolutely accurate citation count

Where formerly only regrst named authors were cited now second andsubsequent authors are also cited in the ISIrsquos indexes thus the comment thatco-authors were uncited within the index and hence uncounted no longerapplies This has improved the citation count achieved and removed asigniregcant objection However Lange suggests that some caution needs to beexercised in assuming that a regrst and subsequent cited author search can bedeemed to be absolutely exact (Lange 2001) Self-citation has long beencriticised in citation analysis such practice it has been suggested leads todistortion and bias in conclusions drawn from counts The evidence andstudies conducted so far however show that this practice has little effect on theresults of citation analysis Snyder and Bonzi (1998) in a study whichexamined the patterns of self-citation in six disciplines clearly showed itsprevalence but were able to demonstrate that the patterns of self-citation arevery similar between disciplines and that the lowest self-citation rate of 3 percent was found in the humanities including presumably archaeology

An author making a submission to the RAE may choose material that hasbeen published at any time during the qualifying period This may make theitems submitted potentially ordfoldordm or very ordfnewordm material Counting thecitations to a population of ordfnewordm articles against ordfoldordm could easily give askewed result in favour of the old articles given that more citations are likely to

Archaeology andthe 2001 RAE

717

accrue to the older material We did not examine this in detail but a simpleoverview indicated that the pattern of age of items submitted did not differmuch between the departments examined

Humanities scholars tend to publish more monographs than those in otherdisciplines A random sample of 40 submitted authors examined here credited atleast 32 of them with either having published a monograph or chapter in onesubmitted for peer review In many cases they had published more than onemonograph or chapter in the assessment period Cronin et al (1997) have examinedthe differing author citation patterns when comparing journals and monographs inthe regeld of sociology They suggest that journals and monographs attract twoquite distinct populations of authors amongst the very highly cited this is certainlya concern and it can be assumed that insufregcient citations totals will result fromthis However we had no evidence that this affected one department more thananother and it is worth reiterating that it is the comparative ranking of citationcounts that is studied here rather than the precise count

A criticism may be levelled at the scope and coverage of the AHCI in termsof its bias towards USAEuropean publications as a distorting feature whenmaking citation counts (Braun et al 2000) Therefore where an author hasmade contributions to non-USAEuropean literature there is less likelihood ofhis or her work being noted V Nanda of University College London did notreceive any citations for his work which has been published in Asia LikewisePM Carroll of the University of Shefregeld did not receive any citations to hisworks when published in German While this is not conclusive evidence as toany particular bias the work of Braun et al (2000) clearly shows a strongpreference for a USAEuropean dominance of journal titles and publisherswithin Web of Science Examining the archaeology journal list in the AHCIconregrms a preference for USAEuropean journals Table II shows thedistribution of journal title by place of publication

This shows there is a deregnite bias toward USAEuropean journals so thelikelihood of regnding the cited work of UK archaeologists is considerable plusmnunless they publish elsewhere This bias in practice adds credibility to thisparticular study This it is contended will help support the results obtainedand the conclusions drawn

Place of publication Number of journals

Denmark 1France 4Germany 3Israel 1South Africa 1UK 10USA 17Total 37

Table IIDistribution of journalsindexed by ISI Web ofScience for Archaeology

JD596

718

ResultsThe results have been summarised in a number of Tables These Tables giveseveral views of the data notably by a ranking of the RAE score obtained byeach archaeology department and several different citation statistics Allcorrelations found were signiregcant thus conregrming earlier studies We alsofound that correlations for just staff returned for assessment is even higherthereby arguing that this is an even better predictor of RAE results

Table III gives an overall view of the RAE score obtained by eachdepartment and a comparative ranking by the four basic citation statistics thathave been calculated for each of them Table IV describes the core statisticscollected and of the range of values obtained for each of the departments

University name

RAEresult2001

Ranking bysubmitted

staffcitations

Rankingby all

citations

Ranking byaverage

citations forsubmitted staff

only

Ranking byaverage

citations for allstaff members

Trinity College Carmarthen 3b 25 23 20 24King Alfredrsquos College

Winchester 3a 24 24 26 25University of Wales College

Newport 3a 23 25 25 26University of Edinburgh 3a 21 21 22 21Bournemouth University 3a 18 18 21 20University of Newcastle 3a 17 17 17 17University of York 3a 15 15 18 18Birkbeck College (one staff

member only) 4 26 26 12 13University of Bristol 4 22 22 24 23University of Nottingham 4 19 19 15 15University of Birmingham 4 16 16 23 22University of Glasgow 4 14 14 19 19University of Wales Lampeter 4 13 11 13 9University of Exeter 5 20 20 16 16University of Liverpool 5 12 13 11 11Cardiff University 5 11 12 7 8University of Leicester 5 9 9 14 14University of Southampton 5 8 8 8 7University of Bradford 5 7 6 5 4University of Durham 5 6 7 9 10University of Shefregeld 5 5 5 4 3University College London 5 4 4 10 12The Queens University of

Belfast 5 2 2 1 1University of Reading 5 10 10 6 6University of Cambridge 5 3 3 3 5University of Oxford 5 1 1 1 2

Table IIIComparative rankingwith the RAE rating

result 2001 staffcitation counts and

citation count averagesfor UoA 58

(archaeology)

Archaeology andthe 2001 RAE

719

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Bir

kbec

kC

olle

ge

60

61

06

000

006

00U

niv

ersi

tyof

Bir

min

gham

660

6630

22

200

002

06B

ourn

em

outh

Univ

ersi

ty47

047

160

294

000

294

Univ

ersi

tyof

Bra

dfo

rd23

878

316

193

125

326

00

143

6U

niv

ersi

tyof

Bri

stol

370

3718

02

060

002

06U

niv

ersi

tyof

Cam

bri

dge

811

8889

943

2318

86

383

136

2U

niv

ersi

tyof

Durh

am25

22

254

363

700

067

651

Univ

ersi

tyof

Exet

er41

041

110

373

000

373

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

115

167

21

572

501

78U

niv

ersi

tyof

Lei

cest

er16

63

169

352

474

150

457

Univ

ersi

tyof

Liv

erpoo

l11

10

111

180

617

000

617

Univ

ersi

tyof

New

cast

le51

657

152

340

300

335

Univ

ersi

tyof

Not

tingham

450

4512

03

750

003

75U

niv

ersi

tyof

Oxfo

rd1

192

761

268

5911

202

06

9118

11

Univ

ersi

tyof

Rea

din

g16

40

164

200

820

000

820

Univ

ersi

tyof

Shef

regel

d31

462

376

202

157

031

00

170

9U

niv

ersi

tyof

Sou

tham

pto

n22

423

247

303

747

767

748

Univ

ersi

tyC

olle

ge

Lon

don

550

955

988

46

252

256

08U

niv

ersi

tyof

Yor

k77

077

230

335

000

335

Univ

ersi

tyof

Edin

burg

h38

038

140

271

000

271

Univ

ersi

tyof

Gla

sgow

103

010

331

03

320

003

32

(continued

)

Table IVSummary results ofcitation counts staffnumbers and citationcount averages for UoA58 (archaeology)

JD596

720

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Tri

nit

yC

olle

ge

Car

mar

then

612

182

73

001

712

00C

ardif

fU

niv

ersi

ty14

30

143

191

753

000

715

Univ

ersi

tyof

Wal

esL

ampet

er10

653

159

195

558

106

06

63U

niv

ersi

tyof

Wal

esC

olle

ge

New

por

t14

014

80

175

000

175

The

Quee

ns

Univ

ersi

tyof

Bel

fast

977

698

315

365

13

200

546

1G

rand

tota

ls5

790

423

621

360

973

951

579

911

Table IV

Archaeology andthe 2001 RAE

721

Calculations using the core data and the RAE score give the four measures ofcorrelation required Two of the measures deal with total citation counts andthe other two are concerned with average citations per member of staff The2001 RAE rating achieved by each archaeology department and theirrespective citation statistics are shown in Table V Table V has been sortedwithin each RAE rating by the number of submitted staff citations receivedgiving a snapshot view of the range of citations received and also how thiscompares between ratings To compare the 1996 and 2001 RAE ratingsTable VI shows the scores obtained in both assessments and the movementbetween the two periods

Table VII summarises the results obtained for the Spearman rank-ordercorrelation analyses for the four differing citation measures that were usedThe correlation scores have been rounded to two signiregcant reggures Each ofthese correlation scores is highly signiregcant ( p 001)

The results obtained by Oppenheim in his earlier study on archaeology areshown in Table VIII for comparison They are consistent with the currentresults

Analysis of results682 staff members were checked for their citation counts Between them theyaccumulated 6213 citations although 262 of the authors did not receive anycitations at all in the assessment period The individual citation count rangedfrom 0 to 565 with an overall average of 911 Of the cited authors just over 50per cent of them have a citation level of between one and six citations each Theexceptional score of 565 citations obtained by PJ Reimer can be attributed tojust three articles These were articles dealing with radiocarbon dating linkedto dendrochronolgy and were as follows

Quaternary Science Reviews 1996 15(7) 655 (10 citations)

Radiocarbon 1998 40(3) 1041 (491 citations)

Radiocarbon 1998 40(3) 1127 (64 citations)

Reimerrsquos nearest rival was P Forster from Cambridge University with 471citations from four articles They deal with DNA studies tracing the migrationof humans from Africa

American Journal of Human Genetics 1996 59(4) 935 (108 citations)

American Journal of Human Genetics 1996 61(3) 691 (73 citations)

American Journal of Human Genetics 1996 59(1) 185 (164 citations)

Genetics 1995 141(2) 743 (126 citations)

These two authors account for one-sixth of all the citations counted and theirarticles appear in just four journals all of which are available in both print andelectronic form

JD596

722

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

Archaeology was chosen following an assessment carried out to regnd a subjectthat was both sufregciently discrete to allow its careful measurement and was ofthe right size and scale that it could be comfortably managed Archaeology isdenoted in the 2001 RAE as unit of assessment (UoA) 58 A total of 26university departments returned submissions into UoA 58 They were rankedin the 2001 RAE with scores from 3b to 5 for their research excellence Asnoted above the report of the 2001 RAE (HEFCE 2001b) for the regrst time madeavailable details of those academics that had been submitted for peer review bydepartments All the names of the academics who were submitted for UoA 58were thus obtained Each academic was allowed to submit up to four itemswhich they had produced during the assessment period We noted details of all682 staff including name institution and the details of the items submitted forthe RAE ISI Web of Science was used to carry out the citation counting Thesearches were conregned to the Art and Humanities Citation Index (AHCI) sinceit covers the regeld of archaeology The cited reference searches were carried outbetween August and September 2002 The searches were conregned to the RAEassessment period ie 1994 to 2000 and looked for citations to material thatwas published during this interval by the authors concerned noting thenumber of citations this material had received Named authors were normallyentered into the search as given In the case of hyphenated or compound namesstandard ISI practice of contraction was used For example the name Van DerLeeum was entered as Vanderleeum Some difregculty was encountered withauthors who had several initials Where the search returned a signiregcantnumber of hits for all the initials it was considered that the author routinelyused their fully initialled name in authorship Where a nil or very low returnwas encountered for a multiple initialled author further searches were carriedout by progressively reducing the number of initials until an appropriate resultwas yielded The cited references were regrst scrutinised to ensure that they fellwithin the RAE assessment period ie 1994 to 2000 Each cited reference wasthen examined to ensure it was the right subject and that it could be attributedto the author concerned Use was made of the authorrsquos submission to the RAE

Author Citation count Afregliation

PJ Reimer 565 BelfastP Forster 471 CambridgeREM Hedges 268 OxfordEJ Rhodes 127 OxfordMGL Baillie 120 BelfastMB Roberts 94 UCLJ Elsner 94 OxfordJ Pollard 92 BradfordFG McCormac 81 BelfastIR Hodder 81 Cambridge

Table IThe top ten most cited

authors and theirafregliation

Archaeology andthe 2001 RAE

715

which indicated his or her subject area Where ambiguity was apparent inauthorship records were marked and the abstract and cited referencesexamined to verify that the correct author and subject had been identiregedSimilarly ambiguous records which had direct electronic links to an abstractand cited references were also scrutinised

A small number of cited works were ignored ie those listed withoutpublication dates unpublished works and those ordfin pressordm Where ambiguitywas still apparent in attributing authorship a search was made at the authorrsquosdepartmental web site and access to their publication record clariregedresponsibility for the cited work

Each authorrsquos citation count was then calculated Anomalous records wererechecked as necessary The 682 named authors were transcribed onto aspreadsheet into their respective departments and citation counts attributableto them were likewise entered The authors were differentiated to show whichof them had been submitted for peer review Of the 682 authors listed 73 hadnot been submitted for review From this general data a number of speciregccitation statistics by department including averages were calculated to allowsubsequent correlation calculations to be made

In order to carry out a Spearman rank-order correlation calculation (themethod used in all these studies) we ranked the archaeology departments bytheir RAE scores and their different citation statistics The 26 archaeologydepartments were listed in ascending order of their 2001 RAE ratings Eachdepartment was then assigned a separate ascending numerical rank Wherethere were two or more equal values in the departmental rating an average ofthe numerical ranking was applied For example Reading Oxford andCambridge each had a 5 (the highest possible) rating and so each were given a2 ranking

The citations from the authors were collated into their respective universitydepartments and totalled From these collations a set of four different tableswere produced each containing the 26 departments their RAE rating and thetotal number of citations Two of the tables were differentiated by a calculatedaverage citation rate dependent on the status of the cited author The other twotables were differentiated by the total citations received by each departmentbut again dependent on the status of the cited authors In both sets of tables thecriteria for differentiation were submitted and non-submitted authors and theircitation scores To again obtain a Spearman rank-order correlation eachdepartment in each table was then assigned a numerical rank and if necessaryan average numerical rank applied

Four sets of calculations were then performed

(1) a ranking for all citations received by each department irrespective ofwhether the author had been submitted or not

(2) a ranking by the total citations received by each department countingsubmitted authors only

JD596

716

(3) a ranking which shows the average citation rate for each departmentusing all departmental staff irrespective of whether the author had beensubmitted or not and

(4) a ranking which shows the average citation rate by department usingsubmitted departmental staff only

Issues arising out of the method adoptedCitation counts can however well carried out lead to erroneous counts Themethods adopted in this study go a long way to minimise such erroneouscounts A number of commonly voiced criticisms of citation counting (Liu1993 MacRoberts and MacRoberts 1989) were potentially present in this studyand are worthy of comment The general problem of mistakenly countingauthors with the same surname and initials but in different disciplines wasunderstood and careful inspection was exercised to limit such miscounting

This process was considerably helped by the RAE submissions whichidentireged the authorsrsquo interests and some of their publications Examination ofthe bibliographical record of the citing article for its subject and its fellowcitations helped minimise potential error Unquestionably however judgementwas required Where it was very difregcult to assess the validity of a particularcitation the record was included Such cases were infrequent and are not likelyto distort the total counts or their relative rankings In any case it is thecomparative ranking of the results which is being sought rather than anabsolutely accurate citation count

Where formerly only regrst named authors were cited now second andsubsequent authors are also cited in the ISIrsquos indexes thus the comment thatco-authors were uncited within the index and hence uncounted no longerapplies This has improved the citation count achieved and removed asigniregcant objection However Lange suggests that some caution needs to beexercised in assuming that a regrst and subsequent cited author search can bedeemed to be absolutely exact (Lange 2001) Self-citation has long beencriticised in citation analysis such practice it has been suggested leads todistortion and bias in conclusions drawn from counts The evidence andstudies conducted so far however show that this practice has little effect on theresults of citation analysis Snyder and Bonzi (1998) in a study whichexamined the patterns of self-citation in six disciplines clearly showed itsprevalence but were able to demonstrate that the patterns of self-citation arevery similar between disciplines and that the lowest self-citation rate of 3 percent was found in the humanities including presumably archaeology

An author making a submission to the RAE may choose material that hasbeen published at any time during the qualifying period This may make theitems submitted potentially ordfoldordm or very ordfnewordm material Counting thecitations to a population of ordfnewordm articles against ordfoldordm could easily give askewed result in favour of the old articles given that more citations are likely to

Archaeology andthe 2001 RAE

717

accrue to the older material We did not examine this in detail but a simpleoverview indicated that the pattern of age of items submitted did not differmuch between the departments examined

Humanities scholars tend to publish more monographs than those in otherdisciplines A random sample of 40 submitted authors examined here credited atleast 32 of them with either having published a monograph or chapter in onesubmitted for peer review In many cases they had published more than onemonograph or chapter in the assessment period Cronin et al (1997) have examinedthe differing author citation patterns when comparing journals and monographs inthe regeld of sociology They suggest that journals and monographs attract twoquite distinct populations of authors amongst the very highly cited this is certainlya concern and it can be assumed that insufregcient citations totals will result fromthis However we had no evidence that this affected one department more thananother and it is worth reiterating that it is the comparative ranking of citationcounts that is studied here rather than the precise count

A criticism may be levelled at the scope and coverage of the AHCI in termsof its bias towards USAEuropean publications as a distorting feature whenmaking citation counts (Braun et al 2000) Therefore where an author hasmade contributions to non-USAEuropean literature there is less likelihood ofhis or her work being noted V Nanda of University College London did notreceive any citations for his work which has been published in Asia LikewisePM Carroll of the University of Shefregeld did not receive any citations to hisworks when published in German While this is not conclusive evidence as toany particular bias the work of Braun et al (2000) clearly shows a strongpreference for a USAEuropean dominance of journal titles and publisherswithin Web of Science Examining the archaeology journal list in the AHCIconregrms a preference for USAEuropean journals Table II shows thedistribution of journal title by place of publication

This shows there is a deregnite bias toward USAEuropean journals so thelikelihood of regnding the cited work of UK archaeologists is considerable plusmnunless they publish elsewhere This bias in practice adds credibility to thisparticular study This it is contended will help support the results obtainedand the conclusions drawn

Place of publication Number of journals

Denmark 1France 4Germany 3Israel 1South Africa 1UK 10USA 17Total 37

Table IIDistribution of journalsindexed by ISI Web ofScience for Archaeology

JD596

718

ResultsThe results have been summarised in a number of Tables These Tables giveseveral views of the data notably by a ranking of the RAE score obtained byeach archaeology department and several different citation statistics Allcorrelations found were signiregcant thus conregrming earlier studies We alsofound that correlations for just staff returned for assessment is even higherthereby arguing that this is an even better predictor of RAE results

Table III gives an overall view of the RAE score obtained by eachdepartment and a comparative ranking by the four basic citation statistics thathave been calculated for each of them Table IV describes the core statisticscollected and of the range of values obtained for each of the departments

University name

RAEresult2001

Ranking bysubmitted

staffcitations

Rankingby all

citations

Ranking byaverage

citations forsubmitted staff

only

Ranking byaverage

citations for allstaff members

Trinity College Carmarthen 3b 25 23 20 24King Alfredrsquos College

Winchester 3a 24 24 26 25University of Wales College

Newport 3a 23 25 25 26University of Edinburgh 3a 21 21 22 21Bournemouth University 3a 18 18 21 20University of Newcastle 3a 17 17 17 17University of York 3a 15 15 18 18Birkbeck College (one staff

member only) 4 26 26 12 13University of Bristol 4 22 22 24 23University of Nottingham 4 19 19 15 15University of Birmingham 4 16 16 23 22University of Glasgow 4 14 14 19 19University of Wales Lampeter 4 13 11 13 9University of Exeter 5 20 20 16 16University of Liverpool 5 12 13 11 11Cardiff University 5 11 12 7 8University of Leicester 5 9 9 14 14University of Southampton 5 8 8 8 7University of Bradford 5 7 6 5 4University of Durham 5 6 7 9 10University of Shefregeld 5 5 5 4 3University College London 5 4 4 10 12The Queens University of

Belfast 5 2 2 1 1University of Reading 5 10 10 6 6University of Cambridge 5 3 3 3 5University of Oxford 5 1 1 1 2

Table IIIComparative rankingwith the RAE rating

result 2001 staffcitation counts and

citation count averagesfor UoA 58

(archaeology)

Archaeology andthe 2001 RAE

719

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Bir

kbec

kC

olle

ge

60

61

06

000

006

00U

niv

ersi

tyof

Bir

min

gham

660

6630

22

200

002

06B

ourn

em

outh

Univ

ersi

ty47

047

160

294

000

294

Univ

ersi

tyof

Bra

dfo

rd23

878

316

193

125

326

00

143

6U

niv

ersi

tyof

Bri

stol

370

3718

02

060

002

06U

niv

ersi

tyof

Cam

bri

dge

811

8889

943

2318

86

383

136

2U

niv

ersi

tyof

Durh

am25

22

254

363

700

067

651

Univ

ersi

tyof

Exet

er41

041

110

373

000

373

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

115

167

21

572

501

78U

niv

ersi

tyof

Lei

cest

er16

63

169

352

474

150

457

Univ

ersi

tyof

Liv

erpoo

l11

10

111

180

617

000

617

Univ

ersi

tyof

New

cast

le51

657

152

340

300

335

Univ

ersi

tyof

Not

tingham

450

4512

03

750

003

75U

niv

ersi

tyof

Oxfo

rd1

192

761

268

5911

202

06

9118

11

Univ

ersi

tyof

Rea

din

g16

40

164

200

820

000

820

Univ

ersi

tyof

Shef

regel

d31

462

376

202

157

031

00

170

9U

niv

ersi

tyof

Sou

tham

pto

n22

423

247

303

747

767

748

Univ

ersi

tyC

olle

ge

Lon

don

550

955

988

46

252

256

08U

niv

ersi

tyof

Yor

k77

077

230

335

000

335

Univ

ersi

tyof

Edin

burg

h38

038

140

271

000

271

Univ

ersi

tyof

Gla

sgow

103

010

331

03

320

003

32

(continued

)

Table IVSummary results ofcitation counts staffnumbers and citationcount averages for UoA58 (archaeology)

JD596

720

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Tri

nit

yC

olle

ge

Car

mar

then

612

182

73

001

712

00C

ardif

fU

niv

ersi

ty14

30

143

191

753

000

715

Univ

ersi

tyof

Wal

esL

ampet

er10

653

159

195

558

106

06

63U

niv

ersi

tyof

Wal

esC

olle

ge

New

por

t14

014

80

175

000

175

The

Quee

ns

Univ

ersi

tyof

Bel

fast

977

698

315

365

13

200

546

1G

rand

tota

ls5

790

423

621

360

973

951

579

911

Table IV

Archaeology andthe 2001 RAE

721

Calculations using the core data and the RAE score give the four measures ofcorrelation required Two of the measures deal with total citation counts andthe other two are concerned with average citations per member of staff The2001 RAE rating achieved by each archaeology department and theirrespective citation statistics are shown in Table V Table V has been sortedwithin each RAE rating by the number of submitted staff citations receivedgiving a snapshot view of the range of citations received and also how thiscompares between ratings To compare the 1996 and 2001 RAE ratingsTable VI shows the scores obtained in both assessments and the movementbetween the two periods

Table VII summarises the results obtained for the Spearman rank-ordercorrelation analyses for the four differing citation measures that were usedThe correlation scores have been rounded to two signiregcant reggures Each ofthese correlation scores is highly signiregcant ( p 001)

The results obtained by Oppenheim in his earlier study on archaeology areshown in Table VIII for comparison They are consistent with the currentresults

Analysis of results682 staff members were checked for their citation counts Between them theyaccumulated 6213 citations although 262 of the authors did not receive anycitations at all in the assessment period The individual citation count rangedfrom 0 to 565 with an overall average of 911 Of the cited authors just over 50per cent of them have a citation level of between one and six citations each Theexceptional score of 565 citations obtained by PJ Reimer can be attributed tojust three articles These were articles dealing with radiocarbon dating linkedto dendrochronolgy and were as follows

Quaternary Science Reviews 1996 15(7) 655 (10 citations)

Radiocarbon 1998 40(3) 1041 (491 citations)

Radiocarbon 1998 40(3) 1127 (64 citations)

Reimerrsquos nearest rival was P Forster from Cambridge University with 471citations from four articles They deal with DNA studies tracing the migrationof humans from Africa

American Journal of Human Genetics 1996 59(4) 935 (108 citations)

American Journal of Human Genetics 1996 61(3) 691 (73 citations)

American Journal of Human Genetics 1996 59(1) 185 (164 citations)

Genetics 1995 141(2) 743 (126 citations)

These two authors account for one-sixth of all the citations counted and theirarticles appear in just four journals all of which are available in both print andelectronic form

JD596

722

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

which indicated his or her subject area Where ambiguity was apparent inauthorship records were marked and the abstract and cited referencesexamined to verify that the correct author and subject had been identiregedSimilarly ambiguous records which had direct electronic links to an abstractand cited references were also scrutinised

A small number of cited works were ignored ie those listed withoutpublication dates unpublished works and those ordfin pressordm Where ambiguitywas still apparent in attributing authorship a search was made at the authorrsquosdepartmental web site and access to their publication record clariregedresponsibility for the cited work

Each authorrsquos citation count was then calculated Anomalous records wererechecked as necessary The 682 named authors were transcribed onto aspreadsheet into their respective departments and citation counts attributableto them were likewise entered The authors were differentiated to show whichof them had been submitted for peer review Of the 682 authors listed 73 hadnot been submitted for review From this general data a number of speciregccitation statistics by department including averages were calculated to allowsubsequent correlation calculations to be made

In order to carry out a Spearman rank-order correlation calculation (themethod used in all these studies) we ranked the archaeology departments bytheir RAE scores and their different citation statistics The 26 archaeologydepartments were listed in ascending order of their 2001 RAE ratings Eachdepartment was then assigned a separate ascending numerical rank Wherethere were two or more equal values in the departmental rating an average ofthe numerical ranking was applied For example Reading Oxford andCambridge each had a 5 (the highest possible) rating and so each were given a2 ranking

The citations from the authors were collated into their respective universitydepartments and totalled From these collations a set of four different tableswere produced each containing the 26 departments their RAE rating and thetotal number of citations Two of the tables were differentiated by a calculatedaverage citation rate dependent on the status of the cited author The other twotables were differentiated by the total citations received by each departmentbut again dependent on the status of the cited authors In both sets of tables thecriteria for differentiation were submitted and non-submitted authors and theircitation scores To again obtain a Spearman rank-order correlation eachdepartment in each table was then assigned a numerical rank and if necessaryan average numerical rank applied

Four sets of calculations were then performed

(1) a ranking for all citations received by each department irrespective ofwhether the author had been submitted or not

(2) a ranking by the total citations received by each department countingsubmitted authors only

JD596

716

(3) a ranking which shows the average citation rate for each departmentusing all departmental staff irrespective of whether the author had beensubmitted or not and

(4) a ranking which shows the average citation rate by department usingsubmitted departmental staff only

Issues arising out of the method adoptedCitation counts can however well carried out lead to erroneous counts Themethods adopted in this study go a long way to minimise such erroneouscounts A number of commonly voiced criticisms of citation counting (Liu1993 MacRoberts and MacRoberts 1989) were potentially present in this studyand are worthy of comment The general problem of mistakenly countingauthors with the same surname and initials but in different disciplines wasunderstood and careful inspection was exercised to limit such miscounting

This process was considerably helped by the RAE submissions whichidentireged the authorsrsquo interests and some of their publications Examination ofthe bibliographical record of the citing article for its subject and its fellowcitations helped minimise potential error Unquestionably however judgementwas required Where it was very difregcult to assess the validity of a particularcitation the record was included Such cases were infrequent and are not likelyto distort the total counts or their relative rankings In any case it is thecomparative ranking of the results which is being sought rather than anabsolutely accurate citation count

Where formerly only regrst named authors were cited now second andsubsequent authors are also cited in the ISIrsquos indexes thus the comment thatco-authors were uncited within the index and hence uncounted no longerapplies This has improved the citation count achieved and removed asigniregcant objection However Lange suggests that some caution needs to beexercised in assuming that a regrst and subsequent cited author search can bedeemed to be absolutely exact (Lange 2001) Self-citation has long beencriticised in citation analysis such practice it has been suggested leads todistortion and bias in conclusions drawn from counts The evidence andstudies conducted so far however show that this practice has little effect on theresults of citation analysis Snyder and Bonzi (1998) in a study whichexamined the patterns of self-citation in six disciplines clearly showed itsprevalence but were able to demonstrate that the patterns of self-citation arevery similar between disciplines and that the lowest self-citation rate of 3 percent was found in the humanities including presumably archaeology

An author making a submission to the RAE may choose material that hasbeen published at any time during the qualifying period This may make theitems submitted potentially ordfoldordm or very ordfnewordm material Counting thecitations to a population of ordfnewordm articles against ordfoldordm could easily give askewed result in favour of the old articles given that more citations are likely to

Archaeology andthe 2001 RAE

717

accrue to the older material We did not examine this in detail but a simpleoverview indicated that the pattern of age of items submitted did not differmuch between the departments examined

Humanities scholars tend to publish more monographs than those in otherdisciplines A random sample of 40 submitted authors examined here credited atleast 32 of them with either having published a monograph or chapter in onesubmitted for peer review In many cases they had published more than onemonograph or chapter in the assessment period Cronin et al (1997) have examinedthe differing author citation patterns when comparing journals and monographs inthe regeld of sociology They suggest that journals and monographs attract twoquite distinct populations of authors amongst the very highly cited this is certainlya concern and it can be assumed that insufregcient citations totals will result fromthis However we had no evidence that this affected one department more thananother and it is worth reiterating that it is the comparative ranking of citationcounts that is studied here rather than the precise count

A criticism may be levelled at the scope and coverage of the AHCI in termsof its bias towards USAEuropean publications as a distorting feature whenmaking citation counts (Braun et al 2000) Therefore where an author hasmade contributions to non-USAEuropean literature there is less likelihood ofhis or her work being noted V Nanda of University College London did notreceive any citations for his work which has been published in Asia LikewisePM Carroll of the University of Shefregeld did not receive any citations to hisworks when published in German While this is not conclusive evidence as toany particular bias the work of Braun et al (2000) clearly shows a strongpreference for a USAEuropean dominance of journal titles and publisherswithin Web of Science Examining the archaeology journal list in the AHCIconregrms a preference for USAEuropean journals Table II shows thedistribution of journal title by place of publication

This shows there is a deregnite bias toward USAEuropean journals so thelikelihood of regnding the cited work of UK archaeologists is considerable plusmnunless they publish elsewhere This bias in practice adds credibility to thisparticular study This it is contended will help support the results obtainedand the conclusions drawn

Place of publication Number of journals

Denmark 1France 4Germany 3Israel 1South Africa 1UK 10USA 17Total 37

Table IIDistribution of journalsindexed by ISI Web ofScience for Archaeology

JD596

718

ResultsThe results have been summarised in a number of Tables These Tables giveseveral views of the data notably by a ranking of the RAE score obtained byeach archaeology department and several different citation statistics Allcorrelations found were signiregcant thus conregrming earlier studies We alsofound that correlations for just staff returned for assessment is even higherthereby arguing that this is an even better predictor of RAE results

Table III gives an overall view of the RAE score obtained by eachdepartment and a comparative ranking by the four basic citation statistics thathave been calculated for each of them Table IV describes the core statisticscollected and of the range of values obtained for each of the departments

University name

RAEresult2001

Ranking bysubmitted

staffcitations

Rankingby all

citations

Ranking byaverage

citations forsubmitted staff

only

Ranking byaverage

citations for allstaff members

Trinity College Carmarthen 3b 25 23 20 24King Alfredrsquos College

Winchester 3a 24 24 26 25University of Wales College

Newport 3a 23 25 25 26University of Edinburgh 3a 21 21 22 21Bournemouth University 3a 18 18 21 20University of Newcastle 3a 17 17 17 17University of York 3a 15 15 18 18Birkbeck College (one staff

member only) 4 26 26 12 13University of Bristol 4 22 22 24 23University of Nottingham 4 19 19 15 15University of Birmingham 4 16 16 23 22University of Glasgow 4 14 14 19 19University of Wales Lampeter 4 13 11 13 9University of Exeter 5 20 20 16 16University of Liverpool 5 12 13 11 11Cardiff University 5 11 12 7 8University of Leicester 5 9 9 14 14University of Southampton 5 8 8 8 7University of Bradford 5 7 6 5 4University of Durham 5 6 7 9 10University of Shefregeld 5 5 5 4 3University College London 5 4 4 10 12The Queens University of

Belfast 5 2 2 1 1University of Reading 5 10 10 6 6University of Cambridge 5 3 3 3 5University of Oxford 5 1 1 1 2

Table IIIComparative rankingwith the RAE rating

result 2001 staffcitation counts and

citation count averagesfor UoA 58

(archaeology)

Archaeology andthe 2001 RAE

719

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Bir

kbec

kC

olle

ge

60

61

06

000

006

00U

niv

ersi

tyof

Bir

min

gham

660

6630

22

200

002

06B

ourn

em

outh

Univ

ersi

ty47

047

160

294

000

294

Univ

ersi

tyof

Bra

dfo

rd23

878

316

193

125

326

00

143

6U

niv

ersi

tyof

Bri

stol

370

3718

02

060

002

06U

niv

ersi

tyof

Cam

bri

dge

811

8889

943

2318

86

383

136

2U

niv

ersi

tyof

Durh

am25

22

254

363

700

067

651

Univ

ersi

tyof

Exet

er41

041

110

373

000

373

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

115

167

21

572

501

78U

niv

ersi

tyof

Lei

cest

er16

63

169

352

474

150

457

Univ

ersi

tyof

Liv

erpoo

l11

10

111

180

617

000

617

Univ

ersi

tyof

New

cast

le51

657

152

340

300

335

Univ

ersi

tyof

Not

tingham

450

4512

03

750

003

75U

niv

ersi

tyof

Oxfo

rd1

192

761

268

5911

202

06

9118

11

Univ

ersi

tyof

Rea

din

g16

40

164

200

820

000

820

Univ

ersi

tyof

Shef

regel

d31

462

376

202

157

031

00

170

9U

niv

ersi

tyof

Sou

tham

pto

n22

423

247

303

747

767

748

Univ

ersi

tyC

olle

ge

Lon

don

550

955

988

46

252

256

08U

niv

ersi

tyof

Yor

k77

077

230

335

000

335

Univ

ersi

tyof

Edin

burg

h38

038

140

271

000

271

Univ

ersi

tyof

Gla

sgow

103

010

331

03

320

003

32

(continued

)

Table IVSummary results ofcitation counts staffnumbers and citationcount averages for UoA58 (archaeology)

JD596

720

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Tri

nit

yC

olle

ge

Car

mar

then

612

182

73

001

712

00C

ardif

fU

niv

ersi

ty14

30

143

191

753

000

715

Univ

ersi

tyof

Wal

esL

ampet

er10

653

159

195

558

106

06

63U

niv

ersi

tyof

Wal

esC

olle

ge

New

por

t14

014

80

175

000

175

The

Quee

ns

Univ

ersi

tyof

Bel

fast

977

698

315

365

13

200

546

1G

rand

tota

ls5

790

423

621

360

973

951

579

911

Table IV

Archaeology andthe 2001 RAE

721

Calculations using the core data and the RAE score give the four measures ofcorrelation required Two of the measures deal with total citation counts andthe other two are concerned with average citations per member of staff The2001 RAE rating achieved by each archaeology department and theirrespective citation statistics are shown in Table V Table V has been sortedwithin each RAE rating by the number of submitted staff citations receivedgiving a snapshot view of the range of citations received and also how thiscompares between ratings To compare the 1996 and 2001 RAE ratingsTable VI shows the scores obtained in both assessments and the movementbetween the two periods

Table VII summarises the results obtained for the Spearman rank-ordercorrelation analyses for the four differing citation measures that were usedThe correlation scores have been rounded to two signiregcant reggures Each ofthese correlation scores is highly signiregcant ( p 001)

The results obtained by Oppenheim in his earlier study on archaeology areshown in Table VIII for comparison They are consistent with the currentresults

Analysis of results682 staff members were checked for their citation counts Between them theyaccumulated 6213 citations although 262 of the authors did not receive anycitations at all in the assessment period The individual citation count rangedfrom 0 to 565 with an overall average of 911 Of the cited authors just over 50per cent of them have a citation level of between one and six citations each Theexceptional score of 565 citations obtained by PJ Reimer can be attributed tojust three articles These were articles dealing with radiocarbon dating linkedto dendrochronolgy and were as follows

Quaternary Science Reviews 1996 15(7) 655 (10 citations)

Radiocarbon 1998 40(3) 1041 (491 citations)

Radiocarbon 1998 40(3) 1127 (64 citations)

Reimerrsquos nearest rival was P Forster from Cambridge University with 471citations from four articles They deal with DNA studies tracing the migrationof humans from Africa

American Journal of Human Genetics 1996 59(4) 935 (108 citations)

American Journal of Human Genetics 1996 61(3) 691 (73 citations)

American Journal of Human Genetics 1996 59(1) 185 (164 citations)

Genetics 1995 141(2) 743 (126 citations)

These two authors account for one-sixth of all the citations counted and theirarticles appear in just four journals all of which are available in both print andelectronic form

JD596

722

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

(3) a ranking which shows the average citation rate for each departmentusing all departmental staff irrespective of whether the author had beensubmitted or not and

(4) a ranking which shows the average citation rate by department usingsubmitted departmental staff only

Issues arising out of the method adoptedCitation counts can however well carried out lead to erroneous counts Themethods adopted in this study go a long way to minimise such erroneouscounts A number of commonly voiced criticisms of citation counting (Liu1993 MacRoberts and MacRoberts 1989) were potentially present in this studyand are worthy of comment The general problem of mistakenly countingauthors with the same surname and initials but in different disciplines wasunderstood and careful inspection was exercised to limit such miscounting

This process was considerably helped by the RAE submissions whichidentireged the authorsrsquo interests and some of their publications Examination ofthe bibliographical record of the citing article for its subject and its fellowcitations helped minimise potential error Unquestionably however judgementwas required Where it was very difregcult to assess the validity of a particularcitation the record was included Such cases were infrequent and are not likelyto distort the total counts or their relative rankings In any case it is thecomparative ranking of the results which is being sought rather than anabsolutely accurate citation count

Where formerly only regrst named authors were cited now second andsubsequent authors are also cited in the ISIrsquos indexes thus the comment thatco-authors were uncited within the index and hence uncounted no longerapplies This has improved the citation count achieved and removed asigniregcant objection However Lange suggests that some caution needs to beexercised in assuming that a regrst and subsequent cited author search can bedeemed to be absolutely exact (Lange 2001) Self-citation has long beencriticised in citation analysis such practice it has been suggested leads todistortion and bias in conclusions drawn from counts The evidence andstudies conducted so far however show that this practice has little effect on theresults of citation analysis Snyder and Bonzi (1998) in a study whichexamined the patterns of self-citation in six disciplines clearly showed itsprevalence but were able to demonstrate that the patterns of self-citation arevery similar between disciplines and that the lowest self-citation rate of 3 percent was found in the humanities including presumably archaeology

An author making a submission to the RAE may choose material that hasbeen published at any time during the qualifying period This may make theitems submitted potentially ordfoldordm or very ordfnewordm material Counting thecitations to a population of ordfnewordm articles against ordfoldordm could easily give askewed result in favour of the old articles given that more citations are likely to

Archaeology andthe 2001 RAE

717

accrue to the older material We did not examine this in detail but a simpleoverview indicated that the pattern of age of items submitted did not differmuch between the departments examined

Humanities scholars tend to publish more monographs than those in otherdisciplines A random sample of 40 submitted authors examined here credited atleast 32 of them with either having published a monograph or chapter in onesubmitted for peer review In many cases they had published more than onemonograph or chapter in the assessment period Cronin et al (1997) have examinedthe differing author citation patterns when comparing journals and monographs inthe regeld of sociology They suggest that journals and monographs attract twoquite distinct populations of authors amongst the very highly cited this is certainlya concern and it can be assumed that insufregcient citations totals will result fromthis However we had no evidence that this affected one department more thananother and it is worth reiterating that it is the comparative ranking of citationcounts that is studied here rather than the precise count

A criticism may be levelled at the scope and coverage of the AHCI in termsof its bias towards USAEuropean publications as a distorting feature whenmaking citation counts (Braun et al 2000) Therefore where an author hasmade contributions to non-USAEuropean literature there is less likelihood ofhis or her work being noted V Nanda of University College London did notreceive any citations for his work which has been published in Asia LikewisePM Carroll of the University of Shefregeld did not receive any citations to hisworks when published in German While this is not conclusive evidence as toany particular bias the work of Braun et al (2000) clearly shows a strongpreference for a USAEuropean dominance of journal titles and publisherswithin Web of Science Examining the archaeology journal list in the AHCIconregrms a preference for USAEuropean journals Table II shows thedistribution of journal title by place of publication

This shows there is a deregnite bias toward USAEuropean journals so thelikelihood of regnding the cited work of UK archaeologists is considerable plusmnunless they publish elsewhere This bias in practice adds credibility to thisparticular study This it is contended will help support the results obtainedand the conclusions drawn

Place of publication Number of journals

Denmark 1France 4Germany 3Israel 1South Africa 1UK 10USA 17Total 37

Table IIDistribution of journalsindexed by ISI Web ofScience for Archaeology

JD596

718

ResultsThe results have been summarised in a number of Tables These Tables giveseveral views of the data notably by a ranking of the RAE score obtained byeach archaeology department and several different citation statistics Allcorrelations found were signiregcant thus conregrming earlier studies We alsofound that correlations for just staff returned for assessment is even higherthereby arguing that this is an even better predictor of RAE results

Table III gives an overall view of the RAE score obtained by eachdepartment and a comparative ranking by the four basic citation statistics thathave been calculated for each of them Table IV describes the core statisticscollected and of the range of values obtained for each of the departments

University name

RAEresult2001

Ranking bysubmitted

staffcitations

Rankingby all

citations

Ranking byaverage

citations forsubmitted staff

only

Ranking byaverage

citations for allstaff members

Trinity College Carmarthen 3b 25 23 20 24King Alfredrsquos College

Winchester 3a 24 24 26 25University of Wales College

Newport 3a 23 25 25 26University of Edinburgh 3a 21 21 22 21Bournemouth University 3a 18 18 21 20University of Newcastle 3a 17 17 17 17University of York 3a 15 15 18 18Birkbeck College (one staff

member only) 4 26 26 12 13University of Bristol 4 22 22 24 23University of Nottingham 4 19 19 15 15University of Birmingham 4 16 16 23 22University of Glasgow 4 14 14 19 19University of Wales Lampeter 4 13 11 13 9University of Exeter 5 20 20 16 16University of Liverpool 5 12 13 11 11Cardiff University 5 11 12 7 8University of Leicester 5 9 9 14 14University of Southampton 5 8 8 8 7University of Bradford 5 7 6 5 4University of Durham 5 6 7 9 10University of Shefregeld 5 5 5 4 3University College London 5 4 4 10 12The Queens University of

Belfast 5 2 2 1 1University of Reading 5 10 10 6 6University of Cambridge 5 3 3 3 5University of Oxford 5 1 1 1 2

Table IIIComparative rankingwith the RAE rating

result 2001 staffcitation counts and

citation count averagesfor UoA 58

(archaeology)

Archaeology andthe 2001 RAE

719

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Bir

kbec

kC

olle

ge

60

61

06

000

006

00U

niv

ersi

tyof

Bir

min

gham

660

6630

22

200

002

06B

ourn

em

outh

Univ

ersi

ty47

047

160

294

000

294

Univ

ersi

tyof

Bra

dfo

rd23

878

316

193

125

326

00

143

6U

niv

ersi

tyof

Bri

stol

370

3718

02

060

002

06U

niv

ersi

tyof

Cam

bri

dge

811

8889

943

2318

86

383

136

2U

niv

ersi

tyof

Durh

am25

22

254

363

700

067

651

Univ

ersi

tyof

Exet

er41

041

110

373

000

373

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

115

167

21

572

501

78U

niv

ersi

tyof

Lei

cest

er16

63

169

352

474

150

457

Univ

ersi

tyof

Liv

erpoo

l11

10

111

180

617

000

617

Univ

ersi

tyof

New

cast

le51

657

152

340

300

335

Univ

ersi

tyof

Not

tingham

450

4512

03

750

003

75U

niv

ersi

tyof

Oxfo

rd1

192

761

268

5911

202

06

9118

11

Univ

ersi

tyof

Rea

din

g16

40

164

200

820

000

820

Univ

ersi

tyof

Shef

regel

d31

462

376

202

157

031

00

170

9U

niv

ersi

tyof

Sou

tham

pto

n22

423

247

303

747

767

748

Univ

ersi

tyC

olle

ge

Lon

don

550

955

988

46

252

256

08U

niv

ersi

tyof

Yor

k77

077

230

335

000

335

Univ

ersi

tyof

Edin

burg

h38

038

140

271

000

271

Univ

ersi

tyof

Gla

sgow

103

010

331

03

320

003

32

(continued

)

Table IVSummary results ofcitation counts staffnumbers and citationcount averages for UoA58 (archaeology)

JD596

720

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Tri

nit

yC

olle

ge

Car

mar

then

612

182

73

001

712

00C

ardif

fU

niv

ersi

ty14

30

143

191

753

000

715

Univ

ersi

tyof

Wal

esL

ampet

er10

653

159

195

558

106

06

63U

niv

ersi

tyof

Wal

esC

olle

ge

New

por

t14

014

80

175

000

175

The

Quee

ns

Univ

ersi

tyof

Bel

fast

977

698

315

365

13

200

546

1G

rand

tota

ls5

790

423

621

360

973

951

579

911

Table IV

Archaeology andthe 2001 RAE

721

Calculations using the core data and the RAE score give the four measures ofcorrelation required Two of the measures deal with total citation counts andthe other two are concerned with average citations per member of staff The2001 RAE rating achieved by each archaeology department and theirrespective citation statistics are shown in Table V Table V has been sortedwithin each RAE rating by the number of submitted staff citations receivedgiving a snapshot view of the range of citations received and also how thiscompares between ratings To compare the 1996 and 2001 RAE ratingsTable VI shows the scores obtained in both assessments and the movementbetween the two periods

Table VII summarises the results obtained for the Spearman rank-ordercorrelation analyses for the four differing citation measures that were usedThe correlation scores have been rounded to two signiregcant reggures Each ofthese correlation scores is highly signiregcant ( p 001)

The results obtained by Oppenheim in his earlier study on archaeology areshown in Table VIII for comparison They are consistent with the currentresults

Analysis of results682 staff members were checked for their citation counts Between them theyaccumulated 6213 citations although 262 of the authors did not receive anycitations at all in the assessment period The individual citation count rangedfrom 0 to 565 with an overall average of 911 Of the cited authors just over 50per cent of them have a citation level of between one and six citations each Theexceptional score of 565 citations obtained by PJ Reimer can be attributed tojust three articles These were articles dealing with radiocarbon dating linkedto dendrochronolgy and were as follows

Quaternary Science Reviews 1996 15(7) 655 (10 citations)

Radiocarbon 1998 40(3) 1041 (491 citations)

Radiocarbon 1998 40(3) 1127 (64 citations)

Reimerrsquos nearest rival was P Forster from Cambridge University with 471citations from four articles They deal with DNA studies tracing the migrationof humans from Africa

American Journal of Human Genetics 1996 59(4) 935 (108 citations)

American Journal of Human Genetics 1996 61(3) 691 (73 citations)

American Journal of Human Genetics 1996 59(1) 185 (164 citations)

Genetics 1995 141(2) 743 (126 citations)

These two authors account for one-sixth of all the citations counted and theirarticles appear in just four journals all of which are available in both print andelectronic form

JD596

722

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

accrue to the older material We did not examine this in detail but a simpleoverview indicated that the pattern of age of items submitted did not differmuch between the departments examined

Humanities scholars tend to publish more monographs than those in otherdisciplines A random sample of 40 submitted authors examined here credited atleast 32 of them with either having published a monograph or chapter in onesubmitted for peer review In many cases they had published more than onemonograph or chapter in the assessment period Cronin et al (1997) have examinedthe differing author citation patterns when comparing journals and monographs inthe regeld of sociology They suggest that journals and monographs attract twoquite distinct populations of authors amongst the very highly cited this is certainlya concern and it can be assumed that insufregcient citations totals will result fromthis However we had no evidence that this affected one department more thananother and it is worth reiterating that it is the comparative ranking of citationcounts that is studied here rather than the precise count

A criticism may be levelled at the scope and coverage of the AHCI in termsof its bias towards USAEuropean publications as a distorting feature whenmaking citation counts (Braun et al 2000) Therefore where an author hasmade contributions to non-USAEuropean literature there is less likelihood ofhis or her work being noted V Nanda of University College London did notreceive any citations for his work which has been published in Asia LikewisePM Carroll of the University of Shefregeld did not receive any citations to hisworks when published in German While this is not conclusive evidence as toany particular bias the work of Braun et al (2000) clearly shows a strongpreference for a USAEuropean dominance of journal titles and publisherswithin Web of Science Examining the archaeology journal list in the AHCIconregrms a preference for USAEuropean journals Table II shows thedistribution of journal title by place of publication

This shows there is a deregnite bias toward USAEuropean journals so thelikelihood of regnding the cited work of UK archaeologists is considerable plusmnunless they publish elsewhere This bias in practice adds credibility to thisparticular study This it is contended will help support the results obtainedand the conclusions drawn

Place of publication Number of journals

Denmark 1France 4Germany 3Israel 1South Africa 1UK 10USA 17Total 37

Table IIDistribution of journalsindexed by ISI Web ofScience for Archaeology

JD596

718

ResultsThe results have been summarised in a number of Tables These Tables giveseveral views of the data notably by a ranking of the RAE score obtained byeach archaeology department and several different citation statistics Allcorrelations found were signiregcant thus conregrming earlier studies We alsofound that correlations for just staff returned for assessment is even higherthereby arguing that this is an even better predictor of RAE results

Table III gives an overall view of the RAE score obtained by eachdepartment and a comparative ranking by the four basic citation statistics thathave been calculated for each of them Table IV describes the core statisticscollected and of the range of values obtained for each of the departments

University name

RAEresult2001

Ranking bysubmitted

staffcitations

Rankingby all

citations

Ranking byaverage

citations forsubmitted staff

only

Ranking byaverage

citations for allstaff members

Trinity College Carmarthen 3b 25 23 20 24King Alfredrsquos College

Winchester 3a 24 24 26 25University of Wales College

Newport 3a 23 25 25 26University of Edinburgh 3a 21 21 22 21Bournemouth University 3a 18 18 21 20University of Newcastle 3a 17 17 17 17University of York 3a 15 15 18 18Birkbeck College (one staff

member only) 4 26 26 12 13University of Bristol 4 22 22 24 23University of Nottingham 4 19 19 15 15University of Birmingham 4 16 16 23 22University of Glasgow 4 14 14 19 19University of Wales Lampeter 4 13 11 13 9University of Exeter 5 20 20 16 16University of Liverpool 5 12 13 11 11Cardiff University 5 11 12 7 8University of Leicester 5 9 9 14 14University of Southampton 5 8 8 8 7University of Bradford 5 7 6 5 4University of Durham 5 6 7 9 10University of Shefregeld 5 5 5 4 3University College London 5 4 4 10 12The Queens University of

Belfast 5 2 2 1 1University of Reading 5 10 10 6 6University of Cambridge 5 3 3 3 5University of Oxford 5 1 1 1 2

Table IIIComparative rankingwith the RAE rating

result 2001 staffcitation counts and

citation count averagesfor UoA 58

(archaeology)

Archaeology andthe 2001 RAE

719

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Bir

kbec

kC

olle

ge

60

61

06

000

006

00U

niv

ersi

tyof

Bir

min

gham

660

6630

22

200

002

06B

ourn

em

outh

Univ

ersi

ty47

047

160

294

000

294

Univ

ersi

tyof

Bra

dfo

rd23

878

316

193

125

326

00

143

6U

niv

ersi

tyof

Bri

stol

370

3718

02

060

002

06U

niv

ersi

tyof

Cam

bri

dge

811

8889

943

2318

86

383

136

2U

niv

ersi

tyof

Durh

am25

22

254

363

700

067

651

Univ

ersi

tyof

Exet

er41

041

110

373

000

373

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

115

167

21

572

501

78U

niv

ersi

tyof

Lei

cest

er16

63

169

352

474

150

457

Univ

ersi

tyof

Liv

erpoo

l11

10

111

180

617

000

617

Univ

ersi

tyof

New

cast

le51

657

152

340

300

335

Univ

ersi

tyof

Not

tingham

450

4512

03

750

003

75U

niv

ersi

tyof

Oxfo

rd1

192

761

268

5911

202

06

9118

11

Univ

ersi

tyof

Rea

din

g16

40

164

200

820

000

820

Univ

ersi

tyof

Shef

regel

d31

462

376

202

157

031

00

170

9U

niv

ersi

tyof

Sou

tham

pto

n22

423

247

303

747

767

748

Univ

ersi

tyC

olle

ge

Lon

don

550

955

988

46

252

256

08U

niv

ersi

tyof

Yor

k77

077

230

335

000

335

Univ

ersi

tyof

Edin

burg

h38

038

140

271

000

271

Univ

ersi

tyof

Gla

sgow

103

010

331

03

320

003

32

(continued

)

Table IVSummary results ofcitation counts staffnumbers and citationcount averages for UoA58 (archaeology)

JD596

720

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Tri

nit

yC

olle

ge

Car

mar

then

612

182

73

001

712

00C

ardif

fU

niv

ersi

ty14

30

143

191

753

000

715

Univ

ersi

tyof

Wal

esL

ampet

er10

653

159

195

558

106

06

63U

niv

ersi

tyof

Wal

esC

olle

ge

New

por

t14

014

80

175

000

175

The

Quee

ns

Univ

ersi

tyof

Bel

fast

977

698

315

365

13

200

546

1G

rand

tota

ls5

790

423

621

360

973

951

579

911

Table IV

Archaeology andthe 2001 RAE

721

Calculations using the core data and the RAE score give the four measures ofcorrelation required Two of the measures deal with total citation counts andthe other two are concerned with average citations per member of staff The2001 RAE rating achieved by each archaeology department and theirrespective citation statistics are shown in Table V Table V has been sortedwithin each RAE rating by the number of submitted staff citations receivedgiving a snapshot view of the range of citations received and also how thiscompares between ratings To compare the 1996 and 2001 RAE ratingsTable VI shows the scores obtained in both assessments and the movementbetween the two periods

Table VII summarises the results obtained for the Spearman rank-ordercorrelation analyses for the four differing citation measures that were usedThe correlation scores have been rounded to two signiregcant reggures Each ofthese correlation scores is highly signiregcant ( p 001)

The results obtained by Oppenheim in his earlier study on archaeology areshown in Table VIII for comparison They are consistent with the currentresults

Analysis of results682 staff members were checked for their citation counts Between them theyaccumulated 6213 citations although 262 of the authors did not receive anycitations at all in the assessment period The individual citation count rangedfrom 0 to 565 with an overall average of 911 Of the cited authors just over 50per cent of them have a citation level of between one and six citations each Theexceptional score of 565 citations obtained by PJ Reimer can be attributed tojust three articles These were articles dealing with radiocarbon dating linkedto dendrochronolgy and were as follows

Quaternary Science Reviews 1996 15(7) 655 (10 citations)

Radiocarbon 1998 40(3) 1041 (491 citations)

Radiocarbon 1998 40(3) 1127 (64 citations)

Reimerrsquos nearest rival was P Forster from Cambridge University with 471citations from four articles They deal with DNA studies tracing the migrationof humans from Africa

American Journal of Human Genetics 1996 59(4) 935 (108 citations)

American Journal of Human Genetics 1996 61(3) 691 (73 citations)

American Journal of Human Genetics 1996 59(1) 185 (164 citations)

Genetics 1995 141(2) 743 (126 citations)

These two authors account for one-sixth of all the citations counted and theirarticles appear in just four journals all of which are available in both print andelectronic form

JD596

722

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

ResultsThe results have been summarised in a number of Tables These Tables giveseveral views of the data notably by a ranking of the RAE score obtained byeach archaeology department and several different citation statistics Allcorrelations found were signiregcant thus conregrming earlier studies We alsofound that correlations for just staff returned for assessment is even higherthereby arguing that this is an even better predictor of RAE results

Table III gives an overall view of the RAE score obtained by eachdepartment and a comparative ranking by the four basic citation statistics thathave been calculated for each of them Table IV describes the core statisticscollected and of the range of values obtained for each of the departments

University name

RAEresult2001

Ranking bysubmitted

staffcitations

Rankingby all

citations

Ranking byaverage

citations forsubmitted staff

only

Ranking byaverage

citations for allstaff members

Trinity College Carmarthen 3b 25 23 20 24King Alfredrsquos College

Winchester 3a 24 24 26 25University of Wales College

Newport 3a 23 25 25 26University of Edinburgh 3a 21 21 22 21Bournemouth University 3a 18 18 21 20University of Newcastle 3a 17 17 17 17University of York 3a 15 15 18 18Birkbeck College (one staff

member only) 4 26 26 12 13University of Bristol 4 22 22 24 23University of Nottingham 4 19 19 15 15University of Birmingham 4 16 16 23 22University of Glasgow 4 14 14 19 19University of Wales Lampeter 4 13 11 13 9University of Exeter 5 20 20 16 16University of Liverpool 5 12 13 11 11Cardiff University 5 11 12 7 8University of Leicester 5 9 9 14 14University of Southampton 5 8 8 8 7University of Bradford 5 7 6 5 4University of Durham 5 6 7 9 10University of Shefregeld 5 5 5 4 3University College London 5 4 4 10 12The Queens University of

Belfast 5 2 2 1 1University of Reading 5 10 10 6 6University of Cambridge 5 3 3 3 5University of Oxford 5 1 1 1 2

Table IIIComparative rankingwith the RAE rating

result 2001 staffcitation counts and

citation count averagesfor UoA 58

(archaeology)

Archaeology andthe 2001 RAE

719

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Bir

kbec

kC

olle

ge

60

61

06

000

006

00U

niv

ersi

tyof

Bir

min

gham

660

6630

22

200

002

06B

ourn

em

outh

Univ

ersi

ty47

047

160

294

000

294

Univ

ersi

tyof

Bra

dfo

rd23

878

316

193

125

326

00

143

6U

niv

ersi

tyof

Bri

stol

370

3718

02

060

002

06U

niv

ersi

tyof

Cam

bri

dge

811

8889

943

2318

86

383

136

2U

niv

ersi

tyof

Durh

am25

22

254

363

700

067

651

Univ

ersi

tyof

Exet

er41

041

110

373

000

373

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

115

167

21

572

501

78U

niv

ersi

tyof

Lei

cest

er16

63

169

352

474

150

457

Univ

ersi

tyof

Liv

erpoo

l11

10

111

180

617

000

617

Univ

ersi

tyof

New

cast

le51

657

152

340

300

335

Univ

ersi

tyof

Not

tingham

450

4512

03

750

003

75U

niv

ersi

tyof

Oxfo

rd1

192

761

268

5911

202

06

9118

11

Univ

ersi

tyof

Rea

din

g16

40

164

200

820

000

820

Univ

ersi

tyof

Shef

regel

d31

462

376

202

157

031

00

170

9U

niv

ersi

tyof

Sou

tham

pto

n22

423

247

303

747

767

748

Univ

ersi

tyC

olle

ge

Lon

don

550

955

988

46

252

256

08U

niv

ersi

tyof

Yor

k77

077

230

335

000

335

Univ

ersi

tyof

Edin

burg

h38

038

140

271

000

271

Univ

ersi

tyof

Gla

sgow

103

010

331

03

320

003

32

(continued

)

Table IVSummary results ofcitation counts staffnumbers and citationcount averages for UoA58 (archaeology)

JD596

720

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Tri

nit

yC

olle

ge

Car

mar

then

612

182

73

001

712

00C

ardif

fU

niv

ersi

ty14

30

143

191

753

000

715

Univ

ersi

tyof

Wal

esL

ampet

er10

653

159

195

558

106

06

63U

niv

ersi

tyof

Wal

esC

olle

ge

New

por

t14

014

80

175

000

175

The

Quee

ns

Univ

ersi

tyof

Bel

fast

977

698

315

365

13

200

546

1G

rand

tota

ls5

790

423

621

360

973

951

579

911

Table IV

Archaeology andthe 2001 RAE

721

Calculations using the core data and the RAE score give the four measures ofcorrelation required Two of the measures deal with total citation counts andthe other two are concerned with average citations per member of staff The2001 RAE rating achieved by each archaeology department and theirrespective citation statistics are shown in Table V Table V has been sortedwithin each RAE rating by the number of submitted staff citations receivedgiving a snapshot view of the range of citations received and also how thiscompares between ratings To compare the 1996 and 2001 RAE ratingsTable VI shows the scores obtained in both assessments and the movementbetween the two periods

Table VII summarises the results obtained for the Spearman rank-ordercorrelation analyses for the four differing citation measures that were usedThe correlation scores have been rounded to two signiregcant reggures Each ofthese correlation scores is highly signiregcant ( p 001)

The results obtained by Oppenheim in his earlier study on archaeology areshown in Table VIII for comparison They are consistent with the currentresults

Analysis of results682 staff members were checked for their citation counts Between them theyaccumulated 6213 citations although 262 of the authors did not receive anycitations at all in the assessment period The individual citation count rangedfrom 0 to 565 with an overall average of 911 Of the cited authors just over 50per cent of them have a citation level of between one and six citations each Theexceptional score of 565 citations obtained by PJ Reimer can be attributed tojust three articles These were articles dealing with radiocarbon dating linkedto dendrochronolgy and were as follows

Quaternary Science Reviews 1996 15(7) 655 (10 citations)

Radiocarbon 1998 40(3) 1041 (491 citations)

Radiocarbon 1998 40(3) 1127 (64 citations)

Reimerrsquos nearest rival was P Forster from Cambridge University with 471citations from four articles They deal with DNA studies tracing the migrationof humans from Africa

American Journal of Human Genetics 1996 59(4) 935 (108 citations)

American Journal of Human Genetics 1996 61(3) 691 (73 citations)

American Journal of Human Genetics 1996 59(1) 185 (164 citations)

Genetics 1995 141(2) 743 (126 citations)

These two authors account for one-sixth of all the citations counted and theirarticles appear in just four journals all of which are available in both print andelectronic form

JD596

722

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Bir

kbec

kC

olle

ge

60

61

06

000

006

00U

niv

ersi

tyof

Bir

min

gham

660

6630

22

200

002

06B

ourn

em

outh

Univ

ersi

ty47

047

160

294

000

294

Univ

ersi

tyof

Bra

dfo

rd23

878

316

193

125

326

00

143

6U

niv

ersi

tyof

Bri

stol

370

3718

02

060

002

06U

niv

ersi

tyof

Cam

bri

dge

811

8889

943

2318

86

383

136

2U

niv

ersi

tyof

Durh

am25

22

254

363

700

067

651

Univ

ersi

tyof

Exet

er41

041

110

373

000

373

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

115

167

21

572

501

78U

niv

ersi

tyof

Lei

cest

er16

63

169

352

474

150

457

Univ

ersi

tyof

Liv

erpoo

l11

10

111

180

617

000

617

Univ

ersi

tyof

New

cast

le51

657

152

340

300

335

Univ

ersi

tyof

Not

tingham

450

4512

03

750

003

75U

niv

ersi

tyof

Oxfo

rd1

192

761

268

5911

202

06

9118

11

Univ

ersi

tyof

Rea

din

g16

40

164

200

820

000

820

Univ

ersi

tyof

Shef

regel

d31

462

376

202

157

031

00

170

9U

niv

ersi

tyof

Sou

tham

pto

n22

423

247

303

747

767

748

Univ

ersi

tyC

olle

ge

Lon

don

550

955

988

46

252

256

08U

niv

ersi

tyof

Yor

k77

077

230

335

000

335

Univ

ersi

tyof

Edin

burg

h38

038

140

271

000

271

Univ

ersi

tyof

Gla

sgow

103

010

331

03

320

003

32

(continued

)

Table IVSummary results ofcitation counts staffnumbers and citationcount averages for UoA58 (archaeology)

JD596

720

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Tri

nit

yC

olle

ge

Car

mar

then

612

182

73

001

712

00C

ardif

fU

niv

ersi

ty14

30

143

191

753

000

715

Univ

ersi

tyof

Wal

esL

ampet

er10

653

159

195

558

106

06

63U

niv

ersi

tyof

Wal

esC

olle

ge

New

por

t14

014

80

175

000

175

The

Quee

ns

Univ

ersi

tyof

Bel

fast

977

698

315

365

13

200

546

1G

rand

tota

ls5

790

423

621

360

973

951

579

911

Table IV

Archaeology andthe 2001 RAE

721

Calculations using the core data and the RAE score give the four measures ofcorrelation required Two of the measures deal with total citation counts andthe other two are concerned with average citations per member of staff The2001 RAE rating achieved by each archaeology department and theirrespective citation statistics are shown in Table V Table V has been sortedwithin each RAE rating by the number of submitted staff citations receivedgiving a snapshot view of the range of citations received and also how thiscompares between ratings To compare the 1996 and 2001 RAE ratingsTable VI shows the scores obtained in both assessments and the movementbetween the two periods

Table VII summarises the results obtained for the Spearman rank-ordercorrelation analyses for the four differing citation measures that were usedThe correlation scores have been rounded to two signiregcant reggures Each ofthese correlation scores is highly signiregcant ( p 001)

The results obtained by Oppenheim in his earlier study on archaeology areshown in Table VIII for comparison They are consistent with the currentresults

Analysis of results682 staff members were checked for their citation counts Between them theyaccumulated 6213 citations although 262 of the authors did not receive anycitations at all in the assessment period The individual citation count rangedfrom 0 to 565 with an overall average of 911 Of the cited authors just over 50per cent of them have a citation level of between one and six citations each Theexceptional score of 565 citations obtained by PJ Reimer can be attributed tojust three articles These were articles dealing with radiocarbon dating linkedto dendrochronolgy and were as follows

Quaternary Science Reviews 1996 15(7) 655 (10 citations)

Radiocarbon 1998 40(3) 1041 (491 citations)

Radiocarbon 1998 40(3) 1127 (64 citations)

Reimerrsquos nearest rival was P Forster from Cambridge University with 471citations from four articles They deal with DNA studies tracing the migrationof humans from Africa

American Journal of Human Genetics 1996 59(4) 935 (108 citations)

American Journal of Human Genetics 1996 61(3) 691 (73 citations)

American Journal of Human Genetics 1996 59(1) 185 (164 citations)

Genetics 1995 141(2) 743 (126 citations)

These two authors account for one-sixth of all the citations counted and theirarticles appear in just four journals all of which are available in both print andelectronic form

JD596

722

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

Univ

ersi

tynam

e

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

all

cita

tion

s

Tot

alsu

bm

itte

dst

aff

Tot

alnon

-subm

itte

dst

aff

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

aff

mem

ber

s

Tri

nit

yC

olle

ge

Car

mar

then

612

182

73

001

712

00C

ardif

fU

niv

ersi

ty14

30

143

191

753

000

715

Univ

ersi

tyof

Wal

esL

ampet

er10

653

159

195

558

106

06

63U

niv

ersi

tyof

Wal

esC

olle

ge

New

por

t14

014

80

175

000

175

The

Quee

ns

Univ

ersi

tyof

Bel

fast

977

698

315

365

13

200

546

1G

rand

tota

ls5

790

423

621

360

973

951

579

911

Table IV

Archaeology andthe 2001 RAE

721

Calculations using the core data and the RAE score give the four measures ofcorrelation required Two of the measures deal with total citation counts andthe other two are concerned with average citations per member of staff The2001 RAE rating achieved by each archaeology department and theirrespective citation statistics are shown in Table V Table V has been sortedwithin each RAE rating by the number of submitted staff citations receivedgiving a snapshot view of the range of citations received and also how thiscompares between ratings To compare the 1996 and 2001 RAE ratingsTable VI shows the scores obtained in both assessments and the movementbetween the two periods

Table VII summarises the results obtained for the Spearman rank-ordercorrelation analyses for the four differing citation measures that were usedThe correlation scores have been rounded to two signiregcant reggures Each ofthese correlation scores is highly signiregcant ( p 001)

The results obtained by Oppenheim in his earlier study on archaeology areshown in Table VIII for comparison They are consistent with the currentresults

Analysis of results682 staff members were checked for their citation counts Between them theyaccumulated 6213 citations although 262 of the authors did not receive anycitations at all in the assessment period The individual citation count rangedfrom 0 to 565 with an overall average of 911 Of the cited authors just over 50per cent of them have a citation level of between one and six citations each Theexceptional score of 565 citations obtained by PJ Reimer can be attributed tojust three articles These were articles dealing with radiocarbon dating linkedto dendrochronolgy and were as follows

Quaternary Science Reviews 1996 15(7) 655 (10 citations)

Radiocarbon 1998 40(3) 1041 (491 citations)

Radiocarbon 1998 40(3) 1127 (64 citations)

Reimerrsquos nearest rival was P Forster from Cambridge University with 471citations from four articles They deal with DNA studies tracing the migrationof humans from Africa

American Journal of Human Genetics 1996 59(4) 935 (108 citations)

American Journal of Human Genetics 1996 61(3) 691 (73 citations)

American Journal of Human Genetics 1996 59(1) 185 (164 citations)

Genetics 1995 141(2) 743 (126 citations)

These two authors account for one-sixth of all the citations counted and theirarticles appear in just four journals all of which are available in both print andelectronic form

JD596

722

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

Calculations using the core data and the RAE score give the four measures ofcorrelation required Two of the measures deal with total citation counts andthe other two are concerned with average citations per member of staff The2001 RAE rating achieved by each archaeology department and theirrespective citation statistics are shown in Table V Table V has been sortedwithin each RAE rating by the number of submitted staff citations receivedgiving a snapshot view of the range of citations received and also how thiscompares between ratings To compare the 1996 and 2001 RAE ratingsTable VI shows the scores obtained in both assessments and the movementbetween the two periods

Table VII summarises the results obtained for the Spearman rank-ordercorrelation analyses for the four differing citation measures that were usedThe correlation scores have been rounded to two signiregcant reggures Each ofthese correlation scores is highly signiregcant ( p 001)

The results obtained by Oppenheim in his earlier study on archaeology areshown in Table VIII for comparison They are consistent with the currentresults

Analysis of results682 staff members were checked for their citation counts Between them theyaccumulated 6213 citations although 262 of the authors did not receive anycitations at all in the assessment period The individual citation count rangedfrom 0 to 565 with an overall average of 911 Of the cited authors just over 50per cent of them have a citation level of between one and six citations each Theexceptional score of 565 citations obtained by PJ Reimer can be attributed tojust three articles These were articles dealing with radiocarbon dating linkedto dendrochronolgy and were as follows

Quaternary Science Reviews 1996 15(7) 655 (10 citations)

Radiocarbon 1998 40(3) 1041 (491 citations)

Radiocarbon 1998 40(3) 1127 (64 citations)

Reimerrsquos nearest rival was P Forster from Cambridge University with 471citations from four articles They deal with DNA studies tracing the migrationof humans from Africa

American Journal of Human Genetics 1996 59(4) 935 (108 citations)

American Journal of Human Genetics 1996 61(3) 691 (73 citations)

American Journal of Human Genetics 1996 59(1) 185 (164 citations)

Genetics 1995 141(2) 743 (126 citations)

These two authors account for one-sixth of all the citations counted and theirarticles appear in just four journals all of which are available in both print andelectronic form

JD596

722

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

Univ

ersi

tynam

e

RA

Ere

sult

2001

Subm

itte

dst

aff

cita

tion

sN

on-s

ubm

itte

dst

aff

cita

tion

sT

otal

cita

tion

s

Aver

age

cita

tion

sper

subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sper

non

-subm

itte

dst

aff

mem

ber

Aver

age

cita

tion

sal

lst

affm

ember

s

Tri

nit

yC

olle

ge

Car

mar

then

3b6

1218

30

17

20

Kin

gA

lfre

drsquos

Col

lege

Win

ches

ter

3a11

516

16

25

18

Univ

ersi

tyof

Wal

esC

olle

ge

New

por

t3a

140

141

80

01

8U

niv

ersi

tyof

Edin

burg

h3a

380

382

70

02

7B

ourn

emou

thU

niv

ersi

ty3a

470

472

90

02

9U

niv

ersi

tyof

New

cast

le3a

516

573

43

03

4U

niv

ersi

tyof

Yor

k3a

770

773

30

03

3B

irkbec

kC

olle

ge

46

06

60

00

60

Univ

ersi

tyof

Bri

stol

437

037

21

00

21

Univ

ersi

tyof

Not

tingham

445

045

38

00

38

Univ

ersi

tyof

Bir

min

gham

466

066

22

00

21

Univ

ersi

tyof

Gla

sgow

410

30

103

33

00

33

Univ

ersi

tyof

Wal

esL

ampet

er4

106

5315

95

610

66

6U

niv

ersi

tyof

Exet

er5

410

413

70

03

7U

niv

ersi

tyof

Liv

erpoo

l5

111

011

16

20

06

2C

ardif

fU

niv

ersi

ty5

143

014

37

50

07

2U

niv

ersi

tyof

Lei

cest

er5

166

316

94

71

54

6U

niv

ersi

tyof

Sou

tham

pto

n5

224

2324

77

57

77

5U

niv

ersi

tyof

Bra

dfo

rd5

238

7831

612

526

014

4U

niv

ersi

tyof

Durh

am5

252

225

47

00

76

5U

niv

ersi

tyof

Shef

regel

d5

314

6237

615

731

017

1U

niv

ersi

tyC

olle

ge

Lon

don

555

09

559

63

23

61

The

Quee

ns

Univ

ersi

tyof

Bel

fast

597

76

983

651

20

546

Univ

ersi

tyof

Rea

din

g5

164

016

48

20

08

2U

niv

ersi

tyof

Cam

bri

dge

581

188

899

189

38

136

Univ

ersi

tyof

Oxfo

rd5

119

276

126

820

26

918

1

Table VResults with rankedRAE rating citationcounts and citation

count averages for UoA58 (archaeology)

Archaeology andthe 2001 RAE

723

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

University name RAE result 2001 RAE result 1996 Rating movement

Trinity College Carmarthen 3b 3b 0King Alfredrsquos College Winchester 3a 2 +University of Wales College Newport 3a 3b +University of Edinburgh 3a 4 plusmnBournemouth University 3a 3b +University of Newcastle 3a 3b +University of York 3a 4 plusmnBirkbeck College (one staff member only) 4 na naUniversity of Bristol 4 4 0University of Nottingham 4 3a +University of Birmingham 4 4 0University of Glasgow 4 4 0University of Wales Lampeter 4 3a +University of Exeter 5 3a +University of Liverpool 5 4 +Cardiff University 5 4 +University of Leicester 5 5 0University of Southampton 5 5 0University of Bradford 5 5 0University of Durham 5 5 0University of Shefregeld 5 5 plusmnUniversity College London 5 5 0The Queens University of Belfast 5 5 0University of Reading 5 5 +University of Cambridge 5 5 0University of Oxford 5 5 0

Table VIA comparison of theratings achieved overthe last twoassessments

Statistic Correlation

Total submitted staff citations 081Total of all staff citations 079Total staff average citations 085Total of all staff citations 084

Table VIISpearman rank-ordercorrelation coefregcientfor the 2001 RAEranking and citationcounts

Statistic Correlation

Total of all staff citations 082Total staff average citations 074

Source Oppenheim (1997)

Table VIIISpearman rank-ordercorrelation coefregcientfor the 1992 RAEranking and citationcounts for archaeology

JD596

724

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

Taking the top ten most cited authors and their collective citation counts showsthem to account for 1993 of the totals citations received ie 32 per cent of the6213 citations received by all authors Table I gives the details of the top tenmost cited authors and their afregliations

Birkbeck College obtained the lowest citation score of six for just one staffmember This low score would have by citation count alone placed it at thebottom of the ranking He did however achieve a four rating by assessmentand by average citation count here would have been rated as a borderline fouror regve Both Bradford and Lampeter Universities did not submit all of theirstaff for assessment The citation counts for those not submitted was asigniregcant percentage (68 per cent) of the total citation count achievedTable IX shows those universities with authors with a signiregcant number ofcitation counts who were not submitted

Carmarthen retained its rating of 3b with a citation count of six In the caseof Lampeter their ranking by total citation count would have placed them witha probable RAE rating of 5 against their current 4 however they did move froma 3a to the 4 rating in the current exercise Bradford would have remainedunchanged at 5 Perhaps the most signiregcant change however occurred forShefregeld who lost their previous 5 rating Had Shefregeld submitted KJEdwards who accounted for 58 of the non-submitted citations and incidentallythe highest citation count of all of their staff then perhaps they may haveretained their 5 rating Given the relative simplicity and accuracy of makingthese ranking assessments by citation counting the technique could be readilyused as a guide to help decision making in borderline cases Oppenheim (1996)has suggested that RAE co-ordinators would do well to consider a citationanalysis of their authors before selecting them for submission Proposals forthe next round of assessment in 20072008 suggest a staged approach whichwould give universities much more control of the process and which wouldallow panels to select whatever methods of assessment they felt were mostappropriate for their subject area (Roberts 2003)

The RAE submissions included 2342 separate items made up as shown inTable X

The 927 articles were submitted to 341 different journal titles with numbersranging from a single submission to a number of journals through to 72 articlespublished in Antiquity a UK-based journal Of the 37 archaeology journalsindexed by Web of Science 23 had articles submitted to them from amongst

University All citations Not submitted citations Percentage

Carmarthen 18 12 67Lampeter 159 53 33Bradford 316 78 25Shefregeld 376 62 16

Table IXPercentage of

non-submitted citations

Archaeology andthe 2001 RAE

725

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

those presented for assessment In total the number of articles submitted to the23 ISI journals was 203 22 per cent of the total number of articles presentedThe 23 journals represent only 67 per cent of the journals represented in thesubmissions

When 5- and 5-rated departments are considered separately it is evidentthat the concentration of publishing authors of ISI indexed journals is greaterthan the lower rated universities Table XI shows that there is a relationshipbetween the RAE rating and the number of ISI indexed archaeology journalsthat authors contribute to from each of the departments Given that the journalsselected by the ISI are high impact journals it is not surprising that they attractthe attention of leading authors

Ten universities improved their score 12 maintained their position and threelost a grade There was also one new entrant making up the total of 26departments This is in line with the general ordfrating inmacrationordm experienced inthe 2001 RAE

Overall the rankings obtained by citation analysis correlate very stronglywith the rankings achieved by the RAE assessment The results obtained arevirtually identical to those found by Oppenheim on the previous RAE Thisconsistency of results over the two RAEs using an almost identical methodsupports the notion that citation counting is a robust and reliable method formaking an initial ranking assessment of archaeology departments

DiscussionEarlier citation studies that attempted to regnd a correlation between citationcounts and the awarded RAE rating had to do so without knowing whichacademics had been submitted for assessment The researchers in these studiesgathered a listing of academics that they thought would be included in theassessment and then carried out a citation analysis of them The results fromthese studies showed a high correlation between the citation count of theacademics in a particular department and the RAE rating The citation studyundertaken here had the beneregt of knowing which academic authors had beensubmitted for the 2001 RAE and consequently could additionally quantifytheir citation counts alone and calculate a correlation between their scores andthe RAE rating The results from this study like its predecessors show asigniregcant correlation between citation count and the RAE rating Therobustness of the methods used and consequent results are consistent with theconregdence with which this assessment tool has been used here and in the pastThe process is not however perfect and it has its limitations

Monograph Conference Reports Internet Articles Other

n 1273 121 4 12 927 5Percentage 5436 517 017 051 3958 021

Table XBreakdown of the itemssubmitted for 2001 RAEassessment itemssubmitted andpercentage split

JD596

726

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

Overall the citation study ranked the universities in line with the RAE ratingHowever there were anomalies Birkbeck College achieved a 4 rating whilstsubmitting just one member of staff for assessment and recording only sixcitations Trinity College Carmarthen likewise recorded six citations with twostaff and retained its 3b rating None of the authors published in the ISIrsquosindexed journals Confusingly Carmarthen declined to submit seven other stafffor assessment one of which (M Patton) would have tripled its citation scoreBirkbeck with its average of six citations was correctly placed by the citationstudy at a borderline 45 rating but on total count should have been placedwith Carmarthen At the other extreme Shefregeld lost its 5 status whilst stillhaving a very high average citation count Belfast with three times the averagecitation rate of its nearest rival and top of the table did not gain 5 status butretained its 5 rating Belfastrsquos very high citation rate is attributable to PJReimerrsquos exceptional score of 565 citations These outliers at the extremes of therating scale where citation counts are very high or low is where the RAE panelhad to exercise difregcult judgements[2]

The RAE process and publishingThis study was supplemented with an interview with Professor Barker ofLeicester Universityrsquos School of Archaeology and Ancient History a member ofthe archaeology RAE panel In it he addressed a number of questionsregarding the effect of the RAE on individuals on departments and on workingmethods[2]

It would seem that there is a link between achieving 5 or 5 status andpublishing in high impact journals ie those identireged for indexing in theAHCI Table XI demonstrates that the higher the number of AHCI indexedjournals you publish in then the higher your RAE rating and citation scoreThis is most deregnitely not a cause and effect relationship and universitymanagers who try to make their academic staff only publish in high impactjournals are being astonishingly naotildeEgraveve Important papers tend to be submittedto prestigious journals Prestigious journals are indexed in the AHCI Importantpapers tend to be submitted to the RAE and tend to lead to higher RAE ratings

RAE assessment rating3-4 5 5 All

Articles submitted 297 443 187 927Number of universities 13 10 3 26Total submitting staff in each rating group 196 291 122 609Average number of articles per staff member 152 152 153 152ISI indexed journals authored 49 104 50 203Average number of ISI authored index articles per

university 4 10 17 8Staff members per ISI authored article 400 280 244 300

Table XIAnalysis of the articles

submitted forassessment includingISI indexed authored

articles

Archaeology andthe 2001 RAE

727

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

Professor Barker[2] was at pains to point out that where the article or book waspublished did not affect the panelrsquos judgement regarding its quality Thethreshold to obtain publication in these journals is that much higher than manyothers and so by default the quality of submissions will need to be higher thuspromoting better quality articles and hence a better RAE assessment

Conclusions and recommendationsThe citation study undertaken here demonstrates that a high correlation existsbetween the 2001 RAE scores obtained by archaeology departments and acitation count of the academics in those departments This result conregrms theevidence from other studies which also demonstrate the robustness andaccuracy of the correlation Staff-submitted correlation scores were marginallybetter than all-staff correlation scores but both were highly statisticallysigniregcantly correlated with RAE scores We hope that with the publication ofthese results the debate about the robustness and consistency of the correlationwill be laid to rest It remains of course right and proper that the implicationsof the correlation should be debated

Assessment of the quality of research output from UK universities willcontinue but how it should be conducted is open to debate a debate that isstrongly encouraged in the subtitle of (Roberts 2003) ordfIssued for consultationordmThere can be little doubt that whatever the changes in the future the quality ofresearch outputs will be a major component of the metrics adopted Whilstcitation analysis is not a perfect tool it is recommended that it should beadopted as the primary procedure for the initial ranking of universitydepartments Once complete the rankings could then be distributed to thepanels for consideration We do not recommend that citation counting would bethe only assessment tool For one thing by deregnition it only looks backwardsand takes no account of future plans Peer review of the written evidenceconsideration of other factors and the careful scrutiny of marginal cases mustcomplement the process Nonetheless we believe despite the well-knownsuspicion by academics of citation-based measures that there is a convincingcase that citation analysis should form the regrst part of any future assessmentof research quality

Notes

1 Ahmed T Oppenheim C and Parker S unpublished results

2 Personal communication from Professor Barker Member of the Archaeology RAE Panel in2001

References

Baird L and Oppenheim C (1994) ordfDo citations matterordm Journal of Information Science Vol 20No 1 pp 2-15

Bence V and Oppenheim C (2001) ordfJournals scholarly communication and the RAE a casestudy of the business and management sectorordm Serials Vol 14 No 3 pp 265-72

JD596

728

Bernard GW (2000) ordfHistory and Research Assessment Exercisesordm Oxford Review ofEducation Vol 26 No 1 pp 95-106

Braun T Glanzel W and Schubert A (2000) ordfHow balanced is the Science Citation Indexrsquosjournal coverageordm in Atkins H (Ed) The Web of Knowledge A Festschrift in Honor ofEugene Garregeld Information Today Medford NJ pp 251-77

Campbell K (1999) ordfJournal publishing journal reputation and the United Kingdomrsquos ResearchAssessment Exerciseordm Journal of Law and Society Vol 26 No 4 pp 470-501

Cronin B and Barsky Atkins H (2000) The Web of Knowledge A Festschrift in Honor ofEugene Garregeld Information Today Medford NJ

Cronin B Snyder H and Atkins H (1997) ordfComparative citation rankings of authors inmonographic and journal literature a study of sociologyordm Journal of DocumentationVol 53 No 3 pp 263-73

Garregeld E (1979) Citation Indexing plusmn Its Theory and Application in Science Technology andHumanities John Wiley amp Sons New York NY

Harnad S Carr L Brody T and Oppenheim C (2003) ordfMandated online RAE CVs linked touniversity eprint archives enhancing UK research impact and assessmentordm AriadneVol 35

HEFCE (2001a) Review of Research Report Consultation (March) (0117) available atwwwhefceacukpubshefce2001 (accessed 3 October 2002)

HEFCE (2001b) RAE 2001 Submissions available at wwwheroacukraesubmissions(accessed 20 August 2002)

Henkel M (1999) ordfThe modernisation of research evaluation the case of the UKordm HigherEducation Vol 38 No 1 pp 105-22

Holmes A and Oppenheim C (2001) ordfUse of citation analysis to predict the outcome of the 2001RAE for Unit of Assessment 61 Library and Information Managementordm InformationResearch Vol 6 No 2

Johnson I (2001) ordfLetter to the Editorordm Information Research Vol 6 No 2

Lange LL (2001) ordfCitation counts of multi-authored papers plusmn regrst-named authors and furtherauthorsordm Scientometrics Vol 52 No 3 pp 457-70

Liu M (1993) ordfProgress in documentation the complexities of citation practice a review ofcitation studiesordm Journal of Documentation Vol 49 No 4 pp 370-408

MacRoberts MH and MacRoberts BR (1989) ordfProblems of citation analysis a critical reviewordmJournal of the American Society for Information Science Vol 40 No 5 pp 342-9

Martin B (1996) ordfThe use of multiple indicators in the assessment of basic researchordmScientometrics Vol 36 No 3 pp 343-62

Meadows AJ (1998) Communicating Research Academic Press San Diego CA

Mynott J (1999) ordfPublishing the view from Cambridge University Pressordm History of the HumanSciences Vol 12 No 4 pp 127-31

Naylor B (2001) ordfThe king is in the altogetherordm Serials Vol 14 No 2 pp 149-51

Oppenheim C (1995) ordfThe correlation between citation counts and the 1992 ResearchAssessment ratings for British library and information science university departmentsordmJournal of Documentation Vol 51 No 1 pp 18-27

Oppenheim C (1996) ordfDo citations count Citation indexing and the Research AssessmentExercise (RAE)ordm Serials Vol 9 No 2 pp 155-61

Archaeology andthe 2001 RAE

729

Oppenheim C (1997) ordfThe correlation between citation counts and the 1992 ResearchAssessment Exercise ratings for British research in genetics anatomy and archaeologyordmJournal of Documentation Vol 53 No 5 pp 477-87

Publications R (2002) A Guide to the 2001 Research Assessment Exercise HEFCE available atwwwheroacukraePubsindexhtm (accessed 20 August 2002)

Roberts G (2003) Review of Research Assessment HEFCE available atwwwraereviewacukreportsrobertsasp (accessed 30 May 2003)

Rogers J (2000) ordfThe intellectual consequences of the Research Assessment Exercise aresponseordm History of the Human Sciences Vol 13

Sarwar S (2000) ordfA publication and citation analysis of civil engineering departments in the UKwhich participated in the 1996 Research Assessment Exercise (RAE)ordm unpublished MScShefregeld University Shefregeld

Seglen P (1992) ordfThe skewness of scienceordm Journal of the American Society for InformationScience Vol 43 No 9 pp 628-38

Seng LB and Willett P (1995) ordfThe citedness of publications by United Kingdom libraryschooolsordm Journal of Information Science Vol 21 No 1 pp 68-71

Smith A and Eysenck M (2002) ordfThe correlation between RAE ratings and citation counts inpsychologyordm available at httppsyserverpcrhbncacukcitationspdf (accessed13 November 2002)

Snyder H and Bonzi S (1998) ordfPatterns of self-citation across disciplines (1980-1989)ordm Journalof Information Science Vol 24 No 6 pp 431-5

Van Raan AFJ (1996) ordfAdvanced bibliometric methods as quantitative core of peer reviewbased evaluation and foresight exercisesordm Scientometrics Vol 36 No 3 pp 397-420

Walford L (2000) ordfThe Research Assessment Exercise its effect on scholarly journalpublishingordm Learned Publishing Vol 13 No 1 pp 49-52

Warner J (2000) ordfA critical review of the application of citation studies to the ResearchAssessment Exercisesordm Journal of Information Science Vol 26 No 6 pp 453-60

JD596

730