Guillaume WarnanSenior Customer Consultant, Research Intelligence, Elsevier

Rankings workshop –

How Scopus and SciVal

data are used for global

university rankings

2nd North American SciVal User Group MeetingMcGill University, October 11th 2019

| 1

Overview

➢ WHY are International Rankings important?

➢ WHAT are the International Rankings Organizations?

➢ HOW are SciVal & Scopus involved? (Brief recap overview)

➢ WHICH rankings? Deep dive:

1. QS

2. THE Rankings

➢ EXAMPLE institution data for THE: McGill University

➢ How can data be reproduced from SciVal/ Scopus?

➢ Scopus Institutional Profile Wizard

➢ HOW can citation performance/ranking be improved?

| 2

The importance of Rankings

| 3

Source: University selection by students (IDP Research)

The goal of the program is that by 2020 at

least 5 Russian universities will be in the top

100 universities as denoted by QS World

University ranking.

Rankings are cited by many stakeholders as being

important

85% of students find university ranking as

important in their selection of institute to study

33% of students find university ranking as the

most important factor (number 1 factor, followed by

21% employer recognition, etc.)

Students & Parents

University Management

David Willets (former UK Universities & Science

minister):

“We broadly accept the criteria used by the THE,

which is why our policies are focused on the

same areas.”Policy Makers

| 4

Students primarily use a small number of powerhouse

rankings systems, although there are significant

variations by nationality

Source: Hobsons, INTERNATIONAL STUDENT SURVEY 2015, Value and the Modern International Student

| 5

Academic performance with

league table

Academic performance without

league table

Broad based league table

“Multidimensional” ranking

Employability based league table

Web presence league table

Retroactive

Environmental Focus

Innovation Focus

Global University Rankings

’03 ’04 ’05 ’06 ’07 ’08 ’09 ’10 ’11 ’12 ’13 ’14 ’15 ’16

1 ARWU/Shanghai

2 Webometrics

3 QS

4 4icu

5 NTU/HEEACT

6 Leiden

7 URAP

8 SCImago R R R R R R R

9 THE

10 Trendence/Emerging

11 RUR R R R

12 U-Multirank

13 UI GreenMetric E E E E E

14 CWUR

15 Youth Incorporated

16 nature INDEX

17 RankPro

18 US News

19 Reuters I I

Source: Illuminate Consulting Group

R

E

I

| 6

Other prestigious university rankings where

Scopus is used as bibliometric data source:

National Institutional Ranking Framework

IndiaNational University Ranking

Poland

Best Chinese University Ranking

| 7

All rankings have their strengths and potential

disadvantages and we do not rank the rankings!

• We believe in working on fundamentals with a “basket of

indicators”, always as a complement to peer opinion

• Informed decisions are better decisions

• Metrics should complement, not replace human judgment

• Well-selected metrics drive positive behaviors

• Metrics does not only mean bibliometrics

• Metrics can help monitor and eliminate biases

• Assessments are costly, but availability of new tools help bring

cost down

• Data sources to cover humanities are becoming more

complete

Elsevier’s position on university rankings and metrics

in general

| 8

Brief recap of Scopus and

SciVal

| 9

Scopus is the largest curated abstract and citation database of peer-

reviewed literature, and features smart tools that allow you to track,

analyze and visualize scholarly research.

5000+Publishers

Scopus delivers a comprehensive view on the world of research.

No packages, no add-ons. One all-inclusive subscription.

24,000+

Serial titles

200,000+

Books

76 millionItems

16 millionAuthor profiles

~70,000Affiliation Profiles

1.4 billion cited references

dating back to 1970

Identify and analyze which

journals to read/submit to

Help researchers manage career-

citation counts and h-index

Decide what, where and with

whom to collaborate

Track impact of research;

monitor global research trends

Find out what already exists in

the global world of research

Determine how to differentiate

research topics, find ideas

| 10

As a primary publisher and information aggregator, Elsevier understands the needs of Authors, Editors and Publishers and

provides resources to support the community. Available resources to help journals with successful title review process:

publication ethics resources | FAQs | advisory documents | reviewer comments | editor and publishing services

https://www.elsevier.com/solutions/scopus/content/content-policy-and-selection or titlesuggestion@scopus.com

Coverage of high quality journals via selection by the

independent Content Selection & Advisory Board (CSAB)

The CSAB is an independent board of subject experts from all over the world.

Board members are chosen for their expertise in specific subject areas; many have (journal) Editor experience.

| 11

Unbiased, comprehensive journal coverage with titles

from many reputable scholarly publishers

Source: May 2016 title list at https://www.elsevier.com/solutions/scopus/content

Other

60%

7%

5%

4%

2%

1%

1%

1%

1%

1%

1%

1%

1%

11%

1%

| 12

Global Representation means global discovery

Across all subjects and content types

North

America

6,000+50% more

than nearest

competitor

Middle East

& Africa

750+212% more

than nearest

competitor

Western

Europe

11,000+69% more

than nearest

competitor

East Europe

incl. Russia

1,400+168% more

than nearest

competitor

Latin

America

700+168% more

than nearest

competitor

Asia

Pacific

2,000+230% more

than nearest

competitor

Australia/

New Zealand

300+206% more

than nearest

competitor

Global

Representation (number of titles)

| 13

And Analyzed in SciVal

SciVal offers quick, easy access to the research performance of

15,500 research institutions and 231 nations worldwide.

| 14

QS

| 15

NEW

Source: QS

Affiliations: Primarily Scopus institutions

Support: Self-citations, subject mapping, etc.

| 16

Source: QS

| 17

Bibliometric parameters

Institutions based

compiled by QS

based on Scopus profiles

NTCC (= Normalized Total

Citation Count) as the core

citation metric

Subject mapping to

ASJC developed

by QS

Scopus data April

2019 snapshot

| 18

Main changes between 2015 and 2018 in QS’ WURM

etr

ics

calc

ula

tion

End Q1 Metr

ics

calc

ula

tion

Q3 (~June/July)

Metr

ics

calc

ula

tion

Q2 (April) Metr

ics

calc

ula

tion

Unchanged

Z-s

core Calculated for the

entire population

Z-s

core Using a fixed range

for z-score calculations to limit divergent effects

Z-s

core

Unchanged

Z-s

core Unchanged

Kilo

papers > 10 affiliations

excluded

Kilo

papers Variable threshold

per subject implemented

Kilo

papers

Unchanged Kilo

papers

Unchanged

Cita

tio

n

Imp

act

NTCC

(introduced in 2015 for the first time)

Cita

tio

n

Imp

act

Unchanged

Cita

tio

n

Imp

act

Unchanged

Citation

Impact

Unchanged

Docum

ent

types

AR RE CP BK CH AIP BZ

Docum

ent

types

UnchangedD

ocum

ent

types

Unchanged

Docum

ent

types

Unchanged

Num

be

r o

f u

niv

ers

itie

s

890 Num

be

r o

f U

niv

ers

itie

s

916 Num

be

r o

f U

niv

ers

itie

s

979 Num

be

r o

f U

niv

ers

itie

s

1,021

Ma

in

Su

bje

cts

5

Ma

in

Su

bje

cts Unchanged

Ma

in

Su

bje

cts Unchanged M

ain

S

ubje

cts

Unchanged

2015 20172016 2018

| 19

THE

| 20

Affiliations: Primarily SciVal institutions

Support:

• Customized bibliometric data from Scopus: Citations (30% of score),

research productivity (6%) & international collaboration (2.5%)

• Reputation data: Elsevier runs the reputation survey (15% of score)

using Elsevier author list for THE

• Affiliation handling: Affiliation corrections, mergers, splits, etc. handled

with THE for the respective universities

Source: THE

| 21

Source: THEElsevier contributes by providing Scopus

bibliometric data

Elsevier contributes by conducting the

reputation survey: Randomly selected authors

from Scopus are invited to complete the survey

| 22

Minimum requirements in order to be ranked

Institutional Data

uploaded through the THE data

portal

1,000 papers for the 5 years

window

At least 150 papers per

year

Variable* threshold

per subject area

(*): 500 papers for large disciplines, 250 for smaller disciplines

TITLE OF PRESENTATION| 23

23|

Main changes between 2015 and 2018 in THE’s WURT

hre

sh

old

o

f p

ap

ers

p

er

ye

ar

200

Th

resh

old

o

f p

ap

ers

p

er

ye

ar

150

Th

resh

old

o

f p

ap

ers

p

er

ye

ar

Unchanged

Th

resh

old

o

f p

ap

ers

p

er

ye

ar

Unchanged

Th

resh

old

p

er

su

bje

ct

0

Th

resh

old

p

er

su

bje

ct

500/250

Th

resh

old

p

er

su

bje

ct

Unchanged

Th

resh

old

p

er

su

bje

ct

Unchanged

Kilo

papers

Excluded Kilo

papers Re-introduced but

with fractionalized counting

Kilo

papers

Unchanged Kilo

papers

Unchanged

Jo

urn

als All

Jo

urn

als Suspended

titles excluded

Jo

urn

als

Unchanged

Jo

urn

als

Unchanged

Docum

ent

types

AR, RE and CP D

ocum

ent

types

AR, RE, CP, BK and CH D

ocum

ent

types

Unchanged

Docum

ent

types

Unchanged

Num

be

r o

f u

niv

ers

itie

s

801 Num

be

r o

f u

niv

ers

itie

s

980 Num

be

r o

f u

niv

ers

itie

s

1,103 Num

be

r o

f u

niv

ers

itie

s

1,258

Su

bje

cts 6

Su

bje

ct

8

Su

bje

ct

11 main disciplines + 32 sub-disciplines

Su

bje

ct

Unchanged

2015 20172016 2018

TITLE OF PRESENTATION| 24

24|

Full description of THE’s methodology to calculate the World University Rankings:

https://www.timeshighereducation.com/world-university-rankings/world-

university-rankings-2020-methodology

TITLE OF PRESENTATION| 25

25|

How to easily find THE’s methodology page:

TITLE OF PRESENTATION| 26

26|

Last year, THE launched a new ranking, the University Impact Rankings

TITLE OF PRESENTATION| 27

27|

Methodology highlights

Only 11 SDGs

evaluated out the 17

To be in the overall

ranking, universities

need to submit data on at least 3

SDGs + SDG 17

Each SDG has a

specific series of metrics

Final score is calculated

by combining its score in

SDG 17 with its top

three scores

TITLE OF PRESENTATION| 28

28|

Elsevier data is being used to assess the “Research” criteria

• Research metrics are derived from data supplied by Elsevier.

• For each SDG, a specific query has been created that narrows the scope of the metric to papers relevant to that SDG.

• As with the World University Rankings, THE is using a five-year window (for the 2019 ranking, 2013-2017). The only exception is the metric on patents that cite research under SDG 9, which relates to the timeframe in which the patents were published rather than the timeframe of the research itself.

• The metrics chosen for the bibliometrics differ by SDG and there are always at least two bibliometric measures used.

TITLE OF PRESENTATION| 29

29|

The weighting of the Research criteria as well as the metrics used vary from one SDG to another

Weighting

# metrics

used

27% 27% 27% 27% 11.6%* 27% 27% 27% 27% 27% 27%

3 3 3 2 1 3 3 3 3 3 2

Main

metric

2 (10%)

3 (10%)

6 (7%)

* + 15.4% on patents, i.e. the number of patents that cite research conducted by the university

Metrics used:

1. Proportion of research papers in the top 10% of journals as defined by Citescore

2. Proportion of research papers that are viewed or downloaded

3. Proportion of research papers that are cited in clinical guidance

4. Proportion of a university’s total research output that is authored by women

1 (10%)

2 (10%)

6 (7%)

1 (10%)

4 (10%)

6 (7%)

1 (14%)

6 (13%)

6 (12%) 1 (10%)

5 (10%)

6 (7%)

1 (10%)

5 (10%)

6 (7%)

1 (10%)

5 (10%)

6 (7%)

1 (10%)

5 (10%)

6 (7%)

1 (10%)

5 (10%)

6 (7%)

7 (14%)

8 (13%)

5. Field-weighted citation index of papers produced by the university

6. Number of publications related to the SDG

7. Proportion of academic publications with co-author from other country

8. Number of publications that relate to the 11 SDGs

| 30

Example of how Scopus data is

prepared for THE’s World

University Ranking

| 31

Process

Reputation

Oct-Nov Dec-Mar Ongoing-May

Preparation for the

reputation survey:

- Questionnaire

- Random selection

of survey invitees

(Scopus author

profiles)

- List of institutions

for selection (all

SciVal institutions)

Reputation Survey

run:

- Regular update to

THE re. response

rate

- Addressing

possible issues

(e.g. emails block)

Bibliometrics

Affiliation validation

and clean up

- THE list against

Elsevier (re)mapped

and checked for

completeness

- SciVal institutions are

default Elsevier profiles

- If not in SciVal, then

added into SciVal or

otherwise from Scopus

- Ongoing feedback

processing from

institutions

- Potential methodology

changes tested and

validated

May-July

Bibliometric

computation:

- Using May-finalized

list, the

bibliometrics:

Citation Score, total

# of papers and

total # of

international papers

- Analysis of the

results, validation

and sign-off

WURLaunch

Sep

World

University

Ranking

launch

27/07/2012

Milestone DescriptionReputation Data

(responses) delivery

27/07/2012

Milestone DescriptionBibliometric Data Delivery

| 32

THE’s WUR 2017-18 Scopus Citation dataset

ALL SCOPUS 2012-2016 Articles,

Reviews, Conf.

papers, Books

and Book

chapters

Disciontinued

titles carved out

12,431,51412,737,67814,061,95667,775,649

Scopus – 1 May 2017

| 34

Example of a citation score calculation

the_name country

rank_

label

overall_

score

teaching

_score

research

_score

citation

_score

industry

_score

internation

al_score

Paris Sciences et Lettres

Ð PSL Research

University Paris France 72 65.2 57 52 85 48.2 76.7

PSL Research University (Paris

Sciences Et Lettres Research

University) publications fwci pubs_intl fwci_intl normdist

Citation

Score

Country Normalized 34185.132 1.476 18987.132 2.137 86.6785.0

Non-normalized 34185.132 1.705 18987.132 2.137 83.32

| 36

Example of THE citation data

from Scopus/SciVal for a

Canadian university

| 37

How THE citation data are calculated for McGill

University...

| 38

Publications of McGill University in SciVal, 2014-2018....

however, not all pub types are eligible...

| 39

Now let’s look at which Scopus profiles this SciVal

profile is sourced from!

| 40

The data for McGill University Scival profile are drawn

from 21 Scopus Institution profiles:

| 41

Scopus profile of McGill University

| 42

Detailed hierarchy of McGill University Scopus profile

TITLE OF PRESENTATION| 43

43|

to analyze more closely the data used by THE?

1

2

3

Use the affiliations from the SciVal profile

Limit to the

document types

eligible for THE

Limit to the past 5 years

(2014-2018)

TITLE OF PRESENTATION| 44

44|

Finally, also add the discontinued titles to the Scopus Advanced Search with: AND NOT(...)

TITLE OF PRESENTATION| 45

45|

You can then re-import your data in SciVal

TITLE OF PRESENTATION| 46

46|

https://www.elsevier.com/solutions/scopus/how-scopus-works/content

Where to find the most recent Scopus discontinued titles list:

TITLE OF PRESENTATION| 47

47|

Use source IDs for your query

TITLE OF PRESENTATION| 48

48|

Excellent new white paper (June 2019) by CSAB member Karen Holland et al. – on the rationale behind the Scopus discontinued titles process and details of how titles are identified:

https://www.elsevier.com/research-intelligence/resource-library/scopus-high-quality-content

...and see recent webinar, Sep. 11 2019 : Scopus Curation and Re-Evaluation = High Quality Content

https://blog.scopus.com/webinars

| 49

Further “VIP” support re. how Scopus/ SciVal data are

used for the World University Rankings – outside the

“regular” Scopus helpdesk channels:

WURSupport@elsevier.com

(Disclaimer: as this email address is managed by only 1-2

colleagues within the Scopus product team, and resources

are therefore limited, please only send emails to this

address for any questions for which you cannot find the

information yourself, either on our Scopus website

https://www.elsevier.com/solutions/scopus/how-scopus-

works/content, on THE’s website

https://www.timeshighereducation.com/world-university-

rankings/world-university-rankings-2020-methodology or

within this slide deck.)

| 50Open Access |

User Guide for the Scopus

Institution Profile Wizard (IPW)

| 51Open Access |

How does an institute access the IPW?

• The Scopus IPW was launched in July 2018

• Use is restricted to named authorized users selected by the institution. Normally this should

not be more than three users per institution

• Selected IPW users from an organization should have:

o Familiarity with Scopus.com

o Knowledge of the organization’s hierarchical structure

o Knowledge of all campus and satellite locations and the names they are known by

• A form for the institution to provide authorization of their users has been prepared by the

Elsevier legal team and can be obtained from Elsevier customer consultants

• The signed letter of authorization for the users should be sent along with their individual Scopus

usernames to your Customer Consultant

Important: IPW users will need to register their Scopus account if they have not yet done

so, and log in on Scopus.com at least once before access can be granted

51

| 52Open Access |

Where can I find the IPW on Scopus?

• If you are an authorized IPW user, log in to Scopus

• From the Affiliation search, find your organization

• Open the Affiliation details page. The Modify institution profile link opens the wizard

| 53Open Access |

Function 1: Modify hierarchy

• Click start on the ‘Modify hierarchy’ tile to review and edit the relationships within your

institution

53

(For more detailed instructions, also on the other functionalities within the

Scopus Institutional Profile Wizard, please see the slides in the Appendix.)

| 54

What can an institution do to

improve its Citation Score and

THE ranking?

| 55

What does FWCI represent?

FWCI = Field-Weighted Citation Impact indicates how the number of

citations received by an entity’s publications compares with the

average number of citations received by all other similar publications in

the data universe

A FWCI of 1.00 indicates that the entity’s publications have been cited

exactly as would be expected based on the global average for similar

publications

THE’s Citation Score is based on the normal distribution (Z-score)

of the FWCI population of ALL ranked institutions

| 56

Canadian universities: FWCI & International

Collaboration (2012-2018)

Source: SciVal (data cut of 08/30/19); sample of 48 Canadian universities

| 57

How does international collaboration affect

institutional FWCI on a global scale?

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

FW

CI

% INTERNATIONAL

Asia World

* FWCI for the top 100 Asian institutions by output from each region vs the top 500 international institutions by output

Source: Scopus.com, May 1, 2018

| 58

Impact of Corporate Collaboration for

Universities

Ben Shneiderman, Twin-Win Model: A human-centered approach to research success, Proceedings of

the National Academy of Sciences (2018). DOI: 10.1073/pnas.1802918115

| 59

www.elsevier.com/research-intelligence

Any question?

| 60

Appendix (FWCI methodology)

| 61

Recap on the Citations Score indicator methodology

• The Citation Score for a university is the average of the Z-scores of the

Non-normalized FWCI and the Country normalized FWCI of the university.

More on Z-score here at: https://nl.wikipedia.org/wiki/Z-score

• This means the following:

𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛 𝑆𝑐𝑜𝑟𝑒 =𝑁𝑜𝑛−𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛 𝑆𝑐𝑜𝑟𝑒 + 𝐶𝑜𝑢𝑛𝑡𝑟𝑦−𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛 𝑆𝑐𝑜𝑟𝑒

2

𝑁𝑜𝑛 − 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛 𝑆𝑐𝑜𝑟𝑒 = 𝑍𝑠𝑐𝑜𝑟𝑒 (𝐹𝑊𝐶𝐼)

𝐶𝑜𝑢𝑛𝑡𝑟𝑦 − 𝑁𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛 𝑆𝑐𝑜𝑟𝑒 = 𝑍𝑠𝑐𝑜𝑟𝑒 (𝐹𝑊𝐶𝐼

𝐹𝑊𝐶𝐼_𝑐𝑜𝑢𝑛𝑡𝑟𝑦)

• For the FWCI calculation methodology please refer to Appendix F7

| 62

FWCI: the formula

The Field-Weighted Citation Impact (FWCI) for a set of N publications is

defined as:

FWCI ≡1

𝑁

𝑖=1

𝑁𝑐𝑖𝑒𝑖

• 𝑐𝑖 = citations received by publication i

• 𝑒𝑖 = expected number of citations received by all similar publications in the publication

year plus following 3 years

When a similar publication is allocated to more than 1 discipline, the

harmonic mean is used to calculate 𝑒𝑖. For a publication k that is part of 2

disciplines:1

𝑒𝑘≡1

2(1

𝑒𝐴+

1

𝑒𝐵)

𝑒𝐴, 𝑒𝐵 = fractional counts of publications and citations, so that publication k will be counted as

0.5 publications in each of 𝑒𝐴 and 𝑒𝐵, and the citations it has received will also be shared

between A and B.

Field-Weighted Citation Impact is an indicator of mean citation impact, and compares the actual number of citations received by an article with the expected number of citations for articles of the same document type (article, review or conference proceeding paper), publication year and subject field*.

| 63

Example

Publication Discipline Citations

received

Expected citations for

discipline

FWCI

1 Biophysics 5 (5+6+(50/2))/(1+1+(1/2)) = 14.4 (5/14.4) = 0.35

2 Biophysics 6 (5+6+(50/2))/(1+1+(1/2)) = 14.4 (6/14.4) = 0.42

3 Chemistry 120 (120+130+(50/2)/(1+1+(1/2)) =

110.0

(120/110.0) = 1.09

4 Chemistry 130 (120+130+(50/2)/(1+1+(1/2)) =

110.0

(130/110.0) = 1.18

5 Biophysics

and

Chemistry

50 Harmonic mean of citations

received, when publication is in

more than 1 discipline:

2/((1/110) + (1/14.4)) = 25.46

(50/25.46) = 1.96

FWCI database of 5

publications:

((0.35+0.42+1.09+1.18+1.96)/5

) = 1.00

FWCI Biophysics ((0.35+0.42+1.96)/3) = 0.91

FWCI Chemistry ((1.09+1.18+1.96)/3) = 1.41

| 64Open Access |

T&F book published in 2017: overview of all worldwide

rankings & main bibliographic databases used

https://www.taylorfrancis.com/books/9781315155890

(In case you would like to receive a PDF copy of a preprint of our Scopus book

chapter, please send me an email at M.Schotten@elsevier.com)