information retrieval to informed action with research metrics thomson scientific research services...
TRANSCRIPT
Information Retrieval to Informed Action with Research Metrics
Thomson Scientific Research Services Group2007
• Science is very dynamic and growing exponentially
• Research is happening in a global and multidisciplinary context
• Emerging fields make it difficult to determine relevancy
• Selectivity becomes more challenging, e.g. how to find the right journals
Challenge of Managing and Evaluating Research
• Keep up to date on new and developing science
• Nurture new research through direction and funding
• Measure the impact of scientific efforts
• Access timely, objective and informed information to make policy and budget decisions
Responsibilities of leaders in research
Metrics – How to measure science?
• Scientists communicate through the literature in academic journals• Citations are considered the “coin of the realm” in science
Bibliometrics uses statistical analysis to draw a landscape of scientific communication and productivity.
• Citation analysis:• examines the elements of bibliographic records (citations) • studies relationships between citations• measures the amount of scientific activity in a subject category,
journal, country, topic or other areas of interest.
Why Metrics?
• Uses readily available data
• Employs transparent and reproducible methodology
• Provides quantitative information that policy makers can apply to their decision process
“If you can measure that of which you speak, and can express it by a number, you know something of your subject; but if you cannot measure it, your knowledge is meager and unsatisfactory”
- Lord Kelvin (William Thomson)
“If you can measure that of which you speak, and can express it by a number, you know something of your subject; but if you cannot measure it, your knowledge is meager and unsatisfactory”
- Lord Kelvin (William Thomson)
Elements of Evaluation using Bibliometric Methods• Theory of citation:
– Citations are a form of peer recognition
• Basic measures: – Papers, citations, citations per paper (impact)
• Relative measures: – impact relative to baselines;
– expected citation rates for articles;
– percentiles
• Units of analysis: – papers, authors, institutions, countries, journals
• Measures of interaction: – research fronts, interdisciplinarity, collaboration
A National view of Turkish Research
Economics informs but citation analysis provides more detail
Turkish research motivation is seen in the OECD records on percentage of GDP dedicated to Research & Development. Turkey is surging and soon to exceed Portugal.
Source: OECD Factbook 2008
Science and technology - Research and Development (R&D) - Expenditure on R&D
0.0
0.2
0.4
0.6
0.8
1.0
1.2
2000 2001 2002 2003 2004 2005
Greece Hungary Poland Portugal
Slovak Republic Turkey Estonia
Turkey
Research Growth and Impact in Turkey, 1981-2007
Turkish research has grown steadily over the last 27 years. On average, Turkey produced over 14,000 scientific papers per year from 2003-2007, with large movement in aggregate citations between 2003 and 2007.
0
20000
40000
60000
80000
100000
120000
140000
160000
Papers Citations
Source: National Science Indicators, 1981-2007 – WOS version
Annual Research Growth in Turkey, 1981-2007
In 2003, Turkey broke through to go above 10,000 research papers produced in a single year.
Source: National Science Indicators, 1981-2007 – WOS version
Annual Growth in Papers, 1981-2007
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
Total Papers
Research Growth in Turkey, 1981-2007, compared to Greece
Among the countries in the region, Greece is the closest in publications but continues to lag behind Turkey’s strong growth trend.
Source: National Science Indicators, 1981-2007 – WOS version
Annual Growth in Papers, 1981-2007
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
Turkey Greece
Total Papers
Percentage of papers in the world, 1981-2007
Source: National Science Indicators, 1981-2007 – WOS version
Turkey also exceeds Greece in overall contribution to research in the world.
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
Turkey Greece
%Papers InWorld
Research in Turkey – Citation ImpactCitation impact is the average citation rate: total cites/total papers.
Turkey’s average citation rate has risen consistently since the mid-1990’s and is trending to continue its steep movement into the world scientific community.
Source: National Science Indicators, 1981-2007 – WOS version
00.20.40.60.8
11.21.41.61.8
22.22.4
1981
-198
5
1982
-198
6
1983
-198
7
1984
-198
8
1985
-198
9
1986
-199
0
1987
-199
1
1988
-199
2
1989
-199
3
1990
-199
4
1991
-199
5
1992
-199
6
1993
-199
7
1994
-199
8
1995
-199
9
1996
-200
0
1997
-200
1
1998
-200
2
1999
-200
3
2000
-200
4
2001
-200
5
2002
-200
6
2003
-200
7
Baseline=1
Avg.Cite Per Paper
Research in Turkey – Percentage Cited
While publication growth is strong, Turkey is still working to exceed Greece in percentage of papers cited.
Source: National Science Indicators, 1981-2007 – WOS version
0
10
20
30
40
50
60
70
19
81
-19
85
19
82
-19
86
19
83
-19
87
19
84
-19
88
19
85
-19
89
19
86
-19
90
19
87
-19
91
19
88
-19
92
19
89
-19
93
19
90
-19
94
19
91
-19
95
19
92
-19
96
19
93
-19
97
19
94
-19
98
19
95
-19
99
19
96
-20
00
19
97
-20
01
19
98
-20
02
19
99
-20
03
20
00
-20
04
20
01
-20
05
20
02
-20
06
20
03
-20
07
Greece Turkey
% of Papers Cited
Chemistry, 11%
Materials Sci 4%
Ecol/Envir3%
Biology & Biochem, 4%
Plant & Animal Sci, 6%
Physics6%
Engineering, 10%
Pharmacology2%
Clinical Medicine, 38%
Agricult. Sci4%
Turkey: Field Specialization
The following 10 fields make up over 85% of Turkish scientific research
Source: National Citation Report on the nation of Turkey (ESI), 1981-2007
Turkey: Field Specialization
Generally Turkey contributes 1.48% of all world papers in 2003-2007.
The following fields exceed that level and indicate a national strength for Turkey in world contribution
Field % papers in
field in the world
Agricultural Sciences 2.76
Clinical Medicine 2.62
Engineering 1.94
Plant & Animal Science 1.89
Field % papers in
field in the world
Ecology/Environment 1.59
Pharmacology 1.54
Materials Science 1.50
Source: National Science Indicators, 1981-2007, ESI version
Turkey: International Collaboration
Turkish research is written more often within the country. Here are the top 20 countries represented in Turkish collaborations since 2000.
Source: National Citation Report on the nation of Turkey, 1981-2007
Key Universities in Turkish Research
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
SELCUK UNIV
ONDOKUZ MAYIS UNIV
BASKENT UNIV
AKDENIZ UNIV
MARMARA UNIV
ERCIYES UNIV
CUKUROVA UNIV
FIRAT UNIV
ISTANBUL TEKNIK UNIV
DOKUZ EYLUL UNIV
ATATURK UNIV
ORTA DOGU TEKNIK UNIV
GAZI UNIV
EGE UNIV
ANKARA UNIV
HACETTEPE UNIV
ISTANBUL UNIV
Number of Papers, 2002-2006
University Publications
Source: National Science Indicators customized for Turkish Institutions, 1981-2006
These 17 universities published 1000 or more papers over the period 2002-2006. This illustrates academic contribution to the national performance
Growth of Publications in Specific Universities
Source: National Science Indicators customized for Turkish Institutions, 1981-2006
In the 2002-2006 period, these 5 leading Turkish universities fit into the mid-range of publication counts for the US, UK, Australia, and Canada university indicators.
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Ankara University
Hacettepe University
Instanbul University
Ege University
Gazi University
Relation of University Strengths to National Strengths
Source: National Science Indicators customized for Turkish Institutions, 1981-2006
Most active fields for top 5 schools
# of papers Turkey overall % of country contribution
Agricultural Sciences 472 2266 21%
Biology & Biochemistry 806 2519 32%
Chemistry 2086 6754 31%
Clinical Medicine 7807 22909 34%
Ecology/Environment 163 1771 9%
Engineering 819 5807 14%
Geosciences 256 1358 19%
Materials Science 394 2234 18%
Neurosciences & Behavior 312 1247 25%
Pharmacology 560 1401 40%
Physics 586 3833 15%
Field Specialization, 2002-2006
Source: National Science Indicators customized for Turkish Institutions, 1981-2006
0 100 200 300 400 500 600 700 800
ORTA DOGU TEKNIK UNIV (METU)
ISTANBUL TEKNIK UNIV
GAZI UNIV
ISTANBUL UNIV
FIRAT UNIV
ATATURK UNIV
BOGAZICI UNIV
YILDIZ TEKNIK UNIV
ERCIYES UNIV
BILKENT UNIV
Papers Citations
Other fields of Turkish strength like Engineering are represented by a unique set of universities.
Growth of Publications in Leading Technical Universities, 1981-2007
Source: National Science Indicators customized for Turkish Institutions, 1981-2006
Highlight comparisons between Middle East Technical University and Instanbul Tecknick
0
50
100
150
200
250
300
350
400
450
500
1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007
ISTANBUL TEKNIK UNIV MIDDLE EAST TECH UNIV
Engineering subfields at METU
Source: National Science Indicators customized for Turkish Institutions, 1981-2006
Cites Field Papers Avg. Cites/Paper
2,845 ENGINEERING, CHEMICAL 640 4.45
1,426 ENGINEERING, BIOMEDICAL 139 10.26
1,168 ENGINEERING, ELECTRICAL & ELECTRONIC
328 3.56
920 MATERIALS SCIENCE, BIOMATERIALS 65 14.15
706 ENGINEERING, MECHANICAL 194 3.64
Engineering at METU: ChemE journals
Source: National Science Indicators customized for Turkish Institutions, 1981-2006
Journal Cites Papers Avg. Cites/Paper
2006 JournalImpact factor
ENERGY SOURCE 55 54 1.02 0.43
TURKISH JOURNAL OF CHEMISTRY 90 54 1.67 0.65
CHEMICAL ENGINEERING COMMUNICATIONS
61 38 1.61 0.35
FUEL 151 29 5.21 1.36
INDUSTRIAL & ENGINEERING CHEMISTRY REESARCH
173 26 6.65 1.52
CHEMICAL ENGINEERING SCIENCE 119 25 4.76 1.63
ENERGY & FUEL 105 25 4.2 1.52
JOURNAL OF FOOD ENGINEERING 109 25 4.36 1.7
AIChE JOURNAL 256 22 11.64 2.15
MINERALS ENGINEERING 37 22 1.68 0.94
Engineering at METU: Authors in ChemE
Source: National Science Indicators customized for Turkish Institutions, 1981-2006
Cites Author Papers Avg. Cites/Paper
Cites2 H index
C index Average Percentile
548 DOGU, T 66 8.3 3,747 14 0.86 43.41
318 DOGU, G 36 8.83 1,864 11 0.91 37.66
258 YILMAZ, L 27 9.56 2,929 10 1.43 54.13
139 KOK, MV 49 2.84 500 6 0.65 62.87
69 TOSUN, I 28 2.46 572 3 0.34 58.74
52 HICYILMAZ C 25 2.08 82 4 0.58 71.52
18 BAGCI, S 23 0.78 18 1 0.2 71.68
18 TURKER, L 20 0.9 115 2 0.14 78.11
Engineering at METU: A well cited ChemE paper from Dr. T. Dogu
Source: National Science Indicators customized for Turkish Institutions, 1981-2006
Comments
• Because science is such a dynamic enterprise, no one metric is adequate for evaluation.
• Evaluation is the culmination of quantitative and qualitative considerations, within a particular environment.
• Citation analysis can help highlight researchers and the work that impacts the overall impact of an organization, group, or nation.– Examines the elements of publıcatıon records
– Studies relationships between citations
– Allows for the measurement of the amount of scientific activity in varıous areas, e.g. Organızatıons, nıdıvıduals, journals, subject areas
Thank you!