arra’s experience in the identification of top research teams: ©arra, october 2011 ivan...
TRANSCRIPT
ARRAARRA’’s experience s experience in the identification in the identification
of top research teams:of top research teams:
©ARRA, October 2011©ARRA, October 2011
Ivan OstrovskIvan Ostrovský, Miroslav Medveď,ý, Miroslav Medveď, Ferdinand Devínsky, Juraj BartaFerdinand Devínsky, Juraj Barta
the the case study of case study of the the Slovak Academy of SciencesSlovak Academy of Sciences
IREG-5 ForumThe National Rankings on the Rise
Bratislava 10-11 October 2011Hosted by the Comenius University in Bratislava
CONCEPT:Fénix 2CONCEPT:Fénix 2
2
Common and standardised system of periodic Common and standardised system of periodic assessment of R&D for HEIs and other research assessment of R&D for HEIs and other research institutionsinstitutions – basis of long-term institutional financing of R&D
CConcept oncept of national and international excellenceof national and international excellence
International criteria for R&D assessmentInternational criteria for R&D assessment Outputs - publications, citations, innovations........
Leading companies are hindered by the collaboration Leading companies are hindered by the collaboration
with average/mediocre HEIs or research institutions.with average/mediocre HEIs or research institutions.
Main Goals of the ProjectMain Goals of the Project
SAS SAS asked for asked for the the identifiidentification ofcation of
top and at the same time top and at the same time prospective prospective research teamsresearch teams
3
3 critical questions to achieve the goal:3 critical questions to achieve the goal:
HowHow is is a a “research team”“research team” defined?defined?
When is the team classified as “top”?When is the team classified as “top”?
When When the the team team has has “good prospects”?“good prospects”?
PrinciplesPrinciples
TTeaeam m and Teamand Team LeaderLeader Top team should be led by a top leader (leaders); however, a top
researcher is not necessarily the leader of the top team
ComparisonComparison is correct and accurate among the SAS teams, but only in the same discipline/set of fields
NNoo K.O. K.O. criteria appliedcriteria applied, but complex assessment (not “blind” scientometry), with needs for: identifying the team’s main scientific field deciding whether a particular researcher is a member of the team or
not assessing criteria in the context
4
Number of Articles (author)
> 10% of co-operative articlesFyzio-VascDisOrd Kristek Cacanyiova Torok Cebova Zemancikova Malekova
František Kristek 27 15/56 4/15 10/37
Soňa Čačányiová (Kyselá) 15/94 16 3/19
Jozef Török 4/40 10 4/40 3/30
Martina Cebová 10/77 3/23 4/31 13
Anna Zemančíková 3/100 3
Magdaléna Maleková 5
Teams’ identificationTeams’ identification
Fyzio-VascDisOrd Kristek Cacanyiova Torok Cebova Zemancikova Malekova
František Kristek
Soňa Čačányiová (Kyselá)
Jozef Török
Martina Cebová
Anna Zemančíková
Magdaléna Maleková
Fyzio-VascDisOrd Kristek Cacanyiova Torok Cebova Zemancikova Malekova
František Kristek 27 15/56 4/15 10/37
Soňa Čačányiová (Kyselá) 15/94 16 3/19
Jozef Török 4/40 10 4/40 3/30
Martina Cebová 10/77 3/23 4/31 13
Anna Zemančíková 3/100 3
Magdaléna Maleková
Primary listPrimary list: : SAS’s formal division (institutes/departments) SAS’s formal division (institutes/departments)
> 30% of co-operative articles
> 60% of co-operative articles
Level and degree of mutual collaboration Level and degree of mutual collaboration AutAuthhor(or(ss), ), with whomwith whom the other team membersthe other team members have the highesthave the highest % % of collaboration, are considered as team leadersof collaboration, are considered as team leaders Team Team ‘Core’ a‘Core’ andnd ‘ ‘NNon-core’on-core’ members members Collaboration outside the institute/department, or outside Collaboration outside the institute/department, or outside SASASS
ABCDEFGH
IJKL
AB CD EF GH IJ KLPHYSIOLOGY
SAS structureSAS structure
6
SLOVAK ACADEMY OF SCIENCESSLOVAK ACADEMY OF SCIENCES
PHYSICAL, SPACE, EARTH, AND
ENGINEERING SCIENCES
LIFE, CHEMICAL,MEDICAL,
AND ENVIRONMENTAL SCIENCES
SOCIAL SCIENCES, HUMANITIES, ARTS
AND CULTURE
SECTION 1 SECTION 2 SECTION 3
FirstFirst ““filterfilter””
In Sections 1 and 2 - more than 200 teams were identifiedIn Sections 1 and 2 - more than 200 teams were identified
First selection criteria: (First selection criteria: (who is definitely who is definitely NOT in the TOPNOT in the TOP))
Team leader has less than 10 WoK Articles since 2006Team leader has less than 10 WoK Articles since 2006
Team leader/most members have at most a part-time (no more Team leader/most members have at most a part-time (no more
than 50%) contract with SASthan 50%) contract with SAS
Modified H-index for particular scientific field (ModH) is lower Modified H-index for particular scientific field (ModH) is lower
than 15than 15
Borderline cases were also evaluatedBorderline cases were also evaluated
63 teams matched these criteria63 teams matched these criteria 7
BasicBasic PPrincrinciplesiples Time scaleTime scale – 2001 – 2001 to 2010 to 2010 9 9 stepssteps to teams assessmentto teams assessment – n – noo K.O. K.O. criteriacriteria Steps Steps 1-2 1-2 focus on individual team membersfocus on individual team members – – their overall their overall
performanceperformance a andnd the the contribution to the quality of the teamcontribution to the quality of the team Steps Steps 3-5 3-5 focus on the team’s focus on the team’s performanceperformance – quantity, quality – quantity, quality
aandnd the the efeffectiveness of published papersfectiveness of published papers Steps 6-8 assess team’s exceptional articles and their relative Steps 6-8 assess team’s exceptional articles and their relative
impact impact Step 9 describes the progress of the team during Step 9 describes the progress of the team during the the
evaluated periodevaluated period All comparisons were done against international benchmarks All comparisons were done against international benchmarks ‘ ‘Avg citation per article’ was also compared with the world’s Avg citation per article’ was also compared with the world’s
best 35 universities (top 25 from: ARWU, best 35 universities (top 25 from: ARWU, QSQS, TimesHE), TimesHE) 8
AnorChem-TeoChem Art Cit Cit/Art H-index DoB Age PhD 1st Art ActiveResm-quotientFTEM OL [100] 70 3663 52,33 28 1956 55 1998 1990 27 1,04 100MVG [100] 73 3526 48,30 28 1956 55 1990 1979 21 1,33 100N J [53] 88 4856 55,18 30 1955 56 1995 1980 16 1,88 67,1KO S [100] 8 53 6,63 4 1982 29 2009 2006 2 2,00 53,3KE S [100] 8 144 18,00 6 1981 30 2009 2003 2 3,00 63,3RE M [0] 10 74 7,40 5 1981 30 2009 2004 2 2,50 55,8HO P [0] 16 160 10,00 9 1981 30 2008 2001 3 3,00 43,3 PA A [100] 1 1 1,00 1 1985 26 2010 1 1,00 13,3MA E [outside SAV] 3 15 5,00 2 2006 5 0,40 KA M [outside SAV] 188 4941 26,28 36 1990 21 1,71
H-index 01-11 (Lead) 21
Art 01-10 Cit 01-10 Cit/Art 01-10W Avg Cit/ArtT/World BestUni ACAT/BestUniAvgCoAuthUnCit H-index 01-11 (Rest w/Lead) 9
63 1088 17,27 10,74 1,61 19,41 0,89 4,5 8 H-index 01-11 (Rest w/o Lead) 2
.1st Percentile 1st Percentile1st Decile2nd DecileAvg% 03-10H-index 01-10ModH 2 Angew. Chem. Int. Edit. 0 0 18 14 30,9 21 17 2 J. Am. Chem. Soc.
Art% Proc% Rev% 17 J. Chem. Phys. 33,17 26,46 12,59 1 J. Chem. Theor. Comput.
4Art Cit Cit/Art W Avg Cit/ArtT/World
2001 7 175 25,00 17,62 1,422002 8 251 31,38 17,61 1,782003 7 122 17,43 16,07 1,082004 4 71 17,75 14,88 1,192005 10 178 17,80 13,09 1,362006 6 100 16,67 10,57 1,582007 6 54 9,00 8,35 1,082008 4 97 24,25 5,79 4,192009 5 31 6,20 2,98 2,082010 6 9 1,50 0,44 3,412011 3 0,002001-2010 63 1088 17,27 10,74 1,61
Phys. Chem. Chem. Phys.
26Top 5 Journals
0,00
0,50
1,00
1,50
2,00
2,50
3,00
3,50
4,00
4,50
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
Collected data were assessed according to four main
criteria: publication productivity (quantity of outputs)publication productivity (quantity of outputs), , total influence (no. of citations, H-index)total influence (no. of citations, H-index), , efeffectiveness (avg cit/art, uncited articles)fectiveness (avg cit/art, uncited articles),, relatrelatiivvee impa impacctt (top papers, 1 (top papers, 1stst percentile/decile) percentile/decile)
Each of the abovementioned information is important for Each of the abovementioned information is important for different aspect of the research evaluationdifferent aspect of the research evaluation
Data evaluationData evaluation
10
22 top teams were identified
Only Sections Only Sections 1 a1 andnd 2 2 are presented here are presented here
Section 3 (SSH) has different publication patterns (more on Section 3 (SSH) has different publication patterns (more on
Slide 13)Slide 13)
17 above-average teams were detected
Several “top-teams” criteria were not metSeveral “top-teams” criteria were not met
SumSummary 1mary 1
11
SumSummary 2mary 2
from from 118800 00 assessed researchers in Sections assessed researchers in Sections 1 a1 andnd 2 2, , ~ 1000 are~ 1000 are members of some teammembers of some team
questionnairequestionnaire sent tosent to 63 t 63 teams,eams, covering some covering some ~ 400 researchers
870 researchers do not belong to any team
580 researchers do not belong to top or above-average teams
The majority are individual researchers (occasional pairs)The majority are individual researchers (occasional pairs)
eveneven the the review of mutual citationsreview of mutual citations hashas led to the same data led to the same data
PProblroblemsems with the performance evaluation usingwith the performance evaluation using WoK, WoK,
InCites, ESI – InCites, ESI – due to the due to the lack of datalack of data regarding regarding SASAS SSH S SSH
fields fields ((exceptexcept for for several individuals within Psychology and several individuals within Psychology and
Economics,Economics, none with H>none with H>2)2)
Additional sources from SAS LibraryAdditional sources from SAS Library
Lack ofLack of proper proper scientometricscientometric data data ( (averageaverage percentile, etc.) percentile, etc.)
NNon-on-existexistinging benchmark benchmarkss ( (no systematic information about no systematic information about
works and citations outside main databasesworks and citations outside main databases))
Section 3 (SSH)Section 3 (SSH)
13
Social SciencesSocial Sciences in the world in the world,, regionregion a andnd Sl Slovovakiaakia
1994 SVK CZE HUN
2011 Art Cit top1
top10
Art Cit top1
top10
Art Cit top1
top10
Econ 1264 770 22 9 2927 2665 123 30 1033 2682 230 45
Psy 836 1608 41 23 1686 5941 383 72 1447 12442 466 132
2001-2010 World The most cited Article
100th most cited Article
Economics 883 239
Social Sciences 620 214
Psychology/Psychiatry 2037 399
Matematics 2280 144
Technical Sciences 1822 288
Due to the different publication patterns, it was impossible to Due to the different publication patterns, it was impossible to apply all the assessmentapply all the assessment methods/criteria used for Sections 1 methods/criteria used for Sections 1 and 2and 2
Moreover, no benchmarks were availableMoreover, no benchmarks were available BBibliometric analibliometric analysesyses were possible only to some extent; and were possible only to some extent; and
the four main categories of evaluation (similar to those used for the four main categories of evaluation (similar to those used for Section 1 and 2) were applied:Section 1 and 2) were applied:
QuantityQuantity: : the the number andnumber and the the ‘‘qualityquality’ (’ (categories: books, categories: books, articles, conf.papers, etc.articles, conf.papers, etc.)) of publications, of publications, includingincluding articles and articles and books outside international databasesbooks outside international databases
Global influenceGlobal influence: : number of citationsnumber of citations for these publicationsfor these publications EfEffectivenessfectiveness: : an average number of citations per publicationan average number of citations per publication RelatRelatiivvee impa impacctt: : publicationspublications ci cited more thanted more than 110/20/50/100 0/20/50/100
timestimes ( (roughrough e equiquivalent valent of papers inof papers in 20/10/1/0.1/.01 percentile) 20/10/1/0.1/.01 percentile)
HumaHumanitiesnitiesChanges to evaluation methodologyChanges to evaluation methodology
16
CONCLUSIONSCONCLUSIONS COMPLICATED, COMPLEX PROJECT AND DIFFICULT TASK COMPLICATED, COMPLEX PROJECT AND DIFFICULT TASK
NO ONE SINGLE PARAMETER! NO ONE SINGLE PARAMETER! SET OF CRITERIA + COMPREHENSIVE ASSESSMENT! SET OF CRITERIA + COMPREHENSIVE ASSESSMENT!
NO DIFFERENCE BETWEEN “HARD” AND “SOFT“ NO DIFFERENCE BETWEEN “HARD” AND “SOFT“ SCIENCESSCIENCES
RESEARCH QUALITY ASSESSMENTRESEARCH QUALITY ASSESSMENT = NEW APPROACH = NEW APPROACH
OFFICIAL ACESS TO RELEVANT DATABASES! OFFICIAL ACESS TO RELEVANT DATABASES!
DIFFERENCES FOR HUMANITIES AND ARTSDIFFERENCES FOR HUMANITIES AND ARTS
THE VERY FIRST COMPREHENSIVE PICTURE ABOUT THE VERY FIRST COMPREHENSIVE PICTURE ABOUT RESEARCH QUALITY OF PARTICULAR INSTITUTION IN RESEARCH QUALITY OF PARTICULAR INSTITUTION IN SLOVAKIA SLOVAKIA
NNEW APPROACH = NEW COMPREHENSIVE METHODEW APPROACH = NEW COMPREHENSIVE METHOD
FROM INDIVIDUAL PERFORMANCE TO TEAMS AND INSTITUTION FROM INDIVIDUAL PERFORMANCE TO TEAMS AND INSTITUTION CLASSIFICATIONCLASSIFICATION
17
CONCLUSIONSCONCLUSIONS
“ “THAT IS ONLY A FIRST AND THAT IS ONLY A FIRST AND
SMALL STEP FOR ONE SLOVAK SMALL STEP FOR ONE SLOVAK
RESEARCH INSTITUTION BUT A RESEARCH INSTITUTION BUT A
HUGE LEAP FOR MEASURING HUGE LEAP FOR MEASURING
THE RESEARCH PERFORMANCE THE RESEARCH PERFORMANCE
OF THE SLOVAK RESEARCH!”OF THE SLOVAK RESEARCH!”
Final list of teamsFinal list of teams
The fThe fininalal list of teamslist of teams contains following contains following informinformation:ation:
Name of the instituteName of the institute//departmentdepartment, , where the majority of where the majority of team membersteam members are basedare based
Team collaboration within Team collaboration within SASASS ( (with with other other departmentsdepartments))
FTE of each team memberFTE of each team member Team Leader(s) Team Leader(s) ((including their including their H-index)H-index) ‘‘CCore’/ ‘ore’/ ‘NNon-core’ ton-core’ teaeamm members members Team members outside Team members outside SASASS ( (if if such such existexist))
19
Comparison of total dataComparison of total data in a in a 1010--year periodyear period in the in the
world world
((selectedselected ESI ESI fieldsfields))
WorldWorld benchmark benchmarkss((averageaverage cit citation/articleation/article))
Fields 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 01-10Biology&Biochem 29.64 27.36 24.84 21.97 18.12 14.21 10.98 7.22 3.62 0.52 15.85Chemistry 17.62 17.61 16.07 14.88 13.09 10.57 8.35 5.79 2.98 0.44 10.74Computer Science 7.66 7.93 5.35 3.99 3.51 2.51 3.26 2.13 0.98 0.15 3.75Economics 11.5 11.72 10.39 9.25 7.43 5.58 4.03 2.22 0.97 0.2 6.33Engineering 8.11 7.57 7.04 6.69 5.63 4.54 3.8 2.41 1.2 0.16 4.72Geosciences 17.88 15.62 14.47 12.82 10.79 9.21 6.24 4.07 2 0.43 9.35Mathematics 6.05 5.99 5.39 4.8 4.19 3.36 2.51 1.67 0.86 0.14 3.50Mol Bio&Genetics 46.34 42.79 37.83 33.71 27.86 22.34 16.94 11.1 5.81 0.9 24.56Physics 14.32 13.23 12.2 11.52 10 8.18 5.77 3.8 1.94 0.39 8.14Psychology 21.11 18.94 18.08 15.84 12.89 10.21 7.33 4.42 1.88 0.34 11.10Social Sciences 8.63 8.37 7.67 7.21 6.19 4.82 3.49 2.02 0.88 0.2 4.95
No. of articles with the top citation rates in No. of articles with the top citation rates in a particular field,a particular field, in the last in the last 1010 years years
((ESI ESI classifications)classifications)
WorldWorld benchmark benchmarkss(top percentile)(top percentile)
Physics 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20100.01% 1427 1377 1041 966 671 717 630 336 348 244 370.10% 517 462 395 348 313 248 192 143 103 70 151.00% 148 139 121 109 100 82 66 51 36 19 6
10.00% 36 33 31 29 27 24 20 14 10 5 220.00% 20 19 18 17 16 14 12 9 6 3 150.00% 6 6 6 5 5 5 4 3 2 1 0
SumSummary 2 (chart) mary 2 (chart)
from from 119900 00 assessed researchers in Sections assessed researchers in Sections 1 a1 andnd 2 2, , ~ 1000 are~ 1000 are members of some teammembers of some team
questionnairequestionnaire sent tosent to 63 t 63 teamseams ~ 400 researchers
Social Sciences AssessmentSocial Sciences Assessment
Oddelenie Meno Art WoSCit WoSC/A WoSH WoSm-quotient1% 10% 20% Art Cit Cit/Art100+ Cit50+ Cit30+ Cit20+ Cit10+ CitTeamsSocCom Bianchi G 29 133 4.59 4 0.16 1 4 1 46 201 4.37 2 2 2
Luksik I 14 132 9.43 4 0.19 1 4 1 38 201 5.29 2 2 1Popper M 17 127 7.47 4 0.22 1 4 1 29 189 6.52 2 2 1
XPsi Adamovova L 6 77 12.83 2 0.29 2 3 21 140 6.67 1 2Fickova E 23 146 6.35 5 0.19 2 3 28 165 5.89 1 2 1
IndividualsProgn Balaz V 47 142 3.02 8 0.30 3 2 62 249 4.02 2 1 4Socio Gajdos P 18 14 0.78 2 0.12 3 1 28 147 5.25 1 2 1
23
Only two small teams and two other individuals above the average Almost all involved in wider international collaboration
A
B
C
D
E
X
Y