beyond bibliometric indicators: a 360° evaluation of research impact
TRANSCRIPT
![Page 1: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/1.jpg)
BEYOND BIBLIOMETRIC
INDICATORS: A 360°
EVALUATION OF RESEARCH
IMPACT
W. Nicol Keith, Glasgow University
![Page 2: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/2.jpg)
The professional development
framework to realise the
potential of researchers
Major new approach to researcher
development
Builds the UK research base
Develops world-class
researchers
Enhances the personal, professional
and career development of
researchers
We train & expect our
researchers to be well
rounded
Should we consider research
assessment & impact in the same way?
![Page 3: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/3.jpg)
Why do we need research assessment?
![Page 4: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/4.jpg)
Who do we assess?
Institute: a group of researchers with an articulated shared mission, operating within one or
more research programmes/units under the same management
![Page 5: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/5.jpg)
Defining the context: What do we assess?
![Page 6: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/6.jpg)
Examples of complex impact assessments in practice
![Page 7: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/7.jpg)
How do we assess? Evidence for the REF
![Page 8: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/8.jpg)
REF Examples of Impact & evidence
![Page 9: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/9.jpg)
Examples of complex impact assessments in practice:
Training future leaders in field of cancer control
![Page 10: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/10.jpg)
Fellowship Portfolio • The UICC has a range of Fellowship schemes & these can be viewed as
a portfolio.
• An impact evaluation of the UICC Fellowship portfolio was conducted The survey spanned years 1997- 2010 and over 600 Fellows reported on the perceived impact of the Fellowship on their research capacity, skills/knowledge & professional influence levels.
• Evaluation to date highlights the power of the portfolio as each has unique features which together deliver training relevant to gaps in skills and knowledge.
![Page 11: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/11.jpg)
Who do we assess?
![Page 12: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/12.jpg)
Background to the review
![Page 13: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/13.jpg)
Summary of Grantees
– ICRETT
• Survey spans years 1997- 2010
• 1120 number of grantees contacted
• 447 grantees responded
• Map shows geographic distribution of respondents
YY
• Survey spans years 1998-2010
• 139 number of grantees contacted
• 95 grantees responded
• Map shows geographic distribution of respondents
ACSBI
• Survey spans years 1998-2010
• 103 number of grantees contacted
• 57 grantees responded
• Map shows geographic distribution of respondents
![Page 14: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/14.jpg)
Overview of Fellowship Satisfaction & Performance
Success is measured across several areas (Research Capacity & Output; Skills & Knowledge; Professional Influence).
YY grants establish scientific projects and as such generate high numbers of joint publications.
ICRETTs are short term grants and for technology/skills transfer which lead to publication, increased knowledge or implementation of skills in for example a clinical setting.
ACSBI fellowships are advanced training grants which establish young investigators
![Page 15: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/15.jpg)
Evidence of activity: Joint publication
![Page 16: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/16.jpg)
Case study: YY Evaluation & Key Findings
• The objectives of the YY Fellowship are – To initiate, set up or pursue bilateral cancer research projects with collaborating investigators abroad
– To exchange and exploit complementary skills and material
– To receive training in advanced experimental research methods and techniques
• It is aimed at individuals with scientific or medical qualifications & a minimum of two years postdoctoral
experience who are actively engaged in cancer research
• Survey spans years 1998-
2010
• 139 number of grantees
contacted
• 95 grantees responded • number of invalid emails
unknown
• Map shows geographic
distribution of respondents
![Page 17: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/17.jpg)
Professional Development
• Making an impact on Project Management & Leadership skills
• Focusing the future Research Direction
• Building the Research Programme
![Page 18: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/18.jpg)
Increasing Research Capacity & Output
![Page 19: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/19.jpg)
Building Skills & Knowledge
![Page 20: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/20.jpg)
Increasing Professional Influence at Home & in the wider
International Community
![Page 21: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/21.jpg)
Reasons for Success
• Reasons for Success – The survey indicates that YY Fellowships have a positive impact on all major key
performance indicators associated with a well rounded, successful international cancer researcher.
– In order to gain additional insight into the possible underlying factors driving success, the associations between variables was analysed using cross tabulation analysis. This analysis complements the basic survey output.
– Cross tabulation analysis is useful for showing a side by side comparison of survey questions to determine how they are interrelated. In statistical terms, it is a joint distribution between two (or more) variables.
• Summary of Findings – An Effective HOST/MENTOR Increases Fellowship Success – Publishing with the MENTOR increases chances of gaining Funding – The increased Knowledge & Skills resulting from the Fellowship help define future
Research Direction & Build the Research Programme – A successful Fellow has increased Project Management & Leadership skills – A successful Fellow is Internationally competitive – A supportive HOME environment is important to the implementation of training
![Page 22: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/22.jpg)
An Effective MENTOR / GRANTEE Relationship Increases
Fellowship Success
Fellowship Success
• An effective Mentor/Grantee relationship
results in:
• Achieving Objectives
• An increase in Research Skills
• Increased Knowledge
• Publications
• This allows for:
• increased Funding
• Increased Professional Influence
• A positive impact on defining Research
Direction & Building a Research
Programme
• Together, these result in Improved
Management & Leadership Skills & an ability to
Compete at an International Level
![Page 23: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/23.jpg)
A supportive HOME environment allows successful
implementation of training
For the benefits of
the Fellowship
experience to be
fully exploited, the
Home Supervisor &
Institution needs to
be fully supportive
of the returning
Grantee
![Page 24: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/24.jpg)
Benefits to the Home Institute
Skills/tech transfer
Publications grantee/host supervisor: publications are an
internationally recognized indicator of success and
reputation. Fellows publications are contributing to the world
knowledge of cancer.
Returning fellows have an increased ability to attract
funding. The flow of funds into home institutes increases the
research capacity and knowledge generation.
The home benefits from the networks established by the
fellows as these extend influence internationally.
![Page 25: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/25.jpg)
Impact Case Study: Leadership, influence &
capacity building
Testimony of Dr Rajesh Dikshit of ICRETT and ACSBI Fellowship opportunities: YouTube Fellowship impact
presentation at WCC
![Page 26: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/26.jpg)
Impact Evaluation of the
American Cancer Society international
fellowships for beginning investigators
(ACSBI)
Aligning Sponsor strategy and training
opportunities
![Page 27: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/27.jpg)
Geographic distribution of grantees
After 2007
Between 2002 -2007
Before 2002
ACSBI
• Survey spans years 1998-2010
• 103 number of grantees contacted
• 57 grantees responded
• Map shows geographic distribution of respondents
![Page 28: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/28.jpg)
Area of work
![Page 29: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/29.jpg)
Area of work & date
![Page 30: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/30.jpg)
Job title
![Page 31: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/31.jpg)
Job title & date
![Page 32: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/32.jpg)
UICC Experience: Who or what do we assess?
• Impact assessment is context dependant – Low-middle income
countries
– Lab training / service development /implementation of technology
• Use it to fulfil current needs & identify future needs
• Complex research situations require more than impact factors, H-index & funding income
![Page 33: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/33.jpg)
A few thoughts on evidence of influence &
impact……Social media & open access
• Some Advantages
– Stay current on new research
– Find resources
– Share & discuss ideas
– Develop local / international hubs
– Career advice
– Job opportunities
– Evidence of reputation, influence & impact
![Page 34: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/34.jpg)
Researcher ID: visualising influence
• Geographic
locations for
publications
that have
cited our
work
• Geographic
locations of
co-authors
![Page 35: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/35.jpg)
• Endorsements for Skills
& Experience
• Recommendations
• Publications
![Page 36: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/36.jpg)
Example of Science Networking Site: ResearchGate
• Assess impact of
your publications
– Downloads
– views
![Page 37: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/37.jpg)
Example of Science
Networking Site:
ResearchGate
• Assess impact of
your institute
– Downloads
– Views
– collaborations
![Page 38: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/38.jpg)
Example of Science Networking
Site: ResearchGate
![Page 39: Beyond bibliometric indicators: A 360° evaluation of research impact](https://reader031.vdocuments.us/reader031/viewer/2022030319/586694121a28ab45408b8475/html5/thumbnails/39.jpg)
In conclusion: Why, Who & What
Choose the right tools to measure impact
Look at the appropriate evidence