Collection and analysis of research outputs and impactsDr Ian Viney

Medical Research Council

• Provide evidence for communicating the outcomes and impacts of medical research (economic, societal and academic)

• Quantitative and qualitative information used to develop/strengthen strategic decisions (what worked, what didn’t)

• Understand better how research leads to impact (how can we maximise the opportunities for impact?)

MRC Evaluation Programme

Input Output/Outcome Impact

Improvements to health (living longer and with better quality of life)

Attracting R&D investment (from global business and non-UK funding sources)

Improving the performance of existing businesses

Creating new businesses (that contribute to economic growth and further R&D)

Delivering highly skilled people to the labour market

Academic impact (effects on further research including other disciplines)

Improving public policy and public services

Evaluation Framework

Funding for Research and Training

Generation of new knowledge

Leveraging funding

Dissemination of research

Intellectual property/licensing

Development of new products/processes

Changes to policy

Development of collaborative networks

Research materials

MRC Vision for increasing impact

MRC Delivery plan 2011/12 – 2014/15

Compiling the evidence

• Tracing the interactions between the producers and users of research, establishing evidenced links across as many steps as possible

• Publications may have a minor impact in transferring knowledge to industry, people exchanges and problem solving (collaboration) are more important

• Similarly focussing on hard commercialisation metrics (spin outs, patents etc.) miss a large range of activities (e.g. product development, policy influences)

• Need a holistic approach to output, challenge is to capture these “softer” processes effectively

• Failures (absence of output, closure/re-direction of projects) also have to be factored in

• Advantage to this “micro-economic” approach is that casual links are established, disadvantage is that these links are rarely “generalisable”

• Has to be gathered from researchers, either by employer or funder – there are no repositories for much of this data

• Need to keep administrative burden as low as possible

• For medical research in particular, research programmes benefit from a variety of funding sources, are highly collaborative, and progress over the long-term. Return on investment studies are extremely difficult

MRC e-Val

• MRC e-Val was an online database used to gather outputs/outcomes/impacts arising from MRC research, it was launched in 2008, and we asked researchers to record output realised since 2006

• Over six years we have learnt a lot about asking the right questions, validating, analysing, and presenting the data

• MRC has built a robust dataset, covering now 7 years of output, which allows progress/productivity and quality of MRC research to be analysed, and this evidence to be utilised in decision making

• Need for both quantitative and qualitative information, e.g. to “set case studies in context” of the productivity and quality of all MRC output

• A requirement for comprehensive data across the whole MRC portfolio against which progress and trends can be prospectively tracked over time

“Federated” e-Val – multiple funding agencies, one route for gathering output data

• MRC licensed e-Val approach to Researchfish Ltd.

• Built a single portal for researchers to record the output from their research across multiple funders (Researchfish.com), latest web technology and cloud hosting

• e-Val provides the structure to capture outputs systematically, using a proven approach, each funder receives the information they need and there is the opportunity to collaborate on cross-funder evaluation

• System is flexible and appears to be equally applicable to disciplines outside of medicine (Science and Technology Facilities Council also use the approach)

• 20 funding organisations initially subscribed to Researchfish with a combined annual research spend of £1.8bn

• 12 Research organisations (11 Universities) have also subscribed to Researchfish, obtaining enhanced real-time reports on progress across multiple funding organisations

Progress to date?

• MRC dataset now covers outputs realised 2006 – 2012, from 98% of our portfolio (£2.5bn expenditure across 4,500 awards, 3,500 researchers)

• 6,500 researchers are now using the federated system and this is rapidly growing. Scottish Health Department, STFC, BHF, Arthritis Research UK are examples of funders that have accumulated significant datasets, some over multiple data gathering exercises.

• A further 50 UK organisations are now preparing to gather data, this will increase the investment tracked to £3bn per year (almost all medical research funding in the UK)

• We have used the data to easily compile case studies, and to set these in the context of the progress, productivity and quality of the whole MRC portfolio

• Researchfish now exchanges data with Europe PubMed Central, enhancing information about the links between funding and papers – useful for open access monitoring

Characteristics of the Researchfish system

Output types Guidance and form for entering data

Grant portfolio

Predictive lookup function within Researchfish, driven by

large datasets of funding agencies and collaborators – easy for researchers to fill in

and data is highly standardised

Reporting new products/interventions in development

Categorisation for products and interventions based on NCI

translational research flowcharts and US DoD technology readiness levels

Simple flow charts, unambiguous categories, allow the data to be structured and reported consistentlyAutomated “lookups” suggest categories to researchers – no keying in of organisation/funder names etc.

MRC use of Researchfish data

• spending review submissions to Government (2010, 2013), medical research treated positively

• published and returned data each year to Universities and MRC Units, some of which are ingesting this into their research information systems

• used to adjust/make the case for new funding initiatives (e.g. a further £20m per year investment in experimental medicine - funding underway)

• 2012 quinquennial review of all three MRC Institutes included bibliometric data, providing an international comparison with other research organisations.

• The UK Government has provided additional science funding (e.g. £90m of the £180m biomedical catalyst fund announced in 2011) referring directly to the MRC data.

• MRC has monitored progress against the objectives set out in its Strategic Plan (Research Changes Lives) using outcomes compiled from Researchfish data. These convey good progress since 2009, http://www.rclprogress.mrc.ac.uk/

Headline Quantitative Results (2012 update)

• MRC research produces high quality output. Annual analysis of citations from MRC publications has shown that MRC reported papers have consistently more than twice the world average impact.

• 110 new products or interventions have been launched onto the market (or are undergoing market authorisation), as a result of MRC research, since 2006. The dataset now includes reports of 686 projects developing such products at all stages.

• Reports of more than 300 influences on clinical guidelines worldwide, including 50 NICE guidelines. In the main these involve impacts on recommendations in these guidelines, and are a subset of 3,300 reports of all types of policy influence.

 • 579 new patents have been reported arising in part as a result of MRC

research since 2006, 180 of these (31%) have already been licensed.

• We have compiled evidence of significant impact on the establishment or growth of 104 companies, representing at least 535 new skilled UK jobs. 56 spin out companies were launched since 2006 with significant input from MRC research. There are substantially more productive links to established UK and global companies.

http://www.mrc.ac.uk/Achievementsimpact/Outputsoutcomes/Researchfish2012/index.htm

Wellcome Trust Project Grants 2006Developed Professor Edward Watkins research independence.

NIHR South-West CLAHRC (2007)Provision of institutional programmatic support to support the further translation of this work into a viable treatment.Total value of Peninsula CLAHRC £10m

Basic Research

Applied Research

HealthcarePractice & Policy

1st translational gap 2nd translational gap

MRC Experimental Medicine Grant (2007)“Cognitive training as a facilitated self-help intervention for depression”Allowed for growth of research team and shift in focus from experimental research to evaluation of therapy via RCT.£464k

MRC Brain Sciences Trial Platform Grant (2005)“Preventing depression relapse in NHS practice using Mindfulness-Based Cognitive Therapy" £237k

Mood Disorders Centre (University of Exeter): Cognitive therapy for depression

OutcomesProof-of-principle evidence that training depressed individuals to think about their problems in a more concrete way reduces their symptoms of depression. Pilot work in a multiple case series has demonstrated in patients with major depression that is safe, tolerable, and acceptable. The final stages of the current MRC grant have tested out the efficacy of the intervention in a small scale controlled trial (n =121). The results of the main outcome analysis indicate that the cognitive training guided self-help treatment adds significant clinical benefit in terms of reducing depression relative to treatment-as-usual alone, and that it is of equivalent efficacy to a relaxation based guided self-help treatment in reducing depression, but superior in changing depressogenic thinking such as rumination. The project has therefore developed a novel treatment derived from experimental findings that is easily accessible, cost-effective and able to be disseminated, and could be made widely available (for example via Improving Access to Psychological Treatments IAPT initiative). This treatment material has been copyrighted. The report of the main outcome of the trial has now been prepared as a paper and submitted to a high impact peer-reviewed journal for publication.Collaborations with researchers in the Netherlands (2009) has extended this work to a new population (adolescents at risk) and to a prevention treatment (rather than an acute treatment).The work has been cited in the NICE guideline update on depression (2009)

Wellcome Trust Capital Grant to build the new Sir Henry Wellcome Mood Disorders Centre at Exeter (due to open in 2011)£3.6m

Health Technology Assessment (2010)“Preventing depressive relapse in NHS Practice through mindfulness-based cognitive therapy (MBCT)”£2.1m

Mood Discorders CentreFounded 2003 by Professor Edward Watkins and Professor Willem Kuyken

DiscoveryClinical trial

New treatment

High impact paper

Policy influence

New collaboration

Intellectual property

Overall, the total cost of health services for depression in England in 2007 was estimated to be £1.7 billion whilst lost employment increased this total to £7.5 billion. By 2020 it is expected that depression will

be the second leading cause of disability in the world, and by 2026 the condition is projected to cost the UK health service £3 billion (£12.2 billion cost to the UK with lost employment).

Pipeline analysis

One of 40 therapeutic behavioural interventions in development10 of which are in early “clinical assessment”3 of which have relevance to treating depression

Prospective tracking provides information on the speed of translation into practice

Case studies easily compiled

Screening by the Cytosponge followed by treatment of patients with dysplasia or

intramucosal cancer would reduce the number of cases of incident symptomatic esophageal

adenocarcinoma by 19%, compared with 17% for screening by endoscopy. 2012

paper2012 paper

Co-publication with the private sector - Bibliometrics

• The MRC e-Val dataset includes reports of all publications arising as a result of MRC funded research since 2006, currently this covers 50,000 reports

• In collaboration with Thompson Reuters the MRC has analysed the available information on the location of all authors on these papers (currently 500,000 data points)

• Our preliminary analysis shows co-authorship with industry increasing from 2006 – 2011 from 5% to nearly 7% of papers, and suggest that this is the result of MRC policies which have encouraged interactions with the private sector. We expect this to rise further as data from 2011 is partial.

In the “International comparative performance of the UK research base (2011)” it was stated that UK academic/private sector co-authorship across all disciplines was low (approximately 1.3%)

The data to the left corresponds to MRC researchers co-authoring with at least 720 organisations, approximately 30% of which are UK-based

Location of co-authors - Europe

Location of co-authors - UK

Challenges

• Researchfish successfully draws information from other datasets, such as PubMed (IPO and clinical trial registry data is in test). Where there are unique identifiers data quality can often be validated, and importantly outputs can be de-duplicated.

• Publishing and exporting data is straightforward. More Researchfish information will be displayed in the RCUK gateway to research, Europe PMC displays Researchfish data, some research organisations are ingesting the data etc.

• Challenge is to “synchronise” data with other systems, identifying the same/similar output and if appropriate merging records – clearly a role for CERIF to continue to be developed to encompass all output types

Next Steps

• Continuing to capture information on outcomes in an efficient way across all scientific disciplines, from medicine to the arts/humanities

• Address areas of under-reporting e.g clinical trials

• Maintaining the support of the research community (good communications, regular feedback, build on success), allow re-use of the data e.g. in CVs and most importantly grant applications

• Continue to extend the approach to acknowledge and involve other research team members (“research teams” functionality now live). Collaborators/co-investigators now enter output once and share it. Outputs are uniquely identified. In 2013 7,000 MRC co-investigator records have been uploaded.

• Continue to improve reporting function

Thank you

• Email – evaluation@headoffice.mrc.ac.uk

• Web - http://www.mrc.ac.uk/Achievementsimpact/Outputsoutcomes/e-Val/index.htm

• Twitter – @MRCeVal

Funding organisations Universities

Academy of Medical Sciences University of Aberdeen

Alzheimers Research UK Cardiff UniversityArthritis Research UK University of DundeeAsthma UK University of GlasgowBritish Heart Foundation Imperial College LondonCancer Research UK Newcastle UniversityChief Scientist Office (Scotland) University of Nottingham

Epilepsy Research UK University of OxfordGreat Ormond Street Hospital Children’s Charity University College London

Kidney Research UK  Medical Research Council  NISCHR (Wales Health Department R&D)  

HSC Public Health Agency (Northern Ireland Health Department R&D)

 

Parkinson’s UK  Science and Technology Facilities Council  

Stroke Association  Tenovus  

Organisations subscribing to Researchfish (February 2013)

Total annual research expenditure of funding organisations = £1.8bn

Researchfish “output types”

Section Output type

1 Publications 

2 Collaborations

3 Further funding

4 Next destination

5 Engagement activities

6 Influence on Policy, Patients and the Public

7 Research Materials (Tools, Methods, Databases)

8 Intellectual property and licensing

9 Development of products or interventions

10 Impacts on the private sector

11 Awards and recognition

12 Use of Facilities

13 Other outputs/future steps