aamc's research evaluation project and rand report release

AAMC’s Research Evaluation Project and RAND Report Release

Webinar

Ann Bonham, PhD Chief Scientific Officer July 8, 2013

Background

AAMC Research Evaluation

Initiative

AAMC Research Evaluation

Initiative

Health Equity Research Expert Panel

New & Shifting Research Models Expert Panel

Foundational & Basic Science Research Expert Panel

Housekeeping

• We welcome you to type in your questions and comments at any time during the webinar, we will save time at the end to address as many of them as we can.

• We also welcome you to submit questions on this initiative at any time via [email protected]

• This webcast will be will also be recorded, and posted on our website, along with a collection of FAQs. If we don’t get to your question at the end of the presentation, we will try to address it online.

mailto:[email protected]

Measuring Research A guide to research evaluation frameworks and tools

Sue Guthrie, Steven Wooding and Jonathan Grant

Webinar, July 2013

7

Outline of presentation

• Background

– Who we are

– Policy background

– Purpose of report

• Frameworks and tools reviewed

• Understanding the challenges and trade-offs in research evaluation

• Developing a customised research evaluation framework

• Next steps and discussion

8

RAND Europe is an independent not-for-

profit public policy research institute

• Not a university or management

consultancy — but with capabilities of both

• Part of the global RAND Corporation

• Work across the breadth and depth of

government

• Strongly held values of quality and

objectivity

• Provider of evidence

• Track record in research evaluation

“help improve policy and decisionmaking

through research and analysis”

9

The project team

Jonathan is a Principal Research Fellow and former president at

RAND Europe with a key interest in research evaluation. He has

conducted a wide range of studies for clients in the UK, Middle East

and Australia.

Steven Wooding is a Research Leader at RAND Europe with an

extensive portfolio of work in research evaluation and science

policy. He has worked for clients in the UK, Canada, USA and

Australia.

Sue Guthrie is a Senior Analyst at RAND Europe working primarily

in the fields of research evaluation and science policy. She had

conducted research for clients in the UK, Europe, Canada and the

Middle East.

10

A brief history of research evaluation (1)

• 1960/70s (USA) – Project Hindsight, Comroe and Dripps, etc.

• 1980s (UK) – Research Assessment Exercise (RAE) and performance

management of research in universities

• 2002 (UK) – Joint funding bodies’ review of research assessment

– “the need to fully recognise all aspects of excellence in research (such as

pure intellectual quality, value added to professional practice, applicability,

and impact within and beyond the research community)”

• 2004-07 (Australia) – Research Quality Framework

• 2006 (USA) – National Institutes of Health (NIH) Clinical and Translational

Science Awards

• 2006 (UK) – Government announces its intention to replace the RAE after

2008 with an assessment system based on metrics

11

A brief history of research evaluation (2)

• 2009 (USA) – American Recovery and Reinvestment Act (ARRA)

• 2010 (USA) – Science and Technology for America's Reinvestment: Measuring

the Effect of Research on Innovation, Competitiveness and Science (STAR

METRICS)

• 2009-11 (UK) – Higher education funding bodies undertake impact pilot and

announce decision to include impact in the Research Excellence Framework

(REF) 2014

– “The assessment of impact will be based on expert review of case studies

[that] include any social, economic or cultural impact or benefit beyond

academia.”

• 2012 (Australia) – Excellence in Innovation (EIA) Trial

– “Universities can generate compelling case studies of impact across the

whole range of disciplines and impact areas”

12

Why research evaluation matters

• Need to demonstrate accountability for the investment of public

funds in research

– demand for good governance, particularly in climate of fiscal

austerity

• Shift in emphasis from purely summative evaluations to more

formative and comprehensive evaluations

– cover wider outputs from research

• Interest in and demand for the evaluation of research is increasing

internationally

13

What did AAMC want from the project?

• Put research evaluation on the agenda

– help institutions align mission and incentives

– move away from the ranking business

– understand what impacts are, including clarity around

definitions for quality and impact

• Consider how AAMC members

might develop research

evaluation frameworks

14

Aims of the study

• Act as a ‘how-to guide’ to

evaluating research

– Understand the challenges

and trade-offs in evaluating

research

– Provide examples of

frameworks and tools used

for evaluating research

internationally

• Promote discussion of this

topic of increasing importance

Report available at: http://www.rand.org/pubs/monographs/MG1217.html

15

Our approach

• Review of existing frameworks and tools for the evaluation of research

– Detailed review of 6 frameworks

– Brief review of a further 8 frameworks

– Review of 10 tools

• Analysis of the characteristics of tools and frameworks using a factor analysis approach

– What are the trade-offs?

– What can be learnt?

• Developed decision tree to aid development of customised research evaluation frameworks

16


• Background


– An example framework – Excellence in Research for Australia (ERA)

– An example tool – case studies




17

We reviewed six frameworks …

• Research Excellence Framework (REF), UK – assesses performance

of UK universities to determine funding allocation

• STAR METRICS, US – uses data mining and other low burden

methods to account for federal R&D spending

• Excellence in Research for Australia (ERA), AU – uses bibliometrics,

and other quantitative indicators, to map R&D output

• Canadian Academy of Health Science (CAHS), CA – aims to provide

consistency and comparability while retaining flexibility

• National Institute of Health Research (NIHR) Dashboard, UK –

provides performance management information at various levels of

aggregation

• Productive Interactions, EU – flexible approach, aims to help

institutions learn and improve their performance against their own

goals

18

… and ten tools

• Bibliometrics

• Surveys

• Logic models

• Case studies

• Economic analysis

• Peer review

• Data mining

• Interviews

• Data visualisation

• Site visits

• Document review

19

An example framework: ERA Origin and rationale: Perceived need to include assessment of quality in block funding allocation (previously volume only). Advocacy purpose - demonstrate quality of Australian research

Scope: Assesses quality, volume, application of research (impact), and measures of esteem for all Australian Universities at disciplinary level

Application to date: First round in 2010, broadly successful. Next round 2012, with minor changes. Intended for funding allocation, but not used for this as yet

Measurement: Indicator approach, uses those appropriate at disciplinary level. Dashboard provided for review by expert panel

Analysis: Broadly positive reception. Meets aims, and burden not too great. Limitation is the availability of appropriate indicators

Wider applicability: Should be widely applicable, criticism limited in Australian context. Implementation appears to have been fairly straightforward

20

SWOT analysis for ERA

Strengths • Acceptable to research community in

Australia

• Burden on participants is moderate

• Indicator driven

• Produces a single performance

indicator, which can be used for

ranking

• Multi-disciplinary

Weaknesses • Indicator driven

• Still moderated through peer review,

reducing objectivity

• Not comprehensive – academic focus

• Summative

• Burden relative to return is high (not

yet used for funding allocation)

• Requires some central expertise

(bibliometric expertise on panel)

Opportunities • Potential to add new indicators

Threats • No funding implications

• Politics informed its development

• Government and public appetite to

include impact is limited in Australia

(limits potential for development)

Broken Link designed by Stephen JB Thomas from The Noun Project

21

An example tool: Case studies

What is it?

Flexible enough to capture a wide variety of impacts, including the

unexpected, and can provide full context around a piece of research,

researcher, or impact

When should it be used? Primary limitation is that findings are less generalisable - best used

where examples are needed rather than full coverage. Often used to

provide the contextual information alongside another approach that

provides generalisable information and to understand process variables

in depth

How is it used? Can be used in a variety of ways depending on aims. Key

considerations are the unit of analysis and the sample selection for

multiple case studies. Structure and approach should reflect the purpose

Watch from The Noun Project; Question designed by Martin Delin from The Noun Project

22


• Background



– Key findings of factor analysis



23

Key findings of analysis

• There is no silver bullet

• The framework should be designed based on the purpose of the

evaluation

• Research evaluation tools typically fall into one of two groups

• There is a range of possible units of aggregation

• There are some perennial challenges to research evaluation that

need to be addressed

• Research evaluation approaches need to suit their wider context

• Implementation needs ownership, the right incentives and support

24

There is no silver bullet

Designing a research evaluation framework requires trade-offs:

• Quantitative approaches tend to produce longitudinal data, do not

require judgement or interpretation and are relatively transparent, but

they have a high initial burden

• Formative approaches tend to be comprehensive, evaluating across

a range of areas, and flexible, but they do not produce comparisons

between institutions

• Approaches that have a high central burden tend not to be suitable

for frequent use

• Approaches that have been more fully implemented tend to have a

high level of central ownership

• Frameworks that place a high burden on participants require those

participants to have a high level of expertise (or should provide

capacity building and training to achieve this)

25

The framework should be designed based

on the purpose of the evaluation

• Analysis

• What works in research funding?

• Advocacy

• ‘make the case’ for research funding

• Accountability

• To taxpayer, donors, etc.

• Allocation

• What to fund (institution, field, people …)

Advocacy

Allocation

Analysis

Accounta

bility

26

Research evaluation tools typically fall into

one of two groups

• Group1: formative tools that are

flexible and able to deal with

cross-disciplinary and multi-

disciplinary assessment

• Group 2: summative tools that do

not require judgement or

interpretation, and are

quantitative, scalable,

transparent, comparable and

suitable for high frequency,

longitudinal use.

Gro

up

2 c

hara

cte

ris

tics

Group 1 characteristics

Bibliometrics Data

mining

Economic analysis

Surveys Logic models

Document

review

Peer

review

Site

visits

Interviews Case

studies

Group 2

Group 1

• Multiple methods are required if researchers’ needs span both groups

27

There is a range of possible units of

aggregation

• The units of aggregation

used for collecting,

analysing and reporting

data will depend on:

• target audience(s)

• tool(s) used

• Units of collection,

analysis and reporting

are interdependent

Research

group

Institution

Department or

programme

Field

Research

system

Project

Researcher

ER

A

RE

F

SM

CA

HS

NIH

R

PI

28

There are some perennial challenges to

research evaluation

29

Research evaluation approaches need to suit

their wider context

• Acceptability and credibility

• Differences between countries

• Need to ensure framework

does not discriminate

30

Implementation needs ownership, the right

incentives and support

• Where compulsory, the challenge is to obtain support from the

academic and wider community

• Where participation is voluntary, incentives need to be in place to

promote and sustain uptake

• In both cases, participants need to be given the skills necessary for

the process, through simplicity, training or a toolkit

• In all cases, strong central ownership is needed for effective large-

scale implementation

31


• Background




– Identified a set of key questions to consider in developing a framework

– Developed a decision tree to support the development of a framework


32

A decision tree for developing a research

evaluation framework

33

A decision tree for developing a research

evaluation framework

34


• Background





– Piloting of set of measures and metrics for AAMC membership

– Discussion?

35

Two expert panels will develop and test set

of metrics and measures for evaluating

research • Two panels have been established of ~7 experts each

– Health Equity and Research

– New and Shifting Research Models

• Aim is to develop evaluation framework(s), metrics and measures

that can be implemented, tested and piloted for wider dissemination

as a potential model for other areas of research

• Focus is on self evaluation, not competitive benchmarking

• RAND Europe working with AAMC to facilitate this process

– Starts in Sept 2013 with two workshop, and will report in Sept 2014

36

Questions and discussion

Q&A

• Please submit your questions and comments via the chat feature.

• We also welcome questions and feedback about this initiative anytime at [email protected]

• This webinar will be recorded and posted online with FAQs at www.aamc.org/researchevaluation

mailto:[email protected]

http://www.aamc.org/researchevaluation

aamc's research evaluation project and rand report release

Documents