rae & ref by hugh mckenna karen cox research research presented at council of deans, seminar,...

RAE & REF

ByHugh McKennaKaren Cox

rresearchesearch

Presented atCouncil of Deans,

Seminar, Manchester, Feb 2009

HEFCE

BRIEF BACKGROUNDBRIEF BACKGROUND

Conducted jointly by the Higher Education Conducted jointly by the Higher Education Funding Councils for England (HEFCE), Scotland Funding Councils for England (HEFCE), Scotland (SHEFC), Wales (HEFCW) and Northern Ireland (SHEFC), Wales (HEFCW) and Northern Ireland (DEL)(DEL)

Purpose:Purpose: to produce ratings of research quality to produce ratings of research quality for the HE funding bodies to determine for the HE funding bodies to determine research funding allocations to HEIsresearch funding allocations to HEIs

RAE 2008RAE 2008 Principles of Equity, Diversity, Equality, Expert Review, Principles of Equity, Diversity, Equality, Expert Review,

Clarity, Consistency, Continuity, Credibility, Efficiency, Clarity, Consistency, Continuity, Credibility, Efficiency, Neutrality, Transparency. Neutrality, Transparency.

All types of research, including practice-based research, applied research, basic/strategic research, interdisciplinary research.

Assessment based on peer review.Assessment based on peer review.

15 Panels and 67 Sub Panels15 Panels and 67 Sub Panels

Sub Panel 11Sub Panel 11• Chair• Prof Hugh McKenna, University of Ulster • Panel Secretaries• Mrs Lilian Caras, Brenda Purkiss• Members• Ms Sarah Buckland, INVOLVE• Prof Charlotte Clarke, University of Northumbria• Prof Karen Cox University of Nottingham • Prof Dame Pauline Fielding, • Prof Martin Johnson, University of Salford • Prof William Lauder, University of Dundee • Prof Karen Luker, University of Manchester• Prof Dame Jill Macleod-Clarke, University of Southampton• Dr Maggs Maguire, Department of Nursing, Scottish Executive• Professor Dave Thompson, University of Leicester• Prof Ruth Northway University of Glamorgan. • Prof Anne Marie Rafferty, Kings College London• Dr Kate Seers, University of Warwick• Prof Roger Watson, University of Sheffield• Prof Anne Williams, University of Wales, Cardiff

Sub Panel 12Sub Panel 12• Chair• Prof Julius Sim, Keele University • Panel Secretaries• Mrs Lilian Caras, Brenda Purkiss• Members• David Billington, Liverpool John Moores • Michael Campbell, University of Sheffield• Aedin Cassidy, University of East Anglia• Stephen Downes, University of Ulster• Pamela Enderby, University of Sheffield• Bernard Gilmartin, Aston University• Martin Griffin, Aston University• Jenny Hewison, University of Leeds• Tracey Howe, Glasgow Caledonian University• Ian Kitchen, University of Surrey• Di Newham, King's College London• Margaret Nicol, Queen Margaret University Edinburgh • David Rogers, University of Portsmouth • Diane Scutt, University of Liverpool• Debbie Sell, Great Ormond Street Hospital• Christina Victor, University of Reading• Keith Wafford, Eli Lilly and Company

Main Panel CMain Panel C

• Professor Maggie Pearson (Chair)Professor Maggie Pearson (Chair)• Professor David Williams (10)Professor David Williams (10)• Professor Hugh McKenna (11)Professor Hugh McKenna (11)• Professor Julius Sim (12)Professor Julius Sim (12)• Professor Robert Hider (13)Professor Robert Hider (13)• Professor Mi Ja KimProfessor Mi Ja Kim• Professor John StammProfessor John Stamm• John Stageman John Stageman • MRC/AMRC/ESRC/DoH/Scot Gov ObserversMRC/AMRC/ESRC/DoH/Scot Gov Observers• SecretariatSecretariat

RAE 2008RAE 2008 January 2005 Guidance to panels issued;

Summer 2005 Guidance on submissions issued;

Autumn 2005 Consultation on criteria and working methods;

February 2006 Final criteria and working methods issued;

31 July 2007 End of assessment period for research income and research student data

31 December 2007 End of publication period (cut-off point for publication of research outputs)

31 October 2007 Census date

30 November 2007 Closing date for submissions

December 2008 Results published

DEFINITION OF RESEARCH • Research - original investigation undertaken in order to gain

knowledge and understanding. It includes work of direct relevance to the needs of commerce and industry, and to the public and voluntary sectors; scholarship*; the invention and generation of ideas... etc, where these lead to new or substantially improved materials, devices, products and processes...

[* The creation, development and maintenance of the intellectual infrastructure of subjects and [* The creation, development and maintenance of the intellectual infrastructure of subjects and disciplines, in forms such as dictionaries, scholarly editions, catalogues and contributions to major disciplines, in forms such as dictionaries, scholarly editions, catalogues and contributions to major research databasesresearch databases.].]

It excludes routine testing and routine analysis of materials, It excludes routine testing and routine analysis of materials, components and processes (such as for the maintenance of national components and processes (such as for the maintenance of national standards…). standards…).

It also excludes the development of teaching materials that do not It also excludes the development of teaching materials that do not embody original research”.embody original research”.

It excludes AuditIt excludes Audit

CRITERIA & WORKING METHODSCRITERIA & WORKING METHODSSub Panel 11Sub Panel 11

““Research activity relevant to the disciplines of nursing Research activity relevant to the disciplines of nursing and midwifery, specialist community public health and midwifery, specialist community public health nursing and all the contexts within which they operate, nursing and all the contexts within which they operate, including policy, practice, education and management”.including policy, practice, education and management”.

Proportional weighting given to quality of Research Proportional weighting given to quality of Research Outputs, Research Environment and Research Esteem; Outputs, Research Environment and Research Esteem;

Opportunity to refer to other Sub Panels and specialist Opportunity to refer to other Sub Panels and specialist advisors;advisors;

CRITERIA & WORKING METHODSCRITERIA & WORKING METHODSSub Panel 12Sub Panel 12

• “The UOA includes (but is not limited to): biomedical sciences; nutrition and dietetics; optometry and orthoptics; radiography; podiatry; occupational therapy; physiotherapy; speech and language therapy; arts therapies; health promotion; psychosocial and ethical aspects of health and healthcare; associated health services research (to include methodological work on quantitative or qualitative procedures)”;

Proportional weighting given to quality of Research Outputs, Proportional weighting given to quality of Research Outputs, Research Environment and Research Esteem; Research Environment and Research Esteem;

Opportunity to refer to other Sub Panels and specialist Opportunity to refer to other Sub Panels and specialist advisors.advisors.

CategorCategoryy

DescriptionDescription

Staff Staff informatioinformationn

•Research-active staff details (RA1)Research-active staff details (RA1)

Category A:Category A: Academic staff in post on census Academic staff in post on census date.date.

Category B:Category B: Academic staff who held contract Academic staff who held contract with institution after 1 January 2001 and left the with institution after 1 January 2001 and left the institution (or, exceptionally, being returned under institution (or, exceptionally, being returned under a different UOA)a different UOA)

Category C:Category C: Other individuals active in research Other individuals active in research in the department as independent investigators at in the department as independent investigators at the census date.the census date.

Category DCategory D: Other individuals active in research : Other individuals active in research in the department as independent investigators in the department as independent investigators after 1 January 2001 but not at the census dateafter 1 January 2001 but not at the census date

* Early Career Staff = entered academic * Early Career Staff = entered academic employment as Category A staff on or after 1employment as Category A staff on or after 1stst Aug Aug 20032003**

Research Research ProfileProfile

•Research output (RA2)Research output (RA2)period 1 January 2001 to 31 December 2007. period 1 January 2001 to 31 December 2007. up to four items of research output for each up to four items of research output for each researcher (subject to panel).researcher (subject to panel).

•Research students (RA3a)Research students (RA3a)No’s full-time & part-time research students No’s full-time & part-time research students and degrees awarded during the period 1 and degrees awarded during the period 1 January 2001 to 31 July 2007.January 2001 to 31 July 2007.

•Research studentships (RA3b)Research studentships (RA3b)No’s research studentships and source of No’s research studentships and source of funding in the period 1 January 2001 to 31 July funding in the period 1 January 2001 to 31 July 2007.2007.

•External research income (RA4)External research income (RA4)Amounts & sources of external funding in the Amounts & sources of external funding in the period 1 January 2001 to 31 July 2007.period 1 January 2001 to 31 July 2007.

Textual Textual descriptiondescriptionss

•Research environment and esteem Research environment and esteem (RA5a)(RA5a)

Research Environment includes info on research Research Environment includes info on research structure, staffing policy, research strategy.structure, staffing policy, research strategy.Indicators of Research Esteem and significance; Indicators of Research Esteem and significance; and any reasonable additional information and any reasonable additional information requested by the sub-panel for period 1 January requested by the sub-panel for period 1 January 2001 to 31 July 2007.2001 to 31 July 2007.

•Individual staff circumstances (RA5b)Individual staff circumstances (RA5b)

Including details of any individual staff Including details of any individual staff circumstances which have significantly affected circumstances which have significantly affected their contribution to the submissions, such as their contribution to the submissions, such as engagement on long-term projects, secondment, engagement on long-term projects, secondment, and any matters covered by equal opportunities and any matters covered by equal opportunities legislation.legislation.

Star Ratings?Star Ratings?FourFourstarstar

Research judged to be Research judged to be world-leadingworld-leading in terms of in terms of originality, significance and rigour originality, significance and rigour regarded as a primary point of reference in its field, which has made or is likely to make an outstanding contribution to knowledge, theory, practice or policy..

Three starThree star Research Research judged to be internationally excellent in terms of originality, significance and rigour, which has made or is likely to make a highly significant contribution to knowledge, theory, practice or policy

TwoTwostarstar

RResearch judged to be at an international standard in terms of originality, significance and rigour, which has made or is likely to make a significant contribution to knowledge, theory, practice or policy

OneOnestarstar

RResearch judged to be original, significant and rigorous, which has made or is likely to make a contribution to knowledge, theory, practice or policy

UnclassifieUnclassifiedd

Work that does not meet the definition of one star, or which does not meet the published definition of

research for the purposes of the RAE.

How did this work in reality?How did this work in reality?

Research outputs Research environment Esteem indicators

25

4*

15201525

u/c1*2*3*

30

4*

20201020

u/c1*2*3*

eg 70%

Overall quality profile

20

4*

10153025

u/c1*2*3*

15

4*

10153525

u/c1*2*3*

eg 20% eg 10%

The overall quality profilecomprises the aggregate of the weighted profiles produced for outputs, research environment and

esteem indicators.

(Minimum 50%)

Quality Level

% of Research Activity

(Minimum 5%)(Minimum 5%)


25

4*

15201525

u/c1*2*3*

25

4*

15201525

u/c1*2*3*

30

4*

20201020

u/c1*2*3*

30

4*

20201020

u/c1*2*3*

eg 70%


20

4*

10153025

u/c1*2*3*

20

4*

10153025

u/c1*2*3*

15

4*

10153525

u/c1*2*3*

15

4*

10153525

u/c1*2*3*

eg 20% eg 10%

The overall quality profilecomprises the aggregate of the weighted profiles produced for research outputs, research environment and

esteem indicators.

(Minimum 50%)

Quality level

% of research activity



25

4*

15201525

u/c1*2*3*

30

4*

20201020

u/c1*2*3*

eg 70%


20

4*

10153025

u/c1*2*3*

15

4*

10153525

u/c1*2*3*

eg 20% eg 10%


esteem indicators.

(Minimum 50%)

Quality Level




25

4*

15201525

u/c1*2*3*

25

4*

15201525

u/c1*2*3*

30

4*

20201020

u/c1*2*3*


25

4*

15201525

u/c1*2*3*

30

4*

20201020

u/c1*2*3*

eg 70%


20

4*

10153025

u/c1*2*3*

15

4*

10153525

u/c1*2*3*

eg 20% eg 10%


esteem indicators.

(Minimum 50%)

Quality Level



Research outputs Research environment Research Esteem

25

4*

15201525

u/c1*2*3*

25

4*

15201525

u/c1*2*3*

30

4*

20201020

u/c1*2*3*

30

4*

20201020

u/c1*2*3*

70%


20

4*

10153025

u/c1*2*3*

20

4*

10153025

u/c1*2*3*

15

4*

10153525

u/c1*2*3*

15

4*

10153525

u/c1*2*3*

25% 5%

The overall quality profilecomprises the aggregate of the weighted profiles produced for research outputs, research environment and

esteem indicators.Quality level

% of research activity

Examples - Quality ProfileExamples - Quality Profile

UoA 11UoA 11 FTE StaffFTE Staff

NumbersNumbers4*4* 3*3* 2*2* 1*1* UnUn

ClassClass

University University AA

5050 15%15% 25%25% 40%40% 15%15% 5%5%

University University BB

2020 0%0% 15%15% 40%40% 45%45% 0%0%

Overview - Sub Panel 11Overview - Sub Panel 11• 761 individuals were returned, submitting 2,851

outputs, mostly from peer reviewed journals;• Selectivity in number of staff returned;• Selectivity in type of staff returned;• Selectivity of panel/sub panel;• 2001 – 43 submissions, 2008 - 35 submissions;

• All thirty-five submissions were reviewed in detail by every sub-panel member;

• All quality profiles represent the consensus of the sub-panel as a whole;

• Strict cognisance was taken of equal opportunity issues and individual staff circumstances.

RAE 2008: Submissions to RAE 2008: Submissions to Sub Panel 11- THE STRENGTHSSub Panel 11- THE STRENGTHS

• 44th out of the 67 subject areas in the RAE;• Internationally recognised research in all submissions;• Strengths in translational/applied research;• Good service user involvement;• Positive impact on patient/client care;• Good interdisciplinary work – various disciplines; • Good PhD completions in review period (446);• Good international collaborations;• Good methodological and theoretical sophistication;• Strategic research appointments;• Research linked to national and international

priorities;


• Impact on national and international health care policy;

• Substantial competitive research grant capture (> £103 M);

• Modern and high quality infrastructure;

• Robust staffing policies;

• Emphasis on capacity building and sustainability;

• Realistic and sustainable research strategies;

• Highly focused, strategically sound, financially viable;

• Good links between NHS, Voluntary bodies and HEIs;

• Esteem indicators - numerous awards, involvement in prestigious national and international committees, funding bodies & editorial boards.

RAE 2008: Submissions to RAE 2008: Submissions to Sub Panel 11- THE WEAKNESSESSub Panel 11- THE WEAKNESSES

• In some institutions service user/public engagement was very limited;

• Few departments were actively engaged in technology and knowledge transfer activities;

• Very few links with industry;• Some researchers working in isolation in poorly

developed research environments;• Lone researchers and small disparate groups

working on the same topic area;• Little evidence of activity around the economic

evaluation of what nurses and midwives do or on the evaluation and impact of new roles;


• Fewer outputs on educational and managerial issues;• Research into aspects of care fundamental to nursing

and midwifery were not sufficiently evident in submissions;

• Over one third of the 1,283 research studentships were supported by the submitting HEIs themselves;

• The doctoral degrees awarded per research student FTE were lower than expected;

• Only fifty-nine research-active staff were returned as designated ‘early career researchers’;

• Insufficient critical mass, inadequate onward investment and inferior succession planning.

Overview- Sub Panel 12Overview- Sub Panel 12• 68 submissions - 18 to this sub-panel for the first time;• 1,622 individuals returned, submitting 6,216 outputs;• External research income totalled £355.7 million;• Varying levels of selectivity exercised by HEIs, in

submissions and outputs;• Heterogeneous in substantive and methodological terms;• Very wide distribution of quality across the submissions;• All quality profiles represent the consensus of the sub-

panel as a whole;• Strict cognisance was taken of equal opportunity issues

and individual staff circumstances;


• Clearly-articulated and focused research strategy;• Coherent and synergistic research groupings, with an

appropriate ratio of researchers to research foci;• Strong evidence of sustainability;• Well-balanced portfolio of research funding, from

competitive peer-review funding sources;• Appropriate and productive links with partners in the NHS

and industry, and with service users;• Good evidence of translational research.; • Outputs showed rigorous and innovative methodology;• Research outputs had clear evidence of actual or potential

impact on practice or policy.


• Evidence of internationally excellent or world-leading research in virtually all the disciplines – especially biomedical science, nutrition and optometry;

• Evidence of investigations into the efficacy or effectiveness of therapeutic interventions;

• Large body of important work on the underpinning basic sciences;

• Studies providing valuable insights in the social and psychological aspects of health, illness, and the delivery of professional care;

• Appropriate and well developed collaborative research links• Some Departments were leading important international

research collaborations• 50% increase over review period in PhDs awarded (2,096).


• Submissions were heterogeneous, consisting of a number of small and seemingly unrelated areas of work, raising questions as to cohesiveness and future proofing;

• Lack of clear evidence of sustainability; • Lack of strategic focus, often with disparate research

groupings;• Underdeveloped research infrastructure;• Inappropriately large number of research foci for the number of

staff submitted;• Weak funding profile;• Discrepancy between the profiles for outputs and environment;


• A number of outputs did not meet the RAE research definition;• Need for an increased focus on the many areas within AHPs

where the evidence base is as yet underdeveloped• 3,426 new research studentships - institutional self-funding

(36%); Research Councils (11%);• High quality doctoral work may not have attracted, or been

unable to access, Research Council funding;• Research outputs that were of poor methodological quality

and/or of little potential impact;• Poorly developed interdisciplinary and/or institutional

collaboration.

Purpose of QRPurpose of QR((points from HEFCE)points from HEFCE)• Creating a sustainable and flexible national baseline Creating a sustainable and flexible national baseline

capacity which enables the sector to respond strategically capacity which enables the sector to respond strategically to a changing external environment to a changing external environment

• To maintain a research base of world leading quality To maintain a research base of world leading quality across the full range of disciplinesacross the full range of disciplines

• To encourage institutions to take investment decisions To encourage institutions to take investment decisions which lead to excellence in research, funded selectively by which lead to excellence in research, funded selectively by reference to robust indicators of research quality.reference to robust indicators of research quality.

• Invest in innovative research, including in new fields and Invest in innovative research, including in new fields and opening new lines of enquiry, supporting early-career staff opening new lines of enquiry, supporting early-career staff who aspire to grant fundingwho aspire to grant funding

• create potential to make connections across subjectscreate potential to make connections across subjects

REF –current state of playREF –current state of play

• HEFCE suggests reduce the burden on institutionsHEFCE suggests reduce the burden on institutions

• Will be designed to take better account of the impact Will be designed to take better account of the impact research makes on the economy and society. research makes on the economy and society.

• A pilot exercise for bibliometric indicators of excellence A pilot exercise for bibliometric indicators of excellence is now underwayis now underway

• HEFCE will set out its plans for all aspects of HEFCE will set out its plans for all aspects of assessment, including user-focused research and assessment, including user-focused research and subjects where bibliometrics are not appropriate, by subjects where bibliometrics are not appropriate, by summer 2009.summer 2009.

Framework for assessment Framework for assessment

RAE assesses a body of research activity in terms of:

• Quality of outputs

• Research environment

• Esteem

HEFCE suggests indicators in REF may also capture

• How a research environment is supportive of User valued research

• Evidence that specific work is esteemed for its quality by particular audiences

Tools and approaches for Tools and approaches for assessmentassessment

HEFCE suggests assessment is likely to include:

• Bibliometric analysis

• Citations?

• Expert review of outputs

• Other available statistical indicators

• Students (PG) and completions

• Income

• Studentships

Cunning PlansCunning Plans• Succession Planning;

• Increase and support Early Career Researchers (ECR)s;

• Ruthless selectivity in all things;

– Staff appointments and support for ECRs -Home grow Vs recruit;

– Publish for high citations;

– Research Grants from prestigious bodies – NIH guidelines;

– Network and collaborate for best esteem nationally and Internationally;

– NHS-HEI collaborations;

– Impact – does the research make a difference?

• Service user involvement from design to dissemination;

• Interdisciplinarity in the widest sense; esteem is important – who you work with

• Translational & policy Research;

• Internal Vs External Studentships;

• Address policy issues;

• Increase the amount and quality of pedagogic research.

ConclusionConclusion• The sub-panel were impressed with most of the submissions and

with the pattern of 3 and 4 star research;

• There are many models of good practice from which developing research groupings can learn in terms of research activities, outputs, environment and esteem;

• Investment by Governments, funding bodies and universities has increased research capacity and developed research leaders;

• These funding streams need to be sustained and enhanced if the upward trajectory and momentum are to continue and if the quality differentials between the strongest and weakest departments are to be addressed.

• Additional slides added from notes Additional slides added from notes taken by Alan Simpson at meeting of taken by Alan Simpson at meeting of Academy of Nursing, Midwifery and Academy of Nursing, Midwifery and Health Visiting Research (UK) Health Visiting Research (UK) 26/02/2009.26/02/2009.

Alison Tierney, Scottish Alison Tierney, Scottish ExecExec• Need a continuous and stable funding stream to Need a continuous and stable funding stream to

maintain and build on progressmaintain and build on progress• In Scotland, over 40% submitted to other panels, In Scotland, over 40% submitted to other panels,

e.g. social policy, psychology – some may be best e.g. social policy, psychology – some may be best researchersresearchers

• What do we do with low quality and sub-Rae What do we do with low quality and sub-Rae researchers?researchers?

• Approx 10% N&M HEI staff submitted to RaeApprox 10% N&M HEI staff submitted to Rae• How do we build structures and processes that How do we build structures and processes that

best support HEI/Health Services partnerships in best support HEI/Health Services partnerships in research and how best to measure outcomes?research and how best to measure outcomes?

• Few joint academic-services authors of papers (7-Few joint academic-services authors of papers (7-10%?)10%?)

Developing HEI/NHS Developing HEI/NHS partnerships for researchpartnerships for research

• Sustainable partnershipsSustainable partnerships• Joint targets for researchJoint targets for research• Capacity building jointly agreedCapacity building jointly agreed• Jointly managed research programmesJointly managed research programmes• More shared infrastructure (as in medicine)More shared infrastructure (as in medicine)• Joint and Honorary appointments at senior levelJoint and Honorary appointments at senior level• Clinical-academic careers (see Post Registration Clinical-academic careers (see Post Registration

Career Framework for Nurses, Welsh Assembly Career Framework for Nurses, Welsh Assembly consultation document, November 2008)consultation document, November 2008)

Professor Maggie PearsonProfessor Maggie PearsonChair, RAe 2008 Main Panel CChair, RAe 2008 Main Panel C

• 70 HEIs in nurse education70 HEIs in nurse education

• 1996 36 HEIs and 396.5 staff submitted 1996 36 HEIs and 396.5 staff submitted

• 2001 43 HEIs and 578.4 staff2001 43 HEIs and 578.4 staff

• 2008 36 HEIs and 641.7 staff2008 36 HEIs and 641.7 staff

• So, over half did not submit – does this So, over half did not submit – does this matter?matter?

• 1996 11.1 staff per submission1996 11.1 staff per submission• 2001 13.5 2001 13.5 • 2008 17.82008 17.8

• 7 HEIs submitted only in 19967 HEIs submitted only in 1996• 5 HEIs submitted only in 20015 HEIs submitted only in 2001• 14 ‘dropped out’14 ‘dropped out’• 9 ‘new entrants’ in 20089 ‘new entrants’ in 2008• 12 HEIs submitted to all 3 exercises12 HEIs submitted to all 3 exercises• 7 high quality7 high quality

2008 C Report2008 C Report

• Stronger groupsStronger groups

• Strategic, strong Strategic, strong leadership, clear focileadership, clear foci

• Peer reviewed, Peer reviewed, competitive fundingcompetitive funding

• Critical mass (no lone Critical mass (no lone researchers)researchers)

• Institutional investmentInstitutional investment

• National/international National/international collaborationscollaborations

• Weaker groupsWeaker groups

• Lack of strategyLack of strategy

• Weak leadershipWeak leadership

• Small, disparate groupSmall, disparate group

• Lone researchersLone researchers

rae & ref by hugh mckenna karen cox research research presented at council of deans, seminar,...

Documents

university of wales

university of leedstracey

university of surreydi

basicstrategic research

research income

applied research

interdisciplinary research

types of research