Systematic reviews: making them policy relevant

A briefing for policy makers and systematic reviewers

Sandy Oliver, Kelly Dickson, Mukdarut Bangpan EPPI-CentreSocial Science Research UnitUCL Institute of EducationUniversity College London

December 2015

EPPI-Centre

i

Authors

This report was prepared by Professor Sandy Oliver, Kelly Dickson and Dr Mukdarut

Bangpan (EPPI-Centre, UCL Institute of Education, University College London).

Acknowledgments

The authors are very grateful to the policy makers and systematic reviewers interviewed

for this study, and to the advisory group: Elie Akl, Karen Daniels, Phil Davies, Etienne

Langlois, Simon Lewin and Kent Ranson. Additional material was provided by Kathryn

Oliver following a systematic review of barriers to and facilitators of the use of evidence

by policymakers.1

The work was funded by the World Health Organization through the Alliance for Health

Policy and Systems Research (research grant no. 2013/323807-0; PO-No. 200770388). The

opinions expressed in this publication are not necessarily those of the EPPI-Centre or the

funders. Responsibility for the views expressed remains solely with the authors.

Competing interest:

The authors are systematic reviewers. They have published systematic reviews under the

auspices of Cochrane (SO, KD), the Campbell Collaboration (SO), the Collaboration for

Environmental Evidence (SO, KD, MB) and the EPPI-Centre (SO, KD, MB) and have ongoing

formal roles with the Alliance for Health Policy and Systems Research (SO), Cochrane (SO),

the Campbell Collaboration (SO) and the EPPI-Centre (SO, KD, MB).

This report should be cited as: Oliver S, Dickson K, Bangpan M (2015) Systematic reviews:

making them policy relevant. A briefing for policy makers and systematic reviewers.

London: EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University

College London.

ISBN: 978-1-907345-83-8

© Copyright 2015

ii

Contents

Abbreviations .......................................................................................... iii

Key messages ........................................................................................... 1

Informing policy decisions with systematic reviews of research evidence ..................... 2

Systematic reviews for strengthening health systems ............................................ 4

Developing policy relevant questions ................................................................ 4

Recognising diversity in policy interest and systematic reviews ................................ 5

Aligning the motivations of policy interest and research efforts ................................ 5

Emphasising engagement between policy and research .......................................... 6

Supporting review activity with conducive structures ............................................ 6

Standardising procedures for engagement .......................................................... 7

Consequences of collaborative working ............................................................. 8

Four models for producing policy relevant systematic reviews .................................. 8

Recommendations for fostering policy relevant systematic reviews .......................... 13

Recommendations for enhancing capacity of policy makers to use evidence ................ 15

iii

Abbreviations

LMIC Low and Middle Income Country/ies

uMIC upper Middle Income Country

lMIC lower Middle Income Country

1

Key messages

Policy relevant systematic reviews are available in different models:

Where multiple policy makers face similar decisions across the world and key

concepts are clear and widely agreed….

….evidence is available in libraries of systematic reviews with standardised formats

that are kept up-to-date and easy to access.

Where specific policy makers face urgent decisions and key concepts are clear and

widely agreed…

…rapid delivery is helped by: systematic review teams being on call; using or

reanalysing existing systematic reviews; limiting the breadth or depth of any new

systematic reviews; or accelerating review procedures with a larger team, less

rigorous methods or advances in information technology.

Where multiple policy makers face similar decisions across the world but key

concepts lack clarity…

…time for wide consultation and debate to understand and agree the key concepts

in advance helps produce widely relevant systematic reviews.

Where specific policy makers face urgent decisions but key concepts lack clarity…

…knowledge brokers can help policy makers and systematic review teams

collaborate to quickly shape a precisely focused review that can be delivered

rapidly.

These systematic review models address policy makers’ requirements for use of evidence:

Relevance is achieved by interaction between policy makers and researchers during

the review process.

Ready access is through publicly available libraries of systematic reviews or close

collaboration with research teams.

Timeliness is addressed by publicly available libraries, knowledge brokers and rapid

review methods.

Policy makers and systematic reviewers making decisions about the production or use of

systematic reviews can learn from research about collective decision making.

Having members representing the full range of stakeholders can bring a fuller range

of relevant knowledge to discussions, although increasing the size of a committee

above 12 members has diminishing returns.

Larger groups offer greater credibility and wider acceptance of decisions.

Time, formal processes and good facilitation allow better sharing and consideration

of specialist knowledge, and exploration of issues requiring judgements.

Policy makers’ use of evidence for decisions can be enhanced by:

Mass mailing a printed bulletin which summarises systematic review evidence…

…when there is a single clear message, if the change is relatively simple to

accomplish, and there is a growing awareness by users of the evidence that a

change in practice is required.

Evidence is lacking about the effects of multifaceted interventions for developing

awareness and knowledge of systematic review evidence and the skills for implementing

this evidence.

2

Informing policy decisions with systematic reviews of research

evidence

Policy decisions are made without research evidence when there is a lack of timely access

to clear and relevant research findings, a lack of research literacy amongst decision

makers or a lack of funds for research or implementation.1 This last barrier is discounted

by the assertion that:

If you are poor, actually you need more evidence before you invest, rather than if

you are rich. [Dr Hassan Mshinda, Director of the Tanzanian Council for Science

and Technology].

Collaboration between policy makers and researchers not only enhances policy makers’

use of research evidence1 but, as this briefing note explains, it can also enhance the policy

relevance of research.

As health systems research grows, study findings are increasingly used in efforts to

strengthen health systems. This includes drawing on multiple studies to inform decisions.

Drawing on multiple studies increases the size of the evidence, offers a choice of better

studies and of studies conducted in a range of contexts so, rather than relying on single

studies, it provides ‘average’ results or results related to different contexts. These are the

benefits driving the production of systematic reviews of evidence.

A systematic review is a review of research literature using methods that are systematic,

explicit and accountable2. Systematic reviews can be considered relevant to policy (and

policy makers) when they present findings clearly for policy audiences to: illuminate

policy problems; challenge or develop policy assumptions; or offer evidence about the

impact or implementation of policy options; and take into account diversity of people and

contexts3. Systematic reviews may configure the findings of studies on an issue to

construct a theory about health systems. More often they aggregate the findings of similar

studies to test a theory or even mix these approaches within frameworks that make sense

to stakeholders to present what can be learnt from the available literature4.

3

Figure 1 shows the steps typically taken for a systematic review that synthesises the

effects of an intervention, highlighting opportunities for input from people designing,

managing, working in or using health systems. Systematic reviews that build or explore

theories, rather than test them, are more iterative.

Figure 1: Typical steps for conducting a systematic review (adapted from Gough et al.

20122)

4

Systematic reviews for strengthening health systems

Systematic reviews increasingly inform national and international policy. However, as a

tool for spanning the worlds of policy and research for health systems, reviews have

encountered two major challenges.

First, policy makers frequently ask questions about broad, complex issues, where causal

pathways between intervention and outcome are long and raise issues that cut across

academic disciplines.

It’s basic sciences, it’s operational, and [it’s] health systems… (uMIC policy maker)

Second, local policy makers who are tasked with implementing packages of interventions

are interested in how interventions are influenced by each other and local contexts and

these methods are not yet well developed.

[In] health systems research, there’s not enough standardised health systems

methodology [so] that you can... contextualise health systems decisions. That’s

part of the problem... So the health systems field for systematic reviews, from

where I am sitting, is very, very (pause) immature. (uMIC policy maker)

Consequently, researchers have often been overwhelmed by the breadth of literature and

number of studies to be navigated, or policy makers have been disappointed either by the

time taken to review the literature, or by the limited findings from narrower, more

manageable investigations.

Developing policy relevant questions

Good policy-relevant systematic review questions emerge from combining knowledge of

the policy problems faced and decisions to be made, with knowledge of the evidence

available and methods required to fill the gaps. The former is largely held by people

responsible for enhancing and managing health services, while the latter is largely held by

researchers.

5

Good systematic review questions…

Are important for policy debates and decisions…

…as judged by people making those

decisions

Have not been answered by systematic

reviews before…

…as apparent from maps of systematic

reviews

Have consensus over the definitions of

their key concepts…

…which is explored in background literature

and discussions between stakeholders

Find sufficient studies to provide useful

answers…

…as judged by maps of primary studies and

initial searches by the review team

Avoid overwhelming numbers to allow a

manageable review…

…as judged by discussions about the initial

searches by the review team

Recognising diversity in policy interest and systematic reviews

Policy makers and systematic reviewers, often with limited experience of each other’s

worlds, need to recognise the diverse interests and responsibilities of the policy world and

the diverse products of systematic reviewing. For instance, research needs will differ

depending on the scale of policy responsibility (local, national and international) or

whether responsibilities are focused on particular populations or particular programmes.

Similarly, systematic reviews vary in scale, depth and the similarity of included studies.4

When reviewing an unfamiliar literature a useful starting point is to describe the literature

available in terms of policy interests and research designs, and subsequently focus effort

on reviewing a subset of studies that are likely to provide usable findings. Only once the

subset is identified will the appropriate reviewing methods be apparent.

‘They’re called systematic because they’re transparent, and because they treat

every piece of research the same way – that’s not the same as every review

treating studies in the same way…’ (uMIC reviewer)

Aligning the motivations of policy interest and research efforts

Given the diversity of policy research needs and systematic review products, aligning the

motivations of policy makers and systematic reviewers is an essential first step for a

satisfactory project. Policy problems may call for imminent decisions, making time the

priority. Alternatively widespread, enduring problems may call for evidence that can

convince multiple users, making generalisability the priority. Imminent decisions may

attract systematic reviewers motivated to inform policy decisions, while enduring

problems drive priorities for review libraries offering evidence as ‘public goods’. Similarly,

systematic review methods, and the skills they require, differ depending on whether their

purpose is to understand a problem, compare policy options, or inform policy

6

implementation. Given this diversity of policy research needs and systematic review

products, aligning the motivations of policy makers and systematic reviewers is an

essential first step for a satisfactory project. This may involve mechanisms for encouraging

policy teams to use evidence in their business cases:

If [staff] get personal kudos for using evidence and it becomes part of their

professional integrity this can strengthen the incentive to use evidence. (Evidence

into Action and What Works Team 2014)

and mechanisms to encourage review teams to focus on policy interests and present

reader-friendly summaries of their findings.

Policy makers are less concerned about the methods compared to the question and

the answer (uMIC reviewer).

Emphasising engagement between policy and research

Most guidance about systematic reviewing addresses the technical aspects of review

methodology. Yet systematic reviewing is a social enterprise – success also depends on

whether and how people work together, particularly how policy teams and review teams

understand each other’s interests, and how they work together to focus a review and

interpret the findings.

In an ideal world, you would not be commissioning a SR unless you’ve already had

a conversation with a practitioner and policy maker and you know that the review

is coming from a need of somebody trying to change the system (local policy

maker uMIC)

Working together is not necessarily easy. Such collaboration can be encouraged by training

and support for both parties, developed over successive reviews by a stable team, or

enforced by contracts whereby policy input for systematic reviews is a requirement.

Successful meetings have been more structured [with] an agenda… for eliciting

ideas… about conceptual definitions of terms in the question, [the] scope and

geographical location of funders’ priorities, to make the review manageable. (HIC

review facility)

Policy expertise can also be valued by asking a few questions pertinent to policy interests

alongside academic peer review, and ensuring the responses are considered as the work

progresses.

Supporting review activity with conducive structures

Various structures conducive to policy-research collaboration have evolved for different

types of systematic reviews. At a national or international level, some systematic review

teams work inside policy organisations or for guideline development panels, where there

are multiple opportunities to discuss the focus of reviews and the meaning of the findings.

Elsewhere, exchanging ideas between policy and research to shape the scope and

interpret the findings can be supported by facilities that oversee the production of

systematic reviews or by knowledge brokers who understand both the research and the

policy makers’ environment. It is useful if they have worked in a government department

7

or agency; bring credibility, respect and a level of humility that made people feel

comfortable; and are at ease in an intermediate space and able to liaise efficiently with

senior, knowledgeable people. Typically the review facilities support reviews destined for

multiple potential users, while knowledge brokers support rapid reviews for imminent,

local decisions.

Standardising procedures for engagement

The policy relevance of reviews assessing effectiveness by pooling the findings of similar

studies is enhanced when policy makers are involved in setting the question and the scope

of the review, identifying studies of interest to them, and discussing the emerging findings

(Figure 1). Reviews that advance understanding by configuring the findings of multiple

different, but related, studies may benefit from policy input throughout the process as the

findings gain coherence through the course of the review. Policy input can be face-to-face

with local advisory groups, or with the help of communication technology when working on

a national or international scale.

Review facilities that support review production encourage review teams to convene

advisory groups with members drawn from policy and practice and also invite other policy

input alongside academic peer review of the review’s protocol and draft report.

Working with Advisory Groups

To make the most of Advisory Groups, systematic review teams can draw on research about committees (or other small decision-making groups) conducted in social psychology and business administration as well as health systems.5

Having members representing the full range of stakeholders can bring a fuller range of relevant knowledge to discussions, although increasing the size of a committee above 12 members has diminishing returns.

Larger groups offer credibility to their networks and wider acceptance of their decisions. However, they need to be well facilitated, especially as numbers increase. Groups with fewer than six members reach less reliable judgements, but more than 12 members brings diminishing returns.

Time is required to allow knowledge brought to the meeting to be shared and evaluated before decisions are made. Particular effort should be made to reveal knowledge initially held by individual, rather than all, members especially if their status is not high. Clear presentation of technical issues may help overcome information asymmetry within the group. More time and effort may be required to explore issues requiring judgements where committee members vary in status.

Formal processes are better than informal processes, for instance the Delphi method (used with geographically dispersed groups) or the Nominal Group Technique (for face to face meetings). The Delphi method may also prevent undue influence by individuals, while the Nominal Group Technique may provide better opportunities for discussion.

8

Consequences of collaborative working

The aim of this collaborative working is to produce accessible, timely reports with

appropriate review methods focused on policy concerns. As policy makers and researchers

accrue experience of working together they become more skilled in combining their

knowledge to shape reviews for decision making.

Four models for producing policy relevant systematic reviews

Listening to policy makers, systematic reviewers and people working with them revealed

four models for achieving policy-relevant evidence products and subsequent decisions

(Figure 2). These models were distinguished largely in terms of their starting point and

their purpose. Reviews could start with or without wide agreement about their key

concepts (see the two rows in Figure 2), and with the purpose of offering ‘public goods’

for multiple audiences (as, for instance, Cochrane or Campbell reviews) or informing

policy decisions with a specific jurisdiction and timescale (see the two columns in Figure

2).

Figure 2: Models for achieving policy-relevant systematic reviews

9

Model 1: Systematic reviews offer evidence as public goods where problems are widespread and key concepts (such as intervention taxonomies and core outcomes) are widely agreed. Methods are required to generate generalizable evidence that convinces multiple users who can find reviews in the public domain with clear messages using plain language.

Systematic reviewing as a public good

These products offer up-to-date, widely available, generic evidence such as is found in the Cochrane or Campbell libraries. However, most of these reviews address effectiveness questions and emphasise the methodological rigour of controlled trials. Where there are few trials, as in many areas of health systems research, this approach often highlights gaps in the evidence rather than what can be learnt from existing evidence.

Example:6 Giving health professionals in low income countries additional training in emergency care probably improves their ability to care for seriously ill newborns. We need additional high-quality studies, including studies in which health professionals are trained to care for seriously ill older children.

Box 1: Institutional mechanisms for supporting systematic reviews as public goods

For conducting systematic reviews

For choosing which reviews to conduct:

priority setting methods (e.g. James Lind Alliance Guidebook7) and methodology8

For categorising interventions:

intervention taxonomies (e.g. for effective practice/ organisation of care9; communication10)

For choosing outcomes:

core outcome sets (e.g. The COMET Initiative11)

For systematic review guidance:

conducting effectiveness reviews of health systems (e.g. Cochrane Effective Practice and Organisation of Care Group: resources for authors12; Handbook: Methodology Centre for Systematic Reviews of Health Policy and System Research in LMICs, Chile13);

conducting reviews of a broader range of quasi-experimental studies (3ie: a tool kit14);

conducting reviews addressing a broader range of questions (e.g. The RAMESES (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) project15); the Collaboration for Environmental Evidence16; the EPPI-Centre2;

reporting systematic reviews (PRISMA)17.

For using systematic reviews

For accessing systematic reviews:

searchable databases of systematic reviews (e.g. The Cochrane Library18; The Campbell Library19; 3ie20; Environmental Evidence Library21)

open access publishing

countrywide licences and subsidised access to reviews

For appraising the quality of systematic reviews

AMSTAR – for Assessing the Methodological Quality of Systematic Reviews22

10

Model 2: Systematic reviews offer evidence for urgent decisions that are driven by

political priorities and can be accelerated by slim-line searching and appraisal methods

and automating some processes. For widespread problems where key concepts are widely

agreed, updating or re-analysing existing ‘public goods’ systematic reviews is an efficient

approach. The products are topical, contextualised, rapid reviews, or ‘locally’ applicable

evidence such as is produced for fast working national guideline panels.

Responding to urgent requests for evidence23 Rapid review practices:24 Summaries or syntheses of evidence may be produced within a matter of days or over weeks or months to meet urgent needs for evidence. Quick methods have been developed for narrower questions and effectiveness questions; they are considered less appropriate for addressing complex interventions, economic implications, ethics or safety. Search strategies tend to be less exhaustive, quality assessment less robust and findings presented as descriptive or tabular summaries. How speed influences the findings is unclear, but there is concern about losing detail and assurance. Rapid review programmes:25 Demand for evidence to inform pressing decisions has led to programmes of reviews that are designed, primarily, to inform funding of health care technologies, services and policy, and programme development. How reviews are produced and disseminated varies widely, and depends upon the time and resources available, the complexity and sensitivity of the research topics, and permission from the decision maker. Example:26 A six week scoping review sought evidence for medical malpractice models, frameworks or policies to control litigation damages in obstetrics. The authors adapted their methods to accommodate the timeline: they did not seek unpublished studies or studies in languages other than English. They did not scan fully the references lists of included studies for further evidence, nor did they contact authors for further information. Finally they did not appraise the methodological quality of the included studies. Nevertheless they were able to inform decision-makers about various initiatives for improving medical malpractice litigation systems, including no fault approaches, safety programmes and practice guidelines, specialized courts and alternative claim resolution, communication and resolution, caps on compensation and attorney fees, alternative payment system and liabilities, limitations on litigation and multi-component models.

Box 2: Institutional mechanisms for supporting systematic reviews for urgent decisions

Review teams housed by operational arms of government agencies, or with ‘on call’ contracts

Rapid review guidance23

Automation technologies, such as text mining27

11

Model 3: Systematic reviews that offer evidence as public goods without prior consensus on key concepts require considerable preparatory work in scoping the nature of the issues in addition to the time required to generate convincing, generalizable evidence for multiple users. A typical product is international guidelines for complex issues.

Developing consensus internationally Example:28 Evidence-informed guidance on optimizing health worker roles to improve access to key maternal and newborn health interventions through task shifting began with extensive stakeholder consultation to identify the critical questions and outcomes. An online consultation engaged policy makers, programme managers, researchers and health-care providers, from 33 countries. Questions and issues raised by the 187 participants were summarized and considered by a guidance scoping panel which included midwives, obstetricians, an associate clinician (non-physician clinician), neonatologists, researchers, experts in research synthesis, and experts in health-care programmes. Only after questions had been clarified and prioritised did a search for relevant evidence begin.

Box 3: Institutional mechanisms for supporting systematic reviews for poorly

conceptualised issues

Open Access publishing

Searchable databases of systematic reviews

Time for facilitated discussion and wide consultation amongst stakeholders

12

Model 4: Systematic reviews that offer evidence for local urgent decisions need rapid

methods as for any urgent decisions but can bypass the time-consuming preparatory

collaborative work for developing consensus on the key concepts by employing a

knowledge broker to mediate between local policy teams and systematic reviewers.

Bespoke systematic reviewing Example:29 The report of a rapid review describing paediatric models of care acknowledged the organisation that commissioned the work; the research team who prepared the review; and the organisation that brokered their working partnership. A disclaimer notes no guarantee of comprehensiveness and emphasises the focus on specific questions from the commissioning agency.

Box 4: Institutional mechanisms for supporting systematic reviews for urgent, local

decisions

Review teams housed by operational arms of government agencies, or with ‘on call’ contracts

Rapid review guidance

Automation, such as text mining

Tools for rapid communication between policy and research teams

Knowledge broker

13

Recommendations for fostering policy relevant systematic reviews

Achieving policy-relevant reviews requires investment in working relationships between

the policy and research worlds. This can be through supporting relationships with a

knowledge broker, benefitting from participating individuals familiar with both worlds, or

giving time to developing relationships and interactive skills gradually over the course of a

review or series of reviews.

Important policy dilemmas tend to spawn broad questions which need reconceptualising

before being readily tackled by research. Mapping out causal pathways or logic models

helps to develop narrower, answerable research questions.

Developing important, answerable review questions is easier when review production is

embedded in facilities that span the worlds of policy and research, such as evidence-

informed guidance producers, knowledge broking organisations, and systematic review

facilities prioritising stakeholder input. Policy makers as authors or co-authors can bring

familiarity with the issues to the conduct of the review.

Existing systematic review guidance may need adapting to accommodate the evidence

available to address complex health systems questions. Where research literatures are

underdeveloped, the value of a review may be in analysing existing models of intervention

or in the thinking required to develop models for intervention.

The capacity for conducting policy-relevant systematic reviews in the relatively young

discipline of health systems research is enhanced by:

Including someone with experience of policy, either as an author of the review, or

to work closely with the authors

Training and support for mutual engagement, through structured or facilitated

discussion, and exposure to the other, less familiar world

Ensuring the financial viability of research centres to retain experienced staff

Successful approaches to conducting health systems systematic reviews vary depending on

how widespread and how urgent the question is.

For widespread issues:

check whether a systematic review already exists

emphasise generalisability of findings

make the protocol and review publicly available

keep the review up-to-date

where concepts are not sufficiently clear and widely agreed in advance, allow time

for a well-facilitated, far-reaching stakeholder dialogue to shape the review and

interpret the findings.

For local issues:

check whether an existing systematic review can be re-analysed

retain systematic review teams to provide an evidence service

For urgent issues, accelerate reviews:

with the help of knowledge brokers for clarifying key concepts and questions in

advance

14

by drawing on existing systematic reviews

by more focused, less rigorous methods

with the help of advances in information technology

subsequently, consider transforming the initial product into a more rigorous and

widely-relevant systematic review.

15

Recommendations for enhancing capacity of policy makers to use

evidence

Policy makers’ use of evidence is constrained largely by lack of timely access to clear and

relevant research findings and enhanced by collaboration with researchers; other barriers

are lack of research literacy and costs associated with either research or policy options.1

A systematic review of interventions to improve the use of systematic reviews in decision-

making by health system managers, policy makers and clinicians found evidence for the

relatively simple intervention of mass mailing a printed bulletin which summarises

systematic review evidence. The review found that this may improve evidence-based

practice when there is a single clear message, if the change is relatively simple to

accomplish, and there is a growing awareness by users of the evidence that a change in

practice is required. However, there was insufficient evidence to support multifaceted

interventions that develop awareness and knowledge of systematic review evidence, and

the skills for implementing this evidence.30

The production of policy relevant systematic reviews may provide such a multifaceted

intervention. The access barriers are overcome by the ‘public goods’ models for

systematic reviews (models 1 and 3). The timeliness barrier is overcome by use of existing

reviews (model 1), by knowledge brokers accelerating the shaping of new reviews (model

4), or by numerous shortcuts during review production (model 2). Collaboration with

researchers can take the form of engaging policy makers in shaping reviews which in turn

leads to clearer and more relevant findings. That collaboration, essential for developing

clarity and consensus over the important issues, is easier if one or more of the people

involved have experience of working in both the policy and research worlds, or if

collaboration is supported by a knowledge broker or skilled group discussion facilitator.

Thus, engaging policy makers in the production of systematic reviews is a mechanism to

maximise both their relevance to policy and consideration in policy development.

1 Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J (2014) A systematic review of

barriers to and facilitators of the use of evidence by policymakers, BMC Health Services

Research 14, 2

2 Gough D, Oliver S, Thomas J (2012) An Introduction to Systematic Reviews. Sage,

London.

3 Oliver S, Dickson K (2015) Policy-relevant systematic reviews to strengthen health

systems: models and mechanisms to support their production. Evidence and Policy Online

ISSN 1744 2656 • http://dx.doi.org/10.1332/174426415X14399963605641

4 Gough D, Thomas J, Oliver S (2012) Clarifying differences between review designs and

methods. Systematic Reviews 1:28 doi:10.1186/2046-4053-1-28

5 Oliver S, Hollingworth K, Briner R (2014) Effectiveness and efficiency of committee work:

a rapid systematic review for NICE by its Research Support Unit. Report of the Research

Support Unit for the National Institute for Health and Care Excellence. London: EPPI-

Centre, Social Science Research Unit, UCL Institute of Education.

16

6 Opiyo N, English M (2015) In-service training for health professionals to improve care of

seriously ill newborns and children in low-income countries. Cochrane Database of

Systematic Reviews 2015, Issue 5. Art. No.: CD007071. DOI:

10.1002/14651858.CD007071.pub3

7 http://www.jla.nihr.ac.uk/

8 http://priority.cochrane.org/

9 https://epoc.cochrane.org/epoc-taxonomy

10 Lowe D, Ryan R, Santesso N, Hill S (2011) Development of a taxonomy of interventions

to organise the evidence on consumers' medicines use. Patient Education and Counselling

85(2): e101-7. doi: 10.1016/j.pec.2010.09.024.

11http://www.comet-initiative.org/

12 http://epoc.cochrane.org/epoc-author-resources

13 http://www.who.int/alliance-

hpsr/projects/alliancehpsr_handbooksystematicreviewschile.pdf?ua=1

14 Waddington H, White H, Snilstveit B, Hombrados JG, Vojtkova M, Davies P, Bhavsar A,

Eyers J, Perez Koehlmoos T, Petticrew M, Valentine JC, Tugwell P (2012) How to do a good

systematic review of effects in international development: a tool kit. Journal of

Development Effectiveness 4 (3) 359-387

15 http://www.ramesesproject.org/index.php?pr=Home_Page

16 The Collaboration for Environmental Evidence (2013) Guidelines for Systematic Review

and Evidence Synthesis in Environmental Management. Version 4.2. Environmental

Evidence. www.environmentalevidence.org/.../Guidelines/Guidelines4.2.pdf

17 http://www.prisma-statement.org/

18 http://www.cochranelibrary.com/

19 http://www.campbellcollaboration.org/lib/

20 http://www.3ieimpact.org/evidence/systematic-reviews/

21 http://www.environmentalevidence.org/

22 http://amstar.ca/index.php

23 Advances in Rapid Reviews series of articles

http://www.systematicreviewsjournal.com/series/ARR

24 Featherstone RM, Dryden DM, Foisy M, Guise JM, Mitchell MD, Paynter RA, Robinson KA,

Umscheid CA, Hartling L (2015) Advancing knowledge of rapid reviews: an analysis of

results, conclusions and recommendations from published review articles examining rapid

reviews. Systematic Reviews 4:50

25 Polisena J, Garritty C, Kamel C, Stevens A, Abou-Setta AM (2015) Rapid review programs

to support health care and policy decision making: a descriptive analysis of processes and

methods. Systematic Reviews 4:26

17

26 Cardoso R, Zarin W, Nincic V, Adams JA, Selvaratnam I, Wilson C, Wilson K, McDonald H,

Kenny M, Warren R, Straus S, Tricco A (2015) Rapid Scoping Review of Medical Malpractice

Policies in Obstetrics. Li Ka Shing Knowledge Institute, St. Michael’s Hospital

27 O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S (2015) Using text mining for

study identification in systematic reviews: a systematic review of current approaches.

Systematic Reviews 4:5 doi:10.1186/2046-4053-4-5

28 World Health Organisation (2012) WHO recommendations: optimizing health worker

roles to improve access to key maternal and newborn health interventions through task

shifting.

29 Zurynski Y, Phu A, Deverell M, Elliott E (2015) Paediatric services capacity: an Evidence

Check rapid review brokered by the Sax Institute (www.saxinstitute.org.au) for the NSW

Kids and Families.

30 Murthy L, Shepperd S, Clarke MJ, Garner SE, Lavis JN, Perrier L, Roberts NW, Straus SE

(2012) Interventions to improve the use of systematic reviews in decision-making by health

system managers, policy makers and clinicians. Cochrane Database of Systematic Reviews

2012, Issue 9. Art. No.: CD009401. DOI: 10.1002/14651858.CD009401.pub2.

The Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) is part of the Social Science Research Unit (SSRU), UCL Institute of Education, University College London.

The EPPI-Centre was established in 1993 to address the need for a systematic approach to the organisation and review of evidence-based work on social interventions. The work and publications of the Centre engage health and education policy makers, practitioners and service users in discussions about how researchers can make their work more relevant and how to use research findings.

Founded in 1990, the Social Science Research Unit (SSRU) is based at the UCL Institute of Education, University College London. Our mission is to engage in and otherwise promote rigorous, ethical and participative social research as well as to support evidence-informed public policy and practice across a range of domains including education, health and welfare, guided by a concern for human rights, social justice and the development of human potential.

The views expressed in this work are those of the authors and do not necessarily reflect the views of the EPPI-Centre or the funder. All errors and omissions remain those of the authors.

This document is available in a range of accessible formats including large print.

Please contact the UCL Institute of Education for assistance: telephone: +44 (0)20 7947 9556 email: info@ioe.ac.uk

First produced in 2015 by:

Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) Social Science Research UnitUCL Institute of Education, University College London18 Woburn SquareLondon WC1H 0NR

Tel: +44 (0)20 7612 6397

http://eppi.ioe.ac.uk/ http://www.ioe.ac.uk/ssru/

ISBN: 978-1-907345-83-8