David Price1,2, Eric D. Bateman2,3, Alison Chisholm2, Nikolaos G. Papadopoulos2,4, Sinthia Bosnic-Anticevich2,5, Emilio Pizzichini2,6, Elizabeth V. Hillyer2, and A. Sonia Buist2,7

1. Academic Primary Care, Division of Applied Health Sciences, University of Aberdeen, Aberdeen, UK

2. Respiratory Effectiveness Group, Cambridge, United Kingdom; 3. Division of Pulmonology, Department of Medicine, University of Cape Town and University

of Cape Town Lung Institute, Cape Town, South Africa; 4. Department of Allergy, 2nd Pediatric Clinic, University of Athens, Athens, Greece; 5. Faculty of Pharmacy, University of Sydney, Camperdown, New South Wales, Australia; 6. Federal University of Santa Catarina, Department of Medicine, Floriano´ polis, Brazil; 7. Oregon Health and Science University, Portland, Oregon

Complementing the Randomized Controlled Trial Evidence Base Evolution Not Revolution

Key messages

• Observational studies and pragmatic trials can complement classical randomized controlled trials (RCTs) o Provide data more relevant to the circumstances under

which medicine is routinely practiced• Evidence should be integrated from a variety of different

study designs and methodologies. • Real-life studies—observational studies and pragmatic trials

— have utility in:o Addressing clinical questions that are unanswered by

RCTs; o Testing new hypotheses and possible license extensions; o Helping to differentiate between available therapies for

a given indication. o Fit within a conceptual framework of evidence relevant

to clinical practice

Efficacy vs Effectiveness

• Efficacy trials: optimize all conditions by using highly-selected patient populations and close clinical monitoring to optimize internal validity, assessing cause and effect between an intervention and an outcome.

• Effectiveness studies: evaluate how interventions work in the diversity of patients treated in routine care when managed in clinical scenarios that differ widely within and between countries, and they seldom (if ever) reflect the highly interventional nature of efficacy trials: “letting the rats out of the cage and seeing what happens in real life.”

The Pros & Cons

Different study designs

RCTs: “the good”

• The RCT is designed to answer precise questions about the efficacy of various types of medical interventions and to gather useful information about treatment-related adverse events.

• RCTs minimize all potential confounders and optimize internal validity by selecting an idealized, “pure” patient population, and by using close patient monitoring, consistently across all trial subjects.

• They provide a confident answer to the question: “Does intervention X work in an ideal (and

specific subgroup of) patients receiving best standards of care?”

RCTs: “the bad”

• RCTs lack external validity.

• RCTs excluding any patients with characteristics that could affect the efficacy signal of an intervention, and by managing and monitoring patients far more intensively than would be feasible in clinical practice.

• RCT findings lack generalizability, particularly in:o Long-term chronic conditions that affect broad and

heterogeneous patient populations, e.g.– Asthma – Chronic obstructive pulmonary disease (COPD)

Pragmatic trials: “the good”

• Compare interventions under more usual clinical conditions than RCTs to improve the applicability of findings to real-life issues and everyday clinical decision making.

• Designed to assess outcomes of healthcare interventions in the context of real-life clinical practice.:o Include heterogeneous populationso Incorporate relevant levels of clinical care to

help answer practical clinical questions for healthcare providers, patients, and policymakers.

Pragmatic trials: “the bad”

• Pragmatic trials can face challenges in:o Maintaining adequate patient follow-upo Detecting a small treatment effect in usual

care settings – requires:– Large study population, and/or– Validated instrument sensitive to the

treatment effect• Although less interventional than RCTs, any level

of monitoring or patient engagement can alter behavior and potentially eliminate differences between two trial interventions

Observational studies: “the good”• Offer high external validity • Routine collection of data means that they are

much greater in extent than predefined RCT datasets, and can often be obtained more quickly and at a lower cost

• Provide valuable data on:o How management approaches are used, o Results associated with interventions in the

real world.• Can detect strong associations between test

interventions and predefined outcomes that are generalizable to a broad patient population

Observational studies: “the bad”• Lack internal validity – limited in the extent to

which they can demonstrate an unequivocal cause-and-effect relationship.

• No randomization or blinding, result in concerns around confounding by:o Indicationo Severity

• Missing data can limit the interpretation of findings.

Observational studies: improvements• Validity of observational studies can be

strengthened by:o Identifying (and preregistering) the:

– Eligible population,– Design– Outcomes– Potential confounders

o Application of rigorous analytic methods to– Reduce bias– Minimise confounding– Adjust for residual confounding factors.

Complimenting RCTs with observational studies

Plugging evidence gaps

RCT: important limitations

• The use of a non-inferiority design• Few comparative efficacy trials• Consistent application of historical (licensing)

precedent, biases can be echoed generations of trials• Do not address use of therapies:

o Diverse populationso Routine care settings

• Expense limits their size and duration• Lack power for subgroup analyses and rare treatment-

related events• Limited data on long-term outcomes and safety issues• Unethical to take patients off “optimized therapy” to

test alternatives

RCT limitations: how to plug the gaps• Do not try to design RCTs to answer every

question about an intervention—they intrinsically cannot.

• Understand the essence of the question being asked and to select the appropriate study design(s) to answer the question at hand.

• Draw on a diversity of different study designs and analytical approaches, a fuller picture of the utility of an intervention can be established.

Plugging the gaps: example

• Patient Behavior and Preferenceo Observational studies and pragmatic trials reflect

(or can be designed to reflect) the level of physician/clinician interaction typical of routine care. They:– Capture patient activity (e.g., consultation

patterns, medication adherence) and patterns of care

– Highlight differences between routine practice appears and guideline recommendations

– Provide important insights into patients’ experiences of their disease and its management and of their preferences, e.g.

Once vs twice-daily therapy Oral vs inhaled therapy

Filling the gaps: more examples• Ethics:

o Observational studies offer a way to address interesting and important clinical questions that are ethically unevaluable or challenging in the RCT setting, e.g.

– It is unethical to take patients off RCT-defined gold-standard therapy to evaluate real-life comparative effectiveness of different treatment options

• Subgroups of Patientso Patients with characteristics that may inhibit treatment

response and who are excluded from RCTs can be included in pragmatic trials and observational studies, e.g.:

– Patients with rhinitis– Smokers

• Where RCT blinding is infeasible, e.g. o Comparative effectiveness of inhaler device studies

Creating a fuller picture of the evidence

Integrating different study designs

Integrating evidence: why?

• One of the major drivers of real-life research is the undue weight and supremacy of evidence that has long been attributed to RCTs.

• There is a need for evolution – there is a need: o To integrate evidence from all sources to arrive at

treatment recommendations. o To recognize the value of evidence from a diversity

of complementary approaches that:– make good each other’s methodology

deficiencies – accommodate the diverse needs and

circumstances under which medicine is practiced.

– For guidance when n = 1.

Integrating evidence: why?

• Dichotomies and hierarchies of the evidence base are not helpful.

• Use of the term “real-life” to refer to observational studies and pragmatic trials implies that RCTs are not real-life.

• All in vivo studies deal with real people – different study designs provide different

pieces of the evidence jigsaw

REG’s integrated evidence framework• The Respiratory Effectiveness Group (REG) has

proposed a new framework to enable classification of clinical research studies in terms of their general design.

• The framework is intended to complement the previously proposed PRECIS wheel (see later slide)

• The REG framework relies on two axes: o One describing the type of studied population in

relation to the broadest target population o The other describing the “ecology of care” (or

management approach) in relation to usual standard of care in the community.

1. Roche N, et al. Lancet Respir Med 2013;1:e29–e30.

REG’s integrated evidence framework• The position of a study within the framework

serves as a description of a study, not as a representation of the quality of evidence it provides.

• The framework is tool for describing the basic characteristics of the study design and population.

• Multiple studies can be placed relative to each other with respect to their relevance to the general target population, and for each study the appropriate quality assessment tools can be identified.

1. Roche N, et al. Lancet Respir Med 2013;1:e29–e30.

REG’s integrated evidence framework

A means of positioning individual studies with respect to their relevance to the general target population.

1. Roche N, et al. Lancet Respir Med 2013;1:e29–e30.

Conclusions: I

• Classical RCTs:o Form the backbone of drug licensing and

registration. o Answer critical questions, but NOT ALL questions

• Pragmatic trials and observational studies lack the internal validity of a registration RCT, but:o Shine light on important aspects of patient care

that are not addressed by RCTso Benefit from being less costly, allowing them to

address longer-term aspects of care and to test hypotheses

RCTs traditionally occupy the preregistration space and pragmatic trials and observational studies that of post-registration, BUT observational studies and pragmatic trials can (more affordably) test a variety of hypotheses to inform the direction of future RCT

expenditure

Conclusions: Redefining the hierarchy• Different study designs should:

o no longer be ranked in pyramids or pitted against each other at opposing ends of the quality spectrum.

o be called on—as appropriate—to answer clinical questions.

• Devising frameworks that unite different streams of research, and establishing standards to appraise the quality of research, are important steps toward achieving a more integrated approach to evidence reviews.