the continuous update project: novel approach to reviewing mechanistic evidence on diet, nutrition,...

The Continuous Update Project: Novel approach to reviewing mechanistic evidence on diet, nutrition, physical activity and cancer 21 October 2015

Martin WisemanWorld Cancer Research Fund International & University of Southampton

The Panel emphasises the importance of not smoking and of avoiding exposure to tobacco smoke

Journal citations WCRF/AICR Reports

1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 20110

50

100

150

200

250

300

350

400

450

500

Doll & Peto 19811997 EXPERT REPORT 2007 EXPERT REPORT

Num

ber o

f cita

tions

KEY FEATURES OF PROCESS

• New method• Systematic reviews• Review of evidence separate from

judgement • Panel of international experts• Predetermined criteria for judgements• Flexibility

Systematic reviewsAnalysis

• Strength• Consistency• Specificity• Timing• Dose Response• Plausibility• Coherence• Experiment• Analogy

Inferring causality

Bradford Hill

• Strength• Consistency• Specificity• Timing• Dose Response• Plausibility and coherence• Coherence• Experiment• Analogy

Inferring causality

Bradford Hill

Systematic reviews

• Expert international Task Force for method• Nine centres - USA, UK, NL, Italy• SLR centre coordinator• Test of reproducibility• Standardised search, analysis and display • Epidemiology and mechanisms• Quality assessment• Peer review - protocol, report• Defined expertise required

– Nutrition, epidemiology, systematic review, cancer biology, statistics

GRADING CRITERIAPredefined requirements for:

–Number and types of studies–Quality of exposure and outcome assessment–Heterogeneity within and between study

types–Exclusion of chance, bias or confounding–Biological gradient–Evidence of mechanisms–Size of effect

• To help identify causal links• Information from mechanistic studies-narrative reviews • Evidence on mechanisms-predefined requirement for

grading criteria• Considerations:

reviews not systematiccould select a mechanism to explain epidemiological associations

2007 Second Expert Report and Continuous Update Project

29 MARCH 2012 | VOL 483 | NATURE | 531

• Reproducibility• Relevance of model• Relevance of exposure• Relevance of dose• Route of administration• Publication bias

• < 10% highly promising basic science discoveries enter clinical use

• Evidence distorted – Weak internal validity (randomisation, allocation concealment,

blinding, drop outs, co-morbidities) – Publication bias (98% of animal studies ‘significant’)

Having confidence in the evidence

van Luijk J et al (2014) Systematic Reviews of Animal Studies; Missing Link in Translational Research?. PLoS ONE 9(3): e89981.

Internal validity of animal intervention studies

available in principle(e.g. thesis, obscure journal)

easily available(Medline-indexed)

activelydisseminated(e.g. reprint fromdrug company)

unavailable(unpublished)

Publication bias

Meta-analysis of the association between TP53 status and the risk of death at 2 years

Kyzas P A et al. JNCI J Natl Cancer Inst 2005;97:1043-1055

• To develop a method for conducting systematic reviews of mechanistic evidence linking food, nutrition and physical activity exposures to cancer– Multidisciplinary team (informatics, statistics,

epidemiology, systematic reviews, cancer biology, pathology, nutrition, cancer site of interest)

– Search terms/inclusion-exclusion criteria– How to manage vast number of papers– What information to be extracted– How to analyse/display results– Identify criteria for grading the evidence

WCRF mechanisms Project at Bristol

Importance• Extensive mechanistic data from animals & cell lines linking diet & cancer• Systematic reviews

•Allow objective appraisal of evidence •Reduce false-positive & false-negative results•Identify sources of bias, improving study quality•Rigorous methods for conducting & reporting systematic reviews of mechanistic studies are lacking

• This project should increase the value of mechanistic data•Enable rigorous reviews •Increased precision of estimated effects •Identify gaps in the research evidence•Reduce selective citation of mechanistic evidence•Inform generalisability to humans (e.g. heterogeneity across species & models)

• A potential tool in the translation of basic sciences into policy & practice

Challenges• Developing a one size fits all template

although we expect that when the template is developed research groups will choose to search for mechanistic targets relevant to their own question• Finding the relevant studies

this relies on having a good search strategy and validating this• Determining study quality

some quality criteria could be adopted from epidemiology• Determining the strength of evidence for different study types

by discussions within the team, tapping into to our multidisciplinary team• Determining the relevance to humans

consensus following examination of the studies and discusses within the team• Publication bias

We will use recognised methods to quantify the extent to which this is likely to have occurred• Collating the evidence

Key issues• Mechanisms discovery

_ Validation (not missing important studies)_ Inter-relationships between mechanisms _ Stand-alone automation

• Risk of bias / internal validity– Animal and in-vitro studies

• Relevance / external validity – What relevance criteria form part of eligibility?– How to incorporate into evidence synthesis?

• Evidence synthesis– Publication bias, consistency (heterogeneity), precision in absence

of meta-analyses• Categorising overall conclusions

- magnitudes of effect and causality

Summary of the process

Research question

• Identify

• Appraise individual studies

• Integrate body of evidence

• Risk of bias• Relevance

• Mechanism discovery (unbiased)

• Specific mechanisms (targeted)

• Within an evidence stream (human, animal, in vitro)

• Across evidence streams

• Confidence conclusion- High- Moderate- Low

Eligibility criteriaRelevance

Step 9: Synthesis of supporting evidence from in vitro and xenograft models underpinning biological plausibility

Step 8: Integrate human and animal studies to develop an evidence based conclusion

Step 7: Assess strength of overall body of evidence for human and animal studies separately (based on study design, risk of bias, relevance, imprecision, inconsistency, publication bias,

magnitude of effect, dose-response & confounding)

Step 6: Synthesis of data from individual studies

Step 5: Assess the quality of individual studies (risk of bias)

Step 4: Extract data

Step 3: Apply inclusion/exclusion criteria, including an assessment of relevance

Step 2: Search for studies

Step 1: Specify research objectives

Step 1

• Identifying potential mechanisms by which an exposure causes an outcome

summarise the literature on all potential mechanisms linking a modifiable exposure to cancer outcomes (may be bypassed if the objective is to

review the evidence underlying a pre-specified mechanism )

Step 1, Stage 1

• Defining the research question– specify the modifiable exposure and the cancer outcomes of interest. – develop a comprehensive list of possible mechanisms or intermediate

phenotypes

A diverse range of studies provide evidence on mechanisms of explaining an exposure – cancer outcome relationship, including cell-line, animal and human

Step 1, Stage 2

Searching for studies

-Develop of list of search terms-Carry out searches in Medline, Embase and Biosis

Automated mechanisms discovery

Tom Gaunt, University of Bristol

Exposure terms

Linking mechanisms

Outcome terms

Tom Gaunt, University of Bristol

Relevance (external validity)

• Mechanistic studies inform plausibility: Bradford-Hill

• Animal studies vs human studies– Population: homogeneity (genetic, clinical, environment)– Exposure: biological window– Outcome: induced or transplanted cancers

• Need clarity over role of relevance in: – Setting eligibility criteria (only assess bias if relevant?)– Integrating the body of evidence (hierarchy?)

Heterogeneity in animal studies exploring effect of statins

Evidence integration

• Within an evidence stream (human, animal, in vitro)– Assess confidence about data quality– Summarise magnitude of effects where possible

• Across evidence streams - causal conclusions• Integrate the highest level of evidence from

each of the evidence streams (human RCT, observational, animal, in-vitro)

Integrate evidence to develop causal conclusions

• Integrate the highest level of evidence from each evidence stream (human, animal, in-vitro)

• Consider evidence from human data with animal evidence, then in-vitro data– E.g. If human level of evidence high – conclusion based on

human data only– If human level of evidence moderate / low – conclusion based

on animal evidence– Upgrade if in vitro data provide strong evidence of biological

plausibility (downgrade if weak?)• Develop a categorisation schema (high,

moderate, low evidence of causality)

Level of evidence

in human studies

High Convincing

ModerateSuggestive Probable

Low

No conclusion Suggestive Probable

Low Moderate High

Level of evidence in animal studies

Integrating evidence

Discussion points

• Eligibility criteria (retrieval stage) vs stratification– design / conduct / reverse causation / relevance considerations?

• Risk of bias for in vitro studies• Discussion over role of relevance in:

- Setting eligibility criteria (only assess RoB if relevant)- Integrating the body of evidence

• Signalling questions for relevance • Evidence integration • Publication bias • Causal conclusions: categorisation and relative weights

– animal studies upgrade confidence if human evidence moderate or low?– in-vitro studies: biological plausibility to support recommendations

• Testing of the tool by others – utility, generalisability and replicability

• Increasing the sensitivity and specificity of mechanism discovery algorithm

• Incorporation of methodological developments– e.g. risk of bias in animal & in-vitro studiesunderstanding of relevance

• Review of other templates: – Office of Health Assessment & Translation, US National

Toxicology Program

Future work

Research question (PECO)

• Identify

• Appraise individual studies

• Integrate body of evidence

• Risk of bias• Relevance

• Mechanism discovery (unbiased)

• Specific mechanisms (targeted)

• Within an evidence stream (human, animal, in vitro)

• Across evidence streams

• Confidence conclusion- High- Moderate- Low

Eligibility criteria relevance

Integrating causal

conclusions about

mechanisms into pathways

Summary of the process

The TeamUniversity of Bristol: PI- Dr Sarah Lewis –Genetic epidemiology/systematic reviews of genetic studies Co-PI- Prof Richard Martin –Epidemiology/systematic reviews Dr Mona Jeffreys- Cancer Epidemiology/systematic reviews Dr Mike Gardner – Animal biology/systematic reviews  Prof Jeff Holly- Molecular biology – IGF and cancer Dr Tom Gaunt – Genetic epidemiology/bioinformatics Prof Jonathan Sterne- Meta-analysis and systematic review methodology Professor Julian Higgins – Meta-analysis and systematic review methodology Prof George Davey Smith – Epidemiology Prof Christos Paraskeva –Molecular biology Prof Steve Thomas –Epidemiology of head and neck cancer Dr Pauline Emmett - Nutritional epidemiology Dr Kate Northstone – Nutritional Epidemiology Cath Borwick – Librarian/ Search strategies

University of Cambridge: WCRF InternationalDr Suzanne Turner- Animal models Prof Martin Wiseman

Dr Pierre Hainut (advisor to WCRF Int)

Dr Panagiota Mitrou

Dr Rachel Thompson

IARC:Dr Sabina Rinaldi- Hormones and cancer