appraisal, extraction and pooling of quantitative data for reviews of effects
DESCRIPTION
Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects . - from experimental, observational and descriptive studies. JBI/CSRTP/ 2013-14/0002 . Introduction. Recap of Introductory module Developing a question (PICO) Inclusion Criteria Search Strategy - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/1.jpg)
Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects
- from experimental, observational and descriptive studies
JBI/CSRTP/2013-14/0002
![Page 2: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/2.jpg)
Introduction• Recap of Introductory
module – Developing a question (PICO)– Inclusion Criteria– Search Strategy– Selecting Studies for Retrieval
• This Module considers how to appraise, extract and synthesize evidence from experimental, observational and descriptive studies.
![Page 3: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/3.jpg)
Program OverviewDay 1
Time Session Group Work0900 Introductions and overview of Module 3
0930 Session 1: The Critical Appraisal of Studies
1000 Morning Tea1030 Session 2: Appraising RCTs and experimental
studiesGroup Work 1: Critically appraising RCTs and experimental studies. Report back
1145 Session 3: Appraising observational Studies
1230 Lunch1330 Group Work 2: Critically appraising
observational studies. Report back
1415 Session 4: Study data and data extraction
1515 Afternoon tea1530 Group Work 3: Data extraction. Report back
1600 Session 5: Protocol development Protocol development1700 End
![Page 4: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/4.jpg)
Program OverviewDay 2
Time Session Group Work
0900 Overview of Day 1
0915 Session 6: Data analysis and meta-analysis
1030 Morning Tea
1100 Session 7: Appraisal extraction and synthesis using JBI MAStARI
Group Work 4: MAStARI trial.Report back
1230 Lunch
1330 Session 8: Protocol Development Protocol development
1415 Session 9: Assessment MCQ Assessment
1445 Afternoon tea
1500 Session 10: Protocol Presentations Protocol Presentations
1700 End
![Page 5: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/5.jpg)
Session 1: The Critical Appraisal of Studies
![Page 6: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/6.jpg)
Why Critically Appraise?
• Combining results of poor quality research may lead to biased or misleading estimates of effectiveness
1004 references
832 referencesScanned Ti/Ab
172 duplicates
117 studiesretrieved
715 do not meetIncl. criteria
82 do not meetIncl. criteria
35 studies forCritical Appraisal
![Page 7: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/7.jpg)
The Aims of Critical Appraisal
• To establish validity– to establish the risk of bias
![Page 8: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/8.jpg)
Internal & External Validity
Internal Validity
External Validity
Relationship between IV and EV?
Used locally?
![Page 9: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/9.jpg)
Strength & Magnitude
Strength Magnitude & Precision
How internally valid is the
study?
How large is the effect?
![Page 10: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/10.jpg)
Clinical Significance and Magnitude of Effect
• Pooling of homogeneous studies of effect or harm• Weigh the effect with cost/resource of change• Determine precision of estimate
![Page 11: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/11.jpg)
Assessing Methodological Quality
• Numerous tools are available for assessing methodological quality of clinical trials and observational studies.
• JBI requires the use of a specific tool for assessing risk of bias in each included study.
• ‘High quality’ research methods can still leave a study at important risk of bias (e.g. when blinding is impossible).
• Some markers of quality are unlikely to have direct implications for risk of bias (e.g. ethical approval, sample size calculation).
![Page 12: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/12.jpg)
Sources of Bias
• Selection;• Performance;• Detection; and• Attrition.
![Page 13: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/13.jpg)
Selection Bias
• Systematic differences between participant characteristics at the start of a trial.
• Systematic differences occur during allocation to groups.
• Can be avoided by concealment of allocation of participants to groups.
![Page 14: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/14.jpg)
Type of bias Quality assessment
Population
Allocation
Selection Allocation concealment
Treatment Control
![Page 15: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/15.jpg)
Performance Bias
• Systematic differences in the intervention of interest, or the influence of concurrent interventions.
• Systematic differences occur during the intervention phase of a trial.
• Can be avoided by blinding of investigators and/or participants to group.
![Page 16: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/16.jpg)
Type of bias Quality assessment
Population
Allocation
Selection Allocation concealment
Treatment Control
Performance Blinding Exposed to intervention
Not exposed
![Page 17: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/17.jpg)
Detection Bias
• Systematic differences in how the outcome is assessed between groups.
• Systematic differences occur at measurement points during the trial.
• Can be avoided by blinding of outcome assessor.
![Page 18: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/18.jpg)
Type of bias Quality assessment
Population
Allocation
Selection Allocation concealment
Treatment Control
Performance Blinding Exposed to intervention
Not exposed
Detection Blinding Population Population
![Page 19: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/19.jpg)
Attrition Bias
• Systematic differences in withdrawals and exclusions between groups.
• Can be avoided by:– Accurate reporting of losses and reasons for withdrawal– Use of ITT analysis
![Page 20: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/20.jpg)
Type of bias Quality assessment
Population
Allocation
Selection Allocation concealment
Treatment Control
Performance Blinding Exposed to intervention
Not exposed
Detection Blinding Population Population
Attrition ITT follow up Follow up Follow up
![Page 21: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/21.jpg)
Ranking the “Quality” of Evidence of Effectiveness
• To what extent does the study design minimize bias/demonstrate validity.
• Generally linked to actual study design in ranking evidence of effectiveness.
• Thus, a “hierarchy” of evidence is most often used, with levels of quality equated with specific study designs.
![Page 22: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/22.jpg)
JBI Levels of Evidence - Effectiveness
Level of Evidence
EffectivenessE (1-4)
1 SR (with homogeneity) of experimental studies (e.g. RCT with concealed allocation)OR 1 or more large experimental studies with narrow confidence intervals
2 One or more smaller RCTs with wider confidence intervals OR Quasi-experimental studies (e.g. without randomisation)
3 3a. Cohort studies (with control group)3b. Case-controlled3c. Observational studies (without control groups)
4 Expert opinion, or based on physiology, bench research or consensus
![Page 23: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/23.jpg)
The Critical Appraisal Process
• Every review must set out to use an explicit appraisal process. Essentially,– A good understanding of research design is required in
appraisers; and– The use of an agreed checklist is usual.
![Page 24: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/24.jpg)
Session 2: Appraising RCTs and experimental studies
![Page 25: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/25.jpg)
RCTs• RCTs and quasi (pseudo) RCTs provide the most robust form
of evidence for effects– Ideal design for experimental studies
• They focus on establishing certainty through measurable attributes
• They provide evidence related to:– whether or not a causal relationship exists between a stated
intervention, and a specific, measurable outcome, and– the direction and strength of the relationship
• These characteristics are associated with the reliability and generalizability of experimental studies
![Page 26: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/26.jpg)
Randomized Controlled Trials
• Evaluate effectiveness of a treatment/therapy/ intervention.
• Randomization critical.• Properly performed RCTs reduce bias, confounding
factors, and results by chance.
![Page 27: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/27.jpg)
Experimental studies
• Three essential elements– Randomization (where possible).– Researcher-controlled manipulation of the independent
variable.– Researcher control of the experimental situation.
![Page 28: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/28.jpg)
Other experimental studies
• Quasi-experiments without a true method of randomization to treatment groups
• Quasi experiments– Quasi-experimental designs without control groups– Quasi-experimental designs that use control groups but not
pre-tests– Quasi-experimental designs that use control groups and
pre-tests
![Page 29: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/29.jpg)
Sampling
• Selecting participants from population.• Inclusion/exclusion criteria.• Sample should represent the population.
![Page 30: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/30.jpg)
Sampling Methods
• Probabilistic (Random) sampling • Consecutive• Systematic• Convenience
![Page 31: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/31.jpg)
Randomization
![Page 32: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/32.jpg)
Randomization Issues
• Simple methods may result in unequal group sizes– Tossing a coin or rolling a dice– Block randomization
• Confounding factors due to chance imbalances– stratification – prior to randomization– ensures that important baseline characteristics are even
in both groups
![Page 33: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/33.jpg)
Block Randomization• All possible combinations ignoring unequal
allocation
1 AABB 4 BABA2 ABAB 5 BAAB3 ABBA 6 BBAA
• Use table of random numbers and generate allocation from sequence e.g. 533 2871
• Minimize bias by changing block size
![Page 34: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/34.jpg)
Stratified Randomization
![Page 35: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/35.jpg)
Blinding
• Method to eliminate bias from human behaviour• Applies to participants, investigators, assessors etc• Blinding of allocation• Single, double and triple blinded
![Page 36: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/36.jpg)
Schulz, 2002
Blinding
![Page 37: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/37.jpg)
Intention to Treat• ITT analysis is an analysis based on the initial treatment
intent, not on the treatment eventually administered. • Avoids various misleading artifacts that can arise in
intervention research. – E.g. if people who have a more serious problem tend to drop out at a
higher rate, even a completely ineffective treatment may appear to be providing benefits if one merely compares those who finish the treatment with those who were enrolled in it.
• Everyone who begins the treatment is considered to be part of the trial, whether they finish it or not.
![Page 38: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/38.jpg)
Minimizing Risk of Bias
• Randomization;• Allocation;• Blinding; and• Intention to treat (ITT) analysis.
![Page 39: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/39.jpg)
Appraising RCTs/quasi experimental studies JBI-MAStARI Instrument
![Page 40: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/40.jpg)
Assessing Study Quality as a Basis for Inclusion in a Review
Included studies
Excluded studies
poor quality
cut off point
high quality
![Page 41: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/41.jpg)
Group Work 1
• Working in pairs, critically appraise the two papers in your workbook
• Reporting Back
![Page 42: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/42.jpg)
Session 3: Appraising Observational Studies
![Page 43: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/43.jpg)
Rationale and potential of observational studies as evidence
• Account for majority of published research studies • Need to clarify what designs to include• Need appropriate critical appraisal/quality assessment
tools• Concerns about methodological issues inherent to
observational studies– Confounding, biases, differences in design– Precise but spurious results
![Page 44: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/44.jpg)
Appraisal of Observational Studies
• Critical appraisal and assessment of quality is often more difficult than RCTs.
• Using scales/checklists developed for RCTs may not be appropriate.
• Specific tools are available to appraise observational research.
![Page 45: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/45.jpg)
Confounding Factors
• The apparent effect is not the true effect.• May be other factors relevant to outcome in
question.• Can be important threat to validity of results.• Adjustments for confounding factors can be made
– E.g. has multivariate analysis been conducted?• Authors often look for plausible explanation for
results.
![Page 46: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/46.jpg)
Bias
• Selection bias– differ from population with same condition.
• Follow up bias– attrition may be due to differences in outcome.
• Measurement/detection bias– knowledge of outcome may influence assessment of
exposure and vice versa.
![Page 47: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/47.jpg)
Observational Studies - Types
• Cohort studies• Case-control studies• Case series/case report• Cross-sectional studies
![Page 48: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/48.jpg)
Cohort Studies
• Group of people who share common characteristic.• Useful to determine natural history and incidence of
disorder or exposure.• Two types
– prospective (longitudinal)– retrospective (historic)
• Aid in studying causal associations.
![Page 49: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/49.jpg)
Prospective Cohort Studies
Taken from Tay & Tinmouth, 2007
![Page 50: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/50.jpg)
Prospective Cohort Studies
• Longitudinal observation through time• Allows investigation of rare diseases or long latency
• Expensive• Increased likelihood of attrition• Long time to see useful data
![Page 51: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/51.jpg)
Retrospective Cohort Studies
Taken from Tay & Tinmouth, 2007
![Page 52: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/52.jpg)
Retrospective Cohort Studies
• Mainly data collection• No follow up through time• Cheaper, faster
![Page 53: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/53.jpg)
Case-Control Studies
• Cases’ already have disease/condition• Controls’ don’t have disease/condition• Otherwise matched to control confounders• Frequently used• Rapid means of study of risk factors• Sometimes referred to as retrospective study
![Page 54: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/54.jpg)
Case-Control Studies
Biomedical Library, University of Minnesaota, 2002
![Page 55: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/55.jpg)
Case-Control Study
• Inexpensive• Little manpower required• Fast• No indication of absolute risk
![Page 56: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/56.jpg)
Case series/Case reports
• Tracks patients given similar treatment– prospective
• Examines medical records for exposure and outcome– retrospective
• Detailed report of individual patient• May identify new diseases and adverse effects
![Page 57: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/57.jpg)
Case series/Case reports
![Page 58: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/58.jpg)
Cross-sectional Studies• Takes ‘slice’ or ‘snapshot’ of target group• Frequency and characteristics of disease/variables
in a population at a point in time• Often use survey research methods• Also called prevalence studies
![Page 59: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/59.jpg)
Appraising comparable Cohort and Case-control studies JBI-MAStARI Instrument
![Page 60: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/60.jpg)
Appraising descriptive/case series studies JBI-MAStARI Instrument
![Page 61: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/61.jpg)
Group Work 2
• Working in pairs:– critically appraise the cohort study in your workbook– critically appraise the case control study in your
workbook• Reporting Back
![Page 62: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/62.jpg)
Session 4: Study data and Data Extraction
![Page 63: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/63.jpg)
Considerations in Data Extraction
• Source - citation and contact details• Eligibility - confirm eligibility for review• Methods - study design, concerns about bias• Participants - total number, setting, diagnostic criteria • Interventions - total number of intervention groups• Outcomes - outcomes and time points• Results - for each outcome of interest: sample size, etc• Miscellaneous - funding source, etc
![Page 64: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/64.jpg)
Quantitative Data Extraction
• The data extracted for a systematic review are the results from individual studies specifically related to the review question.
• Difficulties related to the extraction of data include:– different populations used– different outcome measures– different scales or measures used– interventions administered differently– reliability of data extraction (i.e: between reviewers)
![Page 65: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/65.jpg)
Minimizing Error in Data Extraction
• Strategies to minimize the risk of error when extracting data from studies include:– utilizing the standard JBI data extraction form.– pilot testing the extraction form prior to commencement of
the review.– training and assessing data extractors
![Page 66: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/66.jpg)
Data most frequently extracted
1004 references
832 referencesScanned Ti/Ab
172 duplicates
117 studiesretrieved
715 do not meetIncl. criteria
82 do not meetIncl. criteria
35 studies forCritical Appraisal
26 studies incl.in review
9 excludedstudies
![Page 67: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/67.jpg)
Outcome Data: Effect of Treatment or Exposure
• Dichotomous– Effect/no effect– Present/absent
• Continuous– Interval or ratio level data– BP, HR, weight, etc
![Page 68: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/68.jpg)
What do you want to know?
• Is treatment X more effective than treatment Y?• Is exposure to X more likely to result in an outcome
or not?• How many people need to receive an intervention
before someone benefits or is harmed?
![Page 69: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/69.jpg)
Risk
• Risk =# times something happens
# opportunities for it to happen• “Risk” of birthing baby boy?
– One boy is born for every 2 opportunities: 1/2 = .5That is: 50% probability (risk) of having a boy
• One of every 100 persons treated with aspirin for a headache, has a side-effect,
1/100 = .01
![Page 70: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/70.jpg)
Relative Risk (RR)
• Ratio of risk in exposed group to risk in not exposed group – The RR of feeling better with treatment with i.m.
magnesium injection for chronic fatigue syndrome = the risk of improvement for chronic fatigue patients treated with i.m. magnesium divided by those patients treated with placebo.
![Page 71: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/71.jpg)
‘Risk’ of improvement on magnesium = 12/ 15 = 0.80‘Risk’ of improvement on placebo = 3/ 17 = 0.18 Relative risk (of improvement on Mg2+ therapy vs placebo) = 0.80/0.18 = 4.5Thus patients on magnesium therapy are 4 times more likely to feel better on magnesium
rather than placebo
RR example• A trial examined whether patients with chronic fatigue syndrome
improved 6 weeks after treatment with i.m. magnesium. The group who received the magnesium were compared to a placebo group and the outcome was feeling better
![Page 72: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/72.jpg)
Interpreting Relative Risk
• What does a relative risk of 1 mean? – That there is no difference in risk in the two groups. – In the magnesium example it would mean that patients are
as likely to “feel better” on magnesium as on placebo– If there was no difference between the groups the
confidence interval would include 1• It is important to know whether relative or absolute risk
is being presented as this influences the way in which it is interpreted
![Page 73: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/73.jpg)
Treatment A Treatment B
SuccessFailure
0.960.04
0.990.01
Issues with RR – defining success
• If the outcome of interest is success then RR=0.96/0.99=0.97• If the outcome of interest is failure then RR=0.04/0.01=4
![Page 74: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/74.jpg)
Absolute Risk Difference
• Is the absolute additional risk of an event due to an exposure.– Risk in exposed group minus risk in unexposed (or
differently exposed group).
• Absolute risk reduction (ARR) = Pexposed - Punexposed • If the absolute risk is increased by an exposure we
sometimes use the term Absolute Risk Increase (ARI)
![Page 75: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/75.jpg)
ARR example• From the previous example of comparing magnesium therapy and placebo:
‘Risk’ of improvement on magnesium = 12/ 15 = 0.80‘Risk’ of improvement on placebo = 3/ 17 = 0.18 Absolute risk reduction = 0.80 - 0.18 = 0.62
![Page 76: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/76.jpg)
Number Needed to Treat
• The additional number of people you would need to give a new treatment to in order to cure one extra person compared to the old treatment.
• For a harmful exposure, the number needed to harm is the additional number of individuals who need to be exposed to the risk in order to have one extra person develop the disease, compared to the unexposed group.– Number needed to treat = 1 / ARR– Number needed to harm = 1 / ARR, ignoring negative sign.
![Page 77: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/77.jpg)
NNT exampleFrom the previous example of comparing magnesium therapy and placebo:
‘Risk’ of improvement on magnesium = 12/ 15 = 0.80‘Risk’ of improvement on placebo = 3/ 17 = 0.18 Absolute risk reduction = 0.80 - 0.18 = 0.62Number needed to treat (to benefit) = 1 / 0.62 = 1.61 ~2Thus on average one would give magnesium to 2 patients in order to expect one extra patient (compared to placebo) to feel better
![Page 78: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/78.jpg)
Odds
• Odds =# times something happens# times it does not happen
• What are the odds of birthing a boy? – For every 2 births, one is a boy and one isn’t
1/1 = 1That is: odds are even
• One of every 100 persons treated with aspirin for a headache, has a side-effect,
1/99 = .0101
![Page 79: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/79.jpg)
OR example• From the previous example of comparing magnesium therapy and placebo:
Odds of improvement on magnesium = 12/3 = 4.0Odds of improvement on placebo = 3/14 = 0.21 Odds ratio (of Mg2+ vs placebo) = 4.0 / 0.21 = 19.0Therefore, improvement was 19 times more likely in the Mg2+ group than the placebo group.
![Page 80: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/80.jpg)
Relative Risk and Odds Ratio
• The odds ratio can be interpreted as a relative risk when an event is rare and the two are often quoted interchangeably
• This is because when the event is rare (b+d)→ d and (a+c)→c. – Relative risk = a(a+c) / b(b+d)
– Odds ratio = ac / bd
![Page 81: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/81.jpg)
Relative Risk and Odds Ratio
• For case-control studies it is not possible to calculate the RR and thus the OR is used.
• For cohort and cross-sectional studies, both can be derived.
• OR have mathematical properties which makes them more often quoted for formal statistical analyses
![Page 82: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/82.jpg)
Continuous data
• Means, averages, change scores etc.– E.g. BP, plasma protein concentration,
• Any value often within a specified range• Mean, Standard deviation, N
• Often only the standard error, SE, presented• SD = SE x √ N
![Page 83: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/83.jpg)
MAStARI Data Extraction Instrument
![Page 84: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/84.jpg)
![Page 85: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/85.jpg)
Group Work 3
• Working in pairs:– Extract the data from the two papers in your workbook
• Reporting Back
![Page 86: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/86.jpg)
Session 5: Protocol development
![Page 87: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/87.jpg)
Program OverviewDay 2
Time Session Group Work
0900 Overview of Day 1
0915 Session 6: Data analysis and meta-analysis
1030 Morning Tea
1100 Session 7: Appraisal extraction and synthesis using JBI MAStARI
Group Work 4: MAStARI trial.Report back
1230 Lunch
1330 Session 8: Protocol Development Protocol development
1415 Session 9: Assessment MCQ Assessment
1445 Afternoon tea
1500 Session 10: Protocol Presentations Protocol Presentations
1700 End
![Page 88: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/88.jpg)
Overview
• Recap Day 1– Critical appraisal– Study design– Type of studies
(experimental and observational)
– Data extraction• Today focus is on data
analysis and synthesis.
![Page 89: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/89.jpg)
Session 6: Data Analysis and Meta-synthesis/Meta-analysis
![Page 90: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/90.jpg)
General Analysis - What Can be Reported and How
– What interventions/activities have been evaluated;– The effectiveness/appropriateness/feasibility of the
intervention/activity;– Contradictory findings and conflicts;– Limitations of study methods;– Issues related to study quality;– The use of inappropriate definitions;– Specific populations excluded from studies; and– Future research needs.
![Page 91: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/91.jpg)
Meta Analysis1004 references
832 referencesScanned Ti/Ab
172 duplicates
117 studiesretrieved
715 do not meetIncl. criteria
82 do not meetIncl. criteria
35 studies forCritical Appraisal
26 studies incl.in review
6 studies incl.in meta analysis
20 studies incl.in narrative
9 excludedstudies
![Page 92: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/92.jpg)
Statistical methods for meta-analysis
• Quantitative method of combining results of independent studies
• Aim is to increase precision of overall estimate• Investigate reasons for differences in risk estimates
between studies• Discover patterns of risk amongst studies
![Page 93: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/93.jpg)
When is meta-analysis useful?
• If studies report different treatment effects.• If studies are too small (insufficient power) to detect
meaningful effects.• Single studies rarely, if ever, provide definitive
conclusions regarding the effectiveness of an intervention.
![Page 94: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/94.jpg)
When meta-analysis can be used
• Meta analysis can be used if studies:– have the same population– use the same intervention administered in the same way.– measure the same outcomes
• Homogeneity– studies are sufficiently similar to estimate an average
effect.
![Page 95: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/95.jpg)
Calculating an Overall Effect Estimate
• Odds Ratio – for dichotomous data e.g. the outcome present or absent– 51/49 = 1.04– (no difference between groups = 1)
• Weighted mean difference– Continuous data, such as weight – (no difference between groups = 0)
• Confidence Interval– The range in which the real result lies, with the given degree of
certainty
![Page 96: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/96.jpg)
Confidence Intervals
• Confidence intervals are an indication of how precise the findings are
• Sample size greatly impacts the CI– the larger the sample size the smaller the CI, the greater
the power and confidence of the estimate
![Page 97: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/97.jpg)
CIs indicate:
• When calculated for OR, the CI provides the upper and lower limit of the odds that a treatment may or may not work
• If the odds ratio is 1, odds are even and therefore, not significantly different – recall the odds of having a boy
![Page 98: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/98.jpg)
![Page 99: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/99.jpg)
Favours treatment Favours controlNo effect
Results of different studies combined
The Meta-view Graph
![Page 100: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/100.jpg)
Heterogeneity
• Is it appropriate to combine or pool results from various studies?
• Different methodologies?• Different outcomes measured?• Problem greater in observational then clinical
studies
![Page 101: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/101.jpg)
Favours treatment Favours controlNo effect
Difference between studies
Heterogeneity
![Page 102: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/102.jpg)
Tests of Heterogeneity
• Measure extent to which observed study outcomes differ from calculated study outcome
• Visually inspect Forest Plot. Size of CI• 2 Test for homogeneity or Q Test can be used
– low power (use p < 0.1 or 0.2)
![Page 103: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/103.jpg)
Favours treatment Favours controlNo effect
Studies too small to detect any effect
Insufficient Power
![Page 104: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/104.jpg)
Meta-analysis
• Overall summary measure is a weighted average of study outcomes.
• Weight indicates influence of study.• Study on more subjects is more influential.• CI is measure of precision.• CI should be smaller in summary measure.
![Page 105: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/105.jpg)
Subgroup analysis
• Subgroup analysis • Some participants, intervention or outcome you thought
were likely to be quite different to the others• Should be specified in advance in the protocol• Only if there are good clinical reasons
• Two types• Between trial – trials classified into subgroups• Within trial – each trial contributes to all subgroups
![Page 106: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/106.jpg)
Taken from Egger, M. et al. BMJ 1998;316:140-144
Example subgroup analysis
![Page 107: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/107.jpg)
Sensitivity Analysis
• Exclude and/or include individual studies in the analysis
• Establish whether the assumptions or decisions we have made have a major effect on the results of the review
• ‘Are the findings robust to the method used to obtain them?’
![Page 108: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/108.jpg)
Meta-analysis
• Statistical methods– Fixed effects model– Random effects model
![Page 109: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/109.jpg)
Fixed Effects Model
• All included studies measure same outcome• Assume any difference observed is due to chance
– no inherent variation in source population– variation within study, not between studies
• Inappropriate where there is heterogeneity present• CI of summary measure reflects variability between
patients within sample
![Page 110: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/110.jpg)
Random Effects Model• Assumed studies are different and outcome will
fluctuate around own true value– true values drawn randomly from population– variability between patients within study and from
differences between studies• Overall summary outcome is estimate of mean from
which sample of outcomes was drawn• More commonly used with observational studies due
to heterogeneity
![Page 111: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/111.jpg)
Random Effects Model
• Summary value will often have wider CI than with fixed effects model
• Where no heterogeneity results of two methods will be similar
• If heterogeneity present may be best to do solely narrative systematic review
![Page 112: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/112.jpg)
Session 7: Appraisal, extraction and synthesis
using JBI-MAStARI
![Page 113: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/113.jpg)
Meta Analysis of Statistics Assessmentand Review Instrument (MAStARI)
![Page 114: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/114.jpg)
![Page 115: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/115.jpg)
![Page 116: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/116.jpg)
![Page 117: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/117.jpg)
![Page 118: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/118.jpg)
![Page 119: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/119.jpg)
![Page 120: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/120.jpg)
![Page 121: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/121.jpg)
![Page 122: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/122.jpg)
![Page 123: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/123.jpg)
![Page 124: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/124.jpg)
![Page 125: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/125.jpg)
![Page 126: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/126.jpg)
![Page 127: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/127.jpg)
![Page 128: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/128.jpg)
![Page 129: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/129.jpg)
![Page 130: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/130.jpg)
![Page 131: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/131.jpg)
![Page 132: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/132.jpg)
![Page 133: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/133.jpg)
![Page 134: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/134.jpg)
![Page 135: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/135.jpg)
![Page 136: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/136.jpg)
![Page 137: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/137.jpg)
![Page 138: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/138.jpg)
![Page 139: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/139.jpg)
![Page 140: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/140.jpg)
Group Work 4
MAStARI Trial and Meta Analysis
![Page 141: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/141.jpg)
Session 8: Protocol development
![Page 142: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/142.jpg)
Session 9: Assessment
![Page 143: Appraisal, Extraction and Pooling of Quantitative Data for Reviews of Effects](https://reader036.vdocuments.us/reader036/viewer/2022062310/56816334550346895dd3bb32/html5/thumbnails/143.jpg)
Session 10: Protocol Presentations