Download - Why do we read research articles?
Why do we read research Why do we read research articles?articles?
To increase our knowledge in a particular area To increase our knowledge in a particular area of interestof interest
To search for evidence for our practiceTo search for evidence for our practiceTo review literature in preparation for a To review literature in preparation for a
researchresearch
Should we believe everything Should we believe everything we read?we read?
CRITICAL CRITICAL APPRAISALAPPRAISAL
Kristofferson G. Mendoza, Kristofferson G. Mendoza, PTRPPTRP
Department of Physical TherapyDepartment of Physical TherapyCollege of Allied Medical ProfessionsCollege of Allied Medical ProfessionsUniversity of the Philippines ManilaUniversity of the Philippines Manila
27 May 200927 May 2009
OPST 199 OPST 199 Methods of Methods of ResearchResearch
Learning Objectives
At the end of the session, the learner should be able to:
Identify the level of evidence presented in a research study on the basis of the research design used
Discuss theoretical and methodological aspects of a research study in relation to internal and external validity
Rate the usefulness of evidence presented in a research study using key criteria
Critical Appraisal
the process of systematically examining research evidence to assess its validity, results and relevance before using it to inform a decision.
LEVELS OF LEVELS OF EVIDENCEEVIDENCE
OPST 199 OPST 199 Methods of ResearchMethods of Research
Critical AppraisalCritical Appraisal
Levels of Evidence
Various study designs have different levels of rigour
Classic Levels of Evidence
By Sackett et al. For Studies on intervention, prevention,
etiology and harm RCT the most rigorous study design Modification of the classic level of
evidence later on included systematic reviews bec of the proliferation of RCTs
Level 1Level 1
Level 2Level 2
Level 3Level 3
Level 4Level 4
Level 5Level 5
LOW
HIGH
Levels of Evidence
For studies on Prognosis optimal individual study – individual
inception cohort study with greater than 80% ff up
Where the clinical decision rule has been validated in a single population
Clinical decision rule are algorithms or scoring systems that lead to prognosis estimation or diagnostic category
Levels of Evidence
For studies on Diagnosis optimal individual study – cohort study with
good reference standards or clinical decision rule tested within one clinical center
Levels of Evidence
For studies on Differential diagnosis / Symptom prevalence Cohort study with good follow up
Levels of Evidence
Despite differences in the optimal study design, certain consistencies are evident across different types of questions
A systematic review of high-quality studies always
provides the highest level of rigor An individual study using the optimal design for
that type of clinical question is considered level 1 Prospective data collection indicates higher study
quality than retrospective data collection Expert opinion, bench research, conceptual
framework/theories/first principles are always considered the lowest (level 5) evidence
Other Levels of evidence Classification System
Greenhalgh (1997)
Primary Research
Experimental
Observational
For qualitative studies? No consensus about the relative rigour of different methods
Levels of evidence
Systematic reviews &
Meta-analyses
Randomised controlled trials
Cohort studies
Case control studies
Case series and case reports
For quantitative studies
RESEARCH RESEARCH DESIGN ISSUESDESIGN ISSUES
OPST 199 OPST 199 Methods of ResearchMethods of Research
Critical AppraisalCritical Appraisal
Research Design Issues in Quantitative Research Sampling of Subjects
Objective: A sample that represents the target population
Optimal Design: A random sample selected from the population
Threats to Validity: Differential sampling between groups will affect internal validity; convenience sampling may affect external validity
Research Design Issues in Quantitative Research Sample Size
Objective: The optimal number of subjects is large enough to detect important treatment effects but small enough to conduct the study in a timely, efficient manner
Optimal Design: Correct number of subjects is determined by an accurate sample size calculation
Threats to Validity: Small sample size
Research Design Issues in Quantitative Research Allocation of treatment
Objective: Unbiased allocation
Optimal Design: Random allocation
Threats to Validity: Nonrandom allocation (observational)
Research Design Issues in Quantitative Research Blinding (to treatment allocation and/or outcomes
Objective: To minimize sources of bias introduced by study personnel or participants
Optimal Design: Ideally everyone involved in the study would be blinded to the full extent possible
Threats to Validity: Blinding in rehab is difficult; may result in bias from differential diagnosis, outcome assessments, attention and follow-up procedures
Research Design Issues in Quantitative Research Outcome ascertainment
Objective: To accurately determine pretreatment status; To reflect all important changes in outcome post-treatment
Optimal Design: Outcomes measured at all likely relevant time points; Outcomes measured using reliable, valid, and responsive measures
Threats to Validity: important effects will be missed if relevant time points are not assessed or the outcome measures are not sensitive; outcome measures that reflect only certain domains may be biased towards specific treatments; poor reliability/validity may invalidate the conclusion
Research Design Issues in Quantitative Research Follow-up
Objective: To accurately portray the treatment effects obtained by all the participants
Optimal Design: 100% follow-up
Threats to Validity: differential loss to follow-up can introduce bias in estimate of effects
Research Design Issues in Quantitative Research Statistical analysis
Objective: To provide accurate estimates of the size and significance of the observed effects
Optimal Design: Accurate and appropriate analysis of all data
Threats to Validity: Inappropriate analysis may lead to faulty conclusion
CRITICAL CRITICAL REVIEW FORMREVIEW FORM
OPST 199 OPST 199 Methods of ResearchMethods of Research
Critical AppraisalCritical Appraisal
Law, M., Stewart, D., Pollock, N., Letts, L., Bosch, J., & Westmorland, M., 1998. McMaster University
www.fhs.mcmaster.ca/canchild
Comments
STUDY PURPOSE:Was the purpose stated clearly?YesNo
Outline the purpose of the study. How does the study apply to OT/PT/SP and/or your research question?
LITERATURE:Was relevant background literature reviewed?YesNo
Describe the justification of the need for the study.
DESIGN:RCTCohortSingle case designBefore and afterCase-controlCross-sectionalCase study
Describe the study design. Was the design appropriate for the study question (e.g., for knowledge level about this issue, outcomes, ethical issues, etc.)?
Specify any biases that may have been operating and the direction of their influence on the results.
Appropriateness of Study Design Knowledge of topic/issue
If little is known, more exploratory (case study, cross-sectional design)
as level of knowledge increases, study design must be more rigorous (RCT most rigorous)
Outcomes If outcomes are easily quantifiable and well-
developed standardized assessment tools are available, design should be rigorous
If outcomes are not fully understood yet (e.g., quality of life), use design that explore different factors (case control)
Appropriateness of Study Design Ethical Issues:
If there is no ethical issue re: withholding treatment, design should have control groups
Study purpose/questions Effectiveness of treatment: RCTs, before-
after design, single-case studies Learn more about an issue, pilot study to
determine treatment: case control an cross sectional
Biases
Sample/Selection Bias Volunteer bias – usually favors treatment
group Seasonal bias – could work either way Attention bias – favors treatment group
(extra attention)
Biases
Measurement/Detection Bias Number of outcome measures used
Only one outcome measure – could favor either groups
Too many outcome measures for sample size – favors control group
Lack of masked or independent evaluation – treatment group is usually favored
Recall or memory bias – favors treatment group
Biases
Intervention/Performance Bias Contamination – favors control group Co-intervention – can influence either
groups Timing of intervention Site of treatment Different therapists
Comments
SAMPLE:N=
Was the sample described in detail?YesNo
Was the sample size justified?YesNo
Sampling (who; characteristics; how many; how was sampling done?) If more than one group, was there similarity between groups?
Describe ethics procedure. Was informed consent obtained?
OUTCOMES:Were the outcome measures reliable?YesNoNot addressed
Were the outcome measures valid?YesNoNot addressed
Specify the frequency of outcome measurement (i.e. pre, post, follow-up).
Outcome areas (e.g., self care, productivity, leisure)
List measures used
Comments
INTERVENTION:Intervention was described in detail?YesNoNot addressed
Contamination was avoided?YesNoNot addressedN/A
Cointervention was avoided?YesNoNot addressedN/A
Provide a short description of the intervention (focus, who delivered it, how often, setting). Could the intervention be replicated in OT/PT/SP practice?
Comments
RESULTS:Results were reported in terms of statistical significance?YesNoN/ANot addressed
Were the analysis method(s) appropriate?YesNoNot addressed
Were the analysis method(s) appropriate?YesNoNot addressed
What were the results? Were they statistically significant (i.e., p<0.05)? If not statistically significant, was study big enough to show an important difference if it should occur? If there were multiple outcomes, was that taken into account for the statistical analysis?
Drop-outs were reported?YesNo
Did any participant drop out from the study? Why? (Were reasons give and were drop outs handled appropriately?)
Comments
CONCLUSIONS AND CLINICAL IMPLICATIONS:Conclusions were appropriate given study methods and results?YesNo
What did the study conclude? What are the implications of these results for OT/PT/SP practice? What were the main limitations or biases in the study?
THANK YOUTHANK YOU
OPST 199 OPST 199 Methods of ResearchMethods of Research
Critical AppraisalCritical Appraisal