outcomes research

Outcomes Outcomes ResearchResearch

Chapter 5Chapter 5

Cummings 5Cummings 5thth ed. ed.

Darshni ViraDarshni Vira

AKA clinical epidemiologyAKA clinical epidemiology Study of the effectiveness of

treatment in a nonrandomized, real-world setting (observational data)

Outcome measures - survival, costs, physiologic measures, QOL

Study OutlineStudy Outline

Pt presents at baseline with a condition

Receives treatment for that condition

Experiences a response to treatment

Bias and ConfoundersBias and Confounders

Bias - “Compared components are not sufficiently similar” Selection bias Treatment bias

Confounders – “VConfounders – “Variable thought to cause an outcome is actually not responsible because of the unseen effects of another variable age, gender, ethnicity, race, comorbidities

Assessment of Baseline Assessment of Baseline ConditionCondition

Definition of diseaseDefinition of disease Inclusion criteriaInclusion criteria

Disease severityDisease severity TNMTNM Sinusitis – Lund-Mackay, Harvard, etc Sinusitis – Lund-Mackay, Harvard, etc

reproducible resultsreproducible results ComorbidityComorbidity

Adult Comorbidity Evaluation 27 (ACE-27) is a validated instrument for evaluating comorbidity in cancer patients

Assessment of TreatmentAssessment of Treatment

Control GroupsControl Groups

Assessment of OutcomesAssessment of Outcomes

EfficacyEfficacy Health intervention, in a controlled

environment, achieves better outcomes than does placebo

Effectiveness Retains its value under usual clinical

circumstances

Study DesignStudy DesignDesignDesign AdvantagesAdvantages DisadvantagesDisadvantages Level of EvidenceLevel of Evidence

Randomized clinical trial (RCT)

Only design to prove causationUnbiased distribution of confounding

Expensive and complexTypically targets efficacy

1, if high-quality RCT2, if low-quality RCT

Observational (cohort) study

Cheaper than RCTClear temporal directionality from treatment to outcome

Difficult to find suitable controlsConfounding

2, with control group4, if no control group

Case-control study

Cheaper than cohort studyEfficient study of rare diseases or delayed outcomes

Must rely on retrospective dataDirectionality between exposure and outcome unclear

33

Case series Cheap and simple No control groupNo causal link betweentreatment and outcome

44

Expert opinion n/an/a n/an/a 55

Grade of Grade of RecommendationRecommendation

(EBM)(EBM)

Level of EvidenceLevel of Evidence

AA 11

BB 2 or 32 or 3

CC 44

DD 55

Measurement of Clinical Measurement of Clinical OutcomesOutcomes

Psychometric Validation Psychometric Validation (questionnaires)(questionnaires) ReliabilityReliability ValidationValidation ResponsivenessResponsiveness BurdenBurden

Categories of OutcomesCategories of Outcomes

Health StatusHealth Status Individual’s physical, emotional, and

social capabilities and limitations Function

How well an individual is able to perform important roles, tasks, or activities

QOL Central focus is on the value that

individuals place on their health status and function

Examples of Outcome Examples of Outcome MeasuresMeasures

Medical Outcomes Study Short Form-36 (SF-36) European Organization for Research and

Treatment of Cancer Quality of Life Questionnaire (EORTC-HN35)

Hearing Handicap Inventory in the Elderly (HHIE)

Sinonasal Outcome Test (SNOT-20) Child Health Questionnaire (CHQ) Voice Handicap Index Functional Outcomes of Sleep Questionnaire

(FOSQ)

Interpreting Interpreting Medical DataMedical Data

Chapter 6Chapter 6

Cummings 5Cummings 5thth ed. ed.

Habits of a Highly Effective Habits of a Highly Effective Data UserData User

1. Check quality before quantity

2. Describe before you analyze

3. Accept the uncertainty of all data

4. Measure error with the right statistical test

5. Put clinical importance before statistical significance

6. Seek the sample source

7. View science as a cumulative process

1. Check Quality before 1. Check Quality before QuantityQuantity

Experimental vs observational studyExperimental vs observational study BiasBias ConfoundersConfounders Control groupControl group Placebo responsePlacebo response Prospective studies measure incidence

(new events) whereas retrospective studies measure prevalence (existing events)

2. Describe Before You 2. Describe Before You AnalyzeAnalyze

Begins by defining the measurement scale that best suits the observations Categorical (qualitative) Numerical (quantitative)

Bell-shaped curve with standard deviation Median Survival curve

CategoricalCategorical

ScaleScale DefinitionDefinition ExampleExample

Dichotomous two mutually exclusive categories

Breastfeeding (yes/no), sex (male/female)

Nominal unordered qualitative categories

Race, religion, country of origin

Ordinal ordered qualitative categories, but with no natural (numerical) distance between their possible values

Hearing loss (none, mild, moderate), patient satisfaction (low, medium, high), age group

Odds ratio with retrospective reviewOdds ratio with retrospective review Relative risk with prospective reviewRelative risk with prospective review Rate difference with prospective trialsRate difference with prospective trials Correlation coefficient with ordinal or Correlation coefficient with ordinal or

numerical datanumerical data Coefficient (r) from 0 to 0.25 indicates little or

no relationship, from 0.25 to 0.49 a fair relationship, from 0.50 to 0.74 a moderate to good relationship, and greater than 0.75 a good to excellent relationship. A perfect linear relationship would yield a coefficient of 1.00

3. Accept the Uncertainty 3. Accept the Uncertainty in All Datain All Data

Precision (repeatability) Should be reported with a 95% confidence

interval Precision may be increased by using a more

reproducible measure, by increasing the number of observations (sample size), or by decreasing the variability among the observations

Accuracy measures nearness to the truth measured in an unbiased manner and reflect

what is truly purported to be measured

4. Measure Error with the 4. Measure Error with the Right Statistical TestRight Statistical Test

All statistical tests measure errorAll statistical tests measure error Choosing the right test is Choosing the right test is

determined by determined by (1) whether the observations come from independent or related samples, (2) whether the purpose is to compare groups or to associate an outcome with one or more predictor variables, and (3) the measurement scale of the variables

Null hypothesisNull hypothesis Results observed in a study, experiment, or test that are no different from what might have occurred due to chance alone

Statistical test Procedure used to reject or accept a null hypothesis

Type I (alpha) error Rejecting a true null hypothesis (false-positive error); declaring that a difference exists when in fact it does not

P value Probability of making a type I error; P < .05 indicates a statistically significant result that is unlikely to be caused by chance

Type II (beta) error Accepting a false null hypothesis (false-negative error); declaring that a difference does not exist when in fact it does

PowerPower Probability that the null hypothesis will be rejected if it is indeed false; mathematically, power is 1.00 minus type II error

5. Putting Clinical 5. Putting Clinical Importance Before Importance Before

Statistical SignificanceStatistical Significance The next logical question after “Is there a

difference?” (statistical significance) is “How big a difference is there?” (clinical importance)

Effect size reflects the magnitude of difference between groups Measured by correlation coefficient

Confidence intervals (CI) are more appropriate measures of clinical importance than P values, because they reflect both magnitude and precision If “significant” results, the lower limit of the 95% CI should

be scrutinized; a value of minimal clinical importance suggests low precision (inadequate sample size)

If “nonsignificant” results, the upper limit of the 95% CI should be scrutinized; a value indicating a potentially important clinical effect suggests low statistical power (false-negative finding)

6. Seek the Sample 6. Seek the Sample SourceSource

A statistical test is valid only when the study sample is random and representative

Identifying the sampling method and selection criteria (inclusion and exclusion criteria) that were applied to the target population to obtain the study sample

When the process appears sound, one concludes that the results are generalizable

7. View Science as a 7. View Science as a Cumulative ProcessCumulative Process

Process of Integration Process of Integration Systemic Reviews (meta-analysis) Systemic Reviews (meta-analysis) Clinical practice guidelinesClinical practice guidelines

Popular Statistical TestsPopular Statistical Tests T-test - T-test - comparing the means of two independent or

matched (related) samples of numerical data ANOVA - three or more independent groups of

continuous data differ significantly with regard to a single factor (oneway ANOVA) or two factors (two-way ANOVA)

Contingency tables - association between two categorical variables by using the chi-square statistic

Survival analysis (Kaplan-Meier) - estimates the probability of an event based on the total period of observation

Multivariate (regression) - Multivariate (regression) - Examines the simultaneous effect of multiple predictor variables (generally three or more) on an outcome of interest

Statistical DeceptionsStatistical Deceptions Standard error is used

instead of standard deviation

Small sample study results are taken at face value

Post hoc P values are used for statistical inference

Some results are “significant” but there are a large number of P values

Subgroups are compared until statistically significant results are found

No significant difference is found between groups in a small sample study

Significant P values are crafted through improper use of hypothesis tests

The EndThe End