Types of study designs: from cross-sectional to randomized controlled trials

Types of study designs: from descriptive studies to randomized controlled trialsKirsten Bibbins-Domingo, PhD, MDAssistant Professor of Medicine and of Epidemiology and BiostatisticsUniversity of California, San FranciscoObjectivesTo understand the difference between descriptive and analytic studies

To identify the hierarchy of study designs, and the strengths and weakness of each design

To be able to apply different study designs to the same research questionTypes of StudiesDescriptive Studies

Observational Analytic Studies Cross Sectional studiesCase Control studiesCohort studies

Experimental StudiesRandomized controlled trialsHierarchy of Study TypesDescriptiveCase reportCase seriesSurveyAnalyticObservationalCross sectionalCase-controlCohort studiesExperimentalRandomized controlled trialsStrength of evidence for causality between a risk factor and outcomeDescriptive studiesGetting a lay of the landSurveys (NHIS, MCBS)How many men in the U.S. filled Viagra prescriptions in 2004?

Describing a novel phenomenaCase reports or case seriesViagra-associated serous macular detachment. Sildenafil-associated nonarteritic anterior ischemic optic neuropathy. Descriptive studiesCannot establish causal relationships

Still play an important role in describing trends and generating hypotheses about novel associations

The start of HIV/AIDS researchSquamous cell carcinoma in sexual partner of Kaposi sarcoma patient. Lancet. 1982 Jan 30;1(8266):286. New outbreak of oral tumors, malignancies and infectious diseases strikes young male homosexuals. CDA J. 1982 Mar;10(3):39-42. AIDS in the "gay" areas of San Francisco. Lancet. 1983 Apr 23;1(8330):923-4. Analytic StudiesAttempt to establish a causal link between a predictor/risk factor and an outcome.

You are doing an analytic study if you have any of the following words in your research question: greater than, less than, causes, leads to, compared with, more likely than, associated with, related to, similar to, correlated withHierarchy of Study TypesDescriptiveCase reportCase seriesSurveyAnalyticObservationalCross sectionalCase-controlCohort studiesExperimentalRandomized controlled trialsStrength of evidence for causality between a risk factor and outcomeResearch Question

Is the regular consumption of Red Bull associated with improved academic performance among U.S. medical students?

Rationalefunctional drink designed for periods of mental and physical exertion.

performance, concentration, memory, reaction time, vigilance, and emotional balance

Taurine + glucuronolactone + caffeine

BackgroundAlford C, Cox H, Wescott R. The effects of red bull energy drink on human performance and mood. Amino Acids. 2001;21(2):139-50.

Warburton DM, Bersellini E, Sweeney E. An evaluation of a caffeinated taurine drink on mood, memory and information processing in healthy volunteers without caffeine abstinence. Psychopharmacology (Berl). 2001 Nov;158(3):322-8.

Seidl R, Peyrl A, Nicham R, Hauser E. A taurine and caffeine-containing drink stimulates cognitive performance and well-being. Amino Acids. 2000;19(3-4):635-42.

Horne JA, Reyner LA. Beneficial effects of an "energy drink" given to sleepy drivers. Amino Acids. 2001;20(1):83-9.

Kennedy DO, Scholey AB. A glucose-caffeine 'energy drink' ameliorates subjective and performancedeficits during prolonged cognitive demand. Appetite. 2004 Jun;42(3):331-3.

Great idea, but how do you get started.Interesting, novel, and relevant, but

You only have 25,000 dollars to start investigating this question.

What is feasible?

Study Design #1Cross-sectional study of UCSF medical students taking USMLE Step 2

Questionnaire administered when registering for USMLE 2 Primary predictor: self-report of >3 cans Red Bull per week for the previous yearCovariates: Age, sex, undergraduate university, place of birth

Outcome: Score on USMLE Step 2

Cross-sectional study: structuretimeUSMLE Score

Red Bull consumption

Cross-sectional Study:Descriptive value:How many UCSF medical students drink Red Bull?What is the age and sex distribution of UCSF medical students who drink Red Bull?Analytic value:Is there an association between regular Red Bull consumption and test scores among UCSF med students?UnivariateMultivariate (controlling for confounders)Other cross-sectional surveys:AAMCCalifornia Health Interview Survey (NHIS, CHIS)National Health and Nutrition Exam Survey (NHANES)Cross-sectional Study: Pluses+Prevalence (not incidence)

+Fast/Inexpensive - no waiting!

+No loss to follow up

+Associations can be studied Measures of associationDiseaseYesNoRisk FactorYesABNoCDRisk ratio (relative risk)

AA + BCC + DCross-sectional study: minusestime- Cannot determine causalityUSMLE Score

Red Bull consumption

Cross-sectional study: minusestime- Cannot determine causalityACE inhibitor use and hospitalization rates among those with heart failure

Heart failure patients with a documented DNR status and mortality Cross-sectional study: minuses- Cannot determine causality- Cannot study rare outcomesWhat if you are interested in the rare outcome?The association between regular Red Bull consumption andA perfect score on the USMLE Step 2Graduating top 1% of the medical school classAcceptance into a highly selective residency

ANSWER: A Case-Control study

Study Design #2A case-control studyCases: 4th year med students accepted to residency in highly selective specialty X.Controls: 4th year med students who applied but were not accepted.Predictor: self-reported regular Red Bull consumptionAdditional covariates (age, sex, medical school, undergraduate institution)

Case control studiesInvestigator works backward (from outcome to predictor)

Sample chosen on the basis of outcome (cases), plus comparison group (controls)Case-control study structuretimeTARGET CASESMedical students accepted to highly selective residenciesACTUAL CASES4th year UCSF students who matched in highly selective specialty XTARGET CONTROLSAll unsuccessful applicants to highlyselective residency programsACTUAL CONTROLS4th year students who failed to match in highly selective specialty XRed Bull consumptionYESRed Bull consumptionNOpresentCase control studiesDetermines the strength of the association between each predictor variable and the presence or absence of diseaseCannot yield estimates of incidence or prevalence of disease in the population (why?)Odds Ratio is statisticsCase-control Study: pluses

+Rare outcome/Long latent period

+Inexpensive and efficient: may be only feasible option

+Establishes association (Odds ratio)

+Useful for generating hypotheses (multiple risk factors can be explored)

Case-control study-minusesCausality still difficult to establish

Selection bias (appropriate controls)Caffeine and Pancreatic cancer in the GI clinic

Recall bias: sampling (retrospective)Abortion and risk of breast cancer in Sweden

Cannot tell about incidence or prevalenceStudies of diagnostic tests:Sensitivity, specificityPositive predictive value, negative predictive valueMeasures of associationDiseaseYesNoTestYesABNoCDSensitivity = A/A+C

Specificity = D/B+DPPV = A/A+B

NPV = D/C+DCase-control - the house redRely tampons and toxic shock syndrome:

High rates of toxic shock syndrome in menstruating women

Suspected OCPs or meds for PMS

Cases: 180 women with TSS in 6 geographic areas

Controls: 180 female friends of these patients and 180 females in the same telephone code

Tampon associated with TSS (OR = 29!)

Super absorbency associated with TSS (OR 1.34 per gm increase in absorbency)

Led to RELY brand tampons being taken off the market.Where are we?Preliminary results from our cross-sectional and case-control study suggest an association between Red Bull consumption and improved academic performance among medical students

Whats missing? - strengthening evidence for a causal link between Red Bull consumption and academic performance

Use results from our previous studies to apply for funding for a prospective cohort study!

Study design #3Prospective cohort study of UCSF medical students Class of 2009All entering medical students surveyed regarding beverage consumption and variety of other potential covariates Survey updated annually to record changes in Red Bull consumptionOutcomes: USMLE Step 1 score, USMLE Step 2 score, match in first choice residency

Cohort studiesA cohort (follow-up, longitudinal) study is a comparative, observational study in which subjects are grouped by their exposure status, i.e., whether or not the subject was exposed to a suspected risk factor

The subjects, exposed and unexposed to the risk factor, are followed forward in time to determine if one or more new outcomes (diseases) occurSubjects should not have outcome variable on entryNo new subjects allowed in after initial recruitment

The rates of disease incidence among the exposed and unexposed groups are determined and compared.Elements of a cohort studySelection of sample from populationMeasures predictor variables in sampleFollow population for period of timeMeasure outcome variable

Famous cohort studiesFraminghamNurses Health StudyPhysicians Health StudyOlmsted County, Minnesota timeThe presentThe futureTop USMLE scorersEveryone elseProspective cohort study structureStrengths of cohort studiesKnow that predictor variable was present before outcome variable occurred (some evidence of causality)

Directly measure incidence of a disease outcome

Can study multiple outcomes of a single exposure (RR is measure of association)Weaknesses of cohort studiesExpensive and inefficient for studying rare outcomesHERS vs. WHI

Often need long follow-up period or a very large populationCARDIA

Loss to follow-up can affect validity of findingsFraminghamOther types of cohort studiesRetrospective cohortIdentification of cohort, measurement of predictor variables, follow-up and measurement of outcomes have all occurred in the pastMuch less costly than prospective cohortsInvestigator has minimal control over study designOther types of cohort studiesNested case-control studyCase-control study embedded in a cohort studyControls are drawn randomly from study sampleDouble cohortUsed to compare two separate cohorts with different levels of exposure to predictor variable (e.g., occupational groups)What type of study is this?Among individuals with coronary disease, what is the association between baseline levels of B-type natriuretic peptide and subsequent risk of heart failure?

Among individuals presenting to heart failure clinic, what is the association between self-reported symptoms and risk of hospitalization for heart failure?

Using data from HERS (RCT of HRT in women with coronary disease):Determine the risk factors for developing incident heart failure among women without heart failure at baseline.Determine whether HRT is associated with mortality among women with heart failure.Determine genetic markers for development of heart failure among black women in HERS.Hierarchy of Study TypesDescriptiveCase reportCase seriesSurveyAnalyticObservationalCross sectionalCase-controlCohort studiesExperimentalRandomized controlled trialsStrength of evidence for causality between a risk factor and outcomeWhat distinguishes observational studies from experiments?Ability to control for confoundingPredictorOutcomeConfounderExamples: sex (men are more likely to drink red bull and men are more likely to match in neurosurgery) Undergraduate institution (students from northwest school are more likely to drink red bull and also more likely to score higher onUSMLE)But we measured all of the potential confounders.In a prospective cohort study you can (maybe) measure all potential known confounders, but

You cant control for unanticipated or unmeasured confoundersStudy design # 4Randomized controlled trial of daily Red Bull consumption among entering UCSF medical students Class 2009

Randomized to daily consumption of Red Bull vs. daily consumption of placebo

Outcomes: USMLE Step 1 score, USMLE Step 2 score, match in first choice residency

Randomized controlled trialsInvestigator controls the predictor variable (intervention or treatment)Major advantage over observational studies is ability to demonstrate causalityRandomization controls unmeasured confoundingOnly for mature research questionsBasic Trial DesignPopulationSampleTreatmentDxNo DxControlDxNo DxPlaceboRandomization45For example, select persons with high LDL-cholesterol from among patients presenting to the metabolic clinic at the SFVA and treat with lovastatin to see if this results in lower LDL cholesterol.

Note can have an uncontrolled trial or time series design. These have no concurrent control group and can produce the wrong answer due to learning, regression to the mean or secular trends. Note, for example, that it is common to have a 30-50% improvement in depression scores, pain scores, hot flash scores.

Can have a controlled trial without randomization. This addresses learning, regression to the mean and secular trends, but introduces the issue of baseline differences in the groups. This design is really no different from a double cohort study.

Can have a randomized controlled trial without placebo (blinding). This allows (as will be discussed later) for co-intervention and biases outcome ascertainment.

Also note that these three elements - treatment, randomization and blinding result in the major ethical concerns regarding randomized trials. Steps in a randomized controlled trialSelect participantshigh-risk for outcome (high incidence)Likely to benefit and not be harmedLikely to adhere Measure baseline variablesRandomizeEliminates baseline confoundingTypes (simple, stratified, block)

Steps in a randomized controlled trialBlinding the interventionAs important as randomizationEliminates co interventionbiased outcome ascertainmentbiased measurement of outcomeFollow subjectsAdherence to protocolLost to follow upMeasure outcomeClinically important measuresAdverse eventsWhat is Blinding?Single blind - participants are not aware of treatment groupDouble blind - both participants and investigators unawareTriple blind - various meaningspersons who perform testsoutcome adjudicatorssafety monitoring groupWhy blind?: Co interventionsUnintended effective interventionsparticipants use other therapy or change behaviorstudy staff, medical providers, family or friends treat participants differentlyNondifferential - decreases powerDifferential - causes biasWhy blind?: Biased Outcome Ascertainment or adjudicationIf group assignment is known participants may report symptoms or outcomes differentlyphysicians or investigators may elicit symptoms or outcomes differently Study staff or adjudicators may classify similar events differently in treatment groups

Problematic with soft outcomesinvestigator judgementparticipant reported symptoms, scales

50Analysis of randomized controlled trialAnalyzed like cohort study with RRIntention to treat analysisMost conservative interpretationInclude all persons assigned to intervention group (including those who did not get treatment or dropped out)Subgroup analysisGroups identified pre-randomizationHigh Quality Randomized TrialsTamper-proof randomizationBlinding of participants, study staff, lab staff, outcome ascertainment and adjudicationAdherence to study intervention and protocolComplete follow-upHierarchy of Study TypesDescriptiveCase reportCase seriesSurveyAnalyticObservationalCross sectionalCase-controlCohort studiesExperimentalRandomized controlled trialsStrength of evidence for causality between a risk factor and outcomeA study type of every budget, purpose and research question