C H A P T E R 1

An Overview of Clinical Research:The Lay of the Land

C H A P T E R C O N T E N T S

A Taxonomy of Clinical Research 2What Studies Can and Cannot Do 4Descriptive Study 4Cross-Sectional Study: A Snapshot in Time 4Cohort Study: Looking Forward in Time 5Case-Control Study: Thinking Backwards 5Variations on These Themes 6

Nonrandomised Trial: Penultimate Design? 7Randomised Controlled Trial: Gold Standard 7Measurement of Outcomes 8Confusing Fractions 8Measures of Association: Risky Business 9Conclusion 10

Many clinicians report that they cannot read the medical literature critically. To address this dif-ficulty, we provide a primer of clinical research for clinicians and researchers alike. Clinical researchfalls into two general categories: experimental and observational, based on whether the investigatorassigns the exposures or not. Experimental trials can also be subdivided into two: randomised andnonrandomised. Observational studies can be either analytical or descriptive. In this book, we dis-tinguish between the two by this criterion: analytical studies feature a comparison (control) group,whereas descriptive studies do not. Within analytical studies, cohort studies track people forward intime from exposure to outcome. By contrast, case-control studies work in reverse, tracing back fromoutcome to exposure. Cross-sectional studies are like a snapshot, which measures both exposure andoutcome at one time point. Descriptive studies cannot examine associations, a fact often forgotten orignored. Measures of association, such as relative risk or odds ratio, are the preferred way of expres-sing results of dichotomous outcomes (e.g., sick versus healthy). Confidence intervals around thesemeasures indicate the precision of these results. Measures of association with confidence intervalsreveal the strength, direction, and plausible range of an effect, as well as the likelihood of chanceoccurrence. By contrast, p values address only chance. Testing null hypotheses at a p value of0.05 has no basis in medicine and should be discouraged.

Clinicians today are in a bind. Increasing demands on their time are squeezing out opportunities tostay abreast of the literature, much less read it critically. PubMed currently catalogues 30,000 jour-nals, and its citations include more than 27 million entries. Results of several studies indicate aninverse relation between quality of care and time since graduation from medical school;1 older cli-nicians consistently have a harder time keeping up.2 In many jurisdictions, attendance at a specifiednumber of hours of continuing medical education (CME) courses is mandatory to maintain alicence to practice. Continuing medical education courses have at best a modest effect on the qualityof care, and such courses alone appear unlikely to affect complex clinical behaviours.3 This defi-ciency of traditional CME offerings emphasises the importance of self-directed learning through

1

2 INTRODUCTION

reading. However, many clinicians lack the requisite skills to read the medical literature critically.4,5

Scientific illiteracy6 and innumeracy5 remain major failings of medical education.This book on research methods is designed for busy clinicians and active researchers. The needs

of clinicians predominate; hopefully, this primer will produce more critical and thoughtful con-sumers of research, and thus better practitioners. The needs of clinicians overlap with those ofresearchers throughout the chapters, but that overlap becomes most pronounced in the discussionof randomised controlled trials. For readers to assess randomised trials accurately, they shouldunderstand the relevant guidelines on the conduct of trials; these guidelines have been derived fromempirical methodological research.

The disproportionate coverage of randomised controlled trials is intentional; randomised con-trolled trials are the gold standard in clinical research. Randomised controlled trials help to elim-inate bias, and research has identified the important methodological elements of trials that minimisebias.7,8 Finally, because trials are deemed more credible than observational studies,9,10 cliniciansmight be more likely to act on their results than on those of other study types; hence, investigatorsneed to ensure that trials are done well and reported well. To start, we provide a brief overview ofresearch designs and discuss some of the common measures used.

A Taxonomy of Clinical ResearchAnalogous to biological taxonomy, a simple hierarchy can be used to categorise most studies(Panel 1.1).11 To do so, however, the study design must be known. As in biology, anatomy dictatesphysiology. The anatomy of a study determines what it can and cannot do. A difficulty that readersencounter is that authors sometimes do not report the study type or provide sufficient detail to figureit out. A related problem is that authors sometimes incorrectly label the type of research done.Examples include calling nonrandomised controlled trials randomised,12 and labelling nonconcur-rent cohort studies case-control studies.13-15 The adjective ‘case-controlled’ is also sometimes (inap-propriately) applied to any study with a comparison group. The media compound the confusion byimplying causation16 (when researchers cautiously reported only an association).17

Biology has animal and plant kingdoms. Similarly, clinical research has two large kingdoms: exper-imental and observational research. Fig. 1.1 shows that one can quickly decide the research kingdomby noting whether the investigators assigned the exposures (e.g., treatments) or whether they observed

PANEL 1.1 ■ Rating Clinical Evidence: Assessment System of the US PreventiveServices Task Force11

Quality of Evidence

I Evidence from at least one properly designed randomised controlled trial.

II-1 Evidence obtained from well-designed controlled trials without randomisation.

II-2 Evidence from well-designed cohort or case-control studies, preferably from more than one

centre or research group.

II-3 Evidence from multiple time series with or without the intervention. Important results in

uncontrolled experiments (such as the introduction of penicillin treatment in the 1940s) could

also be considered as this type of evidence.

III Opinions of respected authorities, based on clinical experience, descriptive studies, or reports of

expert committees.

Strength of Recommendations

A Strongly recommends

B Recommends

C Makes no recommendation for or against

D Recommends against

I Concludes evidence insufficient to recommend for or against

Did investigatorassign exposures?

Experimental study

Random allocation?

Observational study

Comparison group?

Analyticalstudy

Direction?

Descriptivestudy

Yes

Yes No

No

Yes No

Randomisedcontrolled

trial

Cohortstudy

Case-controlstudy

Cross-sectional

study

Non-randomisedcontrolled

trial

Exposure Outcome

Exposure Outcome

Exposure and outcome at the same time

Fig. 1.1 Algorithm for classification of types of clinical research.

31—AN OVERVIEW OF CLINICAL RESEARCH: THE LAY OF THE LAND

usual clinical practice.18-20 At the top of the hierarchy, for experimental studies one needs to distin-guish whether the exposures were assigned by a truly random technique (with concealment of theupcoming assignment from those involved) or whether some other allocation scheme was used, suchas alternate assignment.21 Many reports fail to include this critical information.22

With observational studies, which dominate the literature, especially in lower-impact journals,23

the next step is to ascertain whether the study has a comparison or control group. If so, the study istermed analytical.24 If not, it is a descriptive study (see Fig. 1.1). If the study is analytical, the tem-poral direction needs to be identified. If the study determines both exposures and outcomes at onetime point, it is termed cross-sectional. An example would be measurement of serum cholesterol ofpatients admitted to a hospital with myocardial infarction versus that of their next-door neighbour.This type of study provides a snapshot of the population of sick and well at one time point.25 Aweakness of cross-sectional studies is that the temporal sequence may be unclear: did the exposureprecede the outcome? In addition, cross-sectional studies are inappropriate for rare diseases or thosethat resolve quickly; severe diseases causing early death can mean that those studied are not repre-sentative of all those with a given disease.19

If the study begins with an exposure (e.g., oral contraceptive use) and follows women for years tomeasure outcomes (e.g., ovarian cancer), then it is deemed a cohort study. Cohort studies can beeither concurrent or nonconcurrent. By contrast, if the analytical study begins with an outcome(e.g., ovarian cancer) and looks back in time for an exposure, such as use of oral contraceptives, thenthe study is a case-control study.

Studies without comparison groups are called descriptive studies.24 At the bottom of theresearch hierarchy is the case report. Indeed, the literature is replete with reported oddities.26

4 INTRODUCTION

When more than one patient is described, it becomes a case-series report. Some new diseases firstappear in the literature this way.27

What Studies Can and Cannot DoDESCRIPTIVE STUDY

Starting at the bottom of the research hierarchy, descriptive studies are often the first foray into a newarea ofmedicine, a first ‘toe in the water’. Investigators do descriptive studies to describe the frequency,natural history, and possible determinants of a condition.18,19 The results of these studies show howmany people develop a disease or condition over time, describe the characteristics of the disease andthose affected, and generate hypotheses about the cause of the disease. These hypotheses can beassessed through more rigorous research, such as analytical studies or randomised controlled trials.An example of a descriptive study would be the early reports of legionnaires’ disease28 and toxic-shocksyndrome.29 An important caveat (often forgotten or intentionally ignored) is that descriptive studies,which do not have a comparison group, do not allow assessment of associations.30 Only comparativestudies (both analytical and experimental) enable assessment of possible causal associations.

CROSS-SECTIONAL STUDY: A SNAPSHOT IN TIME

Sometimes termed a frequency survey or a prevalence study,24 cross-sectional studies are done toexamine the presence or absence of disease and the presence or absence of an exposure at a particulartime. Thus prevalence, not incidence, is the focus. As both outcome and exposure are ascertained atthe same time (Fig. 1.2), the temporal relation between the two can be unclear. For example, assumethat a cross-sectional study finds obesity to be more common among women with arthritis thanamong those without this condition. Did the extra weight load on joints lead to arthritis, or didwomen with arthritis become involuntarily inactive and then obese? This ‘chicken-egg’ questionis unanswerable in a cross-sectional study.

Cohort study

Exposure Outcome

Case-control study

Cross-sectional study

Exposure

Outcome

Time

Exposure Outcome

Fig. 1.2 Schematic diagram showing temporal direction of three study designs.

51—AN OVERVIEW OF CLINICAL RESEARCH: THE LAY OF THE LAND

COHORT STUDY: LOOKING FORWARD IN TIME

Cohort studies proceed in a logical sequence from exposure to outcome (see Fig. 1.2). Hence, thistype of research is easier to understand than case-control studies. Investigators identify a group withan exposure of interest and another group or groups without the exposure. The investigators thenfollow the exposed and unexposed groups forward in time to determine outcomes. If the exposedgroup develops a higher incidence of the outcome than the unexposed, then the exposure is asso-ciated with an increased risk of the outcome, and vice versa.

Cohort studies can be prospective, retrospective, or both. In a prospective cohort study, theexposed and unexposed are identified and followed forward in time to outcomes. In a retrospectivecohort study, the investigator goes back in time through medical records to identify exposuregroups, then tracks them to outcomes through existing medical records. In some disciplines, ret-rospective cohort studies are more common than prospective cohort studies. As described inChapter 4, ambidirectional cohort studies look backward and forward in time.

The cohort study has important strengths and weaknesses. Because exposure is identified at theoutset, one can usually determine that the exposure preceded the outcome. Recall bias is less of aconcern than in the case-control study. The cohort study enables calculation of true incidence rates,relative risks, and attributable risks. However, for the study of rare events or events that take years todevelop, this type of research design can be slow to yield results and thus prohibitively expensive.Nonetheless, several famous, large cohort studies31,32 continue to provide important information.

CASE-CONTROL STUDY: THINKING BACKWARDS

Case-control studies work backwards. Because thinking in this direction is not intuitive for clini-cians, case-control studies are widely misunderstood. Indeed, in one sample from the rehabilitationliterature, 97% of research articles designated ‘case-control’ were mislabelled.33

Starting with an outcome, such as disease, this type of study looks backward in time for expo-sures that might be related to the outcome. As shown in Fig. 1.2, investigators define a group withan outcome (e.g., ovarian cancer) and a group without the outcome (controls). Then, through chartreviews, interviews, electronic medical records, or other means, the investigators ascertain the prev-alence (or amount) of exposure to a risk factor (e.g., oral contraceptives, ovulation-induction drugs)in both groups. If the prevalence of the exposure is higher among cases than among controls, thenthe exposure is associated with an increased risk of the outcome.

Case-control studies are especially useful for outcomes that are rare or that take a long time todevelop, prototypes being cardiovascular disease and cancer. These studies often require less time,effort, and money than would cohort studies. The Achilles heel of case-control studies is choosingan appropriate control group, discussed further in Chapter 6. Controls should be similar to cases inall important respects except for not having the outcome in question. Inappropriate control groupshave ruined many case-control studies and caused much harm. Additionally, recall bias (better rec-ollection of exposures among the worried cases than among the healthy controls) is inevitable instudies that rely on memory. Because the case-control study samples on cases and controls, noton exposures, investigators cannot calculate incidence rates, relative risks, or attributable risks.Instead, odds ratios are themeasure of association used; when the outcome is uncommon (e.g., mostcancers) the odds ratio provides a good proxy for the true relative risk.34

Investigations of outbreaks of food-borne diseases often use case-control studies. On a cruiseship, the entire universe of those at risk is known. Those with vomiting and diarrhoea (cases)are asked about food exposures, as are a sample of those not ill (controls). If a higher proportionof those who are ill reports having eaten a specific food than those who are well, the food becomessuspect. In 50 outbreaks associated with cruise ships from 1970 to 2003, the most common culpritwas found to be seafood (28% of outbreaks).35

6 INTRODUCTION

VARIATIONS ON THESE THEMES

Nested Case-Control Studies

Here, a group of persons being studied is followed forward in time until outcomes occur. Instead ofanalysing everyone in the group in usual cohort fashion, those with the outcome are deemed cases,and a sample of those without the disease become controls.36 Then, the analysis proceeds in typicalcase-control fashion. The smaller case-control study is ‘nested’ within a larger cohort underscrutiny.24

Why analyse a cohort study this way? This approach may be useful when determining the expo-sure involves an expensive, painful, invasive test or ponderous interview. Determining exposureinformation for all participants might be impractical or prohibitively expensive. All cases have expo-sure determined, but only a small sample of all healthy persons (controls) need this done. A study ofinflammatory markers and postpartum depression is illustrative. A prospective cohort of womenbeing followed through pregnancy had blood samples collected and banked at specified times.Sixty-three women with depression had a sophisticated protein panel run on their blood sampleto examine 92 inflammation markers. The same panel was run on blood from 228 controls withoutdepression.37 Performing this extensive testing on all women in the cohort would have been costly.

Case-Cohort Studies

In this modification of a case-control study, ‘…controls are drawn from the same cohort as the casesregardless of their disease status. Cases of the disease of interest are identified, and a sample of theentire starting cohort (regardless of their outcomes) forms the controls’24 Here, one control groupcan be used for multiple groups of cases, increasing the efficiency of the exploration. Controls arerandomly selected from the population at risk; these controls are called the ‘reference subcohort’.38

In a traditional case-control study, researchers would need to recruit a different control group foreach group of cases being studied; here, one control group can be recycled repeatedly.

A case-cohort study examined risk factors for hospitalisation after dog-bite injury.39 The cohortconsisted of 1384 dog-bite patients (Fig. 1.3) seen at one hospital emergency department; 111 ofthese who were admitted formed the case group. A simple random sample of the other patients

Fig. 1.3 Dog bite.

71—AN OVERVIEW OF CLINICAL RESEARCH: THE LAY OF THE LAND

comprised the control subcohort (221 patients). As noted previously, controls were selected inde-pendent of their disease status; by chance, 21 patients randomly chosen as controls were themselvescases (admitted to the hospital). Infected bites and bites to multiple sites were most strongly relatedto the risk of being hospitalised.

Case-Crossover Studies

This study type can be used to assess adverse drug reactions or events rapidly triggered by an expo-sure. For example, a case with an adverse event is queried about exposures in the prior hour, deemedthe hazard window.25 The control interval might be the same time window in the same patient1 day earlier. This helps to control for confounding by indication, because each study participantserves as both the case and the control.40 An example would be studying the association betweensedatives and car wrecks.41

The possibility that chiropractic manipulation might precipitate carotid artery stroke prompteda case-crossover study in Canada.42 All incident strokes over a 9-year period were the cases; eachserved as his or her own control. The exposure windows used were 1, 3, 7, and 14 days before thestroke. No significant differences in risk between visits to chiropractors and primary care providerswere evident, suggesting no role of spinal manipulation in triggering stroke.

Case-Time-Control Studies

This approach is a variant of the case-crossover design in which a traditional control group providesthe background exposure history. This type of study is defined as ‘a study in which exposure of casesand controls during one period is compared in matched-pair analyses to their own exposure duringanother period of similar length’.24 This approach is used in pharmacological studies to study briefexposures with acute effects, such as nonsteroidal antiinflammatory drugs and cardiac arrest outsideof a hospital.43 This approach helps to control for time trends in exposures.40

NONRANDOMISED TRIAL: PENULTIMATE DESIGN?

Some experimental trials do not randomly allocate participants to exposures (e.g., treatments orprevention strategies). Instead of using truly random techniques, investigators may use methodsthat fall short of the mark (e.g., alternate assignment).44 The US Preventive Services Task Force11

designates this research design as class II-1, indicating less scientific rigour than randomised trials,but more than analytical studies (see Panel 1.1).

After the investigators have assigned participants to treatment groups, the way a nonrandomisedtrial is done and analysed resembles that of a cohort study. The exposed and unexposed are followedforward in time to ascertain the frequency of outcomes. Advantages of a nonrandomised trialinclude use of a concurrent control group and uniform ascertainment of outcomes for both groups.However, selection bias can occur, because allocation concealment may be impossible to maintain.

RANDOMISED CONTROLLED TRIAL: GOLD STANDARD

The randomised controlled trial is the only known way to avoid selection and confounding biases inclinical research. This design approximates the controlled experiment of basic science. It resemblesthe cohort study in several respects, with the important exception of randomisation of participantsto exposures (see Fig. 1.2).

The hallmark of randomised controlled trials is assignment of participants to exposures (e.g.,treatments) purely by the play of chance.When properly implemented, random allocation precludesselection bias. Randomised controlled trials reduce the likelihood of bias in determination of out-comes. Trials feature uniform diagnostic criteria for outcomes and, often, blinding of treatmentassignment, thus reducing the potential for information bias. A unique strength of this study design

8 INTRODUCTION

is that it eliminates confounding bias, both known and unknown (discussed in Chapter 3). Trialstend to be statistically efficient. If properly designed and done, a randomised controlled trial is likelyto be free of bias and is thus especially useful for examination of small or moderate effects. In obser-vational studies, bias can easily account for weak associations (i.e., small differences).45

Randomised controlled trials have drawbacks, too. External validity is a concern. Whereas therandomised controlled trial, if properly done, has internal validity (i.e., it measures what it sets out tomeasure), it might not have external validity. This term indicates the extent to which results can begeneralised to the broader community.20 Unlike the observational study, the randomised controlledtrial includes only volunteers who pass through a screening process before inclusion. Those whovolunteer for trials tend to be different from the population of those with the condition; only a tinyfraction of those with a condition find their way into trials.46 Another limitation is that a rando-mised controlled trial cannot be used in some instances, because intentional exposure to harmfulsubstances (e.g., toxins, bacteria, or other noxious substances) would be unethical. As with cohortstudies, the randomised controlled trial can be prohibitively expensive. Indeed, the cost of largetrials can run into the hundreds of millions of US dollars.47

Measurement of OutcomesCONFUSING FRACTIONS

Identification and quantification of outcomes is the business of research. However, slippery termi-nology often complicates matters for investigators and readers alike. For example, the term ‘rate’ (asin ‘maternal mortality rate’) has been misused in textbooks and journal articles for decades. Addi-tionally, ‘rate’ is often used interchangeably with proportions and ratios.24 Fig. 1.4 presents a simpleapproach to classification of these common terms.

A ratio is a value obtained by dividing one number by another.24 These two numbers can beeither related or unrelated. This feature (i.e., relatedness of numerator and denominator) separatesratios into two groups: those in which the numerator is included in the denominator (e.g., rate andproportion) and those in which it is not.

A rate measures the frequency of an event in a population, usually over a period of time.As shown in Fig. 1.4, the numerator (those with the outcome) of a rate must be contained in

Ratio

Is numerator includedin denominator?

Measure: Rate Proportion Ratio

Example: Incidence rate Prevalence rate Maternal mortality ratio

Is time includedin denominator?

Yes No

Yes No

Fig. 1.4 Algorithm for distinguishing rates, proportions, and ratios.

91—AN OVERVIEW OF CLINICAL RESEARCH: THE LAY OF THE LAND

the denominator (those at risk of the outcome). Although all ratios feature a numerator and denom-inator, rates have two distinguishing characteristics: time and a multiplier. Rates indicate the timeduring which the outcomes occur and a multiplier, commonly to a base ten, to yield whole numbers.For example, the age-adjusted incidence of all unprovoked seizures in Minnesota is 61 per 100,000person-years. In contrast, the prevalence of epilepsy there is 6.8 per 1000 population.48 The lattermeasure is the total number of persons with a condition at a point in time divided by the populationat risk at that point in time (or midway in the time interval).24

Proportion is often used synonymously with rate, but the former does not have a time compo-nent. Like a rate, a proportion is a ratio with the numerator contained in the denominator.24

Because the numerator and denominator have the same units, these divide out, leaving a dimen-sionless quantity (a number without units). An example of a proportion is prevalence (e.g., 27 of 100at risk have hay fever). This number indicates howmany of a population at risk have a condition at aparticular time (here, 27%); as documentation of new cases over time is not involved, prevalence ismore properly considered a proportion than a rate.

Although all rates and proportions are ratios, the opposite is not true. In some ratios, the numer-ator is not included in the denominator. Themost notorious examplemay be the venerable maternalmortality ratio. The definition includes women who die of pregnancy-related causes in the numer-ator, and women with live births (usually 100,000) in the denominator. However, not all those inthe numerator are included in the denominator (e.g., a woman who dies of an ectopic pregnancy at7 weeks’ gestation cannot be in the denominator). Thus this ‘rate’ is actually a ratio, not a rate, a factstill not widely understood.49

MEASURES OF ASSOCIATION: RISKY BUSINESS

Relative risk (also termed the risk ratio) is another useful ratio: the frequency of outcome in theexposed divided by the frequency of outcome in the unexposed. If the frequency of the outcomeis the same in both groups, then the ratio is 1.0, indicating no association between exposureand outcome. By contrast, if the outcome is more frequent in those exposed, then the ratio willbe greater than 1.0, implying an increased risk associated with exposure. Conversely, if the fre-quency of disease is less among the exposed, then the relative risk will be less than 1.0, implyinga protective effect.

Also known as the cross-products ratio or relative odds,24 the odds ratio has different meaningsin different settings. In case-control studies, this ratio is the usual measure of association. It indi-cates the odds of exposure among the case group divided by the odds of exposure among controls.If cases and controls have equal odds of having the exposure, the odds ratio is 1.0, indicating noassociation. If the cases have a higher odds of exposure than the controls, then the ratio is greaterthan 1.0, implying an increased risk associated with exposure. Conversely, odds ratios less than 1.0indicate a protective effect.

An odds ratio can also be calculated for cross-sectional, cohort, and randomised controlled stud-ies. Here, the disease-odds ratio is the ratio of the odds in favour of disease in the exposed versus thatin the unexposed. In this context, the odds ratio has some appealing statistical features when studiesare aggregated in meta-analysis, but the odds ratio does not indicate the relative risk when the pro-portion with the outcome is greater than 5% to 10%.34 Stated alternatively, the odds ratio is always avalid measure of association, but it is not a good proxy for the relative risk unless the outcome isuncommon (‘rare disease assumption’).

The confidence interval (CI) reflects the precision of study results. The interval provides a rangeof plausible values for a point estimate, such as a proportion, relative risk, or odds ratio. The con-fidence interval has a specified probability of containing the true value for the entire population fromwhich the study sample was taken. Although 95% CIs are the most commonly used, others, such as90%, are sometimes seen.50 The wider the confidence interval, the less precision exists in the result,

10 INTRODUCTION

and vice versa. For relative risks and odds ratios, when the 95% CI does not include 1.0, the dif-ference is significant at the usual 0.05 level. However, use of this feature of confidence intervals as aback-door means of hypothesis testing is inappropriate.51

ConclusionUnderstanding what kind of study has been done is a prerequisite to thoughtful reading of research.Clinical research can be divided into experimental and observational; observational studies are fur-ther categorised into those with and without a comparison group. Only studies with comparisongroups allow investigators to assess possible causal associations, a fact often forgotten or ignored.Dichotomous outcomes of studies should be reported as measures of association with confidenceintervals; testing null hypotheses at arbitrary p values of 0.05 has no basis in medicine.

References

1. Goulet, F., Hudon, E., Gagnon, R., Gauvin, E., Lemire, F., Arsenault, I., 2013. Effects of continuing

professional development on clinical performance: results of a study involving family practitioners in Que-

bec. Can. Fam. Physician 59, 518–525.2. Decker, S.L., Jamoom, E.W., Sisk, J.E., 2012. Physicians in nonprimary care and small practices and those

age 55 and older lag in adopting electronic health record systems. Health Aff. (Millwood) 31, 1108–1114.3. Forsetlund, L., Bjorndal, A., Rashidian, A., et al., 2009. Continuing education meetings and workshops:

effects on professional practice and health care outcomes. Cochrane Database Syst. Rev. CD003030.

4. Windish, D.M., Huot, S.J., Green, M.L., 2007. Medicine residents’ understanding of the biostatistics andresults in the medical literature. JAMA 298, 1010–1022.

5. Rao, G., Kanter, S.L., 2010. Physician numeracy as the basis for an evidence-based medicine curriculum.

Acad. Med. 85, 1794–1799.6. Hadley, J.A., Wall, D., Khan, K.S., 2007. Learning needs analysis to guide teaching evidence-based med-

icine: knowledge and beliefs amongst trainees from various specialities. BMC Med. Educ. 7, 11.

7. Schulz, K.F., Altman,D.G.,Moher, D., 2010. CONSORT 2010 statement: updated guidelines for report-

ing parallel group randomised trials. BMJ 340, c332.

8. Moher, D., Hopewell, S., Schulz, K.F., et al., 2010. CONSORT 2010 explanation and elaboration:

updated guidelines for reporting parallel group randomised trials. J. Clin. Epidemiol. 63, e1–37.9. Martinez,M.E.,Marshall, J.R., Giovannucci, E., 2008. Diet and cancer prevention: the roles of observation

and experimentation. Nat. Rev. Cancer 8, 694–703.10. Albert, R.K., 2013. “Lies, damned lies…” and observational studies in comparative effectiveness research.

Am. J. Respir. Crit. Care Med. 187, 1173–1177.11. Harris, R.P., Helfand, M., Woolf, S.H., et al., 2001. Current methods of the US Preventive Services Task

Force: a review of the process. Am. J. Prev. Med. 20, 21–35.12. Wu, T., Li, Y., Bian, Z., Liu, G.,Moher, D., 2009. Randomized trials published in some Chinese journals:

how many are randomized? Trials 10, 46.

13. Grimes, D.A., 2009. “Case-control” confusion: mislabeled reports in obstetrics and gynecology journals.

Obstet. Gynecol. 114, 1284–1286.14. Hellems, M.A., Kramer, M.S., Hayden, G.F., 2006. Case-control confusion. Ambul. Pediatr. 6, 96–99.15. Caro, J.J., Huybrechts, K.F., 2009. Case-control studies in pharmacoeconomic research: an overview.

Pharmacoeconomics 27 (8), 627–634.16. RossmanS.Diet soda can increase risk of dementia and stroke, study finds. https://www.usatoday.com/story/

news/nation-now/2017/04/21/diet-soda-can-increase-risk-dementia-and-stroke-study-finds/100736786/,

accessed 23 April 2017.

17. Pase, M.P., Himali, J.J., Beiser, A.S., et al., 2017. Sugar- and artificially sweetened beverages and the risks

of incident stroke and dementia: a prospective cohort study. Stroke 48, 1139–1146.18. Meyer, A.M., Wheeler, S.B., Weinberger, M., Chen, R.C., Carpenter, W.R., 2014. An overview of

methods for comparative effectiveness research. Semin. Radiat. Oncol. 24, 5–13.19. DiPietro, N.A., 2010. Methods in epidemiology: observational study designs. Pharmacotherapy

30, 973–984.

111—AN OVERVIEW OF CLINICAL RESEARCH: THE LAY OF THE LAND

20. Stang, A., 2008. Appropriate epidemiologic methods as a prerequisite for valid study results. Eur. J. Epi-

demiol. 23, 761–765.21. Hill, C.L., LaValley, M.P., Felson, D.T., 2002. Discrepancy between published report and actual conduct

of randomized clinical trials. J. Clin. Epidemiol. 55, 783–786.22. Strech, D., Soltmann, B., Weikert, B., Bauer, M., Pfennig, A., 2011. Quality of reporting of randomized

controlled trials of pharmacologic treatment of bipolar disorders: a systematic review. J. Clin. Psychiatry

72, 1214–1221.23. Kuroki, L.M., Allsworth, J.E., Peipert, J.F., 2009. Methodology and analytic techniques used in clinical

research: associations with journal impact factor. Obstet. Gynecol. 114, 877–884.24. Porta, M., 2014. A Dictionary of Epidemiology. Oxford University Press, New York.

25. Hartung, D.M., Touchette, D., 2009. Overview of clinical research design. Am. J. Health Syst. Pharm.

66, 398–408.26. Goldberg, H.R., Allen, L., Kives, S., 2017. Fetiform teratoma in the ovary of a 7-year-old girl: a case report.

J. Pediatr. Adolesc. Gynecol. 30, 256–258.27. Pneumocystis pneumonia—Los Angeles, 1981. MMWR Morb. Mortal. Wkly Rep. 30, 250–252.28. Keys, T.F., 1977. A sporadic case of pneumonia due to legionnaires disease.Mayo Clin. Proc. 52, 657–660.29. McKenna, U.G., Meadows 3rd, J.A., Brewer, N.S., Wilson, W.R., Perrault, J., 1980. Toxic shock syn-

drome, a newly recognized disease entity. Report of 11 cases. Mayo Clin. Proc. 55, 663–672.30. Caillouette, J.C., Koehler, A.L., 1987. Phasic contraceptive pills and functional ovarian cysts. Am. J.

Obstet. Gynecol. 156, 1538–1542.31. Wei, E.K., Colditz, G.A., Giovannucci, E.L., et al., 2017. A comprehensive model of colorectal

cancer by risk factor status and subsite using data from the nurses’ health study. Am. J. Epidemiol.

185, 224–237.32. Iversen, L., Sivasubramaniam, S., Lee, A.J., Fielding, S., Hannaford, P.C., 2017. Lifetime cancer risk and

combined oral contraceptives: the Royal College of General Practitioners’ Oral Contraception Study.

Am. J. Obstet. Gynecol. 216, 580.e1–580.e9.33. Mayo, N.E., Goldberg,M.S., 2009.When is a case-control study not a case-control study? J. Rehabil.Med.

41, 209–216.34. Grimes, D.A., Schulz, K.F., 2008. Making sense of odds and odds ratios. Obstet. Gynecol. 111,

423–426.35. Rooney, R.M., Cramer, E.H., Mantha, S., et al., 2004. A review of outbreaks of foodborne disease asso-

ciated with passenger ships: evidence for risk management. Public Health Rep. 119, 427–434.36. Etminan, M., 2004. Pharmacoepidemiology II: the nested case-control study—a novel approach in phar-

macoepidemiologic research. Pharmacotherapy 24, 1105–1109.37. Brann, E., Papadopoulos, F., Fransson, E., et al., 2017. Inflammatory markers in late pregnancy in

association with postpartum depression—A nested case-control study. Psychoneuroendocrinology

79, 146–159.38. Checkoway, H., Pearce, N., Kriebel, D., 2007. Selecting appropriate study designs to address specific

research questions in occupational epidemiology. Occup. Environ. Med. 64, 633–638.39. Rhea, S., Weber, D.J., Poole, C., Cairns, C., 2014. Risk factors for hospitalization after dog bite injury: a

case-cohort study of emergency department visits. Acad. Emerg. Med. 21, 196–203.40. Schneeweiss, S., Sturmer, T., Maclure, M., 1997. Case-crossover and case-time-control designs as alter-

natives in pharmacoepidemiologic research. Pharmacoepidemiol. Drug Saf. 6 (Suppl 3), S51–S59.41. Lu, C.Y., 2009. Observational studies: a review of study designs, challenges and strategies to reduce

confounding. Int. J. Clin. Pract. 63, 691–697.42. Cassidy, J.D., Boyle, E., Cote, P., Hogg-Johnson, S., Bondy, S.J., Haldeman, S., 2017. Risk of carotid

stroke after chiropractic care: a population-based case-crossover study. J. Stroke Cerebrovasc. Dis.

26, 842–850.43. Sondergaard, K.B., Weeke, P., Wissenberg, M., et al., 2017. Non-steroidal anti-inflammatory drug use is

associated with increased risk of out-of-hospital cardiac arrest: a nationwide case-time-control study. Eur.

Heart J. Cardiovasc. Pharmacother. 3, 100–107.44. Sharma, P.K., Yadav, T.P., Gautam, R.K., Taneja, N., Satyanarayana, L., 2000. Erythromycin in pityriasis

rosea: A double-blind, placebo-controlled clinical trial. J. Am. Acad. Dermatol. 42, 241–244.45. Grimes, D.A., Schulz, K.F., 2012. False alarms and pseudo-epidemics: the limitations of observational epi-

demiology. Obstet. Gynecol. 120, 920–927.

12 INTRODUCTION

46. Rothwell, P.M., 2005. External validity of randomised controlled trials: “to whom do the results of this trial

apply?”. Lancet 365, 82–93.47. Roth, J.A., Etzioni, R., Waters, T.M., et al., 2014. Economic return from the Women’s Health Initiative

estrogen plus progestin clinical trial: a modeling study. Ann. Intern. Med. 160, 594–602.48. Logroscino, G., Hesdorffer, D.C., 2005. Methodologic issues in studies of mortality following epilepsy:

measures, types of studies, sources of cases, cohort effects, and competing risks. Epilepsia 46 (Suppl

11), 3–7.49. MacDorman, M.F., Declercq, E., Cabral, H., Morton, C., 2016. Recent increases in the U.S. maternal

mortality rate: disentangling trends from measurement issues. Obstet. Gynecol. 128, 447–455.50. Catton, C.N., Lukka, H., Gu, C.S., et al., 2017. Randomized trial of a hypofractionated radiation regimen

for the treatment of localized prostate cancer. J. Clin. Oncol. 35, 1884–1890.51. Sterne, J.A., Smith, G.D., 2001. Sifting the evidence—what’s wrong with significance tests? BMJ

322, 226–231.