8/10/2019 Spotting Bad Science

http://slidepdf.com/reader/full/spotting-bad-science 1/1

BAD SCIENCE

A ROUGH GUIDE TO SPOTTING

Aa

1. SENSATIONALISED HEADLINESHeadlines of articles are commonly designed toentice viewers into clicking on and reading thearticle. At best, they over-simplify the ndings ofresearch. At worst, they sensationalise and mis-represent them.

2. MISINTERPRETED RESULTSNews articles sometimes distort or misinterpretthe ndings of research for the sake of a goodstory, intentionally or otherwise. If possible, tryto read the original research, rather than relyingon the article based on it for information.

3. CONFLICT OF INTERESTSMany companies employ scientists to carryout and publish research - whilst this does not

necessarily invalidate research, it should beanalysed with this in mind. Research can also bemisrepresented for personal or nancial gain.

4. CORRELATION & CAUSATIONBe wary of confusion of correlation & causation.Correlation between two variables doesn’tautomatically mean one causes the other. Globalwarming has increased since the 1800s, andpirate numbers decreased, but lack of piratesdoesn’t cause global warming.

5. SPECULATIVE LANGUAGESpeculations from research are just that -speculation. Be on the look out for wordssuch as ‘may’, ‘could’, ‘might’, and others, as itis unlikely the research provides hard evidencefor any conclusions they precede.

6. SAMPLE SIZE TOO SMALLIn trials, the smaller a sample size, the lowerthe condence in the results from that sample.Conclusions drawn should be considered withthis in mind, though in some cases small samplesare unavoidable. It may be cause for suspicion ifa large sample was possible but avoided.

7. UNREPRESENTATIVE SAMPLESIn human trials, researchers will try to selectindividuals that are representative of a largerpopulation. If the sample is dierent from thepopulation as a whole, then the conclusionsmay well also be dierent.

8. NO CONTROL GROUP USEDIn clinical trials, results from test subjects shouldbe compared to a ‘control group’ not given thesubstance being tested. Groups should also beallocated randomly. In general experiments, acontrol test should be used where all variablesare controlled.

9. NO BLIND TESTING USEDTo prevent any bias, subjects should not know ifthey are in the test or the control group. In double-

blind testing, even researchers don’t know whichgroup subjects are in until after testing. Note,blind testing isn’t always feasible, or ethical.

10. ‘CHERRY-PICKED’ RESULTSThis involves selecting data from experimentswhich supports the conclusion of the research,whilst ignoring those that do not. If a researchpaper draws conclusions from a selection of itsresults, not all, it may be cherry-picking.

11. UNREPLICABLE RESULTSResults should be replicable by independentresearch, and tested over a wide range ofconditions (where possible) to ensure they aregeneralisable. Extraordinary claims requireextraordinary evidence - that is, much more thanone independent study!

12. JOURNALS & CITATIONSResearch published to major journals will haveundergone a review process, but can still beawed, so should still be evaluated with thesepoints in mind. Similarly, large numbers ofcitations do not always indicate that research ishighly regarded.

???

2014 COMPOUND INTEREST - WWW.COMPOUNDCHEM.COMC