Dealing with biasBoyce Sigweni

Martin Shepperd NB no cats were harmed in the making of this presentation!

What is bias?

“[L]et us define bias as the combination of various design, data, analysis, and presentation

factors that tend to produce research findings when they should not be produced.”

John Ioannidis (2005)

What is (the) scientific method?

• Guidelines, methods, procedures and evaluation criteria to reduce bias– reduce subjectivity, promote objectivity

• Other issues but bias is central

Blinding

Withholding information to reduce bias, e.g.,– blind reviewing– blind treatments to participants (placebo)– blind treatments to experimenters (double blind)– blind analysis

Blinding Hierarchy

• Open• Single blind• Double blind

• Complete blind (including the analyst)

Some work on blind analysis

Over to Boyce …

Bias harms randomisation

Random implies selection cannot be predicted in advance beyond the natural odds.

• Sample selection to estimate population parameters

• Assignment of experimental units / participants to treatments

• Choosing data sets

Cats and survival odds

• A study shows the odds slightly improve the further a cat falls.

• Used data supplied by vets in NY.

• Why?• What's the problem?

Do Finches help novices learn to code?

• Analysis by cohort• What's the problem?• Solutions?

Predicting software defects

• Same old …• But inference into all software?• Some data sets may favour some learners

Random doesn't mean unusual things can't happen!

• Consider stratified randomisation (reduce confounding)

• Type I and II errors can, and do, occur!

• Measurement error (random or biased?)

• Meta-analysis

Meta-analysis and bias

• Publication bias– file drawer problem– selective reporting of results– refereeing bias against 'negative' results

Problems with blinding

• Impractical• Unethical

• Sometimes can address by a crossover design