8 guidelines for critically evaluating a statistical study 1.identify the goal, population, and type...
TRANSCRIPT
8 Guidelines for Critically Evaluating a Statistical Study
1. Identify the Goal, Population, and Type of Study
2. Consider the Source
3. Examine the Sampling Method
4. Look for Problems in Defining or Measuring the Variables of Interest
5. Watch Out for Confounding Variables
6. Consider the Setting and Wording of Any Survey
7. Check That Results Are Fairly Represented in Graphics or Concluding Statements
8. Stand Back and Consider the Conclusions
Identify the Goal, Population, and Type of Study
The goal should be stated precisely; that is, who or what is being studied and exactly what it is we’d like to learn about it.
Population – complete set of people or things being studied.
3 Types of Studies – Observational, Experiment, Meta-Analysis
Consider the Source
Who is conducting the study and why?Watch for researcher biasPeer review is a process in which
several experts in a field evaluate a research report before the report is published.
Examine the Sampling Method
Simple Random SamplingSystematic SamplingCluster SamplingStratified SamplingConvenience SamplingMulti-Stage Sampling
BIAS
ResearcherSample
Selection bias - researchers select ther sample in a biased way
Participation bias – occurs any time participation in a study is voluntary
Self-selected or Voluntary Response survey
BIAS
DataExperimental Controls
Control GroupConfounding FactorsPlacebo EffectExperimenter EffectsBlindingCase-Control Studies
Watch Out for Confounding Variables
If, for example, subjects in one group are simultaneously tested in a room with the heat set at 70 degrees whereas subjects in another group are simultaneously tested in a nearby identically appointed room with the heat set at 60 degrees, the obtained differences in performance could be attributed to any of three factors. It could be due to the random assignment of subjects (i.e. to chance). It could be due to the different temperatures in the two rooms. It could, however, be due to some confounding factor such as differences in ambient illumination that result from unnoticed differences in the orientation of each room with respect to the sun. In any experiment an appropriate statistical test can help in the decision as to whether or not to attribute the results to chance, but only the most careful analysis of the actual conditions of the experiment can suggest whether or not the result might be due to a confounding factor.
Consider the Setting and Wording of Any Survey
Do you think Pat cheated on her test?Don’t you think Pat cheated on her test?
Would you say that traffic contributes more or less to air pollution than industry?
Would you say that industry contributes more or less to air pollution than traffic?
BIAS
Check That Results Are Fairly Represented in Graphics or Concluding Statements
Watch the scale on the axes of graphs
Stand Back and Consider the Conclusions
Did the study achieve its goals?Do the conclusions make sense?Can you rule out alternative explanations
for the results?If the conclusions make sense, do they
have any practical significance?