the analysis of population-based survey experiments
DESCRIPTION
The Analysis of Population-Based Survey Experiments. Diana C. Mutz University of Pennsylvania. Analysis of Experiments. Simple, straightforward No fancy statistical techniques required Very few questions required Comparison of means (analysis of variance) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/1.jpg)
Diana C. MutzUniversity of Pennsylvania
![Page 2: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/2.jpg)
Simple, straightforward No fancy statistical techniques required Very few questions required Comparison of means (analysis of
variance) Many problems result from using
observational analysis techniques on experimental data
People make it more complicated than it needs to be!
![Page 3: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/3.jpg)
1. Well measured Dependent Variable(s)2. Manipulation check (to ensure that the
Independent Variable was successfully manipulated by the experimental treatment)
![Page 4: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/4.jpg)
Causality requires meeting only 3 conditions:
1. Association (The easy part!)2. Precedence in Time of Independent Variable
(We manipulate the Independent Variable)3. Non-spuriousness of relationship
(Random assignment eliminates this problem)
![Page 5: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/5.jpg)
1. Well measured Dependent Variable2. Manipulation checks (to ensure that the
Independent Variable was successfully manipulated by the experimental treatment)
OPTIONAL:1. Potential Moderators/Contingent
conditions2. Covariates
![Page 6: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/6.jpg)
Does Social Trust Influence Willingness to Engage in Online Economic Transactions?
CONTROLCONDITION
POSITIVESOCIALTRUST
NEGATIVESOCIALTRUST
![Page 7: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/7.jpg)
1. Randomization checks/Balance tests2. Statistical models for analysis3. Weighting data to population parameters4. Use and misuse of covariates
![Page 8: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/8.jpg)
Randomization checks/balance tests: They can’t tell us what we want to know, and they can lead to inferior model choices
Statistical models for analyzing population-based survey experiments often altogether ignore the fact that they are, indeed, experiments.
![Page 9: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/9.jpg)
We assume…. Researcher has control over
assignment to conditions Respondents do not undergo attrition
differentially as a result of assignment to a specific experimental condition
Researcher can ensure that those assigned to a given treatment are, in fact, exposed to treatment.
![Page 10: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/10.jpg)
If any one of those 3 requirements is not met, then balance tests can make sense
If the randomization mechanism requires pretesting, then balance tests make sense
Otherwise, not.
![Page 11: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/11.jpg)
Rationales for balance tests
Credibility of findings Efficiency of analyses
![Page 12: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/12.jpg)
Lack of faith in or thorough understanding of probability theory
Confusion between frequentist and Bayesian paradigms
Mistakenly applying methods for observational analyses to experimental results
Field experimental literature in which exposure to treatment cannot always be controlled
![Page 13: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/13.jpg)
What does it mean for a randomization to “succeed”?
A well-executed random assignment to experimental conditions does not promise to make experimental groups equal on all possible characteristics, or even a specified subset of them.
![Page 14: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/14.jpg)
“Because the null hypothesis here is that the samples were randomly drawn from the same population, it is true by definition, and needs no data.” (Abelson)
Randomization checks are “philosophically unsound, of no practical value, and potentially misleading.” (Senn)
“Any other purpose [than to test the randomization mechanism] for conducting such a test is fallacious.” (Imai et al.)
![Page 15: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/15.jpg)
“p<.05” already includes the probability that randomization might have produced an unlikely result
Thus experimental findings are credible without any balance tests at all.
![Page 16: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/16.jpg)
Can balance tests profitably inform the analyses of results?
What should one do if a balance test fails?
![Page 17: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/17.jpg)
Inclusion of covariates Post-stratification Re-randomization
![Page 18: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/18.jpg)
Is a failed balance test useful for purposes of choosing covariates?
Covariates should be chosen in advance, not based on the data.
Covariates are chosen for anticipated relationship with the DV; balance tests evaluate the relationship with the IV.
So is a balance test informative for model selection?
![Page 19: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/19.jpg)
NO! If inclusion of a variable as a covariate
in the model will increase the efficiency of an analysis, then it would have done so, and to a slightly greater extent, had it not failed the balance test.
Thus balance tests are uninformative when it comes to the selection of covariates.
![Page 20: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/20.jpg)
“Failed” randomization with respect to a covariate should not lead a researcher to include that covariate in the model. If the researcher plans to include a covariate for the sake of efficiency, it should be included in the model regardless of the outcome of a balance test.
![Page 21: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/21.jpg)
Changes the appropriate p-value Always excludes X: p1
Always includes X: p2
Not the same p-value that should result after the 2-stage process
But most researchers simply report p1 or p2
![Page 22: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/22.jpg)
If they have no implications for the credibility of our findings…
If they cannot improve the efficiency of our analyses…
![Page 23: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/23.jpg)
They can’t tell us what we want to know They can lead to inferior model choices They can lead to unjustified changes in
the interpretation of findings
![Page 24: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/24.jpg)
Balance tests do not provide rationales for including additional variables
Three examples of model and analysis choices made for the wrong reasons
![Page 25: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/25.jpg)
EXAMPLE 1: “In order to ensure that the experimental conditions were randomly distributed—thus establishing the internal validity of our experiment—we performed difference of means tests on the demographic composition of the subjects assigned to each of the three experimental conditions.”
![Page 26: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/26.jpg)
“Having established the random assignment of experimental conditions, regression analysis of our data is not required; we need only perform an analysis of variance (ANOVA) to test our hypotheses as the control variables that would be employed in a regression were randomly distributed between the three experimental conditions.”
![Page 27: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/27.jpg)
Regressions run amok with survey-experimental findings!
Five dummies for 6 conditions
EXAMPLE 2:
![Page 28: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/28.jpg)
Regression versus analysis of variance is a red herring. So are balance tests.
Especially in an experimental analysis, everything needs a reason for being there.
True experiments should not have “control” variables! (A few covariates are OK.)
The presence of unnecessary variables in a statistical model should be viewed with suspicion; they can hurt and bias results.
![Page 29: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/29.jpg)
1. Randomization checks/Balance tests2. Statistical models for analysis3. Weighting data to population parameters4. Use and misuse of covariates
![Page 30: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/30.jpg)
Should population-based experiments use population weights supplied by survey houses?
Some studies do, some don’t; no particular rationale typically given
No one correct answer but need to consider: Possibility of heterogeneous effects Power needs Emphasis on generalizability
![Page 31: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/31.jpg)
1. No use of weights2. Weighting sample as a whole to underlying
population parameters3. Weighting formulated so that individual
experimental conditions reflect population parameters
Either (1) or (2) benefits through increasing generalizability to full population; (2) is better at reducing noise due to uneven randomization
But all weighting sacrifices power .
![Page 32: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/32.jpg)
If all the full sample weights are squared for a sample of size n, and then summed across all subjects, this sum (call it M1) provides a sense of just how much power is lost through weighting:
If M1 =3000 and n=2000, then the equation will come out to .33.
![Page 33: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/33.jpg)
Weighting in this example lowers power as if we had reduced the sample size by one-third. Instead of a sample of 2000, we effectively have the power of a sample size of 1340.
![Page 34: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/34.jpg)
Calculate via same formula for within-subject
Compare loss of power in within versus whole sample weighting
![Page 35: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/35.jpg)
Request both whole sample and within-condition weights
Decision can be made on basis of importance of power relative to generalizability
Ultimately depends on expectations about heterogeneity of effects.
![Page 36: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/36.jpg)
1. Randomization checks/Balance tests2. Statistical models for analysis3. Weighting data to population parameters4. Use and misuse of covariates
![Page 37: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/37.jpg)
Because population-based survey experiments involve survey data, often analyzed as if they were observational studies
Mistaken use of unnecessary “control” variables
![Page 38: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/38.jpg)
Because population-based survey experiments involve survey data, often analyzed as if they were observational studies
Mistaken use of unnecessary “control” variables
Not a cure for an unlucky randomization (which isn’t necessary in any case)
But what’s the harm? Biased results
![Page 39: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/39.jpg)
Treatment effects and their interactions with other variables
EXAMPLE 3:
![Page 40: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/40.jpg)
But then what are these? “Control variables”
Treatment effects and their interactions with other variables
![Page 41: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/41.jpg)
To improve efficiency when selected in advance from pretest measures based on advance knowledge of predictors of dependent variable
Better yet, use blocking if equality across conditions on that particular variable is THAT important.
![Page 42: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/42.jpg)
Too many available variables leads to sub-optimal data analysis practices.
Researchers need to rely more on the elegance and simplicity of their experimental designs.
Equations chock full of “control” variables demonstrate a fundamental misunderstanding of how experiments work.
Failed randomization checks should never be used as a rationale for inclusion of a particular covariate.
![Page 43: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/43.jpg)
![Page 44: The Analysis of Population-Based Survey Experiments](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813952550346895da0f224/html5/thumbnails/44.jpg)