power and sample size
DESCRIPTION
Power and Sample Size. Adapted from: Boulder 2004 Benjamin Neale Shaun Purcell. I HAVE THE POWER!!!. Overview. Introduce Concept of Power via Correlation Coefficient (ρ) Example Discuss Factors Contributing to Power Practical: Simulating data as a means of computing power - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/1.jpg)
PowerPower and Sample Size and Sample Size
Adapted from:Adapted from:Boulder 2004Boulder 2004
Benjamin NealeBenjamin NealeShaun PurcellShaun Purcell
I HAVE THE
POWER!!!
![Page 2: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/2.jpg)
OverviewOverview
Introduce Concept of Power via Introduce Concept of Power via Correlation Coefficient (ρ) ExampleCorrelation Coefficient (ρ) Example
Discuss Factors Contributing to Discuss Factors Contributing to PowerPower
Practical:Practical:• Simulating data as a means of Simulating data as a means of
computing powercomputing power• Using Mx for Power CalculationsUsing Mx for Power Calculations
![Page 3: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/3.jpg)
3
Simple exampleSimple exampleInvestigate the linear relationship between Investigate the linear relationship between two random variables X and Y: two random variables X and Y: ρρ=0 vs. =0 vs. ρ≠ρ≠0 0
using the Pearson correlation coefficient.using the Pearson correlation coefficient.
Sample subjects at random from Sample subjects at random from populationpopulation Measure X andYMeasure X andY Calculate the measure of association Calculate the measure of association ρρ Test whether Test whether ρρ ≠≠ 0. 0.
![Page 4: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/4.jpg)
4
How to Test How to Test ρρ ≠≠ 00
Assume data are normally Assume data are normally distributeddistributed
Define a null-hypothesis (Define a null-hypothesis (ρρ = 0) = 0) Choose an Choose an αα level (usually .05) level (usually .05) Use the (null) distribution of the Use the (null) distribution of the
test statistic associated with test statistic associated with ρρ=0=0 t=t=ρρ √√ [(N-2)/(1- [(N-2)/(1-ρρ22)] )]
![Page 5: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/5.jpg)
5
How to Test How to Test ρρ ≠≠ 00
Sample N=40Sample N=40 r=.303, t=1.867, df=38, p=.06 r=.303, t=1.867, df=38, p=.06 αα
=.05=.05 Because observed p > Because observed p > αα, we fail , we fail
to reject to reject ρρ = 0 = 0
Have we drawn the correct Have we drawn the correct conclusion that p is genuinely conclusion that p is genuinely zero?zero?
![Page 6: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/6.jpg)
6
= type I error rate = type I error rate probability of deciding probability of deciding ρ≠ρ≠ 0 0
(while in truth (while in truth ρρ=0)=0)
ααis often chosen to is often chosen to equal .05...why?equal .05...why?
DOGMA
![Page 7: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/7.jpg)
7
N=40, r=0, nrep=1000, central N=40, r=0, nrep=1000, central t(38), t(38),
αα=0.05 (critical value 2.04)=0.05 (critical value 2.04)
![Page 8: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/8.jpg)
8
Observed non-null Observed non-null distribution (distribution (ρρ=.2) =.2)
and null distributionand null distribution
![Page 9: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/9.jpg)
9
In 23% of tests that In 23% of tests that ρρ=0, |t|=0, |t|>2.024 (>2.024 (αα=0.05), and thus =0.05), and thus correctly conclude that correctly conclude that ρρ 0. 0.
The probability of correctly The probability of correctly rejecting the null-hypothesis rejecting the null-hypothesis ((ρρ=0) is 1-=0) is 1-ββ, known as the power. , known as the power.
![Page 10: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/10.jpg)
Hypothesis TestingHypothesis Testing
Correlation Coefficient hypotheses:Correlation Coefficient hypotheses: hhoo (null hypothesis) is ρ=0 (null hypothesis) is ρ=0
hha a (alternative hypothesis) is ρ ≠ 0(alternative hypothesis) is ρ ≠ 0 Two-sided test, where ρ > 0 or ρ < 0 are Two-sided test, where ρ > 0 or ρ < 0 are
one-sidedone-sided
Null hypothesis usually assumes no Null hypothesis usually assumes no effecteffect
Alternative hypothesis is the idea Alternative hypothesis is the idea being testedbeing tested
![Page 11: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/11.jpg)
11
Summary of Possible Summary of Possible ResultsResults
H-0 trueH-0 true H-0 H-0 falsefalse
accept H-0accept H-0 1-1-ααββreject H-0reject H-0 αα 1-1-ββ
αα=type 1 error rate=type 1 error rate
ββ=type 2 error rate=type 2 error rate
1-1-ββ=statistical power=statistical power
![Page 12: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/12.jpg)
Rejection of H0 Non-rejection of H0
H0 true
HA true
STATISTICS
R E
A L I T
Y
Nonsignificant result(1- α)
Type II error at rate β
Significant result(1-β)
Type I error at rate α
![Page 13: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/13.jpg)
PowerPower The probability of rejecting the The probability of rejecting the
null-hypothesis depends on: null-hypothesis depends on: the significance criterion (the significance criterion (αα)) the sample size (N) the sample size (N) the effect size (NCP)the effect size (NCP)
“The probability of detecting a given effect size in a population from a sample of size N, using significance criterion α”
![Page 14: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/14.jpg)
P(T)
T
alpha 0.05
Sampling distribution if HA were true
Sampling distribution if H0 were true
β α
POWER = 1 - β
Standard Case
Effect Size (NCP)
![Page 15: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/15.jpg)
P(T)
T
alpha 0.1
Sampling distribution if HA were true
Sampling distribution if H0 were true
POWER = 1 - β ↑
Impact of less Impact of less conservative conservative α
β α
![Page 16: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/16.jpg)
P(T)
T
alpha 0.01
Sampling distribution if HA were true
Sampling distribution if H0 were true
POWER = 1 - β↓
Impact of more Impact of more conservative conservative α
β α
![Page 17: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/17.jpg)
P(T)
T
alpha 0.05
β α
Impact of increased sample size
Reduced variance of sampling distribution if HA is true
Sampling distribution if H0 is true
POWER = 1 - β↑
![Page 18: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/18.jpg)
P(T)
T
alpha 0.05
Sampling distribution if HA were true
Sampling distribution if H0 were true
β α
POWER = 1 - β↑
Impact of increase in Effect Size
Effect Size (NCP)↑
![Page 19: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/19.jpg)
Summary: Factors affecting Summary: Factors affecting powerpower
Effect SizeEffect Size Sample SizeSample Size Alpha LevelAlpha Level <Beware the False Positive!!!><Beware the False Positive!!!> Type of Data:Type of Data:
Binary, Ordinal, ContinuousBinary, Ordinal, Continuous Research DesignResearch Design
![Page 20: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/20.jpg)
Uses of power Uses of power calculationscalculations
Planning a studyPlanning a study
Possibly to reflect on ns trend resultPossibly to reflect on ns trend result
No need if significance is achievedNo need if significance is achieved
To determine chances of study To determine chances of study successsuccess
![Page 21: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/21.jpg)
Power Calculations via Power Calculations via SimulationSimulation
Simulate Data under theorized modelSimulate Data under theorized model
Calculate Statistics and Perform TestCalculate Statistics and Perform Test
Given α, how many tests p < αGiven α, how many tests p < α
Power = (#hits)/(#tests)Power = (#hits)/(#tests)
![Page 22: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/22.jpg)
Practical: Empirical Practical: Empirical Power 1Power 1
Simulate Data under a model onlineSimulate Data under a model online
Fit an ACE model, and test for CFit an ACE model, and test for C
Collate fit statistics on boardCollate fit statistics on board
![Page 23: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/23.jpg)
Practical: Empirical Practical: Empirical Power 2Power 2
First get First get http://www.vipbg.vcu.edu/neale/gen619/phttp://www.vipbg.vcu.edu/neale/gen619/power/power-raw.mx and put it into your ower/power-raw.mx and put it into your directorydirectory
Second, open this script in Mx, and note Second, open this script in Mx, and note both places where we must paste in the both places where we must paste in the datadata
Third, simulate data (see next slide)Third, simulate data (see next slide) Fourth, fit the ACE model and then fit the Fourth, fit the ACE model and then fit the
AE submodelAE submodel
![Page 24: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/24.jpg)
Practical: Empirical Practical: Empirical Power 3Power 3
Simulation ConditionsSimulation Conditions 30% A30% A22 20% C20% C22 50% E 50% E22
Input:Input: A 0.5477 C of 0.4472 E of 0.7071A 0.5477 C of 0.4472 E of 0.7071 350 MZ 350 DZ350 MZ 350 DZ Simulate and use “Space Delimited” Simulate and use “Space Delimited”
option atoption at http://statgen.iop.kcl.ac.uk/workshop/http://statgen.iop.kcl.ac.uk/workshop/
unisim.html or click unisim.html or click herehere in slide show mode in slide show mode Click submit after filling in the fields and you Click submit after filling in the fields and you
will get a page of datawill get a page of data
![Page 25: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/25.jpg)
Practical: Empirical Practical: Empirical Power 4Power 4
With the data page, use ctrl-a to select the With the data page, use ctrl-a to select the data, control-c to copy, switch to Mx (e.g. data, control-c to copy, switch to Mx (e.g. with alt-tab) and in Mx control-v to paste with alt-tab) and in Mx control-v to paste in both the MZ and DZ groups.in both the MZ and DZ groups.
Run the ace.mx script with the data Run the ace.mx script with the data pasted in and modify it to run the AE pasted in and modify it to run the AE model.model.
Report the -2log-likelihoods on the Report the -2log-likelihoods on the whiteboardwhiteboard
Optionally, keep a record of A, C, and E Optionally, keep a record of A, C, and E estimates of the first model, and the A and estimates of the first model, and the A and E estimates of the second model E estimates of the second model
![Page 26: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/26.jpg)
Simulation of other Simulation of other types of datatypes of data
Use SAS/R/Matlab/MathematicaUse SAS/R/Matlab/Mathematica
Any decent random number Any decent random number generator will dogenerator will do
See See http://www.vipbg.vcu.edu/~neale/gehttp://www.vipbg.vcu.edu/~neale/gen619/power/sim1.sasn619/power/sim1.sas
![Page 27: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/27.jpg)
27
RR
R is in your futureR is in your future Can do it manually with rnormCan do it manually with rnorm Easier to use mvrnormEasier to use mvrnorm
runmx at Matt Keller’s site:runmx at Matt Keller’s site: http://www.matthewckeller.com/html/mhttp://www.matthewckeller.com/html/m
x-r.htmlx-r.html27
library (MASS)mvrnorm(n=100,c(1,1),matrix(c(1,.5,.5,1),2,2),empirical=FALSE)
![Page 28: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/28.jpg)
Mathematica ExampleMathematica ExampleIn[32]:=(mu={1,2,3,4}; sigma={{1,1/2,1/3,1/4},{1/2,1/3,1/4,1/5},{1/3,1/4,1/5,1/6},{1/4,1/5,1/6, 1/7}}; Timing[Table[Random[MultinormalDistribution[mu,sigma]],{1000}]][[1]])
Out[32]=1.1 Second
In[33]:=Timing[RandomArray[MultinormalDistribution[mu,sigma],1000]][[1]]
Out[33]=0.04 Second
In[37]:=TableForm[RandomArray[MultinormalDistribution[mu,sigma],10]]
Obtain mathematica from VCU http://www.ts.vcu.edu/faq/stats/mathematica.html
![Page 29: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/29.jpg)
Theoretical Power Theoretical Power CalculationsCalculations
Based on Stats, rather than Based on Stats, rather than SimulationsSimulations
Can be calculated by hand Can be calculated by hand sometimes, but Mx does it for ussometimes, but Mx does it for us
Note that sample size and alpha-Note that sample size and alpha-level are the only things we can level are the only things we can change, but can assume different change, but can assume different effect sizeseffect sizes
Mx gives us the relative power levels Mx gives us the relative power levels at the alpha specified for different at the alpha specified for different sample sizessample sizes
![Page 30: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/30.jpg)
Theoretical Power Theoretical Power CalculationsCalculations
We will use the power.mx script to We will use the power.mx script to look at the sample size necessary for look at the sample size necessary for different power levelsdifferent power levels
In Mx, power calculations can be In Mx, power calculations can be computed in 2 ways:computed in 2 ways: Using Covariance Matrices (We Do This Using Covariance Matrices (We Do This
One)One) Requiring an initial dataset to generate Requiring an initial dataset to generate
a likelihood so that we can use a chi-a likelihood so that we can use a chi-square testsquare test
![Page 31: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/31.jpg)
Power.mx 1Power.mx 1! Simulate the data! Simulate the data
! ! 30% additive genetic30% additive genetic
! ! 20% common environment20% common environment
! ! 50% nonshared environment50% nonshared environment
#NGroups 3#NGroups 3
G1: model parametersG1: model parameters
CalculationCalculation
Begin Matrices;Begin Matrices;
X lower 1 1 fixedX lower 1 1 fixed
Y lower 1 1 fixedY lower 1 1 fixed
Z lower 1 1 fixedZ lower 1 1 fixed
End Matrices;End Matrices;
Matrix X 0.5477Matrix X 0.5477
Matrix Y 0.4472Matrix Y 0.4472
Matrix Z 0.7071Matrix Z 0.7071
Begin Algebra;Begin Algebra;
A = X*X' ; A = X*X' ;
C = Y*Y' ;C = Y*Y' ;
E = Z*Z' ;E = Z*Z' ;
End Algebra;End Algebra;
EndEnd
![Page 32: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/32.jpg)
Power.mx 2Power.mx 2G2: MZ twin pairsG2: MZ twin pairs
CalculationCalculation
Matrices = Group 1Matrices = Group 1
Covariances A+C+ECovariances A+C+E || A+C _A+C _
A+CA+C || A+C+E /A+C+E /
Options MX%E=mzsim.covOptions MX%E=mzsim.cov
EndEnd
G3: DZ twin pairsG3: DZ twin pairs
CalculationCalculation
Matrices = Group 1Matrices = Group 1
H Full 1 1 H Full 1 1
Covariances A+C+ECovariances A+C+E || H@A+C _H@A+C _
H@A+CH@A+C || A+C+E /A+C+E /
Matrix H 0.5Matrix H 0.5
Options MX%E=dzsim.covOptions MX%E=dzsim.cov
EndEnd
![Page 33: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/33.jpg)
Power.mx 3Power.mx 3! Second part of script! Second part of script
! Fit the wrong model to the simulated data ! Fit the wrong model to the simulated data
! to calculate power! to calculate power
#NGroups 3#NGroups 3
G1 : model parametersG1 : model parameters
CalculationCalculation
Begin Matrices;Begin Matrices;
X lower 1 1 freeX lower 1 1 free
Y lower 1 1 fixedY lower 1 1 fixed
Z lower 1 1 freeZ lower 1 1 free
End Matrices;End Matrices;
Begin Algebra;Begin Algebra;
A = X*X' ;A = X*X' ;
C = Y*Y' ; C = Y*Y' ;
E = Z*Z' ; E = Z*Z' ;
End Algebra;End Algebra;
EndEnd
![Page 34: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/34.jpg)
Power.mx 4Power.mx 4G2 : MZ twinsG2 : MZ twins
Data NInput_vars=2 NObservations=350Data NInput_vars=2 NObservations=350
CMatrix Full File=mzsim.covCMatrix Full File=mzsim.cov
Matrices= Group 1Matrices= Group 1
Covariances A+C+ECovariances A+C+E || A+C _A+C _
A+C A+C | | A+C+E /A+C+E /
Option RSidualsOption RSiduals
EndEnd
G3 : DZ twinsG3 : DZ twins
Data NInput_vars=2 NObservations=350Data NInput_vars=2 NObservations=350
CMatrix Full File=dzsim.covCMatrix Full File=dzsim.cov
Matrices= Group 1Matrices= Group 1
H Full 1 1H Full 1 1
Covariances A+C+ECovariances A+C+E || H@A+C _H@A+C _
H@A+C H@A+C | | A+C+E /A+C+E /
Matix H 0.5Matix H 0.5
Option RSidualsOption RSiduals
! Power for alpha = 0.05 and 1 df! Power for alpha = 0.05 and 1 df
Option Power= 0.05,1 Option Power= 0.05,1
EndEnd
![Page 35: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/35.jpg)
35
Model IdentificationModel Identification
Necessary ConditionsNecessary Conditions
Sufficient ConditionsSufficient Conditions
Algebraic TestsAlgebraic Tests
Empirical TestsEmpirical Tests
35
![Page 36: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/36.jpg)
36
Necessary ConditionsNecessary Conditions
Number of Parameters < or = Number of Parameters < or = Number of StatisticsNumber of Statistics
Structural Equation Model usually Structural Equation Model usually count variances & covariances to count variances & covariances to identify variance componentsidentify variance components
What is the number of What is the number of statistics/parameters in a univariate statistics/parameters in a univariate ACE model? Bivariate?ACE model? Bivariate? 3
6
![Page 37: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/37.jpg)
37
Sufficient ConditionsSufficient Conditions
No general sufficient conditions for No general sufficient conditions for SEMSEM
Special case: ACE modelSpecial case: ACE model Distinct Statistics (i.e. have different Distinct Statistics (i.e. have different
predicted valuespredicted values VP = a2 + c2 + e2VP = a2 + c2 + e2 CMZ = a2 + c2CMZ = a2 + c2 CDZ = .5 a2 + c2CDZ = .5 a2 + c2
37
![Page 38: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/38.jpg)
38
Sufficient Conditions 2Sufficient Conditions 2 Arrange in matrix formArrange in matrix form
1 1 1 a2 VP1 1 1 a2 VP 1 1 0 c2 = CMZ1 1 0 c2 = CMZ .5 1 0 e2 CDZ.5 1 0 e2 CDZ
A x = bA x = b
If A can be inverted then can find AIf A can be inverted then can find A--
11bb 38
![Page 39: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/39.jpg)
39
Sufficient Conditions 3Sufficient Conditions 3
39
Solve ACE modelCalc ng=1Begin Matrices; A full 3 3 b full 3 1End Matrices;Matrix A1 1 11 1 0.5 1 0Labels Col A A C ELabels Row A VP CMZ CDZMatrix b ! Data, essentially1.8.5Labels Col B StatisticLabels Row B VP CMZ CDZBegin Algebra; C = A~; x = A~*b;End Algebra;Labels Row x A C EEnd
![Page 40: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/40.jpg)
40
Sufficient Conditions 4Sufficient Conditions 4
What if not soluble by inversion?What if not soluble by inversion? Empirical:Empirical:
1 Pick set of parameter values T1 Pick set of parameter values T11
2 Simulate data2 Simulate data 3 Fit model to data starting at T3 Fit model to data starting at T22 (not T (not T11)) 4 Repeat and look for solutions to step 3 4 Repeat and look for solutions to step 3
that are perfect but have estimates not that are perfect but have estimates not equal to Tequal to T11
If equally good solution but different If equally good solution but different values, reject identified model values, reject identified model hypothesis hypothesis
40
![Page 41: Power and Sample Size](https://reader036.vdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c5f/html5/thumbnails/41.jpg)
ConclusionConclusion
Power calculations relatively simple Power calculations relatively simple to doto do
Curse of dimensionalityCurse of dimensionality Different for raw vs summary Different for raw vs summary
statisticsstatistics Simulation can be done many waysSimulation can be done many ways No substitute for research designNo substitute for research design