• Power the ability to find a difference if there is one. – Effect size, alpha-level, sample size, (all increase

power) variance (decrease power)

PTP 560

• Research Methods

Week 6

Thomas Ruediger, PT

SamplingHow can we generalize our study to the world?

Our sample responses are representative of population!

• Population– All members– All measurements possible

• Sample– Subgroup of members– Measurements actually taken

• Bias– Conscious– Unconscious (sub?)

Sampling Target PopulationAccessible population Sample

• Inclusion Criteria– Trait of the Target or accessible population– Qualifies someone as a subject– Restrictions here will limit ability to generalize

• Exclusion Criteria– Precludes someone being a subject– Excluded because they may interfere with interpreting findings

• Selection– Plan – Fig 8.2

Sampling Techniques

• Probability– Everyone in the population has an equal chance of being

selected– Through random selection– Not the same as random assignment– Every member has equal chance of being selected– Considered (but not guaranteed to be) representative– Allows estimate of sampling error

The difference between population average and sample average

• Non-probability– Non-random methods– Limits ability to generalize outcomes

Probability Sampling

• Simple random– Also known as sampling without replacement– Table of random numbers (Table 8.1)

• Systematic – Every 2nd one, every other person, etc.

• Stratified random– Subsets (strata) established– Random selection from the strata– May also be proportional to amount in population.– May be more representative than random

• Grade in Class, Age, Fitness level.

Probability Sampling

• Disproportional– Select random samples of appropriate size– Correct it with proportional weighting

• Allows us to use a smaller sample size to project to the a larger population, through proportional weighting.

• Cluster– Successive random sampling– Convenient and efficient…………….BUT, increased sampling error

– Example• Area probability sampling• Random digit dialing

Non-Probability Sampling• Convenience– Also known as accidental sample– Consecutive sampling is common method– Self selection is a major limitation

• Quota– Enroll subjects (Ex: selecting by decade, fill up quota) – Stop for certain strata when they are represented

• Purposive– Hand picked by criteria– Prone to bias

• Snowball– Chain-referral: friends talk to friends

Recruitment• Feasibility issues can be daunting• Advertisements• Other healthcare providers/institutions• Track and report

– Screened for eligibility– Number actually eligible– Number enrolled

• POWER– “The ability to find significant differences when they exist”– Important to know a priori to get appropriate sample size

• Higher alpha level• Increase Sample Size• Estimate Effect Size, compared to chart (30 typically the best per group)

Validity in Experimental Design

• Experiment has three essential characteristics:1. Manipulation of independent variables

2. Random assignment to groups

3. Control or comparison group

Supports (Does NOT prove) cause-and effect relationship

-stronger design the better cuaseial reationship

• Extraneous variables– Must be controlled OR,– They can confound

`

Handling incomplete (or lost) data

• On-Protocol (Completer) Analysis– Only those who complete the study– Tends to bias in favor of the treatment

• As there is a reason why they dropped out.

• Intentions to treat (ITT) PREFERRED approach– What did we intend to do?– More conservative than On-Protocol– Considered to reflect clinical situations– Analysis?

• Non-completer equals failure• Last observation carried forward

– When they drop out, carry their score forward as if no change (more conservative)

Validity in Experimental Design

• Blinding– Single Blind

• Subject blinded to treatment or placebo

– Double Blind• Subject and Tester blinded to treatment or placebo

condition

– Triple Blind• Researcher, tester, and subject blinded• Data analyzed by independent source

Controlling Inter-subject Differences

• Options– Homogenize on certain characteristic(s)–Manipulate attribute variables into “Blocks”– Consider matching– Use subjects as own control– Handle statistically with ANCOVA

• Table 9.1

Threats to validityFour Threats correspond to four major questions

• Is there a relationship between IV and DV?– Statistical Conclusion Validity

• Evidence of causal relationship?– Internal Validity

• Can results be generalized to a theoretical construct?– Construct Validity

• Can it be generalized to other persons/settings/times?– External Validity

Statistical Conclusion Validity

• Is there a relationship between IV and DV?

• Threats– Low Statistical Power– Violated Assumptions of Statistical Tests– Error Rate– Reliability– Variance (to small)– Failure to use ITT

Internal Validity• Evidence of causal relationship?The extent to which the results of a study/experiment can be attributed

to the treatment or intervention rather than to flaws in the research design

• Threats to internal validity– History– Maturation– Attrition– Testing– Instrumentation– Regression– Social Threats

Internal Validity

• Threats to Internal Validity – Testing Interactions• Pre-tests or subsequent testing has an effect• Second test scores tends to move toward the mean• Standard Deviation decreases

Construct Validity

• Can results be generalized to a theoretical construct?

• Threats– Limits of Operational Definitions– Time Frame Within Operational Definitions– Multiple Treatment Interactions– Experimental Bias– Hawthorne Effect

External Validity

• Can results be generalized to other persons/settings/times?

• Threats– Interaction of treatment and selection– Interaction of treatment and setting– Interaction of treatment and history

Research Designs

• Common Sources of Error – Experimental Bias• Post Hoc Error

– Events that occur in sequence without cause & effect– Change related to coincidence; rival hypothesis

• Error of Misplaced Precision– Statistical significance not clinically important– Measuring blood pressure to the 0.001 mm Hg

Research Designs

• Common Sources of Error – Experimental Bias• “Typical” Case Studies

– Typically not typical – Typically IDEAL

• The Law of the Instrument– Always use the same instrument– Always calibrate the instrument

Experimental Bias

• Halo Effect– Irrelevant factors effect outcome favorably or unfavorably– Ex: Health care worker with a favorable/unfavorable

characteristic influences outcome of study

• Rating Errors– Over/Under/Central tendency raters

• Hawthorne Effect– 1920’s Hawthorne Plant of Western Electric– Productivity & efficiency– Factory that owners changed conditions for productivity

Experimental Bias

• “Self-Fulfilling Prophecy”– Find what researchers expect to find

• “John Henry” Effect–Control group discovers their status and

outperforms experimental group

• Placebo Effect– True effect of intervention versus ‘suggestibility’

EXAM• Types of Data• Application of the ICF Model-clinical purpose• Evidence Based Practice/ APTA Core Values• IV/DV, labels, levels, intervention• Single subject, multifactorial, etc know details about them level of strength in design (case report to RCT) • Hypotheses• Determinates of Power• Steps in Research process• Correlation/ Association=spearman rho,

– Inter vs. Intra– MDD, statistical difference– Reliability and Validity, Sources of Error

• No Research Articles• Shapes of Distribution, skews, • Levene’s Test, Mean’s difference, • Specify and Sensitivity, Likelihood Ratios

• Multiple Choice and Fill in the Blank (40 questions) Paper Exam 8am