1

Chapter 14

External Validity of Research

2

External Validity

• Concept of External Validity• Structural Component• Functional & Conceptual Components• Assessing External Validity• Lab Research, Natural Setting & External

Validity• Summary

3

External Validity:Concept

• Do results generalize “across”:– Other settings– Populations– Times

• Do results generalize “to”:– Particular setting– Particular population

• ? Which is more important to– Basic researchers?

• Across

– Applied researchers?• To

– ? Why?• Give examples of limits to from your project

4

External validity:concepts

• Boundary conditions?– Limitations of generalization

• Situations where the theory doesn’t apply

• E.g. Newtonian physics in outer space

• “Reinforcement theory” when behavior is constrained

– ? What are some boundary conditions in your study?

• Ecological validity?– Similarity of conditions to the natural setting

• Substance abuse in half-way house. v. living at home

• Training offsite v. on the job

5

External Validity(Whitley’s view)

• Generalizing Across – “generalizability”

• Generalizing to– “ecological validity”

6

External Validity Components:

• Structural– Methods, how its carried out

• Setting, procedures, sample

• Functional – How it operates; works

• Similarity of psych process in study and natural setting

• Conceptual– How important in real world

• Faking on personality tests used for applicant selection

• Marihuana use at work

7

External Validity:Components (Vidmar, ’79)

• Structural – Method– Procedure– Sample– Setting

• Example: – Does student participation in GS -> task satisfaction?

• Generalize to other students?

• To workers in a factory?

8

External ValidityComponents (con’t) (Vidmar, 79)

• Functional (primarily ecological validity)

– How close do the psychological processes in study match natural setting?

– E.g. mock jury v. real jury

• Conceptual (primarily ecological validity)

– How close does the problem in study match the natural setting?

– E.g. crowding in lab study = crowding in prison?

9

Structural Components:Generalize findings to other

• Settings• Participant samples• Research procedures• Times • Cultures

• ? How are the structural components of your study related to external validity?

10

Structural Component

• Settings (Physical & Social)– Physical (e.g. Wickland)– Reactivity (behavior changes when being observed)– Research attributes (of E)– Co-participant attributes (when they interact)– Ecological validity (mundane realism)

• What factors in your project could interact with IVs?

11

Structural Component

• Participant sample factors– Convenience sample

• availability v. representative

– Restricted sampling• E.g. Young, white, lesbian female postal carrier

– Volunteer participants• Would you “devolunteer”?

– Person by person situation interactions• Personal characteristics may interact with IV

– Ecological validity• Are college students representative of…?

12

Research Procedure Factors

• Artificiality• Operational Definitions (use multiple op defs)• Levels of IV (what is the range of values?)• Ecological Validity (Too many college Ss?)• Cultural Factors (Cross cultural factors)• Time Factors

– (how many psychotherapy sessions are optimal?)

13

Functional and Conceptual Componentsverisimilitude = appearance of truth

• Functional

– Do the psychological processes act the same way in other settings, populations?

– E.g. when the stakes are high in jury decisions v. mock juries?

• Conceptual

– E.g. “sub-clinical” depression in college Ss = form of depression in clinical population? (Kazdin, ’97)

– Researchers v. policy makers

• Estimator vars cannot be changed (e.g. level of victim anxiety)

• Policy vars can (e.g. time between crime and testimony)

– Utility of DV

• E.g. actual prisoner violence v. perceptions of control (theoretical interest)

14

Relationships Among Components

• Components of structural, functional, conceptual are interrelated

– Acute v. chronic manipulations• Related to both functional and conceptual

– E.g. are long and short term anxiety the same?

15

Assessing External Validity

• Assessing Generalizability– 1. include generalizability factors

• E.g. moderators such as age, gender, race

– 2. replicate findings– 3. acknowledge limitations of study

• ? What are some of yours?

• Assessing Ecological Validity – consider:– Conceptual (policy relevance of IV; utility of DV)– Functional (similar psychological processes?)– Structural (naturalism?)

16

Lab v. natural Settings & Ecological Validity

• Lab research & Ecological Validity– Defending lab research (Purposes are different)– Lab research

• Tests causal hypotheses (Berkowitz 7 Donnerstein, ’82)– “ A behavioral phenomenon reliably deonstrated is a genuine

phenomenon” • Falsify theoretical porpositions (Higgens & Marlatt, ’73)

– Alcoholics don’t drink to reduce tension• Dissect complex phenomena (Haney, Banks, & Zimbardo, ’73)

– Role assignment alone can determine behaviors• Discover new phenomena (Henshel, ’80)

– Would ecological research have discovered that biofeedback works?

17

Lab v. Natural Settings (con’t)

• Ecological Validity as Empirical Question

– Can be tested

– (e.g. meta of meta-analysis shows effect size r=.73)

• Natural Settings and Generalizability

– No guarantee of generalizability for natural setting research

• Structural v. Functional & Conceptual Verisimilitude– Ecological focuses on structural, ignores functional/conceptual (Locke, ’86)

• Analog Research– Reproducing natural setting in lab

• What most of you would like to do with your manipulation

• Philosophies of Science and Naturalism in Research

– A matter of preference! Basic or Applied?

– What’s in your wallet?

18

External Validity & Internal Valitidy

• It’s a trade off, but…– Internal is more important– Must rule out other plausible causes to draw

conclustions• E.g. which diet is best? Carbs or Fat?

• Who really knows?

• ?Which is more important for your project?

19

External Validity: Summary

• Concept of External Validity• Structural Component• Functional & Conceptual Components• Assessing External Validity• Lab Research, Natural Setting & External

Validity