measurements and validity - division on addiction · 2/4/2014  · common threats to internal...

Measurements and Validity

Julia Braverman, PhDDivision on Addictions

Types of measures

Michael John

ANXIETY??

Types of measurement

1. Objective/Physiological measures• Bodily activity, nervous system.• Response time

2. Observational measures• Direct observing participants.

3. Self-report• Participants provide information about

themselves.

Converging operations

Using several measurement approaches to measure a particular variable

Basics of psychometrics: How to build a trait/state assessment measure?

l Concept• Affect

• E.g. I feel sad• Behavior

• E.g. I cannot sleep, I cry a lot• Cognition

• E.g. I think about suicide.

l Question format (Likert scale, yes/no, reverse scale)

Measure quality

1. Reliability2. Validity

Reliability

l The degree of consistency between observations made by the same measurement tool.

Measurement Error. No measure is perfect.

l Observed score = True score + Measurement error.

l True score – is the score that the participant would have obtained if our measure were perfect.

Sources of measurement errors

1. Transient states• Mood, health, anxiety

2. Stable attributes• Suspicious participant may distort their answers

3. Situational factors• Weather outside, baseball game.

4. Characteristics of the measure• E.g. instruction ambiguity

5. Actual mistakes

Theoretical concept of reliability.

Systematic varianceReliability =

Total variance

0 < Reliability < 1

Assessing reliability

1. Test-retest reliability• Measuring the same thing twice.• Reliability = correlation ( r) between

results of the first and the second measurements.

• High reliability > .70

Assessing reliability

1. Test-retest reliability• Problems

•Memory•Experience

Assessing reliability

l Interitem Reliability - Measure of consistency among the items on a scale.1.Item-total correlation

qFor each item how it is correlated with the sum of other items. >. 30

2.Split-item reliabilityqDivide the items on the scale into 2 sets and test

the correlation (instead of test-retest).3.Cronbach’s alpha coefficient

qAverage of all possible split-half reliabilities.

Benevolent sexism scale: 1 (disagree) – 7 (agree)

1. Women should be cherished and protected by men.

2. Women, compared to men, tend to have a superior moral sensibility.

3. Men should be willing to sacrifice their own well-being in order to provide financially for the women in their lives.

4. Many women have a quality of purity that few men possess.

5. A good woman should be set on a pedestal by her man.

6. Men are complete without women.

Made-up table of item-total correlations

Item # r

1 .7

2 .5

3 .9

4 .6

5 .7

6 .4

Made-up table of item-total correlationsChronbach α = .85

Item # r Chronbach α(without the item)

1 .7 .8

2 .5 .8

3 .9 .7

4 .6 .8

5 .7 .8

6 .4 .9

Assessing reliability

l Interrater reliability – consistency between two or more raters or judges who observe the same behavior.• High reliability > .70

Increasing the Reliability Measures

1. Standardize administration of the measure• Same test conditions

2. Clarify instructions and questions. • To reduce ambiguity and misinterpretations.• Pretest questionnaires if possible.

3. Train observers.• To increase interrater reliability.

4. Minimize error in coding data.

Validity

l If the measurement actually measures what it is supposed to measure • Different from reliability• Same measure maybe valid for one purpose

and invalid for another one.

Assessing validity

1. Face validity – if a measure appears to be valid.• Does not mean actual validity. • E.g. SAT reading comprehension test

•Does it measure reading comprehension or common sense? (Katz et al., 1990)

• Affect motivation to participate?

Assessing validity

2. Construct validityn Relation to other measures.

§ Convergent validity§ High correlation with conceptually relevant

measures. n Discriminate validity

• Low correlation with conceptually unrelated constructs

Assessing validity

3. Criterion-Related validity – the correlation between the measure and some current behavior.• E.g. IQ and GPA• Doctor’s productivity

•Peer evaluation•Patient evaluation

Assessing validity

3. Predictive validity – the ability of a measure to predict a certain behavior/situation in a future. • E.g. SAT and GPA or GPA and after-college

salary. • Doctor’s productivity• ?

Test bias

l Test is biased if it is not equally valid for everyone who takes the test. • Groups with the same ability obtain different

scores on the test.

Reliability and Validity

l If reliable• May be valid or not.

l If not reliable • Not valid

Threats to measurement validity

l Using non-validated measures

l Solution• Validate the measure• Use pre-validated measures

Threats to measurement validity

l Loose connection between theory and method. • Disagreement between conceptional and

operational definitions. • E.g. putting more pepper as a measurement of

aggression?

• Solution• Validate your measure with previous

measurements

Threats to measurement validity

l Social desirability (evaluation apprehension) –Desire to look “normal” or to be judged favorably by another person (including the experimenter).

l Solutions• Anonymity• Ask indirect questions

• “How many drinks an average college student have during a party?”

Threats to measurement validity

l Yes-biasl Extreme-score bias

l Solution• Reverse score. • Z-transformation within an individual.

Threats to measurement validity

l Testing effects• Most participants perform better on a test of

personality/behavior/IQ measure the second time they take it.

l Reasons• Learning (e.g. IQ test)• Practice (e.g. physical skills)• Learn the test goal (e.g. personality test)• Attitude polarization

• Thinking about their attitudes

Threats to measurement validity

l Testing effectsl Solutions

• Control group• No pretest• Long waiting period

Validity of experiment

l Internal validity• Extent to which a study provides evidence of

a cause-effect relationship between the variables.

l External validity• The ability to generalize results of the

experiment.

Internal validity

l 3 conditions to determine causality• Covariation• Temporal sequence• No confounds

l Low internal validity – the conclusion that A affects B is wrong.

Threats to internal validity

l Role demands – participants’ expectations to what an experiment requires them to do• Good-subject tendency

• E.g. hypnosis and antisocial acts• Participants reactance

• E.g. What is the weather today?

Threats to internal validity

l Role demands

l Solution• Cover story

• E.g. Independent studies• Add non-relevant tasks, items (For

measurements)

Threats to internal validity

l Experimenter bias• E.g. Gratitude study

l Solution• Double-blind

Threats to internal validity

l Hawthorne effect – Increases in productivity that occur when participants know they are being studied. • Workers responded to any change in working

conditions by working harder than usual. l Solution

• Control group

Common Threats to Internal Validity of Quasi-experimentsl History

• Something occurred between the pretest and posttest.l Maturation

• Normal time changesl Regression to the mean

• If extreme scored Ss. were selected. l Pretest sensitization

• Pretest affects the posttest resultsl Selection bias

• Comparison groups differed from the beginningl Local historyl Contemporary historyl Attrition/mortality

• Only most motivated participants stay• Only participants who experience less adverse effects of treatment stay

External validity

l How well the findings of an experiment generalize to other situations or populations.

Threats to external validity

l Other subjects• Sampling/selection bias

l Other timesl Other settings

Threats to external validity

l Sampling bias• Motivated volunteers• Those available (at home, have phone)

Threats to external validity

l Other setting• Artificial experimental environment

External validity

l External validity - the ability to generalize results of the experiment.

l Tight control - highly specific and artificial situation -> less external validity.

Internal validity External validity

l You are a researcher. In your experiment, you assign the first 20 people in your study to the experimental condition and the second 20 people to your control condition. This could pose a threat to:• Internal validity• Reliability• External validity• Construct validity

l Saying that some measure is ________ definitely means it is also __________.• valid, reliable• reliable, valid• nominal, numerical• observational, self-report• none of the above

l An experimenter wants to examine if a new behavioral intervention program increases compliance among hypertension patients. For this purpose she recruits hypertension patients with low medication compliance and tests their compliance before and after the intervention. What are the potential threats to internal validity: • Regression to the mean• Maturation• History• Pretest sensitization• All of the above

Find a threat/threats to internal validityl The Alzheimers Center wants to evaluate the

effectiveness of their support groups for caregivers of individuals with Alzheimers Disease. The caregivers are given the choice when they first come to the center as to whether they want to join these support groups. The center gives a stress measure to the caregivers that attend these weekly meetings, once they have attended meetings for three months. They also administer the same stress measure to the caregivers who have not attended the support groups, as a control group. Both groups of caregivers are married to the person with Alzheimers disease and both groups have been involved with the center for the same length of time

Any questions?