chapter 2 personality assessment, measurement, and research design © 2015 m. guthrie yarwood
TRANSCRIPT
© 2015 M. Guthrie Yarwood
CHAPTER 2
Personality Assessment, Measurement, andResearch Design
© 2015 M. Guthrie Yarwood
Outline
I. 4 Sources of personality data
II. Reliability / Validity in Personality
III. A little more on observer reports
© 2015 M. Guthrie Yarwood
I. Sources of Personality Data
Self-Report Data (S-Data)Observer-Report Data (O-Data) Test-Data (T-Data) Life-Outcome Data (L-Data)
© 2015 M. Guthrie Yarwood
Self-Report Data (S-Data)
Information provided by a person, such as through a survey or interview
Limitations of S-data?
© 2015 M. Guthrie Yarwood
O-Data
Information provided by someone else about another person Professional personality assessors People who actually know the target person
Naturalistic vs. Artificial Observation
Limitations?
© 2015 M. Guthrie Yarwood
Test-Data (T-Data)
Information provided by standardized tests or testing situations
Situation designed to elicit behaviors that serve as indicators of personality
© 2015 M. Guthrie Yarwood
Test-Data Creativity Example
What are unusual uses for common objects – bricks, knives, newspapers?
Answers to hypothetical events What would happen if people went blind? What would happen is people shrank to 12 inches tall?
(Paul Silvia’s work)
© 2015 M. Guthrie Yarwood
Test-Data: Other Examples
Mechanical recording devices, e.g., “Actometer” used to assess children’s activity
Physiological data
Projective Tests
Ex: Fairy Tale Test
Limitations?
© 2015 M. Guthrie Yarwood
Life-Outcome Data (L-Data)
Information that can be gleaned from events, activities, and outcomes in a person’s life that is available for public scrutiny—e.g., marriage, speeding tickets
Can serve as important source of “real life” information about personality
Ex: implicit egotism: people gravitate toward people, places, things that resemble the self
(Pelham and colleagues’ work)
© 2015 M. Guthrie Yarwood
Issues in Personality Assessment
Links among different data sourcesFallibility of personality measurement
All sources of data have limitations Results that replicate through “triangulation” are
most powerful
© 2015 M. Guthrie Yarwood
You are a personality psychologist and would like to measure the personality trait risk-taking (i.e., sensation seeking).
How could you measure risk-taking using each of the four data sources?S-DataO-DataT-DataL-Data
© 2015 M. Guthrie Yarwood
Evaluation of Personality Measures How do we know whether our scale is a “good scale?”
Types of ErrorsReliability ValidityThreats to Reliability and Validity
© 2015 M. Guthrie Yarwood
Extraneous vs. Confounding Variables
Extraneous variables are variables that may compete with the independent variable in explaining the outcome of a study.
Confounding variable: an extraneous variable that does indeed influence the dependent variable.
A confounding variable systematically varies or influences the independent variable and also influences the dependent variable.
© 2015 M. Guthrie Yarwood
Random vs. Constant Errors
Random Errors (unsystematic): extraneous variables whose average influence on the outcome is the same in both (or all) conditions (Aronson et al., 1990) Affects reliability AND validity
Constant (systematic) Errors: influences all the scores in one condition in the same direction and has no effect or a different effect on the other condition. Affects ONLY validity Confounding variables
© 2015 M. Guthrie Yarwood
Clarification Check!
IV: Type of show Condition 1: Extraverts watch comedy
show Condition 2: Extraverts watch neutral
show
DV: Self-reported emotions Extraverts report more positive emotions
in Comedy Condition than in Neutral Condition
© 2015 M. Guthrie Yarwood
Clarification Check! – Random Errors
Extraverts in conditions 1 and 2 are placed in very hot rooms. The hot temperature lowers their self-reported levels of positive emotions.
But, we still find that extraverts report more positive emotions when watching the comedy show than the neutral show.
Accurate: We find differences in self-reported positive emotions (on 5-point scale): Comedy Cond. = 4.8, Neutral Cond. = 2.5
Random Errors: In presence of hot temperature, we find differences in self-reported positive emotions Comedy Cond. = 3.0, Neutral Cond. = 1.5
© 2015 M. Guthrie Yarwood
Clarification Check! – Random Errors
Extraverts in conditions 1 and 2 are placed in very hot rooms. The hot temperature lowers their self-reported levels of positive emotions.
But, we still find that extraverts report more positive emotions when watching the comedy show than the neutral show.
Accurate: We find differences in self-reported positive emotions (on 5-point scale): Comedy Cond. 1 = 4.8, Neutral Cond. 2 = 2.5
Random Errors: In presence of hot temperature, we find differences in self-reported positive emotions Comedy Cond = 3.0, Neutral Cond.2 = 1.5
Hot temperature lowers the scores in both
conditions!!
© 2015 M. Guthrie Yarwood
Clarification Check! – Constant Errors
Extraverts in Comedy Cond. are in a 70◦ room. Extraverts in Neutral Cond. are placed in a 78◦ room. In Neutral Cond. only, the hot temperature lowers self-reported levels of positive emotions.
We find that the comedy show results in more positive emotions than the neutral show.
Accurate: We do not find differences in self-reported positive emotions Comedy Cond. = 4.8, Neutral Cond. = 4.6
Constant Error: We find differences in positive emotions due to the hot temperature, not due to the manipulation! Comedy Cond. = 4.8, Neutral Cond. = 2.5
© 2015 M. Guthrie Yarwood
Clarification Check! – Constant Errors
Extraverts in Comedy Cond. are in a 70◦ room. Extraverts in Neutral Cond. are placed in a 78◦ room. In Neutral Cond. only, the hot temperature lowers self-reported levels of positive emotions.
We find that the comedy show results in more positive emotions than the neutral show.
Accurate: We do not find differences in self-reported positive emotions Comedy Cond. = 4.8, Neutral Cond. = 4.6
Constant Error: We find differences in positive emotions due to the hot temperature, not due to the manipulation! Comedy Cond. = 4.8, Neutral Cond. = 2.5
Hot temperature lowers the scores in Condition 2, but not Condition 1!!
© 2015 M. Guthrie Yarwood
Reliability
The ability of a test to measure an attribute consistently Does this extraversion scale measure the true
level of each participant’s extraversion (over time)?
Are coders following the same method?
To achieve reliability we want to reduce random error.
© 2015 M. Guthrie Yarwood
Threats to Reliability (random error)
Participant Characteristics: fatigue, motivation, boredom
Testing situation: time of day, room temperature
Testing Instrument: instructions, rating scale, items, reading level.
Experimenter Characteristics and Errors: interactions with participants; incorrect observations
of participants; incorrect coding of behavior
© 2015 M. Guthrie Yarwood
Estimating Reliability
Test-Retest Coefficient
Parallel-Forms Coefficient
Internal Consistency Coefficient
Interrater (interobserver) Reliability
© 2015 M. Guthrie Yarwood
Validity
Validity: The degree to which a test or measurement accurately measures or reflects what it claims to measure.
Internal Validity: Did the experimental treatments make a difference in this specific experimental instance?
External Validity: Generalizability; To what populations, settings, treatment variables, and measurement variables can the effect be generalized? **Never completely answerable
© 2015 M. Guthrie Yarwood
Internal Validity
Degree to which test measures what it claims to measure
5 types of internal validity Face validity Predictive or criterion validity Convergent validity Discriminant validity Construct validity
© 2015 M. Guthrie Yarwood
Threats to Internal Validity (errors)
Affected by random and constant errors
Random (unsystematic) Errors: same errors that affect reliability
Constant Errors (systematic): errors that affect measurement of variable; does not affect reliability
© 2015 M. Guthrie Yarwood
Knowledge Check!You are conducting a study on the personality traits associated with the frequency of exercising.
For your study, which of the following poses a threat to validity, but not reliability? A. All the participants are bored.B. The construction outside the laboratory
window is very loud.C. You recruited participants from the Rec Hall.D. The experimenter who greets all
participants is very rude.
© 2015 M. Guthrie Yarwood
Knowledge Check!You are conducting a study on the personality traits associated with the frequency of exercising.
For your study, which of the following poses a threat to validity, but not reliability? A. All the participants are bored.B. The construction outside the laboratory
window is very loud.C. You recruited participants from the Rec
Hall.D. The experimenter who greets all participants
is very rude.
© 2015 M. Guthrie Yarwood
Threats to Internal Validity – MRS SMITH
MaturationRegression to the MeanSelection of SubjectsSelection by Maturation InteractionMortalityInstrumentationTestingHistory
© 2015 M. Guthrie Yarwood
Regression to the Mean Example
In an experiment involving reading instruction, subjects grouped because of poor pre-test reading scores show considerably greater gain than do the groups who scored average and high on the pre-test.
© 2015 M. Guthrie Yarwood
Regression to the Mean Example
In an experiment involving reading instruction, subjects grouped because of poor pre-test reading scores show considerably greater gain than do the groups who scored average and high on the pre-test.
Poor Average / Mean
HighPre-test
© 2015 M. Guthrie Yarwood
Regression to the Mean Example
In an experiment involving reading instruction, subjects grouped because of poor pre-test reading scores show considerably greater gain than do the groups who scored average and high on the pre-test.
Poor Average / Mean
HighPre-test
After reading Instruction
© 2015 M. Guthrie Yarwood
Selection by Maturation Interaction
Group Pre-test Post-Test (after head start)
Head Start Intervention
Middle-class children 65/100
Disadvantaged children
65/100
Control Group – No Intervention
Middle-class children 65/100
Disadvantaged children
65/100
© 2015 M. Guthrie Yarwood
Selection by Maturation Interaction
Group Pre-test
Post-Test: 6 months
Post-Test: 12 months
Post-Test: 18 months
Head Start Intervention
65/100
70/100 75/100 80/100
Control Group – No Intervention
65/100
65/100 65/100 65/100
© 2015 M. Guthrie Yarwood
Selection by Maturation Interaction
Group Pre-test
Post-Test: 6 months
Post-Test: 12 months
Post-Test: 18 months
Head Start Intervention
65/100
70/100 75/100 80/100
Control Group – No Intervention
65/100
65/100 65/100 65/100
Ss in Intervention are middle class, while Ss in control group are disadvantaged. Over time, Intervention Ss show
improvement of post-test due to better health care, greater parental support, greater access to resources, etc.
© 2015 M. Guthrie Yarwood
Identify the Threat to Validity!
In a short experiment designed to investigate the effect of computer-based instruction, Ss missed some instruction because of a power failure at school.
A. HistoryB. Mortality C. TestingD. Instrumentation
© 2015 M. Guthrie Yarwood
Identify the Threat to Validity!
In a short experiment designed to investigate the effect of computer-based instruction, Ss missed some instruction because of a power failure at school.
A. HistoryB. Mortality C. TestingD. Instrumentation
© 2015 M. Guthrie Yarwood
Identify the Threat to Validity!
In a health experiment designed to determine the effect of various exercises, those Ss who find the exercise most difficult stop participating.
A. Selection of SubjectsB. Mortality C. TestingD. Maturation
© 2015 M. Guthrie Yarwood
Identify the Threat to Validity!
In a health experiment designed to determine the effect of various exercises, those Ss who find the exercise most difficult stop participating.
A. Selection of SubjectsB. Mortality C. TestingD. Maturation
© 2015 M. Guthrie Yarwood
Estimating Internal Validity
Content ValidityCriterion-Related Validity
Concurrent Validity Predictive Validity
Construct Validity Convergent Validity Discriminant Validity
© 2015 M. Guthrie Yarwood
Content Validity
Definition: whether the content of a test elicits a range of responses that are representative of the entire domain or universe of skills, understandings, and other behaviors a test is designed to measure.
To assess: compare tests’ content with an outline of specifications concerning subject matter to be covered in test.
© 2015 M. Guthrie Yarwood
Openness to Experience
?????
?????
?????
?????
© 2015 M. Guthrie Yarwood
Openness to Experience
Behaviors?
Perceptions?
Thoughts/Cognitions?
Feelings/Emotions?
© 2015 M. Guthrie Yarwood
Criterion-Related Validity
Definition: procedures in which the test scores of a group of people are compared with ratings, classifications, or other measures of performance.
© 2015 M. Guthrie Yarwood
Concurrent Validity – A type of criterion-related validity
Concurrent Validity: when a test is administered to people in various categories, to determine whether test scores of people in 1 category are significantly different from people in other categories. Clinical vs. non-clinical group Different socioeconomic levels
© 2015 M. Guthrie Yarwood
Group A shows more openness to experience
Group B shows less openness to experience
Concurrent Validity
© 2015 M. Guthrie Yarwood
High IQ group shows more openness
to experience
Low IQ group shows less openness to
experience
Concurrent Validity
© 2015 M. Guthrie Yarwood
Predictive Validity – A type of criterion-related validity
Predictive Validity: how accurately test scores predict criterion scores.
Indicated by correlation between test score (the predictor) and a criterion of future performance (what the test predicts)
© 2015 M. Guthrie Yarwood
Predictive Validity
Openness to Experience
(Predictor)
?????
(Criterion)
© 2015 M. Guthrie Yarwood
Predictive Validity
Openness to Experience
College Major
Well-being, Psychological Adjustment
Openness to Experience
© 2015 M. Guthrie Yarwood
Construct-Related Validity
Definition: extent to which scale measures a particular construct or psychological concept
To assess: need to determine whether an assessment instrument that presumably measures a certain personality variable is actually doing so
© 2015 M. Guthrie Yarwood
Convergent Validity – A Type of Construct-Related Validity
Convergent Validity: the measure has high correlations with other measures or methods of measuring the same construct
© 2015 M. Guthrie Yarwood
Convergent Validity
Our Openness to Experience
Self-Report Scale
Same Construct, Same Measurement
Our Openness to Experience
Self-Report Scale
Same Construct,Different Measurement
© 2015 M. Guthrie Yarwood
Convergent Validity
Our Openness to Experience Self-
Report Scale
Costa & McCrae’s Self-Report OE
Dimension
Observer-Report OE Dimension
Our Openness to Experience Self-
Report Scale
© 2015 M. Guthrie Yarwood
Discriminant Validity – A Type of Construct-Related Validity
Discriminant Validity: the measure has low correlations with measures of different constructs
© 2015 M. Guthrie Yarwood
Discriminant Validity
Our Openness to Experience
Self-Report Scale
Different Construct, Same Measurement
Different Construct Different Measurement
Our Openness to Experience
Self-Report Scale
OR
© 2015 M. Guthrie Yarwood
Discriminant Validity
Our Openness to Experience
Self-Report Scale
Self-Report Sensation Seeking
Scale
Clinical Diagnosis of Schizotypal
PersonalityDisorder
Our Openness to Experience
Self-Report Scale
© 2015 M. Guthrie Yarwood
Discriminant Validity
Our Openness to Experience
Self-Report Scale
Self-Report Sensation Seeking
Scale
Clinical Diagnosis of Schizotypal
PersonalityDisorder
Our Openness to Experience
Self-Report Scale
These correlations should be significant,
but lower than correlations for
convergent validity.
© 2015 M. Guthrie Yarwood
To assess convergent/discriminant validity
The same construct using the same method (c)The same construct using different methods
(c)Different constructs using the same method
(d)Different constructs using different methods
(d)
Note: c = convergent; d=discriminant
© 2015 M. Guthrie Yarwood
Reliability & Validity
A test can be reliable, but not valid
In other words, to be valid a test must first be reliable.
NO VALDITY WITHOUT RELIABILITY!
© 2015 M. Guthrie Yarwood
Reliability Internal Validity
Test-Retest Coefficient
Parallel-Forms Coefficient
Internal Consistency Coefficient
Interrater (interobserver) Reliability
Content ValidityCriterion-Related
Validity Concurrent Validity Predictive Validity
Construct Validity Convergent Validity Discriminant Validity
Reliability and Validity: Summary
© 2015 M. Guthrie Yarwood
A LITTLE MORE ON OBSERVER REPORTS (O-DATA)
© 2015 M. Guthrie Yarwood
•Self-report
•Observer report 1
Correlation 1
•Observer report 1
•Observer report 2
Correlation 2
•Self-report
•Several different observers
Correlation 3, 4, 5…
© 2015 M. Guthrie Yarwood
Aggregating Scores
Averaging the self-report and the observer report/s provides a clearer picture of personality than the self or observer report alone (Kolar et al., 1996)
© 2015 M. Guthrie Yarwood
Let’s discuss the purpose of observer reports
What is the purpose of an observer report?Compared to a self-report measure, what
results would you expect from an observer report?
What would you think if you did not obtain these results?
Can you think of any threats to validity that observer reports may pose?
© 2015 M. Guthrie Yarwood
Aggregating Scores
A positive correlation between an observer report and a self-report, would provide evidence for which type of validity?
A. ConvergentB. PredictiveC. ConcurrentD. Discriminant
© 2015 M. Guthrie Yarwood
Recall Reflection Post and TIPI!
Disagree
Strongly
(1)
Disagree
Moderately
(2)
Disagree a
little
(3)
Neither
Agree nor
Disagree
(4)
Agree a
little
(5)
Agree
Moderately
(6)
Agree
Strongly
(7)
1 Extraverted,
enthusiastic
2 Reserved, quiet r
3 Critical,
quarrelsomer
4 Sympathetic,
warm
5 Dependable,
self-disciplined
6 Disorganized,
carelessr
7 Anxious, easily
upset r
8 Calm,
emotionally
stable
9 Open to new
experiences,
complex
10 Conventional,
uncreativer
© 2015 M. Guthrie Yarwood
Self Scores!
Extra
vers
ion
Agree
ablene
ss
Consc
ient
ious
ness
Emot
iona
lly S
tabl
e
Openn
ess to
Exp
erienc
e1
2
3
4
5
6
7
3.96
5.135.54
4.46
5.71
TIPI Dimensions
© 2015 M. Guthrie Yarwood
Observer Scores!
Extra
vers
ion
Agree
ablene
ss
Consc
ient
ious
ness
Emot
iona
lly S
tabl
e
Openn
ess to
Exp
erienc
e1
2
3
4
5
6
7
4.294.92 5.13
4.38
5.25
TIPI Scores
© 2015 M. Guthrie Yarwood
Based on the following graphs, which dimension do you think shows the weakest correlation between self and observer scores?
A. ExtraversionB. AgreeablenessC. ConscientiousnessD. Emotional StabilityE. Openness to Experience
© 2015 M. Guthrie Yarwood
1 2 3 4 5 6 71
2
3
4
5
6
7
Openness to Experience
SELF SCORE
OB
SE
RV
ER
SC
OR
E
1 2 3 4 5 6 71234567
Emotional StabilityLOW NEUROTICISM
SELF SCORE
OB
SE
RV
ER
SC
OR
E
1 2 3 4 5 6 71
2
3
4
5
6
7
Conscientiousness
SELF SCORE
OB
SE
RV
ER
SC
OR
E
1 2 3 4 5 6 71
3
5
7
AGREEABLENEESS
SELF SCOREOB
SE
RV
ER
SC
OR
E
1 2 3 4 5 6 71234567
EXTRAVERSION
SELF SCORE
OB
SE
RV
ER
SC
OR
E
© 2015 M. Guthrie Yarwood
Based on the following graphs, which dimension do you think shows the weakest correlation between self and observer scores?
A. ExtraversionB. AgreeablenessC. ConscientiousnessD. Emotional StabilityE. Openness to Experience
© 2015 M. Guthrie Yarwood
LET’S CORRELATE THE SCORES!
TIPI DIMENSION TIPI TRAIT r
1EXTRAVERSION
Extraverted, enthusiastic
.742 Reserved, quiet r
3AGREEABLENESS
Critical, quarrelsomer
.484 Sympathetic, warm
5
CONSCIENTIOUSNESSDependable, self-disciplined
.526 Disorganized, carelessr
7EMOTIONAL STABILITY
(LOW NEUROTICISM)
Anxious, easily upset r
.078 Calm, emotionally stable
9 OPENNESS TO
EXPERIENCE
Open to new experiences,
complex .3910 Conventional, uncreativer
Note. Red = Self; Yellow = Observer
© 2015 M. Guthrie Yarwood
1 2 3 4 5 6 7 8 9 10 11 121
2
3
4
5
6
7
EXTRAVERSION
STUDENT
SC
OR
E
1 2 3 4 5 6 7 8 9 10 11 121
2
3
4
5
6
7
AGREEABLENESS
STUDENT
SC
OR
E
1 2 3 4 5 6 7 8 9 10 11 121
2
3
4
5
6
7
CONSCIENTIOUSNESS
STUDENT
SC
OR
E
1 2 3 4 5 6 7 8 9 10 11 121
2
3
4
5
6
7
EMOTIONAL STABILITY(LOW NEUROTICISM)
STUDENT
SC
OR
E
1 2 3 4 5 6 7 8 9 10 11 121
2
3
4
5
6
7
OPENNESS TO EXPERIENCE
STUDENT
SC
OR
E
Note. Red = Self; Yellow = Observer
© 2015 M. Guthrie Yarwood
1 2 3 4 5 6 7 8 9 10 11 121
2
3
4
5
6
7
EXTRAVERSION
STUDENT
SC
OR
E
1 2 3 4 5 6 7 8 9 10 11 121
2
3
4
5
6
7
AGREEABLENESS
STUDENT
SC
OR
E
1 2 3 4 5 6 7 8 9 10 11 121
2
3
4
5
6
7
CONSCIENTIOUSNESS
STUDENT
SC
OR
E
1 2 3 4 5 6 7 8 9 10 11 121
2
3
4
5
6
7
EMOTIONAL STABILITY(LOW NEUROTICISM)
STUDENT
SC
OR
E
1 2 3 4 5 6 7 8 9 10 11 121
2
3
4
5
6
7
OPENNESS TO EXPERIENCE
STUDENT
SC
OR
E
© 2015 M. Guthrie Yarwood
Summary and Evaluation
Decisions about data source and research design depend on (1) the purpose of study and (2) threats to validity/reliability
There is no perfect data sourceThere is no perfect research designAssessing threats to reliability and validity will
assist in selecting a data source and research design.
Observer reports improve validity and reduce social desirability concerns
© 2015 M. Guthrie Yarwood
Reminder – Paper Topic!!
Find a group member (can do on ANGEL!)Select a topic!Submit Paper topic to drop-box
Due: Friday, January 30th at 9 AM ETPaper Topic Outline available on ANGEL.Guidance: Meet with Michelle or Celina
Questions???
© 2015 M. Guthrie Yarwood
Chapter 3 (Dispositional Domain) Survey
HEXACO Follow link to access survey Score your survey according to the instructions on
ANGEL
Big Five Model Access through Chapter 3 Survey on course website