William van der Veld (University of Amsterdam)Willem Saris (ESADE Business School)Barcelona, 2005, European Association for Survey Research
THE NATURE OF MEASUREMENT ERRORIN PANEL DATA
Reliability and Opinion Stability
2
Overview• The RUSSET panel (www.vanderveld.nl)• Reliability estimation in a panel design• Reliability estimation in a cross-section
design• Comparison of Panel & Cross-section• The VAS Model• Conclusion
3
The RUSSET panelPlease, tell me, how satisfied are you with your current life as a whole?
1 2 3 4 5 6 7 8 9 10 Not at all Very Satisfied Satisfied
123456
1994 1995 1998
Aggregate level [means] Individual level (Correlations)
1993 1995 19981993 1.001995 0.34 1.001998 0.30 0.27 1.00
4
Reliability estimation in a panel design
Quasi-simplex model: Assumptions• The random error variance is the identical for the repeated measures;• For each respondent the attitude changes according to a lag-1 quasi-
simplex;• The repeated measures are independent.
T1
y1
λ1
β21 T2
y2
λ2
β32 T3
y3
e
λ3
e e
d2 d3 Interpretation• y: The observed variable• e: the random error in y• T = y + e• When T and y standardized,• β: the stability coefficient• λ: reliability coefficient
5
Reliability estimation in a panel design• Estimation of Reliability & Stability• RUSSET Data: 3 waves, 3 times the question:• How satisfied are you with your current life…?• n=837; Chi-square=0; df=0 [Lisrel, ML estimation]
Completely Standardized Solution
λ1 λ2 λ3 s21 s32 0.59 0.58 0.59 0.99 0.82
1993 1995-6
1998
1993 1.001995-6
0.34 1.00
1998 0.30 0.27 1.00
6
Reliability estimation in a panel design• The observed stability is:
0.34 & 0.27.• After correction for measurement error, the
stability is:0.99 & 0.82.
• The reliability coefficient of a simple question as ‘How satisfied are you with your current life as a whole?’ is quite poor: ±0.6 (reliability=0.36)
• Is there an alternative way to estimate the reliability,so that we can compare both estimates?
7
Reliability estimation in a cross-section design
Parallel-test model: Assumptions• The random error variance is the identical for the
repeated measures;• For each respondent the attitude has not changed
during the interview;• The repeated measures are independent.
Interpretation• y: The observed variable• e: the random error in y• T = y + e• When T and y
standardized,• λ: reliability coefficient
λ1 λ2
x1 x2
T
e e
8
Reliability estimation in a cross-section design• Estimation of Reliability (not Stability)• RUSSET Data: 1 wave, 2 times the question:• How satisfied are you with your current life…?• Same respondents (exactly), same question.• n=837; Chi-square=1; df=1 [Lisrel, ML estimation]
1995-6 995-1371995-6 1.001995-137
0.72 1.00
Completely Standardized Solution
λ1 λ2-6 λ2-137 λ3 - 0.85 0.85 -
9
Reliability estimation in a cross-section design• THAT’S STRANGE!!!!• Same question, same respondents, same moment• Same definition:
• Still estimated reliability PTR(6)=0.85 vs. QSR(6)=0.58• No confirmation because they are different.• So, this result does not help• Is there a way to check which reliability is correct?
10
Comparison of QSR & PTR• Use the reliability-estimates to
correct the observed correlation between different variables for measurement error.
• ρ21 = r21/(λTx1 * λ Tx2)• We have observed the
correlation: r21• We have estimated the reliability
coefficients with QS & PT model.• We are interested in: ρ21• Which ρ21 is more plausible?
That obtained with the QS-estimates or that with the PT-estimates
Ts1
xs1
λTx1
ρ21 Ts
2
xs2
λTx2
e'2 e'
1 r21
Tx21 21Tx21 ρr
11
Comparison of QSR & PTR• Exactly the same data (n=837), Wave 1995(6,7).• Correction of the observed correlation between:
Satlife & Satinc.• Satlife:
How satisfied are you with your current life as a whole?
• Satinc:How satisfied are you with your family’s current financial situation?
• Observed correlation between Satlife & Satinc = 0.51
• The reliability coefficient estimates are:
12
Comparison of QSR & PTRStandardized estimates of reliability coefficients
QSRC PTRCλ1 λ2-6 λ3 λ2-6 λ2-m2
SATLIFE 0.59 0.58 0.59 0.85 0.85SATINC 0.57 0.53 0.63 0.84 0.84Cor(Satlife, Satinc)=
.51 .51
ρ21=
13
Comparison of QSR & PTRStandardized estimates of reliability coefficients
QSRC PTRCλ1 λ2-6 λ3 λ2-6 λ2-m2
SATLIFE 0.59 0.58 0.59 0.85 0.85SATINC 0.57 0.53 0.63 0.84 0.84Cor(Satlife, Satinc)=
.51 .51
ρ21=
.71
14
Comparison of QSR & PTRStandardized estimates of reliability coefficients
QSRC PTRCλ1 λ2-6 λ3 λ2-6 λ2-m2
SATLIFE 0.59 0.58 0.59 0.85 0.85SATINC 0.57 0.53 0.63 0.84 0.84Cor(Satlife, Satinc)=
.51 .51
ρ21=
1.66 .71
15
Comparison of QSR & PTR• It appears that the QSR-estimates are wrong.
What’s wrong with the quasi-simplex reliability?
• The key is provided by the VAS model (Van der Veld & Saris, 2004)
• It can be derived that: QSR=q(vas)*c(vas)• This is proven in the accompanying paper, but
it can be illustrated too.
16
The VAS ModelFor details, visit
Session:Nonattitudes and informed opinions at Thursday 11:30Chair: Peter Neijens
17
The quasi-simplex model
T1
y1
λ1
β21 T2
y2
λ2
β32 T3
y3
e
λ3
e e
d2 d3
18
The VAS ModelFor details, visit
Session:Nonattitudes and informed opinions at Thursday 11:30Chair: Peter Neijens
19
The VAS Model - Estimation of Q, S, & C• RUSSET Data: 3 waves, 6 times the question (n=627).• Satlife:
How satisfied are you with your current life as a whole? • Satinc:
How satisfied are you with your family’s current financial situation?
Completely Standardized Solution χ2(6) q11 q12 q21 q22 q31 q32 c1 c2 c3 s21 s32 SATLIFE 14.19 0.85 0.88 0.81 0.91 0.83 0.84 0.79 0.75 0.76 0.81 0.79 SATINC 34.92 0.80 0.90 0.78 0.94 0.90 0.89 0.73 0.70 0.81 0.81 0.66
20
Comparison of QSR & PTR & VASStandardized estimates of reliability coefficients
QS PT VASλ2-6 λ2-6 q2-6 c2-6 q*c
SATLIFE 0.58 0.85 0.81 0.75 0.60SATINC 0.53 0.84 0.78 0.70 0.54
• QSRC=q*cQSRCsatlife=.81*.75=.60QSRCsatinc=.78*.70=.54
21
Conclusion• The quasi-simplex model has parameters for
– Stability– Reliability
• The reliability in that model is not correct, It is the product of:– The quality of the measurement
instrument– The opinion crystallization
• The stability is correct [see paper].
22
Conclusion• How do you feel about public security these days?• With the QSM all the considerations that are unique
for a specific time are not in the variable S. Hence such considerations are not part of the stability.
• One should be aware of this.• It depends on the object of study, whether this
model is appropriate to estimate the stability!• It is definitely not correct for reliability estimation!