1 dependent interviewing: evidence from field tests the sipp methods panel nancy bates and joanne...
TRANSCRIPT
1
Dependent Interviewing: Evidence from Field Tests
The SIPP Methods PanelNancy Bates and Joanne Pascale
U.S. Census Bureau
Seminar on Dependent Interviewing Univ. of Essex, September 17, 2004
2
Acknowledgements
• Pat Doyle, Jeff Moore, Joanne Pascale, Julia Klein Griffiths, Anna Chan
• Heather Holbert, Elaine Hock, Johanna Rupp, Aniekan Okon, Ceci Villa, Judy Eargle
• U.S. Census Bureau Field Division
3
Methods Panel Research Project
• 4-yr research project to study and refine recommendations– Extant data and research, cognitive assessments,
large-scale field experiments, interviewer assessments
• 3 Field Experiments– Treatment group—experimental instrument– Control group—SIPP production instrument– 1000 households per treatment– Conducted in 6 Regional Offices– CAPI interview (CASES language)
4
Methods Panel Research Project
• 3 Field Experiments
– Experiment in 2000 — Wave 1– Experiments in 2001 & 2002 — Waves 1 and 2– Interviewer assessments, data analysis– Response Rates from 83% - 92%
• Produced SIPP 2004 instrument
5
Cognitive Assessments
• Wave 1: 34 households interviewed
• Wave 2: 4 months later, 23 households re-interviewed
• Cognitive interview followed by debriefing
• Reactions to dependent interviewing techniques:– Sharing of data with other members– Reaction to Respondent Identification Policy (RIP)
6
Cognitive Assessments• RIP Question: “We re-contact households every 4 months
to update information. If we talk to someone else in your household next time, instead of you, is it OK if we use your answers as a starting point?”
• D.I. did not elicit privacy concerns• Respondents expected previous information to be
stored• Few respondents declined RIP request
(misunderstood?)
7
Interviewer Debriefings• Pencil and paper debriefing after each wave
(Wave 1 n=152 ; Wave 2 n=131)
• Section in Wave 2 about D.I. Techniques: – Use of “Same as last time” reminders– Dependent questions vs. “from scratch”– Dependent question nonresponse follow-ups– Correcting mis-reported breaks in receipt
• Rated on 5 point scale (Strongly disagree – Strongly agree)
8
Interviewer Debriefings
• D.I. preferred over “from scratch” (assets)• D.I. perceived as easier for respondent; smoother
interview; easier to administer; more accurate (earnings, unearned income, assets, & health insurance)
• D.I. not perceived more accurate in correcting seam bias (not sig. different from control).
• D.I. perceived more accurate regarding health insurance coverage
9
Attrition
• Response Rates (Unit-level attrition)
• “Learning Curve” effect (year)
• D.I. effect (treatment)
10
Attrition: Unit-Level Nonresponse by Year and Treatment by Wave
0%
10%
20%
2000 W1 2001 W1 2002 W1 2001 W2 2002 W2
% n
onre
pons
e
Control
Test
11
Attrition
• On surface, little evidence D.I. impacted attrition• No significant difference in test/control
comparisons• Closer look at Wave 1 across years and treatment
groups• Logistic regression
– Significant (p<.10) main effect for year (negative)– Borderline (p=.11) year*treatment interaction – Nonresponse decline across years is sig. for test group
but not control– Wave 2 exhibits similar trend, but interaction not sig.
12
Dependent Interview Nonresponse Follow-up Techniques
If “Don’t know” “Refused” or “Same as last time”….
• Earnings: Things may have changed since then, but I have
recorded from last time that [NAME] earned about $X,XXX a
month from this job with [EMPLOYER NAME]. Does that still
sound about right?
• Unearned Income: It says here that [NAME] received [$XXX] in
food stamp benefits last [MONTH]. Does that still sound about
right?
• Asset Income: Things may have changed a lot but I have
recorded from last time your income from these rental properties
was about $[X,XXX] annually. Does that still sound about right?
13
Assessing Dependent Interview Nonresponse Follow-up Techniques
• Item nonresponse for income amounts (don’t know, refused)
• Computed aggregate nonresponse ratio: # D.K. + Refused
# questions asked
• Average of ratios across all adults in universe
14
% Nonresponse to Earnings Items - 2002 Wave 2 (All Jobs/Businesses Combined)
20.0%
11.0%
0%
10%
20%
30%
Non
-Res
pons
e R
ate
Control
Test
15
% Nonresponse to Asset Income Items - 2002 Wave 2 (All Asset Types Combined)
20.2%
27.8%
0%
10%
20%
30%
Non
-Res
pons
e R
ate
Control
Test
16
% Nonresponse to Unearned Income Items - 2002 Wave 2 (All Programs Combined)
25.9%
17.3%
0%
10%
20%
30%
Non
-Res
pons
e R
ate
Control
Test
17
SIPP interview months and their associated reference period months (example):
OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG
M1 M2 M3 M4 M5 Wave 1 Interview
M1 M2 M3 M4 M5 Wave 2
M1 M2 [etc.]
18
Hypothetical Panel Survey Month-to-Month Transitions illustrating "seam" bias (Burkhead and Coder, 1985)
050
100150200250300350
Num
ber o
f Rep
orte
d Ch
ange
s
Food Stamps
Social Security
First Interview Second Interview
19
Dependent Interview Seam-Bias Reduction Techniques
• Unearned Income (and School Enrollment):
– If reported in M5 previous: “Last time I recorded that you received
[food stamps] in [month]. Is that correct?”
– If reported in M4 but not M5: “Last time I recorded that you received
[food stamps] in [month 4]. Did you continue to receive any benefits
from [food stamps] after [month 5] 1st?
• Health Insurance:
– “Last time I recorded [NAME] was covered by Medicaid. Is
[NAME] still covered by that type of insurance plan?”
20
Assessing Dependent Interview Seam-Bias Reduction Techniques
• Month-to-month transitions in recipiency/enrollment/coverage (yes/no)
• Transitions at the “seams” (between M4-M5) where fewer transitions = improvement
• Calculated % of month-to-month transitions for: unearned income, school enrollment, and health insurance coverage
21
% of Month-to-Month Transitions in Unearned Income Observed at the "Seam" - 2002 Wave 2
74.7%
62.2%
0%
20%
40%
60%
80%
% tr
ansi
tions
at s
eam
Control
Test
Expected % of transitions if no seam bias
22
% of Month-to-Month Transitions in School Enrollment Observed at the "Seam" - 2001 Production SIPP vs. 2004
40.4%
30.7%
0%
20%
40%
60%
80%
% tr
ansi
tions
at s
eam
2001 SIPP
2004 SIPP
Expected % of transitions if no seam bias
23
% of Month-to-Month Transitions (Yes-No and No-Yes) in School Enrollment Observed at the "Seam" -
2001 Production SIPP vs. 2004
13.1%
1.0%2.0% 1.4%
0%
20%
40%
Yes-No No-Yes
% tr
ansi
tions
at s
eam
2001 SIPP
2004 SIPP
24
% of Month-to-Month Transitions in Health Insurance Observed at the "Seam" (MPSIPP 2002 Control vs. Test)
73.8% 73.4%
0%
20%
40%
60%
80%
% tr
ansi
tions
at s
eam
Control
Test
Expected % of transitions if no seam bias
25
% of Month-to-Month Transitions (Yes-No and No-Yes) in Health Insurance Observed at the "Seam" -
MPSIPP 2002 Control vs. Test
4.7%
21.4%
5.2%
26.6%
0%
20%
40%
Yes-No No-Yes
% tr
ansi
tions
at s
eam
Control
Test
26
Conclusions
• D.I. In 2004 SIPP represents major departure from past practice
• Thorough testing program suggest:– Substantially improved interview experience from
Interviewer perspective– Respondents had few concerns with D.I.
(qualitative tests)– R.I.P didn’t have big impact on D.I. (3-5%)
27
Conclusions• D.I. beginning to impact attrition?• D.I. Nonresponse follow-up techniques
successful • Decreased item nonresponse for earning,
asset, and unearned income amounts• D.I. techniques helped to reduce seam bias in
income recipiency & enrollment• D.I. helped reduce “on/off” seam transitions?• Still much room for improvement• SIPP D.I. techniques first fielded in June 2004