let me tell you what you told me: dependent interviewing ......• panel surveys often ask for same...
TRANSCRIPT
Let Me Tell You What You Told Me:Dependent Interviewing in
Establishment Surveys
Heather RidolfoNational Agricultural Statistics Service
Jennifer EdgarBureau of Labor Statistics
AAPOR 2015
Background
• Panel surveys often ask for same informationacross data collection points
• Burden• Measurement error can produce false rates of
change• Dependent Interviewing one solution
AAPOR 2015
Dependent Interviewing (DI)
• ProactiveThe last time we interviewed you on January 21,2013, you had 10 employees. How many employeesdo you have now?• Reactive(after a change in status is reported) Last time youreported that there were 5 employees at thiscompany. Does knowing that change your answer inany way?
AAPOR 2015
DI in Population Surveys
• Burden– Increases & Decreases
• Rapport– Improve (PDI)– Decreases (RDI)– Little evidence of
confidentiality concerns
• Data Quality ImprovedSome of the Time– Reduces misclassification– Reduces spurious change– Question & respondent
characteristics matter– Decreases editing– Little evidence of
satisficing– Less standardization
AAPOR 2015
DI in Establishment Surveys
• Reactive is more effective than proactive inimproving data quality but still adds burden
• Establishment characteristics (size)• Construct characteristics (stability, complexity)• Satisficing is a problem
AAPOR 2015
Research Questions
• How do establishment respondents integratepreviously reported data into the questionresponse process?
• Do establishment respondents use previouslyreported data differently when it’s presentedproactively vs. reactively?
AAPOR 2015
Methods
• Occupational Employment Survey (OES)Cognitive Interviews
• Agricultural Labor Survey Behavior Coding
AAPOR 2015
• Collects data on employment, job title and wages• Considering DI to improve data quality for phone
interviews (generally small establishments)• 17 cognitive interviews conducted over the phone• Randomly assigned to DI condition
– Proactive (n=8)– Reactive (n=9)
• Employment no. captured 12 and 18 months prior
Methods: OES Cognitive Interviews
AAPOR 2015
OES Results: Response Process
• With small companies (average employment count was6), most respondents simply knew the employmentcount
• Almost all respondents reported employment withoutconsulting records or any mental calculations– “I own the company, I know who works here”
AAPOR 2015
OES Results: Proactive DIEx: Last time, you reported that there were 5employees at this company. How many employeeswork there now?• Some respondents confirmed the PRD rather than
answering with an employment count• Most respondents who had a change in
employment were able to explain it in thedebriefing
AAPOR 2015
OES Results: Reactive DIEx: (after a different employment count was provided.) Last timeyou reported that there were 5 employees at this company. Doesknowing that change your answer in any way?• Most respondents (5 out of 8) reported the same employment
so reactive DI was not done• For the three that received reactive DI, differing results
• Easily able to explain difference• Correction of answer “Oh, you must be talking about New
York”• Unwillingness to reconcile differences “maybe I made a
mistake back then”
AAPOR 2015
OES Results: Reaction to DI
• Respondents don’t recall reporting last year• No confidentiality concerns• In both conditions, respondents reported that PRD didn’t
impact their response process much
AAPOR 2015
OES Conclusions
• Little impact of DI on data quality• Some evidence of satisficing
– Proactive: “it’s the same”– Reactive: Unwilling to reconcile differences in PRD
and current reports• No discernable impact on detecting change• No evidence of burden reduction
AAPOR 2015
Methods: Ag Labor Behavior Coding
• Collects data on types and numbers of farmworkers, hours worked and wages paid
• October 2013 Ag Labor Survey• Recorded 759 interviews CATI calls• 139 contained questions of interest & quality
audio• 2,689 observations coded• 4 coders (Kappa = .62 - .66)
AAPOR 2015
Methods: Ag Labor Behavior Coding
Interviewer BehaviorCodes• Asked as worded• Major change• Used PRD• Shortcutted
Respondent BehaviorCodes• Yes• Elaborated• Corrected• Requested clarification• Answered don’t know• Gave no response
AAPOR 2015
Ag Labor Results
• PRD used 19% of the time- Quarter 1: 14%- Quarter 2: 24%
AAPOR 2015
Respondents’ Reactions to PreviouslyReported Data (n=166)
Respondent Reaction Percent
Cognitive EngagementElaborated 15Corrected 18Requested Clarification 1
Possible SatisficingYes 50Gave No Response 23
OtherAnswered Don’t know 1Inaudible 3
AAPOR 2015
Cognitive Engagement By Question
0
20
40
60
80
100
Screener WorkerType
Describe No. Workers Hours Wages
Perc
ent
Questions
Cognitive Engagement Possible Satisficing
AAPOR 2015
Ag Labor Conclusions
• Some evidence of cognitive effort acrossquestions; majority of respondents possiblesatisficing
• No evidence that question type matters• Interviewers need more training
AAPOR 2015
Recommendations
• More research is needed to understand theestablishment respondent’s response process
• Initial evidence suggest satisficing problem• There was no strong evidence promoting one DI
method over another (proactive or reactive)• For either method, interviewer training is crucial
– Interviewers need to be aware of the potential risksassociated with DI and equipped to address them
AAPOR 2015
