the labour force survey - data quality issuesthe labour force survey – data quality issues matthew...

The Labour Force Survey – Data Quality Issues

Matthew Steel

• Response rates • Factors • Improving response • Research • Refusal follow-up study • Future work

Data Quality Issues

Response rates

• Falling contact rates • Increasing interview length • Respondents less willing • External shocks • Reduced budgets • Change in survey design

Factors affecting response

• Interviewer training • Incentives • Questionnaire reviews • Varying calling patterns • Survey materials • IT infrastructure

Improving response

• Analysis of attrition bias • Analysis on key estimates • Analysis of paradata • Refusal follow-up study

Research

Refusal follow-up study - Introduction Study:

Eurostat funded project - result of one of the recommendations of the Quality Task Force Review of the LFS, to collect information on the characteristics of non-respondents in order to assess and adjust for non-response bias and to improve fieldwork strategies.

Issue

• Declining response rates:

- August 2011 64% response rate at wave 1 - 10 years ago 75% in wave 1

• Key LFS estimates - sensitive to ethnicity, household type and nationality But – these are not included in the calibration totals

Refusal follow-up study - Introduction

• Therefore, if achieved sample proportions of… - ethnic groups - household groups - national groups… …are not representative of those proportions in… - allocated sample - population… …then the estimates derived from the LFS may be biased.

• RFU survey aimed to:

- identify whether key characteristics of responders and non-responders differentiate - identify if RFU group reasons for refusal differ to general refusing group

Refusal follow-up study – Data Collection

Mainstage RFU exercise: - Main and boost LFS survey - Wave 1 and outright/HQ refusals only - April to June 2011 LFS questionnaire

Data collected in 4 ways… 1. Face-to-face 2. Telephone Unit (wave 1) 3. Survey Enquiry Line (SEL) 4. Self completion

♦ Face-to-face (left at doorstep) ♦ SEL (posted to respondent)

Questionnaire process: - Interviewer/ SEL attempt achieve LFS interview - Not possible, offer RFU as last resort

RFU Letter RFU Questionnaire

Refusal follow-up study – Response Rate

• Total AJ11 outright refusals and HQ refusals = 4,137

• LFS RFU respondents = 787

• LFS RFU response rate = 19 %

• ONS received responses from each of the 4 methods • No interviewer estimate data was allowed, for:

- Ethical reasons - Accuracy/ validity reasons

Refusal follow-up study – Analysis

• LFS collected data on all members of household • LFS RFU collected data:

- Fully on one member of household - Partially on all other members

• Therefore, comparisons between LFS and LFS RFU enabled by… - Selection of one random adult from each LFS responding household - Selection of full data respondent from LFS RFU - Created analysis dataset containing both

Percentage comparison of reasons for RFU and LFS

Refusal follow-up study – Reasons for refusal

Refusal follow-up study - Characteristic comparisons of LFS and RFU respondents

Age

Refusal follow-up study - Characteristic comparisons of LFS and RFU respondents

Sex

Refusal follow-up study - Characteristic comparisons of LFS and RFU respondents

Nationality

Refusal follow-up study - Characteristic comparisons of LFS and RFU respondents

Ethnicity

Refusal follow-up study - Characteristic comparisons of LFS and RFU respondents

Household type

Refusal follow-up study - Characteristic comparisons of LFS and RFU respondents

Workless households

Refusal follow-up study - bias

• Aim? - to determine whether bias exists in LFS data

• How? - Created 1 weight for LFS responders and RFU combined dataset, and - Created 1 weight for LFS responders dataset only (excluding RFU cases)

• Analysis? - Investigated whether proportions in data differed when RFU group included

This indicates: - potential bias in the current estimates from a sample that included these ‘refusers’ in the LFS - or that the current LFS estimates are themselves potentially biased.

Dataset Indicator Status LFSRA only Unemployed No bias

LFSRA plus LFSRFU Employed No bias

LFSRA plus LFSRFU Unemployed Bias

LFSRA plus LFSRFU Inactive Bias

LFSRA only Employed Bias

LFSRA only Inactive Bias

Refusal follow-up study – bias investigations

Refusal follow-up study - Caveats

• LFSRFU data collected from one adult per residence

• LFSRFU sample may not be representative of all ‘refusers’ to LFS

• Not possible to produce standard errors in comparing the two weighted datasets.

• Continue to monitor • Census non-response link study • Regularly review length of survey • Data collection modes • Non-response weights

Future work

Contact details LFS Research Team: nina.parry-langdon@ons.gsi.gov.uk debra.leaker@ons.gsi.gov.uk matthew.steel@ons.gsi.gov.uk Social Survey Data Service: socialsurveys@ons.gsi.gov.uk

Questions?