q2008 – european conference on quality in official statistics rome • july 9-11, 2008
Post on 03-Jan-2016
21 Views
Preview:
DESCRIPTION
TRANSCRIPT
The Effect of Questionnaire Length on Response Rates and Level of Estimates in the German Job Vacancy Survey
Hans Kiesl
Institute for Employment Research (IAB), Germany
Q2008 – European Conference on Quality in Official Statistics
Rome • July 9-11, 2008
2
Background
Regulation (EC) No. 453/2008 of the European Parliament and of the
Council of 23 April 2008 on quarterly statistics on Community job
vacancies
Member states have to provide quarterly data on job vacancies (broken down to NACE section level) quality reports
In Germany, the data will be provided by the IAB.
3
Background (2)
Information on job vacancies in Germany
Business units might report job vacancies to the Federal
Employment Agency
Federal Employment Agency publishes monthly statistics on number
of registered job vacancies (by NACE-sector)
Since 1989, IAB conducts a yearly (4th quarter) sample survey among
business units to estimate number of job vacancies (registered or
not) and to get additional information (e.g. about recruiting strategies)
Mail questionnaire (8 pages in length); voluntary
CATI interviews in quarters 1 - 3
4
Basic sampling design
stratified SRS; 16 sectors 7 size classes x West/East
sampling rates and sample sizes in different size classes (IV/2006):
West East
rate sample size rate sample size
< 10 0.9% 11067 3.3% 10963
10 - 19 5.9% 8533 26.1% 9546
20 - 49 7.3% 6745 33.6% 8282
50 - 199 8.6% 4541 13.3% 1815
200 - 499 32.9% 3249 50.3% 1111
500 - 999 68.3% 1806 73.0% 395
>= 1000 75.1% 984 84.8% 193
total 2.4% 36925 8.0% 32305
5
Highly skewed distribution of job vacancies0
100
200
300
400
job
vaca
ncie
s
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
6
Highly skewed distribution of job vacancies0
20
40
60
80
100
job v
acancie
s
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
91 88 84 76 72 68 86 89 88 84 78 77 81 87 83 87% of Zeros:
7
Problem: extremely low response rates
0%
5%
10%
15%
20%
25%
30%
35%
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007
8
Reasons for non-response
After fall 2007 survey, CATI sub-sample of non-respondents to
find out main reasons for non-response
sample of 1292 business units
6% no/wrong telephone number or late respondents
15% not willing to respond
79% respondents; their reasons for non-response in job vac. survey:
54%: no time; too much work (72% for largest units)
20%: no job vacancies (25% for smallest units)
4%: no relevant topic (44% for smallest units)
7%: take part in surveys only if mandatory
7%: never take part in surveys
9
Impact of length of questionnaire (1)
During the 4th quarter of 2006 (at the same time of the regular survey
with 8 pages) a separate survey was conducted (1 page, basic infos,
e.g. number of job vacancies; 9,450 sampling units).
Research questions:
Has length of questionnaire significant impact on response
rates? (Prediction: yes)
If so, do different response rates lead to different estimates of
number of job vacancies? (Prediction: yes)
10
Impact of length of questionnaire (2)
Response rates by size of business units:
8 pages 1 page abs. diff. rel. diff.
< 10 21.0% 27.3% + 6.3% + 29.9%
10 - 19 17.0% 30.7% + 13.7% + 80.1%
20 - 49 18.9% 28.2% + 9.3% + 49.1%
50 - 199 23.2% 28.6% + 5.4% + 23.2%
200 - 499 18.1% 27.1% + 9.0% + 49.9%
500 - 999 18.4% 28.5% + 10.1% + 54.9%
>= 1000 26.8%
11
Impact of length of questionnaire (3)
Response rates by sector:
8 pages 1 page abs. diff. rel. diff.
Agriculture, farming, forestry, fishing 24.8% 36.5% + 11.6% + 46.9%
Manufacturing 19.6% 29.8% + 10.2% + 52.3%
Construction 21.5% 28.8% + 7.3% + 34.1%
Sale, trade, hotels and restaurants, transport, communications 14.9% 23.3% + 8.4% + 56.1%
Financial intermediation, insurance, renting and business activities 17.3% 24.9% + 7.6% + 44.0%
Public administration, education, health, social work, other service activities 23.0%
12
Weighting and estimation
Calibrate Horvitz-Thompson-estimator to totals from auxiliary data:
sampling frame out of date (> 1 year)
up-to-date estimates on number of units by size and by sector (no
cross-classification)
up-to-date estimates on number of employees by size and sector (no
cross-classification)
registered number of job vacancies by sector
Linear GREG, using CLAN (Statistics Sweden)
Results quite robust with regard to underlying model
e.g. constant variance or variance proportional to (1 + number of registered vacancies)
13
Impact of length of questionnaire on estimated number of job vacancies (1)
8 pagesestimated
standard error estimated cv
1 page
West 791,000 27,000 3.4%
843,000 51,000 6.0%
East 212,000 10,000 4.7%
229,000 32,000 14.0%
Germany 1,003,000 30,000 3.0%
1,072,000 60,000 5.6%
(Note: differences to officially published results due to slightly different weighting)
14
Estimated number of job vacancies
0
100000
200000
300000
400000
500000
600000
700000
800000
900000
< 10 10 - 19 20 - 49 50 - 199 200 - 499 500 - 999 >= 1000
8 pages
1 page
Impact of length of questionnaire on estimated number of job vacancies (2)
15
Impact of length of questionnaire on estimated number of job vacancies (2)
Estimated number of job vacancies
0
50000
100000
150000
200000
250000
300000
350000
400000
450000
Agriculture,farming, forestry,
fishing
Manufacturing Construction Sale, trade,hotels and
restaurants,transport,
communications
Financialintermediation,
insurance,renting andbusinessactivities
Publicadministration,
education,health, socialwork, other
service activities
8 pages
1 page
16
Impact of length of questionnaire on estimated number of job vacancies (3)
8 pages 1 page
(higher
response)
weighting without number of registered vacancies 211,000 256,000
registered job number of vacancies 529,000 676,000
vacancies share of registered vacancies 40% 38%
number of units with vacancies 166,000 176,000
weighting with
registered job number of registered vacancies 465,000 465,000
vacancies number of vacancies 791,000 843,000
share of registered vacancies 59% 55% number of units with vacancies 172,000 199,000
17
Conclusions
Length of questionnaire has considerable effect on response rates
(as expected).
Change in response rates seem to have only little effect on main
survey estimates (job vacancies)
(not as expected).
Indication that non-response bias might be only modest.
Thank you very much for your attention!
Q2008 – European Conference on Quality in Official Statistics
Rome • July 9-11, 2008
top related