from postal surveys to electronic data capture: building higher response rates christopher taylor,...
Post on 17-Dec-2015
212 Views
Preview:
TRANSCRIPT
From postal surveys to electronic data capture: building higher response rates
Christopher Taylor, CBI
Technical Development Executive
EU BCS Workshop
November 13th 2014
INTRODUCTION
• THE JOURNEY TO ELECTRONIC DATA CAPTURE (E-CAPTURE)
• THE THINKING BEHIND OUR DECISIONS
• BROADER IMPACTS ON OUR METHODOLOGY
• THE IMPACT ON RESPONSE RATES
• THE FUTURE…
Background context
● The CBI has a long and well established relationship with its participants using paper based surveys dating back to the first Industrial Trends survey in 1958
● At present all CBI business surveys are paper based with the majority being distributed by post and returned via the same method or faxed with some being scanned and emailed
● Participants are also offered the opportunity to return responses over the phone by phoning us themselves or receiving a pre-arranged call
● All our current methods of return are known to be reliable and traditionally popular within our survey samples.
● The long standing nature of these processes mean we have also developed several mitigation processes if a problem such as a postal strike might arise. This contingency planning is a key aspect of our sample management
Is there demand?
● As part of our usual practice we provide space in our questionnaires for participants to comment on any subject they choose
● In some of our samples we were capturing increased interest in online response possibilities
● The Survey management group decided to follow up these comments by asking our broader sample
● We did this by asking about online response in our 2007-09 cycle of Answering Practice surveys. These are research surveys conducted to gain detailed insight into participant response behaviour
Previous APS Findings
We first asked participants about online response in our 2007-09 Answering practices surveys
APS 07-09 Response0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
51%55%
60%
67%
DTS ITS SSS FSS
The findings were positive with an Average of 58% in favour
ITS APS 07 Yes - 55%SSS APS 08 Yes - 60% DTS APS 08 Yes - 51%FSS APS 09 Yes - 67%
The results weren’t absolute but clearly showed there was a encouraging majority in favour of having an online response option
This led to an internal discussion paper investigating online response with the assistance of a third party developer.
A paper proposing a road map to online response was developed but deferred due to the significant amount of work required for the NACE Rev 2/SIC2007 project and updating of the survey weights
Recent APS findings
The ambition to expand our response offering to participants remained, so we turned to our participants again. In the 2012-14 APS surveys we asked participants if they would like the option to submit their response online
● SSS APS 2012 demand had risen to 75%● ITS APS 2013 demand had risen to 73%● DTS APS 2014 demand had risen to 75%
With a clear increase in demand we began looking into a variety of online options that would meet our business and stakeholder requirements as well as our participants response needs
APS 07-09 APS 10-140%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
51%
75%
55%
73%
60%
75%
0.67
DTS ITS SSS FSS
GETTING STARTED: project stakeholders
Electronic Data Capture
SMG
Systems
I.T
ITIM
EconomicAnalysis
Participants
Resources
Focus Groups
CBI
Bank of England
European Commission
H.R
FinancePress
Media
Membership
Academic Community
Software Developers
Economic Community
Public
Design
Web Team
Policy
Survey Sponsor
Data Vendors
Key throughout project lifespan
Key to multiple aspects of the project
Key to a particular aspect of project
A strong interest in project
Key consideration when developing the new system
To enable the SMG to collect
data from participants in an
online format
reliability, security and accuracy of
the system created are absolutely paramount
It must be easy to create, adapt and
maintain
Any impact on established systems and methodologies must have clear positive gains
The participants must find it time and cost efficient whilst being simple to use
E-methodology
The method chosen is E-questionnaires using a PDF platform which are distributed by a mail merge system
● What does this mean?
Questionnaire Distributed by post
Participant receives and populates questionnaire
Participant returns response via fax,
post, phone or scan/email
SMG Scrutinises response
Response information entered into main data set
E-questionnaire Distributed via
Participant receives part populated E
questionnaire enters response information
Participant returns response via Submit
button, fax, post, phone
SMG & E-system Scrutinise response
Response information entered into main data set
Why this method?
Existing Response behaviour, brandContinuity
Online and Offline, pre-populationFunctionality
Existing software platform, known process, multiple fail safesStability
Added scrutiny points, maintains existing high levels of scrutinyScrutiny
Non hosted, quick turnaround, internal management and controlProduction
We have several established survey process and analytical systems which have been utilised in accordance with some of the functionality now available as a result of our new E-methodology
● Firm Level Data system – Built in 2008/9 this in access data interface linked directly to our survey system response data. It allows the SMG to extract unweight raw data directly from the system in a specified format
● Non-Response model – this is an excel analytical tool that uses participant ID numbers to track response behaviour. The system identifies differing levels of response and non response and helps the SMG identify dormant or non trading participants
● Participant Emails rounds – these are enacted based upon data provided by both the afore mentioned systems
How can we use this opportunity?
Participant Email groups
● Before we make an online option fully available, an e-relationship was established with our samples● This serves as an excellent testing ground for the distribution system and growing our focus groups● It continues and enhances our existing dialogue with participants throughout this key development phase
How did we do this?- The introduction of 2 email rounds to opted in participants, the 1st includes a fully mail merged version of the
E-questionnaire and the 2nd has a non mail-merged version of e-questionnaire - The first round is enacted 3 days before the survey closing date and then a final reminder the day before
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Start of field
period
Field period closes
11 12 13 14
1st Email Round
2nd Email round
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Day 12 Day 13 Day 14 Day 15 Day 160
10
20
30
40
50
60
70
2013 2014
Average daily response - April-October 2013/14
Below is the average daily response to the Industrial trends survey over a normal field period prior to the introduction of the Email rounds and post. The impact of the Email rounds is clearly illustrated by the appearance of the upward curve on the final 4 days of the 2014 trend line
Email round impact on final 4 days of field period 2013 vs. 2014
Day 1 Day 2 Day 3 Day 4
19
12
18
3
46
38
36
16
Industrial trends
2013 2014
The charts show the year on year increase in response rates over the final 4 days of the September field periods for the Service sector and Industrial trends survey
Day 1 Day 2 Day 3 Day 4
9
6
4
1
21
12
11
6
Service Sector
2013 2014
Percentage increase in Total response over the last 6 months
The percentage increase in Response rates April-October 2014 in relation to the equivalent period 2013
May June July August September October
-20.0%
-10.0%
0.0%
10.0%
20.0%
30.0%
40.0%
8.6%
27.7%
22.8%
1.0%
22.6%20.5%
-11.0%
26.7%
21.4%
33.7% 34.8%33.1%
Industrial Trends Service Sector
The Distributive Trades survey total response over the final 4 days 2013/14
The distributive trades sample has reacted positively although slightly differently from the other samples
In place of a large spike in response post the email rounds we’ve seen an evenly proportioned increase in response.
April May June July August September October0
10
20
30
40
50
60
3537 37
2730
40
33
47
56 55
46
55
49
41
Distributive Trades final 4 days of Field period response totals 2013 2014
20
Average growth
The average proportion of survey responses received within the final 4 days of our surveys has grown from 18% to just over 30%
Since elements of our E-methodology were introduced we have observed an average growth rate of 14.9% in our sample
Average
Distributive Trades
Industrial Trends
Service Sector
0.0% 5.0% 10.0% 15.0% 20.0% 25.0%
14.9%
9.7%
15.7%
19.4%
Average growth in sample size
18.6%
81.4%
30.3%
69.7%
This is a work in progress but the initial results are very encouraging, we’re seeing growth in all our samples
Our participants have greeted our work with enthusiasm and are providing invaluable feedback
Development work will continue with additional functionality continuing to be made available to participants
2015 will be a key year for rolling out and promoting our full E-methodology to a wider audience
Summary
Prepare Survey folder
Run E-participant report
Participants opt in via Questionnaire/email/phone
Mail merge distribution
Participants receive e-questionnaire as an email attachment
Scrutiny
Pre distribution Email
Prepare E-questionnaire
E-Capture 4 stage cycle
The operational step-by-step run through of the processes incorporated within our E-methodology
Response
Late/non response
S c r u t i n y
Participant opens pre-populated E-questionnaire
Participants enter answers
Response returned to designated Email inbox
Participants Submit response
printed version
returned Via Fax or
Post
Via Submit button
As email attachment
Scrutiny
Responses saved to response folder
Data mapping conducted
Mapped response data placed into Import folder
Results uploaded onto survey system via importer
Response data extracted into spreadsheet
S c r u t i n y
All imported data will be marked as manual or E-capture
3 calculation reports will be run for each survey
Non web responses
only
Web responses
only
Combo of All
Reponses
Data supplied to Economists
Economists conduct analysis of results
The future and the E-community
What do we expect in the future…● Instant and easy dialogue survey sample● Resilient process, provision of support to postal system● Possible ad-hoc questions● Embedded links● Social media● Online metrics/analytics● New response metrics● New soft material delivered to participants● Greener ● Broader distribution of soft media
top related