tracer survey of qualified vocational educational and ... reporttsqvta.pdf · final consolidated...
TRANSCRIPT
SEPTEMBER 5, 2014
F I N A L C O N S O L I D A T E D S U R V E Y R E P O R T
Tracer Survey of Qualified Vocational Educational and Training Applicants
MCA‐N/COM/RCQ/5A01003‐C
PRESENTED TO: Millennium Challenge Account Namibia
PRESENTED BY: NORC at the
University of Chicago
commissionedbytheMillenniumChallengeAccountNamibia
withfundingfromtheMillenniumChallengeCorporation
Table of Contents
I. List of Acronyms .................................................................................................................... 3
II. Executive Summary ............................................................................................................... 4
III. Introduction ............................................................................................................................ 6
Implementation Timeline ........................................................................................................... 7
IV. Survey Results – Baseline ..................................................................................................... 9
Sampling ................................................................................................................................... 9
Response Rates ...................................................................................................................... 12
Quality Control ......................................................................................................................... 15
Data Entry and Cleaning using the Liberty System ................................................................. 16
V. Review of Survey Events – Baseline ................................................................................... 17
Field Problems/Issues ............................................................................................................. 17
Questionnaire Changes ........................................................................................................... 18
Integrating Previous Data ........................................................................................................ 19
VI. Survey Results – Endline ..................................................................................................... 20
Sampling ................................................................................................................................. 20
Response Rates ..................................................................................................................... 21
Quality Control ......................................................................................................................... 24
Data Entry and Cleaning using the Liberty System ................................................................. 25
VII. Review of Survey Events – Endline ..................................................................................... 26
Field Problems/Issues ............................................................................................................. 26
VIII. Lessons Learned ................................................................................................................ 27
Baseline & Follow-up Rounds ................................................................................................. 27
I. List of Acronyms
FS Field Supervisor
MCA‐N Millennium Challenge Account Namibia
MCC Millennium Challenge Corporation
MPR Mathematica Policy Research
NORC NORC at the University of Chicago
NTA Namibia Training Authority
NTF National Training Fund
SW Survey Warehouse
TP Training Providers
TSQVTA Tracer Survey of Qualified Vocational Training Applicants
VET Vocational Education and Training
VTGF Vocational Training Grant Fund
II. Executive Summary
MCA‐N has contracted NORC at the University of Chicago (NORC) to implement a Tracer Survey of
Qualified Vocational Education and Training (VET) Applicants (TSQVTA). The objective of this survey is to
collect high‐quality data to support the impact evaluation (conducted by Mathematica Policy Research
(MPR)) of the training funded through the Vocational Training Grant Fund (VTGF) established by
Millennium Challenge Account Namibia (MCA‐N) and through the Namibia Training Authority’s (NTA)
National Training Fund (NTF). The impact evaluation will focus on the impact on employment and
income related outcomes. The data collection includes two rounds – a baseline survey and an endline
survey.
This Final Consolidated Survey Report for the Baseline and 12‐Month Follow‐up Rounds is the 14th
deliverable under the contract MCA‐N/COM/RCQ/5A01003‐C, and provides a summary of NORC’s
baseline and endline data collection. It includes the number of interviews completed and various actions
that were taken over the course of data collection, as well as lessons learned for future rounds of similar
data collections for both baseline and endline.
The vocational training programmes under MCA‐N’s intervention occur on a rolling basis and have
different durations. Accordingly, data collection was an ongoing activity to ensure that the evaluation
team would be able to assess the situation before the impact of the training programs for each cohort.
NORC and Survey Warehouse (SW) administered the survey by phone to each group of training
applicants at baseline (immediately after random selection took place) and twelve months1 after
finishing the training. Training applicants are divided in two groups: trainees (treatment group) and
other qualified applicants not receiving training (control group).
NORC’s data collection began on February 26, 2013 and a total of 3,497 surveys were successfully
completed – 1,237 with applicants accepted into training, 2,245 with other qualified applicants, and 15
with respondents for which NORC did not receive selection status.
The sample for NORC’s endline data collection is comprised of 351 eligible2 respondents, from 373
names received from three training providers (TPs).3 NORC’s endline data collection began on March
17, 2014 and a total of 185 surveys were successfully completed.
The Liberty system allows enumerators to directly enter survey responses into the web based system,
allowing for real time logic and consistency checks. Moreover, data is available for download and review
1 Although follow‐up was planned for 12 months after training concluded, preparation of the follow‐up survey (developing the instrument, programming the survey, and training) took longer than expected. Respondents were therefore typically interviewed between 15 to 18 months after their training ended. 2 Names received may be ineligible because (1) they were already listed on a previous TP’s list, (2) the TP did not provide contact information for the individual, or (3) MPR instructed NORC not to collect data due to the absence of a control group. 3 MPR will oversee the data collection for the remainder of follow‐up cases.
instantly and no further data entry is needed. After changes to the survey instruments, NORC updated
the program to reflect these changes (these are documented in the Survey Data and Documentation
Report).
The substantive sections of the report following the Executive Summary start in Section 3, which
provides an introduction to the project. Section 4 provides a summary of the baseline survey data
collection efforts, including tables and graphs. A review of baseline survey events is found in Section 5.
This includes issues with the Liberty system, gaps in interviewing, some sampling issues, and several
changes to the questionnaire.
Section 6 provides a summary of the endline survey data collection efforts, along with endline tables
and charts. A review of endline survey events is found in Section 7. This includes issues with the Liberty
system, gaps in interviewing, some sampling issues, and several changes to the questionnaire.
Finally, in Section 8 we outline some important lessons learned during baseline and endline data
collection. These lessons will allow us to improve the data collection processes for future similar surveys.
III. Introduction
The current Millennium Challenge Corporation (MCC) Compact with the Republic of Namibia aims to
reduce poverty through economic growth in the Education, Tourism and Agriculture sectors. In
Education, the programme seeks to bring the quality of the work force closer to the requirements of
industry and the labour market at large. More specifically, the Millennium Challenge Account Namibia
(MCA‐N) Education Project aims to improve the competency and knowledge of young Namibians by
supporting new and innovative methods of learning in addition to the traditional approaches, and
improved physical infrastructure for learning and teaching in schools, Regional Study and Resource
Centres, and Community Based Skills Development Centres.
As part of the Education component of the compact, the Vocational Training Grant Fund (VTGF) aims to
provide grants to support 1,638 trainees enrolled in high‐priority skills training areas at eligible
institutions. Additionally, by giving grants to Training Providers, it aims to improve the training and
functional capacity of their centers for the delivery of high priority training that is relevant to market
demands.
MCA‐N contracted NORC at the University of Chicago (NORC) to implement a Tracer Survey of Qualified
Vocational Education and Training (VET) Applicants. The objectives of this exercise are to provide up‐to‐
date, reliable, and comprehensive information on the outcome of training funded through the VTGF
established by MCA‐N. It is expected that the information elicited through this data collection exercise
will allow the impact evaluator, Mathematica Policy Research (MPR), to generate reliable statistics on
qualified applicants for VTGF‐supported vocational training programmes, which are all in high‐demand
skill areas. Specifically, it will help to compare income levels and employment of VTGF‐supported
trainees before and after receiving training, relative to qualified applicants who did not receive training
through the VTGF or NTA. The survey and evaluation will also contribute to a broader effort to support
the development of a monitoring and evaluation approach and culture at the NTA.
When dealing with large scale social investment projects, it is essential to ensure that funds are being
used effectively and resulting in the intended outcomes. MCA‐N’s results‐based approach, with its
strong focus on monitoring, impact evaluation, and data quality, will drive programme designs and allow
MCA‐N to make necessary improvements to interventions over the course of the compact. Monitoring
and evaluation efforts, however, are only as good as the data on which they are based. Towards this
end, NORC views itself as a vested partner in MCA‐N’s efforts to collect and employ quality data to
support its M&E activities.
In regards to the evaluation design, two approaches will be used. These include (1) an experimental
design and a double differences analysis; and (2) a complementary qualitative implementation analysis.
As for the evaluation objectives, the key variables of interest are:
Employment
Earnings
Vocational training received
Through measurement of these variables, the current survey will provide data to aid in answering the
question of whether the VTGF‐supported vocational training programmes lead to better employment
and increased income. As such the survey has a strong focus on employment and earnings.
The main research questions the evaluation is expected to address are:4
To what extent did those offered the opportunity of training through the VTGF receive more
training relative to nonfunded, qualified applicants?
To what extent did the VTGF improve employment outcomes for VTGF‐funded trainees relative
to nonfunded, qualified applicants?
To what extent did VTGF‐funded trainees have higher earnings and income relative to
nonfunded, qualified applicants?
To what extent did increased earnings result from increased wages while employed, versus
increased employment?
Did the effects of the training funded by the VTGF vary by trainee characteristics?
What key characteristics or practices of training providers were associated with stronger
impacts on employment and earnings?
This report presents a summary of the endline data collection executed by NORC It includes updates on
the number of interviews completed and various actions that have been taken over the course of data
collection, as well as lessons learned. A Pilot Survey Implementation Report was delivered to MCA‐N on
March 18, 2013 and contains information on the training of enumerators and piloting of the instrument
and field procedures. A Survey Design Report submitted March 4, 2013 describes the sampling
methodology, plans for conducting the baseline survey (including pilot test plan and training plan) and
the questionnaire.
Implementation Timeline
The following timeline is based on contract and project realities as of August 2014. The timeline includes
which organization is responsible for each item.
4 Evaluation questions may change once the evaluation design is finalized.
Table 1: Implementation Timeline
Task Responsible Start Date End Date
Survey Design Report
Draft Survey Design Report NORC/SW Dec. 10, 2012 Jan. 7, 2013
Review of Draft Survey Design Report
MCA‐N & Partners
Jan. 7, 2013 Jan. 18, 2013
Final Survey Design Report NORC/SW Jan. 18, 2013 Mar. 4, 2013
Review of Final Survey Design Report
MCA‐N & Partners
Jan. 24, 2013 Jan. 31, 2013
Sampling
Sample Draw NORC/SW Feb. 1, 2013 Jun. 18, 2014
Review of Sample Draw MCA‐N & Partners
Feb. 3, 2013 Jun. 22, 2014
Baseline Enumerator Training
Manuals/Training Plan NORC/SW Dec. 19, 2012 Jan. 31, 2013
Classroom Training NORC/SW Feb. 12, 2013 Feb. 13, 2013
Pilot Test/Debriefing NORC/SW Feb. 14, 2013 Feb. 15, 2013
Endline Enumerator Training
Manuals/Training Plan NORC/SW November 2013
Classroom Training NORC/SW Feb. 28, 2014 Mar. 4, 2014
Pilot Test/Debriefing NORC/SW Mar. 5, 2014 Mar. 8, 2014
Fielding
Baseline Data Collection NORC/SW Feb. 26, 2013 Aug. 5, 2014
Endline Data Collection5 NORC/SW Mar. 17, 2014 Aug. 5, 2014
5 NTA‐based enumerators conducted baseline data collection for three TPs, and these were ready for follow‐up at the time NORC began data collection. However, NORC and MCA‐N agreed that NORC would begin follow‐up data collection for these respondents only after the evaluator (MPR) joined the team in order to incorporate MPR’s input for the follow‐up version of the survey.
IV. Survey Results – Baseline
Sampling
The core population of interest for TSQVTA consists of qualified applicants to MCA‐N‐supported
vocational training programs. As TPs completed their random selection exercises, NORC obtained the list
of applicants who were randomly selected to receive the training and the list of other qualified
applicants who were not selected. Since the program runs over the life of the Compact over several
intakes of applications to many TPs, the sample is cumulatively constructed and updated each time a
new intake develops. Intakes of training programs were broken down into four waves, which were
divided arbitrarily based on natural breaks in the commencement of the training programs. Applicants
are divided between trainees and qualified applicants not receiving training:
Trainees: Applicants who meet minimum selection criteria determined by training providers
and are randomly selected to receive training (intervention group);
Qualified applicants: Applicants who meet minimum selection criteria determined by training
providers to receive the training but are randomly assigned to not receive the training (control
group).
For each member, we obtained the following information (where available): name, ID number/date of
birth, phone number, gender, training provider, and training level. As instructed by MCA‐N and
discussed with MPR, we included all qualified applicants in the sample. In other words, a complete
census will be conducted to collect data from all qualified applicants, both trainees and other qualified
applicants not receiving training.
The sample of qualified applicants from each control group was expected to be equal to the number of
intervention cases whenever possible to achieve a balanced design. However, this was not always
possible and, in many situations, the control group is much smaller or larger than the intervention
group.
The sample for the first three waves of NORC’s baseline data collection comprises 4,695 eligible6
respondents from a total of 5,121 names received from the twelve training providers (TPs) listed in
Table 2 below.
6 Respondents may be ineligible because (1) they were already listed on a previous TP’s list, (2) they already completed the survey through the NTA‐based enumerators, (3) the TP did not provide contact information for the individual, or (4) MPR instructed NORC not to collect data due to the absence of a control group.
Table 2: Sample Lists Received
Training
Provider Sectors
Names Received
from TP*
Eligible
Trainees
Eligible Qualified
Applicants
Wave 0 (NTA data collection)
NATH Tour Guiding 157 59 98
Wolwedan’s Hospitality Core; Front
Office, House Keeping, F &
B and Food Prep
184 95 89
COSDEC
Luderitz
Plumbing & Pipefitting,
Welding & Metal
Fabrication, Office
Administration
198 78 120
Wave 1
COSDEC
Luderitz
Office Administration,
Welding, Plumbing &
Pipefitting
221 53 86
ABTCC Food & Beverage;
Housekeeping Operations
32 16 16
ILSA Reception Management;
Office Administration
147 100 47
IUM Hospitality & Tourism 390 240 150
OVTC Hospitality & Tourism 57 30 26
NamCol Automotive Mechanics,
Office Administration,
Plumbing & Pipefitting,
Welding & Metal Fabrication
104 96 8
NamWater Electrical General, Water
Care, Plumbing & Pipefitting
1661 69 1176
Wave 2
IUM Tourism 89 56 29
RVTC Auto Electrical, Auto
Mechanic, Elec Gen, Joinery,
Bricklaying, Plumbing
40 40 0
ZVTC Bricklaying, Joinery,
Hospitality, Office Admin,
Welding, Plumbing
599 112 454
NamCol Office Admin, Welding,
Plumbing, Automotive
Mechanics
156 91 42
Training
Provider Sectors
Names Received
from TP*
Eligible
Trainees
Eligible Qualified
Applicants
Wave 3
COSDEC Office Admin, Welding,
Plumbing
331 60 188
NIMT Fitter & Turner, Electrical,
Instrumentation
114 48 66
Kayec Carpentry, Concrete,
Shuttering
84 44 35
Wave 4
COSDEC Office Administration,
Welding, Plumbing &
Pipefitting
93 30 16
VVTC Food Preparation, Food &
Beverage Services, Front
Office, Housekeeping &
Food Preparation
96 35 29
NAMWATER Bulldozer, Forklift, Front‐End
Loader, Grader
71 39 11
NIMT Fitter & Turner, Electrical,
Instrumentation
76 42 27
Kayec Construction Carpentry,
Shuttering, Concrete
221 92 66
Total All 5121 1525 2779
*The number here represents the total number of names received, which is greater than the total number of
eligible respondents loaded into the Liberty system for interviewing. Respondents are counted in this column but
not loaded into Liberty if either: (1) they were already loaded into Liberty based on a previous TP’s list, (2) they
already completed the survey through the NTA‐based enumerators, (3) the TP did not provide contact information
for the individual, or (4) MPR instructed NORC not to collect data due to the absence of a control group.
A key consideration for the current project, sampling plan, and endline analysis deals with the issue of
diminishing controls. Two related concerns have been identified which require some description:
1. Some applicants who were in the previous intake’s control group may appear in the treatment
group of a new intake. In other words, an applicant may have been placed into the control
population, applied a second time for a new intake and then placed in the treatment group.
2. Similar to the above, some applicants may have been placed in the control group for one intake
and then placed in the control group again for another intake after applying unsuccessfully a
second time.
For applicants in both cases, MPR instructed NORC not to re‐interview them, but rather track the status
of each respondent, flagging them in the dataset so we do not repeat a baseline interview with them
each time their name comes up in an intake list. The NORC sample database flags these respondents
based on national ID number and/or phone number for review.
Response Rates
NORC began baseline data collection on February 26, 2013 and a total of 3,497 surveys were successfully
completed (Figure 1).7 An additional 213 interviews were partially completed, i.e. started, paused, and
not yet completed.8 Interviews with the remaining 985 respondents were not started (unable to be
interviewed) – this number includes (1) respondents who were not contacted successfully, (2)
respondents who were contacted but did not begin the survey, (3) respondents with poor contact
information, and (4) refusals, as well as (5) any other respondent who has not answered any survey
question.
Figure 1: Current Status of All Cases
7 In the previous wave of surveys, conducted by enumerators based at the Namibian Training Authority (NTA), an
additional 461 surveys were successfully completed between January and October, 2012. These interviews are
included in the tables and figures presented in this document, except where noted.
8 This includes respondents who have made appointments to complete the survey as well as those who have not.
985
3497
213
Not started CompletePartial
(5%)
Partial
(74%)
Complete
(21%)
Not started
The average length of completed interviews is 23 minutes. On average, respondents were contacted 2
times before completing an interview. 1,869 respondents (55%) completing interviews are female, while
the remaining 1551 (45%) are male (Figure 2).
Figure 2: Gender of Survey Respondents
Of the 3,497 completed interviews, the majority (1,286) were completed in the month of March 2013
(Table 3). In early April 2013, the NORC/SW team of enumerators reported that they had finished
working through the entire available sample. Teams resumed data collection for one week in June (June
7‐13, 2013) and permanently on August 21, 2013 when we had acquired a substantial number of new
respondents to interview. The pauses in data collection were due to the need for additional sample
respondents. Once SW worked through the lists we had, it was necessary to pause until new names
could be added to our list.
Table 3: Completed Interviews by Month
Month Completed Interviews Percent of Total Completes
December 2011 22 0.63
January 2012 200 5.72
February 2012 66 1.89
March 2012 3 0.09
1551
1869
Male Female
(55%) Female
(45%)
Male
Month Completed Interviews Percent of Total Completes
May 2012 50 1.43
August 2012 30 0.86
September 2012 111 3.17
October 2012 57 1.63
February 2013 194 5.55
March 2013 1286 36.77
April 2013 23 0.66
June 2013 42 1.20
August 2013 221 6.32
September 2013 418 11.95
October 2013 118 3.37
November 2013 23 0.66
December 2013 41 1.17
January 2014 70 2.00
February 2014 137 3.92
March 2014 10 0.29
April 2014 108 3.09
May 2014 60 1.72
June 2014 46 1.32
July 2014 161 4.60
TOTAL 3,497 100
There are 985 respondents that were eligible for interview but did not begin the survey. Of these, the
majority (524) are due to bad contact information – either wrong or non‐working/disconnected phone
numbers. The list of respondents with bad contact information was continuously shared with MCA‐N
and the various TPs to obtain updated contact information for these respondents, but updated contact
information was received for only a small subset of respondents.
A summary of the current status of all eligible respondents from NORC’s data collection is shown in
Table 4 below.
Table 4: Final Status of Respondents (NORC Data Collection Only)
Disposition Code Number of Cases Percent
No contact attempts yet made 319 7.68
Non-working/disconnected phone number 175 4.21
Wrong number 36 0.87
Disposition Code Number of Cases Percent
No answer/phone turned off 125 3.01
Busy signal 2 0.05
Network problem 13 0.31
Respondent not at home 1 0.02
Language barrier 1 0.02
Respondent busy/unavailable and rescheduled 2 0.05
Phone problem 1 0.02
Completed by phone9 2,988 71.90
Completed by email/web10 2 0.05
Deceased (no surviving spouse) 1 0.02
Permanently out of country 1 0.02
Final respondent refusal 8 0.19
Final break-off 2 0.05
Final unlocatable 451 10.85
Other 28 0.67
TOTAL11 4,156 100
Quality Control
NORC and SW place a strong emphasis on high quality data gathered in the field. To this end, multiple
supervision and quality control measures took place:
The supervisors regularly reviewed cases to ensure surveys were properly completed,
consistent, and that the respondent was correctly identified. While doing so, the supervisors
were responsible for documenting errors by question and by enumerator as well as corrective
action taken in a Log File. This Log File was used by supervisors as a tool for recognizing
problems or patterns during data collection. This was documented using a paper‐based form.
These forms were periodically reviewed by senior staff.
During data collection supervisors back‐checked 15% of the sample, spread out evenly across all
the enumerators and respondent types (in terms of trainee versus non‐trainee). For the back‐
checks, supervisors called the survey respondents to ask them key survey questions using a
validation form. This validation form includes information such as confirmation that the
interview took place; the approximate time taken by the interview; and checking critical
9 Note that the number marked “completed by phone” does not match the number reported as complete elsewhere in the report. This discrepancy is due to the fact that this table reports on what disposition the enumerator chose, and the numbers reported elsewhere are determined by looking at the survey data to confirm that an entire interview was finished. Additionally, this table does not include the 539 baseline interviews conducted by enumerators based at the NTA. 10 We had briefly entertained the idea of hosting a web survey, but decided early‐on not to move ahead with this. 11 See footnote #9.
variables for completeness. The responses obtained during validation were data entered by
Survey Warehouse staff for comparison to responses obtained during the interview itself.
Given that data is directly entered into the Liberty System, the NORC office in the United States
can track the progress and conduct quality controls in an ongoing basis as data is collected
assuring timely addressing any potential issues that may arise. MCA‐N was kept up‐to‐date on
these issues through weekly reports while data collection is active.
Supervisors observed (involving listening in on) 5% of interviews to ensure the proper execution
of the survey and provide constructive feedback to enumerators.
Directly after conducting a call, enumerators documented (in the Liberty System) the result of
the call, delineating the reasons for non‐response (i.e., if due to lack of availability or refusal),
documenting the efforts made to find the respondent (if they were not spoken to), when and
how many additional attempts were made, and contact information of any people they spoke to
about the missing person’s whereabouts. If the respondent refused to participate, the
enumerator documents the reason for refusal. In all cases the enumerators were instructed to
make at least three attempts to engage the respondent. Disposition data is kept in a dataset
separately from the survey data.
While conducting interviews, enumerators used a paper‐based form to track issues such as item
non‐response or system/computer errors during the interview, and enter the data directly into a
comment section in Liberty once the interview is complete.
Data Entry and Cleaning using the Liberty System
The NORC/SW team utilized a web based data entry system named Liberty. The Liberty survey
application allows for consistency and range checks, question skip patterns, and other logic features
necessary for surveys. Liberty is a generalized data collection and performance measurement
framework which includes modules for user management, surveys, and email campaigns. These
modules have been integrated into Liberty, providing a robust base layer on top of which custom data
collections instruments can be built and deployed.
In terms of the current project, the Liberty system allowed enumerators to directly enter survey
responses into the web based system, allowing for real time logic and consistency checks. Moreover,
data was available for download and review instantly and without additional data entry.
Changes to the survey instruments occurred after the evaluator, MPR, joined the project and NORC
updated the Liberty program to reflect these changes. A document highlighting the changes to the
instrument is available and included in the Survey Data and Documentation deliverable.
V. Review of Survey Events – Baseline
This section reviews various important events that have occurred during the course of data collection to
date. These events were detailed in weekly reports to MCA‐N while data collection was active.
Field Problems/Issues
Monday, February 25, 2013. In the process of making adjustments to the data collection instrument
between pilot testing and data collection, all enumerator accounts were inadvertently deleted. The user
accounts were re‐created as soon as this issue was encountered, but it caused a delay in the start of
data collection from the planned date of Monday, February 25, 2013 to the actual date of Tuesday,
February 26, 2013. As this issue was related to the transition from pilot testing to data collection, there
is little to no risk of reoccurrence.
Thursday, February 28, 2013. NORC’s server was shut down for regular maintenance for approximately
3 hours on the morning of Thursday, February 28, 2013, preventing SW’s morning team from working
for the duration of the maintenance. The problem was resolved in time for the afternoon team at SW to
complete their shift on February 28, 2013. The maintenance schedule for this server has been altered to
avoid future disruptions in the data collection activities.
Wednesday, March 6‐13, 2013. On February 27, 2013, MCA‐N provided NORC with sample files
including respondents from Cosdec Luderitz’s 2nd Intake and NAMWATER. NORC found duplicate
respondents in this file, thereby delaying the upload of these respondents into the Liberty system until
the duplicates could be documented and removed. This process temporarily stalled data collection on
the afternoon of March 6, 2013 as SW teams ran out of respondents and sat idle as NORC prepared the
new sample for upload. On March 6, 2013, SW notified NORC of some performance issues within Liberty
whereas page loading times can take longer than expected within a survey. The issue seems to be
infrequent and SW suggested that their internet service provider may be a bottleneck; other NORC
projects utilizing the Liberty system have reported no issues. Nevertheless, SW continues to document
such instances so NORC can investigate to see if there are any improvements that can be made to
mitigate the issue. SW has sent bug reports on March 8 and March 13, 2013; NORC investigated possible
system improvements, but no issues were found. No impact on overall survey progress was noted by
SW.
Friday, March 22, 2013. SW notified NORC that they had exhausted the respondent lists received to
date. As a result, the teams worked for only half of the day.
Tuesday, March 26 to Thursday, April 4, 2013. Survey teams did not work during this period due their
exhausting cases on Friday, March 22, 2013. Supervisors used this week to clear back logs of planned
validations, and teams resumed collecting data once an appropriate number of new respondents were
loaded into the Liberty system.
Saturday, April 6 to Tuesday, June 6, 2013. Survey teams did not work during this period due their
exhausting cases on Friday, April 5, 2013.
Friday, June 14 to August 20, 2013. Survey teams did not work during this period due their exhausting
cases on Thursday, June 13, 2013.
Tuesday, August 20, 2013. Prior to resumption of data collection on August 21, 2013 Survey Warehouse
held a one‐day refresher training for all enumerators, due to questionnaire changes that occurred when
MPR came on board and due to the long break.
Friday, August 23, 2013. Shortly after data collection resumed, MPR visited Survey Warehouse’s offices
to observe data collection. During this visit, the team observed enumerators translating surveys to
respondents on the fly. Survey Warehouse reiterated to enumerators that this behavior is not allowed,
and asked them to conduct interviews in English only and, when interviews cannot be conducted in
English, to choose the disposition code for “Language barrier”. Survey Warehouse monitored this issue
in the following week, and found that only one language barrier was encountered among 178 completed
interviews (depending on whether there was any non‐reporting, this number may actually be slightly
higher). NORC created a tracking sheet for enumerators to further track language issues in more detail.
Between October 25 and November 6, 2013, SW enumerators used the tracking sheet for a total of 110
attempted interviews. A “Memo on Tracer Survey Language Barriers” was sent to MCA‐N and MPR on
November 21, 2013.
Tuesday, October 29, 2013 to Friday, November 8, 2013. Survey Warehouse contacted training
providers through Martin Wilkinson at MCA‐N to ask for updated contact information for the 779
respondents that enumerators found to have incorrect or disconnected phone numbers. We received
updated contact information for subsets of respondents from OVTC and ZVTC; the remaining TPs
contacted either reported that they sent us all of the information that they have or were unresponsive.
Friday, November 8, 2013 to Monday, December 16, 2013. No data was collected during this period for
two reasons. First, Survey Warehouse thought that they had exhausted all available cases but upon
investigation NORC identified a number of cases warranting further attempts. Second, for a portion of
this period Survey Warehouse’s phone shop was occupied for another project.
Saturday, December 21, 2013 to Sunday, January 5, 2014. No data was collected during this period due
to the Christmas and New Year holidays.
Friday, March 7, 2014 to Thursday, April 10, 2014. No data was collected during this period due to the
exhaustion of all available cases.
Questionnaire Changes
Since the launch of data collection on February 26, 2013 several important changes to the questionnaire
have been implemented:
A Facebook page was created to assist in tracking respondents across survey rounds. Text has
been added to the end of the questionnaire instructing enumerators to invite respondents to
“Like” this Facebook page. The page received three “Likes”.
MCA‐N has requested more detailed information on the town of origin of respondents.
Previously, relevant information was collected in three places: (1) The region of the
respondent’s current physical address; (2) The region of the respondent’s current postal
address, and; (3) The full address where the respondent can best be reached (either physical or
postal). However, NORC found that the addresses noted in (3) were often incomplete and
sometimes did not include the actual town name. As a result, the information was not available
in the level of detail necessitated by MCA‐N. After consultation with MCA‐N, a new question has
been added, which has two multiple choice options (one for town, and one for region):
o What is the name of your town of origin and its region? By “town of origin” I mean the
town where you grew up, and the region where it is located.
After a review of the questionnaire, a number of changes were requested by the evaluator
(MPR). The changes ranged in nature from wording changes to question additions. NORC made
the requested changes before data collection resumed on August 21.
During the re‐training session on August 20, enumerators noted that respondents typically have
difficulty in answer question E.2. MPR instructed NORC to remove this question.
Integrating Previous Data
For the current project it is necessary to understand the relationship between previous data and
database structures and the new methods being employed. The current project employs a web based
data collection methodology in which data is automatically entered as the interview takes place. The
Liberty system allows the developer to define variables either in advance or after data collection has
taken place. Variable names and labels, value labels, and response categories are all defined in Liberty to
match the questionnaire.
Prior to NORC’s involvement in the project, NTA‐based enumerators conducted baseline interviews with
trainees and qualified applicants of three TPs – NATH, Wolwedan’s, and COSDEC Luderitz. From a
sample of 551 respondents, 461 interviews were reported as complete.
NORC obtained a copy of NTA’s response database from MCA‐N. Though it used its own proprietary
variable names and labels, NORC has successfully adapted the NTA‐based enumerators’ database to
match the database produced by Liberty. This will allow the project team an integrated view of all data
that has been collected since January, 2012.
The evaluation consultant, MPR, has joined the team in mid‐2013; the database has been shared with
MPR as part of the survey data and documentation deliverable.
VI. Survey Results – Endline
Sampling
The core population of interest for TSQVTA consists of qualified applicants to MCA‐N‐supported
vocational training programs. As TPs completed their random selection exercises, NORC obtained the list
of applicants who were randomly selected to receive the training and the list of other qualified
applicants who were not selected. Since the program runs over the life of the Compact over several
intakes of applications to many TPs, the sample is cumulatively constructed and updated each time a
new intake develops. Applicants are divided between trainees and qualified applicants not receiving
training:
Trainees: Applicants who meet minimum selection criteria determined by training providers
and are randomly selected to receive training (intervention group);
Qualified applicants: Applicants who meet minimum selection criteria determined by training
providers to receive the training but are randomly assigned to not receive the training (control
group).
For each member, we obtained the following information (where available): name, ID number/date of
birth, phone number, gender, training provider, and training level. As instructed by MCA‐N and
discussed with MPR, we included all qualified applicants in the sample, both those selected to receive
training and other qualified applicants not receiving training.
The sample of qualified applicants not receiving training (the control group) was expected to be equal to
the number of those selected to receive training (the intervention group) for each training whenever
possible to achieve a balanced design. However, this was not always possible and, in many situations,
the control group is much smaller or larger than the intervention group for a particular training.
The sample for NORC’s endline data collection was comprised of 351 eligible respondents from a total
of 373 names received from the three training providers (TPs) listed in Table 5 below.
Table 5: Sample Lists Received
Training
Provider Sectors
Names
Received from
TP*
Eligible
Trainees
Eligible
Qualified
Applicants
NATH Tour Guiding 157 58 96
Wolwedan’s Hospitality Core; Front
Office, House Keeping, F
& B and Food Prep
184 90 75
ABTCC Food & Beverage;
Housekeeping Operations
32 16 16
Total All 373 164 187
*The number here represents the total number of names received, which is greater than the total number of
eligible respondents loaded into the Liberty system for interviewing. Respondents are counted in this column but
not loaded into Liberty (not counted as eligible) if either: (1) they were already loaded into Liberty based on a
previous TP’s list, (2) the TP did not provide contact information for the individual, or (3) MPR instructed NORC not
to collect data due to the absence of a control group.
A key consideration for the endline survey, sampling plan, and endline analysis deals with the issue of
diminishing controls. Two related concerns have been identified which require some description:
3. Some applicants who were in the previous intake’s control group may appear in the treatment
group of a new intake. In other words, an applicant may have been placed into the control
population, applied a second time for a new intake and then placed in the treatment group.
4. Similar to the above, some applicants may have been placed in the control group for one intake
and then placed in the control group again for another intake after applying unsuccessfully a
second time.
For applicants in both cases, MPR instructed NORC not to re‐interview them, but rather track the status
of each respondent, flagging them in the dataset so we do not repeat an endline interview with them
each time their name comes up in an intake list. The NORC sample database flags these respondents
based on national ID number and/or phone number for review.
Response Rates
NORC began endline data collection on March 17, 2014 and a total of 185 surveys were successfully
completed (Figure 3). An additional 54 interviews were partially completed, i.e. started, paused, and not
yet completed.12 Interviews with the remaining 112 respondents were not started (unable to be
interviewed) – this number includes (1) respondents who were not contacted successfully, (2)
respondents who were contacted but did not begin the survey, (3) respondents with poor contact
12 This includes respondents who have made appointments to complete the survey as well as those who have not.
information, and (4) refusals, as well as (5) any other respondent who has not answered any survey
question.
Figure 3: Current Status of All Cases
The average length of completed interviews was 36 minutes13. On average, respondents were contacted
5 times before completing an interview14. 107 respondents (58%) completing interviews were female,
while the remaining 78 (42%) were male (Figure 4).
13 This is computed using only observations under 90 minutes to remove surveys completed in multiple sessions. 14 Although enumerators are asked to attempt 3 times per case, there are a handful of cases in which the enumerators attempted many more times, which significantly increased the average.
112
185
54
Not started CompletePartial
(15%)
Partial
(53%)
Complete
(32%)
Not started
Figure 4: Gender of Survey Respondents
a
Of the 185 completed interviews for which date information is available, the majority (148) were
completed in the month of March 2014 (Table 6).
Table 6: Completed Interviews by Month
Month Completed Interviews Percent of Total Completes
March 2014 148 80.00
April 2014 16 8.65
May 2014 5 2.70
June 2014 3 1.62
July 2014 13 7.03
TOTAL 185 100
There were 112 respondents that were eligible for interview but did not begin the survey. Of these, the
majority (57) were due to bad contact information – either wrong or non‐working/disconnected phone
78
107
Male Female
(58%) Female
(42%)
Male
numbers. 15 MPR provided a list of updated contact information based on baseline survey responses;
however, only 2 respondents were able to be contacted through use of the list (in other cases,
respondents were able to be contacted with use of the primary contact information, or unable to be
reached with the alternate contact information due to a nonworking phone number).
A summary of the current status of all eligible respondents from NORC’s data collection is shown in
Table 7 below.
Table 7: Current Status of Respondents
Disposition Code Number of Cases Percent
No answer/phone turned off 7 1.99
Busy signal 1 0.28
Network problem 1 0.28
Respondent busy/unavailable and did not reschedule 1 0.28
Completed by phone16 183 51.99
Deceased (no surviving spouse) 2 0.57
Final unlocatable17 99 28.13
Other18 57 16.48
TOTAL 351 100
Quality Control
NORC and SW place a strong emphasis on high quality data gathered in the field. To this end, multiple
supervision and quality control measures took place:
The supervisors regularly reviewed cases to ensure surveys were properly completed,
consistent, and that the respondent was correctly identified. While doing so, the supervisors
were responsible for documenting errors by question and by enumerator as well as corrective
action taken in a Log File. This Log File was used by supervisors as a tool for recognizing
problems or patterns during data collection. This was documented using a paper‐based form.
These forms were periodically reviewed by senior staff.
During data collection supervisors back‐checked 15% of the sample evenly across all the
enumerators and respondent types (treatment and control). For the back‐checks, supervisors
called the survey respondents to ask them key survey questions using a validation form. This
15 The remaining were largely “Other”, an explanation for which is provided in footnote #12. 16 Note that the number marked “completed by phone” (183) does not match the number reported as complete elsewhere in the report (185). This discrepancy is due to the fact that this table reports on what disposition the enumerator chose, and the numbers reported elsewhere are determined by looking at the survey data to confirm that an entire interview was finished. 17 Cases have been attempted an average of 11 times before being marked as “final unlocatable”. They mostly seem to be cases with bad contact information. 18 The explanation for these are typically left blank or miscoded.
validation form includes information such as confirmation that the interview took place; the
approximate length of the interview; and checking critical variables for completeness. The
responses obtained during validation were data entered by Survey Warehouse staff for
comparison to responses obtained during the interview itself. When soliciting feedback
supervisors noted that although respondents largely did not have feedback on the survey, those
who did were mostly concerned about how the information would be used and what was the
purpose of the survey.
Given that data is directly entered into the Liberty System, the NORC office in the United States
tracked the progress and conduct quality controls on an ongoing basis as data is collected
assuring timely addressing any potential issues that arose. MCA‐N was kept up‐to‐date on these
issues through weekly and bi‐weekly reports while data collection was active.
Supervisors directly observed 5% of interviews to ensure the proper execution of the survey and
provide constructive feedback to enumerators.
Directly after conducting a call, enumerators documented (in the Liberty System) the result of
the call, delineating the reasons for non‐response (i.e., if due to lack of availability or refusal),
documenting the efforts made to find the respondent (if they were not spoken to), when and
how many additional attempts were made, and contact information of any people they spoke to
about the missing person’s whereabouts. If the respondent refused to participate, the
enumerator documented the reason for refusal. In all cases the enumerators were instructed to
make at least three attempts to engage the respondent. Disposition data was kept in a dataset
separately from the survey data.
While conducting interviews, enumerators used a paper‐based form to track issues such as item
non‐response or system/computer errors during the interview, and entered the data directly
into a comment section in Liberty once the interview was complete.
Data Entry and Cleaning using the Liberty System
The NORC/SW team utilized a web based data entry system named Liberty. The Liberty survey
application allows for consistency and range checks, question skip patterns, and other logic features
necessary for surveys. Liberty is a generalized data collection and performance measurement
framework which includes modules for user management, surveys, and email campaigns. These
modules have been integrated into Liberty, providing a robust base layer on top of which custom data
collections instruments can be built and deployed.
In terms of the current project, the Liberty system allowed enumerators to directly enter survey
responses into the web based system, allowing for real time logic and consistency checks. Moreover,
data was available for download and review instantly and without additional data entry.
VII. Review of Survey Events – Endline
This section documents all issues encountered on the TSQVTA endline data collection since the first
interviews were completed on March 17, 2014.
Field Problems/Issues
Monday, March 17, 2014. Survey Warehouse reported two errors in the Liberty programming – both D.2 and D.5 were not allowing responses of zero. Both issues were fixed immediately.
Monday, March 17, 2014 to Friday, April 18, 2014. During this period, NORC was unable to produce weekly reports due to issues with reporting within Liberty whereas the baseline and follow‐up data were exported as one database. NORC’s IT staff resolved the issue without any loss of data or interruptions in data collection.
Tuesday, March 18, 2014. Survey Warehouse reported an instance where data was erased upon moving backwards in the survey instrument. NORC’s IT staff checked for errors and tested the instrument, but was unable to replicate the problem. On March 25, Survey Warehouse confirmed that the issue was not widespread and that there were no new instances. Survey Warehouse and NORC will continue to monitor any similar issues.
Wednesday, March 26, 2014. Survey Warehouse reported that case 6234 was inadvertently entered in
the survey for case 5989. Upon discovering the error, Survey Warehouse addressed the issue with the
enumerator.
Friday, April 4, 2014 to Wednesday, April 9, 2014. No data was collected during this period due to the
exhaustion of all available cases.
Wednesday, April 16, 2014 to Monday, April 21, 2014. No data was collected during this period due to
the exhaustion of all available cases.
Monday June 2, 2014 to Monday, June 23 2014. During this period, there was a lapse in data collection
since Survey Warehouse’s phone data collection center was finishing up another activity.
VIII. Lessons Learned
Baseline & Follow‐up Rounds
The nature of the TSQVTA, with several waves and multiple rounds of data collection, is advantageous in
that it allows the refining of methods between survey rounds. It is important that we take advantage of
this opportunity to ensure the highest quality data possible. While the first wave of data collection has
went smoothly, there are, of course, areas for improvement that can make the project even better. To
this end, we present some lessons learned from our data collection.
Participant tracking and contact information. For the purpose of participant tracking, it is
important to have complete information prior to submitting a list to be added to the sample.
This includes contact information, such as name and phone number, as well as
treatment/control distinctions. In several cases, TPs submitted multiple versions of the same list,
sometimes after interviewing had already begun on the list by the NORC/SW team. This
situation adds a layer of difficulty to maintaining the database of respondents and heightens the
risk for error. Similarly, many lists had instances where phone numbers were non‐existent or
incorrect, resulting in an inability to interview these respondents until TPs were able to provide
updated information.
Enumerator attrition. Long lags between survey rounds heighten the risk of enumerator
attrition. Upon resumption of data collection activities in August 2013, NORC/SW attempted to
rehire all 12 enumerators who had previously worked on the study, but only half were available
for employment. Given the rate at which enumerators completed surveys in wave 1 and at
which selections have taken place, we decided not to hire any new enumerators.
Disposition results. It is vitally important for enumerators to consistently select accurate
disposition results for each contact attempt. Disposition codes are used for tracking survey
progress and reporting to MCA‐N, so inaccuracies can cause deficiencies in reporting. During the
first wave, about 50 cases received disposition codes which made them appear to be cases for
which an interview was not possible; upon review, the NORC/SW management team found that
they should be pursued further. This topic was covered in the enumerator re‐training session
prior to the next wave of data collection.
Language issues must be tracked carefully. During the initial months of baseline data collection,
language issues were tracked intermittently and enumerators were found translating on the fly.
In the future, these issues should be tracked closely so that the questionnaire can be translated
if necessary.
Contact information should be carefully recorded and maintained by training providers. The
NORC/SW team received many names and phone numbers which were found to be inaccurate.
Enumerators found that a number of phone numbers were disconnected, out of service, never
picked up, and had other problems. While NORC/SW followed up with training providers, we
achieved only limited success in obtaining updated information. As such, these respondents
were never reached for surveying. MPR provided a list of updated contact information based on
baseline survey responses for endline survey follow‐up; however, only 2 respondents were able
to be contacted through use of the list (in other cases, respondents were able to be contacted
with use of the primary contact information, or unable to be reached with the alternate contact
information due to a nonworking phone number).