redirected inbound call sampling - reconnect research · redirected inbound call sampling (rics)–...
TRANSCRIPT
www.rti.orgRTI International is a registered trademark and a trade name of Research Triangle Institute.
Redirected Inbound Call Sampling (RICS)–An Emerging Fit for Purpose Non-probability Telephone Sample Design
Burton Levine
Insights Webinar
April 9, 2019
Need and Problem
Need:
Organizations must be in a position to provide objective, accurate, and timely
information that informs decision making.
Problem:
“Continued reliance on (traditional*) sample surveys as the principal means of
collecting statistical data is threatened by the increasing difficulty and cost in
conducting the surveys, with consequent threats to data quality, and by the
increasing demand for more and faster information.”
*added by the presenter
-Innovations in Federal Statistics: Combining Data Sources While Protecting Privacy (2017)
RICS is a Potential Solution
Redirected Inbound Call Sampling (RICS)
Low cost
– About one-tenth the cost compared to outbound telephone sampling
Quick
– Can collect data on thousands of respondents a day
Low burden
– The respondent is already on the phone
Introducing Redirected Inbound Call Sampling (RICS) Surveys
RICS survey participants come from:
Calls to nonworking toll-free numbers (e.g., Area Code: 800, 888)
Direct inward dialing (toll) calls that fail to connect to their intendent target
RICS Data Collection Methods
VoiceKeypad
RICS Data Collection
Methods
Interactive
Voice
Response
WebLive
Interviewer
Redirecting Inbound Toll-free Calls is Commonplace
Redirecting inbound calls is
commonplace
9% of toll-free number are redirected
5.4 million toll-free telephone numbers
are redirected.
Area
code
n Redirected
(%)
800 126 29.4
844 126 0.8
855 126 0.0
866 126 7.9
877 126 7.1
888 126 8.7
Total 756 9.0 (7.0, 11.0)
Innovation is applying RICS to population surveillance.
RICS Surveys Fielded (and in the field) by Presenter
Date SurveyFunding
organizationRespondents
September 2015 BRFSS Evaluation Study RTI 6,799
October 2016 National Adult Tobacco Survey NYS-DOH 4,302
March 2017 NHIS Evaluation Study RTI 9,478
April 2017 New York City Sleep Study NYC-DHMH 1,532
August 2017 National Adult Tobacco Survey NYS-DOH 4,599
February 2018 National Adult Tobacco Study NYS-DOH 4,091
August 2018 RICS-to-Web pilot FL-DOH 500
Date SurveyFunding
organizationRespondents
March 2019 Informed consent user experience
survey
RTI 6,000
March 2019 National Adult Tobacco Survey NYS-DOH 4,000
Completed studies
Studies in progress
Studies in red are evaluation studies.
Studies in black are commissioned by government clients.
Future RICS Surveys by Presenter
Future studies
Date SurveyFunding
organizationRespondents
2019 NY State Tribal Cigarette sales NYS-DOH 14,000
2019 May---2024
(quarterly)
NY- Adult Tobacco Survey RICS-to-CATI Pilot NYS-DOH 1,500 per quarter
(for 5 years)
2019--2024 National Adult Tobacco Survey RICS-to-CATI* NYS-DOH 4,000 per year
(for 5 years)
2019--2024 California Tobacco Control Program—Media
Evaluation Study
CA-DOH 700 per month
(for 5 years)
NHIS Evaluation Study
The results of the NHIS evaluation study have been accepted for publication
in the AAPOR journal Public Opinion Quarterly
We created a 27-question instrument that mimics questions from the
National Health Interview Survey (NHIS) - Adult sample
We recruited inbound callers to complete the interview via interactive voice
response (IVR) system.
Issue: we fielded continuous questions as categorical because of high item
nonresponse.
NHIS Evaluation Study—Data collection metrics
Data collection for two separate one-week periods in 2017
Inbound calls Respondents
AAPOR4
response rate
137,840 9,478 7.7%
Median interview length:
NHIS Evaluation Study— Comparison of demographic distributions among: population, RICS respondents and BRFSS respondents
RICS UWE : 1.25 /1.0 = 1.25
BRFSS UWE : 4.1 / 3.2 = 1.28
NHIS Evaluation Study—Night owls
We define night owls as individuals who respond
to the survey between 9pm and 9am.
31% of respondents are night owls
NHIS Evaluation Study—Night owls (continued)
On average, compared to day-timers, night owls
are:
• More male
• Younger
• Less White-NH
• Lower educated
These are the groups that are underrepresented
in outbound telephone surveys.
NHIS Evaluation Study— Calculating Sampling Weights
Demographic distributions for calibration: Sex (2-levels)
Age category (6-levels)
Race/ethnicity (5-levels)
Educational attainment (4-levels)
Census division (9-levels)
We created delete-one jackknife weights (30 groups)
Base
Weight
or
Pseudo-
Inclusion
Probabilities
Calibrate to
demographic
distributions
NHIS Evaluation Study— Comparing categorical outcomes
NHIS Evaluation Study— Comparing continuous outcomes
NHIS Evaluation Study—Primacy Effect
If you get sick or have an accident, how worried are you that you
will be able to pay your medical bills?
Order RICS Order 1 RICS Order 2
1 Very worried Not at all worried
2 Somewhat worried Somewhat worried
3 Not at all worried Very worried
IVR Design Recommendations
Recommendations based on the IVR evaluation study:
Allow continuous responses
Use global “prefer not to answer” prompt
Add a softball question before the first study question
For the most important questions do not allow the respondent to
“barge-in”
Comparison of demographic distributions—Investigate Representativeness
Standardized calibration adjustment index (SCAI)
Design weights used in the SCAI in general
Are based on the inverse of the selection probability.
Have not been adjusted for nonresponse nor trimmed.
Sum to the population total.
In RICS the design weights are
Equal for all respondents (i.e., the sampling mechanism mimics a SRS), and
Sum to the population total
Standardized calibration adjustment index (SCAI)
SCAI = 0.5 ∗ 100 ∗σ⋁𝑖 𝐷𝑒𝑠𝑖𝑔𝑛 𝑊𝑇𝑖 − 𝐴𝑛𝑎𝑙𝑦𝑠𝑖𝑠 𝑊𝑇𝑖
σ⋁𝑖 𝐴𝑛𝑎𝑙𝑦𝑠𝑖𝑠 𝑊𝑇𝑖
SCAI—Interpretation
If there was only one calibration category with two levels (i.e., sex) the SCAI is the absolute
value of the difference in the percentage of one category, using the design weight, and the
population.
Percent male
SCAIPopulation
(%)Design
weights (%)
50 30 20
50 40 10
50 50 0
50 60 10
50 70 20
Investigate Representativeness with SCAI
Year Study Frame Mode N SCAI
2014 NY-Adult Tobacco Study ABS PAPI 3,899 24.6
2015 NY-Adult Tobacco Study ABS Web/PAPI 2,802 29.8
2016 NY-Adult Tobacco Study ABS Web/PAPI 1,872 25.3
2017 NY-Adult Tobacco Study Telephone CATI 3,286 16.9
2016 National Adult Tobacco Study Telephone CATI 1,582 19.5
2017 National Adult Tobacco Study Telephone CATI 1,690 19.6
2016 National Adult Tobacco Study RICS IVR 4,302 19.0
2017 National Adult Tobacco Study RICS IVR 4,599 20.2
2018 National Adult Tobacco Study RICS IVR 4,060 22.2
Average
SCAI
26.6
18.7
20.5
Take away—The RICS IVR data matched the population better than ABS and
almost as well as outbound telephone.
ABS-address based sample PAPI-paper and pencil interview
CATI-computer assisted telephone interview IVR-interactive voice response
Next Frontier of RICS Methodology at RTI—RICS IVR-to-CATI
We piloted collecting New York Adult Tobacco Survey data using
RICS IVR to screen calls
Eligible sample members are redirected to CATI data collection
We obtained 500 CATI respondents using 500 interviewer hours
In comparison outbound telephone sampling takes about 4 interviewer hours
per complete.
Good Fits for RICS IVR Methodology
RICS methodology with IVR data collection require:
Short survey instrument (up to 35 questions)
– Break-offs
Simple questions
Target geographies comprised of an area code or groups of area codes
Minimal Screening Criteria
Extensive screening increases costs
Examples of good fits for RICS IVR
Surveillance of population fleeing a natural disaster (Hurricane Harvey)
National or state surveillance of gastroenteritis (stomach flu)
Next Steps
Work in progress
Informed consent / user experience study
Evaluate RICS IVR-to-CATI data for representativeness and bias
Develop methodology to optimally design and combine data
collections that use both RICS and outbound CATI
