Mode Effects in aNational Survey ofBusinessEstablishments
May 19, 2013
Introduction: GoalTo explore several aspects related to the introductionof multi-mode survey administration in the 2012 FamilyMedical Leave Act (FMLA) Survey of Establishments
– Compare item response rates toadministrative data questions between the2000 and 2012 surveys
– Potential mode effects between web andtelephone responses (Phillips, Harris,Turakhia 2012. Potential mode effects)
Acknowledgements
• Colleagues at Abt Associates: JacobKlerman, Alyssa Pozniak, Katherine Wen
• Abt SRBI team: Lisa Currie, CourtneyKennedy, Julie Pacer, Chintan Turakhia
• Department of Labor, Wage and HourDivision
3
2012 FMLA Worksite Survey• Sponsor: United States Department of Labor (DOL)
• Research Goal: obtain estimates of the use ofleave under the FMLA and examine the impact onU.S. private business establishments
• History:– Conducted twice prior: 1995 and 2000, telephone only– 2012: questionnaire revisions to capture changes in statutes– Addition of web mode
Web Option for 2012
• Reporting administrative or historical datamay be difficult or burdensome tocomplete via telephone survey.
• Internet option flexibility• Flexibility:
increase response rates, produce higher quality data, potentially cost reduction (See Pacer et al,poster)
Study Design• Sample Frame: 2012 Dun & Bradstreet Market Identifiers
• Target Population: private business establishments– (excluded: self-employed respondents without employees,
government and quasi-government units)
• Stratification: the cross-classification of size (number ofemployees) and North American Industry Classification System(NAICS) grouping
• Over-sampling: establishments in two size categories (20-49 and50-99) to accommodate analysis needs (straddle the FMLAcoverage threshold of 50 employees)
• Survey Questions: based on 2000 Survey, updated to reflecteconomic and statutory changes, several new questions, newoptions for reporting period
Data Collection ProtocolTwo Phases:
1. Screener in CATI (Verified, eligible 6,943/8229)• Verify existence of establishment & matched sample• At least 1 employee• Identify MOST KNOWLEDGEABLE RESPONDENT
(MKR)• Gather contact information for MKR
2. MAIN interview (1812: 634 WEB, 1178, Phone)• Pre-notification letter with WEB link & password• Email reminders sent when available• Follow-up telephone calls, maximum recall 10 attempts
Item Response 2000 and 2012QuestionNumber Question Text 2000 2012
PERCENT GAVE A RESPONSE
Q19 How many of those employees took leave thatyou classified as being under FMLA? 50% 95%
98%Web
92%Phone
Q20
Can you please provide the total number ofseparate LEAVES taken in this same time period?A leave is time taken off for a single reason; thiscould be taken all at once or intermittently overtime.
65% 78%
90%Web
67%Phone
8
Potential Mode EffectsTwo types of mode effects:
1. Reporting administrative data
2. Social desirability– Expected to be minimal
• Compliance issues related to FMLA might be harder toreport to an interviewer
• Opinion questions on impact of FMLA on the business(ease of administration, productivity, profitability)
• Other behaviors that signal “less generous” employer
Estimating Mode Effect
• Use matching design to estimate averagetreatment effect for the treated (ATT)– How telephone respondents would have answered
questions had they received web mode– Sum of responses of web respondents minus sum of
responses of their alters• Matching designs increasingly used for causal
analysis in quasiexperimental designs (c.f. Morganand Winship 2007)– ATT is equal to mode effect
• Variables recoded in direction of expected effect– Negative ATT coefficient associated with social
desirability/satisficing on phone mode– Z-scored for uniform interpretation
Distance Measure• Matching by propensity score generated by logit
model predicting web mode by:– Firm Size– Industry classification (2-digit NAICS code)– Email address provided in screener
• Site size and single site tested but not significant• Resulting propensity scores used in matching
algorithm• Radius matching with radius of .05 (i.e.,
probability of selection into web more ± 5% ofeach web case)
Estimating Mode Effects I• 8 “reporting” variables used;
– “Satisficing” measured as rounded to 10 versus not-rounded number to 10 on:
• Q19 Number of employees who took leave under FMLA• Q20 Number of separate leave occasions• Q21 Number of employees who took INTERMITTENT leave• Q26 Number of medical certifications accepted• Q26a Number of medical certifications returned• Q40 Number of suspected “misused” leaves• Q58 Number of employees who took leave (non-covered
establishments)
Estimating Mode Effects I• 3 variables have ATT≠0 variables have significant
differences (α=.05) after Šidák correction for multipletesting
• Q19 Number of employees who took leave under FMLA• Q20 Number of separate leave occasions• Q21 Number of employees who took INTERMITTENT leave• Q26 Number of medical certifications accepted• Q26a Number of medical certifications returned• Q40 Number of suspected “misused” leaves• Q58 Number of employees who took leave (non-covered
establishments)
Estimating Mode Effects II
• 50 “Social Desirability Variables Used”
– 12 FMLA compliance questions– 14 “Opinion” questions (ease of administration,
impact on profitability, productivity)– 24 Employee benefits questions
Social DesirabilityQ11e Paid Paternity LeaveQ46_2 Do you offer the same family and medicalleave benefits to employees who are NOT eligiblefor FMLA because they are {hourly employees}Q46_3 Over the years, has complying with the FMLAincreased, decreased, or not changed the following?Q49d “Other” costs change as a result of FMLA(decreased)Q52 Has complying with FMLA resulted in any costsavings at this location? (opposite direction)Q67a How easy or difficult is it for your company todeal with the following types of leaves?
15
Conclusions
• Item response rates on reporting dataimproved from 2000• Multiple strategies to ease burden (pre-notification
letter; choice in reporting period; percentageestimation)
• Evidence of satisificing as measured via“rounding”, although we do not know if anyloss of precision
• Implications for data quality unclear
Considerations & Next Steps
• Limited in the data available to predictmode
• Did not collect characteristics ofrespondent Exploring Phase I data forfurther possibilities
• Did not collect information about whetherrespondent consulted records Exploringparadata
17
Appendix Matching Design
Propensity Score
Mea
sure
Web
Phone
Included matches
Treatment Effect
Radius matching used. Average of matches is used. Usedpsmatch2.ado in Stata.
Radius
Appendix: Social Desirability
20
-1.500
-1.000
-0.500
0.000
0.500
1.000
q11_
aq1
1_b
q11_
cq1
1_d
q11_
eq1
1_f
q11_
gq1
1_1
q14_
aq1
4_b
q14_
cq1
4_d
q15
q16_
aq1
6_b
q16_
cq1
6_d
q16_
eq1
6_f
q16_
gq1
6_h
q16x
_1q1
6x_3
q16x
_5q1
6x_7
q16x
_8 q17
q21a q2
7q2
8q3
0q3
2q3
4q4
0sd
q46_
1q4
6_2
q46_
3q4
9_a
q49_
bq4
9_c
q49_
dq4
9_e
q52
q53
q54c
ombo
q55c
ombo q5
6q6
7_a
q67_
bq6
7_c
q67_
dq6
7_e
Aver
age
Trea
tmen
t Effe
ct fo
r the
Tre
ated
Variables
Average ATT
95% confidence intervals shown
28 variables have ATT≠0 (p ≤ .05), 23 in expecteddirection; 6 variables have significant differences(α=.05) after Šidák correction for multiple testing,5 in expected direction. Negative mean indicatesresponses consistent with social desirabilityhypothesis.
Less phone socialdesirability
Less web social
desirability
Rounding
21
-1.000
-0.500
0.000
0.500
1.000
q19r
nd10
q20r
nd10
q21r
nd10
q26r
nd10
q26a
rnd1
0
q40r
nd10
q58r
nd10
Aver
age
Trea
tmen
t Effe
ct fo
r the
Tre
ated
Variables
95% confidence intervals shown
4 variables have ATT≠0 (p ≤ .05), all in expected direction; 3variables have significant differences (α=.05) after Šidákcorrection for multiple testing, all in expected direction.Negative mean indicates net greater rounding in phonemode.
Average ATT
Less phone roundingLess w
eb rounding