Mode Effects in aNational Survey ofBusinessEstablishments

May 19, 2013

Introduction: GoalTo explore several aspects related to the introductionof multi-mode survey administration in the 2012 FamilyMedical Leave Act (FMLA) Survey of Establishments

– Compare item response rates toadministrative data questions between the2000 and 2012 surveys

– Potential mode effects between web andtelephone responses (Phillips, Harris,Turakhia 2012. Potential mode effects)

Acknowledgements

• Colleagues at Abt Associates: JacobKlerman, Alyssa Pozniak, Katherine Wen

• Abt SRBI team: Lisa Currie, CourtneyKennedy, Julie Pacer, Chintan Turakhia

• Department of Labor, Wage and HourDivision

3

2012 FMLA Worksite Survey• Sponsor: United States Department of Labor (DOL)

• Research Goal: obtain estimates of the use ofleave under the FMLA and examine the impact onU.S. private business establishments

• History:– Conducted twice prior: 1995 and 2000, telephone only– 2012: questionnaire revisions to capture changes in statutes– Addition of web mode

Web Option for 2012

• Reporting administrative or historical datamay be difficult or burdensome tocomplete via telephone survey.

• Internet option flexibility• Flexibility:

increase response rates, produce higher quality data, potentially cost reduction (See Pacer et al,poster)

Study Design• Sample Frame: 2012 Dun & Bradstreet Market Identifiers

• Target Population: private business establishments– (excluded: self-employed respondents without employees,

government and quasi-government units)

• Stratification: the cross-classification of size (number ofemployees) and North American Industry Classification System(NAICS) grouping

• Over-sampling: establishments in two size categories (20-49 and50-99) to accommodate analysis needs (straddle the FMLAcoverage threshold of 50 employees)

• Survey Questions: based on 2000 Survey, updated to reflecteconomic and statutory changes, several new questions, newoptions for reporting period

Data Collection ProtocolTwo Phases:

1. Screener in CATI (Verified, eligible 6,943/8229)• Verify existence of establishment & matched sample• At least 1 employee• Identify MOST KNOWLEDGEABLE RESPONDENT

(MKR)• Gather contact information for MKR

2. MAIN interview (1812: 634 WEB, 1178, Phone)• Pre-notification letter with WEB link & password• Email reminders sent when available• Follow-up telephone calls, maximum recall 10 attempts

Item Response 2000 and 2012QuestionNumber Question Text 2000 2012

PERCENT GAVE A RESPONSE

Q19 How many of those employees took leave thatyou classified as being under FMLA? 50% 95%

98%Web

92%Phone

Q20

Can you please provide the total number ofseparate LEAVES taken in this same time period?A leave is time taken off for a single reason; thiscould be taken all at once or intermittently overtime.

65% 78%

90%Web

67%Phone

8

Potential Mode EffectsTwo types of mode effects:

1. Reporting administrative data

2. Social desirability– Expected to be minimal

• Compliance issues related to FMLA might be harder toreport to an interviewer

• Opinion questions on impact of FMLA on the business(ease of administration, productivity, profitability)

• Other behaviors that signal “less generous” employer

Estimating Mode Effect

• Use matching design to estimate averagetreatment effect for the treated (ATT)– How telephone respondents would have answered

questions had they received web mode– Sum of responses of web respondents minus sum of

responses of their alters• Matching designs increasingly used for causal

analysis in quasiexperimental designs (c.f. Morganand Winship 2007)– ATT is equal to mode effect

• Variables recoded in direction of expected effect– Negative ATT coefficient associated with social

desirability/satisficing on phone mode– Z-scored for uniform interpretation

Distance Measure• Matching by propensity score generated by logit

model predicting web mode by:– Firm Size– Industry classification (2-digit NAICS code)– Email address provided in screener

• Site size and single site tested but not significant• Resulting propensity scores used in matching

algorithm• Radius matching with radius of .05 (i.e.,

probability of selection into web more ± 5% ofeach web case)

Estimating Mode Effects I• 8 “reporting” variables used;

– “Satisficing” measured as rounded to 10 versus not-rounded number to 10 on:

• Q19 Number of employees who took leave under FMLA• Q20 Number of separate leave occasions• Q21 Number of employees who took INTERMITTENT leave• Q26 Number of medical certifications accepted• Q26a Number of medical certifications returned• Q40 Number of suspected “misused” leaves• Q58 Number of employees who took leave (non-covered

establishments)

Estimating Mode Effects I• 3 variables have ATT≠0 variables have significant

differences (α=.05) after Šidák correction for multipletesting

• Q19 Number of employees who took leave under FMLA• Q20 Number of separate leave occasions• Q21 Number of employees who took INTERMITTENT leave• Q26 Number of medical certifications accepted• Q26a Number of medical certifications returned• Q40 Number of suspected “misused” leaves• Q58 Number of employees who took leave (non-covered

establishments)

Estimating Mode Effects II

• 50 “Social Desirability Variables Used”

– 12 FMLA compliance questions– 14 “Opinion” questions (ease of administration,

impact on profitability, productivity)– 24 Employee benefits questions

Social DesirabilityQ11e Paid Paternity LeaveQ46_2 Do you offer the same family and medicalleave benefits to employees who are NOT eligiblefor FMLA because they are {hourly employees}Q46_3 Over the years, has complying with the FMLAincreased, decreased, or not changed the following?Q49d “Other” costs change as a result of FMLA(decreased)Q52 Has complying with FMLA resulted in any costsavings at this location? (opposite direction)Q67a How easy or difficult is it for your company todeal with the following types of leaves?

15

Conclusions

• Item response rates on reporting dataimproved from 2000• Multiple strategies to ease burden (pre-notification

letter; choice in reporting period; percentageestimation)

• Evidence of satisificing as measured via“rounding”, although we do not know if anyloss of precision

• Implications for data quality unclear

Considerations & Next Steps

• Limited in the data available to predictmode

• Did not collect characteristics ofrespondent Exploring Phase I data forfurther possibilities

• Did not collect information about whetherrespondent consulted records Exploringparadata

17

Contacts

• Kelly Daley k.daley@srbi.com• Ben Phillips b.phillips@srbi.com

18

Appendix Matching Design

Propensity Score

Mea

sure

Web

Phone

Included matches

Treatment Effect

Radius matching used. Average of matches is used. Usedpsmatch2.ado in Stata.

Radius

Appendix: Social Desirability

20

-1.500

-1.000

-0.500

0.000

0.500

1.000

q11_

aq1

1_b

q11_

cq1

1_d

q11_

eq1

1_f

q11_

gq1

1_1

q14_

aq1

4_b

q14_

cq1

4_d

q15

q16_

aq1

6_b

q16_

cq1

6_d

q16_

eq1

6_f

q16_

gq1

6_h

q16x

_1q1

6x_3

q16x

_5q1

6x_7

q16x

_8 q17

q21a q2

7q2

8q3

0q3

2q3

4q4

0sd

q46_

1q4

6_2

q46_

3q4

9_a

q49_

bq4

9_c

q49_

dq4

9_e

q52

q53

q54c

ombo

q55c

ombo q5

6q6

7_a

q67_

bq6

7_c

q67_

dq6

7_e

Aver

age

Trea

tmen

t Effe

ct fo

r the

Tre

ated

Variables

Average ATT

95% confidence intervals shown

28 variables have ATT≠0 (p ≤ .05), 23 in expecteddirection; 6 variables have significant differences(α=.05) after Šidák correction for multiple testing,5 in expected direction. Negative mean indicatesresponses consistent with social desirabilityhypothesis.

Less phone socialdesirability

Less web social

desirability

Rounding

21

-1.000

-0.500

0.000

0.500

1.000

q19r

nd10

q20r

nd10

q21r

nd10

q26r

nd10

q26a

rnd1

0

q40r

nd10

q58r

nd10

Aver

age

Trea

tmen

t Effe

ct fo

r the

Tre

ated

Variables

95% confidence intervals shown

4 variables have ATT≠0 (p ≤ .05), all in expected direction; 3variables have significant differences (α=.05) after Šidákcorrection for multiple testing, all in expected direction.Negative mean indicates net greater rounding in phonemode.

Average ATT

Less phone roundingLess w

eb rounding