1

Kenneth E. Wallena,b,c, Adam C. Landona,b, Gerard T. Kylea,b,c,

Michael A. Schuetta,c, Jeremy Leitzd, & Ken Kurzawskid

a Department of Recreation, Park and Tourism Sciences , Texas A&M University  

b Human Dimensions of Natural Resources Lab , Texas A&M University  

c Applied Biodiversity Science NSF-IGERT Program , Texas A&M University 

d Texas Parks and Wildlife Department

Sampling Efficacy & Bias in Mode of Response for Survey-Based Research

22

Background

‘’89

‘90

‘93

‘97

‘04

‘08

‘01

‘12

33

Background

44

Background

55

Background

66

Background

77

Background

88

Background

99

Background

1010

Background

1111

Background

1212

Response Rates

•Why do we care about response rates?• Need (desire) to reduce survey error… the difference between the

estimate using the collected data and the true value of the variables in the population• Non-response error… emerges when those who do not respond differ

from those who respond

•Mail questionnaires are costly… printing, postage & labor

1313

Trends in Response Rates

•All modes of collection area are on the decline•Mixed-mode designs (mail/web) have demonstrated success

for reducing non-response error •Respondents to mail and web-based collection modes are

different demographically and with their use of technology•Mixed-mode benefit is twofold… opportunity in preferred

manner & remind respondent of the opportunity to respond

1414

Texas Recreational Anglers

•Heterogeneous• Demographics•Motivations• Preferred resources• Target species• Avidity• Expenditures

• Imperative to reduce non-response error to obtain data that is reflective of the angling population to make informed decisions that will impact this population

1515

Study Purpose

• Compare three modes of questionnaire administration in terms of:1. Response rate2. Respondent characteristics

1616

Survey Methodology

•Data Collection• Sample drawn randomly from TPWD database of licensed

fresh/saltwater anglers• Three modes of contact commencing the last week of September,

2012• Mixed-mode (n=4,000)• Email-Only (n=4,000)• Combination (n=1,000)

•Web-based questionnaire was designed/administered thru Qualtrix with the URL: WWW.TPWD-SURVEY.ORG

1717

Mixed-Mode

1. Initial contact: Invitation letter with web push 2. One week follow-up: Thank you/reminder postcard with

web push 3. Two week follow-up: Second contact letter to non-

respondents with web push 4. Three week follow-up: Survey packet to non-respondents…

cover letter (with web push), questionnaire, and postage-paid reply envelope

1818

Email-Only

• Four email invitations, one week apart• Similar to the cover letter sent to the Mixed-Mode group, the

email invitations outlined the study purpose and invited respondents to complete the questionnaire online• Both a URL to be entered into respondents’ browser and

hyperlinked icon (“Take Survey”) were provided for respondents to access the questionnaire

1919

Combination

•A combination of contacts replicating the “Mixed Mode” and “Email Only” methods• Eight invitations (4 hardcopy, 4 email invitations) were sent

to respondents, one week apart • The sending of email and postal invitations were synchronized to arrive

simultaneously; i.e., email invitations were sent approximately two days following the mailing of hard copies

2020

Survey Response

• Effective Response Rates

Email-only29.9%

(784/2685)

Mixed-mode20.0%

(697/3486)

Combination63.4%

(407/640)

2121

Survey Response

•53.8% of Mixed-mode respondents completed hard copies of the questionnaire

•29.5% of Combination respondents completed hard copies

2222

Survey Analysis

•Multinomial Logistic Regression Model (logit)• Dependent variable: survey response mode

•Null-model Approach

• Independent Variables • Socio-demographics, motivations, & avidity

•Reference group: Mixed-mode

Ø = Mixed-mode1 = Combination2 = Email only

Age + gender + income + race + ethnicity + motivation + avidity

2424

Table 1. Chi-square comparison of observed and expected income by survey mode

Under $20,000

$20,000- $59,999

$60,000- $99,999

$100,000- above

Mixed-modeObserved 8 33 59 139Expected 7.3 41.6 65 125.2Percentage 3.35 13.81 24.69 58.16

Email-only Observed 9 66 150 312Expected 16.3 93.4 146 281.2Percentage 1.68 12.29 27.93 58.10

Combination Observed 19 107 113 169Expected 12.4 71 111 213.6

Percentage 4.66 26.23 27.7 41.42

Notes. χ2 = 49.90, p < 0.001, Cramér's V = 0.15

Income

2525

Table 2. Chi-square comparison of observed and expected income by survey mode

Under $20,000

$20,000- $59,999

$60,000- $99,999

$100,000- above

Mixed-modeObserved 8 33 59 139Expected 7.3 41.6 65 125.2Percentage 3.35 13.81 24.69 58.16

Email-only Observed 9 66 150 312Expected 16.3 93.4 146 281.2Percentage 1.68 12.29 27.93 58.10

Combination Observed 19 107 113 169Expected 12.4 71 111 213.6

Percentage 4.66 26.23 27.7 41.42

Notes. χ2 = 49.90, p < 0.001, Cramér's V = 0.15

Income

2626

Table 3. Chi-square comparison of male and female respondents by survey mode

Female MaleMixed-mode

Observed 11 228Expected 17.8 221.2Percentage 4.6 95.4

Email-onlyObserved 24 513Expected 39.9 497.1Percentage 4.47 95.53

CombinationObserved 53 355Expected 30.3 377.7

Percentage 12.99 87.01

Notes. χ2 = 27.95, p < 0.001, Cramér's V = 0.15

Gender

2727

Within Groups – Socio-demographics

Table 4. T-test results between respondents who completed survey by paper or online within mixed-mode and combination groups

Mixed-mode Combination

Paper Online p-value Paper Online p-value

Socio-demographics

Age 51.3 47.3 .006 54.4 48.8 .007

2929

Table 5. Results of ANOVA for Motives

Effect Mean SD SS F-value dfTo be outdoors1

Mixed-Mode 4.49 0.67Email-Only 4.56a 0.64Combination 4.40a 0.72Between-groups 6.16 6.76*** 2

To be with friends1

Mixed-Mode 3.92 0.99Email-Only 3.95b 1.01Combination 3.75b 1.08Between-groups 10.17 4.78** 2

Notes. 1Mean score value is on a scale ranging from 1 (not important) to 5 (extremely important). Like superscripts indicate significant differences at p ≤ 0.01 (Scheffe post-hoc analysis).** p-value < 0.01***p-value < 0.001

Motives

3030

Table 6. Chi-square comparison of observed and expected importance/non-importance of angling as primary recreational activity by survey mode

Most important Not most importantMixed-mode

Observed 69 170Expected 71.5 167.5Percentage 28.87 71.13

Email-only Observed 141 396Expected 160.6 376.4Percentage 26.26 73.74

Combination Observed 144 264Expected 122 286Percentage 35.29 64.71

Notes. χ2 = 9.19, p < 0.01, Cramér's V = 0.09

Avidity

3232

Within Groups - MotivationTable 7. T-test results between respondents who completed survey by paper or online within mixed-mode and combination groups Mixed-mode Combination Paper Online p-value Paper Online p-valueMotivations To be outdoors 4.3 4.5 .001 4.1 4.6 .000To get away from the regular routine 3.9 4.2 .005 3.8 4.2 .003For the challenge or sport 3.4 3.8 .002 3.7 3.8 .570Most important/Not most important 35.7% 35.0% .862 30.0% 28.6% .843 Mixed-mode: N=408, LR X2 = 66.63, p = .000, Pseudo R2 = .12 Combination: N=239, LR X2 = 51.93, p = .000, Pseudo R2 = .21

3333

Discussion – Response Rates

•Combination (surface mail & email) yielded strongest response rate• Lowest non-response error?

•Respondents (or non-respondents) are reluctant to go from the paper invitation to their computer or device

3434

Discussion - Socio-demographics

• Socio-demographic variation• Variations in age•Web-based respondents slightly higher household incomes•Men more inclined than women to complete online

3535

Discussion – Motivation & Avidity

Motivation• On items where there was significant variation web-based

respondents considered these facets more important•Avidity• Some indication that hard copy respondents most avid

3636

Next Steps

•2015 Survey of licensed Texas Anglers•Mixed-Mode – Mail survey packet with a web push• Incentive – “lifetime license”

3737

Acknowledgements

• Texas Parks and Wildlife Department• Robin Riechers• John Taylor