Effects of Differential Incentives - aapor.org€¦ · Core of the ITC Project, ... (which also includes French respondents as a sub-group not used here), ... Appeal to researchers
19
1 Effects of Differential Incentives Four-Country Re-contact Survey Kathleen McSpurren, Senior Manager, Survey Research Centre, Statistics Dept, University of Waterloo Christian Boudreau, Assistant Professor, Statistics Dept and Associate Director of the Data Management Core of the ITC Project, University of Waterloo AAPOR Conference May 13, 2011
We are pleased to acknowledge those who contributed to the study design: Dr. Andrew Hyland, Associate Professor, Roswell Park Cancer InstituteDr. Geoff Fong, Professor, Psychology, University of WaterlooDr. Mary Thompson, Professor Emeritus, Statistics and Actuarial Science, University of WaterlooCheryl Rivard, Research Associate, Roswell Park Cancer InstituteJanine Ouimet, Project Manager, ITC, University of Waterloo
3
ITC Four-Country OverviewInternational Tobacco Control Project (ITC) currently running in 20 countries
Four Country project – Australia, Canada, UK and US – just completed its 8th wave of longitudinal data collection
Pre-paid incentive – cheques mailed prior to contact – has helped retain over 70% of respondents from one wave to the next, with 12 months or more between contact
Questionnaire is long – 60 minutes to complete
Presenter
Presentation Notes
The International Tobacco Control Project involves research in 20 countries, focusing on the impact of tobacco control policies on adult smokers. The Four Country component of the ITC project is a longitudinal, quasi-experimental design using four English speaking countries: Australia, Canada (which also includes French respondents as a sub-group not used here), the United Kingdom and the United States. Eight waves of data have been collected, with a ninth planned for 2012. Replenishment of each country’s cohort is carried out during most waves. The quasi-experimental aspect involves pre- and post-treatment measures of smoking behaviour and attitudes when legislative or policy changes are enacted. A detailed description of the ITC project methods is contained in Thompson et al, 2006 and the conceptual model is explained in Fong et al, 2006.
4
Mode Choice and IncentivesAppeal to researchers of web self-administered for cost savings
Suitable to wealthy developed nations with broad access to internet
Length of questionnaire can be accommodated on web –start and stop as needed
Respondents may appreciate convenience and flexibility of timing
Presenter
Presentation Notes
The Recontact survey is long, taking 45 to 60 minutes to complete, depending on smoking status. The appeal of web-based self-completed interviews for cost savings (in wealthy countries where is this mode is a viable option) is clear. Other benefits include convenience for the respondent, the possibility to stop and restart the interview when necessary, and the possibility that smokers prefer this mode over the telephone survey (Brogger et al., 2007). However, the disadvantages may be lesser data quality because of no interviewer presence, interruption leading to non-completion of the interview, technical problems due to the web skills of the respondent, and a difference in response distributions with this mode (Nagelhout et al., 2010).
5
Mode Choice and Incentives
Can an extra incentive (post-paid) encourage web response?
Is data quality an issue?
Presenter
Presentation Notes
Incentives paid to all respondents to encourage retention in Wave 8 were $15 CAN/US/AUS and £10 UK. The sub-study of web experience used random assignment to a “bonus arm” consisting of an additional $10 CAN/US/AUS or £6 for the UK respondents on top of the regular token of appreciation, which is a pre-paid incentive by cheque. To qualify for the bonus, respondents were required to answer a brief questionnaire on the web experience. The bonus amount was post-paid. Random assignment was stratified by country, gender, age and presence of email address. The purpose of the experiment was to determine whether the extra incentive could encourage greater web uptake, and whether this incentive differential had an impact on data.
6
Four Country W8 ExperimentExperiment in Wave 7 invited subgroup to the web survey
In Wave 8, all respondents could potentially choose web mode
participants who shared a valid email address at W7 were sent clickable linkAll others sent letter with log in instructions
Web-only period of first three weeks of wave
Randomly selected respondents offered additional incentive for answering 5 extra questions – stratified by gender, age, country and presence of email
Presenter
Presentation Notes
In the previous wave, only a subgroup was asked to complete by web, and retention by web was considerably lower than by phone (48% compared to 68% to 76%). In this wave, all respondents were invited to the web, and all were followed up by telephone after the initial period of web-only responses. The web version of the questionnaire was available throughout the wave, but was offered as the sole mode of response for the first three weeks. Where possible, respondents were emailed an invitation with the survey link embedded in a clickable URL, and those who had not responded were sent email reminders on Days 5, 8, 12 and 15 of the three week period. At the same time, letters were mailed to all participants to invite responses by web through a log-in process from a URL hosted on the call centre’s main web page.
7
Four Country Web UptakeComparison of web uptake by country
*French respondents excluded
Australia Canada* UK US
Telephone 740 560 893 749
Web 513 483 407 371
Percent Uptake
40.9% 46.3% 31.3 33.1%
Presenter
Presentation Notes
The table indicates the percent uptake for each of the four countries. As with a previous web survey pilot study in Wave 7 of Four Country, Canadian respondents had the highest uptake, while UK participants had the lowest.
8
ITC Four Country ProjectComparison of web uptake by random assignment to
incentive groups by country
Australia Canada UK US
Web no bonus
47% 45% 48% 45%
Web with bonus
53% 55% 52% 55%
Presenter
Presentation Notes
The bonus incentive did lead to a higher web survey participation in Canada and the U.S., as evidenced in the proportion of bonus/no bonus completes online. Bonus arm assignment was approximately equal: 49% without and 51% with the bonus amount. Difference of proportions tests showed only Canada has among its web respondents a significantly higher (at the 5% level) proportion responding in the bonus group
9
Age Differences in Web UptakeAll four countries had significant differences in uptake by age regardless of incentive group
AU – 46% of under 25s by web; 28% of 55+ by web
CA – 59% of under 25s by web; 36% of 55+ by web
UK – 48% of under 25s by web; 18% of 55+ by web
US – 42% of under 25s by web; 28% of 55+ by web
Presenter
Presentation Notes
10
Four Country W8 ExperimentThree groups compared:
TelephoneWeb no bonusWeb with extra incentive
Demographics compared: age, education, gender
Smoking measures examined:Behavioural measures included number of cigarettes
smoked per day, time in minutes to first cigarette, number of quit attempts since last contactAttitudinal measures included ‘success’ of quit attempt,
and attitude towards smoking
Presenter
Presentation Notes
For the analysis which follows, all comparisons refer to the three mode/incentive groups, rather than comparing phone versus web first and then comparing the two web groups. This approach should control for multiple comparisons, leading to possible significance just by chance. Given the number of comparisons, the accepted level of probability of a type I error is lowered to .01, rather than .05.
11
Age Differences in Mode Choice
Presenter
Presentation Notes
This is typical of the distribution of age by mode for all countries, but only Australia is shown. On basic demographics, there were some predictable differences in proportion responding by web, in particular, depending on age and education. The tables illustrate the recurring pattern within all four countries by age group, where web proportion is higher among younger respondents. Subgroup comparisons showed that the two web choices did not differ by age for any country (results not shown here). A logistic regression model was significant, predicting that an additional 10 years of age leads to a reduction of likelihood of web uptake by one quarter. An addition of 25 years leads to a reduction by half in the likelihood.
12
Differences in Attitudes by Mode or Incentive
Presenter
Presentation Notes
Education was related to web uptake, as the table displays for Canada. Education categories are as follows: low – primary and pre-vocational secondary levels; medium – secondary level; and high – pre-university and professional education. Once again, the pattern is seen across all four countries, although only Canada is shown here. Respondents with the lowest level of education were less likely to complete the survey by web (Chi-Square=41.0, p.<.0001). This is expected given that highest level of education achieved tends to decrease with increasing age. Subgroup comparisons demonstrated that only the two web groups did not differ by education level in any of the four countries. There were no significant differences by gender for any country for web uptake. In more complex modeling conducted by others, gender had a small effect.
13
Behaviour Differences in Mode Choice
Time in Minutes to First Cigarette
Country Telephone Web no bonus Web with bonus
Statistic
Australia 57.4* 34.2 36.2 F=3.7, p.=.02
Canada 47.6* 23.3 24.2 F=8.8, p=.0002
UK 57.6* 26.6 25.5 F=10.7, p.<.0001
US 35.1 30.9 21.5* F=4.3, p.=.01
Presenter
Presentation Notes
A number of common smoking research measures were compared by mode choice and incentive grouping. Basic smoking measures included: a) number of cigarettes per day, b) time in minutes before smoking first cigarette, c) attitude towards smoking, asked as “what is your overall opinion of smoking?” – very positive, positive, neither positive nor negative, negative or very negative d) number of quit attempts, e) self efficacy of quit attempt (current or hypothetical next, asked as “how sure are you that you would/will succeed” - not at all sure, slightly sure, moderately sure, very sure, extremely sure). The approach here is similar to that of Nagelhout et al., 2010, although not having two independent samples means measurement error is the focus of this paper, ie. in the Four Country Wave 8 Recontact survey there are mode effects but no difference in sampling frames, since the follow-up sample consists of one group. Two questions are investigated: do respondents’ answers in each mode vary – a mode effect; and is there an effect with the differential incentive within the web uptake group? For number of reported quit attempts, there were no differences across mode or incentive amount for any of the four countries, except the UK; however, with a p value of .02, the results do not meet the higher threshold for significance. Self-reported efficacy of quitting (a scale measure) also showed no differences. However, for number of cigarettes smoked, time to first cigarette, and attitude towards smoking, some differences were observed - only time to first cigarette is shown in the table. The ANOVA test was used for the three-group comparison and Bonferroni post hoc tests were used in the subgroup analysis. The table summarizes the tests for four countries on number of cigarettes smoked per day. Only Australia had a significance difference with the web with bonus incentive group differing from the phone group. The average amount of time reported by respondents before the first cigarette of the day was significantly different for three of the four countries, with Australia being just below the threshold alpha of .01. An asterisk indicates the group that was statistically different in the post hoc test. The reported time to first cigarette was consistently longer for the telephone respondents. Only the US group showed a difference between the incentive groups. Minutes to first cigarette had more outlier values in the telephone group than the two web incentive groups; however, parsing the data to remove high values did not nullify the significant differences. This suggests that there is measurement error with this item. It is unclear whether the self-administered quality of the web survey leads to lower reported times.
14
Differences in Attitudes by Mode or Incentive
Attitude Towards Smoking
Country Three Group Comparison
Phone v Web no bonus
Phone v Web w bonus
Web no bonus v Web w bonus
Australia Χ2=33.6p.<.0001
Χ2=15.8p.=.003
Χ2=20.1p.=.0005
Not sig
Canada Χ2=25.4p.=,0013
Not sig Χ2=23.4p.=.0001
Not sig (.04)
UK Χ2=49.0p.<.0001
Χ2=32.6p.<.0001
Χ2=23.2p.=.0001
Not sig
US Χ2=21.0p.=.007
Not sig (.05) Χ2=14.0p.=.007
Not sig
Presenter
Presentation Notes
One measure consistently showed differences by mode across all four countries – attitude towards smoking. The table displays the results of the three-way comparison, and the post hoc two-way comparisons. The phone versus web group with extra incentive consistently had significant differences in all four countries. However, rather than show all results for the two-way comparisons, a case was made to compare phone to all web (see next slide), since the complete lack of significant differences between the two web incentive groups suggest that there is no incentive effect. The next table shows the attitudinal measure by mode choice for one of the four countries (UK), with Chi-Square results displayed below the table. The trend in each country is a tendency to have fewer positive responses and more neutral responses on the web survey. The phone group tends to have more negative responses. A number of factors could explain this pattern: in self-administered web/paper surveys, there is a tendency towards the earlier listed categories, while on the phone, there is a pattern of choosing the later (last heard) responses. However, this pattern was not seen in the other scale item analyzed here – the evaluation of quit success. There may be a social desirability trend with the attitude about smoking question, since it is placed very near the end of the interview, and after a long “conversation” with the interviewer about anti-smoking policies, there may be a tendency to state more negative attitudes than in the self-administered mode, or that web respondents will take the middle or neutral position more often.
15
Differences in Attitudes by Mode Only
16
Four Country W8 ExperimentResults:
Very few and inconsistent differences seen within each country across most behaviours – none are significant between two web groups, only telephone group stands out
Attitudinal differences in all countries for opinion of smoking, but two web groups are similar
17
Missing Data Don’t knows and refused
Presenter
Presentation Notes
Data quality is examined using missing and don’t know/refused categories. The telephone survey allows respondents to offer “don’t know” or to refuse a question, but these options are never read to the participant. In order to replicate that as closely as possible by web, and to deter speeding through the survey, the options DK or refused were not offered on the page with the question, but a prompt would pop up if the person attempted to go to the next question without providing an answer. The example below illustrates how this pop up worked. The first screen shows the initial offering of categories, while the pop up in red alerts the participant to a missing answer, and the extra selections are shown in red at the bottom of the screen. This pop up feature was rarely used. For example, on the opinion of smoking item, only two respondents indicated “DK” (one in each of the UK and Australia) and one person in the UK selected “skip to the next question” from the screen once an attempt was made to go forward without selecting an option. In contrast, missing data imputed by interviewers on the telephone was as high as 2% in Australia and the US, 1.6% in Canada, and 1.3% in the UK on this item. Some anecdotal evidence (taken from the web experience questions) suggests that participants were not aware they could go forward without selecting an item, and offered their “best” answers for the questions.
18
Mode Effect versus Incentive Effect
Age differences in mode choice
Education differences as well but again only in mode choice
Differential incentive did not affect data quality
But mode choice will affect data quality
Presenter
Presentation Notes
Mode has more effect than incentive in possible measurement error in this example. Only attitude and number of minutes to first cigarette had a strong pattern of differences across four countries, which were more evident in the mode comparison than with the mode/incentive groupings. Incentives do not significantly increase web uptake, and do not seem to encourage a distinct subgroup to respond. Differences in responses were not consistent across the four countries, and many differences were not large, suggesting that web is a viable alternative to telephone interviewing in a follow up survey.
ITC Project Research Support
Core support provided by the U.S. National Cancer Institute
