Journal of Safety Research 46 (2013) 13–20

Contents lists available at SciVerse ScienceDirect

Journal of Safety Research

j ourna l homepage: www.e lsev ie r .com/ locate / j s r

The comparison of road safety survey answers between web-panel and face-to-face;Dutch results of SARTRE-4 survey

C. Goldenbeld a,⁎, S. de Craen b

a Duindoorn 32, 2262 AR Leidschendam, Netherlandsb Institute for Road Safety Research SWOV, Netherlands

⁎ Corresponding author. Tel.: +31 70 3173374; fax: +E-mail address: charles.goldenbeld@swov.nl (C. Gold

0022-4375/$ – see front matter © 2013 National Safetyhttp://dx.doi.org/10.1016/j.jsr.2013.03.004

a b s t r a c t

a r t i c l e i n f o

Article history:

Received 19 November 2012Received in revised form 13 February 2013Accepted 12 March 2013Available online 23 March 2013

Keywords:SurveyRoad safetyWeb-based panelFace-to-face surveyComparison

Introduction: In the Netherlands, a comparison of an online and a face-to-face sample of car drivers was madeto study differences on a number of selected questions from the SARTRE-4 road safety survey. Results: Con-trary to expectations, there was no indication that online respondents were more likely to come from highereducated or more privileged social groups. Confirming earlier research, the results indicated that online re-spondents were less inclined to give socially desirable answers and were less inclined to use more extremeratings in their opinions about measures. Contrary to expectations, face-to-face respondents did not tendto give more positive answers in judgment of road safety measures. Weighting to make samples comparableon gender, age, and education had almost no effect on outcomes. Conclusions: The implications for a transi-tion from face-to-face survey to online panel method are discussed.

© 2013 National Safety Council and Elsevier Ltd. All rights reserved.

1. Introduction

In the field of road safety, survey research is frequently used to studytraffic behavior, and underlying cognitive andmotivational determinants.In 2010, as part of the SARTRE-4 project, a large scale face-to-face surveywas conducted in 19 countries (18 European countries and Israel).SARTRE stands for Social Attitudes to Road Risk in Europe. This survey,also performed in 1991, 1996, and 2002, focuses on a number of coreroad safety issues — speeding, impaired-driving (drink-driving, drugs),and seat-belt wearing (Cauzard & Wittink, 1998; SARTRE, 1994, 2004).In 2010, several new issues were included such as ‘eco-driving’ and mo-bility, cross-border traffic control, driver fatigue, safety of motorizedtwo-wheelers, risk to pedestrians in urban areas, and new traffic enforce-ment technologies. In 1991, 1996, and 2002 the SARTRE-surveys focusedsolely on car drivers. In the 2010 survey, car drivers were still the majorsampling group (N ≈ 600 in each country), but smaller subsamples(N ≈ 200) were also obtained of motorcyclists, and of other road userswho indicated that they did not use either car or motorcycle as theirmajor travelmode. The SARTRE-4 data provide the EC andmember statesa European picture of road users’ self-reported behavior and opinions,with possibilities to compare between states, and over time, to identifypossible reasons for differences (SARTRE-4, 2012).

Since face-to-face surveys are costly, changing the future set-up ofthe SARTRE-4 survey to an internet survey or to a mixed mode surveyhas been considered. Within the SARTRE-4 project, a study was set upin the Netherlands to compare the responses to SARTRE-4 questions

31 70 3201261.enbeld).

Council and Elsevier Ltd. All rights

between a probability-based face-to-face survey and a non-probabilitybased online panel survey of car drivers. It was decided to restrict thecomparative analysis to car drivers since they were the major samplinggroup in the SARTRE-4 survey and because national statistics for thisgroup were available (and not for the other groups), which enabledweighing results. This paper presents a comparison of demographiccharacteristics and substantive answers between both surveymethods.Such comparisons have been conducted in various other fields, but thisis thefirst time such a systematic overall comparison ismade in thefieldof road safety.

The use of internet panels for purposes of survey research has grownsteadily since the 1990s. Widely recognized advantages of internet sur-veys are the relatively low costs, the high speed, the possibility to reachdifficult to contact individuals, and the increased possibilities to moni-tor data quality (Frippiat &Marquies, 2010). However, critical questionshave been raised whether internet surveys can be considered as repre-sentative for the population and whether the specific mode of answer-ing questions on a computer leads to different answers (Couper, 2000).

1.1. Representativeness of internet surveys

The Dutch SARTRE-4 online web survey of car drivers used a sam-ple from a large volunteer online panel. Several authors have voicedconcerns about the representativeness of non-probability onlinepanels such as used in this study (Couper, 2000; Baker et al., 2010;Dillman, 2007; Smyth & Pearson, 2011). Non-probability samplingdoes not involve random selection and thus non-probability samplescannot depend upon the rationale of probability theory. With a prob-abilistic sample, the odds or probability that the population is rightly

reserved.

14 C. Goldenbeld, S. de Craen / Journal of Safety Research 46 (2013) 13–20

represented is known. Couper (2000) concluded that internet surveysare vulnerable to coverage and sampling errors. In several studies ithas been found that internet surveys tend to be biased towards havingmore privileged social groups, with respondents more often beingyoung, urban, having higher income, or having higher education(e.g., Couper, 2000; Baker et al., 2010; Heiervang & Goodman, 2011;Marta-Pedroso, Freitas, & Domingos, 2007). Duffy, Smith, Terhanian,and Bremer (2005) concluded that online research using panel ap-proaches attracts a more knowledgeable, viewpoint-oriented samplethan face-to- face surveys. This could be because this is a prior charac-teristic of internet-users or panel-members, or it could be a learned be-havior from taking part in a number of surveys. In recent years,concerns about coverage have decreased somewhat as the percentageof homes with internet access has sharply risen. In 2009, one year be-fore the fieldwork of this study was conducted, 89% of households inthe Netherlands had internet access (TNS Opinion and Social, 2010).

The debate on the representativeness and accuracy of internet sur-veys is not yet resolved. For various specific subjects, such as personalityand well-being scales (Howell, Rodzon, Kurai, & Sanchez, 2010), atti-tudes towards immigration (Duffy et al., 2005), political behavior(Duffy et al., 2005; Stephenson & Crête, 2010), contingent valuation(Marta-Pedroso et al., 2007; Lindhjem & Ståle, 2010; Nielsen, 2011),risk factors associated with alcohol-related problems (Heeren et al.,2008), and willingness to pay for collective goods (Olsen, 2009;Taylor, Nelson, Grandjean, Amatchkova, & Aadland, 2009), authors haveconcluded that internet surveys or internet panels deliver nearly equiva-lent estimates or outcome patterns as more traditional probability-based surveys (telephone, face to face, ormail). However, for other sub-jects such as political activism (Duffy et al., 2005; Malhotra & Krosnick,2007), knowledge about cholesterol (Duffy et al., 2005), and attitudesand preferences concerning wildlife management (Duda & Nobile,2010; Gigliotti, 2011), authors have reported divergent findings be-tween internet and other type surveys, and less representative or accu-rate results for internet surveys.

Baker et al. (2010) conclude that few studies have systematicallydisentangled effects from differences between samples and effectsfrom differences in interviewmode (internet, phone, mail, face-to-face)by the use of an experimental set-up. Based on the scarce number ofstudies that have done this, these authors conclude that non-probability samples that are used in most internet surveys are less rep-resentative and less accurate than probability samples. Comparing sur-vey results with objective benchmarks from official governmentrecords or established high-quality surveys with high response rates,Yeager et al. (2011) also report greater accuracy for probability-basedsamples than non-probability based samples.

1.2. Mode or instrument effects

Specific mode or instrument effects such as question-order effects,social desirability biases, response-order effects, and satisficing havebeen found to be different according to the mode of data-collection(i.e., self-administered internet or paper-and-pencil questionnaire,interviewer-administered telephone or face-to-face interview; Ye,Fulton, & Tourangeau, 2011).

A general effect found in comparative survey research is that websurveys tend to produce less extreme, more neutrally toned re-sponses (Frippiat & Marquies, 2010). Measuring agreement with pos-itive statements about a target corporation, Roster, Rogers, Albaum,and Klein (2004) found that the web survey generated more negativeand neutral evaluations than did a telephone survey. Measuring atti-tudes towards immigration, Duffy et al. (2005) found that online sur-vey respondents were more likely to select a neutral scale-category(‘neither agree nor disagree’) than face-to-face respondents. In thecontext of a comparative survey on attitudes towards immigrantsand asylum seekers, Heerwegh and Loosveldt (2008) again foundthat internet participants supplied more mid-scale responses than

face-to-face participants. In research on satisfaction with a long dis-tance company, Dillman et al. (2009) found that both internet andmail respondents gave less positive answers at the scale end than re-spondents interviewed via phone by an interviewer or by an Interac-tive Voice Response (IVR) procedure.

A meta-analysis by Ye et al. (2011) used 18 comparisons based on12 studies to determine differences in frequency of positive answers.This analysis showed that telephone respondents more frequentlygave positive responses compared to mail or internet respondents,but not to face-to-face respondents. The authors posit that a positivitybias in answering questions is prompted by the actual presence of aninterviewer. According to the authors, theoretical explanations un-derlying this effect may be that either personally interviewed respon-dents need to make a favorable impression by giving positiveresponses, or that personally interviewed respondents experience agreater burden on working memory making them more vulnerablefor response-order, such as recency effects.

Social desirability refers to the tendency of respondents to provide re-searchers with responses that will give a favorable image of themselves,or responses they think correspond to the social norm (Frippiat &Marquies, 2010). There is consistent evidence that socially-desirableresponding is more likely with interviewer-administered modes of datacollection than self-administered modes (Joinson, 1999; Heerwegh,2009; Baker et al., 2010). For example, consistentwith the social desirabil-ity hypothesis, Link andMokdad (2004) found that after correction for de-mographic differences, internet respondents reported higher rates ofdiabetes, high blood pressure, obesity, and binge drinking, and lowerrates of efforts to prevent contracting sexually transmitted diseases,when compared to those interviewed by telephone.

Satisficing refers to a failure to put in the necessary effort to optimal-ly answer a survey question (i.e., shortcutting the response process;Krosnick, 1991). The general hypothesis of a web survey inducingmore satisficing than a face-to-face survey leads to the expectationthat web survey respondents will use the “don’t know” (DK) responsealternative more frequently, and that they will differentiate less on rat-ing scales than face-to-face respondents (Holbrook, Green, & Krosnick,2003). Non-differentiation refers to the respondent’s limited use ofthe available response alternatives on rating scales. In support of thishypothesis, Heerwegh and Loosveldt (2008) found that web survey re-spondents produced a higher “don’t know” response rate, differentiatedless on rating scales, and produced more item nonresponse thanface-to-face survey respondents.

1.3. Hypotheses

Based on the literature, we expected that respondents from the on-line panel would tend to come from more socially privileged groups(i.e., higher income or higher education groups) (H.1), that online re-spondents would report more socially undesirable traffic behaviors andattitudes than face-to-face respondents (H.2), that online respondentswould show more neutral ratings on traffic topics than face-to-face re-spondents (H.3), more specifically that online respondents would beless inclined to choose extreme positive ratings of approval for roadsafetymeasures than face-to-face respondents (H.4), and that online re-spondents would show less answer differentiation than face-to-face re-spondents (H. 5).

2. Method

2.1. Description of on-line panel

The online panel of the performing research bureau Motivactioncontained approximately 80,000 Dutch respondents, who had indi-cated to be willing to participate in online research every now andthen. A gross sample of 17,942 persons in the ages 18–70 years wasdrawn from the panel. From the gross sample, 12,686 respondents

15C. Goldenbeld, S. de Craen / Journal of Safety Research 46 (2013) 13–20

were invited to participate in the SARTRE-4 survey. In total 6,318responded. Subsequently, 5,466 potential candidates were not used be-cause they either did not qualify for the criteria for one of the subsamples(car driver, motorcyclist or other road user), or they were not neededbecause of the fixed quota. Finally, 852 Dutch respondents completedthe SARTRE-4 online survey. After data cleaning, 799 responded remainedin the dataset, of which 381 in the sample car drivers (and 208 in thesamplemotorcyclists, and 210 in the sample other road users; these lattersamples are not relevant and not used in the present study).

Since specific groups are either under- or over-represented in apanel, random sampling from an internet panel will result in a non-representative sample.With the SARTRE-4 online survey, the techniqueof propensity sampling was used to address this problem. This is an ad-vancedmethod used for obtaining amore valid sample from an internetpanel (Joffe & Rosenbaum, 1999). The chance of being selected differsper respondent and is based on the combination of standard demo-graphics (including age, gender, education, region), internet usage,and the score on the social segmentation model developed by researchinstitute Motivaction. The latter model has been extensively used inmarketing research (Franzen, 2006; Lampert, van der Lelij, Knoop, &Egberink, 2002).

The advantage of the propensity method over the quota samplingmethod is that more and different variables are incorporated into thesample drawing than simply socio-demographic variables. For exam-ple, in internet panels people with a low education tend to beunder-represented but with propensity sampling these persons have alarger chance of being selected for the sample. Besides a correction forsocio-demographic variables (age, gender, education), the propensitysampling method also corrects for characteristics of non-internetusers, and for social and/or cultural characteristics. The research bureauMotivaction uses the Mentality database as a national representativereference on those variables. This database consists of data from morethan 24,000 respondents, originating from an annual face-to-face sur-vey among the Dutch population and is considered by the research bu-reau to be a representative sample of the Dutch population.

2.2. The SARTRE-4 face-to-face survey

A random sample of postal codes of Dutch addresses was drawn. Thepostal codes were a good representation by region and the degree of ur-banization of the Netherlands. Within the sample a total of 38 locationswere selected. Teams consisted of a maximum of 4 interviewers whoon average completed 10 interviews a day (in total). They visited 38 lo-cations and performed a total of 350 interviews in approximately38 days. Interviewers started to bring in interview respondents in thepre-selected street at every location. After completing an interview theyskipped seven houses and started the procedure to bring in respondentsagain. When interviewers met an intersection they turned right. Theteams consisted of males and females, aged 19 to 25. During the inter-views, interviewers used a laptop to read the questions and answer op-tions to the respondents and interviewers coded answers into a laptop.

The face-to-face sample of car drivers consisted initially of 376 re-spondents, of which 350 respondents with fully completed recordsfor basic variables such as age, gender, and education, were finally in-cluded in the data file.

2.3. Data analysis

2.3.1. Selection of questionsTo test the hypothesis concerning social desirability (H.2), SARTRE-4

questions were chosen regarding traffic behaviors that are avoided ordisapproved of by a clear majority of road users: drinking and driving,exceeding the speed limit in a built-up areawith 20 km/h, following ve-hicle in front too closely, not giving way to a pedestrian at a pedestriancrossing, and not making children in a car use appropriate restraints(seat belt or other restraints) (SARTRE-4, 2012).

To test hypotheses 3 and 4 concerning differential answer tenden-cies three batteries of questions were used: questions about agree-ment with existing road safety measures (30 km/h zones, speedcameras etc.), about agreement with new car safety provisions (fa-tigue detection system, black box etc.), and about personal willing-ness to reduce car usage for a cleaner environment. Finally, twoitem batteries were used to study possible differences in answer dif-ferentiation (H.5).

For results presented in tables, percentages are given for answer cate-gories on which the two samples diverged most. Since survey reportsoften condense respondents’ answers on various issues to ‘For’or ‘Against’statement, it was decided to also look exploratory at answer differenceson the three batteries of questions after reducing the four-point scales(1 = very in favor…4 = not at all in favor; 1 = very accept .. not at allaccept) to bipolar scales (‘In favor vs. not in favor’ and ‘Accept vs. not ac-cept’). These results are quite straightforward and are summarilyreported in text (and not reported in a separate table).

2.3.2. Weighting of the dataIn order to reduce the possible influence of sample differences on

answers, the survey data were weighted to make both the onlinesample and face-to-face sample comparable to official national refer-ence data on car drivers’ gender, age, and education. By doing this, weattempted to minimize the effects of initial sample differences and toincrease the chance that observed differences would reflect differ-ences in method rather than sample.

The weighting was done sequentially. First the sample wasweighted for the gender-age variable; population data were availablefor this nested variable. After this new frequency distributions werecalculated for the variable Education and a new weight (Gender-age *Education) was calculated.

2.3.3. General significance testingThe differences between the face-to-face and online sample on the

selected SARTRE-4 questions were analyzed with chi-square analysis.Chi-square analysis was preferred above an analysis of variance be-cause the rating scales were considered to be ordinal rather than in-terval. For the comparison of the face-to-face and online sample,multiple chi-square significance tests were performed. This increasesthe chances of finding significant results when, in fact, there are ab-sent (Type I error). With 20 tests and α = .05, there is a 100% chanceof finding 1 (false) significant result. Since we have 28 tests we usedthe Bonferroni correction (e.g., Field, 2009) to find an adjusted α. TheBonferroni correction (dividing the desired α by the number of tests)suggest an α of 0.0018. Since the Bonferroni correction is just an ap-proximation, comparisons with α b .01 will be considered significantin this paper.

Besides the significance testing itself, the pattern of answer differ-ences and the type of answer categories involved (midscale, endscale) were inspected as they were relevant for H.3 and H.4.

2.3.4. Testing for differences in response differentiationFor two item batteries (questions on agreement with road safety

measures and questions on willingness to reduce car usage) two mea-sures of answer differentiation were looked at: a response differentia-tion index Pd (rho) (McCarty & Shrum, 2000, p. 278) and the standarddeviation of answers on the battery per respondent. Thefirstmentionedindex refers to the number of different scale points used by a respon-dent and it varies between minimum 0 and maximum 1. A higher Pdvalue indicates more differentiation and signifies that the respondentused more of the response options. Only respondents answering allthese questions were included in these analyses. Analysis of variance,with α = 0.05, was used to test for significant differences on these in-dicators by face-to-face and online respondents. Because of the largesample size (N = 731), even small differences result in significant re-sults. Therefore, besides significance, also the effect size (Partial èta

Table 1Population and sample distribution before and after weighting.

National reference data from Statistics Netherlands (CBS) 2010 Face-to-face sample N = 350 Online sample N = 381

% Unweighted % Weighted % Unweighted % Weighted %

Male 18–24 4 4 4 3 425–39 14 8 13 13 1340–49 12 9 12 12 1250–59 10 7 10 10 1060+ 13 16 14 14 13

53 44 53 52 53Female 18–24 4 4 4 4 4

25–39 13 17 12 14 1340–49 11 16 10 10 1150–59 9 10 9 11 960+ 10 10 11 9 10

47 57 47 48 47Education Primary education 20 10 20 17 20

Secondary education 46 46 46 45 46Further education 34 44 34 39 34

100 100 100 100 100Region North 11 14 12 11 11

East 21 20 19 21 20West 45 41 42 40 41South 23 25 27 28 28

100 100 100 100 100Work status Self-employed 7 25 27 11 10

Employed 54 36 35 50 49Not employed 40 38 38 40 41

100 100 100 100

16 C. Goldenbeld, S. de Craen / Journal of Safety Research 46 (2013) 13–20

squared,η2)was consideredwithη2 ≈ .01 as a small,η2 ≈ .06 as ame-dium, and η2 ≈ .14 as a large effect size (Cohen, 1988).

3. Results

The next sections present results on sample background differ-ences (3.1), social desirability answer tendencies (3.2), opinions onroad safety measures (3.3.), response differentiation (3.4), and a re-sults summary (3.5). The results in Sections 3.2, 3.3, and 3.4 concerntests of differences after both samples have been weighted to correctfor demographic differences on gender, age, and level of education.

3.1. Sample comparison

Table 1 shows the population and sample distribution before andafter weighting.

Table 2Differences between face-to-face and online respondents on questions concerning anti-nor

Questions Fa

3a. Concerning driving a car 20 km/h over the speed limit in a residential area….a) It makes driving more pleasant

N

9a. You can drink and drive if you drive carefully N

10. Over the last month, how often have you driven a car after having drunk even asmall amount of alcohol?.

N

11. Over the last month, how often did you drive a car, when you may have beenover the legal limit for drinking and driving?

N

23a. When driving a car how often do you follow the vehicle in front too closely? N

23b. When driving a car how often do you give way to a pedestrian at pedestriancrossings?

Al

N8a How often do you make children travelling with you wear seat belt or useappropriate restraint on motorways

Al

8d. How often do you make children traveling with you wear seat beltor use appropriate restraint on roads within built-up areas?

Al

⁎ Since cells had expected counts less than 5, some answer categories were taken togeth⁎⁎ These questions were only answered by those respondents who indicated that they ha

The chi-square-analysis on unweighted samples indicated thatface-to-face and online sample differed on educational background(χ2

(2,N = 731) = 8.2; p = .016) and work status (χ2(2,N = 731) = 30.9;

p = .000). In contrast to H.1 there were more respondents with highereducation in the (unweighted) face-to-face sample than in the onlinesample. After weighing, the two samples did not differ on education,but there remained a significant difference in work status. As can beseen in the Table 1, there was an overrepresentation in both samplesfor self-employed respondents compared to the population mean. Theface-to-face sample was especially overrepresented by respondentsowning a business or a shop. This overrepresentation was probablydue to the time of day the interviewers conducted their interviews,and the higher chance of finding people at home to participate in theinterviews.

The next sections present results on weighted samples. It shouldbe mentioned here that the effects of weighting were negligible:cell percentages and significance levels of chi-square analyses were

mative traffic behaviours.

ce-to-face % answers N = 350 Online % answers N = 381 Significance

ot at all agree 24.0 Not at all agree 21.5 Ns

ot at all agree 52.7 Not at all agree 65.1 χ2 = 13.5, df = 3;p = .004

ever 66.0 Never 70.6 Ns

ever 92.6 Never 93.2 Ns

ever 40.9 Never 26.1 χ2 = 37.1, df = 5;p = .000

ways 59.8 Always 48.0 Ns

= 235⁎⁎ N = 232⁎⁎

ways 97.4 Always 90.1 χ2 = 10.8, df = 1⁎;p = .001

ways 95.3 Always 90.5 Ns

er in this analysis reducing degrees of freedom.ve transported children in their car.

Table 3Differences between face-to-face and panel respondents on agreement with road safety measures.

Question Howmuch would you be in favour of in favour of… (scale 1 = very .. 4 = not at all) Face-to-face % answersN = 350

Online % answersN = 381

Significance

7a. Automated cameras for red light surveillance Not at all 10.0 Not at all 3.4 χ2 = 14.6, df = 3; p = .0027b. Surveillance of speeding at a single point by automated cameras Not at all 17.4 Not at all 7.6 χ2 = 17.8, df = 3; p = .0007c. Surveillance of speeding between two distant points by automated cameras Not much 21.4 Not much 30.3 χ2 = 12.7, df = 3; p = .0057d. More “30 km/h” zones in built-up areas Not much 26.9 Not much 35.7 Ns7e.More bicycle lanes Fairly 32.3 Fairly 44.2 χ2 = 44.4, df = 3; p = .0007f. More sidewalks for pedestrians Fairly 28.0 Fairly 44.5 χ2 = 60.0, df = 3; p = .0007g. More car and motorcycle free zones in built-up areas Not at all 25.7 Not at all 18.3 Ns

17C. Goldenbeld, S. de Craen / Journal of Safety Research 46 (2013) 13–20

nearly similar for the weighted and un-weighted samples (findingsavailable author, here not further reported).

3.2. Results social desirability

Table 2 presents results concerning differences between face-to-facesample and internet sample on questions selected to test the social desir-ability hypothesis. The questions concern traffic behaviors that aredisapproved of by a majority of Dutch road users.

Table 2 shows the eight questions for which a social desirabilitybias was expected to occur. For two self-reported behavior questionssignificant differences were found (8a: ‘how often children proper re-straint motorways;’ 23a: ‘how often follow vehicle in front to close’)in the expected direction of the hypothesis 2. For one attitudinalquestion a significant difference was found in contrast to hypothesis2 (9a: ‘you can drink/drive if careful’). Although not significant, thedifferences on two questions (8d: ‘how often children restraint inbuilt-up areas;’ 23b: ‘how often give way to pedestrian’) were closeto significance (p = .044, resp. p = .012) and also in the directionof H.2.

3.3. Differences in answer tendencies on questions on road safetymeasures

It was tested whether face-to-face and online respondents dif-fered in their agreement on three different sets of questions: agree-ment with existing road safety measures (Table 3), agreement withnew car safety provisions (Table 4), and agreement with the personalreduction of own car usage (Table 5).

Table 3 shows that on 5 out of 7 questions about road safety mea-sures, significant differences were found (7a: ‘automated red lightcameras,’ 7b: ‘automated speed cameras,’ 7c: ‘automated speed cam-eras distance,’ 7e: ‘more bicycle lanes,’ 7f: ‘more sidewalks pedes-trians’). For one question (7d: ‘more 30 km/hr. zones’) the resultwas close to significance (p = .025). Comparing percentage differ-ences per answer category, the relative differences between thegroups were most clear in the ‘Not at all,’ ‘Not much,’ or ‘Fairly’ an-swer categories. The differences were consistent in the same directionindicating that, in agreement with H3, online respondents wereless inclined to choose answer categories at the end of the scale(i.e., ‘Not at all’) and more inclined to choose mid-scale values(i.e., ‘Not much,’ ‘Fairly’). In contrast to hypothesis 4, face-to-face

Table 4Differences between face-to-face and panel respondents on agreement with introduction o

Question: How much would you be in favour of ....? (scale 1 = very ….4 = not at all)

6a. Speed limiting devices fitted to cars that prevented drivers exceeding the speed limit6b. A 'black box' to identify what caused an accident6c. An “alcolock” that prevented the car to start if the driver exceeds the legal alcohol limit for d6d. An “alcolock” that prevented the car to start for recidivist driver that exceeds thelegal alcohol limit for driving

6e. Fatigue detection devices that warn the driver to stop if he/she was too tired to drive

respondents were less positive about (i.e. less in favour of) various roadsafety measures than online respondents.

Concerning the five questions on new car safety devices (seeTable 4), significant differences between face-to-face and online re-spondents were found on three questions (6a: ‘speed limiting de-vices,’ 6c: ‘an alcolock,’ 6e: ‘fatigue detection device’). On two ofthese three questions (6a: ‘speed limiting devices,’ 6e: ‘fatigue detec-tion device’), online respondents were less likely to choose end ofscale answers (‘Not at all’) and more likely to choose midscale an-swers (‘Fairly’) than face-to-face respondents. On one question 6c(‘alcolock’) however, the pattern is reversed with online respondentsmore likely to choose the extreme answer category than face-to-facerespondents (‘Very’). The pattern of differences indicated partial sup-port for H.3 with online respondents less likely to choose end scaleanswer categories (‘Not at all’) and no support for H. 4, withface-to-face respondents being less positive about road safety mea-sures than online respondents.

Finally, analysis of sample differences on the eight questions aboutwillingness to accept personal measures to reduce car usage indicatedthat on four questions (21a: 'reduce the usage of your car'; 21b: 'sharea car'; 21d: 'use public transport more frequently'; 21e: 'a car free dayeach month'; 21f: 'use a bicycle more frequently') face-to-face re-spondents indicated lesswillingness to acceptmeasures than online re-spondents (see Table 5). For another question (21h: ‘spend moneyhybrid engine’), the result was close to significance (p = .018) andthe difference was in the same direction.

Again, the pattern of differences in Table 5 is fully consistent withhypothesis 3, with online respondents preferring midscale answercategories (‘Not much’ or ‘Fairly’) and face-to-face respondents pre-ferring end scale answer categories (‘Not at all’). Again, the strongernegative ratings (‘Not at all’) given by face-to-face respondents arein contrast to H.4.

3.3.1. Overview extent of percentage differences 20 questionsWhen we compared face-to-face versus online percentage differ-

ences on the three batteries of questions on measures (i.e., 20 ques-tions in Tables 3; Table 4; 5) with percentages of 4-point ratingscale reduced to ‘For/In Favor’ or ‘Against/Not in favor,’ the followingpicture emerged (results not reported in table). For 4 of 20 questionsthere was a fairly large percentage difference between the online andface-to-face respondents, difference varying between 9 to 17 percent-age points; for 3 of 20 questions the answer difference was moderate,slightly over 4 percentage points; for the 13 remaining questions the

f new car safety provisions.

Face-to-face % answersN = 350

Online % answersN = 381

Significance

Not at all 36.6 Not at all 25.5 χ2 = 12.4, df = 3; p = .006Not at all 16.6 Not at all 10.8 Ns

riving Very 48.9 Very 60.9 χ2 = 19.8, df = 3; p = .000Very 69.8 Very 75.3 Ns

Fairly 33.7 Fairly 41.6 χ2 = 14.1, df = 3; p = .003

Table 5Differences between face-to-face and panel respondents on willingness to reduce car usage behaviour.

Question: In order to reduce air pollution, how much are you willing to accept thefollowing (scale 1 = very .. 4 = not at all)

Face-to-face % answersN = 350

Online % answersN = 381

Significance

21a.Reduce the usage of your car Not much 13.4 Not much 26.4 χ2 = 32.2, df = 3; p = .00021b. Share a car with colleagues to go to work place (car pooling) Not much 9.7 Not much 20.7 χ2 = 20.2, df = 3; p = .00021c. Renting a car when you just need it (car sharing) Not much 13.7 Not much 19.9 Ns21d. Use public transport more frequently Not at all 55.4 Not at all 40.4 χ2 = 27.2, df = 3; p = .00021e. A car free day each month Not much 15.4 Not much 26.1 χ2 = 14.1, df = 3; p = .00321f. Use a bicycle more frequently Fairly 33.7 Fairly 40.7 χ2 = 12.9, df = 3; p = .00521g. Use a moped/motorcycle more frequently Not at all 76.0 Not at all 67.5 Ns21h. Spend an extra amount of money on a hybrid or electric engine when buying a new car Not much 23.4 Not much 30.4 Ns

18 C. Goldenbeld, S. de Craen / Journal of Safety Research 46 (2013) 13–20

answer difference was small and most often between 2 and 4 per-centage points.

3.4. Response differentiation

Table 6 presents analyses on indicators of differentiation for twobatteries of questions.

Although the difference is small, for the on the road safety measuresquestions, the online respondents had a lower average differentiationrate (.470) than the face-to face respondents (.505), confirming H.5.For the battery of questions onwillingness to reduce car usage no statis-tical difference was found between face-to-face and online respondents.In support of H.5, for both question batteries, the standard deviation ofitem scores per respondent was significantly lower for the online sam-ple than the face-to-face sample.

3.5. Summary of findings

Only few differences in background characteristics were observedbetween the sample of online and face-to-face respondents. In con-trast to H.1, face-to-face respondents tended to be better educatedand more often self-employed. The effects of weighting were negligi-ble: cell percentages and significance levels of sample comparisonswere nearly similar for the weighted and un-weighted samples.

In total, the analysis of sample differences on 28 selected SARTREquestions, produced 16 significant and 5 near significant results (sig-nificance level p b .010). This number of differences is far more thancould be expected by chance alone.

There was partial support for the social desirability hypotheses(H.2). On four out of eight selected questions concerning social unac-ceptable traffic behaviors, online respondents were more likely to re-port that they had engaged in these behaviors than face-to-facerespondents (two significant results and two near significant).

In support of H.3, on several questions concerning road safetymeasures, new safety car provisions, and willingness to accept mea-sures to reduce car usage where significant or near significant differ-ences had been found, face-to-face respondents consistently hadlarger percentages end scale rating (e.g., ‘Not at all’) for a number ofquestions, whereas online respondents consistently had larger per-centages midscale ratings (‘Not much;’ ‘Fairly’).

Contrary to H.4, face-to-face respondents were consistently lesspositive and more outspoken in their disagreement on three sets ofquestions concerning road safety measures, new safety car provi-sions, and willingness to accept measures to reduce car usage. In

Table 6Differences in answer differentiation between face-to-face and online sample.

Battery questions Indicator Face-to

questions road safety measures 7a–7g Pd (Rho) .503St. Dev. per person .744

questions willingness to reduce car usage 21a–h Pd (Rho) .499St. Dev. per person .913

support of H.5, online respondents showed less differentiation ontwo batteries of questions than face-to-face respondents, but thesedifferences were small.

4. Discussion

The present study compared answers of face-to-face sample andan online sample of respondents on questions concerning road safetymeasures and traffic behaviors. Before we focus on the results wewould like to point out some limitations of the present study. Thepresent results were obtained in the Netherlands, a densely populat-ed, highly urbanized country with a high internet penetration, andbelonging to the top 20 of the economic high ranking countries. It isnot certain to which extent the results may generalize to other coun-tries in Europe or worldwide.

In the present study we were only partially able to correct for possi-ble sample differences that may have affected results. Therefore it wasnot possible to completely separate effects of sampling (face-to-facerandom walk sample vs. internet panel propensity sample) and instru-ment (face-to-face interviewing with interviewer reading questionsand answer categories aloud from a laptop vs. self-completed online).

The results of the present study could not be meaningfully com-pared to more objective benchmarks, such as provided by official sta-tistics or high quality surveys. Therefore it is not possible to concludewhich results of both samples better approximate population values.

Population differences between face-to-face and online samplewere modest. There was no difference in age and gender. This is prob-ably due to the fact that the performing survey bureau Motivactionused a sampling method to correct for different representations ofpeople in the complete panel of 80,000 Dutch respondents. Respon-dents were selected to participate in the study based on the popula-tion means. The face-to-face sample had significant higher educatedrespondents and more self-employed respondents. They were over-represented compared to the online sample, as well as the populationmean. The fact that there were more self-employed respondents inthe face-to-face sample is probably due to logistic reasons (e.g., timeof day and chances of finding people at home) rather than a preferencefor a certain questionnaire/interview style.

There was no evidence from the present research that online respon-dents tended to come from more privileged social groups (H. 1). On thecontrary, the online respondents more often only had primary educationcompared to face-to-face respondents, and the online grouphad a smallershare of self-employed, independent professionals. There are possiblytwo factors that can explain this result. First, one of the main aims of

-face N = 350 Online N = 381 Significance

.470 F(1,778) = 6.4, p = .012, η2 = .008

.613 F(1,778) = 34.8, p = .000, η2 = .043

.512 Ns

.802 F(1,778) = 19.3, p = .000, η2 = .024

19C. Goldenbeld, S. de Craen / Journal of Safety Research 46 (2013) 13–20

the used technique of propensity sampling for the online panel was tocorrect for underrepresentation of specific social groups. Second, in theNetherlands internet penetration among households is among thehighest in the world with 9 out of 10 households having internet access.

We found partial support for a social desirability tendency (H.2). Onfour questions concerning social unacceptable traffic behaviors, onlinerespondents were more likely to report that they had engaged inthese behaviors than face-to-face respondents. However, the evidencewas not consistent. On three other behavioral questions no statisticaldifference was found between face-to-face and online respondents,and on one attitudinal question online respondents were more likelyto report disagreement with an unacceptable attitude statement thanface-to-face respondents.

Differences between online and face-to-face respondents werelooked at in terms of significance testing, percentages, midscale versusend scale answer preferences, and answer differentiation. On threeout of four selected questions significant or near significant differenceswere found between online and face-to-face respondents. This numberof significant differences is farmore than could be expected on the basisof statistical chance alone and it suggests that either sample differencesand/or instrument effects have contributed to the different answer pat-tern. In support of H.3, on three item batteries on traffic measuresconcerning road safety measures, vehicle-related measures, or carusage reduction measures, the general pattern was that online respon-dents tended to use moremidscale ratings, supporting H.3. This findingreplicates earlier findings by Heerwegh and Loosveldt (2008) whofound that in five item batteries using a five-point rating scale and ac-counting for a total of 38 items, themiddle categorywas selected signif-icantly more often in the online than in the face-to-face survey.

The evidence concerning a possible positivity bias (H.4) was in theopposite direction than predicted: on three item batteries, face-to-facerespondents were less positive and were more outspoken in their dis-agreementwith these road safetymeasures. Onmost of these questionsthe difference between online and face-to-face respondents was mostpointed in ‘Not at all’ or ‘Not much’ category. Face-to-face respondentshad larger ‘Not at all’ percentages for a number of questions on roadsafety measures, whereas online respondents had larger ‘Not much’percentages for some questions.

There are three possible explanations for the above described answerpatterns. First, visual presentation of answer-categories on-screen morestrongly prompts the use of mid-scale categories than verbal presenta-tion of questions (e.g., Dillman et al., 2009). Second, it can be assumedthat online panel respondents who likely have a greater experiencewith internet surveys than face-to-face respondents have ‘learned’ touse the extreme answer categories more discriminately in order to bebetter able to discriminate for themselves on issues that really matter.If, as Duffy et al. (2005) have concluded, online research using panel ap-proaches attracts a more knowledgeable, viewpoint-oriented samplethan face-to-face surveys, than this could possibly manifest itself in apreference for more cautious disagreement ratings (‘Not much’) thantotal disagreement ratings (‘Not at all’). Thirdly, it seems that if impres-sion management motives have played a role in the face-to-face inter-views, the general tendency for subjects may have been to appear‘decisive’ rather than to appear ‘positive.’ It could be that in face-to-faceinterviews, respondents tended to choose more extreme answer catego-ries to show how certain they were.

Taking a look at answer differences after dichotomizing 4-point an-swer scales to bipolar scales (‘In favor/accept’vs. ‘Not in favor/not accept’),the answer differences were small for 13 out of 20 questions, moderatefor 3 questions, and large for 4 questions. The effects of weighting werenegligible: cell percentages of weighted and un-weighted samples werenearly similar. In other research it has also been found that weightinghardly reduces differences between face to face and online respondents(e.g., Duffy et al., 2005). Given that sample differences in gender, age,and education, were not large to begin with it is not surprising thatweighting did not make much difference.

5. Conclusion

In countries with similar characteristics as the Netherlands (highinternet penetration among various groups of population), the transi-tion from face-to-face to online survey research can be expected to beassociated with: (a) on the positive side, equal representation of lesssocially privileged groups, less socially desirable and likely more ac-curate answers on behavior questions, and, (b) perhaps less positive-ly, a systematic tendency to use midscale answer categories moreoften, and to differentiate slightly less on opinion questions. Whenthe opinion results were looked at in terms of bipolar distinctions‘Agree-Disagree,’ ‘In favor’- ‘Not in Favor,’ most differences werequite small in the range of 2–4 percentage points or slightly over 4percentage points. All in all, the changes between samples were sys-tematic but modest. In view of the considerably lower costs of inter-net panels compared to face-to-face surveys and in view of thepresent results, the use of an internet panel is a serious option for afuture European SARTRE-survey, at least in those countries with ahigh internet penetration (preferably > 90%).

References

Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., et al.(2010). AAPOR Report on online panels. Public Opinion Quarterly, 74(4), 771–781.

Cauzard, J. -P., & Wittink, R. W. (Eds.). (1998). The attitude and behaviour of Europeancar drivers to road safety; SARTRE 2 reports part 1. Report on principal results.Leidschendam: The Institute for Road Safety Research SWOV.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale NJ: LawrenceErlbaum Associates.

Couper, M. (2000). Web surveys. A review of issues and approaches. Public OpinionQuarterly, 64(2), 464–494.

Dillman, D. A. (2007). Mail and Internet surveys: The tailored design method (2nd ed.).New York: John Wiley & Sons.

Dillman, D. A., Phelps, G., Tortora, R., Swift, K., Kohrell, J., Berck, J., et al. (2009). Re-sponse rate and measurement differences in mixed-mode surveys using mail, tele-phone, interactive voice response (IVR) and the Internet. Social Science Research,38(1), 1–18.

Duda, M. D., & Nobile, J. L. (2010). The fallacy of online surveys: No data are better thanbad data. Human Dimensions of Wildlife: An International Journal, 15(1), 55–64.

Duffy, B., Smith, K., Terhanian, G., & Bremer, J. (2005). Comparing data from online andface-to-face surveys. International Journal of Market Research, 47(6), 615–639.

Field, A. (2009). Discovering Statistics using SPSS. London: SAGE Publications Ltd.Franzen, G. (2006). The SWOCC book of brand management models. Amsterdam: SWOCC.Frippiat, D., & Marquies, N. (2010). Web surveys in the social sciences: An overview.

Population_E, 65(2), 285–312.Gigliotti, L. M. (2011). Comparison of an internet versus mail survey: A case study.

Human Dimensions of Wildlife: An international Journal, 16(1), 55–62.Heeren, T., Edwards, E. M., Dennis, J. M., Rodkin, S., Hingson, R. W., & Rosenblom, D. L.

(2008). A comparison of results from an alcohol survey of a prerecruited internetpanel and the national epidemiologic survey on alcohol and related conditions.Alcoholism, Clinical and Experimental Research, 32(2), 222–229.

Heerwegh, D. (2009). Mode differences between face-to-face and web surveys: an ex-perimental investigation of data quality and social desirability effects. InternationalJournal of Public Opinion Research, 21(1), 111–121.

Heerwegh, D., & Loosveldt, G. (2008). Face-to-face versus web surveying in a high-internet-coverage population. Differences in response quality. Public Opinion Quarterly,72(5), 836–846.

Heiervang, E., & Goodman, R. (2011). Advantages and limitations of web-based sur-veys: evidence from a child mental health survey. Social Psychiatry and PsychiatricEpidemiology, 46(1), 69–76.

Holbrook, A. L., Green, M. C., & Krosnick, J. A. (2003). Telephone versus face-to-faceinterviewing of national probability samples with long questionnaires: Compari-sons of respondent satisficing and social desirability response bias. The Public Opin-ion Quarterly, 67(1), 79–125.

Howell, R. T., Rodzon, K. S., Kurai, M., & Sanchez, A. H. (2010). A validation of well-beingand happiness surves for administration via the internet. Behavior ResearchMethods, 42(3), 775–784.

Joffe, M. M., & Rosenbaum, P. R. (1999). Invited commentary: Propensity scores. AmericanJournal of Epidemiology, 150(4), 327–333.

Joinson, A. (1999). Social desirability, anonymity, and Internet-based questionnaires.Behavior Research Methods, Instruments, & Computers, 31(3), 433–438.

Krosnick, J. (1991). Response strategies for coping with the cognitive demands of atti-tude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236.

Lampert, M., van der Lelij, B., Knoop, L., & Egberink, W. (2002). Awareness and attitudeswith respect to genetic modification. In G. Bartels, & Wil Nelissen (Eds.),Marketingfor sustainability, towards transactional policy-making. Amsterdam: IOS Press.

Lindhjem, H., & Ståle, N. (2010). Can cheap panel-based internet surveys substitute costlyin-person interviews in CV surveys? MPRA Paper 24069. Munich: University Library ofMunich.

20 C. Goldenbeld, S. de Craen / Journal of Safety Research 46 (2013) 13–20

Link, M. W., & Mokdad, A. H. (2004). Effects of survey mode on self-reports of adult al-cohol consumption: A comparison of mail, web and telephone approaches. Journalof Studies on Alcohol, 66(2), 239–245.

Malhotra, N., & Krosnick, J. A. (2007). The effect of survey mode and sampling on in-ferences about political attitudes and behavior: Comparing the 2000 and 2004ANES to internet surveys with nonprobability samples. Political Analysis, 15(3),286–323.

Marta-Pedroso, C., Freitas, H., & Domingos, T. (2007). Testing for the survey mode effecton Contingent valuation data quality: a case study of web based versus in-personinterviews. Ecological Economics, 62(3/4), 388–398.

Nielsen, J. S. (2011). Use of the Internet for willingness-to-pay- surveys. A comparison offace-to-face and web-based interviews. Resource and Energy Economics, 33(1), 119–129.

McCarty, J. A., & Shrum, L. J. (2000). The measurement of personal values in survey re-search: A test of alternative rating procedures. The Public Opinion Quarterly, 64(3),271–298.

Olsen, S. B. (2009). Choosing between internet and mail survey modes for choice ex-periment surveys considering non-market goods. Environmental and ResourceEconomics, 44(4), 591–610.

Roster, C. A., Rogers, R. D., Albaum, G., & Klein, D. (2004). A comparison of responsecharacteristics from web and telephone survey. International Journal of MarketResearch, 46(3), 359–373.

SARTRE (1994). European drivers and traffic safety. Paris: Presses des Ponts et Chaussées.SARTRE (2004). SARTRE 3, European drivers and road risk. Part 1: Report on principal re-

sults. Paris: Institut National de Recherche sur les Transport et leur Sécurité (INRETS).SARTRE-4 (2012). European road users’ risk perception and mobility. The SARTRE 4 sur-

vey. Paris: IFSTTAR.Smyth, J. D., & Pearson, J. E. (2011). Internet survey methods: A review of strengths,

weaknesses and innovations. In M. Das, P. Ester, & L. Kaczmirek (Eds.), Social andbehavioral research and the Internet (pp. 11–44) New York: Routledge.

Stephenson, L. B., & Crête, J. (2010). Studying political behavior: A comparison of inter-net and telephone surveys. International Journal of Public Opinion Research, 23(1),24–55.

Taylor, P. A., Nelson, N. M., Grandjean, B. D., Amatchkova, B., & Aadland, D. (2009).Mode effects and other potential biases in panel-based internet surveys: final report.Laramie: Wyoming Survey & Analysis Center, University of Wyoming.

TNS Opinion and Social (2010). E-Communications Household Survey. Special Eurobarometer,335, Brussels: European Commission.

Ye, C., Fulton, J., & Tourangeau, R. (2011).More positive ormore extreme?Ameta-analysisof mode differences in response choice. Public Opinion Quarterly, 75(2), 349–365.

Yeager, D. S., Krosnick, J. A., Linchiat, C., Javitz, H. S., Levendusky, M. S., Simpser, A., et al.(2011). Comparing the accuracy of RDD telephone surveys and internet surveysconducted with probability and non-probability samples. Public Opinion Quarterly,75(4), 709–747.

Mr. Charles Goldenbeld (Ph.D. social sciences University Utrecht), is 51 years old. Af-ter having worked 5 years as a researcher and lecturer at the Department of Social andOrganization Psychology at the University of Utrecht, he received his Ph.D in the socialsciences in 1992. He joined the Institute for Road Safety Research SWOV in 1992 wherehis current position is that of senior researcher. His main research interests includetraffic enforcement and evaluation of traffic behaviour interventions.

Saskia de Craen has been a researcher at SWOV Institute for Road Safety Researchsince 2000. As psychologist specialized in Methodology and Psychometrics (MSc,Leiden University) she has worked on several traffic safety subjects, such as accompa-nied driving, (2nd phase) driver education and practical driver training for mopeddrivers. In 2010, she received her PhD after a study on risk perception and assessmentof driving skills of young, novice drivers.