mixed mode surveys

Mixed-mode surveys: Do different modes produce different data?

Evidence from the Greek Candidate Survey

Ioannis Andreadis

Aristotle University of Thessaloniki

Greek Candidate Survey 2007

• Target population: PASOK and ND candidates

• Mixed-mode survey (different data collection modes in sequence)

• Web Survey (self-administered)• Nonresponse follow-up by (CATI)

including a very limited number of FTF interviews conducted as CATI (no show cards)

To mix, or not to mix?

• A survey conducted without spending a single Euro

• Mixed-mode to reduce total survey error• Avoid coverage error

– Avoid omissions (lists from political parties)– Avoid duplications and erroneous inclusions

(PIN must be used to enter the survey)

• Without introducing measurement error?

Mode Impact of CATI

• Impact of Interviewer– Social desirability– Positive in interview

– Acquiescence (tendency to agree)

• Aural channel– Recency effect (the last or one of the last

offered answer categories is chosen)

Web survey

• Coverage: “Non-Internet households are older, have less education, and lower incomes”.

• Multitasking (Skype call, new email, etc)• Possible primacy effect?

• Do the answers provided by respondents differ by mode?

Age, Education, Income

• YoB: sig. difference (t=3.47, p=0.001)– Web average: 1955.4, CATI average: 1960

• Education– No difference (circa 95% university in both

groups)

• Income– No data, but all of them can afford to pay for

an internet connection

Outcome of election and party

• Web: 160 responses RR: (20%-60%)• 30% of the total population have been elected;

only 20% of web survey participants• 116 (72.5%) are PASOK candidates.

• Why the ratio of elected MPs is smaller in the group of Web respondents? (discussion)

• Why PASOK candidates are more than ND candidates? (discussion)

Matching (which variables to include?)

• Age reported by 76 of 81 (CATI) and 104 of 160 (Web) – Web: break-offs (age one of the last questions of a

long questionnaire); Research in other sources raised the numbers to 79 and 139. Web average age raised by 0.84 years (older people – more dropouts?)

• On the other hand the impact of this difference (by 5.6 years) on the questions asked in CCS may not be worth the trouble

• Exact matching without age (outcome and party)– Only 4 sub-classes– No need to drop cases

Chances to win

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

WEB CATI WEB matched

I could (not/hardly) win It was an open race I could (hardly/not) lose

X2=14.6 p=0.006

X2=5.8 p=0.218

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


0-2 3-4 5 6-7 8-10

Left-Right Self-placement

X2=24.6 P<0.001

X2=2.4 p=0.661

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


satisfied not satisfied

Satisfaction with Democracy

X2=6.0 p=0.014

X2=1.4 p=0.234

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


satisfied not satisfied

Satisfaction with Democracy in EU

X2=4.4 p=0.037

X2=0.2 p=0.885

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


strongly agree agree neither agree nor disagree disagree

Our democracy needs serious reform

X2=12 p=0.007

X2=4.3p=0.233

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


strongly agree agree other

Income and wealth should be redistributed to ordinary people

X2=13.4 p=0.001

X2=3.5p=0.177

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


agree neither agree nor disagree disagree

Women should be free to decide on matters of abortion (direction)

X2=6.4 p=0.040

X2=4.5 p=0.107

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%



Women should be free to decide on matters of abortion (intension)

X2=15.8 p<0.001

X2=11.1 p=0.004

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%



Providing a stable network of social security prime goal (intension)

X2=17.0 p<0.001

X2=11.9 p=0.003

Findings contradict previous research on mode effects

• It is reported that CATI is associated with extreme positive replies (strongly agree)

• Reverse order of agree disagree in CCS?• Do Recency and Acquiescence drive CATI

respondents to give the last offered answer category that matches their direction?

Conclusions

• Web and CATI give similar results after matching Web respondents to CATI respondents

• Some differences when (almost) all replies are located in the same direction and differences are due to intensity

• An idea (for discussion) on how recency and acquiescence influences CATI respondents

Explanation of different ratios

• Elected are more busy. Their ratio is still lower in 2009• Those who failed to elect want to review the situation

and learn more about what they have done wrong or missed?

• In the sample of 2009 ND candidates are slightly more than PASOK candidates. From this observation I conclude that the larger ratio of PASOK candidates in 2007 was partly due to different rates of elected MPs between the two political parties. The proportion of 2007 elected MPs was 37% in the group of ND candidates and 24% in the group of PASOK candidates. This picture was reversed in the 2009 elections.

The future of web surveys in Greece

• Large increase in penetration of Internet use in Greece (Eurostat 2009) . – The percentage of Internet users increased

from 18% in 2003 to 45%

– The percentage of households with Internet access rose from 12% in 2002 to 38% in 2009

– In 2009 33% of households have broadband Internet connection

Candidate Survey Objects

• Campaigns – No differences between the two modes

• Issues and Policies– A lot of differences – Most of them corrected by matching– Direction vs Intensity

• Democracy and Representation– Some differences

Dillman et al (2009) Social Science Research

• For example, in Treatment 2 (Form A), overall satisfaction was measured by interviewers reading from the most positive rating labels to the most negative rating labels, ‘‘where ‘5’ means extremely satisfied, and ‘1’ means not at all satisfied. . ..” In Treatment 3 (Form B) respondents heard the question with the most negative rating label first, as ‘‘where ‘1’ means not at all satisfied and ‘5’ means extremely satisfied. . .” You may use any number.

• If a recency effect occurs we would expect that those responding in Treatment 2 ‘‘Not at all satisfied” (i.e., last category presented to the respondent) would be significantly different in comparison to the same response in Treatment 3. Likewise, those responding in Treatment 3 ‘‘Extremely satisfied” (i.e. last category presented to the respondent) would be larger than in the other treatment. This is not the case. A difference of means test shows that none of the differences are statistically significant and there is no consistent trend in either direction. All chi-square tests for differences were also insignificant. Because of the lack of differences, or ever slight trends, these results are not shown in the table. Also, because of the lack of differences we have combined Treatments 2 and 3 together for all other analyses reported in this paper.

Comment on the previous finding

• Recency occurs when all five (or more categories) are read. If polar points are the only labeled points and respondent is requested to provide a number, the possibility of recency is small.

Proposed model for CATI 1/2

• If order: strongly disagree – strongly agree– Respondent agrees– Recency towards strongly agree– Acquiescence towards agreement

• Outcome: strongly agree, Conclusion: extreme responses

• If order: strongly agree – strongly disagree– Respondent agrees– Recency towards strongly disagree– Acquiescence towards agreement

• Outcome: agree, Conclusion: non extreme responses

Proposed model for CATI 2/2

• If order: strongly disagree – strongly agree– Respondent disagrees– Recency towards strongly agree– Acquiescence towards agreement

• Outcome: disagree – neither/nor (significant impact of acquiescence)

• If order: strongly agree – strongly disagree– Respondent disagrees– Recency towards strongly disagree– Acquiescence towards agreement

• Outcome: disagree or strongly disagree

mixed mode surveys

Technology