rss oct 2011 mixed modes pres3

1

RSS Social Statistics Section, 11 October 2011

The Role of Visual and Aural Communication in Producing Mode Effects on Answers to Survey Questions

Peter Lynn1, Steven Hope2, Annette Jäckle1, Pamela Campanelli3,

Gerry Nicolaas4

1 ISER, University of Essex2 University College London3 Independent Survey Methods Consultant4 National Centre for Social Research

1/39 RSS Social Statistics Section, 11 October 2011

Channels of Communication

In surveys, communication is primarily via sight (visual) and sound

(aural), of different forms and in different combinations

The nature of communication can affect:

• Comprehension

• Retrieval from memory

• Judgement

• Response

Via task difficulty - affecting whether the effort made is sufficient

Via context - affecting question processing

(Jäckle et al presentation)

2


Communication in Different Survey Modes

• Telephone surveys: communication is usually restricted to aural

form;

• Postal surveys: communication is restricted to visual form;

• Web surveys: usually just visual, but sometimes aural too;

• Face-to-face: both visual and aural

Administering Survey Items

The administration of a survey item has three main components:

• Asking the question;

• Offering response options;

• Receiving the answer.

Channels of communication can differ between items in a survey and

between components of an item.

Example: From the respondent’s perspective, a question in CAPI,

using a show card, involves these channels of communication:

Aural: respondent hears the question spoken by the interviewer;

Visual: respondent reads the response options on the card;

Oral: respondent speaks his/her response to the interviewer.


3

Our General Hypotheses I

H1. Visual rather than aural presentation of a survey question or

response options tends to increase the propensity that a

respondent will correctly understand the question

H2. Visual rather than aural presentation tends to reduce the

propensity that a respondent will take short cuts in responding due

to satisficing


Our General Hypotheses II

H1b, H2b. Effects on understanding or shortcutting will be mediated

by the cognitive ability and motivation of the respondent.

Specifically, visual rather than aural presentation will have positive

effects (better understanding; less shortcutting) for respondents of

higher ability and motivation and negative effects for respondents of

lower ability and motivation.

H1c, H2c. Effects on understanding or shortcutting will be mediated by

characteristics of the question. Specifically, positive effects of visual

presentation will be confined to questions that are relatively difficult

or burdensome and will be stronger for non-factual questions that

factual ones;


4

Our Study

4 sets of experimental manipulations, each involving between 1 and 6

survey questions.

Collectively, experiments address our hypotheses


Experiment 1

Visual vs. Aural Communication of Response Options in

CAPI

Randomised experiment, with and without show cards

Face to face interviews in respondents’ homes

6 items: 4 factual and 2 satisfaction

Satisfaction items: 7-category, ordinal

Factual items: 7-cat ordinal, 8-cat ordinal, 7-cat nominal, 8-cat nominal


5

Experiment 1: Methods

Logistic regression with controls for variables predictive of unit non-

response (sex, age, economic status, marital status, ethnic group)

Dependent variables:

a) Indicator of choice of middle or extreme categories (satisficing)

b) Indicator of choice of first n (primacy) or last n (recency) categories

presented (n = 1, 2 or 3 depending on question and response

distribution)

Tested main effect of treatment and interactions with:

a) Cognitive ability indicators (age and education level)

b) Motivation indicator (interviewer perception)


Experiment 1: Results

Visual communication (show card) reduced shortcutting (H2):

• Fewer middle/extreme responses for 1 of 2 satisfaction items

• Fewer first categories for 3 of 6 items

Interactions with cognitive ability and motivation (H2b):

• For one item (monthly expenditure on leisure activities), the

reduction in primacy was restricted to respondents aged under 65;

• For one item (proximity of features to home), Visual

communication increased primacy for less motivated respondents

Dependency on question characteristics (H2c):

• Effects observed for 2/2 difficult factual Qs, 0/2 easy factual, 1/2

non-factual


6

Experiment 2

Visual vs. Aural Communication of Question and Response

Options (CAWI vs CATI)

Same 6 items: 4 factual and 2 satisfaction

Web has only visual communication; CATI has only aural

communication

Note that both modes also have communication of the question in the

same mode as the response options (unlike experiment 1)

Interviewer ‘presence’ can also affect the CATI responses, but effect

likely to be weaker than for face-to-face, so it is possible that the

effect of channel of communication is the dominant one

Methods same as experiment 1



Visual communication (CAWI) and shortcutting (H2):

• No main effect;

Interactions with cognitive ability and motivation (H2b):

• Shortcutting increased for respondents without qualifications only

(1 Q, middle+extremes) and for less motivated respondents (1 Q,

primacy)

Dependency on question characteristics (H2c):

• No support for H2c: effects found were for 1/2 difficult and 1/2

easy factual Qs


7

Spend on Leisure Activities


0

5

10

15

20

25

30

35

< £20 < £40 < £60 < £80 < £100 < £120 < £140

CATI

CAWI

Experiment 3

Visual Communication of Response Options in CAWI

One item with end-labelled 10-point scale

Version one: ten radio buttons, displayed linearly

Version two: number box


8

A. On a scale of 1 to 5, with one being

very satisfied and 5 being very

dissatisfied, how satisfied are you with

the classes you are taking this

semester?

�1 Very Satisfied

�2

�3

�4

�5 Very Dissatisfied

B. On a scale of 1 to 5, with one

being very satisfied and 5 being

very dissatisfied, how satisfied are

you with the classes you are taking

this semester?

Number of your rating

Evidence that 10% of people changed their answer on B because they had

forgotten the direction of the scale! Further evidence that others answering B who

had forgotten the direction of the scale, did not realise their mistake.

Christian and Dillman, 2004


Experiment 3: Version 1


9

Experiment 3: Version 2


Experiment 3: Methods

Our specific hypotheses are that visual presentation (radio buttons)

will produce:

• Less shortcutting, indicated by middle or extreme categories;

• Better understanding, indicated by internal validity (correlations);

• Greater use of categories 2 to 9.

Methods:

• Logistic regression with controls as before. Dependent variables

are a) indicator of choice of middle or extreme categories,

b) indicator of choice of categories 2 to 9

• Comparison of correlations with conceptually-related Qs


10


Visual communication (radio buttons) brought:

Less shortcutting (H2);

More choice of options 2 to 9;

Greater validity (H1) (significantly higher correlations for 5 of 9 items,

though lower correlation for 1 item);

Increase in validity was for respondents with both high and low

cognitive ability and motivation (H2b):


Experiment 4

Visual vs. Aural Communication of Response Options for

End-labelled Scales (CAPI with show card vs CATI)

Four items with end-labelled scales: response task similar to experiment 3

All scales had 7 points

2 satisfaction questions and 2 difficult factual questions

Hypotheses and methods same as for experiment 3


11


Visual communication and shortcutting (H2):

• Less shortcutting with visual (CAPI) – significant for one item,

borderline for one (P = 0.08), consistent direction for all 4;

Mediation by question characteristics (H2c):

• Effects for two difficult factual items, not for non-factual items;

Use of options 2 to 9:

• Greater with visual (CAPI) for one item (other 3 no difference);

Visual communication and validity (H1):

• For 3 items, not possible to identify correlated items; for 4th item,

lower correlation with visual (CAPI).


Summary

Visual communication and respondent understanding (H1):

• Strong support from expt 3 (CAWI); no support from expt 4 (CAPI

vs CATI)

Visual communication and shortcutting (H2):

• Modest or strong support from all 4 expts;

Mediation by cognitive ability and motivation (H1b, H2b):

• Support from expts 1, 2, 4; no support from expt 3;

Mediation by question characteristics (H1c, H2c): :

• Indicative support from expts 1 & 4; no support from expt 2


12

Conclusions and Implications

• Visual communication can aid understanding and full processing;

• But only for respondents of higher ability and motivation;

• And effects may be noticeable only for difficult/ complex questions.

For surveys of high-ability respondents, visual may be preferable

For surveys with only simple questions, channel of communication

may not affect measurement

For surveys of respondents with a range of abilities, and including

some difficult/complex questions, it may be preferable to utilise

both visual and aural channels, e.g.

• CAPI: Show cards and interviewer read out, when appropriate

• A-CASI / A-CAWI with question text displayed


Discussion I

Use of Show Cards

• We know little about how and when respondents process them and how

this influences measurement

• We don’t know how best to control the use of cards

Dual-channel Communication

• We know little about how respondents process such stimuli:

• Respondents may give primary attention to one channel;

• They may switch between channels;

• Processing behaviour may vary between respondents and, in particular,

be associated with cognitive ability, vision, hearing, etc;

• All of which may produce response effects


13

Discussion II

And for a better understanding of the role of cognitive ability and other

relevant between-respondent differences:

• Better measures of cognitive ability and motivation in mode

experiments;

• Measures of hearing and vision;

• Control of these features through design.

We should perhaps not be asking ourselves which channel is better/best,

but how best to combine them


Cognitive Interview ResultsExperiment 1: Showcard Effects (1)• Of the 6 questions tested, the best example of a primacy effect on the

aural (no show card version) was for expenditure on leisure activities

(except eating out)

• For the cognitive interviewing an additional question on “expenditure on

eating out” was added

• One group received the leisure activities question first, the other group

received that eating out question first

• For both groups, the first question did not have a show card and the

second one did

• Main focus on expenditure on leisure activities


14

Cognitive Interview ResultsExperiment 1: Showcard Effects (2)

• Note first category was lowest value

• There were misunderstandings of the question. Themes

observed:

• Some Rs incorrectly thinking about family expenditure rather than

personal expenditure

• Some Rs incorrectly including eating out

• Thus Rs answering the question correctly would have much lower

expenditures

• The ‘average month’ part of the question was occasionally missed:

• One respondent actually answered about a week and thus had very

small expenditure


Cognitive Interview ResultsExperiment 1: Showcard Effects (3)

• These confusions, although affecting the final category

chosen by Rs did not appear to explain the show card effect

as Rs who had had the show card reported similar

confusions.

• But when asked, “How did you find it when you had a card

compared to when you did not have a card?” the majority of

respondents clearly preferred the use of a card.


15

End-Labelled Scales(only tested in CAPI and CATI in cognitive interviews)


Survey Questions Used – End-Labelled Versions

Satisfaction Questions - Common

• Difficult: Satisfaction with democracy and personal freedom in Great

Britain (where 1 is very satisfied and 7 is very dissatisfied) - from ESS,

2006 (with addition of ‘and personal freedom’ to make the question more

difficult)

• Easy: Satisfaction with state of the economy in Great Britain (where 1 is

very satisfied and 7 is very dissatisfied) - from ESS, 2006

Behavioural Frequency Questions - Uncommon

• Difficult: Frequency of hot beverages purchased outside the home (where

0 is none and more than 25 is 6) – newly developed to fit hypotheses

• Easy question, but difficult due to scale: Frequency of grocery

shopping (where 1 is every day and 7 is never) – newly developed to fit

hypotheses


16

Aural delivery and confusion of direction of scale –

Experiment 4


Cognitive Interview Findings

• Support for Christian and Dillman (2004).

Although both CAPI and CATI Rs spontaneously

mentioned that they would have liked to have had

labels on all the categories, there is evidence

from respondent comments that the lack of visual

aid in CATI made the questions more difficult than

in CAPI.

In line with the findings of Christian and Dillman

(2004), a few Rs did confuse the direction of the

scale (slightly more in CATI than CAPI).


17

Middle Categorieson End-Labelled Scales –Other analyses


Middle Categories on End-labelled Scales - Quantitative

• Tables in Dillman & Christian (2005) show a higher percentage

of middle category answers in the end-labelled format than fully

labelled format

• Same pattern on attitudinal and behavioural questions in Mixed

Modes data

• Except for state of the economy - Respondents in both

formats and all modes gave economy a bad rating

• The middle category effect less in CATI


18

Cognitive Interview Findings

• How easy or difficult is it to choose the middle category on an

aural end-labelled scale? (This was explored through cognitive

probing.)

Although roughly equal numbers said easy and difficult, only a

third of Rs could correctly identify the mid-point!

True for the 7-point satisfaction scales (correct answer is 4: Rs

also said 3, 3.5, 3 or 4)

Larger issue for behavioural frequency ones

– Correct answer for grocery shopping question is 4: Rs also said 3, 3

or 4, between 3 and 4

– Correct answer for hot beverage question is 3: Rs also said 3.5, 3 or

4, 4, 1 or 2


Other analyses -

Similar Attitudinal and Behavioural

Frequency Distribution

on End-Labelled Scales


19

Similar Attitudinal and Behavioural Frequency Distribution on End-Labelled Scales - Quantitative

• Dillman & Christian (2005) found a higher percentage of

answers at the top of the scale (more positive answers and higher

mean) in the fully-labelled format than end-labelled (end-labelled

more spread out)

• Replicated in Mixed Modes study on attitudinal and

behavioural questions (except state of the economy).

• Remember middle category pattern on attitudinal and

behavioural questions also replicated Dillman & Christian

(2005) (except state of the economy).

• Given attitudinal and behavioural show such similar

distributions, it would be tempting to say that the same

response process is responsible


Cognitive Interview Findings (1)

Attitudinal & behavioural questions processed differently

• On satisfaction questions . . .

Rs were using the 1 to 7 scale

Choice of middle category was a way to ‘satisfice’, say ‘don’t know’, or choose a valid option.

• On frequency of grocery shopping question . . .

Some Rs tried to use the scale (1 everyday to 7 never), e.g., everyday is code 1; nearly every day is code 2

‘Once a week’ very difficult to place (answers ranged from 3 to 6!)

Others patterns

– Actual number of days of shopping

– Idiosyncratic methods, e.g., 4 means every 4th day; 5 means twice a

week


20

Cognitive Interview Findings (2)

• On frequency of hot beverages purchased outside the house . . .

– Rs didn’t use the scale (0 is none and 6 is more than 25);

– Common answer was the actual number of drinks;

– But a few Rs using this strategy became confused. If 5 drinks is code 5 why the large gap with ‘more than 25’ being code 6;

– Some Rs used multiples of 2 or 5.


Thank You!


21

Annex


References

Christian, L. M. & Dillman, D.A. (2004) The influence of graphical and symbolic language

manipulations on responses to self-administered questions, Public Opinion Quarterly 68, 57-

80.

Holbrook A L, Krosnick J A, Moore D & Tourangeau R (2007) Response Order Effects in

Dichotomous Categorical Questions Presented Orally. Public Opinion Quarterly 71, 325-348

Stern, M.J., Dillman, D.A. & Smyth, J.D. (2007) Visual design, order effects, and respondent

characteristics in a self-administered survey, Survey Research Methods 1, 121-138.


22

Survey Questions

N39x.

Which of the following is closest to where you live?

READ OUT

A primary school...........................1

A secondary school......................2

A 6th form college.........................3

A river............................................4

A lake............................................5

A cinema.......................................6

Or a theatre...................................7


N43y.

I would like you to tell me how satisfied or dissatisfied you are with local household waste

collection, recycling collection and other re-cycling collection points. Would you say you are…

READ OUT

Very satisfied....................................1

Moderately satisfied..........................2

Slightly satisfied................................3

Neither satisfied nor dissatisfied......4

Slightly dissatisfied............................5

Moderately dissatisfied......................6

Or very dissatisfied?..........................7


23

N44y

And how satisfied or dissatisfied are you with street cleaning? Would you say you are…

READ OUT

Very satisfied....................................1

Moderately satisfied..........................2

Slightly satisfied................................3

Neither satisfied nor dissatisfied......4

Slightly dissatisfied............................5

Moderately dissatisfied......................6

Or very dissatisfied?..........................7


FM75y.

Which of these best describes your home? Would you say a . . .

READ OUT

Detached house..................1

Semi-detached house ........2

Terraced house...................3

Bungalow ...........................4

Flat in a block of flats..........5

Flat in a house.....................6

Maisonette...........................7

Or other? ............................8


24

FM81x.

How much do you personally spend in an average month on leisure activities, and entertainment

and hobbies, other than eating out? Would you say. . .

READ OUT

Less than £20..............1

£20 - £39......................2

£40 - £59 .....................3

£60 - £79 .....................4

£80 - £99......................5

£100 - £119..................6

£120 - £139 .................7

Or £140 or more.?.......8


FM82x. How long have you lived in this area? Would you say. . .

READ OUT

Less than 12 months

12 months or more but less than 2 years

2 years or more but less than 3 years




Or 20 years or longer


25

GB16x. On the whole, how satisfied are you with the way democracy and personal freedom

work in Great Britain? Please look at this card and tell me your answer.

Very Satisfied .......... 1

2

3

4

5

6

Very dissatisfied…....7


GB17x. And on the whole, how satisfied are you with the present state of the economy in Great

Britain? Please look at this card and tell me your answer.

Very Satisfied .......... 1

2

3

4

5

6

Very dissatisfied…....7


26

FM68x.

The next question is about grocery shopping which includes food, drinks, cleaning products,

toiletries and household goods. How often do you personally do grocery shopping? Please

look at this card and tell me your answer.

Every day.................. 1

2

3

4

5

6

Never …….……….....7


FM74x. In the last two weeks, how many teas, coffees and other hot beverages have you

purchased outside the home? Please look at this card and tell me your answer.

None…........................ 0

1

2

3

4

5

More than 25..................6


rss oct 2011 mixed modes pres3

Documents

aural presentation

sight visual

role of visual

visual form web surveys

aural form postal surveys

question h2

negative effects

mode effects