the measurement effect in pc smartphone and tablet surveys valerija kolbas university of essex iser...
TRANSCRIPT
![Page 1: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/1.jpg)
The measurement effect in PC smartphone and tablet surveys
Valerija Kolbas
University of EssexISER
Ipsos-MORI
The European Survey Research Association Conference 13-17 July, 2015 Reykjavik
![Page 2: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/2.jpg)
Background and motivation Limiting surveys to the PC mode
affects the size and representativeness of the sample
PC, smartphones, tablets differ in: size of the screen input method speed of connection processing power
Different effect on the measurement error
![Page 3: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/3.jpg)
Background and motivation
Smartphone more break-offs longer completion time More straightlining
Mixed results
Other indicators question order effect primacy effects open-ended answers response distribution
… but non-optimized mobile design affects completion rates, satisfaction with the survey
Tablet Less primacy effect Less straightlining Less or comparable
completion time
![Page 4: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/4.jpg)
Background and motivation
PC and mobile response distributions are equally affected by response formats
Drop-boxes –preference for first options Grids – preference for visible options,
straightlining
- no conclusive evidence which format is better for mobiles
![Page 5: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/5.jpg)
Research Question
How mode of administration and response format affect survey responses
Indicators of measurement error:
overall satisfaction rates straightlining response distribution length of open-end questions
![Page 6: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/6.jpg)
Survey and Questionnaire design National Satisfaction Survey 2014 Administered to final year higher education
students in UK 22 core questions using a 5-point Likert scale 2 open-end questions Mixed-mode: self-selected mail, phone, web 5 response formats – randomly allocated
![Page 7: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/7.jpg)
Sample composition
PCN=5529
SmartphoneN=3196
TabletN=551
Radio-button
Drop-box ()
Drop-box (0)
Drop-box (-)
Drop-box (+)
Radio-buttonWeb
Survey SampleN=9276
![Page 8: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/8.jpg)
Screenshots: response design
• Once clicked a list with options appears on a separate screen.• Responses always in the same order
![Page 9: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/9.jpg)
Screenshots: response design
• All questions are visible on the screen.
• Requires horizontal scrolling .• Portrait or landscape viewing.
• Requires vertical scrolling.
![Page 10: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/10.jpg)
10
Measurement Effect between PC smartphone
tablet responses
Comparisons madeacross all three modes, but within one radio-button response format
![Page 11: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/11.jpg)
Measurement Effect between PC smartphone tablet
F p PC smartphone tablet
Straightlining
3.9 <.05 6.9% 10.3% 6.4%
Modal responses
<1 >.05‘Mostly Agree’
‘Definitely Agree’
&‘Mostly Agree’
‘Mostly Agree’
Mean values
2.7 .07 4.1 4.1 4
Positive feedback
1.3 >.05 195 180 191
Negative feedback
1 >.05 257 240 243
MANOVA to test for differences F=2.3, p<.05
![Page 12: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/12.jpg)
ME between PC, smartphone, tablet summary
Smartphone straightlining significantly higher
Tablet straightlining rate the lowest Signs of the visibility effect for smartphone Other quality indicators comparable
between all three modes
![Page 13: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/13.jpg)
13
Measurement Error between different response designs
presented on a smartphone and a tablet
Comparisons made across five survey response formats within smartphone and tablet modes
![Page 14: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/14.jpg)
MANOVA test of differences F=2.7, p<.05
F p
device 2.7 <.05
format 1.7 <.05
interaction 1.3 .18
Both device and format affected data quality
![Page 15: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/15.jpg)
Responses in Drop-box with a positive initial option
Def.A
gree
Mos
tly A
gree
Neith
er
Mos
tly D
isag
ree
Def. D
isag
ree
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
smartphone
tablet
Smartphones
higher selection of initially suggested response.
less moderately positive responses 11.1% straightliners Mean 4.3
Tablets
Similar selection of extreme and moderately positive responses
More negative responses 5.6% straightliners Mean 4.1
![Page 16: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/16.jpg)
Responses in Drop-box with a negative initial option
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
45.00%
50.00%
smartphone tablet
Smartphones
More extreme negative responses Less positive responses 8.4% straightliners Mean 4
Tablets
More positive responses Extremely low ‘Definitely Disagree’
frequency 2.5% straightliners Mean 4.1
![Page 17: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/17.jpg)
Def.A
gree
Mos
tly A
gree
Neith
er
Mos
tly D
isag
ree
Def. D
isag
ree
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
45.00%
smartphonetablet
Responses in Drop-box with a middle initial option
Smartphone and Tablet
Comparable results
Weak evidence of selecting the middle option
Most frequent ‘Mostly Agree/Disagree’ selection across formats
6.5% and 6.1% of straightliners Mean 4 an 3.9
![Page 18: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/18.jpg)
ME indicators summary
Smartphone – answers affected by response formats.
Initially suggested response is selected more often
Tablet – no strong effect of response formats.
No significant differences in the length of open answers between formats.
![Page 19: The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association](https://reader033.vdocuments.us/reader033/viewer/2022051417/5697bff11a28abf838cbb193/html5/thumbnails/19.jpg)
Potential Limitations
No reverse-coding Similar question wording Not counterbalanced question or response
order Instructions universal for each response
format Self selected device condition Survey sample: highly educated, IT literate,
similar age, highly motivated