CASRO PANEL CONFERENCE 2008 Miami

Maximizing respondent engagement through survey design

Presented By:

Su Ning Strube, Vision Critical

Yola Zdanowicz, Angus Reid Strategies

Presentation Outline

• User Interface Design• Hypothesis• Methodology• Respondent Engagement• Impact on Survey Data• Conclusions

Survey Terminology

• Flat– Traditional online survey approach with check boxes and

radio buttons from the host operating system– Familiar “flash card” navigational metaphor

• Fusion– Advanced design utilizing rich media techniques– Developed in Adobe Flash

Hypothesis

• Fusion relative to flat:

– Higher respondent engagement

– Survey perceived as shorter

– Higher completion rates

Methodology and Sample Design

• Online data collection

• Sample sourced from Angus Reid Forum

• Respondent qualification– Flash Player version 8

• Split sample design (1246 Fusion/1227 flat)

• Two languages (English/French)

• Data weighted to Canadian population demographics

Design and Presentation of Controls

Combined Questions

Follow-on and Pop-up Questions

Scale/Sort Questions

Respondent Engagement

Fusion is clearly more fun to complete, enjoyable and engaging.

Future Participation

Fusion is a more powerful vehicle for promoting participation in future research.

Perceived Length

Fusion and flat were perceived as equal in length. In reality, Fusion took longer to complete.

Response and Completion Rates

Completion rates high for both surveys but higher for flat.

Impact on Survey Data

• No difference between Fusion and Flat for the majority of questions:– Food consumption (behavior)– Purchasing of food (behavior and knowledge)– Demographics– Computer technical profile

• No difference between Fusion and Flat across different types of questions:– Combined questions– Follow-on & Pop-Up Questions

• Exceptions were open-ended questions, knowledge questions with “don’t know” option, scale/sort exercise.

Impact on Survey Data :: Open Ends

Fusion encourages more detail.

Open Ended Questions

Impact of Survey Data :: Knowledge Questions with “Don’t Know”

Fusion results in more “don’t know” responses.

Knowledge Questions with “Don’t Knows”

Impact of Survey Design :: Scale/Sort Exercise

Greater scale use with Fusion, towards stronger disagreement

Impact of Survey Data: Scale/Sort Exercise

Conclusions

• Fusion relative to flat:

– Higher respondent engagement– Perceived as equal in length – in reality Fusion

takes longer – High completion rates for both but lower for Fusion – Greater detail for open ended questions– Greater use of “don’t know” for select knowledge

questions– Greater scale use – particularly toward stronger

disagreement

Conclusions

• Promise for Fusion bodes well for the future of panels

– Helps to keep panelists interested and engaged– Could improve retention rates– Encourages panelists to provide thoughtful, detailed responses to

research questions– Challenges the mantra of “shorter is better”

• More research is needed (i.e. usability testing) to expand our knowledge of:

– Respondent engagement– Strengths & weakness of specific question design– Overall completion rates

Q&A