Self-Administered CognitiveInterviewing

or

Do Respondents Really Need Us?

Jennifer Edgar

AAPORMay 16, 2013

Background

Cognitive interviews are typically face to faceinterviews used for problem identification

In-depth probing used to explore responsestrategies

Small sample sizes are common, thoughsome research suggests larger sample sizescan increase results (Blair & Conrad, 2011)

Traditional cognitive interviews are timeintensive

2

Past Research

It is possible that the goals of cognitiveinterviewing can be met using a selfadministered format

Promising findings (Edgar, 2012)Quantity and quality of data was

comparableEfficiencies in cost & data collection time

Research focused on response strategye.g. “How did you arrive at that answer?”

3

Research Questions

Can information typically obtained throughcognitive interviews be collected using a selfadministered web survey?

Are self administered web interviews anefficient way to collect data about questioncomprehension?Are substantive conclusions the same?Are potential issues identified the same?

4

Method

43 traditional cognitive interviews 34 self administered, web interviews Same protocol used when possible

5

Questions

Testing of new and revised questions for BLSproduction survey

Focused on comprehension

1. Swimsuits, warm-up and ski suits2. Delivery and installation charges3. Mortgages

6

Lab Cognitive Interviews

Traditional cognitive interviewsSemi-structured, combination of scripted

and spontaneous probing Testing purpose and method explained to

participants Conducted in three US cities Most interviews lasted 45-50 minutes

7

Self Administered Interviews TryMyUI: remote usability testing serviceVolunteer panelParticipants rated on past work

Questions administered via web survey Participants given survey question and follow-

up questions (following lab protocol)Five spaces given for examples

Video and audio recording of computerscreen and participant voices

Sessions limited to 20 minutes8

Tasks

Respond to survey question Explain what you were thinking of when

you answered the question Give examples of items in category Complete inclusion/exclusion exercise

Participants

CI WebGender

Male 51% 26%Female 49% 74%

AgeAverage 47.6 38.2Median 48 36

Same demographic requests for both modes

Analysis

Transcripts were made of all think aloudresponsesFor cognitive interviews, number of probes

and responses to follow up probes were alsorecorded

Web survey data compared with data enteredfrom cognitive interviews

Results: Sportswear

Straightforward test of question comprension,little evidence of issues

Lab participant issues were with task (“give meexamples of what a swimsuit is”) not question

Web participants gave more examples Additional probing (average 1.2) did not

encourage lab participants to add examples

Avg. # of examples% of Ps with issue First Probe Total

CI 5% 2.6 2.7Web 0% 3.6

Results: Delivery and Installation

13

More complicated question construct Lab participants had more issuesIncluding shipping or other incorrect

charges (e.g. pizza)Naming items that were delivered for free

Avg. # of examples% of Ps with issue First Probe Total

CI 44% 4.1 / 4 4.2 / 4Web 2.7% 3.7 / 4

Results: Mortgage

14

Many people showing evidence of issues Lab debriefing revealed a ‘show stopper’ issue related

to refinancing, no evidence of this found on the web Web participants made comments indicating

sensitivity issues, no evidence of this in the lab

GeneralUncertainty

ComprehensionRefinance Issues

SensitivityIssues

CI – think aloud 46% 18% 0%Web –whileanswering

21% 3% 3%

Web – thinkaloud

29% 3% 25%

Substantive Conclusions

15

SportswearCI: question works fineWeb: question works fine

DeliveryCI: question has potential comprehension

issueWeb: question works fine

MortgageCI: question has show stopping issueWeb: question is potentially sensitive

Conclusion: Comprehension

16

Minor differences found in how peopleunderstood the question, or in how theyexplained their understanding of it

Web participants had fewer issues, but alsonamed fewer examples

Could be evidence of the lab probingtechnique creating questionsParticipants working hard to give more

examples, and ended up making mistakes

Conclusion: Sensitivity

17

Different conclusions about question sensitivityCIs included more expenditure questions

preceding the mortgage questions,participants may have gotten used toanswering this type of questionsBoth groups a ‘spiel’ about the study at the

beginning, but web participants skimmed it Self-administration is often thought to be

better for sensitive questions, maybe not forpretesting?

Conclusion: Question Function

‘Show stopper’ issue found in the lab notfound online

Follow-up probing identified the issue, whichwasn’t identifiable through the question-answer process or the single probe usedonline

Additional follow-up questions could havebeen added to the web survey, if potentialissue was caught in time

18

Observed Limitations

Some answers or explanations requiredfollow-up or re-focusingComments which would have prompted

probing went un explainedSeveral participants spent time going to

their banking site to look up mortgageinformation

Technical issues held some participants upand they spent their time working them outrather than completing the survey

19

Observed Strengths

Some web participants talked aloud the wholetime, giving useful information throughoutthe process.Lab participants generally talk only when

asked a question Some web participants gave un-requested,

honest feedback about the questions

20

Method Advantages

Self administered interviewing saves time andmoney

Large amounts of data can be easily collectedMaking the potential ‘bad’ data less

important as those cases are easilyreplaced

There is no interviewer effect on the web

21

Method Disadvantages

Potential for bias, if participants systematicallydiffer by mode (web participants were younger)

Instructions must be clear and work foreveryone

Cannot provide probes to follow up onparticipants’ commentsAll probes must be scripted and therefore

must be applicable to all participants No way to bring participants ‘back on track’

22

Conclusions

Can information typically obtained throughcognitive interviews be collected using a selfadministered web survey? YES

Are self administered web interviews anefficient way to collect data about questioncomprehension and recall? YESAre conclusions about the questions the

same? Not AlwaysAre potential issues identified the same? NO

23

Potential Applications Combination of standard CI and web studies

is likely an effective approachTake advantage of the cost and time

efficiency of web collectionFocus lab testing on in-depth probing

Run studies concurrently, modifying webprobes based on lab testing results

24

Contact

Edgar.Jennifer@Bls.gov

Study Costs

Total Cost PerParticipant

ParticipantIncentive

Web $27.30 $10

CI $42.66 $40

26

Study TimesTask CI Web

Requestingparticipants

20 minutes; explainingcriteria to recruiter

30 minutes total;specifying test groups

and criteria

Screening 10 minutes perparticipant

0 minutes; done byTryMyUI

Scheduling &confirming

15 minutes perparticipant

0 minutes; study doneat participantconvenience

Preparing forinterviews

10 minutes perparticipant

60 minutes total;setting up Web survey

and tasksConductinginterviews

45 minutes perparticipant

0 minutes, self-administered

Total 31.7 hours 1.5 hours