a comparison of cognitive testing methods and sources - final · jessica l. holzberg, gerson...

A Comparison of Cognitive Testing Methods and Sources:

In-Person versus Online Nonprobability and Probability

MethodsJessicaL.Holzberg,GersonMorales,Aleia ClarkFobia,

andJenniferHunterChildsU.S.CensusBureau

QDET2November11,2016

Disclaimer:AnyviewsexpressedarethoseoftheauthorsandnotnecessarilythoseoftheU.S.CensusBureau.

Motivation§ Wecangetvaluablecognitivetestingfeedbackusingunmoderated,onlineservices(e.g.,Edgar2013;Fowleretal.,2015;Cooketal.,2015)

§ Feedbackcanvarybynonprobabilitysamplesource(e.g.,Murphy,Edgar,&Keating,2014)

§ Onlineopt-innonprobabilitysamplessometimesdemographicallyskewed§ Maynotbeviableforsometypesofcognitivetesting

2

Motivation (cont.)§ Presentstudyisacomparisonbetween:

§ Twoonlinenonprobabilitymethods§ Traditionalin-personcognitiveinterviews§ Probabilitysample(coldcontact)

§ ResearchQuestions§ Howdoconclusionsaboutcomprehensionfromcognitivetestingfeedbackdifferbysamplesource?

§ Howdothedemographiccharacteristicsofrespondentsdifferbysamplesource?

3

Cognitive testing§ Assessedcomprehensionof36officialCensusBureaumessages*§ “TheCensusBureauwillneveruseyourresponsesforanythingotherthanstatisticalresearch.”

§ “Wewillnevershareyourinformationwithlawenforcementorallowittobeusedtodetermineyoureligibilityforgovernmentbenefits.”

*seeFobia andChilds,2016AAPOR

4

Instrument for online testing§ Respondentswererandomlyshown9messageson9screens,inrandomorder:§ Fivemessagesonprivacyandconfidentiality§ Onemessagefromeachoffoursub-sectionsonrequiredlanguage(burden,mandatoryresponse,OMBnumber,otherconfidentialityprotections)

§ Aftereachmessage,respondentswereasked,“Inyourownwords,whatisthismessagetellingyou?”withanopen-endedtextbox

§ Demographicquestions

5

Data sources (cont.)Nonprobab.Census opt-in

Nonprobab.MTurk

Prob. In-person

Description Emailaddressessigneduptobeinresearchoncensus.gov

Crowd-sourcing site;previouslyusedinonlinecognitivetesting

EmailaddressesmatchedtomasterlistofU.S.addresses

Think-aloud,concurrentlyprobedinterviewsonasubsetofthemessages

#Responses 303(8% RR) 200 330(3%RR) 30

Incentive - $0.50 - $40Datacollection

Twoweeks Twohours Twoweeks Overamonth

6

Analysis§ Comprehension?

§ Qualitativeassessmentoffeedback§ Codingofresponses

§ Diversityofrespondents?§ Demographicsofrespondents

7

Coding comprehension§ Understood§ Misunderstood

§ Misinterpretsthemeaningofthemessagebyparaphrasingincorrectly

§ Explicitlystatesthattheydonotunderstandthemessage

§ Requestsclarification§ Comprehensionofthemessage=

#𝑢𝑛𝑑. 𝑐𝑜𝑑𝑒𝑠#𝑢𝑛𝑑. 𝑐𝑜𝑑𝑒𝑠 + #𝑚𝑖𝑠𝑢𝑛𝑑. 𝑐𝑜𝑑𝑒𝑠

8

Findings: Comprehension

9

Online v. in-person generally§ Online

§ Morenegativity§ Moreoff-topicresponsesthatwecouldnotcodeforcomprehension

§ In-person§ Morerequestsforclarification

10

Comprehension§ Formostmessages,generalconsensusacrosssamplesourcesincomprehension§ Leadstosimilarconclusionaboutclarityofmessage

§ Sometimesunpredictablespikesinnegativityandoff-topicresponses

11

“Very few authorized individuals actually see your name or other personal information that could identify you. Most of the time, personal information that could identify you is removed from the file that contains your census or survey answers.”

12

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

100%

Und. Misund.

Census(nonprob.)

MTurk(nonprob.)

Prob.

Inperson

Mandatory response§ Short:Youarerequiredbylawtorespondtothecensus(Title13U.S.CodeSections141and193).

§ Wordy:CollectionoftheinformationismandatoryandiscollectedunderTitle13U.S.CodeSections141and193.

13

Mandatory response (cont.)

14

Short Wordy

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

100%

Und. Misund.

Census(nonprob.)

MTurk(nonprob.)

Prob.

Inperson

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

100%

Und. Misund.

Census(nonprob.)

MTurk(nonprob.)

Prob.

Inperson

Comprehension results (cont.)

§ Therewereafewinstanceswhereoneortwosourcesdifferfromtheotherincomprehension§ Noclearpatternacrosssamplesources§ Especiallyofconcernwhenin-persondiffersfromonline

15

“By law, we only allow access to data to conduct research that would help carry out the Census Bureau’s mission and benefit the public good.”

16

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

100%

Und. Misund.

Census(nonprob.)

MTurk(nonprob.)

Prob.

Inperson

Findings: Demographics of respondents

17

Age

18

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0%

100.0%

18-24 25-34 35-44 45-54 55-64 65-74 75+ Samplesource

Census(non-prob.),n=228

Mturk(nonprob.),n=183

Prob.,n=224

Race

19

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0%

100.0%

White Black AIAN Asian NHOPI 1+race Samplesource

Census(non-prob.),n=221

Mturk(nonprob.),n=170

Prob.,n=223

Education

20

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0%

100.0%

Samplesource

Census(non-prob.),n=226

Mturk(nonprob.),n=183

Prob.,n=223

Sex

21

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0%

100.0%

Male FemaleSamplesource

Census(non-prob.),n=228

Mturk(nonprob.),n=183

Prob.,n=224

Demographics of respondents: in-person

§ Obviously,abilitytobemoreselective§ Weneededtotestwith:non-white,lesseducated,men§ 22/30black§ 24/30somecollegeorless§ Stillendedupwith20/30women!

22

Conclusion§ Feedbackoncomprehension

§ Formostmessages,nomajordifferencesinconclusiononcomprehensionbyonlinesamplesource

§ Generallymorenegativityandoff-topicresponsesonline,fewerrequestsforclarification

23

Conclusion (cont.)§ Demographicsofrespondents

§ Useofprobabilitysamplenotaclearimprovementformostdemographics

§ Ifyou’reonlyusingoneonlinesample,demodifferencesmaybeusedtoguidechoice,dependingonwhatyouaretesting

24

Future Research§ Lengthofresponsesandotherqualitymeasures

§ Demographicdifferencesinresponses?§ Diggingintodiscrepanciesandusingothertypesofprobes

§ Inthemeantime:usein-person,too!

25

Thank you!AComparisonofCognitiveTestingMethods

andSources:In-PersonversusOnlineNonprobabilityandProbabilityMethods

JessicaL.Holzberg,GersonMorales,Aleia ClarkFobia,andJenniferHunterChilds

U.S.CensusBureau

Jessica.Holzberg@census.gov

26