1 protocol & preliminary results from a transnational cognitive interviewing study: ensuring...
TRANSCRIPT
1
Protocol & preliminary results from a Protocol & preliminary results from a transnational cognitive interviewing study: transnational cognitive interviewing study: Ensuring valid cross-cultural comparisonsEnsuring valid cross-cultural comparisons
INSTITUTO NACIONAL DE SALUD PÚBLICA MÉXICO
James F. Thrasher, Anne C. K. Quah, Ron Borland, Rahmat Awang,
Maizurah Omar, Buppha Sirirassamee, Marcelo Boado, Ashlee
Watts, Ana Dorantes Alonso
TTURC Planning Meeting,
Portland, OR, USAMarch 2, 2008
2
Purpose of the studyPurpose of the study
• Develop a protocol for assessing and reducing Develop a protocol for assessing and reducing systematic response error in multi-lingual and systematic response error in multi-lingual and multi-cultural surveysmulti-cultural surveys
• Focus on existing ITC survey questions:Focus on existing ITC survey questions:– Assess equivalence of comprehension and Assess equivalence of comprehension and
response error across 6 countries, 4 languages;response error across 6 countries, 4 languages;– Recommend alterations to questions to increase Recommend alterations to questions to increase
equivalence in later studies.equivalence in later studies.
• Increase data quality Increase data quality more confidence in more confidence in results from comparative data analysesresults from comparative data analyses
3
Questionnaire protocolQuestionnaire protocol
1. Ask participant survey question.
2. Passive recording of behaviors that indicate comprehension problems (i.e., behavioral coding*).
3. Ask follow-up questions that aim to assess response error across the steps involved in cognitive processing of survey questions (i.e., cognitive interviewing).
*Johnson, TP et al (2006). Cultural variability in the comprehension of health survey *Johnson, TP et al (2006). Cultural variability in the comprehension of health survey questions. questions. Annals of EpidemiologyAnnals of Epidemiology, , 16,16, 661-668. 661-668.
4
Study StepsStudy Steps
Identify candidate questions
Determine likely
biases
Develop probes to
assess biases
Develop translator
guide
Translate protocol
Train interviewers
Pilot protocol
Review pilot data
Adjust protocol
Implement protocol
Enter data
Translate responses to English
Review quantitative
data
Identify themes in qualitative
data
Determine evidence for systematic
bias
Suggest changes
5
Sample Characteristics*Sample Characteristics*
USA Australia Uruguay Mexico Malaysia Thailand
Male 65% 50% 50% 65% 100% 80%
Age 36 36 40 38 31 39
<HS 5% 30% 30% 20% 10% 85%
Edu HS 65% 30% 40% 55% 45% 10%
Uni 30% 40% 30% 25% 30% 5%Daily smoker 85% 90% 80% 85% 90% 85%
Average cigs/day 23 15 14 15 13 13
*n=20 for each country
6
Nicotine in cigarettes is the chemical Nicotine in cigarettes is the chemical that causes most of the cancer: Falsethat causes most of the cancer: False
60%65%
20% 20%25%
5%0%
20%
40%
60%
80%
100%
US Aust Urgy Mex Mal Thai
7
Behavioral codes indicate some Behavioral codes indicate some systematic biassystematic bias
Did the participant… US AUST URGY MEX MAL THAI p…need you to repeat the question?
1 1 0 0 1 1 1.00
…have difficulty with response options?
5 3 0 0 3 0 0.008
…ask for clarification or qualify answer?
4 1 0 1 0 0 0.047
Total 10 5 0 1 4 1
8
Can you tell me what you know Can you tell me what you know about nicotine?about nicotine?
Theme USA AUST URGY MEX MAL THAI
Addictive 80% 60% 20% 20% 15% 35%
Harmful 15% 15% 30% 30% 35% 30%
Chemical-drug 0% 30% 0% 15% 20% 20%
Stimulant 5% 5% 0% 10% 0% 0%
Discoloration 10% 5% 10% 0% 5% 5%
Don’t Know 0% 15% 10% 15% 5% 20%Haven’t heard of nicotine 0% 0% 20% 20% 5% 25%
9
Conclusions & RecommendationConclusions & Recommendation• Behavioral codes systematic bias
– Awkward wording?
• Acquiescence bias?– “don’t know” or haven’t heard of nicotine
• Order effect after health outcome questions?
Recommendation:
Clarify question intent:– Assess knowledge? Why does this matter?
Could first ask if they have heard of nicotine– ITC-SEA W3: Cigarettes contain nicotine
10
Tobacco is addictive: Tobacco is addictive: ““agree” or “strongly agree”agree” or “strongly agree”
80%
100%90% 90% 85% 85%
0%
20%
40%
60%
80%
100%
US Aust Urgy Mex Mal Thai
11
Behavioral Codes indicate no Behavioral Codes indicate no problem with “Tobacco is addictive”problem with “Tobacco is addictive”
Did the participant… USA AUST URGY MEX MAL THAI p
…need you to repeat the question?
0 0 0 0 0 1 1.00
…have difficulty with response options?
1 1 0 0 1 1 0.87
…ask for clarification or qualify answer?
1 0 0 0 0 0 1.00
Total 2 1 0 0 1 2
12
Other things that are addictiveOther things that are addictiveCategory USA AUST URG MEX MAL THAI
Alcohol 70% 70% 70% 50% 15% 15%
Other drinks 5% 5% 25% 25% 10% 0%
Illegal drugs 80% 70% 70% 55% 60% 80%
Legal drugs 35% 35% 0% 10% 25% 0%
Inhalants 0% 0% 0% 5% 20% 15%
Food 40% 20% 25% 5% 0% 0%
Sex 15% 10% 0% 20% 5% 0%
Gambling 5% 15% 10% 5% 0% 0%
Other 35% 15% 15% 0% 20% 5%
DK 5% 0% 5% 25% 15% 0%
13
What does it mean to say that What does it mean to say that something is addictive?something is addictive?
Themes USA AUST URG MEX MAL THAI
Control-General 50% 50% 55% 55% 45% 30%
Physiological 40% 25% 5% 15% 0% 0%
Control-Psych 10% 10% 0% 0% 30% 0%
Control-Quit 0% 10% 5% 10% 10% 15%
Freq-Quantity 5% 50% 40% 15% 20% 30%
Danger 5% 0% 0% 0% 0% 35%
Pleasure 5% 0% 5% 0% 5% 5%
It's got a grip of you and you can't let go. It's a must have,
you've gotta have it. 39 year old Australian male
The body gets tolerance for it and the body won't function
normally with out it.27 year old US female
If he doesn’t get that thing, he cannot do other things…His mind is always imagining that thing…if he doesn’t get it, he cannot sit still.
26 year old Malaysian male
14
Conclusions & RecommendationsConclusions & Recommendations
• Behavioral coding no clear bias
• Connotations of and associations with addiction appear to differ across countries
Recommendation:
• Ask question(s) that focus specifically on:– Control over smoking (e.g., Smokers cannot
go long without smoking, even when they don’t want to smoke)
– Control over quitting (e.g., Completely stopping smoking is extremely difficult)
15
Lights are less harmful than regular Lights are less harmful than regular cigarettes: “Disagree”/”Strongly disagree”cigarettes: “Disagree”/”Strongly disagree”
55%65%
35%
20%
35% 35%
0%
20%
40%
60%
80%
100%
US Aust Urgy Mex Mal Thai
16
Behavioral codes indicate no Behavioral codes indicate no systematic bias systematic bias
Did the participant… USA AUST URGY MEX MAL THAI p
…need you to repeat the question?
2 1 0 1 2 3 .377
…have difficulty with response options?
1 1 0 1 2 3 .343
…ask for clarification or qualify answer?
0 1 0 0 2 1 .279
Total 3 1 0 2 6 7
17
Other differences beween light and Other differences beween light and regular cigarettes?regular cigarettes?
themes USA AUST URGY MEX MAL THAI
Nicotine & Tar 65% 60% 5% 5% 15% 10%
Other Chemicals 30% 30% 15% 5% 15% 15%Filter & Tobacco Qty 35% 5% 5% 0% 5% 10%
Taste & Flavor 20% 15% 50% 30% 50% 60%Other sensory effects 10% 5% 0% 5% 35% 50%
other (danger, packaging) 10% 15% 20% 0% 15% 0%
18
Conclusions & RecommendationsConclusions & Recommendations• Behavioral coding no systematic bias• Perceptions of light vs. regular appear to differ
across countries– Focus on product characteristics in western
countries– Focus on sensory effects of product use in non-
western countries
Recommendation• Be aware of the possibility of the salience of
these characteristics when developing new questions about product risks and effects.
19
Noticed warning labels “often” or “very Noticed warning labels “often” or “very often” in last monthoften” in last month
35%
80%
65%75%
55% 60%
0%
20%
40%
60%
80%
100%
US Aust Urgy Mex Mal Thai
20
Behavioral codes indicate some Behavioral codes indicate some evidence of systematic biasevidence of systematic bias
Did the participant… USA AUST URGY MEX MAL THAI p
…need you to repeat the question?
0 0 3 1 1 1 .377
…have difficulty with response options?
3 1 0 1 1 0 .377
…ask for clarification or qualify answer?
4 4 0 1 0 0 0.011
Total 3 1 0 2 6 7
21
Follow up probes indicate Follow up probes indicate incorrect reference period and incorrect reference period and issues with the term “notice” issues with the term “notice”
USA AUST URGY MEX MAL THAI P
Correct reference period identified (last month)
15% 25% 15% 10% 5% 5% .437
Can you notice without looking closely?
55% 80% 70% 65% 35% 75% .049
Can you notice if you know what label says?
75% 75% 55% 90% 60% 80% 0.146
22
Conclusions & RecommendationsConclusions & Recommendations
• Behavioral coding indicates systematic bias (*problems with US & Australia)
• Consistently incorrect identification of reference period for the question
• Some indications of systematic differences in meanings of “notice”
Recommendation: – Ask about last time when “noticed”– Review terms used for translation of “notice”
23
In last 6 months, noticed ads on…In last 6 months, noticed ads on…Measure USA AUST URG MEX MAL THAI ALL p*
TV 15% 5% 25% 35% 30% 5% 19% 0.054
Dif to remember 25% 10% 50% 15% 60% 40% 33% 0.003
Radio 15% 0% 25% 20% 10% 0% 12% 0.041
Dif to remember 30% 5% 45% 25% 65% 50% 37% <.001
Papers/Mags 75% 0% 40% 50% 30% 10% 34% <.001
Dif to remember 25% 10% 30% 15% 45% 40% 28% 0.095
Billboards 60% 0% 60% 95% 35% 10% 43% <.001
Dif to remember 20% 25% 45% 35% 50% 40% 36% 0.332
In stores 95% 45% 70% 90% 70% 30% 67% <.001
Dif to remember 15% 35% 30% 20% 25% 35% 27% 0.660
Correct period (last 6 months)
0% 20% 30% 10% 5% 15% 5% 0.067
24
Conclusions & RecommendationsConclusions & Recommendations• Behavioral coding no systematic bias• Across channels, participants in all countries
found it difficult to remember whether they had seen an ad– Some indication of systematic bias in difficulty of
remembering exposure for TV and radio.
• Consistently incorrect identification of reference period for noticing
Recommendation:– Ask about last time “saw”, with options for “never”
and for “more than 6 months ago”
25
General ConclusionsGeneral Conclusions
• Cognitive interviewing and behavioral coding techniques can be implemented across linguistic and cultural settings
• Overcoming challenges of assessing equivalence across languages demands collaboration from multilingual partners.
• Comprehension of some questions appeared generally comparable whereas other questions appear to generate response error and need to be adjusted.
26
General RecommendationsGeneral Recommendations
• Any changes to harmonize item comprehension should be verified through another round of cognitive interviews.
• Interviewers should be better trained to clarify ambiguous or repetitive responses.
27
General RecommendationsGeneral Recommendations• Prepare translation notes (i.e., definitions of the
concept being measured & question intent) for new questions.
• Choose source language questions that use simple, translatable language– “Notice” vs. “saw”
• To avoid augmenting “translation loss”, questions should originate from one source.– NOT: English Spanish Malay
• Equivalence of comprehension may demand flexibility in “source” language item
28
General RecommendationsGeneral Recommendations
Systematic documentation of the evolution, translation, and meanings of ITC survey questions, key terms, and phrasing across countries. Translator’s notes May include possible translation equivalents in
the target languages Provides record of original survey item
wording Allows future researchers to understand
intentions behind survey items and any changes to them
International Tobacco Control Policy Evaluation Projecthttp: / / www.itcproject.org
http: / / www.roswelltturc.org
ITC Project Research Support
Core support provided by the U.S. National Cancer Institute to the Roswell Park TTURC
(P50 CA111236)
Additional major funding provided by the Canadian Institutes of
Health Research
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
30
Study StepsStudy Steps
Identify candidate questions
Determine likely
biases
Develop probes to
assess biases
Develop translator
guide
Translate protocol
Train interviewers
Pilot protocol
Review pilot data
Adjust protocol
Implement protocol
Enter data
Translate responses to English
Review quantitative
data
Identify themes in qualitative
data
Determine evidence for systematic
bias
Suggest changes