cognitive pre-testing

Joint Seminar of the Gender Task Force and the Tool Pool:

Cogni:ve Pretes:ng of Cross-‐na:onally Comparable Survey Instruments in a Developing Country Context with applica*on to USAID Feed the Future’s Women’s Empowerment in Agriculture Index Presented at the Interna*onal Food Policy Research Ins*tute Washington, DC 9 May 2014 Kiersten B. Johnson, PhD Senior Study Director, Westat

• Understand the purpose of cogni*ve interviewing/pretes*ng • Become familiar with the cogni*ve processes that are assessed

• Recognize when it is necessary to implement

•  Learn what methodological decisions can be made to tailor the pretes*ng to the needs and characteris*cs of the survey

• Obtain a sense for how to train and implement

Objec:ves

• Overview of the methodology

• Walk-‐through of the steps to implement

• Demonstra*on of how the method was applied for the Women’s Empowerment in Agriculture ques*onnaire in Haï*

Approach

Overview of the Cogni:ve Interview

Method

We want to systema*cally iden*fy and analyze sources of response error in surveys, and use that informa*on to try to improve the quality and accuracy of our survey instruments.

Why cogni:ve pretes:ng?

Especially important with new or revised instruments/ ques*ons

Cri*cal when being used cross-‐

culturally and/or cross-‐linguis*cally

Why Cogni:ve Interviewing?

Four Stages of Cogni:on Required for an Accurate Response to a Survey Ques:on:

Cogni&ve Stages Cogni&ve Stage Defini&on Problems Causes

1. Comprehension Respondent interprets the ques*on

Respondent does not understand

Unknown terms, ambiguous concepts, long and overly complex

2. Retrieval Respondent searches memory for relevant informa*on

Respondent does not remember/does not know

Recall difficulty, ques*ons assume respondent has informa*on

3. Judgment Respondent evaluates and/or es*mates response

Respondent does not want to tell, can’t tell

Biased or sensi*ve, es*ma*on difficulty

4. Response Respondent provides informa*on in the format requested

Respondent can’t respond in the format requested

Incomplete response op*ons, mul*ple responses necessary

What is cogni:ve pretes:ng?

• Qualita*ve field research method embedded in survey interview

• Can be comprised of: •  Direc*ons to “think aloud” while formula*ng a response to a survey ques*on •  Addi*onal probes/ques*ons integrated into dra_ instrument •  Interviewer observa*ons of respondent verbal & nonverbal cues

•  Intended to: •  Help determine whether the ques*on is genera*ng the intended informa*on •  Iden*fy problems related to ques*onnaire design

Can respondents discuss the survey ques3ons with a reasonable degree of validity?

What is Cogni:ve Interviewing? how

people construct their

answers

how people interpret

the ques*ons

difficul*es people had

in answering

Different Approaches

•  Lab-‐based vs field-‐based •  Thinking aloud vs. probing (vs. a combina*on of both) •  Concurrent vs. retrospec*ve •  Standardized vs. “ac*ve” •  Scripted, semi-‐scripted, or en*rely improvised •  Number of respondents and itera*ons •  Rela*vely unskilled data collector or expert inves*gator •  Analysis: systema*c review of interview transcripts, or based on notes

Selec*on of methodological approaches is likely to vary according to: -‐  Type and amount of resources available (human, temporal, financial) -‐  Nature of the survey

Different Methodological Approaches

•  Iden*fica*on of “problems” with ques*ons that turn out not to be “real” during survey implementa*on •  difficult to measure; limited research suggests it’s not significant problem

•  Failure to iden*fy problems that actually exist in survey instrument design and/or administra*on •  almost certain to occur, but possible to reduce

• Cogni*ve interview findings may be inconsistent when interviews are implemented by independent researchers •  not necessarily problema*c

Validity and Reliability of Cogni:ve Pretes:ng: Poten:al Types of Error

Steps in Implemen:ng a Cogni:ve Pretest

Example: Steps in Implementa:on

• Develop the ques*onnaire •  Translate the ques*onnaire •  Iden*fy candidate ques*ons for assessment •  Select methods; develop appropriate probes • Determine number and selec*on of respondents •  Select and train interviewers • Prac*ce interviews, note-‐taking and audio recording

•  Implement the cogni*ve interviews •  Summarize field notes; possibly use so_ware to facilitate data analysis • Write report: background, data and methods, results, conclusions & recommenda*ons • Revise ques*ons • Reiterate

Example: Steps in Implementa:on METHOD-‐OLOGICAL CHOICES

EMBEDDED HERE!

Cogni&ve Stages

Cogni&ve Stage Defini&on Problems Causes

Compre-‐hension

Respondent interprets the ques*on

Respondent does not understand

Unknown terms, ambiguous concepts, long & overly complex

Retrieval Respondent searches memory for relevant informa*on

Respondent does not remember/ does not know

Recall difficulty, ques*ons assume respondent has informa*on

Judgment Respondent evaluates and/or es*mates response

Respondent does not want to tell, can’t tell

Biased or sensi*ve, es*ma*on difficulty

Response Respondent provides informa*on in format requested

Respondent can’t respond in the format requested

Incomplete response op*ons, mul*ple responses necessary

Probes

E.g., “Earlier I asked you who contributes most to decisions regarding a new purchase of different items. How would you put that ques*on in your own words? Is this ques*on easy or difficult? Why is it difficult?”

E.g., “Many people find it difficult to recall every ac*vity done in a day. How well do you remember specific ac*vi*es you were doing in the past 24 hours?” “How do you know what *me you woke up yesterday?”

E.g., “Do you think that any other women you know may be afraid to answer this ques*on? Why do you think they might be afraid?”

E.g., “Earlier I asked you how sa*sfied are you with your available *me for leisure ac*vi*es like visi*ng neighbors, watching TV, listening to the radio, seeing movies or doing sports. Was this ques*on easy or difficult? Why was it difficult?”

Interviewer Training Content

• Understanding what cogni*ve pretes*ng is, why it is necessary for survey ques*onnaire development

• Knowledge of the cogni*ve processes required of survey respondents •  Full understanding of the inten*on of the ques*ons being tested • Knowledge of the probes to be used in cogni*ve tes*ng interviews •  Facility with in-‐depth qualita*ve interview techniques and the no*on of “narra*ves as data”

• Prac*cal exercises in doing cogni*ve interviews, to include cri*cal group discussion



Things to be mindful of when using regular, seasoned survey interviewers to implement cogni*ve pretes*ng:

•  Revised priori*es: iden*fica*on of problems of understanding, not resolu*on of problems of understanding

•  Need to administer slowly and allow *me for thoughkul responses, rather than proceed expedi*ously

Interviewer Training Content (cont.)

Materials for Training •  Slide show on cogni*ve tes*ng and why it is necessary • Ques*onnaire to be tested • A detailed ques*on-‐by-‐ques*on interviewer’s guide, customized for the cogni*ve interview • A set of generic possible probes for use in the interviews; e.g.:

•  “Why did you answer in that way?” •  “Tell me a lille bit more” •  “Can you think of an example of what you are talking about?”

•  Signs of respondent cogni*ve difficul*es •  Long silences •  Contradic*ons •  Reluctance or other discomfort

Materials for Training

Number of Interviews & Respondent Selec:on • Number of interviews:

•  Implement for each country >> Within each country, implement for each language comprising 10% or more of the survey sample popula*on

>> For each language, implement 15 well-‐conducted interviews* (see next slide)

• Respondent selec*on: •  Select according to demographic profile of eligible respondents to the survey •  Lab vs field implementa*on

“The palern of responses across a limited sample of respondents provides insight into the common palerns of understanding and interpreta*ons of people who are typical of

the targeted survey respondents.” -‐-‐ UNESCAP 2010

Number of Interviews & Respondent Selec:on

“For each language, implement 15 well-‐conducted interviews*” • No empirically-‐grounded conclusions on op*mal sample sizes; typical recommenda*ons range from 10-‐15 interviews •  Important to have enough respondents so all relevant aspects of a ques*onnaire are tested

“The point is not to obtain sample sizes large enough to supply precision in sta*s*cal es*mates. Rather, we strive to interview a variety of

individuals who will be useful in informing our decisions.

Sta*s*cians strive to minimize (error) variance, whereas cogni*ve interviewers maximize (subject) variance.”

-‐-‐ Willis 2005

Interview team

• One interviewer to ask the respondent ques*ons • One interviewer to operate the digital recorder, and observe & take notes of the respondent’s verbal & non-‐verbal cues

Interview Team

Materials

•  Tailored consent form • Customized ques*onnaire (hard copy) • Digital recorder • Pens/pencils/clipboard

Interview Materials

Preparing the Respondent

•  Informed consent to record •  Explain that the interview is evalua*ve, and we will be asking for their input on different aspects of the ques*ons and responses

Preparing the Respondent

Data Analysis

• Analysis method will depend on type of data •  Fundamentally qualita*ve; coding of textual responses will likely be needed to be able to compare responses across respondents •  Triangulate:

•  Responses to closed-‐ended CI ques*ons •  Narra*ve responses to probes captured in text •  Interviewer observa*ons recorded in text •  Digital recording of interview

• Applica*on of logic to interpret findings & make revisions accordingly

Data Analysis

Applica:on of Cogni:ve Interview

Method:

Women’s Empowerment in

Agriculture Ques:onnaire in

Haï:

Cogni:ve Pretest of the Women’s Empowerment in Agriculture Instrument in Haï: •  Step 1: Decide if we need to implement cogni*ve tes*ng.

•  New topic being addressed •  New instrument developed for assessing women’s empowerment in ag •  Cross-‐na*onal comparability of ques*ons and responses is important

•  Step 2: Iden*fy selected ques*ons that respondents may find cogni*vely difficult.

Cogni:ve Pretest of the Women’s Empowerment in Agriculture Instrument in Haï:

Module H2 •  H2.01. Recall difficulty •  H2.01. Recall period (last 12 months) •  H2.01. Difficult terms: “in-‐kind or monetary work both agriculture and other wage work”

•  H2.02. & H2.03. Comprehension: “Input into decisions”

•  H2.02. & H2.03. Sensi*vity: Input into decisions

Iden:fy Key Ques:ons (1)

Module H3a – Access to Produc&ve Capital

•  H3.01a: Household items owned: concept “household”, difficult items (nonfarm business equipment, Farm equipment (non-‐mechanized), Farm equipment (mechanized)

•  H3.01b: Number of items owned: number recall difficulty

•  H3.02: Owner of items: judgment difficulty

•  H3.03: Decide to sell: judgment difficulty

•  H3.04: Decide to give away: judgment difficulty

•  H3.05: Decide to mortgage or rent out: judgment difficulty, difficult terms (mortgage, rent out)

•  H3.06: Decide to purchase: comprehension (“contributes most to decisions “), judgment difficulty.

Module H3b – Access to Produc&ve Capital •  H3.07: Anyone taken loans in cash/kind: Comprehension (“Borrow in-‐kind”), recall (last 12 months)

•  H3.08: Decision to borrow: Judgment

•  H3.09: Decision about what to do with loans: Judgment

•  H3.10a: Demand for extra credit: Comprehension (condi*onal sentence)

•  H3.10b: Reasons for lack of demand for extra credit: Judgment

•  H3.11a: Limita*ons to credit supply: Judgment

•  H3.11b: Reasons for limited credit supply: Judgment, sensi*vity

Module H4 •  H4.01: Infrastructure: Sensi*vity •  H4.02: Proper payment of wages for public works: Sensi*vity, comprehension

•  H4.03: Protest the misbehavior of authori*es or elected officials: Sensi*vity

•  H4.04: Group membership: Comprehension of the different groups involved

•  H4.05: Ac*ve membership: comprehension

Module H5a

•  H5.01: Decision-‐maker: Possible sensi*vity if privacy not maintained

•  H5.02: Personal decisions: Possible sensi*vity if privacy not maintained


Module H5b •  H5.03: Get in trouble: comprehension; Act differently: comprehension (item wording, “Gevng inputs for agricultural produc*on”), sensi*vity •  H5.04: Others think poorly: comprehension (item wording, “think poorly”), sensi*vity •  H5.05: Right thing to do: comprehension (item wording, “right thing to do”)


Module H6 •  24 hour recall problems (remembering every ac*vity, *me of the day) •  Variability of daily schedules •  Comprehension: “sa*sfied”

•  PLUS… dura*on of each module, debrief interview ques*ons

•  Lab-‐based vs field-‐based à field-‐based •  Thinking aloud vs. probing (vs. a combina*on of both) à probing • Rela*vely unskilled data collector or expert inves*gator à specially trained data collector • Concurrent vs. retrospec*ve à middle road: between modules •  Standardized vs. “ac*ve” à standardized •  Scripted, semi-‐scripted, or en*rely improvised à scripted • Number of respondents à 12 HHs; 12 female respondents & 8 male respondents • Number of itera*ons à just one was possible

Selec:on of Methods

Table 1. Household and individual samples according to age of the female respondent

Age of female respondent Households sampled

Dual-‐headed household

Single female-‐headed household

18-‐35 4 2 36 and older 4 2

Individuals Sampled

Dual-‐headed household

Single female-‐headed household

18-‐35 8 2 36 and older 8 2

H2.01. Did you par:cipate in [ACTIVITY] in the past 12 months (that is during the last [one/two] cropping seasons)?

•  One fi_h of the 20 respondents reported that this ques*on was difficult. •  In some cases, there were comprehension difficul*es; respondents generated the following feedback on this ques*on:

“I haven’t understood the ques*on, that’s why it seems difficult to me.” (Man, age 49, dual-‐headed household)

“I haven’t understood the ques*on well.” (Woman, age 67, single-‐headed household)

What does the term “in-‐kind work” mean for you? •  Only three respondents approached the correct meaning •  Outright incorrect responses:

•  Don’t know •  Incorrect responses: “own account work” or “work to earn cash or something else” •  Imprecise responses that suggest a comprehension problem: “work for your life,” “work to obtain a benefit,” “work to survive,” etc.

•  Two respondents indicated the item may be sensi*ve as it draws alen*on to income-‐genera*on ac*vi*es, which respondents may consider to be private informa*on:

“The ques*on is difficult because I don’t have anything to make a living.” (Man, age 68, dual-‐headed household)

“(It’s difficult) because I don’t want to tell anyone.” (Woman, age 46, single-‐headed household)

Earlier I asked you if you do what you do regarding different ac*vi*es so that others don’t think poorly of you. How would you put this ques*on in your own words? Is this ques*on easy or difficult? Why was it difficult?

“Penser mal” has a range of meanings…

“To bring forward a cri*cism that does not please you” (Man, age 33, dual-‐headed household) “To think of hur*ng someone” (Woman, age 25, dual-‐headed household)

“It means destroying the life of a person” (Man, age 32, dual-‐headed household)

“When you think evil of a person, you can kill that person” (Woman, age 46, single-‐headed household) “A person who is there to kill someone” (Woman, age 67, single-‐headed household)

Time Use: Respondents were asked how they knew what :me of the day it was. •  50% check the &me using a cell phone they carry with them “With my phone, I check it every *me I start an ac*vity” (Man, age 36, dual-‐headed household) Nearly half rely on others: “I always ask the neighbors what *me is it” (Woman, age 36, dual-‐headed household) “I get the *me from a friend” (Man, age 40, dual-‐headed household)

References:

•  Willis, Gordon B. 2005. Cogni3ve Interviewing: A Tool for Improving Ques3onnaire Design. Thousand Oaks, CA: Sage.

•  Bealy, Paul C. and Gordon B. Willis. 2007. “Research Synthesis: The Prac*ce of Cogni*ve Interviewing.” Public Opinion Quarterly 71(2):287-‐311.

•  UNESCAP. 2010. Guidelines for Cogni3ve and Pilot Tes3ng of Ques3ons for Use in Surveys. Sta*s*cs Division, Economic and Social Commission for Asia Pacific Region.

•  Johnson, Kiersten B. & Diego-‐Rosell, Pablo. 2014. Assessing the Cogni*ve Validity of the Women’s Empowerment in Agriculture Index Instrument in the Hai* Mul*-‐Sectoral Baseline Survey. Manuscript under prepara3on for submission to peer-‐reviewed publica3on.

•  Blair, J., Conrad, F., Ackermann, A. and Claxton, G. 2006. “The Effect of Sample Size on Cogni*ve Interview Findings.” Paper presented at the American Associa3on for Public Opinion Research Conference. Montreal, Canada.

•  Conrad, Frederick G. and Johnny Blair. 2009. “Sources of Error in Cogni*ve Interviews.” Public Opinion Quarterly 73(1):32-‐55.

Key References

Photo credits

• hlp://blogs.agu.org, 12/2012 (slide #1) • Chris Blalman (slide #4) • Dinesh Maneer Photography (slide #10) • Doug Satre (slide #22) • hlp://church*mes.co.uk (slide #34)

Photo Credits

cognitive pre-testing

Government & Nonprofit