1 RTI International is a trade name of Research Triangle Institute

Considerations When Using Family Surveys with Spanish-Speaking Families

Georgina McAvincheyMurrey Olmsted, Ph.D.

Don Bailey, Ph.D.

Measuring Child and Family Outcomes National TA Meeting

Baltimore, MD

August 27, 2007

2

Goals for this Session

Provide background of issues and early findings

Describe cognitive testing as a methodology for understanding how respondents understand and respond to survey directions and items.

Present findings from a study using cognitive testing interviews with 40 Spanish speaking families in early intervention programs

Initiate a forum for discussing ways to maximize valid participation in state surveys by families whose primary language is not English

3

Background Information

Families participating in early intervention programs are ethnically diverse

0

10

20

30

40

50

60

70

White

Hispanic

AfricanAmericanAsian/PI

AmericanIndian/Alaskan

Percent ofParticipants

4

Background Information

The number and proportion of Hispanic/Latino families enrolled in early intervention is increasing

0

10000

20000

30000

40000

50000

60000

1998199920002001200220032004

Number of Hispanic/Latino

FamiliesEnrolled

5

Background Information

Hispanic/Latino families enrollment in early intervention is growing much faster than African American enrollment

0

10000

20000

30000

40000

50000

60000

1998 2004

Hispanic

AfricanAmerican

6

Background Information

The NEILS study found differences by race/ethnicity in initial experiences with early intervention (Bailey et al., 2004)

White Black Hispanic/Latino

A lot of effort required to find early intervention

8 13 13

Aware of IFSP 89 77 63

Have good feelings about professionals 65 64 55

Professionals respect my family’s culture 65 56 55

Disagree that Early Intervention professionals ignore opinion

58 45 41

7

Background Information

A more recent NEILS paper (Bailey et al., 2005) showed that race/ethnicity was a significant predictor of family outcomes at 36 months of age White families significantly higher than black

Black families significantly higher than other minority groups

A forthcoming NEILS paper (Bailey et al., in press) using structural equations modeling found that white families tended to report more positive perceptions of quality of family services and were more likely to report that EI had a positive effect on their family

8

Background Information

A recent pilot study of the Family Outcomes Survey showed substantial differences in outcomes reported by English-speaking versus Spanish-speaking families

9

Background Information

beginning to learn

programs and srvcs

2 3 4 5 6 very aware of program

and srvcs

know available services

0

10

20

30

40

Per

cent

Lang

Eng

Spa

just beginning 2

34

56

understand very well

understand child's development

0

10

20

30

40

50

Per

cen

t

Lang

Eng

Spa

10

Study Goals

Fundamental Question Do differences in responses represent true differences in outcome attainment

or might they reflect problems with translation, instrument format, different cultural interpretations of the meanings of items, or different expectations for what programs can and should do for families?

Research Sponsor This project was sponsored by the Early Childhood Intervention Division

(ECID) of the Texas Department of Assistive & Rehabilitative Services

Research Approach To answer this question, we conducted a set of cognitive testing interviews with

40 Spanish-speaking families in Illinois and Texas

11

What is Cognitive Interviewing?

Cognitive interviewing is a qualitative testing methodology used to explore common problems in the way information is displayed, questions are asked, and tasks are presented to respondents in questionnaires (Willis, Royston, Bercini, 1991, Willis, 1994, 2005).

Based on a cognitive psychology model of the mental processes involved in responding to questionnaires and surveys by Tourangeau (1984) and Eisenhower et al. (1991) Encoding in memory (have some knowledge/memory of the question) Comprehension (understand the question and relevant concepts) Retrieval (retrieve the information from memory) Judgment (assess the completeness/relevance of their memories) Communication (decide whether their answer fits the answer categories

provided and also decide whether they actually want to provide an answer or provide one that might be socially acceptable)

12

What Kinds of Questions Does it Address?

Using structured interview techniques, a methodologist examines the following kinds of questions: What do respondents think the question is asking? What do specific words and phrases in the questions mean to

respondents? Are words used that respondents do not understand? Are the memory and/or decision requirements of the

questionnaire reasonable? How do respondents choose their answers? How well do respondents understand instructions and visual

cues?

13

How Does it Work in Practice?

Identify issues in a questionnaire that may be problematic. This can often be done via a structured instrument review process.

Develop a cognitive testing guide to address each issue

Recruit participants that are representative of the target audience 8-12 participants are typical for an average questionnaire If federally funded, OMB restricts to 9 interviews per questionnaire

Conduct testing interviews in a one-to-one setting Select a quiet location for the interviews where you will not be disturbed Record all interviews and take good hand-written notes Provide an incentive as a “Thank you” gift to participants

Review interviews and summarize the findings and the recommendations for possible changes to the questionnaire

Discuss the findings and implications with the project team and determine which changes are appropriate

14

Probing Techniques Used in Interviewing

Primary approaches for probing used in cognitive interview are concurrent and retrospective probing. Concurrent probing: This involves asking the participant questions as they fill out

the questionnaire to explore their understanding of concepts, questions, and other related factors.

Retrospective probing: Participants are first asked to complete a copy of the questionnaire and then are asked questions about their understanding of terms, questions, and concepts as the interviewer reviews the questionnaire with them.

Some examples of types of probes used include: Comprehension probe: What do you think is meant in the question by “your

child’s development? Paraphrase probe: In your own words, what is this question asking? Confidence judgment probe: Do you feel that you know enough about your

“child’s development? Explain. Elaborative/Expansive probe: Can you explain? Recall probe: Think of the person or persons that provide services to your child.

Do you feel that you can adequately communicate with them? Explain.

15

What Are the Results of Cognitive Interviewing?

Develop a cognitive testing summary report providing an overview of the background, methods, findings, and recommendations Identification of problematic questions

Recommendations for possible solutions

Marked-up questionnaire showing possible changes

Vet possible changes to ensure that solutions address findings and do not cause further problems

Ensure that the meaning of the questions are not lost in recommended changes

Evaluate the language and cultural implications of any changes

Update and test the new version of the questionnaire

16

Background of the Spanish Testing

ECID requested that RTI conduct cognitive testing of the FOS with Spanish speaking families in Texas and Illinois

Earlier pilot data of the original Spanish version of the FOS showed that families had:

Lower average scores on many items

Lower overall response rates

Significant item-level missing data

Testing sought to determine whether predominantly Spanish-speaking parents are able to understand and respond to the survey easily

17

Spanish Cognitive Interview Protocol

Protocol was based on the materials that were used to test the English version of the FOS during the summer of 2006

Also included new questions about access to services and experience with early intervention

Previous research support the idea that cognitive testing interviews can provide valuable insights when testing cross-cultural survey instruments

It can reveal ways in which cultural differences may affect how respondents perform cognitive tasks involved in survey responding

18

Methodology and Participants

Cognitive testing were completed with:

40 family members

6 sites across Texas and Illinois

Qualified participants were Hispanic/Latino families in Texas and Illinois

Interviews conducted in June and July 2007

90 to 120 minute interviews

$70 cash incentive

19

Recruiting Participants

Recruiting methods:

Posting flyers at early intervention centers

Utilized contacts established by coordinators, family contact personnel and parent liaisons to reach out to families

Focused recruiting effort on obtaining:

Parents or family members who had some caretaking role of a child who was currently enrolled in an early intervention program

Spanish speakers who are monolingual or bilingual with Spanish language dominance

20

Methodology: Inclusion Criteria

Inclusion criteria:

Linguistic skills (monolingual and bilingual speakers with Spanish dominance)

Education

Region of origin

Demographic diversity

21

Language Dominance

Language dominance was assessed for each participant: Evaluated the language that the participant spoke in the home and studied

in early school

Current usage of language

Language the participant feels most comfortable using for communication

Participants were classified as: Monolingual English speaker (speaks only English; not eligible)

Monolingual Spanish speaker (speaks only Spanish, eligible)

Bilingual, English dominant speaker (speaks both Spanish and English; dominant in English; not eligible)

Bilingual, Spanish dominant speaker (speaks both Spanish and English; dominant in Spanish; eligible)

22

Respondent Profile

Demographics - 40 family members in 6 sites across Texas and Illinois

Language dominance characteristics 6 monolingual Spanish speakers 34 bilingual speakers with Spanish dominance

3 men and 37 women

Median age 32 (Min: 23, Max: 53)

Exposure to early childhood service ranged from 1 to 4 years

Typically referred to early intervention by hospitals and/or pediatricians, except for a few children whose parents requested a referral

23

Respondent Profile: Spanish-speaking respondents

Countries represented by participants included:

Mexico Peru

El Salvador Dominican Republic

Guatemala Ecuador

Honduras Venezuela

The majority of respondents (75%) reported being of Mexican descent

24

Findings and Recommendations

Presentation of the finding and recommendations will be presented as follows:

General Findings Related to Early Intervention Services

Specific Instrument Design Issues

Example of Findings and Recommendations

Overall Recommendations

25

General Findings Related to Early Intervention Services

Participant’s Experience and Understanding of Early Intervention

Most participants praised the early intervention services

Tenía una venda y me la quitaron. Me hicieron ver las cosas y aprender. Fue una nueva experiencia.”

(I was blindfolded and they took it off. They made me see and learn things. It was a new experience.)

26

General Findings Related to Early Intervention Services

Desire to have early intervention services expanded past the 3 years of age

Parents’ initial expectations originated from first Early Intervention meeting(s)

Parents expectations have been met and exceeded in several cases Expected some type of improvement in their child’s special situation

Eager to learn how to better interact and help their children

27

Impact of Language on Early Intervention Services

Good communication with service providers

Service providers responded very well to the child’s needs

Some language barrier

“Si yo pudiera hablar inglés estaríamos mejor, nos entenderíamos mucho mejor y me entenderían más lo que yo les trato de decir”

(It would be better if I could speak English, we could understand each other much better, and they would understand what I try to tell them).

Fairly high overall satisfaction with having bilingual personnel available

28

Specific Instrument Design Issues

Participants generally liked the FOS questionnaire

A few consistent issues emerged from the cognitive testing which should be addressed including:

Instructions and 7-point Scale

Problems with Concepts and Terminology

Issues Affecting Comprehension of Survey Items and Instruction

29

Instructions for answer option 2, 4, and 6

If a statement almost describes your family, but not quite, circle the number just to the left or the right. For example if you feel that the statement 5 “We know a good amount about dinosaurs” almost describes your family, but not quite—circle the 4.

How much does your family know about dinosaurs?

1 2 3 4 5 6 7

We know a little about dinosaurs

We know some about dinosaurs

We know a good amount about dinosaurs

We know a great deal about dinosaurs

30

7-point Scale

Many participants appeared uncomfortable or struggled using the scale during the first few questions

The spaces between response options were described as confusing by some but not all participants.

“Estos huequitos de en medio confunden.”

(These little holes in the middle are confusing.)

1 2 3 4 5 6 7

We are just beginning to understand our child’s special needs

We understand some about our child’s special needs

We understand a good amount about our child’s special needs

We understand a great deal about our child’s special needs

31

Instructions and 4-point Scale

Recommendations for possible change to the FOS

Eliminate instructions for answer option 2, 4 and 6

Remove spaces between response options (see below)

1 2 3 4

We are just beginning to understand our child’s special needs

We understand some about our child’s special needs

We understand a good amount about our child’s special needs

We understand a great deal about our child’s special needs

32

Problems with Concepts and Terminology

Most of the concepts found in the questionnaire were well understood by parents

Some concepts and terms were confusing or needed clarification

Incorrect assumptions about participant’s knowledge of the terminology

Others were associated with vagueness

Some additional issues are presented in the question by question review of the report

33

Key concepts that caused difficulty

Child - Most participants understood this term as intended but there were a few parents with more than one child receiving early intervention (EI) services who were unsure which child to consider when completing the survey

Early Intervention Services (EI services) - The inclusion of this term in the survey assumed parental knowledge of early intervention services. Most participants reported lacking this knowledge prior to receiving these services for their children

Special needs - This term was found to be sensitive, and widely misunderstood by participants. Several of the participating family members associated this term with severe physical or mental impairments

34

Issues affecting Comprehension

Reading level – Many questions appeared to be over the participant’s current reading level and limited vocabulary.

“Si se me hace difícil a veces el español. El español es diferente. A veces hay palabras que uno no las usa y si no las usa, a veces no está uno seguro si uno sabe lo que quiere decir. Esta muy elevado la forma de preguntar. Esta muy formal. Lo pondría en palabras más sencillas…”

(Sometimes I have a hard time with Spanish. Spanish is different. There are some words that if you do not use them for a while, you are not sure what they mean. The way of asking is too “high”. It’s too formal. I would use simpler words…)

35

Issues affecting Comprehension

Unnecessary wording in some questions – Parents had trouble understanding some questions due to unnecessary or confusing text appearing in the questions

Las preguntas no van directo al punto. Dan primero una… dan como dos preguntas en una sola pregunta. Simplificaría haciendo las preguntas más directas”.

(Questions should go straight to the point. Sometimes they ask like two questions in one. I would simplify them by making them more direct).

36

Issues affecting Comprehension

Cultural issues – There were some cultural issues and assumptions that might have affected comprehension

Creo que es tabú, porque si dices que tu niño es autista, creen que es ‘mongolito’, y como si fuera malo. Como aceptar que tu hijo es un perdedor. No solo en la comunidad mexicana, sino en los latinos. No sabemos del tema.”

(I think its taboo, if you mention that your child is autistic, they think he is retarded, as if it was something bad. It is like admitting that your child is a loser. This not exclusive to Mexicans but the whole Latino community. (In our culture) we do not know about this.)

37

Example of Findings: Question 2

Example:

Question 2 (Original):

Algunos niños tienen necesidades médicas especiales, alguna discapacidad o retraso en el desarrollo. Estas a menudo se conocen como “necesidades especiales”. ¿Qué tan familiarizada está su familia con las necesidades especiales de su hijo?

(Some children have special health needs, a disability, or are delayed in their development. These are often referred to as “special needs.” How familiar is your family with your child's special needs?)

38

Example of Changes: Question 2

Example:

Question 2 (Revised):

¿Qué tan familiarizada está su familia con las necesidades especiales de su hijo? (Las necesidades especiales incluyen necesidades médicas, alguna discapacidad, problemas del habla, visión o audición, o demora en el desarrollo.)

[How familiar is your family with your child's special needs?) (Special needs include health needs, a disability such as a speech, vision or hearing problem, or delayed development).]

39

Overall Recommendations

Simplify the survey items by moving from a two-sentence format to a single statement

Reduce the number of scale points from 7 to 4 by eliminating the current ratings of 2, 4, and 6 for which there are no behavioral anchors

Revise each question to clarify the meanings of key terms/concepts and simplify the provided response options

Make selected changes in the Spanish translation using words that are more likely to be understood by the majority of Spanish speakers

40

Questions & Answers

Any questions?