stacey frank nina sabarre - d3 systems...• don’t use biasing language • ask about one concept...

Photo by OwenBlacker - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/12932804@N00 Created with Haiku Deck

Stacey Frank

Nina Sabarre

D3 Systems, Inc.

Photo by Leo Reynolds - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/49968232@N00 Created with Haiku Deck

GOALTo construct questions that allow for the collection of reliable and valid data that is representative of the population of interest.

• Don’t use biasing language

• Ask about one concept at a time

• Construct exhaustive and mutually exclusive response options

• Specify the response task

Photo by Maria Rosaria Sannino/images and words - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/93154838@N00Created with Haiku Deck

Photo by swimparallel - Creative Commons Attribution-ShareAlike License https://www.flickr.com/photos/31677782@N03 Created with Haiku Deck

Describe attitude, knowledge, behavior, and sensitive questions

Use strategies to write these types of questions with cultural considerations in mind

Photo by wazari - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/65231640@N00 Created with Haiku Deck

Measure someone’s views or feelings on a topic

Cannot be fact checked

Attitude Questions

Measure someone’s views or feelings on a topic

Cannot be fact checked

Attitude Questions

Behavioral: Did you vote in the last election? vs.

Attitudinal: What was your biggest motivation for voting in the last election?

Answers to attitude questions are fluid and can change often.

Beliefs and perceptions are complex ideas.

Fluid AttitudesQ2a. Would you say security in your local area is good, fair or poor? Is that ‘very good/poor'?

Q2b. Would you say security on local roads is good, fair or poor? Is that 'very good/poor'?

70%

Local Roads 59%58%

Local Area 45%

0%

100%

W1 W2 W3 W4

Photo by DVIDSHUB - Creative Commons Attribution License https://www.flickr.com/photos/28650594@N03 Created with Haiku Deck

Understand external conditions that influence attitudes

Photo by Wiertz Sébastien - Creative Commons Attribution License https://www.flickr.com/photos/36169570@N08 Created with Haiku Deck

Question order can have profound effect on responses to your attitude question.

Question Order Effect

Election Survey A and Election Survey B were used to project the number of likely voters in the Afghanistan presidential election (held April 2014).

Question Order Effect

Election Survey A and Election Survey B were used to project the number of likely voters in the Afghanistan presidential election (held April 2014).

The percentage of respondents who were likely to vote was off by 15%....

WHY?

Example of Question Order

Election Survey A: Likelihood of voting question followed questions about past voting behavior

Election Survey B: Likelihood of voting question followed questions about security

Photo by Wiertz Sébastien - Creative Commons Attribution License https://www.flickr.com/photos/36169570@N08 Created with Haiku Deck

Tips for Question Order

• Consider how other questions frame respondents' thinking

• Open-ended questions before closed-ended questions

• Start with warm-up questions

• Consider respondent fatigue

Photo by fusion-of-horizons - Creative Commons Attribution License https://www.flickr.com/photos/9019841@N08 Created with Haiku Deck

Understand & clearly define the attitude object.

Photo by fusion-of-horizons - Creative Commons Attribution License https://www.flickr.com/photos/9019841@N08 Created with Haiku Deck

Understand & clearly define the attitude object.

Revisit your research question.

Research question: How do Egyptians perceive legitimacy of their newly formed government?

Research question: How do Egyptians perceive legitimacy of their newly formed government?

Favorability: Q36a. Do you have a very favorable, somewhat favorable, somewhat unfavorable, or very unfavorable view of the government?

Confidence: Q36b. How do you feel about the government's ability to govern fairly – very confident, somewhat confident, not so confident or not confident at all?

Support: Q36c. How do you feel about most decisions made by the government – do you support them strongly, support them somewhat, oppose them somewhat or oppose them strongly?

Photo by Éole - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/73491156@N00 Created with Haiku Deck

Ask simple questions.

Avoid double barreled questions.

Double Barreled Question

Q-11. Do you think the Iraqi Government provides services and aid to all Iraqis equally or do they favor one group or actor over the others?

1. Yes, provides services and aid to all Iraqis equally

2. No, favors one group over others

______

98. Refused (vol.)

99. Don’t Know (vol.)

Revised Question Wording

Q-11a. Do you think the Iraqi Government provides services and aid to all Iraqis equally?

1. Yes

2. No

____

98. Refused (vol.)


Q-11b. Do you think the Iraqi Government favors one group or actor over the others?

1. Yes

2. No

____

98. Refused (vol.)


Photo by Adam Melancon - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/46062480@N00 Created with Haiku Deck

Break it down if it is too difficult to capture the whole picture in one question.

Drill Down Battery Questions

Drill Down Battery Questions

Q30. Do you believe women in your country have fewer rights

than men, the same or equal rights, or more rights than men

when it comes to the following…?

Fewer

Rights

Equal

Rights

More

Rights

Refused

(vol.)

Don’t know

(vol.)

a. Participating in politics 1 2 3 98 99

b. Access to education 1 2 3 98 99

c. Access to employment 1 2 3 98 99

d. Protection from violence 1 2 3 98 99

e. Access to healthcare 1 2 3 98 99

f. Decisions regarding

reproductive health

1 2 3 98 99

Photo by nevil zaveri ( away : ) - Creative Commons Attribution License https://www.flickr.com/photos/43109416@N00 Created with Haiku Deck

Measure effectiveness of an educational process

Measure knowledge in order to effectively design and implement information programs or advertising campaigns

Measure intelligence, which may be required to explain behavioral and attitudinal variables

Knowledge Questions

Photo by Koshyk - Creative Commons Attribution License https://www.flickr.com/photos/97235261@N00 Created with Haiku Deck

Ask knowledge questions to screen out respondents who lack information about a topic

Q-27. Biometric fingerprint scanners are scheduled to be used in the 2015 presidential election in Yemen. Are you aware of Yemen’s planned new biometric voting registry?

1. Yes

2. No

______

98. Refused (vol.)


Screening Questions

Q-28. (If ‘1’ in Q27) Do you support or oppose the creation of the new voting system, or do you neither support nor oppose it?

1. Support it

2. Neither support nor oppose it

3. Oppose it

______

98. Refused (vol.)


Screening Questions

16%

78%

Yes No

More than three fourths of respondents were not aware of Yemen’s biometric voting registry.

16%

78%

Yes No

More than three fourths of respondents were not aware of Yemen’s biometric voting registry.

93%

1%4% 2%

Support it Neither

support or

oppose it

Oppose it Don't know

(vol.)

Of those who are aware, the overwhelming majority support it.

Photo by USAID_IMAGES - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/46658241@N06 Created with Haiku Deck

Know the education level of your sample

Educational Achievement in Yemen

33%

One-third of Yemenis are illiterate.

Educational Achievement in Yemen

67%

Two-thirds have received primary schooling or less.

Do you have a very favorable, somewhat favorable, somewhat unfavorable, or very unfavorable opinion of the following leaders, or do you not know much about them?


• President Abd Rabuh Mansur Hadi

• Sanaa Mayor Abd al-Qadir Hilal

• Major General Ali Muhsin al-Ahmar

• Huthi movement leader Abd al-Malik al-Huthi

• Former President Ali Abdullah Salih

• Businessman Hamid al-Ahmar

• Former Republican Guard Ahmad Ali Salih

• Defense Minister Mohammed Naser Ahmed


• President Abd Rabuh Mansur Hadi 6% DK

• Sanaa Mayor Abd al-Qadir Hilal 59% DK

• Major General Ali Muhsin al-Ahmar 27% DK

• Huthi movement leader Abd al-Malik al-Huthi 18% DK

• Former President Ali Abdullah Salih 5% DK

• Businessman Hamid al-Ahmar 25% DK

• Former Republican Guard Ahmad Ali Salih 27% DK

• Defense Minister Mohammed Naser Ahmed 53% DK

Photo by Rex Pe - Creative Commons Attribution License https://www.flickr.com/photos/41087714@N00 Created with Haiku Deck

Use pictures and nonverbal procedures

Please look at this list of candidates with their photos and party

symbols. After I read out the list of 11 candidates for president,

please tell me if the election were today, for whom would you vote?

Write knowledge question with

research question in mind.

Purpose of Knowledge Question

Q-5. On what date will the 2014 elections be held? (OPEN-ENDED)

Purpose of Knowledge Question

Q-5. On what date will the 2014 elections be held? (OPEN-ENDED)

1. On April 5

2. Other date (within one month of April 5th,

between March and April)

3. Other date (any other date outside of

March and April)

4. Don't know exact date

_____

98. Refused (vol.)


Intent: to accurately record past, present or future behaviors

• Need to be carefully written to encourage truthful responses and avoid unintended distortions

Behavior Questions

Photo by Sukanto Debnath - Creative Commons Attribution License https://www.flickr.com/photos/7487149@N03 Created with Haiku Deck

Understand the behavioral norms in the country where you’re working

Photo by Agrofilms - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/8148165@N04 Created with Haiku Deck

Q15. In a typical week, please tell me if you engage in each of the following activities AT LEAST 2 DAYS WEEK for at least 10 minutes each day?

a. Exercise or play sportsb. Walk or ride a bicyclec. Do strenuous household chores like

cleaning or gardening


Recall

• Time Elapsed: memory diminishes over time

• Saliency: easier to remember details of salient events

Timeframe and SalienceNegative Interplay

Long time elapse, low salience:

How much did you, yourself spend in total on potato chips in January 2011?


Long time elapse, low salience:

How much did you, yourself spend in total on potato chips in January 2011?

Short time elapse, high salience:

Did you, yourself get married last week?


Photo by j-dub1980(THANK YOU FOR 100k+ Views) - Creative Commons Attribution-ShareAlike License https://www.flickr.com/photos/48048313@N02 Created with Haiku Deck

Timeframe and SaliencePositive Interplay



Long time elapse, high salience:

Where were you when the terrorist attacks in New York and Washington DC on September 11, 2001 happened?



Long time elapse, high salience:

Where were you when the terrorist attacks in New York and Washington DC on September 11, 2001 happened?

Short time elapse, low salience:

How much money, if any, did you spend at restaurants yesterday?


Be aware of how different perceptions of time will impact your survey questions

Photo by emdot - Creative Commons Attribution License https://www.flickr.com/photos/35237093637@N01 Created with Haiku Deck

What Year is it in Afghanistan?

2015: Gregorian calendar

1393: Persian calendar

1436: Islamic calendar

Photo by brongaeh - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/63848374@N04 Created with Haiku Deck

Photo by Sprengben [why not get a friend] - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/37010090@N04 Created with Haiku Deck

Q57. Were you personally or someone in your household injured prior to 2001?

1. Yes

2. No

Q58. Were you personally or someone in your household injured after 2001?

1. Yes

2. No

Landmark Events

Aided recall provides the respondent with memory cues or external sources of information to reduce recall errors.

Aided Recall

Aided recall provides the respondent with memory cues or external sources of information to reduce recall errors.

• Use when concerned with under-reporting, not over-reporting.

Aided Recall

Photo by thompsoe - Creative Commons Attribution License https://www.flickr.com/photos/8158623@N05 Created with Haiku Deck

Aided Recall: External Data

Available Records

• Tax survey

• Education survey

Diaries

• Media survey

• Health survey

Photo by waterdotorg - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/38891222@N03 Created with Haiku Deck

Q53. On average, how long does it take for one round trip to obtain water from this source?

This includes travel time to the source, wait time at the source, and return time. For bottled water, this refers to the amount of time it takes to travel to the shop, buy the water, and return.

Photo by Stuck in Customs - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/95572727@N00 Created with Haiku Deck

Sensitive Questions

Any survey question dealing with a sensitive or potentially threatening topic, where special measures should be taken to increase respondent ease and encourage truthful responses.

Photo by Sukanto Debnath - Creative Commons Attribution License https://www.flickr.com/photos/7487149@N03 Created with Haiku Deck

'Sensitive' subjects will differ from culture to culture.

Photo by Alex E. Proimos - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/34120957@N04 Created with Haiku Deck

Q12. Have you ever given birth to a boy or girl who was born alive but later died?

PROBE: Any baby who cried or showed signs of life but did not survive?

‘Sensitive’ Questions: Haiti

Factors that Impact Sensitivity

• Legal restrictions on free speech

• Religious or ethnic taboos

• Different concepts of public and private information

Photo by openDemocracy - Creative Commons Attribution-ShareAlike License https://www.flickr.com/photos/14071207@N00 Created with Haiku Deck

Understand the true ramifications of asking a sensitive question.

Sensitive Questions: Iraq

Q6. Who provides security in your

neighborhood? (Open-ended with

pre-codes, DO NOT READ LIST)

IMPORTANT: If respondent says “The

Islamic State of Iraq and Al-Sham, ISIS,

Islamic State and Da’esh” Skip to

Demographics and END interview

Photo by Jonathan Rashad - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/32623295@N02 Created with Haiku Deck

The Casual Approach

Q-36. Thinking about voting, if you happen to know,

can you tell me the location where people living in this

voting precinct will go to cast their ballot on election

day, or is this something you don’t know?

1. Yes, knows the location

2. No, does not know location

The Everybody Approach

Q6. As you know, many people in this area grow poppy along with other crops like wheat. Do you yourself happen to grow poppy?

1. Yes, grow poppy.

2. No, do not grow poppy.

Photo by DFID - UK Department for International Development - Creative Commons Attribution-ShareAlike License https://www.flickr.com/photos/14214150@N02Created with Haiku Deck

The Some/Other Approach

Q40. Some people think that sending

girls to school is doing what the

Qur’an and Hadith teach. Other

people think that sending girls to

school goes against the teachings of

the Qur’an and Hadith. Which is

closer to your view?

1. Sending girls to school is doing

what the Quran and Hadith

teach

2. Sending girls to formal school

goes against the teachings of

the Qur’an and Hadith.

Photo by Marco Gomes - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/39802787@N00 Created with Haiku Deck

Deliberately load the question:

• “Everybody does it…”

• Assume the behavior, ask about frequency

Ask “have you ever” before asking “do you currently”

Tips: Socially Undesirable Questions

• Use the casual approach “Do you happen to…”

• Ask “do you currently” rather than “do you usually”

Tips: Socially Desirable Questions

• Use the casual approach “Do you happen to…”

• Ask “do you currently” rather than “do you usually”

In all cases... avoid reliability checks within the same questionnaire!

Tips: Socially Desirable Questions

Comprehension of the question

Retrieval of information

Judgment and estimation

Reporting an answer

Comprehension

• What type of candy is being asked about?

• What is the time frame?

• Does one package equal 1 bar or 4 bars?

Comprehension




Retrieval

• Do I regularly eat Kit Kat bars?

• Has there recently been a holiday or party where I ate a lot of candy?

Comprehension




Retrieval



Judgment

• If I usually eat 5 Kit Kat bars a week, I’ve probably eaten about 20 in the past 30 days.

Comprehension




Retrieval



Judgment

• If I usually eat 5 Kit Kat bars a week, I’ve probably eaten about 20 in the past 30 days.

Reporting

• Does the question provide response options?

• Do I want to admit that I ate 20 Kit Kat bars in the past 30 days?

Comprehension Retrieval Judgment Reporting

Conclusions

• Check your assumptions

• Know the characteristics of your respondent population

• Know the strengths and weaknesses of the different question types

• Test your questions!

ReferencesAsking Questions: The Definitive Guide to Questionnaire Design - For Market Research, Political Polls, and Social and Health Questionnaires. Bradburn, Norman N., Seymour Sudman, and Brian Wansick. John Wiley and Sons Inc., 2004.

Cognitive Interviewing: A Tool for Improving Questionnaire Design. Willis, Gordon B. Sage Publications Inc., 2005.

Cross-Cultural Survey Methods. Janet A. Harkness, et. al. John Wiley and Sons Inc., 2003.

Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method, Third Edition. Don A. Dillman, et. al. John Wiley and Sons Inc., 2008.

Survey Methodology, Second Edition. Robert M. Groves, et. al. John Wiley and Sons Inc., 2009.

"Telescoping of Landmark Events: Implications for Survey Research." Gaskell, George D., Daniel B. Wright, and Colm A. O'Muircheartaigh. Public Opinion Quarterly Volume 64: 77-89.

For more

information,

please

contact:

8300 Greensboro Dr.

Suite 450

McLean, VA 22102

(703) 388-2450

www.d3systems.com

D3 Systems, Inc.

Stacey [email protected]

Nina [email protected]

stacey frank nina sabarre - d3 systems...• don’t use biasing language • ask about one concept...

Documents