366_8. survey design things to consider layout time to complete types of questions question...

366_8

Survey Design

• Things to consider– Layout– Time to complete– Types of questions– Question wording– Sensitive items

– THE RESPONDENT

Survey Design

• THE RESPONDENT– Does she know what you are asking?– Does she understand the question?– Can she recall the needed information?

– Does she have a reason to hide something?

Survey Design

• THE RESPONDENT– May not be paying close attention

• Make survey easy to read, follow

• Use similar response categories for blocks of questions where possible

• Sensitive items at the end

Survey Design

• THE RESPONDENT– Needs to know why the survey is being done• Why bother to do it?

– Needs to know the responses are anonymous.• Phone vs..• Mail vs..• Internet • Handed to them (face-to face)

Types of Questions

• Open ended– What is the most important issue facing the

country today?

– Can you tell me what it is that makes you like the Democratic Party?

Types of Questions

• Open ended– Valuable to measure information

– Might be biased toward people with more information

Types of Questions

• Open ended– Very labor intensive to enter data

– Must create categories for responses, code into spreadsheet

Types of Questions

• Closed ended– Defined response categories

– Overall, how satisfied are you with how democracy is working in the United States• Very satisfied• Satisfied• Not very satisfied• Not satisfied

Types of Questions

• Closed ended– Do you want a neutral

category?• Agree strongly• Agree• Neither agree nor

disagree• Disagree• Disagree strongly

• Closed ended– Neutral category might

be easy response default• Too little• About right• Too much

• More• Same• Less

Types of Questions

• Closed ended– Are you creating an ordinal (or even interval)

measure?• 1) a very great deal 6) Very little• 2) a great deal 7) None• 3) quite a bit• 4) a fair amount• 5) a little

Types of Questions

• Closed ended– Thermometer scales• On a scale from 0 to 100, with 0 being cool, 50 being

neutral, and 100 being warm, tell me how you feel about [whatever]. Just write a number between 0 and 100 in the space below– [whatever1] _________– [whatever2]__________– [whatever3]__________

Types of Questions

• Closed ended– Forced choice?

• Should response categories include “don’t know”

• Phone vs.. print surveys– If phone, we might not offer DK, but code it

– If print, if no DK option, respondent might skip

Closed Ended Question Blocks

Question Blocks

Layout:

Question Wording

• Be specific– “are you liberal or conservative” vs.:

– When it comes to politics, do you consider yourself liberal or conservative?

Question Wording

• Avoid biasing responses– Leading questions• “Do you favor or oppose dirty coal?”

– Loaded words• “welfare” vs.. Income support for the less affluent

– Double – barreled questions• Should we raise taxes and cut spending to lower the

deficit?

Question Wording

• Avoid biasing responses

– Is Clinton an old-style, tax & spend liberal, or a new style Democrat who who will be careful with the nation’s money? [ABC] vs.:

– Is Clinton a liberal Democrat or a Conservative Democrat (or, see slide 14).

Question Wording

• Sensitive questions– If you must ask, ask indirectly

– Drinking, not voting, rioting

• Occasionally, people drink too much and become [intoxicated]. In the past year, how often have you become [intoxicated] while drinking any kind of alcoholic beverage?

Question Wording• Sensitive questions [Why didn’t you vote?]• In any election, some people are not able to vote because they are sick or busy or have some

other reason, and others do not want to vote. Which best describes your situation in the recent local election in [CITY]

• • Illness or disability (your own, or in your family)• Forgot• Registration problems• Not interested; it doesn't matter• Too busy• Disliked the candidates• Out of town• It was inconvenient

Don’t like the voting rules• Other (VOL)• Don't know (VOL)

More points about the layout

• Begin with brief intro– Class project, anonymity, etc.

• Follow with ‘easy’ substantive question– First we’d like to ask you about...

• Start questions casual, chatty, clear– In general, do you think_______– Thinking about _____ , – How about, _______, Do you think....

More points about the layout

• Most important substantive items next

• Qs that test your main hypotheses

• Demographic questions at end

• Check established surveys for coding– Age – Region of state– Education– Party– Ideology– Religion [ or / and ]– Religious attendance– Family income (last)

More points about the layout

• Whatever layout, include:

• Are you registered to vote?– If no, SKIP to Q X– If Yes, are you registered

to vote here in Whatcom County, or elsewhere?

Before you print

• Are you sure everything is in there?– You have what you need to test your main

hypotheses

– You have questions that measure the attitudes you are interested in

– You have the questions designed to explain variation in the attitudes you are interested in

More points about the layout

• Printing– Single sheet– Back to back

– Booklet

– Font size

Pre-test

• Once DRAFT is completed, give survey to a few friends– How long did it take?– Ask them about it after they take it

• Any items not make sense?

Monitor refusals

• Is there a pattern to who is not filling out survey?