Developing Survey ItemsDeveloping Survey Items

Cari BritoPsychology Department

October, 2003

Information NeedsInformation Needs

• Every item should be required for making decisions, choices, selections, or judgments. Obtain allall essential information Obtain onlyonly what is directly applicable Avoid items that only satisfy curiosity

• Classify information items into 3 priority categories High: essential or main reason for survey Medium: substantial value for decisions or actions Low: provide supporting info or background

• List in order of priority or importance can help reduce length

Basic Question AttributesBasic Question Attributes

• FocusFocus – every Q should focus on a single, clear, specific issue or topic. Poor: When do you usually go to class? Better: What time do you ordinarily leave for your first class?

• BrevityBrevity – the longer the Qs, the more difficult the response task. Short Qs are less subject to error Poor: Can you tell me how many siblings you have, whether they are

girls or boys, and how old they are? Better: What is the age and sex of each of your siblings?

• ClarityClarity – meaning of Qs must be clear to all respondents Poor: About how much of the storage space in your dorm room do you

and your roommate use? Better: What proportion of the storage space in your dorm room is used

for your things and what is used for your roommate’s?

Expressing the QuestionExpressing the Question

• Use core vocabularyvocabulary; words and phrases used in casual speech. 1255 surveyed by NY’s American Museum of Natural History:

• 77% interested in plants and trees, but only 39% interested in botany • 48% interested in fossils, but only 39% interested in paleontology• 42% interested in rocks and minerals, but 53% interested in geology

• GrammarGrammar: Use simple sentences where possible; use complex sentences only when required; change long, dependent clauses to words or short phrases where possible Poor: How do you work it out when you want one thing and your

roommate wants another and you both feel very strongly about it? Better: How do you settle disagreements with your roommate when you

both have strong feelings about it?

Avoid Negatively Phrased QuestionsAvoid Negatively Phrased Questions

• 1992 survey: 20% of Americans doubted that the Holocaust had occurred, another 12% were not sure. “As you know, the term Holocaust usually refers to the killing of millions

of Jews in Nazi death camps during World War II. Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?” (Roper Starch Worldwide)

“Do you doubt that the Holocaust happened or not?” (Gallup)• 9% doubted the truth of the Holocaust, another 4% were unsure.

• Poor: Do you believe the university should not have the right to search your dorm room?

• Better: Should the university have the right to search your dorm room?

Use Balanced ItemsUse Balanced Items

• Don’t write items favoring one position or another Poor: Do you support the use of animals in undergraduate laboratory

courses? Better: Do you support or oppose the use of animals in undergraduate

laboratory courses?

Poor: Do you think Textbook Rental provides an excellent, low-cost service to students?

Better: What do you think about Textbook Rental?

Avoid Questions that are Too DemandingAvoid Questions that are Too Demanding

• They may confuse respondents It’s probably too much to ask someone to rank-order 50 universities.

• There are memory limits Qs relying on memory should have clear time referent

• Poor: How many times last year did you study at the library?• Better: How many times during the SP03 semester did you study at the library?

Don’t ask the impossible of someone’s memory• When asking about past behaviors, don’t make the interval too long chances of

memory errors increase.• Poor: How many drinks did you have in the past month?• Better: How many drinks did you have in the past week?• Can aid memory problems by providing a list

o Drank with friends, drank at a bar, bought beer, etc.

Question Format: Open or ClosedQuestion Format: Open or Closed

• Unstructured (“open-ended”) – only the Q is expressed; Structured (“closed”) – alternative answers are listed

• Structured formats are preferred when feasible Dimension of answers is clear

• Respondents know more precisely what is being asked and what info is sought – they can perform more reliably

Data comparability• Open-ended responses need to be categorized when editing & postcoding data• Judgments have to be made about meaning or intent of respondents.

Recording Accuracy• it can be difficult to write and later read open-ended answers

Inappropriate Reasons• It takes time and effort to write effective structured Qs.• Structured items limit the “richness and variety” of people’s answers. That is true.

Using Response ScalesUsing Response Scales• Implicit scale

Exists in the minds of respondents Common understanding between the questioner and the respondent, concerning the

scale to be used.• What is your age? Generally, respond in number of years• Weight, height, eye color, distance between cities, etc.

• Explicit scale The scale can be specified in the question

• Months, pounds, dollars, degrees of temp, number of certain things Scale not clear: How much do you study each week?

• A lot; every day, whenever I need to, etc. Explicit scale: How many hours do you study each week?

• Depicted scale Can’t be included in question, depict as response options: verbally, numerically, or

graphically.

Common Response Categories/ScalesCommon Response Categories/Scales

• For any kind of rating: Excellent / Good / Fair / Poor • Satisfaction with amounts: Too many / Not enough / About right • For measuring amounts: Very much / Quite a bit / Some / Very little• Frequency of activities: Always / Often / Sometimes / Seldom / Never• How important, interested, or satisfied: Very / Fairly / Not at all • For opinions: Favor / Oppose

LikertLikert ScalesScales: State issue or opinion, obtain respondents level of agreement or disagreement with those statements. Strongly agree Agree Neutral Disagree Strongly disagree

• Basic survey topic categories are: attitudes, images, decisions, needs, behavior, lifestyle, affiliations, and demographics

Scale Points to IncludeScale Points to Include

• No clear advantage to a 5- or 7- (or more) point scale. 5-point scale provides sufficient discrimination Might become de facto 3-point scale tendency to avoid

selecting the ends 7-point scale may yield 5 points, but extra level of

discrimination may increase completion time• Avoid mixing formats: use all 5- or all 7-point scales• A 5-point scale can be made into a 7-point scale

For Likert scales, add: Very strongly agree & Very strongly disagree

Likert Scales – On being “Neutral” or Likert Scales – On being “Neutral” or “Undecided” and Not Knowing“Undecided” and Not Knowing• Reasons for not including Neutral or Undecided

May encourage some to take lazy way out not have to ponder an item

May appeal to those who don’t wish to reveal their position on a controversial topic.

• Reason for including May have respondents who are truly neutral frustrating

• Omit If you believe all potential respondents probably have developed positive

or negative attitude toward topic; have an opinion Instruct to select choice in each item that is CLOSEST to their position

• Use “Don’t know” carefully when measuring opinions or attitudes Only if respondents could have very little or no knowledge about topic Can have them respond: “Using your best guess …”

Likert Scales – Balance Favorable & Likert Scales – Balance Favorable & Unfavorable StatementsUnfavorable Statements• If all worded same way, items may not be read carefully

The instructor held my attention during class lectures. The instructor wrote exams that fairly tested the material covered. The instructor seemed to be well prepared for class. The instructor was available for extra help outside of class.

• Some students have a response bias called “response acquiescence” tendency to agree with virtually all statements.

• Have to read each item carefully & make item-by-item decision.• Generally try for ½ and ½

The instructor was seldom able to hold my attention during class lectures. The instructor wrote exams that fairly tested the material covered. The instructor often appeared to be unprepared for class. The instructor was available for extra help outside of class.

• Make sure to note beforehand which are fav & which are unfav so that they can be re-coded later on.

Be Aware of Response BiasesBe Aware of Response Biases

• Social desirability Give socially acceptable responses

• Yea- and nay-saying Global tendency toward pos or neg answers

• Prestige Respondent wants to enhance his or her image

• Threat The Q instills anxiety or fear

• Hostility The response task brings out anger and resentment

• Auspices If know who is sponsoring the survey; their feelings toward the sponsor biases their answers.

• Mental Set Develop set of perceptions and assumptions that act as a frame of reference for responding

• Order Initiation (learn how to respond), routine (develop response strategy), fatigue

• Extremity Scale is dichotomized: pick extreme options; Fence-riding is the opposite – pick only the middle

Collecting Demographic InformationCollecting Demographic Information

• Only include demographic categories important to the objectives of your survey.

• The more demographic items you include, the longer the questionnaire, and the greater the risk respondents will become bored.

• Can also irritate respondents Some requests (e.g., income) can be seen as invasions of privacy.

• Some guidelines When asking about age, ask for DOB When asking about annual income (or other sensitive info), provide a range Don’t let the alternatives overlap If comparing to other surveys, ask for identical demographics Use standard categories: Borrow items from the Census Bureau (www.census.gov)

• Novice: “married, single, divorced, and widowed”• Pro: “single” is ambiguous “married, separated, widowed, divorced, never married”

Miscellaneous TipsMiscellaneous Tips

• Organize questionnaire carefully Cluster items on same topic together – if possible, no more than 10

items in a list Also, cluster by response scale

• Write brief separate instructions for each section.• Provide clear directions – don’t assume they know what to do

tell them• Be careful about the order of items

Start with easy and interesting Demographics last

• Work hard on the Introduction• Work hard on the survey’s visual appeal• Provide a reward for responding

Field Pretest: “Piloting”Field Pretest: “Piloting”

• Rewrite Qs several times• Seek informal reviews of the items for clarity• Do a test run with a small group from the population• Perform a crude item analysis on categorical items

In the past month, how many times have you smoked marijuana?______ 0-10 _____ 11-20 ______ 21-30

If every person chooses 0-10, you know that the categories are too inclusive – include more alternatives…

Don’t Lose Track of ObjectivesDon’t Lose Track of Objectives

• They should guide the construction of the questionnaire• Will also direct the analyses and comparisons performed

after the data are collected.

ReferencesReferences

1. Alreck, P. L., & Settle, R. B. (1985). The survey research handbook. Homewood, IL: Richard Irwin, Inc.

2. Fowler, F. J., Jr. (2001). Survey research methods (3rd ed.). Thousand Oaks, CA: Sage.

3. Patten, M. L. (1998). Questionnaire research: A practical guide. Los Angeles: Pyrczak.

4. Rossi, P. H., Wright, J. D., & Anderson, A. B. (1983). Handbook of survey research. San Diego: Academic Press.