SurveysJulie Demuth

(With input from Jeff Lazo, Mary Hayden, other resources … and hopefully you!)

Summer WAS*IS, July 14, 2007

WAS* ISWAS* ISweather & society * integrated studies

www.sip.ucar.edu/wasis/

CULTURE CHANGEWAS* ISWAS* ISweather & society * integrated studies

www.sip.ucar.edu/wasis/

CULTURE CHANGE

Outline• What is survey research?

• To use surveys or not: Some factors and information to consider

• Survey research components emphasis on design

Remember, this is just a brief introduction!

Remember, this is just a brief introduction!

What is survey research?• Surveys

– Self-administered questionnaires (written)

– Face-to-face interview, telephone interview, etc. (oral)

– Cross-sectional design• More than 1 case, single point in time, analyzed

to detect patterns of association

To use surveys or not?• Objectives

– What information do I need?– What will I do with the information?– How will I analyze the information?– How and to whom will I present the results?

Advantages to surveys• Gathering information

– Efficient for collecting large amount of information about a large population

– Flexible to collect wide range of information (e.g., attitudes, values, beliefs, past behaviors)

• Questions– Subjects gets same questions– Researcher can ask more complex questions– No response effect (willing to divulge more info w/o

face-to-face contact; less likely to try to impress interviewer)

• Administering– Relatively easy and inexpensive to administer– Can be computer-based

Disadvantages to surveys• Gathering information

– Questions must be general enough to be appropriate for all respondents, possibly missing important context

– Useless with non-literate or illiterate populations, only English-speaking populations, or hard-to-reach populations

• Questions– Subjects’ motivation, memory, and ability to respond– Not appropriate for studying complex social phenomena– Subjects may not answer honestly – No control over participant interpretation

• Administering– Low response rates – Respondents usually self-selected– Uncertainty about who actually filled out the

questionnaire

Survey research components

Design

Sampling

Implementation Analysis

Survey design• Parts of a survey

– Introduction– Filtering questions– Content questions– Socio-demographic questions– Debriefing questions

• Other considerations– Instructions– Formatting

The questions• Using focus groups

– A la Mary Hayden!

• Borrowing questions– Longitudinal studies

• Developing them from scratch!

Criteria for assessing questions

• Does the question require an answer?– If you read the area forecast discussion, does

it help you better understand the weather?

• To what extent does the respondent have an accurate, ready-made answer to the question?– What is your favorite episode of Friends?

• Can people accurately recall and report past behaviors?– What time did you leave work on Dec 20,

2006?

More criteria• Is the respondent willing to reveal the

requested information?– How many times have you driven through

flooded roadways?

• Will the respondent feel motivated to answer each question?– Scenario with 10 subparts

Possible scalar categories• Strongly agree strongly disagree

• Very favorable very unfavorable

• Extremely satisfied extremely dissatisfied

• None Extreme

• Scale of 1 to 5, where 1 means lowest possible quality and 5 means highest possible quality– Scales of 1 to 3, 1 to 7, 1 to 10

Question structures• Open-ended

• Close-ended with ordered response categories (like scalar categories)

• Close-ended with unordered categories

• Partially close-ended– With “Other (please specify)_________” option

Principles for writing questions

• Choose simple over specialized words

• Avoid vague quantifiers when more precise estimates can be obtained

• Avoid specificity that exceeds the respondent’s potential for having an accurate, ready-made answer

• Use equal numbers of positive and negative categories for scalar questions

Principles for writing questions

• Distinguish undecided from neutral by placement at the end of the scale

• Avoid bias from unequal comparisons

• State both sides of attitude scales

• Avoid check-all-that-apply question formats

Principles for writing questions

• Develop response categories that are mutually exclusive

• Avoid asking respondents to say yes in order to mean no

• Avoid double-barreled questions

• Avoid asking respondents to make unnecessary calculations

Pretesting• Review by knowledgeable colleagues

• Evaluate cognitive and motivational qualities– Verbal protocol analysis / think alouds– Retrospective interviews

• Small pilot study– 10% of final sample size

• Final check

How often do you get weather forecasts from the sources listed below?

Rarely or never

Once or more

a month

Once a week

Two or more

times a week

Once a day

Two or more

times a day

Local TV stations 1 2 3 4 5 6

Cable TV stations (e.g., CNN, The Weather Channel)

1 2 3 4 5 6

Newspapers 1 2 3 4 5 6

Telephone (dial-in) weather information source

1 2 3 4 5 6

Commercial or public radio 1 2 3 4 5 6

NOAA Weather Radio 1 2 3 4 5 6

National Weather Service (NWS) webpages

1 2 3 4 5 6

Other webpages 1 2 3 4 5 6

Cell phone, personal desk assistant (PDA), pager, or other electronic device

1 2 3 4 5 6

Friends, family, co-workers, etc.1 2 3 4 5 6

More examples

Never SeldomSome-times

Usually Always

I watch television or listen to the radio to get a weather forecast so that I can know what to expect

1 2 3 4 5

More examples• I have a plan for keeping myself and

my family safe in a tornado.– Yes– No

Survey sampling• Population, Units, Subjects and Samples

– Population: entire group of people about which information wanted.

– Units/subjects: Individual members of the population are called units

– Sample: part of population examined

• Error– Sampling: not surveying all elements of

population – Coverage: not allowing all members of the

survey population to have an equal or nonzero chance of being sampled

Survey implementation• Methods for survey implementation

– Telephone– In-person– Internet

• Knowledge Networks type of access

– Mixed mode (e.g., telephone/mail)– Mail

• pre-contact• cover letter• survey instrument• return envelope• incentive• reminder post-card• follow-up

Survey analysis (examples)• Univariate analysis

– Histograms, tables, charts, etc.– Mean, median, mode– Range, standard deviation

• Bivariate analysis– Crosstab/contingency tables– Correlations (Pearson’s r, Spearman’s rho, phi,

Cramers V)

• Regression analysis– Logistic, ordinal, linear, etc.

• So much more!

Final thoughts• Many rules

• Many resources

• Beg, borrow, and steal

NO such thing as a perfect study, survey, or sample!

NO such thing as a perfect study, survey, or sample!

So much more• Constraints -- Paperwork Reduction Act,

Institutional Review Boards

• Reliability and validity

• Errors -- sampling, coverage, measurement, nonresponse

• Biases

Resources• Dillman, D.A. 1999. Mail and Internet Surveys: The Tailored

Design Method, 2nd Ed. John Wiley Company, New York: NY

• Krueger, R. A. and M.A. Casey. 2000. Focus Groups: A Practical Guide for Applied Research. SAGE Publications

• Presser, S., J. Rothgeb, M. Couper, J. Lessler, E. Martin, J. Martin, and E. Singer. 2004. Methods for Testing and Evaluating Survey Questionnaires. New York: Wiley and Sons

• Tourangeau, R., Rips, L.J., and Rasinski, K. 2000. The Psychology of Survey Responses. Cambridge University Press.

• http://www2.chass.ncsu.edu/garson/pa765/survey.htm

• http://srcweb.berkeley.edu/index.html

• http://www.jpsm.umd.edu/jpsm/index.htm

• EACH OTHER!!