Survey questionnaires, a data collection instrument

Outline1. What is a questionnaire?2. Open vs. Closed Questions 3. Modes of questionnaire distribution4. Piloting5. Take-a-ways

Quiz time!

A questionnaire . . .

A. Is what that guy on the Hill has on his clipboard.B. Is something you throw away with Tuesday’s junk mail.C. Causes you to hang-up on someone around election time.D. Is a cost effective means of gathering original data from a

large sample population that can be performed via personal interview, telephone, or mail.

E. All of the above.

What is a questionnaire?• A questionnaire is a form of data collection, a survey

methodology• Questionnaires “pose standardized, formally structured

questions to a group of individuals, often presumed to be a sample of a broader population . . . Questionnaires are useful for gathering original data about people, their behavior and social interactions, attitudes, and opinions, and awareness of events (McGurik and O'Neill 2005).”

• Can be used as part of a mixed methodology

Geographical issues that questionnaires can explore• Perceptions of risk• Social networks• Coping behaviors (HIV, unemployed)• Environmental attitudes• Travel patterns• Mental maps• Power relations in a particular setting• Gender roles in the household• Access to employment, social services, education . . .

Strengths and Weaknesses

Strengths of Questionnaires • Gather original data about opinions, behaviors, attitudes and

awareness• Provide insights to relevant social trends• Cost-effective meaning you can extend your research over a

large area• Very flexible i.e. can be combined with other methodologies

Weaknesses of Questionnaires• Structure of questions can influence responses• Depth and originality of answers can be limited• Low response rates for the more cost effective modes of

delivery

Things to keep in mind

• Requires a lot of preparation and forethought, design is critical

• Make sure you can’t get quality data from somewhere else, don’t just use this to generate your “own” data

• Questions should be relevant to and focused on your research question

• Each question must have a clear purpose and be relevant to your critical analysis

• Be aware of the social and cultural limits of what your respondents are willing or able to answer• Can the respondents understand the questions?• Do they have the knowledge to answer them?• Do your questions make them uncomfortable?

Closed Questions

Closed/ Fixed• Benefits• Can get quantitative data from these• Select an answer, rank, select a point on a scale• Major benefit is that they are easy to code and analyze

• Challenges• Demanding to design• Have to know how the question will be interpreted and must

assume that everyone will understand the meaning of the words and concepts to be the same

• Respondents are limited to the range of answers provided

Open-ended Questions

Open questions• Benefits• Less structured• Respondents get to recall experiences or explain understanding in

their own words and terms• Allows respondents to challenge the structure of the

questionnaire• You can get valuable and unanticipated insights• Major benefit is it gives you the most robust perspective, major

drawback is that it’s more challenging to analyze• Challenges• Can’t use this data for statistical analysis• Requires a lot of effort to interpret

Quantitative, Qualitative, Mixed• Fixed questions allow you to do quantitative analysis on sample

populations, used to make statistical claims about the population at large

• In qualitative analysis questionnaires are used usually as part of mixed methods i.e. non-probability sampling, looking for trends, themes

• Start your mixed method questionnaire with quantitative

(closed-ended) questions and progress to the qualitative (open-ended questions)

• *DON’T try to turn open-ended answers into quantitative data, its mixed methods!*

Mode of Questionnaire Distribution

Mail• The Good• low cost• large coverage area• good for sensitive subjects, anonymity

• The Bad • length and complexity limits• once its mailed it’s out of your hands• low response rate 30-40%

Email• will limit your audience to those with email accounts

(obviously problematic in many researach sites)• some report high (and detailed) response rates

Mode of Questionnaire Distribution

Face to face (interviewer) • The Good• 80-85% response rate• Can usually get longer more detailed answers verbally than in

writing• The interviewer can explain uncertain terms and insist on more

detail• The Bad• Expensive (especially if you hire professionals), time consuming• Spatially limited• Data quality is dependent on interviewers• Interviewees may self-censor in the presence of an interviewer

Mode of Questionnaire Distribution

Telephone• The Good • Historically good response rates• Cheaper and less spatially restricted than face to face interviews• Can give the feeling of anonymity overcoming self-censorship

• The Bad• Can introduce bias since not every has or reacts the same on the

telephone• Not everyone has a land line, do you really expect them to use

there cellphone minutes on a questionnaire?

Piloting, pre-testing, road-testing

. . . is essential.

• Do it with a subset of your target audience, this gives you a chance to work out kinks • Did they interpret the questions as you hoped?• Did they answer open ended questions fully?• Were they uncomfortable with the questions?

• How long did it take to complete the questionnaire? (30 min should be the max)

• If you are doing face to face interviews this is a time to build your confidence

• Vet your questionnaire with experienced surveyors, experience is priceless in crafting a good questionnaire

Take-a-ways• Survey questionnaires can be a useful part of you research

toolkit• Survey questionnaires should be viewed as mixed method• Survey questionnaires can be helpful in reaching large areas

cost affectively• There are tradeoffs between the depth of answers, cost

effectiveness of the questionnaire, and response rate.

Bibliography

• McGurik, Pauline M., and Phillip O'Neill. "Using Quesitonnaires in Qualitative Human Geography." In Qualitative Research Methods in Human Geography, edited by

Iain Hay, 147-162. New York: Oxford University Press, 2005.

• McLafferty, Sara L. "Conducting Questionnaire Surveys." In Key Methods in Geography, edited by Nicholas J. Clifford,

& Gill Valentine, 87-100. London: Sage Publications Ltd., 2003.

Questions and Comments