questionnaire (survey) design. measurement error: respondent’s cognitive process multicomponent...

Questionnaire (Survey) Design

Measurement Error: Respondent’s Cognitive Process

• Multicomponent Approaches

• Two track Theories– High road (Follows multicomponent)– Low road (short circuits steps of multicomponent)

Multicomponent Approach

1. Comprehension: of the question2. Retrieval: of relevant information from the

R’s memory3. Judgment: using the retrieved information to

judge the question4. Selection: of an answer

Overview

• What should be asked?• How should each question be phrased?• In what sequence should the questions be

arranged?• What questionnaire layout will best serve the

research objectives?• Does the questionnaire need to be revised?

What should be asked?

• Relevancy- – If you don’t need the info don’t ask

• Accuracy- – reliable and valid

How should each question be phrased?

• Open-ended

Fixed alternatives

Simple dichotomy

– Determinant choice– Attitude rating scale– Checklist (Check all that apply)

• Code as individual yes/no questions

– Categories should be mutually exclusive and exhaustiveEx. How old are you?

1. 0-12 2. 12-16 3. 16-24 4. 24-85

Choices Matter.

• <30min;30-59min;1-1.5;>1.5-<=2hrs; etc

• <2.5

Choices Matter.Half of the respondents were given a scale that began with <30 minutes and then proceeded in half-hour increments ending with 4.5+ hours. The other respondents were given the same scale except the first five answers were compressed so that it began with <2.5 hours. Only 16 percent of the respondents given the first set of response alternatives reported watching more than 2.5 hours of TV per day, but 32 percent of the respondents given the second set of response alternatives reported watching more than 2.5 hours of TV per day.

The Art of Asking Questions

• Avoid complexity– Ex. In deciding on optimum B & M location, the

franchiser requests community input on the mean travel distance for its clientele, how far do you travel?

The Art of Asking Questions

• Avoid leading and loaded questions– Ex. Culvers sells a new deliciously fattening

custard, in a typical sitting how much would you like to eat?

• Exception likert scales ask people to affiliate with attitudes.

The Art of Asking Questions

• Avoid ambiguity– Ex. How many times have you eaten at culvers?

The Art of Asking Questions

• Avoid making assumptions– Ex. How would you rate the service at the new

Culvers?

The Art of Asking Questions

• Avoid “double-barreled” items– Ex. How much do you like the burgers and custard

at Culvers?

The Art of Asking Questions

• Avoid burdensome questions that may tax the R’s memory– Ex. What did you order on your first visit to

Culvers?

In what sequence should the questions be arranged?

• Order Bias- election ballots– order of answers, – order of questions

• Funnel Technique- general to specific questions

• Filter question- a screening question to be used to avoid making assumptions– Ex. Have you ever contracted an STD?

• Pivot question- for sensitive info, like income– Ex. What was your income last year?

• Was your income greater than 10,000?• Was it greater than 20,000?

What questionnaire layout will best serve the research objectives?

• Instructions to interviewers should be in CAPS, italics,

• The survey should have plenty of free space• Precode the responses• Clearly direct the interviewer with skips etc.

Does the questionnaire need to be revised?

• TEST IT!!

Suggested components

• Questionnaire number• phone number? • Interviewer ID

• Informed consent (part of email). • Demographics section

Example Survey Preamble/EmailDear Respondent: My name is XXXXX and I am conducting a research project on business advisory boards at colleges and universities. We are interested in the opinions of faculty from AACSB schools as The University of Somewhere is in the start-up phase for establishing a business advisory board. Could you please take a few moments and answer this very short survey on the attached file? Even if you have very little contact with your advisory board I am very interested in your opinion. By completing this survey you will be contributing to the goal of identifying best practices for establishing and maintaining a business advisory board and I will send you the study findings upon request.  All answers will be kept confidential to the extent possible, acknowledging that any form of communication over the Internet does carry a minimal risk of loss of confidentiality.  All information will be reported in the aggregate. Participation in this study is voluntary, and all participants may refuse to answer any question found on the survey.  I thank you for your time and consideration in contributing to this project and if you have any questions about your rights as a research subject you may contact my advising professor name here in the XXX Office of Sponsored Research at XXXXXX or email here to discuss them. Sincerely, 

XXXXX

Where are you from?_________

What is your income_______?

Are you interested in studying abroad?

(0)Yes (1)No

We are trying to gauge your level of interest in studying abroad? Are you very interested, somewhat interested, not very interested, or not at all interested?1. Very interested 2. Somewhat interested 3. Not very interested 4. Not at all interested

Is your religion a factor in your decision making?

1. Yes 0. No

Decision guide for questions• A. Does the survey question measure some aspect of the research

question?• B. Does the question provide information needed in conjunction with

some other variable?– {if no to both A and B drop question, if yes to at least one proceed}

• C. Will most Rs understand the Q and in the same way?– {if no revise or drop; if yes proceed}

• D. Will most Rs have the information to answer it?– {if no revise or drop; if yes proceed}

• E. Will most Rs be willing to answer it?– {if no revise or drop; if yes proceed}

• F Is other information needed to analyze this Q?– {if no Proceed. If yes proceed only if the other information can be collected}

• G. Should this question be asked of all Rs or a subset?– {If all proceed, if a subset proceed only after appropriate filter Q}

But First a Sampling Experiment

Each group of students should:1. Pull 5 candies out of the bag2. Weigh the candies3. Write down the weight4. Put the candies back in the bag!!5. Pass the scale and bag to your neighbors6. Silently multiply the weight of the 5 candies by 20.

Type Weight (g)

Hersheys 43

Hot tamales 14

Jr mint 18

peanut butter kiss 7

Carmels 8