semester review. the tailored design method uses multiple motivational features in compatible and...
TRANSCRIPT
The Tailored Design Method
• Uses multiple motivational features in compatible and mutually supportive ways to encourage high quantity and quality of responses
The Tailored Design Method
• Premised on social exchange perspective on human behavior
• Assumes that the likelihood of responding is greater when the expected rewards outweigh the anticipated costs
The Tailored Design Method
• Gives attention to all aspects of contacting and communicating with respondents
• Encourages response by considering survey sponsorship, the nature of the population and variations within it, and content of questions
The Tailored Design Method
• Emphasizes reducing errors of coverage, sampling, nonresponse, and measurement
Coverage Error
• Occurs when all members of a population do not have a known, non-zero probability of selection
• Occurs when those who are excluded are different from those who are included
Sampling Error
• Results from surveying only some rather than all members of a population
• Represented by B, the bound on the error of estimation
Nonresponse Error
• Occurs when people selected do not respond and are different than those who do
• Nonresponse can occur at the level of items within a survey or at the level of the survey– MAR– MCAR
Measurement Error
• Occurs when responses are inaccurate or imprecise
• Primarily related to poor layout and poor design and wording of questions
Social Exchange and Surveys
• Addresses three central questions about design and implementation1. How can the perceived rewards for
responding be increased?2. How can the perceived costs of
responding be reduced?3. How can trust be established so that
people believe the rewards will outweigh the costs of responding?
Increasing Benefits
• Provide information about the survey• Ask for help or advise• Show positive regard• Say thank you• Support group values• Give tangible rewards• Make the questionnaire interesting• Provide social validation• Inform people that opportunities to
respond are limited
Decreasing Costs
• Make it convenient to respond• Avoid subordinating language• Make the questionnaire short and
easy to complete• Minimize requests for personal or
sensitive information• Emphasize similarity to other
requests or tasks to which a person has already responded
Establishing Trust
• Obtain sponsorship by legitimate authority
• Provide a token of appreciation in advance
• Make the task appear important• Ensure confidentiality and security of
information
Features that can be Tailored
• Survey mode– Singular or multiple
• Sample design– Type of sample– Number of units sampled
• Incentives– Type of incentive– Amount or cost of incentive– Before or after
Features that can be Tailored
• Contacts– Number of contacts– Timing of initial and subsequent
contacts– Mode of each contact– Whether contacts will be personalized– Sponsorship information– Visual design of each contact– Text or words in each contact
Features that can be Tailored
• Additional materials– Whether to provide them at all– Type of materials (e.g., research report)– Visual design of materials– Text or wording of materials
Features that can be Tailored
• Questionnaire– Topics included– Length (duration, number of
pages/screens, number of questions)– First page or screen– Visual design– Organization and order of questions– Navigation through questionnaire
Features that can be Tailored
• Individual questions– Topic (sensitive, of interest to the
respondent)– Type (open-ended versus closed-ended)– Organization of information– Text or wording– Visual design
Central Terminology
• An element is an object on which a measurement is taken
• A population is a collection of elements to which an inference is made from a sample
• A sample is a collection of sampling units drawn from a frame or frames
• Sampling units are nonoverlapping collections of elements from the population that cover the entire population
• A frame is a list of sampling units
Central Terminology
• A completed sample is the units that respond
• Sampling error is the result of collecting data from only a subset, rather than all, units from a frame– Again, represented by B, the bound on
the error of estimation
Coverage
• The degree to which the units in a sampling frame correspond to the population of interest
• Coverage is likely one of the most serious problems in most surveys
Coverage and Frame Problems
Target Population Frame Population
Covered Population
Undercoverage
Und
erco
vera
ge
Ineligible Units
Ineligible Units
Reducing Coverage Error
• Central questions:– Does the list contain everyone in the
survey population?– Does the list include people who are not
in the study population?– How is the list maintained and updated?– Are the same sample units included on
the list more than once?– Does the list contain other information
that can be used to improve the survey?
Issues to Consider
1. What survey mode(s) will be used to ask the questions?
2. Is the question being repeated from another survey, and/or will answers be compared to previously collected data?
3. Will respondents be willing and motivated to answer accurately?
4. What type of information is the question asking for?
Choosing Words and Forming Question1. Make sure the question applies to the respondent2. Make sure the question is technically accurate3. Ask one question at a time4. Use simple and familiar words5. Use specific and concrete words to specify the
concepts clearly6. Use as few words as possible to pose the question7. Use complete sentences with simple sentence
structures8. Make sure “yes” means yes and “no” means no9. Be sure the question specifies the response task
Visual Presentation of Survey Questions1. Use darker and/or larger print for the question and lighter and/or
smaller print for answer choices and answer spaces2. Use spacing to create subgrouping within a question3. Visually standardize all answer spaces or response options4. Use visual design properties to emphasize elements that are
important to the respondent and to deemphasize those that are not5. Make sure words and visual elements that make up the question
send consistent messages6. Integrate special instructions into the question where they will be
used rather than including them as freestanding entities7. Separate optional or occasionally needed instructions from the
question stem by font or symbol variation8. Organize each question in a way that minimizes the need to reread
portions in order to comprehend the response task9. Choose line spacing, font, and text size to ensure the legibility of the
text
General Premises
• The design of a questionnaire should consider how to motivate the recipient to respond
• It should also avoid measurement errors, ranging from order effects to item nonresponse
Ordering Questions
• General guidelines1. Group related questions that cover
similar topics, and begin with questions likely to be salient to nearly all respondents
2. Choose the first question carefully3. Place sensitive or potentially
objectionable questions near the end4. Ask questions about events in the order
that they occurred5. Avoid unintended order effects
Visual Stimulus
• General guidelines1. Establish consistency in the visual presentation of
questions (across pages and screens) and use alignment and vertical spacing to help respondents organize information on the page
2. Use color and contrast to help respondents recognize the components of the questions and the navigational path through the survey
3. Visually group related information in regions through the use of contrast and enclosure
4. Use visual elements and properties consistently across questions to emphasize or deemphasize certain types of information
5. Avoid visual clutter6. Minimize the use of matrixes and their complexity
Yes
No
First name/initials(optional) (name)
Relative 1
(name)
Relative 2
(name)
Relative 3
Does this relative live in Kalamazoo? . . . . . . .
If no: about how far away from Kalamazoo do they live? . . . . . . . . Miles Miles Miles
About how old is this relative? . . . . . . . . . . . .
Years Years Years
Every day
Every week
Approximately, how often do you communicate with this relative? . . . . . . . . . . . .
Once a month
Less than once a month
Every day
Every week
Once a month
Less than once a month
Every day
Every week
Once a month
Less than once a month
Mail Questionnaires
• General guidelines1. Determine whether keypunching or optical imaging and
scanning will be used, and assess the limitations imposed on designing and processing questionnaires
2. Construct paper questionnaires in booklet formats, and choose physical dimensions based upon printing and mailing considerations
3. Decide question layout and how questions will be arranged on each page
4. Use symbols, contrast, size, proximity, and pagination effectively when designing branching instructions to help respondents correctly execute them
5. Create interesting and informative front and back pages that will have wide appeal to respondents
6. Avoid placing questions side-by-side on a page so that respondents are asked to answer two questions at once
Web Questionnaires
• General guidelines1. Decide whether an electronic alternative is
appropriate2. Choose how the survey will be programmed and
hosted, commensurate with needs, skills, and sponsorship
3. Take steps to ensure that questions display similarity across platforms, browsers, and user settings
4. Decide how many questions will be presented on each page and how questions will be presented
5. Develop a screen format that emphasizes the respondent rather than the sponsor
Web Questionnaires
• General guidelines6. Use a consistent page layout across screens and
visually emphasize questions information that respondents will need to complete the survey while deemphasizing information not essential to the task
7. Do not require responses unless absolutely necessary
8. Design survey-specific and item-specific error messages to help respondents troubleshoot
9. Evaluate carefully the use of interactive features, balancing improvements in measurement with the impact on respondent burden and the implications with mixed-mode surveys
Web Questionnaires
• General guidelines10. Use audiovisual capabilities sparingly, and evaluate
the differential effect they have on respondents11. Allow respondents to stop the survey and finish
completing it at another time12. Whenever possible, collect paradata that provide
feedback on how respondents interact with questionnaire
13. Test the survey using a variety of platforms, connection speeds, browsers, and user-controlled settings, and test the database to ensure that items are collected and coded accurately
14. Take screenshots of each page of the final questionnaire for testing and documentation
Pretesting Questionnaires
• General guidelines1. Obtain feedback on the draft of the questionnaire
from a number of people, each of whom has specialized of some aspect of questionnaire quality
2. Conduct cognitive interviews of the complete questionnaire in order to identify wording, question order, visual design, and navigational problems
3. When the stakes are high, consider experimental evaluations of questionnaire components
4. Conduct a small pilot study with a subsample of the population in order to evaluate interconnections among questions, the questionnaire, and implementation procedures
Mail Survey Implementation
• When high quality implementation procedures are used, response rates of 50%-70% are not uncommon– In my own work, I have had response
rates as high as 90%
• Implementation must receive considerable thought and planning– It should not occur after questionnaire
design, but simultaneously
Mail Survey Implementation
• General guidelines1. To the extent possible, personalize all
contacts to respondents (even when names are unavailable)
2. Send a token of appreciation with the survey request
3. Use multiple contacts, each with a different look and appeal
4. Carefully and strategically time all contacts5. Select all mail-out dates with the
characteristics of the population in mind
Mail Survey Implementation
• General guidelines6. Place information in the mailing exactly
where it needs to be used7. Take steps to ensure that mailings will not
be mistaken for junk mail or marketing materials
8. Evaluate the size and weight of mailing materials on mailing costs
9. Assemble the mailings in a way that maximizes the appealing aspects of each element when the package is opened
Mail Survey Implementation
• General guidelines10.Ensure that all addresses in the sample comply
with current postal regulations11.Assign an individual ID number to each sample
member12.Establish procedures for dealing with
undeliverable mail13.Establish procedures for dealing with returned
incentives14.Establish procedures for dealing with respondent
inquiries15.Evaluate early returns for problems that can be
addressed mid-stride
Prenotice Letter
• Sent to respondents a few days prior to the questionnaire
• It notes that a questionnaire is for an important survey that will arrive in a few days and that the person’s response will be greatly appreciated
• This letter can increase response rates by 3%-6%– No recent research comparing postcards
to letters (I prefer the former)
Questionnaire Mailing
• Includes a detailed cover letter explaining why a response is important
• The questionnaire• A prepaid postage envelope• A token incentive if one is to be
provided
Thank You Postcard
• Sent a few days to a week after the questionnaire
• The mailing expresses appreciation for responding and indicates that if the completed questionnaire has not yet been mailed it is hoped that it will be returned soon
Replacement Questionnaire
• Sent to respondents 2 to 4 weeks after the previous questionnaire mailing
• It indicates that the person’s completed questionnaire has not yet been received and urges the recipient to respond
Final Contact
• Sent using a different mode of delivery 2 to 4 weeks after the previous mailing
• The different mode of contact distinguishes each type of final contact from regular mail delivery
• Special contacts have been shown to increase response
Web Survey Implementation
• Major concerns associated with web surveys– Mode of initial contact– Extent to which personalized elements
can be used– Whether and how incentives are
delivered– Timing of contacts
Web Survey Implementation
• General guidelines1. To the extent possible, personalize all
contacts to respondents2. Send a token of appreciation with the
survey request3. Use multiple contacts and vary the
message across them4. Carefully and strategically time all
contacts with the population in mind5. Consider contacting respondents by
another mode when possible
Web Survey Implementation
• General guidelines6. Keep e-mail contacts short and to the point7. Take steps to ensure that e-mails are not
flagged as spam8. Carefully select the sender name and
address the subject line text for e-mail communications
9. Provide clear instructions for how to access the survey
10.Make obvious connections between the opening screen and other implementation features
Web Survey Implementation
• General guidelines11.Assign each sample member a unique ID number12.Know and respect the capabilities and limits of
the web server(s)13.Establish a procedure for dealing with bounced
e-mails14.Establish procedures for dealing with returned
incentives15.Establish procedures for dealing with respondent
inquiries16.Implement a system for monitoring progress and
evaluating early completes
Why Consider a Mixed-Mode Survey Design
• Lower costs• Improve timeliness• Reduce coverage error• Deliver incentives
Type II
• Use a second mode to collect responses from the same respondents for specific questions within a questionnaire
Mixed-Mode Surveys and their ImplicationsType Motivation Limitations
I. Use one mode to contact respondents and to encourage response by a different mode
• Improve response rates• Reduce coverage and
nonresponse error
• Increased implementation costs
II. Use a second mode to collect responses from the same respondents for specific questions within a questionnaire
• Reduce measurement error
• Reduce social desirability bias for sensitive question
• Increased design costs• Increased nonresponse if
respondent must respond by other mode at a later time
III. Use alternative modes for different respondents in a the same survey period
• Improve response rates• Reduce coverage and
nonresponse error• Reduce survey costs
• Increased design costs• Measurement error from
differences that may be confounded with differences among subgroups
IV. Use a different mode to survey the same respondents in a latter data collection period
• Different modes become available to survey respondents
• Reduce survey costs
• Increased design costs• Measurement error from mode
differences that impact the ability to measure change over time
Presence versus Absence of an Interviewer
• Locus of control• Normative question order effects• Social desirability• Acquiescence