PowerPoint Presentation

Introduction to Survey DesignModule 7Erika Trapl, PhD

Two-Part Survey SeriesModule 7Matching survey items to research questionsQuestion Selection/DevelopmentResponse categoriesSurvey design and formatting

Module 8Taking your survey liveData collection protocolsFive Stages of Survey Development and CompletionInitial survey design and preliminary planningPre-testingFinal survey design and planningData collectionData coding, data-file construction, analysis, final report

Five Stages of Survey Development and CompletionInitial survey design and preliminary planningPre-testingFinal survey design and planningData collectionData coding, data-file construction, analysis, final report

Survey Design and Preliminary PlanningWhat are the goals of the survey?To test a hypothesis (or a hypothetical model of interrelated hypotheses)eg, Females are more likely than males to believe that a woman should be permitted to have an abortion for any reason. To estimate the proportion of people who hold a certain belief or engage in a specific behavioreg, What proportion of our population believe that our criminal justice system is working well?

Initial Design of the QuestionnaireWhat types of information do you need to collect from participants?How can you best elicit that information?Open-ended questionseg, What is the most difficult problem facing families today? Best for face to face rather than mailed surveysCan provide lists of choices, visual aids, participant can consult recordsClosed-ended questionseg, Do you approve or disapprove of abortion?Respondents choose from a list of provided responses

Final Survey Design and PlanningActivityNo. WeeksReview literature and draft questionnaire8Assemble think alouds with 5 respondents1-2Revise questionnaire2Conduct pretest (n=25-30)2Debrief interviewers and revise questionnaire3Pretest #2 (n=20-25)2Debrief interviewers, revise questionnaire, develop training materials4Select samples4Conduct main data collection8Code data & prepare data files10Analyze data and write report16Develop a Study TimelineDrafting the QuestionnaireBorrowing questions from other research studies is acceptableEnables comparison of your study results to previous studiesNote how researchers in past studies defined concepts Note the questions they used to measure these conceptsFree to use the same wording for items unless copyrightedCaveat: dont assume that the questions are problem-free just because someone else developed and used them

Drafting the QuestionnaireMust pre-test questionnaires even if borrowed most items from prior studiesExpect to do many drafts of the questionnaire to work out all of the problems

Guiding Questionsor, What do you want to know?The first step: write a series of guiding questions or objectives that clearly indicate what you are trying to find out through the survey Examples as questions:How do occupational therapists incorporate play into their practice?Do occupational therapists assess play behaviors and what methods do they use?Are there differences between school-based and non-school-based settings that influence the role of play within pediatric occupational therapy?

Guiding QuestionsHypotheses may be used to examine relationshipsExamples as hypotheses:Men will rate themselves higher in career success compared with womenMen will report greater importance of salary and position compared with womenDeveloping Survey QuestionsOperationalize the Research Questions into survey questions/itemsBe concreteSpecify exactly what information you want from respondentsConsider all possibilities of interpretation and responseHow is a question answered?For each question, the respondent must:Comprehend the question (interpret the subject and the task)Recall information, form judgmentDecide on answerRecord answer on a form or speak the answer to an interviewerAnticipate potential problems at each step for each question

Developing Survey QuestionsHow many college mathematics courses have you taken?

Possible interpretations: Courses being taken now but not yet completedCollege-level math courses taken in high schoolHigh school level remedial math courses taken in collegeMath courses taken by transfer students at their previous collegeA math course taken but failed last semesterA course that primarily involves math but is given by another department or is not listed as a math courseAn audited math course not taken for creditA math course taken on a pass-fail optionCollege level math courses taken and completedDeveloping Survey QuestionsIf the choice of what to count is left to respondents, different people will include different things, introducing measurement error into the dataFor the math question, it may be more useful to break up the question into a number of questions that ask about the different types of courses (ie, regular, audit, pass-fail)But if and how you do this depends on your Research Question(s)!Developing Survey QuestionsNot every possible question can be included in a survey But you should include all the questions you need to answer your Research Question

Each question has a cost and must be justified

So, how do you decide whats in and whats out?

Choosing Survey QuestionsDecision Guide Does the question measure some aspect of the Research Questions?Does the question provide information needed in conjunction with some other variable? (If no to both A & B, drop the question. If yes to one or both, proceed).Will most respondents understand the question in the same way? (If no, revise or drop. If yes, proceed).Will most respondents have the information to answer it? (If no, drop. If yes, proceed)Will most respondents be willing to answer it? (If no, drop. If yes, proceed)Is other information needed to analyze this question? (If no proceed. If yes, proceed if the other information is readily available or can be obtained from the survey)Should this question be asked of all respondents or of a subset? (If all, proceed. If a subset, proceed only if the subset is identifiable beforehand or through questions in the questionnaire/interview)

Choosing Good Survey QuestionsThe questions should pose tasks that the respondent can do well (ie, recall information, make judgments)In wording the question, do not influence the respondents answer in one direction or another Questions should be uncomplicated, explicit, and natural-soundingAvoid ambiguityAvoid double-barreled questionseg, Do you think the police and the courts are doing an excellent, good, fair, or poor job?Avoid agree/disagree response format Acquiescence response set

Structure of Survey QuestionsTwo main categories of survey questions: open questions & closed questionsResponses to open questions are narratives that must be interpreted and coded by the investigatorsRecommend using closed questions as much as possible

Structure of Survey QuestionsClosed questions have 2 partsStatement of the question (aka item)Response categories (aka options)The question should be stated as directly as possibleThe answer must be given in terms of only one of the answer choices provided (typically)Qualifiers and conditions are commonly used in questions to limit the question to match the research objectivesExample QualifiersHow many times have you shopped at the Book Center since January?In the past year, did anything happen to you or your property that you thought was a crime or an attempt at a crime?In the past year, would you say that the violent crime situation in your neighborhood has gotten better, gotten worse, or stayed about the same?Closed Ended Response FormatsResponses should be exhaustive; including all possible responsesProtections to assure this include NA, Dont know, or Other (please specify) _________If interested in more than one answer to a question, offer the mark all that apply optionEach response should be mutually exclusive; each choice should clearly represent a unique answerOrder responses according to an inherent hierarchy if one exists (eg, increasing or decreasing intensity)Response options: NOIRNOMINAL: Mutually exclusive, but not ordered, categories (gender)ORDINAL: Order matters but not the difference between values. For example, you might ask patients to express the amount of pain they are feeling on a scale of 1 to 10. A score of 7 means more pain that a score of 5, and that is more than a score of 3. But the difference between the 7 and the 5 may not be the same as that between 5 and 3. The values simply express an order.INTERVAL: Measurement where the difference between two values is meaningful. The difference between a temperature of 100 degrees and 90 degrees is the same difference as between 90 degrees and 80 degrees.RATIO: has all the properties of an interval variable, and also has a clear definition of 0.0. When the variable equals 0.0, there is none of that variable. Variables like height and weight are ratio variables.Response options have implications for data analysis and interpretation.Categorical Response FormatsWhat do you like most about your job? ___ Interaction with clients___ Interaction with other staff___ Intellectual challenge___ Opportunities for professional growthDisadvantages of this type of response setDoes not allow respondents to express their personal viewpointsMay over look some important responsesMay bias answers by presenting a particular attitudeCategorical - Nominal scale has implications for analysisAlmost like four different yes/no itemsCategorical Response FormatsWhat race or ethnicity do you most identify with?___ Asian___ American Indian or Alaskan Native___ Black or African American___ White or Caucasian___ Native Hawaiian or Pacific Islander___ I identify with more than one___ I dont identify with any of these

In this example, we want respondents to categorize themselves.

Continuous Response FormatsCharacteristics on a continuum, such as attitudes or quality of performanceA range of responses allows the respondent to select the choice representing the intensity of their responseExample:Very importantImportant Somewhat importantNot importantUnsureFrequently usedContinuous scale (ordinal, interval or ratio)Common Response CategoriesOpinionsKnowledgeFrequency of EventsRatingsSatisfied / Somewhat satisfied / Somewhat dissatisfied / Very dissatisfiedVery Familiar / Somewhat familiar/ Not too familiar/ Not at all familiarPer day/Per week/Per month/Per year/NeverExcellent/ Good / Fair/ PoorImportantTrue/FalseAlways/Frequently/ seldom/neverHigh / Medium /LowOppose/SupportA lot/Some/ A little/ NoneAlways/Sometimes/ NeverGotten better / Gotten worse/ Stayed about the sameStrongly oppose/ Oppose/ Support/ Strongly supportAll/Most/Some / A few / NoneVery fair/ Fair / Unfair / Very unfairOften/ Sometimes/ Rarely / NeverMeasurement ScalesA scale is an ordered system of questions (a set of items) that provides an overall rating representing an attitude, value, or characteristic Measures one dimension or characteristicEach item on the scale carries the same weight or point valueResults in a calculated summary score

Examples: Stress, Depression, Parental monitoringTypes of ScalesLikertSemantic differentialVisual analogueGuttmanLikert ScaleUsed to assess an attitude or a valueOriginally a 5-point response category formatNo consensus on the number of response categories that should be usedShould there be a neutral category?ExampleStrongly agreeAgreeNeutralDisagreeStrongly disagree

Likert Scale DevelopmentA large pool of items should be developed that reflect an equal number of favorable and unfavorable attitudesDo not include items that are intended to elicit neutral responsesScale should be validated through item analysisThose who score the highest should agree with positively worded statements, those who score the lowest should disagree Semantic DifferentialMeasures attitudes or feelings about a concept on a continuum that extends between the two extremesTypically a 7-point scale Only extremes are labeledTends to measure evaluation, potency, and/or activityGood ____ ____ ____ ____ ____ ____ ____BadVisual Analogue Scale

Measures intensity of a subjective experienceA 100 mm line is typically drawnWord anchors on either end represent extremes of a characteristicRespondent places a mark along the line to indicate their perceived level for that characteristicScored by measuring in millimetersOften treated as ratio level dataNo pain

_______________________________Pain as bad as it can beGuttman ScaleAddresses problematic issue with Likert scaledifferent individuals with the same summary score on Likert scale may have marked different responsesGuttman cumulative scale ensures that there is only one unique combination of responses that can achieve a particular score

Guttman ScaleExample Bogardus Social Distance Scale

Are you willing to permit immigrants to live in your country? Yes NoAre you willing to permit immigrants to live in your community? Yes NoAre you willing to permit immigrants to live in your neighborhood? Yes NoAre you willing to permit immigrants to live next door to you? Yes NoWould you permit your child to marry an immigrant? Yes NoPutting it togetherYou have all of your questions and responses selectedwhats next?Organization of the QuestionnaireGroup according to categories or topicsQuestions should proceed from general to specificInitial questions should pique the respondents interest or at least be neutralSensitive questions should come later Some put demographic questions at the beginning, others at the end because they are less interesting/engaging for the respondentOrganization of the QuestionnaireMake it easy for the respondent to navigate the surveyUse headers and simple instructions Page should be uncluttered in appearanceLaser quality printClear alignment of questions/responsesFont of 11 or 12 pointWhite or light-colored paper

SourcesCzaja R, Blair J. 1996. Designing Surveys: a guide to decisions and procedures. Thousand Oaks, CA: Sage. Portney LG, Watkins MP. 2000. Foundations of Clinical Research, 2nd Ed. Upper Saddle River, NJ: Prentice Hall.