oct. 31 st, 2005 lisa a. lauxman, ph.d nae4-ha the art and science of creating useful surveys...
TRANSCRIPT
Oct. 31st, 2005Lisa A. Lauxman, Ph.D
NAE4-HA
The Art and Science Of Creating Useful Surveys
Arizona Cooperative Extension
Survey Design Fundamentals
• Survey is a method of gathering data
• Information collected may include people’s ideas, feelings, plans, beliefs, social, educational, and financial background.
• Questionnaires and interviews are the most commonly used surveys
Survey Purposes
• Survey purposes and methods fall on a continuum. Some have far-reaching generalizable effects
• Other are conducted to meet specific needs
• Identify purpose of survey• Program evaluation• Research• Policy needs
Survey & Questionnaire Components
Questions Instructions Sampling and design Data processing and analyses Pilot testing Response rate Reporting results
Most Common Survey Designs
Cross sectional: Data collected at one point (called a census if all members of a population are surveyed)
Most Common Survey Designs
Longitudinal: Data collected over time (note difference between sampling from a population and following the same people over time
– Trend: Sample a population over time e.g. 17 year olds in 1995 and 17 year olds in 2005
– Cohort: Same population sampled over time e.g. 17 years old in 1990 and 27 years old in 2000
– Panel: Same individuals over time e. g. the 17 year olds in 1995 and same people in 2005
Types of Questions
Close-ended survey items Open-ended survey items: Do you get any
satisfaction? – Find out whether people are satisfied with
product, service or program– Are particularly appropriate when collecting info
about what people like best or least
Writing Items
Survey items may take the form of questions– Example of open-ended question
How courteous are the people who make your appointments?
Survey items may be worded as statements– Example of close-ended statement
Circle your agreement or disagreement with the following: 2) The people who make my appointments are
(a) definitely courteous(b) agree(c) disagree)(d) definitely disagree
Rules for Writing Forced Choice
Each question should be meaningful to respondents
Use standard English Make questions concrete Avoid Biased words or phrases Check your own biases Each question should have only one thought
Examples of Scales Used for Forced Choice
Yes or No Checklist- provides a series of answers.
Respondents may choose just one or more answers Rating scales- 4 types
– Nominal or categorical responses, i.e.. Male or Female– Ordinal- require respondents to place answers in a rank
order, i.e.. High, medium or low – Interval – These have measurement choices with real
meaning between the numbers, i.e.. Annual income– Ratio- Has real meaning between numbers with
equidistance between the numbers. Ex- height and weight
Putting Questions in Order
All surveys should be preceded by an introductions First set of questions should be related to topic described Ask relatively objective questions before subjective ones Move from familiar to the least Follow natural sequence of time See to it that all questions are independent Relatively easy-to-answer questions should be asked at end Avoid items that look alike Sensitive questions should be placed well after the start but
also well before the end Questions should be asked in logical order
Response Scales-Likert-Type
Likert-type Scales– Balance all responses ex. 4, 6– Use neutral responses (no response, don’t know)– Use 4 to 7 point scale– Put negative end of scale first– Use meaningful scale
Endorsement ( Def. True > Def. False) Frequency ( Always > Never) Intensity ( Severe > None) Influence ( Big Problem > No Problem Comparison ( Much more than others > Much less than
others
Test the Survey
Ease of recording responses Check for typos, misspelled words Length Understanding of questions- easy? Difficult? Test for readability Are questions appropriate for people who will be
surveyed? What about data collected? Does it make sense?
How useful is the data? Check for cultural sensitivity
Response Scales- Checklist
Use ranking only if respondents can see or easily remember all choices
Respondents over-report frequent behaviors Big events are more easily remembered Use list to help jog memory (Yes/No to each)- Break
into components Behavior
– Time– Duration– Frequency
RELIABILITY
Test-retest reliability– Most commonly used; same respondents, same survey,
different time; correlation of .70 or better is generally considered accepable; item should NOT be likely to change over time; beware of “practice effect” which can falsely inflate test-retest reliability
Alternate-form reliability– Items differ only in wording such as change order of the et,
response or wording of item; correlation; need large enough sample size to increase the “power” ( the ability to detect difference between forms IF they exist)
RELIABILITY
Internal consistency reliability: – How groups of items “hang together”; Cronbach’s
coefficient alpha; Cronbach’s alpha reflects the homogeneity of a scale.
Inter-observer reliability (.a.k.a. inter-rater reliability):
– Correlation between two observer’s scores on same subject
Intra-observer reliability– Correlation between two observer’s scores on
same subject
VALIDITY
Face validity:Subjective; cursory view by untrained judges;
casual
Content validity: Subjective; not quantified; typically an organized view of what the survey covers in terms of subject matter
VALIDITY
Criterion validity: Compared to another instrument for prediction; Types include:
Concurrent validity: – needs to be relevant, well-known and is like
being compared to a “gold standard”
Predictive validity:– Ability to forecast attitudes, behaviors, outcomes,
events
VALIDITY
Construct validity: most valuable, yet most difficult to obtain; meaning of scale/instrument in practical use, a gestalt, rather than a calculated value; typically results from much effort and often years of work;
Types include: Convergent validity: usually from multiple
investigators from different approachesDivergent /Discriminant validity – much
effort, years of work
Survey Monkey.com
Professional Growth and Development Survey
http://surveymonkey.com/Users/32263842/Surveys/33828911949/EF5500DC-AD39-4186-A9FB-5425E2C3ED1F.asp?U=33828911949&DO_NOT_COPY_THIS_LINK
Arizona Master Gardener Survey
file:///C:/Documents%20and%20Settings/Owner/My%20Documents/Work%20Stuff/SurveyMonkey.com/485F788C-85FC-4E68-856A-A16CB2421AB6.asp.html
Creating Your Own Survey
http://surveymonkey.com/Users/32263842/Surveys/33182449893/F47D9320-AC19-4744-8D5B-9376E1AA9FEA.asp?U=33182449893&DO_NOT_COPY_THIS_LINK
http://surveymonkey.com/Users/32263842/Surveys/108271317032/260979FC-5246-47A3-8467-CC7A6854A795.asp?U=108271317032&DO_NOT_COPY_THIS_LINK
http://surveymonkey.com/Users/32263842/Surveys/83749780641/1F6E31E3-6493-4CA2-A355-0A880364380D.asp?U=83749780641&DO_NOT_COPY_THIS_LINK