acc survey research overview. discussion and goals 1.introduction to surveys 1.key considerations...
TRANSCRIPT
ACC Survey Research Overview
Key considerations for practical surveys
“A high response rate does not compensate for a faulty questionnaire or weak data analysis”*
What is the purpose of your survey – state clear objective.
WHAT DO YOU WANT KNOW and WHY?
What is the best way for me to collect this data based on the purpose,
timeline, population, budget and my expertise?
Determine what actions will result from the data collection.
Ask - do we need all the data we want to collect right now? Don’t ask
questions that do not inform an action.
Does your questionnaire help you determine next steps or how to
improve?
4
*Pubic Perspective, May/June 2003
Your Questions
• What’s a good response rate?• How do I get people to respond?• What are the do and don’t dos of surveys?• How do I know what questions to include?• Can I just write my own questions?• Where can I find surveys that others are
doing?• Can I compare x and y?• What do I do with the data?
5
Common Surveys – Modes -Types of Data
• Satisfaction Survey• Program evaluation• Course evaluation• Needs assessment• Interest/perception• Sponsor
• Qualitative• Quantitative• Mixed-Method• Mixed-Mode
6
Paper Web Phone In-Person
Do this Don’t do this Articulate your purpose Have an action plan for the results Share the results and actions you
take Describe your population Use research ambassadors Test your survey Plan your surveys annually Protect anonymity, report groups
over 5 or 10 Check what ACC or industry
surveys have the same information
X Create long surveys (over ten minutes)
X Ask questions just because…X Avoid open-ended questionsX Ask questions with jargon,
acronyms or terms that aren’t clear
X Design a questionnaire without peer review (even if its just one person)
X Promise to share results and not share them
X Make generalizations with small samples
7
Three important questions for members• Value• Net promotion (would you recommend)• Intent to renew
– Yes – What benefit most important/useful, etc.
– No- Reason for not continuing• Engagement
– Last attended event– Last conference– Participate in webinar, feedback, survey,
voting etc.8
Exercise
Take 5 minutes and as a group, write down the number one thing you want to know from members in your chapter. Be ready to share the following:1. The research question (what you want to
know)2. The rationale for wanting/needing to know3. What the question would look like (write it)4. What you would do with the results5. How you might analyze the data
9
Common Survey Topics for Associations
Satisfaction• With services• With programs• With support• With cost/value
Expectations• Assess at sign-up• Track progress
meeting expectations and retention
Needs• What are
members most pressing needs
• How are current needs being addressed
COURSE/EVENT• Measure aspects not
the whole event• Consider a pre-
survey to leverage expectations with the instructor’s plans
10
Determine Mode
Web•Cost effective•Survey large numbers•Automatic skip patterns and data input•Analysis capability•Confidentiality
Phone•Personal contact•More in-depth•Time intensive/costly•Opportunity to test concepts•Not anonymous
Paper•Immediate feedback•Requires data input•Mail and return mail costs•Confidentiality
In-person
•Provides connectivity and observation•Limited participants but more in-depth•Not anonymous
11
Budget, Time, Resources!
Determine Mode• Paper survey or mobile onsite, e-mail web• Give time to complete, collect if in class • Consider pretest/posttest and learning follow-up to
help assess impact
Course evaluations
• Can be cost effective, especially for large population
• Requires good communication to boost response• Best practice is to Include open-ended opportunity
for comments on improving
Satisfaction
•In-person, phone, web, paper•May use open-ended questions to get unknown/unanticipated responses
Memberassessment
12
Designing questions
13
1. Map questions to your purpose2. Have an action item for each question 3. Use easy-to-understand terms- not jargon or acronyms4. Put definitions first on complex questions, not after the
question5. Use words on scales (i.e. use Strongly Agree, Somewhat Agree
etc., rather than 1 to 5)6. Think about how the order of questions influence each other 7. Use forced-answer and required questions sparingly8. Be mindful of confidentiality, anonymity and social desirability
when asking questions9. Scale from negative to positive10. Ask yourself, will the audience know about this? Do I Have an
option for them to select “Don’t Know”, “Not Sure”, “Prefer not to answer”, etc.
Common Pitfalls in Question DesignThe double-
barreled question
• How would you rate the conference and training today?
• Poor,• Fair,• Good• Excellent
Avoid double negatives
• I would not attend a training on cybersecurity
• Yes• No
The loaded question
• How honest are you at work?
• Not very• Somewhat• Mostly• Always
The ambiguous question
• How would you rate the training?
• Poor• Fair• Good• Excellent
14
Ask about the conference and training separately
Be specific to better isolate what you’re measuring, i.e. trainer, content, etc.
Consider asking about comfort speaking openly
Remove ‘not’ or ask about level of interest
Resources for questionnairesCourse Evaluation Topics1. Extent content meets needs of
attendees2. Leader assessment3. Effective teaching approach4. Facilities5. Schedule6. Learning aids effective in improving
communication and maintaining interest
7. Coordination satisfactory8. Opportunities to improve program
TemplatesACC, AAPOR, CASRO and other professional groups, Survey Monkey, QuestionPro, Survey Gizmo have templates for various surveys: http://www.questionpro.com/survey-templates/association-member-survey/ 15
Strongly Disagree
Disagree
Neither
Agree
Strongly Agree
Not applicable
The material covered in the program was relevant to my job The materials was presented in an interesting way The instructor was an effective communicator The instructor was well prepared The audiovisual/technical aids were effective The materials will be helpful to me The facilities were suitable The schedule was suitable There was good balance between presence and group involvement I feel the workshop will help me do my job better What would improve the program?
Response 10% - 15% percent industry average
16
Web-surveys, 5 minutes or less is optimal, over 10 minutes is NOT.– Estimate roughly 11 to 12
seconds per question, 3-5 questions/min
– Roughly 15 questions– 1 to 2 open-ended
questions/min Paper surveys should be no more
than 10 questions, 15 if grid style or with
Phone interviews under 20 minutes can be completed without an appointment Schedule interviews of 30+
minutes
Boost response• Use ambassadors to build buy-in, let members know the survey
is coming and why (census example)• Consider incentives (monetary or in-kind, i.e. free event)• Clearly state the benefit of participating• Let members know how long the survey will take• Mobile optimize• Stay away from long grids on web surveys• Use multiple question pages but limit scrolling to avoid drop-
offs• Use four touch-points
17
Explain what you want, when the
survey will arrive, and reason to participate
Previte
Invite, provide link, assure
confidentiality, share benefits of participation
Invite
Call to action, create
personal link to the results,
remind of incentives
Remind and
thank
Final opportunity to be heard, why
you need participation
Remind and
thank
Incentives• Knowing the data will drive change, improvements or
address concerns is a big incentive, communicate this in pre-vite and invitation
• Share the report or write up results in a newsletter• Use monetary incentives if you have budget
– Studies conflict but small amounts are generally sufficient (little statistical difference between $5 and $20)
– Prepaid are most effective, promised incentives have limited impact.
– Charity donations– Web store (Amazon), local store or common chain
(Starbucks)• Check local laws if you want to have a drawing or
contest18
Generalizability and Coverage• Confidence Interval (how
confident you can be the data falls within the margin of error)
• Margin of error (+/- point or percentage)
• Estimate your sample neededhttp://americanresearchgroup.com/sams.html
http://www.raosoft.com/samplesize.html
http://
www.surveysystem.com/sscalc.htm
19
Confidentiality versus Anonymity: Be specific
Anonymity
• Researchers do not know the identity of the respondent and cannot connect data to individuals. No identifying data is recorded.
• Participants may feel more comfortable answering• Response may be higher• Researchers cannot track individual responses• Reminders go to the entire group as a thank you/reminder to
complete if one had not done so• Use general link to the survey
Confidentiality
• Researchers will not disclose personally identifiable information or share personal data at the individual level. Name, email and other contact variables are often known.
• Respondents must trust personal information is protected• Respondents may not feel comfortable sharing opinions on
sensitive topics• Ability to send respondent unique link makes outreach and
tracking response easier• Can preload demographics such as gender or birthdate
20
Web Surveys and Tools
• http://www.surveygizmo.com/
• http://freeonlinesurveys.com/login-form/
• www.surveymonkey.com
• www.Questionpro.com
Most online survey platforms have templates and free basic access.
21
Analysis
22
• Most online software programs have an analysis capability/reporting built into the platform
• Sample resource to help you work with data: http://aea365.org/blog/ • Only perform analysis that you understand- it doesn’t have to be a complex
analysis to be a good analysis• Distribution is important
• Mode is the answer that occurs most frequently• Mean or Average is the sum of responses divided by the number• Normal distribution is a bell curve
Example: Overall course rating = (40% very dissatisfied, 40% very satisfied) you would never use an average with this bi-modal distribution.
Qualitative Research: Map responses to your purpose, look for themes.Create theme words and code each response by theme to identify patterns.
Frequencies and Crosstabs Say A LotHave you created surveys at any time in the past or in your current role?
Yes No
Strongly disagree
0.0% 0.0%
Disagree 23.1% 0.0%
Neither agree nor disagree
38.5% 100%
Agree 23.1% 0.0%
Strongly agree
15.4% 0.0%
Not Applicable
0.0% 0.0%
100% 100%
23
I k
no
w w
he
re t
o l
oo
k f
or
info
rma
tio
n
wh
en
I n
ee
d h
elp
cre
ati
ng
a s
urv
ey.
Have you created surveys at any time in the past or in your current role?
Yes 92.9%
No 7.1%
1. Please describe your knowledge of survey research on a scale of 1 to 5, with 1 being no knowledge and 5 being an expert level of knowledge.
1 2 3 4 501234567
1
4
6
1 1
Value Percent Count
1 7.7% 1
2 30.8% 4
3 46.2% 6
4 7.7% 1
5 7.7% 1
Total 13
2. Have you created surveys at any time in the past or in your current role?
94
6
Yes No
Value Percent Count
Yes 93.8% 15
No 6.3% 1
Total 16
Please rate your level of agreement or disagreement with the following statement: I know where to look for information when I need help creating a survey.
25
44
19
13 Strongly disagree Disagree Neither agree nor disagree
Agree
Strongly agree Not Applicable
Value Percent Count
Strongly disagree 0.0% 0
Disagree 25.0% 4
Neither agree nor disagree 43.8% 7
Agree 18.8% 3
Strongly agree 12.5% 2
Not Applicable 0.0% 0
Total 16
ACC Resources and Strategy
• Membership• ACC Research
Planned for FY2015• Master calendar• Question Database• Survey resource portal for chapters
27