abcs of satisfaction & impact surveys elizabeth harris, ph.d. evaluation, management &...
TRANSCRIPT
![Page 1: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/1.jpg)
ABCs of Satisfaction & ABCs of Satisfaction & Impact Surveys Impact Surveys
Elizabeth Harris, Ph.D.Elizabeth Harris, Ph.D.Evaluation, Management & Training Evaluation, Management & Training
Associates, Inc.Associates, Inc.
May 21May 21stst , 2008 , 2008
![Page 2: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/2.jpg)
ABCs of Satisfaction & Impact ABCs of Satisfaction & Impact SurveysSurveys
ObjectivesObjectives– Understand the importance and relevance of science-Understand the importance and relevance of science-
based surveyingbased surveying– Understand method of developing telephone and Understand method of developing telephone and
Internet surveys to measure inquirer satisfaction and Internet surveys to measure inquirer satisfaction and key outcomes important to contact centerskey outcomes important to contact centers
– Leave the workshop with the tools necessary to Leave the workshop with the tools necessary to develop surveys tailored to the needs of each contact develop surveys tailored to the needs of each contact center center
Workshop exerciseWorkshop exercise
![Page 3: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/3.jpg)
““Lessons LearnedLessons Learned” ” from surveying for: from surveying for:
Centers for Disease Control & PreventionCenters for Disease Control & Prevention– Consumer Response Services Center (CDC-INFO)Consumer Response Services Center (CDC-INFO)– Monitor calls & e-mail responses Monitor calls & e-mail responses
First 5 LAFirst 5 LA– Countywide Parent Helpline Countywide Parent Helpline
![Page 4: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/4.jpg)
We will discuss:We will discuss:
Survey: Survey: – DesignDesign
To Do’sTo Do’s
Common PitfallsCommon Pitfalls
– Administration Administration OptionsOptions
– Data UseData Use
![Page 5: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/5.jpg)
Satisfaction & Impact Satisfaction & Impact
Satisfaction:Satisfaction:– ImmediateImmediate– Caller’s opinion about Caller’s opinion about
Overall ServiceOverall Service
Various Aspects of the Service (Drill Down)Various Aspects of the Service (Drill Down)
Impact:Impact:– What happened as a result of the call?What happened as a result of the call?
![Page 6: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/6.jpg)
Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 1Step 1
What is important to us when we consider What is important to us when we consider the caller’s perspective and experience? the caller’s perspective and experience?
What questions are our key stakeholders What questions are our key stakeholders asking that we need to answer? asking that we need to answer?
![Page 7: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/7.jpg)
Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 2Step 2
Determine method of administration Determine method of administration – Interactive Voice ResponseInteractive Voice Response– Live SurveyorLive Surveyor– Web-Based Web-Based
FeasibilityFeasibility
Balance between cost & knowledge gainBalance between cost & knowledge gain
![Page 8: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/8.jpg)
Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 3 Step 3
Develop questions Develop questions Response options Response options – ClosedClosed
Yes/NoYes/NoLikert ScaleLikert Scale
– e.g. Very Satisfied to Very Dissatisfied, Press 1 for Very e.g. Very Satisfied to Very Dissatisfied, Press 1 for Very SatisfiedSatisfied
– Allow a middle option? Neither Satisfied or Dissatisfied Allow a middle option? Neither Satisfied or Dissatisfied
– Open Open ProPro
– Information Information
ConCon– Labor considerationLabor consideration
![Page 9: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/9.jpg)
Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 3Step 3
Develop questions Develop questions Handout example of closed and open-Handout example of closed and open-ended questions ended questions
![Page 10: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/10.jpg)
Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 4Step 4
Design Survey Design Survey – Question wordingQuestion wording
NeutralNeutral
– Response order effect Response order effect Carefully consider which question is first and which Carefully consider which question is first and which is last is last
![Page 11: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/11.jpg)
Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 4Step 4
Design SurveyDesign Survey– IVR RulesIVR Rules
Brief and to the pointBrief and to the point
No more than six response optionsNo more than six response options
– Consider lengthConsider lengthLonger = more respondents drop off Longer = more respondents drop off
– Language options Language options
![Page 12: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/12.jpg)
Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 5Step 5
Determine Survey ProtocolDetermine Survey Protocol– Who is surveyed-Immediate SatisfactionWho is surveyed-Immediate Satisfaction
Every callerEvery caller– Works well if no automated transfer optionWorks well if no automated transfer option
TrainingTraining
BiasBias
Random selectionRandom selection– Works well with a robust systemWorks well with a robust system
program algorithm that automatically chooses program algorithm that automatically chooses
![Page 13: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/13.jpg)
Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 5Step 5
Who is surveyed—Impact (Call Back) Who is surveyed—Impact (Call Back) – Again consider:Again consider:
Purpose of the surveyPurpose of the survey– Caller preferenceCaller preference– Part of service follow up effortPart of service follow up effort
How the information will be used How the information will be used
– Other considerations:Other considerations:Labor allocation/resourcesLabor allocation/resources
– To surveyTo survey– To interpret/make use of information To interpret/make use of information
![Page 14: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/14.jpg)
Satisfaction & Impact Surveys: Satisfaction & Impact Surveys: Step 6Step 6
Pilot Testing Pilot Testing – Works out the bugsWorks out the bugs– Improves introductory instructionsImproves introductory instructions– May revise questions and responses based May revise questions and responses based
on pilot test results on pilot test results
![Page 15: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/15.jpg)
What to Expect: Response Rate What to Expect: Response Rate
Immediate Satisfaction SurveyImmediate Satisfaction Survey– 22% to 33%22% to 33%
Follow Up SurveyFollow Up Survey– Will reach half of those you try to callWill reach half of those you try to call
If you try more than onceIf you try more than once
Vary time/day called Vary time/day called
![Page 16: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/16.jpg)
Results: Results:
Over 90% give the First 5 LA Parent Helpline the Over 90% give the First 5 LA Parent Helpline the highest satisfaction rating for overall service!highest satisfaction rating for overall service!
Over half who received a referral successfully Over half who received a referral successfully accessed servicesaccessed services
![Page 17: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/17.jpg)
Results: Results:
Over 75% give CDC-INFO the highest satisfaction rating for Over 75% give CDC-INFO the highest satisfaction rating for overall serviceoverall service
Over 75% report learning something new from their callOver 75% report learning something new from their call
Over half report that what they learned makes them want to Over half report that what they learned makes them want to change a behavior change a behavior
![Page 18: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/18.jpg)
Questions? Questions?
![Page 19: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/19.jpg)
Exercise: Exercise:
See handoutsSee handouts
Select one question you would like to answer Select one question you would like to answer with a satisfaction survey with a satisfaction survey
![Page 20: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/20.jpg)
Exercise—Break Out: Exercise—Break Out:
Develop a survey question and responses:Develop a survey question and responses:– Doable/feasible to implement in your setting Doable/feasible to implement in your setting
![Page 21: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/21.jpg)
Exercise—Debriefing: Exercise—Debriefing:
Share examples from each group Share examples from each group
![Page 22: ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514ddca550346935c8b54be/html5/thumbnails/22.jpg)
Questions?Questions?
Contact information:Contact information:
Elizabeth Harris, Ph.D.Elizabeth Harris, Ph.D.
818-990-8301818-990-8301
[email protected]@emt.org