common measurements tool client feedback workshop

71
Common Measurements Tool Client Feedback Workshop

Upload: sabastian-faulks

Post on 29-Mar-2015

216 views

Category:

Documents


2 download

TRANSCRIPT

  • Slide 1

Slide 2 Common Measurements Tool Client Feedback Workshop Slide 3 Introductions Welcome Introductions Objectives Agenda Slide 4 Objectives Learn why,when,how to use CMT Understand basics of survey research Survey operationalisation, question design, sampling, administration, analysis 101 Produce 2 draft surveys (Homecare, Emergency Care) Produce action plans for implementation Slide 5 Resources To assist you additional resource material has been provided in your binder CMT Managers Guide Customer Feedback and Customer Satisfaction Measurement - EPA Detailed Handouts, Articles, Case Studies Slide 6 Why Customer Feedback? Performance measurement is basic element of management. Management functions Plan Organize Lead Control (Measurement) Slide 7 Managements Contribution To coordinate the behaviour of: individuals groups organizations To attain: individual effectiveness organizational effectiveness Management performs the functions of: planning organizing leading controlling Feedback Slide 8 Customer Service Model Satisfied Customers Principles Systems, Processes Enabling Environment Performance Measurement Team-Oriented Customer-Focused Manage by Fact Continuous Improvement Leadership Customer Feedback Slide 9 What level of service should you get from government, compared to the private sector? Percent of respondents Governments have a more difficult task than the private sector. EXPECTATIONS OF SERVICE QUALITY Percent of respondents Slide 10 Implement, Evaluate, Repeat Develop and implement action strategies, test outcomes Listen to Customers Analyze Results Implement Changes Slide 11 Customer Feedback Framework Plan Construct Conduct Analyze Act Slide 12 Step 1. Plan the Customer Feedback Project Define objectives of feedback What do I want to accomplish with this feedback? Why am I conducting this feedback activity Determine how findings will be used What will we do with these findings Will they be used for performance indicator to improve a process to ID customer needs/expectation planning, decision making Slide 13 What is most appropriate feedback technique? Survey is often not the best technique Options include Focus Groups In Depth Interviews Customer Advisory Panels Employee Observations Performance Audits Customer Site Visits Mystery Shoppers Benchmarking surveys see handbook for description of strengths and weaknesses Slide 14 Strengths/Weaknesses Employee Observations Customer feedback, complaints Service audits Internal customers Walk-throughs Focus groups, interviews Surveys Slide 15 Two key questions What information do you need? Who (which segment of the population) can provide you with this information? Internal vs external customers Demographic segmentation (age, income, geography Program function Slide 16 Exercise # 1 In 2 groups answer the following questions What do you want to know? What technique would you recommend? Who would ask to provide this information? In 2 groups answer the following questions What do you want to know? What technique would you recommend? Who would ask to provide this information? Slide 17 FOI/FIPPA Major Concerns include privacy notification approval etc. Major Concerns include privacy notification approval etc. Slide 18 Warning: How ready is your organization for customer feedback? Do managers and staff sincerely intend to pay attention and take action? Are key managers committed to action? Have staff members participated directly in defining needs and approaches Are there possible barriers to using feedback effectively ( concerns re: workload, negative findings, change ) Slide 19 Top Reasons for Failure Failure to involve the right person in developing customer service measures customers sponsors owners,users Failure to capture data that leads to action Scorekeeping vs.understanding Failure to link customer data to operational, quality, or performance data Absence of closed- loop process for measuring and managing satisfaction Brandt,D.R. Build Actionability Into Your Customer Surveys 1995 Slide 20 Customer Feedback Framework Plan Construct Conduct Analyze Act Slide 21 Step 2.Constructing the Survey Types of Questions Dos and Donts Scales Ordering Number of Questions Layout Types of Questions Dos and Donts Scales Ordering Number of Questions Layout Slide 22 Phase 2 Constructing your survey Operationalizing your research questions What kinds of questions will get you the information you are looking for This crucial step is often overlooked Examples The Common Measurements Tool Slide 23 Common Measurements Tool Objectives: Provide a ready-made tool for public service organizations Enable like-organizations to compare results Enable organizations to build internal benchmarks Facilitate the sharing of information gained and lessons learned between organizations Slide 24 Elements of the Service Experience... Client expectations Client perceptions of service experience Level of satisfaction Level of importance Priorities for improvements Slide 25 CMT - Service Dimensions Responsiveness Timely delivery of service # of contacts to receive service Waiting time Courtesy of service staff Reliability What was promised was delivered Errors detected and corrected Access & Facilities Hours of service Telephone access Communications Slide 26 Slide 27 Question Types Open ended questions allows respondents to provide his or her own answer e.g.. What one thing could we do to improve service? Closed ended questions limit the response an individual can make e.g. (likert scale) CMT includes both types Slide 28 Closed Ended Pros Questions are relatively easy to complete Easy to analyze Unlikely to receive irrelevant or unintelligible responses Pros Questions are relatively easy to complete Easy to analyze Unlikely to receive irrelevant or unintelligible responses Cons no knowledge may over simplify Risk missing the boat: For example if you asked what is the main reason for your visit? and 30% respond other. This is a good reason to pre-test questions to ensure you include all the likeliest responses Slide 29 Open ended Pros will receive more information, often richer (intensity, more specific) more likely to receive surprise insights Pros will receive more information, often richer (intensity, more specific) more likely to receive surprise insights Cons difficult to analyze and code responses may be irrelevant or unintelligible demanding and time consuming for respondent Slide 30 Guidelines Remember you want actionable info what action could you take with this information Avoid questions which are too vague, subject to misinterpretation double barreled questions which imbed more than one item on a scale of 1-6 please indicate how clear and useful the materials are? extraneous or irrelevant sensitive Slide 31 Avoid questions that do not elicit responses which point to specific remedies e.g. How satisfied are you with the quality of service provided Very satisfied Satisfied Neutral Dissatisfied Very Dissatisfied which mix scales e.g. were you checked in within five minutes satisifed neutral dissatisfied Slide 32 Avoid questions which include terminology, jargon etc. which includes words and terminology which is too complex e.g. Are you aware of the impending amalgamation of surrounding constituencies into the New Metro catchement region? with unclear frame of reference e.g. what is your income? your individual family, household? Time reference bi-weekly, monthly, annual Income - salary, tips or other sources Slide 33 Impact of question wording See hand out for examples Implied alternative Slide 34 Considerations in wording Are the words simple, direct, and familiar to all respondents? Are the questions as clear and specific as possible? Are the questions applicable to all respondents Are the categories mutually exclusive and exhaustive Are any questions leading or loaded? Slide 35 Scale types Likert scale 3,5,7,9 point Thurston Slide 36 Constructing Questionnaire Format Introduction - sets forth the purpose of survey and guide respondent through questions Customer experience - establishes customers level of knowledge regarding various parts of questionnaire Measurement - CMT asks customer to characterize experience, expectations, needs, priorities Demographic information - classify Slide 37 Ordering of Questions Order of questions should encourage respondents to complete questionnaire facilitate respondents recall appear sensible focus on the topic follow a sequence that is logical to the respondents follow smoothly from one to next Slide 38 Sensitive Questions Introduce where respondent has developed confidence and trust locate in section where they have most meaning and context introduce slowly Slide 39 Number of Questions Be careful not to ask too many questions CMT (100+) Do not recommend asking all 11 core questions get 5 drivers,access, communication and cost (see list in book) Slide 40 Timing of Feedback When you solicit feedback is very important too long after original event will lower response rate and relevancy. Be cautious re: decisions, administrative process etc. Slide 41 Exercise # 2 Group Exercise Using CMT as a guide each group will be given 1 hour to develop a draft survey based on first exercise. Each group must include a minimum of 5 CMT questions and 5 Custom questions Slide 42 Sampling Census Vs. random What size sample do I need? Depends on use of survey Trade offs Confidence Interval and sampling error How to calculate sample size Why 400 is the magic # (see handouts) Slide 43 On-line tools Slide 44 Pre-testing Questions Where possible pre- test your instrument. Ensure language is clear Have you missed any important items Testing Methods Focus Groups/Interviews Ask test respondents to talk out loud Remember Pre-testing is extremely valuable, if you consider how much time and effort you have already committed to customer feedback Slide 45 Agenda Day 2 Review of Draft Surveys - Discussion Review of CMT Conducting Surveys (Who,When, Where, How) Analysis Action Slide 46 Citizen-Centered Network A network of Federal/Provincial/Municipal civil servants dedicated to improving the quality of public service in Canada Citizens First Erin Research Inc THE CITIZEN-CENTRED Common Measurements Tool Faye Schmidt for the ANAGEMENT DEVELOPMENT IPAC 1999 Gold Award Slide 47 Citizens First Five Drivers of Satisfaction 1. Timeliness 2. Competence 3. Courtesy 4. Fairness 5. Outcome Slide 48 Customers Priorities for Improving Access to Government Services Making government services easier to locate in the phone book Improving telephone service Expanding electronic access Expanding hours of service Standards for routine services Access a major weak spot 29% citizens cant find service 89% experience at least 1 problem Access a major weak spot 29% citizens cant find service 89% experience at least 1 problem Slide 49 Common Measurements Tool Objectives: Provide a ready-made tool for public service organizations Enable like-organizations to compare results Enable organizations to build internal benchmarks Facilitate the sharing of information gained and lessons learned between organizations Slide 50 Elements of the Service Experience... Client expectations Client perceptions of service experience Level of satisfaction Level of importance Priorities for improvements Slide 51 CMT - Service Dimensions Responsiveness Timely delivery of service # of contacts to receive service Waiting time Courtesy of service staff Reliability What was promised was delivered Errors detected and corrected Access & Facilities Hours of service Telephone access Communications Slide 52 The CMT Go to tool Slide 53 Customer Feedback Framework Plan Construct Conduct Analyze Act Slide 54 Step 3. Conducting Surveys Will Focus on Mail Surveys (80-90%) How to achieve a high response rate Data collection and coding Data entry options CMT Data Entry options Slide 55 Mail Surveys Mail surveys can achieve response rate of 60% Begin with good data base unique id #, relevant customer characteristics, mailing address, mail out dates and date responses receive Professionalism in all Materials questionnaire, cover letter, return envelope with self addressed stamp envelope; all professional look Slide 56 Ensuring high response rate Include cover letter; explaining purpose of survey ( print on official letter head ) Booklet format recommended ( see sample ) Use stamp and hand written address Objective is coffee table adhesion factor Follow up letter is a must Generates an additional 50% response Slide 57 Other tips Avoid mailing near holidays use hot-line or contact # use admin # and explain this is to avoid multiple contacts 1st Mail out 2nd Mail out 3rd Mail out Slide 58 Telephone Surveys Very common form of survey administration Advantages are: faster offers more control Call response ratio 10:1 CMT provides Computer Assisted Telephone Interview option Slide 59 Internet Surveys Advantages fast, cheap, automated data collection/analysis Disadvantages susceptible to sample biases too easy to administer security/privacy concerns More info available on request Slide 60 Data entry options CMT External data entry contact Service 1st Manual entry options On-line survey Data entry spreadsheet sheet Bar coding Slide 61 Customer Feedback Framework Plan Construct Conduct Analyze Act Slide 62 Analysis Quantitative Frequencies, percentiles, percentages Central tendencies (mean, mode, median) Cross tabulations that array independent variables against dependent variables e.g. customer type who were satisfied Quantitative Frequencies, percentiles, percentages Central tendencies (mean, mode, median) Cross tabulations that array independent variables against dependent variables e.g. customer type who were satisfied Cross tab example Slide 63 7 Tools of Quality Cause and effect diagram Check Sheets Control charts Flow charts Histogram Pareto charts Scatter Diagram Cause and effect diagram Check Sheets Control charts Flow charts Histogram Pareto charts Scatter Diagram See Handout Slide 64 CMT Analysis For each question Bar/Pie charts Frequencies Percentages For each section Satisfaction/Importance Actual Vs. Expected Priorities for Improvement Service Standards For each question Bar/Pie charts Frequencies Percentages For each section Satisfaction/Importance Actual Vs. Expected Priorities for Improvement Service Standards Slide 65 Exercise 3 In groups of 4 review the Case Study What are the main findings? What are the clients priorities for improvement? What further analysis do you recommend? How would you present the results to executive management? 30 Minutes Slide 66 Customer Feedback Framework Plan Construct Conduct Analyze Act Slide 67 Acting on Results What to do with the feedback? Step 1. Assess what the feedback is telling you. CMT for example allows you to evaluate 5 drivers of satisfaction, estimate service standards, understand relationship b/w satisfaction and importance and ID priorities for improvement. Slide 68 Acting on Results How good is good enough? Satisfied vs.. Very satisfied 4.0/5,0 vs. 4.5/5.0 Nature of organization effects rating How good is good enough? Satisfied vs.. Very satisfied 4.0/5,0 vs. 4.5/5.0 Nature of organization effects rating Slide 69 Action Steps Recover Report Brief Prioritize Communicate Improve Enhance Reward Plan Feed into Strategic/Business Planning Adapted from Customer Service Guidelines EPA Slide 70 Defining Action Paths Process involves identifying owners for each issue or area targeted for improvement Getting owners to own readiness for action linking issue to relevant processes, systems using learning achieved above to define next steps See Brandt Slide 71 Satisfaction and Problem Tracking LOGGING Problem description Action required CLASSIFICATION Bridge to Analysis ANALYSIS Statistics Diagnostics Policy PREVENTION Training/Staffing Process Improvement Innovation INVESTIGATION FORMULATION RESPONSE AND FOLLOW-UP REFERRAL OR SPECIAL HANDLING Technical Assistance Research Programs (TARP) Slide 72 Organizational Readiness DO WE UNDERSTAND WHAT CUSTOMER IS TELLING US? GATHER ADDITIONAL INFORMATION DEVELOP AND IMPLEMENT SOLUTION BEGIN PROCESS PERFORMANCE ANALYSIS ALIGN TARGET/SOLUTION WITH CUSTOMER EXPECTATIONS IS OUR TARGET ALIGNED WITH CUSTOMER EXPECTATIONS? IS THE EXISITNG METHOD WORKING? DO WE HAVE A METHOD OF ADRESSING THE ISSUE? NO Slide 73 Wrap - up Review Objectives Tools and References