Design and AnalysisDesign and Analysis of Open Response of Open Response Surveys: Lessons Surveys: Lessons LearnedLearned
Dr. Joan Burtner Associate Professor of Industrial Engineering Mercer University
Design and Analysis of Design and Analysis of Open Response SurveysOpen Response Surveys
Designing Survey
Administering Survey
Conducting Survey Analysis
Reporting Survey Results
IIE/SHS FEB 2012
Mercer University School of Engineering Slide 2
Open Response SurveysOpen Response SurveysAsk participants to respond to a question such
as “What can we do to improve patient care?
Excellent method for collecting “Voice of the
Customer” data within a Quality Mgt. System
Allows analysts to interpret qualitative
characteristics (tone, frustration, pride)
Opportunity to discover unanticipated
responses
IIE/SHS FEB 2012
Mercer University School of Engineering Slide 3
Designing and Administering Designing and Administering the Surveythe Survey
Purpose of survey
Delivery method (paper, electronic, phone, in-
person, etc.)
Length of survey, allowable response time
Incentives for participation
Anonymity, confidentiality
Who will collate and/or analyze data?
How will data be analyzed?IIE/SHS FEB
2012Mercer University School of Engineering Slide 4
Design of Survey Design of Survey QuestionsQuestions
Structure, specificity of written questions
◦ What can we do to improve the process?
◦ What is the one thing that will make your job easier?
◦ What is the “one idea” that will improve the
registration process?
Text-based questionnaires vs. oral interviews
◦ Limited, prescribed, uniform questions
◦ Non-uniform follow on questions in addition to
prescribed questions
IIE/SHS FEB 2012
Mercer University School of Engineering Slide 5
Example 1: On-line Survey Example 1: On-line Survey Question Question Please describe challenges Please describe challenges that make it difficult to deliver patient that make it difficult to deliver patient
care. care. Over 700 responses submitted
anonymously over a two week period
Data downloaded into an Excel file
Many rows of data contained no text,
just time and date of submission
Two 2-person teams reviewed the data
and developed preliminary codesIIE/SHS FEB
2012Mercer University School of Engineering Slide 6
Example 1: Initial CodingExample 1: Initial CodingCoding done electronically by labeling
each response in Excel
Initial decision to allow only one major
code for each entry to simplify analysis.
Twenty-four categories emerged between
the two teams.
Overlap and clarification of labels resulted
in the elimination of four categories.IIE/SHS FEB
2012Mercer University School of Engineering Slide 7
Example 1: Revised Example 1: Revised CodingCoding
Difficult to gain consensus among all four
coders on the “one” main concept for
each response
Realization that multiple codes increased
validity of work
Excel spreadsheet expanded to four
columns to accommodate multiple codes
for each responseIIE/SHS FEB
2012Mercer University School of Engineering Slide 8
Code - Process Change?Code - Process Change?Patients should not be transferred to med surg
during change of shift. There needs to be a
window of time that patients cannot be
transferred. It is very unfair to the patient to
be transferred at that time as going off nurses
are trying to get caught up and give report;
and coming on nurses are trying to get report.
For example; no transfers between 6:00 and
07:15pm IIE/SHS FEB
2012Mercer University School of Engineering Slide 9
Code - Work Load? Policy?Code - Work Load? Policy? If a patient is computer literate, why do we not
have some of the admission forms on line so that
patients can fill out the questions at home? They
would certainly have access to their medications &
we might have a more exact list of what they take.
They would be in a more comfortable & less
stressful environment when filling out the forms &
the older folks would most likely have a family
member to assist in remembering when they had
previous procedures, etc. IIE/SHS FEB
2012Mercer University School of Engineering
Slide 10
Audience Participation Audience Participation ExerciseExercise
Form teams of two or three
Consider the following responses
(handout)
How would you code the data?
Do multiple codes increase
validity?
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 11
Example 2: Nursing Example 2: Nursing InterruptionsInterruptions
Research agreement with a local
hospital
Students observed nurses at work
Data collected orally and transcribed by
observers
Anecdotal reports from nurses collected
periodically as time permitted
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 12
Example 2: Nursing Example 2: Nursing Interruptions Data Interruptions Data Excerpt 1Excerpt 1
April 15th
Comments by Nurse - Interruptions
Helping with code lift or code response
Family
Telephone calls
Doctors
When technician needs help
Comments by Observer 1
Tech was using chart station when Nurse was trying to chart
Having room in another section causes a lot of extra walking
This could be a system failure interruptionIIE/SHS FEB
2012Mercer University School of Engineering
Slide 13
Example 2: Nursing Example 2: Nursing Interruptions Data Interruptions Data Excerpt 2Excerpt 2
April 16th
Comments by Nurses - Interruptions
Family
Lab Calls asking about results or talking about results
Doctors calling
Patients – getting called by other patients and helping other patients
Unexpected events – Patients getting out of bed, codes
Helping nurses, covering nurses when they are out for lunch
Comments by Observer 2
Quiet day with not many interruptions
Nurse didn’t have much to doIIE/SHS FEB
2012Mercer University School of Engineering
Slide 14
Coding Nursing Coding Nursing InterruptionsInterruptions
Nature of the interruption
◦Phone call , call light, patient’s
family, healthcare professional, etc.
Urgency of the interruption
◦Avoidable, justifiable, etc.
Consequence of the interruption
◦Delay, error, etc.IIE/SHS FEB
2012Mercer University School of Engineering
Slide 15
Example 3: On-line Survey Example 3: On-line Survey Question Question Please list and describe Please list and describe
ideas that will contribute to cost efficiency ideas that will contribute to cost efficiency within our system.within our system.
Over one hundred responses submitted
Data downloaded into an Excel file
Single person reviewed the data and
categorized each response
Fifty-eight usable responses
Data organized into a pivot table
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 16
Reporting Analysis ResultsReporting Analysis ResultsExample 1
◦ Ten-page written report◦ Power-Point presentation to five-person
committee◦ Follow-up submission: frequency distribution
models overall and departmentalizedExample 2
◦ In-person feedback to nurse manager◦ Brief report with Pareto diagrams
Example 3◦ Electronic feedback to client◦ Pivot table demonstration
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 17
Code Frequency - All Code Frequency - All LocationsLocations
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 18
Frequency - AberdeenFrequency - Aberdeen
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 19
Frequency - DallasFrequency - Dallas
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 20
Frequency - GreenvilleFrequency - Greenville
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 21
Example 3: Pivot Table 1Example 3: Pivot Table 1
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 22
Example 3: Pivot Table 2Example 3: Pivot Table 2
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 23
Example 3: Pivot Table 3Example 3: Pivot Table 3
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 24
Lessons Learned 1Lessons Learned 1Nurses are willing to spend time
completing surveys.
Many are eager to have their ideas
heard.
It is important to give feedback, when
possible, with respect to how the survey
results may change the process.
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 25
Lessons Learned 2Lessons Learned 2The process of analyzing and quantifying
open-ended responses is very time-
consuming.
It is very difficult to gain consensus on some
responses.
System-wide data should also be reported at
the unit level if possible.
In the case of “one best idea” it is difficult to
prioritize.IIE/SHS FEB
2012Mercer University School of Engineering
Slide 26
Lessons Learned 3Lessons Learned 3Due to the somewhat subjective nature of
the coding and frequency tabulation, it
may be difficult to use open-ended survey
responses to measure the effectiveness of
process changes.
However, changes in the culture and
attitudes of the respondents may be
apparent.IIE/SHS FEB
2012Mercer University School of Engineering
Slide 27
Questions / Contact Questions / Contact InformationInformation
Dr. Joan Burtner [email protected]
Associate Professor of Industrial
Engineering and Industrial Management
(478) 301- 4127
Mercer University School of Engineering
1400 Coleman Avenue
Macon, GA 31207
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 28