SurveyHandbook

o þþ oo þo þþ o

Archester Houston, Ph.D.Department of the Navy

Total Quality Leadership Office

About the TQLOThe mission of the Department of the Navy (DON) Total QualityLeadership Office (TQLO) is to assist the DON leaders in their quality-focused improvement efforts. The TQLO also provides technicaladvice to organizations inside and outside government. The TQLOhas responsibilities in six key areas:

Information and CommunicationThe TQLO educates the DON about TQL policies and initiativesthrough the TQLeader and through articles and presentations atconferences. It has developed an Internet site to facilitate com-munication.

AssessmentSystems are needed to assess and enhance TQL implementationin the DON. The TQLO designs and develops feedback mecha-nisms in support of mission accomplishment. It also developsnew approaches to improving organizational effectiveness.

Consultant ServicesTQLO members provide technical advice to the Under Secretaryof the Navy and other senior DON leaders on the application ofTQL principles and methods within the DON and on strategicplanning. Advice may also take the form of recommendations onimplementing new laws, such as the Government Performanceand Results Act, as well as on related initiatives.

Education and TrainingThe TQLO is responsible for ensuring the technical accuracy ofthe DON TQL curriculum. Having overseen the design anddevelopment of the courses, the staff now advises on the integra-tion of TQL material into training pipelines. The TQLO continuesto publish handbooks and other materials on organizationalchange and to design new courses.

Networking LiaisonThe TQLO has much to share with other organizations, govern-ment and private, and much to learn from them. Staff membersparticipate in TQL-related networks and organizations.

New TechnologiesTechnology can provide critical support to DON quality improve-ment efforts. The job of the TQLO is to assess new technologiesrelated to organizational change and process improvement andtranslate them into applications for the DON.

SurveyHandbook

o þþ oo þo þþ o

Archester Houston, Ph.D.Department of the Navy

Total Quality Leadership Office

4 Survey Handbook

5Survey Handbook

o þþ oo þo þ

Foreword

The Survey Handbook was designed as abasic reference for organizations collectinginformation through surveys to support improve-ment efforts. This handbook primarily providesassistance on conducting customer satisfactionsurveys and employee attitude surveys. Itcovers such topics as: constructing surveyitems, writing instructions, survey administra-tion, and presentation of results.

The Handbook will not make anyone an expertin surveys. References are provided for thosewho want to learn more, and recommendationsare included as to when to seek the advice ofan expert in survey design and analysis. Thebest advice, however, is to understand thepurpose of a survey and how the results will beused.

The Survey Handbook is also available fromthe Total Quality Leadership Office through theInternet: www.tql-navy.org. Please direct any

6 Survey Handbook

questions you have about it to Dr. ArchesterHouston through the Internet at:Houston.Archester@hq.navy.mil.

Linda M. Doherty, Ph.D.Director, Total Quality Leadership Office

Department of the Navy

7Survey Handbook

o þþ oo þo þ

Foreword .............................................................................. 5

What is a Survey? .................................................................. 9

When Should I Use a Survey? ............................................. 11

Preparing for a Survey ....................................................... 13

What is the purpose of the survey?.................................... 13Who will use the results? ...................................................... 15What specific information is needed? ............................... 15Who will be responsible for developing the survey? ......... 16Who will be surveyed? .......................................................... 16How will the survey be administered? ................................ 16What resources will be needed? ......................................... 17What survey items will be used? .......................................... 17How will survey information be analyzed and reported? 18How many people need to respond to the survey? ......... 19What are the steps in using a survey? ................................ 19

Constructing Survey Items ................................................ 21

Selecting Response Formats.............................................. 23

Rating scales ........................................................................ 23Ranking items ....................................................................... 25Selecting options ................................................................ 26Comments and open-ended questions ............................ 26Demographic questions ...................................................... 27Reviewing Items..................................................................... 31

8 Survey Handbook

Survey Introduction and Instructions .............................. 35

Pilot Testing a Survey ......................................................... 37

Explain the process. ............................................................ 37Request written comments. ............................................... 37Measure time requirements. ............................................... 38Hold a group discussion. .................................................... 38

Administering the Survey ................................................... 39

Mail-out surveys. ................................................................. 39In-person surveys. ............................................................... 40

Summarizing the Survey Results ....................................... 41

Frequency distributions. .................................................... 41Percentages. ........................................................................ 43A note of caution ................................................................ 43

Sampling ............................................................................. 45

Communicating Survey Results ......................................... 47

Conclusion ........................................................................... 49

Resources on Surveys ......................................................... 51

9Survey Handbook

o þþ oo þo þ

What is a Survey?

A survey is a systematic method of collectinginformation from a selected group of people byasking a series of questions. Surveys can beused to collect various types of information.They can collect information on people’s be-haviors, job performance, knowledge, prefer-ences, attitudes, beliefs, feelings, etc. For asurvey to provide useful information, care mustbe taken in its development and use for anumber of reasons, including the cost in termsof time and money.

This handbook provides you with an overviewof developing and conducting a survey. It alsooffers tips for writing or modifying survey itemsto meet your special needs. However, it will notmake you an expert in survey development. It isstrongly advised that you obtain professionalassistance when possible, especially in surveydesign and statistical analysis.

10 Survey Handbook

11Survey Handbook

o þþ oo þo þ

When Should I Use a Survey?

Although surveys are a popular method ofcollecting data, they must be used under theappropriate conditions.

Consider using a survey when it is faster,easier, or less expensive to use than othermethods. Sometimes other data collectionmethods are preferable. For example, to deter-mine the number of people using a clinic, youcan simply count the number of signatures onthe sign-in sheet, or examine the daily records,rather than conduct a survey to obtain thisinformation.

Consider using a survey when the informa-tion does not already exist in some form.Checking whether relevant and accurate dataexist in archives, records, or databases cansave a great deal of time, money, and effort.For example, before asking employees thenames and dates of each course taken withinthe past year, consult their training files toobtain this information.

12 Survey Handbook

13Survey Handbook

o þþ oo þo þ

Preparing for a Survey

A number of issues should be resolved beforedrafting the first survey item. Use the followingquestions to help you in preparing for a survey.

What is the purpose of the survey?

Surveys can be used for many purposes,including:

þ determining customer needs/assessingcustomer satisfaction.

þ identifying organizational strengths andweaknesses.

þ targeting areas needing improvement.

þ assessing the effectiveness of new orexisting policies or programs.

14 Survey Handbook

Without a clear purpose it is likely that thesurvey effort will flounder. This can result inwasted resources, useless data, and disap-pointment on the part of those who initiated thesurvey and those who responded to it.

Be very careful about conducting a survey “justto see what’s going on.” Conducting a surveycan draw attention to an organization from thepeople who receive the survey and from thosewho learn that it is being conducted. Thisattention can lead to expectations that execu-tive actions will soon follow.

Survey information can be valuable in guidingimprovement efforts at the process, organiza-tional, and systems level. Information on plan-ning and conducting improvement efforts canbe found in a variety of DON Total QualityLeadership publications, including:

A Handbook for Strategic Planning (Wellsand Doherty, 1994).

Strategic Management for Senior Leaders:A Handbook for Implementation (Wells,1996).

A Total Quality Leadership Process Im-provement Model (Houston and Dockstader,1993).

The Process Improvement Notebook(Culbertson et. al., 1997).

15Survey Handbook

Use these publications to assist you in deter-mining and clarifying the information needs thatwould require a survey. The publications canbe downloaded from the TQLO Web site atwww.tql-navy.org

Who will use the results?

Identify who will be using the results of thesurvey and what their information needs are.What types of decisions are they going to makebased on the results? How should the informa-tion be sorted for them? Do they need detailedresponses or is the “big picture” sufficient? Howdo they prefer to have information presented tothem?

What specific information is needed?

To meet the purpose of the survey, identify thetopics or issues of interest and the forms ofinformation needed. For example, if you areinterested in maintaining equipment at anathletic center, then you might ask questionsabout how often people use the center, whatfacilities they use, and hours of use. You mightalso ask people to rate the quality of the swim-ming pool, treadmills, exercise machines, showerarea, towel service, etc. If you are interested inmeeting future needs, you might ask people toidentify their anticipated physical fitness needsor interests. Each of these topics could be thesubject of one or more survey items.

16 Survey Handbook

Who will be responsible for developing thesurvey?

Surveys are often developed by small teams.When considering membership, include indi-viduals who have some knowledge of thetopics or issues of interest. Also try to includeindividuals who have experience with planningtools or graphic methods. If possible, have amember with experience in survey develop-ment and statistical analysis.

Who will be surveyed?

Identify the types of people who can provide theinformation you are interested in. Do theybelong to a single rank or organizational levelor do they come from a variety of ranks orlevels? Do they work in the organization or arethey end-users of the organization’s products orservices?

How will the survey be administered?

There are several ways to conduct surveys.Examples include face-to-face interviews,telephone interviews, and written surveys. Thelatter include those conducted by mail, com-puter, and group sessions.

Select the method that will provide sufficientinformation as efficiently as possible. Inter-views can provide very detailed information,

17Survey Handbook

but they require special training to conducteffectively, can be time-consuming, and mightbe difficult to analyze and interpret. Interviewscan be useful when you need to collect detailedinformation from a relatively small group ofpeople. Interviews can be used to exploreissues and options to a greater extent thanwritten surveys. Written surveys typically pro-vide less detailed information, but can be usedto collect large amounts of information in arelatively short period of time.

What resources will be needed?

Identify the staff and other resources requiredfor the survey. Depending on the type of surveybeing conducted, resources can include aproject manager, trained interviewers, statisti-cians, data analysts, printing costs, mailingcosts, data entry, and data processing. Keep inmind that the people who respond to the surveyshould be considered a resource in terms ofnumber involved, time invested, and informa-tion provided.

What survey items will be used?

Based on the information requirements, deter-mine if you can use an existing survey, modifyan existing survey, or if you need to develop anentirely new survey. (Developing survey itemswill be covered later.)

18 Survey Handbook

How will survey information be analyzed andreported?

Decide what methods will be used to summa-rize, interpret, and communicate the surveyresults. Data can be summarized with methodsas simple as tallies, percentages, or averages.Depending on the decisions that are going tobe made based on the survey information,more complex analytic methods might beappropriate. These more complex methodscould involve analysis of variance to comparethe results from different groups such as man-agers and employees or regression analysis tocheck the relationship between two variablessuch as job satisfaction and intent to leave theorganization.

Decide how you will display the survey infor-mation. This decision should be influenced bythe interests and preferences of the audience.Do they prefer simple graphs, statistical charts,or text summaries? The presentation format ofresults to executives who are consideringpolicy changes might be different from thatoffered to external customers. If you need topresent the results to different groups, considertailoring the level of detail and data displaymethods to the different levels of interest.

19Survey Handbook

How many people need to respond to thesurvey?

Due to limited resources, it might not be fea-sible to collect survey information from everymember of a group you’re interested in; forexample, it would be very difficult and costly tosurvey every member of the Department of theNavy. Sampling is a technique used to deter-mine the size and types of group subsetsneeded to provide representative informationabout the larger group. Obtain the help ofsomeone skilled in sampling techniques whenyou try to determine how many survey respon-dents you need. Inappropriate samples canlead to misleading results, inaccurate interpre-tations, and ineffective actions.

What are the steps in using a survey?

If you have addressed the previous questionsand decided that conducting a survey would beuseful and feasible, then here are a set ofactivities to assist your effort. They include:constructing survey items, selecting responseformats, reviewing items, creating a surveyintroduction and instructions, pilot testing asurvey, administering a survey, summarizingsurvey results, sampling, and communicatingresults. These activities can be helpful whetheryou are creating a completely new survey ormodifying an existing one.

20 Survey Handbook

21Survey Handbook

o þþ oo þo þ

Constructing Survey Items

Write survey questions for each topic of inter-est, jotting down whatever comes to mind. Atthe beginning it is better to have too many itemsthan too few. Do not focus on writing the perfectquestion. It is more important at the beginning tocover all of the topics than to be elegant inphrasing. Consider using brainstorming, treediagrams, and other idea-generation methods tocreate survey items. Individuals who haveattended the DON TQL Methods for ManagingQuality course should be familiar with them.Information on these methods can also be foundin Memory Jogger II (Brassard & Ritter, 1994).

Try to create items that are:

þ clearly written.

Statements should be short and easy to read.Do not use jargon, technical terms, or acro-nyms that are unfamiliar to the people answer-ing the survey.

22 Survey Handbook

þ concise.

Get to the point as quickly as possible. Elimi-nate extraneous words or ideas, so the respon-dent can concentrate on the survey item withoutbeing distracted.

þ specific.

Focus on one idea at a time. Each item shouldcollect information on a single behavior, atti-tude, opinion, event or subject.

þ explicit.

Do not force people to guess about what isbeing asked. Be sure they understand whatinformation you want by explicitly stating so. Ifnecessary, highlight or underline what isneeded by way of an answer.

23Survey Handbook

o þþ oo þo þ

Selecting Response Formats

Along with the statements and questions, youneed to provide methods for people to give theiranswers. Typically, survey items are used toask people how much they agree with somestatement, how important something is, or howoften something happens. Following are somecommon types of survey response formats usedto collect information.

Rating scales

Surveys often ask that products or services berated according to some scale. Some surveyitems present statements and ask people torate how much they agree or disagree with thestatements. Some examples of rating scalesfollow.

24 Survey Handbook

My supervisor encourages subordinates to participate inimportant decisions.

1 2 3 4 5 6Strongly Disagree Slightly Slightly Agree StronglyDisagree Disagree Agree Agree

How satisfied are you with the service you received?

1 2 3 4 5 Very Dissatisfied Neutral Satisfied VeryDissatisfied Satisfied

Please rate the condition of the housing you were provided:

1 Bad, in need of total rehabilitation.2 Very Poor, in need of several major repairs.3 Poor, in need of at least one major repair.4 Fair, in need of minor repairs.5 Good Condition.6 Excellent Condition.

25Survey Handbook

Please rank the following five improvement objectives interms of importance by marking a 1 next to the mostimportant objective, a 2 next to the second most importantobjective, and so forth:

_____ Having a quick success_____ Increasing the amount of output_____ Reducing the price charged to customers_____ Reducing work backlog_____ Reducing the number of defects

1 2 3 4 5 Mostly Completely Very Extremely RavingSatisfied Satisfied Satisfied Satisfied Fan

When creating rating scales, be sure that the endpoints or anchors are equal and opposite in mean-ing. If that is not done, there is a risk of biasing theresponses you get from the survey. For example,the following rating scale might provide answersyou like, but not information you need:

Ranking items

Another common response format is to askrespondents to rank-order a list of options interms of some factor, such as importance.These data help to prioritize what is most im-portant to respondents. Thus, if speed of ser-vice and cost for a restaurant on base areranked higher than quality of food and varietyon the menu, then efforts can be focused onthose aspects most important to respondents.

26 Survey Handbook

Selecting options

This response format presents a list of state-ments or options and asks respondents to circleall that apply to them. This format is similar tothe ranking question, but does not requiresurvey respondents to put things in an order.This kind of question is easier to respond tothan having to rank-order questions, for inmany cases respondents cannot prioritizeoptions when they feel that everything is ofequal importance.

Comments and open-ended questions

The fourth question/response format allowsrespondents to provide additional comments orother information in response to general ques-tions. These questions usually leave blankspace where respondents can write whatever isimportant in their own words. Examples arelisted below.

Please circle the numbers next to each of the graphic orstatistical methods that your team has used to analyzeprocess performance data:

1. Pareto diagrams2. Histograms3. Scatter diagrams4. Run charts5. Control charts6. Regression analysis7. Taguchi methods

27Survey Handbook

This information can be very useful in under-standing issues that have not been touchedupon or to gather ideas and generate optionsfor improvement. On the down side, responsesto these types of questions can be difficult tointerpret.

Demographic questions

Demographic information is typically used tosegment respondents into smaller groups basedon specific characteristics such as rank, age, ororganizational level. This segmentation isimportant if one of the purposes of the survey isto determine if there are significant differences inresponses between groups.

Demographic items can be included that askpeople to identify themselves, i.e., give theirnames or Social Security numbers. If you askrespondents to identify themselves in thismanner, they might not feel comfortable provid-ing honest feedback, particularly if it is “badnews.” To help reduce potential concernsabout privacy, you should include a descriptionof how the survey answers will be used and a

Do you have any suggestions on how we can improve?

Are there any products or services you need that we donot currently provide?

Is there anything else you would like us to know?

28 Survey Handbook

promise to keep individual responses anony-mous. For large or more formal surveys it mightbe appropriate to include a Privacy Act State-ment. The statement should read as follows:

Public Law 93-579, the Privacy Act of 1974,requires that you be informed of the pur-poses and uses to be made of the survey.Authority to collect this information is grantedin Title 5 of the United States Code. Provid-ing this information is voluntary. The infor-mation will be used for statistical purposesonly. In no case will the information be usedfor making decisions affecting specific indi-viduals.

Following are examples of demographic ques-tions.

29Survey Handbook

Which one of the following age categories do you currentlyfall in:

_____ Under 20_____ 20-29_____ 30-39_____ 40-49_____ 50-59_____ Over 59

What is the highest educational level you have attained:

_____ Less than high school diploma or GED_____ High school diploma or GED_____ Associate’s Degree_____ Bachelor’s Degree_____ Master’s Degree_____ Doctoral Degree

What is your marital status?

_____ Single (never married)_____ Married_____ Divorced/Separated_____ Widowed

What is your current rank and pay grade? __________

What is your organizational level?

a. Directorateb. Division Directorc. Branch Headd. Section Heade. First-line Supervisorf. Non-Supervisory

30 Survey Handbook

31Survey Handbook

o þþ oo þo þ

Reviewing Items

After you have developed a set of potentialsurvey items and response scales, review themto make sure that they are:

þ relevant to the purpose of the survey.

Items that stray from the purpose will not pro-vide the information needed. Be sure to have aspecific reason for why an item is being askedin a survey. Always focus on the purpose of thesurvey and the type of information that will beneeded to support that purpose. Carefullycompare the items to your survey purpose toensure that they address the issues that havebeen identified.

þ appropriate for the individuals being

surveyed.

Do not include items that people do not havethe knowledge to answer. For example, store

32 Survey Handbook

customers could probably answer questionsabout a store’s layout; but they would not beable to answer questions about a store’s com-pliance with fire regulations.

þ capable of providing the appropriate type of

results.

Think ahead to how the information will besummarized. The summary of results shouldprovide the types and level of informationrequired by the survey users. Will the results bepresented in simple bar graphs or subjected toadvanced analysis? How much detail will berequired to meet the information needs of thoseusing the survey? For instance, if the surveyusers are interested in general impressions,there is little reason to present averages calcu-lated to the seventh decimal point. On the otherhand, if the survey users need to make precisedistinctions among quality features, then justproviding a list of verbatim comments is notgoing to be helpful either.

Eliminate items that are inappropriate or redun-dant and clarify items that are unclear. Checkitems to ensure that they are not:

þ ambiguous.

Avoid words or phrases that can be misin-terpreted:

33Survey Handbook

þ overlapping.

Avoid presenting response choices thatoverlap. Overlapping choices can lead toconfusion on the part of the respondent anddifficulty in interpreting information:

þ leading.

Avoid giving clues that point to the desiredanswer, or limiting the answers to thosedesired:

Do you feel that you are being forced to pay too much fordaycare services?

Circle the number that best represents the number ofhours per week you spend on physical readiness training:

1. None at all2. Less than one hour per week3. One to two hours per week4. Two to three hours per week5. Three to four hours per week6. Four or more hours per week

1. Do you enjoy exciting tours?

2. This organization is energetic and active.

34 Survey Handbook

þ double-barreled.

Avoid having respondents address twodifferent issues in the same item:

þ redundant.

Avoid asking the same question more thanonce:

The clerk listened to me carefully and took a short time tohandle my request.

Evaluate the presence and quality of the listed statementswith the following scale. You will provide a single re-sponse for each statement by circling the appropriatenumber. Use your discretion to balance the presence/quality response.

Our productivity and/or quality improvement effort has astructured, highly participative planning process.

Presence: None Partially Organization- Wide

0 1 2 3 4 5 6 7 8 9 10

Quality: No Satisfactory Excellent Performance

Rewrite or delete inappropriate or repetitive items. Thenreview the remaining ones for clarity and to check thatthey will collect the information required. In addition toreviewing the items yourself, you might consider conduct-ing a pilot test of the survey.

35Survey Handbook

o þþ oo þo þ

Survey Introduction and Instructions

Surveys usually have a short introductionexplaining the purpose of the survey and howthe data will be used. Sometimes this informa-tion is explained in a cover letter attached to thesurvey. Short instructions on how to completethe survey usually follow the introduction.

Sometimes an example is provided to showhow to mark response boxes or fill out ananswer sheet. If demographic items are in thesurvey, provide a statement to reassure peoplethat their responses will remain anonymous.That statement can be reinforced by adding thePrivacy Act Statement (presented in the sectionon demographics) to the end of the instructions.Be sure to give people instructions on how,when, and where they are to return the com-pleted survey.

36 Survey Handbook

37Survey Handbook

o þþ oo þo þ

Pilot Testing a Survey

After you have a draft survey, you can conducta pilot test to further check and refine the sur-vey. Pilot testing involves trying out a surveybefore wide-scale distribution to make sure it iseasy to understand and collects appropriatedata. Pilot testing is usually done with a smallgroup of respondents who are representative ofthe larger group. If you choose to conduct apilot test, consider taking the following steps.

Explain the process.

Gather together the group who will be pilot-testing the survey. Briefly explain that they are toevaluate the survey as they are taking it andidentify questions that are unclear, topics that aremissing, and suggestions about improvements.

Request written comments.

Ask people to make notes on the survey wherethey have questions or comments. Someindividuals will not feel comfortable discussing

38 Survey Handbook

survey problems with the group, but will pro-vide written feedback. Collect the completedsurveys at the end of the meeting and reviewany written comments provided.

Measure time requirements.

Time how long it takes people to complete thesurvey. Use this information to determine if thesurvey is too long, or if it could be expanded inlength. A survey that takes more than 20 min-utes to complete is probably too long.

Hold a group discussion.

After everyone has completed the survey,review each section of the survey with thegroup to gather feedback. Record the informa-tion and study it after the pilot testing session.Sometimes comments that do not seem torepresent a good idea initially can, upon furtherreflection, be very useful. Ask if importantissues or satisfaction areas were left off thesurvey. Also ask if there are any other demo-graphic questions that need to be added to thesurvey.

Use the information obtained through the pilottest to modify the survey. Modifications couldinvolve rewriting items to increase their clarity,eliminating items that do not provide relevantinformation, or adding items to obtain morecomplete information on important issues.Once you have a final version of the survey andhave identified the sample, it is time to adminis-ter the survey.

39Survey Handbook

o þþ oo þo þ

Administering the Survey

Two common methods of administering writtensurveys are by mail and in-person. Here aresome things to consider when using one or theother.

Mail-out surveys.

The most common method of administration isto mail surveys to the customer sample with astamped, business reply envelope. Rememberto budget for postage costs, which can besubstantial if several thousand surveys arebeing mailed. When using the mail-out pro-cess, allow time for the survey to get to itsdestination, time for the respondent to completeit, and time for the survey to be returned. Sur-veys mailed to distant (i.e., overseas) ad-dresses might require 2-16 weeks for theirreturn. Local surveys might take 2-3 weeks tocome back.

40 Survey Handbook

In-person surveys.

In-person surveys can be done in differentways. One format is to ask respondents tocomplete on-the-spot surveys as they receiveproducts or services from the organization.Only very short surveys are practical in this typeof situation. The survey administrator usuallystands by as the customer completes the sur-vey and might give the customer some smalltoken of appreciation for completing the survey.A second format is to ask respondents to cometo a particular location to complete a customersurvey. A third format is to visit respondents attheir work sites and ask them to complete thecustomer satisfaction survey.

41Survey Handbook

o þþ oo þo þ

Summarizing the Survey Results

After surveys have been administered, youneed to summarize, analyze, and interpret theresults. This requires sorting and consolidatingindividual responses to survey items so thatthey can be more easily displayed and under-stood. Some common methods of summarizingdata are presented next. See Brassard (1989),Brassard and Ritter (1994), and the DON BasicTools for Process Improvement (1996) for moreinformation about these and other methods.

Frequency distributions.

Frequency distributions are a very simple methodof displaying the variation in responses to surveyitems. These distributions can be developed bycounting and recording answers according to theresponse scales used in the survey. Frequencydistributions are typically presented as tables orbar graphs for ease of interpretation.

42 Survey Handbook

Figure 1. Example of a table showing a frequency distribution ofresponses to survey items.

Availability oftechnical support

Responsivenessof technical

support

Time to completetechnical support

service

1

Very

Dissatisfied

2

Dissatisfied

3

Neutral

4

Satisfied

5

Very

Satisfied

Courtesy shownby technical

support provider

3 14 3 30 50

10 15 2 45 28

1 3 0 88 8

17 70 0 7 6

Figure 2. Example of a frequency distribution presented in a bargraph.

43Survey Handbook

Percentages.

One of the simplest ways to summarize surveyinformation is with percentages. Percentages arecalculated by dividing the total of a specificresponse choice by the total number of responsesand multiplying by 100. Percentages can bedisplayed using tables, bar graphs, or pie charts.

A note of caution

Unless a statistician or someone well-versed insurvey development and statistical analysis isavailable, keep the analyses focused on fre-quency distributions, rank orderings, counts,and averages. More complicated statisticalanalyses, such as analysis of variance orregression analysis, should be conducted bypeople familiar with them.

Figure 3. Example of percentages presented in a pie chart.

44 Survey Handbook

45Survey Handbook

o þþ oo þo þ

Sampling

Since surveys can be expensive in terms ofprinting, mailing, and data entry costs, it iscommon to select a subset or small group ofpeople from whom to gather data. This subsetor small group is known as a sample.The people targeted to fill out a survey arechosen from a specific population. A populationconsists of all members of an organization orgroup of people who possess the desired traits,knowledge, experience, or characteristics ofinterest to the survey project. A population mayconsist of, for example, all civilian and militaryemployees of the Department of the Navy, allpersonnel within a command, or the users of aproduct or service.

A rule of thumb for selecting the size of asample is to randomly select 10-20% of themembers from the population being investi-gated. Random sampling means that eachmember of the population has an equal chanceof being surveyed. Random sampling improves

46 Survey Handbook

the likelihood that the information obtainedthrough the survey will represent the responsesthe entire population would give. Please con-sult with someone who has statistical expertisewhen developing a sample, especially if youintend to use more advanced analytic methodsto interpret the results.

The number of survey respondents could bequite large. One way to organize the surveyinformation is to think of respondents in terms ofsegmentation. Segmentation means to sortrespondents into groups based on similarcharacteristics, such as their relationship to theorganization (internal, end-user, supplier), rank,or the types of products or services used. Sortingrespondents into groups is a commonly usedmethod to identify and distinguish the experi-ences and perceptions of distinct groups.Demographic items are used in surveys toidentify respondents so that their data can besorted and analyzed as needed.

For instance, a DON organization could seg-ment respondents by whether they are part ofafloat or ashore commands, operational orsupport units, headquarters offices or fieldactivities, or a combination of these work envi-ronments. External respondents could also begrouped by their use of the organization’sproduct: those respondents who use a navalstation’s piers versus tenant commands thatinhabit buildings on the naval station versusrespondents who travel to the naval station touse the legal services, the library, and clubfacilities.

47Survey Handbook

o þþ oo þo þ

Communicating Survey Results

One of the most important actions after collect-ing and analyzing survey data is communicat-ing the results. You might need to tailor presen-tations of results for different groups in theorganization.

Presentations of results to those working in theprocesses should focus on specific product orservice requirements. People working in aprocess have intimate knowledge of that pro-cess and can offer ideas on how to improve it.

A summary of the survey results should beprovided to respondents when possible alongwith a discussion of planned efforts to addressidentified problems and concerns. The surestway to kill respondents’ interest in providinginformation is to do nothing with the informationthey provide.

48 Survey Handbook

49Survey Handbook

o þþ oo þo þ

Conclusion

Surveys can be used to provide information tosupport improvement efforts. Surveys can beused to obtain information on external customerrequirements, current satisfaction levels, ideason how to improve, and the effects of previousefforts. Based on this information, decisionscan be made on where to focus theorganization’s future improvement efforts. Keepthe following in mind when beginning a survey.

þ Have a clear purpose for conducting asurvey.

þ Tailor your survey to the particular respon-dents and issues being assessed.

þ Keep surveys short and simple.

þ Communicate results to executives, manag-ers and respondents.

þ Use results to guide future improvementefforts.

50 Survey Handbook

51Survey Handbook

o þþ oo þo þ

Resources on Surveys

The following resources provide more informa-tion on developing and interpreting surveys.

Brassard, M. (1989). The memory jogger+.Methuen, MA: GOAL/QPC.

Brassard, M.,& Ritter, D. (1994). The memoryjogger II: A pocket guide of tools for continuousimprovement & effective planning. Methuen,MA: GOAL/QPC.

Culbertson, A., Houston, A., Faast, D., White, M.,Aguirre, M., & Behr, C. (1997). The processimprovement notebook (TQLO Pub. No. 97-01).Washington, DC: Department of the Navy.

Department of the Navy. (1996). Basic tools forprocess improvement. Available on the Internet:www.tql-navy.org.

Federal Quality Institute. (1993). Self-assessmentguide for organizational performance andcustomer satisfaction. Washington, DC: U.S.Government Printing Office.

52 Survey Handbook

Fowler, F. J. (1988). Survey research methods(Applied Social Research Methods SeriesVolume 1). Newbury Park, CA: Sage.

Garvin, D. A. (Fall 1984). What does “productquality” really mean? Sloan Management Re-view 26(1), 25-43.

General Accounting Office. (1993). Developing andusing questionnaires (GAO/PEMD-10.1.7).Washington, D.C.: Government Printing Office.

Hayes, B. (1992). Measuring customer satisfaction.ASQC Quality Press: Milwaukee, WI.

Henry, G. T. (1990). Practical sampling. NewburyPark, CA: Sage.

Office of Management and Budget. (1993). Re-source manual for customer surveys. Washing-ton, DC: Executive Office of the President.

Scholtes, P. (1988). The team handbook. Madison,WI: Joiner Associates Inc.

Spendolini, M. J. (1992). Customer satisfactionmeasurement. Laguna Beach, CA: MJS Asso-ciates.

Wells, D. L. (1996). Strategic management forsenior leaders: A handbook for implementation.Washington, DC: Department of the Navy.

Wells, D. L. & Doherty, L. M. (1994). A handbookfor strategic planning. Washington, DC: Depart-ment of the Navy.

Zimmerman, R. E., Steinmann, L., & Schueler V.(October 1996). Designing customer surveysthat work. Quality Digest, 16(10), 22-28.

53Survey Handbook

Edited by Bobbie RyanDesigned by Ines Muniz-Suarez

TQLO Publication Number 97-06

Department of the NavyTotal Quality Leadership Office

Washington Navy YardBuilding 36

901 M Street, SEWashington, DC 20374-5024

TQLOFFICE

DEPARTMENT OF THE NAVY