six sigma final sample project
DESCRIPTION
Sample of Succesful Final Project for Six Sigma Black Belt CertificationTRANSCRIPT
Aveta Six Sigma Online Black Belt Course
1
SAMPLE Six Sigma Black Belt Project Report
Aveta Six Sigma Online Black Belt Course
2
Executive Summary As a new subsidiary of an established company moving into a new business, establishing a reputation for solid customer service at a low price is paramount for the Security Department’s new serivce. Along the way, the business manager discovered that the idea of a solid customer service reputation was in jeopardy, but felt the personal service could not be beat. Under that presumption, he solicited help to use Six Sigma processes to 1) validate the perception, 2) propose a recommendation for solutions and 3) put processes in place to ensure control in the long run. The Problem After research and survey, we found that site managers were pleased to be moving to the new video surveillance technology, but they were disappointed in the number of outages they were seeing in the equipment. After measuring the process, we found there were several reasons for outages:
1) Network service issues 2) Software update issues 3) User training issues 4) Equipment failure
Of these, the latter proved to be the most costly and unexplainable. According to information from the manufacturer, we should not see the kinds of problems we were having. The estimated cost for these repairs exceeds $70,000 annually and has an untold impact on Operations. The Process Using the Six Sigma process of Define-Measure-Analyze-Improve-Control, we determined what the priority is for the Security technical group. We then chartered a project and proceeded to understand and define the problem. Through customer surveys, site visits and data gathering began to focus on the real issues at hand. To help clarify the situation, we mapped the installation and support processes for the security equipment finding and correcting issues as we went. Eventually, the measurement and analysis processes pointed to the fact that we had a real issue that was out of the ordinary for the equipment in service. Fortunately, we had clear concise records and good support from the manufacturer to help research and resolve the problem. Findings As the old adage goes from a well-known Houston media personality, we had “slime in the ice machine!” More importantly we had significant amounts of dirt in the machines. A by-product of the process/project was that we discovered the Information Technology team was having the same issue, just not quite to the same extreme. With some research, the team did find a low cost solution to protect the machines. While it is early in the post-solution phase, we are seeing improvements. Conclusion By using a systematic approach, we identified several areas in the process that needed modification. The net result is a longer machine life with an expected savings of more than $70,000 annually as well as improved customer service. Based on the proactive response to their issues, the Operations Management teams of both the internal and external customers are beyond satisfied and are serving as excellent references for the Security Department.
Aveta Six Sigma Online Black Belt Course
3
Definition Phase Project Opportunities As a new subsidiary business within a well established, large company, the Security Department has several competing projects for a small staff. They find themselves needing to balance on-going operations, roll out of more services to the parent company and new sales opportunities for external customers. For internal customers, the goal is to provide excellent security service -burglar, fire and equipment monitoring - as well as video coverage (with historical recordings) of the operations area. For external customers, their strategy is to utilize their existing staff to provide low cost services for monitoring alarms. Their goal continues to be low price and excellent customer service. Excellent customer service is defined as reducing false alarms and properly maintaining equipment for legitimate alarms as well as providing video services for those wanting to pay for it. In order to fully understand what was important to the customers, we conducted a survey to understand what people are expecting from the Security Department. From that survey, table 1 lists potential projects1 identified from on-going operations and for future customer sales.
Criteria and weights Weight
4 2 3 10 Profitability Easy to
Do Low Cost
Customer Satisfaction
Total
Add new model DVR support 3 6 8 7 118 Resolve perceived DVR failure issue 8 2 3 10 145 Automatic video retrieval on alarm 6 3 3 6 99 Implement IP alarm monitoring function 6 4 5 8 127 Table 1: Security Project List & Impact In discussions with senior management, internal customers, call center employees and potential external customers, the Security Department used a weighting to help determine what to focus on first. Based on input, they weighed customer satisfaction as critical since their predominant advertising mechanism is going to be word of mouth for the first year of the new subsidiary. By using the results from Table 1, the Security Department decided to investigate the perception of DVR failures since it had a direct link to customer services. As a side note, this was not even on their list of potential projects prior to the survey. Voice of the Customer With the relative surprise response of a high rate of Digital Video Recorder (DVR) failure problems, I felt it was necessary for the department management (potential project sponsor) to really understand the nature of what the customer had to say. So rather than just presenting statistics and summaries from surveys, we discussed the comment section from each response.
Site Manager (parent company) - “Being able to review and monitor transaction processing via video is critical. I never would have thought I would say that since I avoided the technology. Now, I can’t live without it. When the camera systems are down, it costs me money. If everyone knows it is down, I can’t image what they are doing.”
1 Actual list is longer. The list was truncated to simplify this report
Aveta Six Sigma Online Black Belt Course
4
Safety Manager (parent company) - “By having video access (and more importantly
recording), I have ‘eyes in the back of my head’. This technology extends my ability to cover a site and find issues before they have a negative impact on the business. An outage, especially an extended one, raises a great deal of concern.”
Security Director (external customer, multiple sites) – “The potential of the service is
incredible, but the number of camera failures is frustrating. If we do not resolve this problem, then I will have to cut services back to the basics.”
While these represent the extreme comments, several people made similar comments. The last one (and others like it) is what made Management sit up and take notice of a problem. Based on the scoring, conversations regarding customer service and potential impacts to their reputation of service challenges, the Company decided to authorize a project to investigate the perception and implement solutions if the allegations proved to be true. Remember this is a new subsidiary and working under “word of mouth” advertising. Project Charter A summary (retyped version to simplify project submission) is shown in Table 2 below. The original has full company and employee names along with telephone numbers and signatures. PROJECT INFORMATION Project Name: Chartered Date: Investigate & Resolve DVR Failure 6/1/06 Sponsoring Organization: Project Start Date: Security Department 6/2/06 Revision Number: Target Completion:
2 6/30/06 TEAM SPONSORS Project Sponsor: Contact Number:
Doug – Security Monitoring Director Project Black Belt Contact Number Tom Project Green Belt Contact Number
None available ADDITIONAL TEAM MEMBERS Name Role Contact Number
Matt Security Supervisor Alex Technician Sharron Customer Service Rep
Aveta Six Sigma Online Black Belt Course
5
ADDITIONAL TEAM MEMBERS (continued) Barry Installation Specialist Jeff IT Analyst
Nick DVR Vendor Representative PRINCIPLE STAKEHOLDERS Name Role Zack VP – Security
Bill ABC Customer – Security Manager Doug Security Monitoring Director & Sponsor SLT Senior Leadership Team
PROJECT GOALS
Identify any deficiency in DVR performance. Assuming a deficiency, implement corrective actions to resolve the problem and return performance to documented performance standards. In the absence of a deficiency, document performance expectations and discuss with the customers.
The following was added in revision 2 of the project charter after finding installation issues: Correct the installation problems and return DVR performance to published performance standards as well as reinstate customer confidence in both the equipment and customer service.
PROCESS PROBLEM
The following was added in revision 2 of the project charter after finding installation issues: Under certain installation conditions, the DVRs are failing prematurely. The apparent cause of the issue is dirt. Certain operational environments are outside recommended standards – the actual offices are temporary spaces that have a high volume of traffic in a construction type environment. The result is the server fans are pulling in too much dust and dirt for the servers to function properly.
SCOPE OF PROJECT
The following are the items that have been identified as the scope requirements: 1) Conduct further surveys to understand the nature of DVR failures 2) Investigate performance standards and expectations of the various DVR
brands 3) Compare installed DVRs and maintenance issues against the performance
standards. 4) If performance is found to be substandard, investigate what is causing the
problem and recommend solutions. 5) Pending findings and approval for #4, implement recommended solutions.
Aveta Six Sigma Online Black Belt Course
6
PROCESS IMPORTANCE
The video support function is a significant value added service provided by the Security department. Some of the potential savings and findings are listed below:
Monitor cash point of sales to discover fraud. One site reported savings of $500-$750 per day. The reports indicated that it had been going on for years. While that kind of fraud is extreme and rare, it does occur. Allows Internal Audit in performing site audits on viewing scale transactions with point of sale entries. Each IA trip is estimated to cost $5000. Using reliable video, IA could save at least 10 trips annually.
In addition to these findings, the external customers are expecting to save a great deal on safety audits and inventory shrinkage discovery. While they have not documented their savings to the company, it is clear they are expecting to save over $100,000 annually. As the project team interviews the stakeholders to confirm their interest in the project, they will document more regarding cost savings.
AUTHORIZED RESOURCES Project sponsor – Doug, Security Monitoring Director.
As project sponsor, Doug is responsible for the following: 1) ensuring staff members are available to work as a team member, 2) coordinating funding with the management team using the figures
provided by the project manager and black belt candidate, 3) coordinating participation from the customer representative. Doug’s
role also has him serving in a sales role, so he maintains the customer relationship.
Key Stakeholder – Bill, ABC Customer Security Manager.
As customer, Bill is a key stakeholder in buying the service. He has a vested interest in seeing the project work as planned due to the amount of hard dollar savings. Additionally, there are some un-quantified savings that he is expecting to achieve. His role on the project is to provide as much detail as possible about the operating environment of his company as well as answer questions regarding
expectations and understanding service deliveries. Key Stakeholder – Zack, VP – Security.
In addition to being a key stakeholder, Zack is also responsible for the entire Secure Department budget (he has fiscal and functional responsibility for the Security Department). As a stakeholder, his interest is to ensure that customer service is maintained. He services as one of the key people in regard to sales and is a major proponent of the group. While the sponsor will facilitate getting a funding stream, Zack will also help. He is also responsible for answering to the Senior Leadership Team
Aveta Six Sigma Online Black Belt Course
7
AUTHORIZED RESOURCES (continued): in regard to external sales. As a result, his role will be to help drive an understanding within the leadership team of 1) problem resolution techniques and 2) capital funding for implementing the project resolution.
Key Stakeholder – Senior Leadership Team (6 people)
The senior leadership team represents the stockholders and outside interest of the company. They have authorized and provided funding for the new subsidiary. As such, they want to receive a return on investment (profit) for that funding; however, they owe support to the operation to ensure that the best quality of service can be delivered. In the event that it cannot produce the profit as expected, they will own the decision to either continue or cease operations.
Team Members –
Matt Security Supervisor Alex Technician Sharron Customer Service Representative Barry Installation Specialist Jeff IT Analyst
Nick DVR Vendor Representative
Duties are to be defined as the project proceeds. Outside Needs
Some travel to locations may be required as part of understanding the installation process. The point of having experienced team members (Jeff, Nick and Barry) on the team is to minimize that need since they have worked in the various sites.
Technical expertise regarding the equipment in service will be required. As a representative of the company providing the equipment, Nick has access to that information. A test system with cameras may be needed for experimentation and configuration changes if required. Training needs are unknown at this time, but expected to be minimal based on the experience of those selected to participate. They were chosen under the theory that they have “seen everything”.
Table 2: Project Charter
Aveta Six Sigma Online Black Belt Course
8
Project Schedule Figure 1 is a schedule drafted early in the discovery process the edited as data became available. The control phase is documented in later sections.
ID Task Name Duration1 Definition Phase 23.32 days
2 High level business need survey 5 days
3 Review findings 1 day
4 Meet with business leaders regarding findings 0.33 days
5 Detailed follow-up on customer issue regarding DVRs 3 days
6 Follow-up meeting with business leaders regarding problem 0.33 days
7 Identify customers, sponsors and process owners 0.33 days
8 Identify resource requirements - high level 0.34 days
9 Team selection 0.33 days
10 Draft project schedule 2 days
11 Draft project budget 1 day
12 Review with sponsor 0.33 days
13 Project Charter approval 0.33 days
14 Detail project plan 2 days
15 Process Mapping 4 days
16 Develop maps 1 day
17 Resolve immediate issues (clear gaps in process) 3 days
18 Develop success metrics (CTQs) 1 day
19 Evaluate existing data for applicability 1 day
20 Sponsor review 1 day
21 Measure Phase 5.32 days
22 Validate metrics 0.33 days
23 Review existing data 1 day
24 Develop checklist measurement & sample sizes 0.33 days
25 Prep base analysis charts 1 day
26 Re-evaluate source data. 2 days
27 Analyze Phase 3.32 days
28 Evaluate preliminary chart results 0.33 days
29 Benchmark to published standards 0.33 days
30 Compare results across location type 0.33 days
31 Compare results with vendor 0.33 days
32 Review results with business leaders 0.33 days
33 Improve Phase 28.33 days
34 Team brainstorming sessions 0.5 days
35 Options analysis (costs & time) 0.5 days
36 Review options with business leaders 0.33 days
37 Capital approval documentation 2 days
38 Pilot on 2 sites 5 days
39 Rollout to remaining sites 20 days
40 Control Phase 2.16 days
41 Determine control variables 0.33 days
Week 1
Figure 1: Project Schedule
Aveta Six Sigma Online Black Belt Course
9
Process map Since the results of the original survey were a surprise to the security team, we documented the setup process (figure 2) as well as the trouble shooting and repair processes (figure 3) as well.
Request from Sitefor Video
Request review bySecurity Dept
(cost & equipmentestimate)
Cost review by sitemanagement &
Security
ImplementVideo?
Counterproposalfor other options Cost Issue?
No
Yes
No implementation
No
Security ordersequipment
Equipment staging(separate process
chart for this)
Yes
Ship equipment tosite
Security arrangesfor local
installation
Local IT orcontractor installs
equipmentSite testing
Remote testing Test OK? Production Ready
Site configuration
Yes
No
DVR Installation Process
Site prepares fordata and powerrequirements
Figure 2: Setup Process Special notes: As a by product of creating the process maps for system setup, we discovered three significant points. First, some items were not being configured consistently or at all. Some of that depended on the technician handling the process. The second item is that the sites were not necessarily prepared for the implementation process. In many cases, the local installation technician needed to arrange for power and data cabling once on site. That loop is shown in the setup process above. The third was the need to create a knowledge base for troubleshooting (figure 3 – troubleshooting). In evaluating the process, one technician was generally perceived as a bottleneck. In reality, he had a better process for handling issues which resulted in a better resolution rate. That is not documented in the results for this particular project.
Aveta Six Sigma Online Black Belt Course
10
DVR Troubleshooting & Repair ProcessAutomaticreportingfunctionsendstroublesignal
Technician callssite to discuss
problem
Problemexists?
Onsite personelnotice issue and
call in
Technician workstroubleshooting
process*
* Common issues are camera outage, focusissue, bad lighting (camera contrast), can not
search video (training issue).
Full systemoutage?
Reboot DVR
Restartnormal?
Review errormessages
Yes
Yes
No
Review cameraspecific issue
Cameraspecific issue
NoYesDispatch regional
camera contractorUser resolvablewith instruction?No
Camera contractcontacts central
technician todiagnose &
resolve
Issue resolved
DVRFunctional
Issue?Issue resolved
YesNo
Return toproductionYes
Yes
Yes
Return toproduction
No
A No
A
Customer ship unitto central support
No
Review DVRprocesses withuser (training as
required)
Yes
No
SoftwareError?
Hardwareerror?
No
Resolve perMicrosoft or DVR
vendorspecifications &
standards
YesIssue resolved
Yes
Hardware repairedand returned perset up process
No
Figure 3: Troubleshooting & Repair Process (Simplified) Key Measures for Evaluation From the troubleshooting and repair process, we found that several machines were being returned to the central support group. The general perception is that anytime a machine is taken out to be shipped, the overall outage is entirely too long and expensive. From that information and perception, we were in a position to evaluate run times, environment and impact to the business. Some of the items that we felt are critical to customer satisfaction or needed to be understood for comparison purposes are as follows (the list grew after the first analysis iteration):
Aveta Six Sigma Online Black Belt Course
11
1) Types of outages for the DVRs. a. Software issues b. Hardware issues c. Network outages (provider service) d. Network issue (hardware other than DVR, non-provider) e. User error f. Miscellaneous
2) Vendor performance expectations a. Warranty and service life for the unit b. Operating conditions
3) Number of units returned to central support in various time frames a. Weekly b. Monthly
4) Total number of support calls resulting in on-site technical support a. Weekly b. Monthly
5) Total duration of outages over various time frames a. Weekly b. Monthly
6) Total number of units in production In addition, we did discover that the central technical support staff was keeping detailed notes on the various things they were finding with each returned machine. They were just not making a point of discussing how many machines they were seeing with the management team. In the long run, the notes saved a great deal of time in getting to the final solution. That is noted in the section discussing the Analyze Phase to follow. Lastly, we also wanted to make sure that we understood the impact to the customer for each outage. The general opinion was certain outages were “less expensive” than others. We would use the Measurement Phase to help drive where to look:
1) Hard dollar cost per repair 2) Soft dollar cost per hour outage 3) Total outage per site over various periods
a. Weekly b. Monthly
Measure & Analyze Phases Using the information from above regarding the key measures, we then determined that the critical quality variable is system availability. That is generally stated as operating hours per month at the DVR level or more importantly as the time in service before failure. In some cases, the camera level data also becomes important, but not all cameras are equivalent. In order to keep the measurement process at a definable level, we focused at the DVR level overall. Camera level analysis at specific locations may become a target for review in the future. Following the decision to treat issues at the DVR level, we worked to understand the relationship among the different items as they relate to cost and business impact. The following are some of the items that we discovered:
Aveta Six Sigma Online Black Belt Course
12
1) The hard dollar costs are a dependent item, so tracking them independently is not required.
2) Network outages generally do not have a long term negative impact. The DVR continues to record and the data can be accessed once the network is available. As a result, we only used 4 weeks worth of data for analysis purposes. A later study will work with the external network provider to improve that service.
3) Opinions on soft dollar outages vary significantly based on location type. For example, sites with high volume of cash transactions (Operations) feel an outage is significantly more expensive than an office environment. As such, those numbers are not included in outage estimates in order to be more conservative on savings. Any presentation to business leaders did discuss potential numbers around soft costs.
The end result is that the key variables for monitoring are the following:
1) Number of hours of outage – relates to total cost for the outage 2) Number of units returned to central support 3) Operating time to failure
The latter uses the definition of failure as some event that requires removing the machine for maintenance (actual failure or diagnostics). Appendix A lists some of the raw data used to arrive at the conclusions listed above and to create the Pareto chart shown in Figure 4 & 5. Due to the correlation between labor effort and outage costs, Figure 5 pointed to the area needing attention – specifically that the DVR hardware maintenance uses more labor. The next question needing research was to find out how the DVRs were performing in relation to performance specifications. Some of the data for figure 5 as well as subsequent analysis came directly from the central support maintenance logs. A sample log sheet is shown in Appendix B.
Total Outage Counts by Type
0
20
40
60
80
100
120
140
160
Network DVR Hardware User issue Software Network Hardware
Count of Cause of Outage
Cause of Outage
Figure 4: Count of Outages by Type Network outages for 4 weeks only. All other data for sample time frame
Aveta Six Sigma Online Black Belt Course
13
Total Technician Hours by Cause of Outage
0
50
100
150
200
250
300
350
DVR Hardware Network Hardware Software User issue Network
Total
Sum of Technician Hours
Cause of Outage
Figure 5: Total Technician Hours by Cause Taking figure 5 a step further, we then analyzed the outages over the sample time. Figure 6 shows the findings. Using the outages per week against an upper limit (lower limit was set to zero since it could not be negative), we found the actual outages to be “in control” meaning that whatever perception site management may have, it was consistent and unless we changed something, it would continue; however, the perception remains that there were too many outages for their expectations.
Outages per week
0
1
2
3
4
5
6
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Week
Cou
nt o
f Out
ages
Outages Average UL
Figure 6: Outages per week
Aveta Six Sigma Online Black Belt Course
14
Taking this information and translating that into business impact, each outage has an average cost of $597 in repairs (labor, parts, shipping), an average outage of 70.15 hours and 2.44 outages per week. (Appendices B and C show cost and outage information.) The distribution of the outages is shown in Figure 6 previously. This does not attempt to quantify the “soft” dollar costs since the various locations have such a diverse definition of what the cameras are saving them. However, we do know that Internal Audit has a concern about the availability of the video to the point that they have not cancelled any trip in lieu of video coverage. In order for Internal Audit to “get comfortable” with video coverage and to ease site managements concern about outages, the management team mandated cuting the outages from an average of over 2 per week to 2 per month (1 every other week). That mandate equated to more than $60,000 savings annually in hard dollars. Again, the soft dollar costs are not documented. The questions needing to be addressed where 1) is this possible within the environment and 2) what would it take? The first question addresses the issue around why are the DVRs failing and is the failure within the manufacturing tolerances. The second question is based on the assumption that there was something within the environment causing the problem and not a manufacturing issue. In order to understand how the failure rate compared to what should be expected, we went back to the original installation information. While much of the information was available, there were some assumptions needed:
1) DVRs would be in production 24 hours per day 2) DVRs would be in production every day from installation 3) Production is defined as turned on so that the hardware components were in use even
if not for the intended software package. 4) The exact installation dates are not available, so we used the first day of the month for
the most conservative estimates of time in production. Using those assumptions, we then compared observed service life with expected performance. While expected performance can be expressed in many terms, we elected to use a conservative definition of warranty period.2 Based on that expectation, we found the following:
Number of Failures 39Total Population 508Average Time to Failure (days) 382.87Warranty Period 1080
Maintenance records were sampled for a 16 week period from1/29 to 5/19. Warranty is 3 years @ 360 days/year.
To further understand the outages and impact, we then analyzed which sites were experiencing the problems. That also relates to the actual business impact of the outages. In theory, zero revenue sites have less at stake (soft cost) than revenue sites. Table 3 shows the largest portion of outages effect operational facilities. Table 4 gives the installation base as background.
Aveta Six Sigma Online Black Belt Course
15
Count of Cause of OutageLocation Type Cause of Outage TotalNot documented User issue 39Not documented Total 39Office DVR Hardware 2
Network Hardware 1Software 2
Office Total 5Operations DVR Hardware 34
Software 2Operations Total 36Transfer DVR Hardware 3
Network Hardware 1Software 1
Transfer Total 5Grand Total 85
Table 3: Outage counts by location type and cause
Count of DVR NumberLocation Type Total % of TotalOffice 26 5%Operations 443 87%Transfer 39 8%Grand Total 508 Table 4: DVR counts by location type
We also took one more assumption – namely that all DVRs that have not failed will not fail before reaching the warranty level (3 years). With that assumption, we then recalculated the average time to failure as 1026.48 days with a standard deviation of 187.84 days. When plotting that in a control chart, most all of the failed units are outside of the control limits. Figures 7 and 8 show that information (the tools are limited so that 508 points cannot be included in one chart). These assumptions putting the failures outside of the control limits are conservative. Most notably is that our sample mean is less than 3 years. The actual mean from the manufacturer is more than three years – thus the warranty set at that point. A call to the manufacturer outlining our problem got us this information. That call and subsequent discussions resulted in finding that they are not seeing out of the ordinary challenges like this. After that call, they have cooperated in our investigation by sending a systems engineer to work with us through the later phases of the project.
Aveta Six Sigma Online Black Belt Course
16
DVR Production Days
0
200
400
600
800
1000
1200
1400
1600
1800
DVR (1-255)
Day
s in
Pro
duct
ion
Figure 7: Production Days by DVR (1-255)
DVR Production Days
0
200
400
600
800
1000
1200
1400
1600
1800
DVR (256-508)
Day
s in
Pro
duct
ion
Figure 8: Production Days by DVR (256-508)
DVR Production Days Against Manufacturer Information
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Failed DVRs Time In Production
Day
s in
Pro
duct
ion
Figure 9: DVR Failures Against Manufacturer's Information
Based on this new data (or more accurately, based on this relevant comparison), we can conclude that we are not seeing the expected life we should see. Something is causing the DVRs to fail
Aveta Six Sigma Online Black Belt Course
17
prematurely. At that point, the team used a brain storming session to develop the cause and effect diagram shown in figure 7. Many of the items shown on the diagram were directly from the technicians’ maintenance notes (sample form shown in Appendix B). While the manufacturing area is a process that could have problems, we tended to focus on the items that were handled internally. Everyone is aware that some machines will fail prematurely due to manufacturing challenges. If we get to the point where that is the only cause, then we will initiate another project to resolve that problem.
DVRFailure
Manufacturing Installation
Configuration& Staging
ProductionEnvironment
Out of datedrivers
Power surges
Dirt
NoPreventiveMaintenance
Damage inshipping Improper
wiring
Missing virussoftware Untrained
users
Figure 10: DVR Failure Cause & Effect
Improvement Phase Using the information from Figure 10 and the maintenance logs, we took each of the causes and evaluated their true impact on the problem:
1) Out of date drivers and missing virus software While these are a problem that this discovery process did find (and have since corrected), this would not cause the kind of outages we were seeing. A follow-up virus scan of the remaining DVRs in production turned up no infestations.
2) Damage in shipping This could be an issue, but we use the shipping containers from the manufacturer to send the machines to the location. The systems engineer confirmed that the manufacture is not seeing out of the ordinary challenges.
3) Improper wiring & power surges This does explain some of the outages we have seen. The maintenance notes on one of the power supply outages clearly documented a surge. It was not a standard policy to have surge protectors. That has changed.
4) No preventive maintenance After analyzing this, the team concluded that this may have saved the faulted units by finding the other conditions before they caused an outage.
Aveta Six Sigma Online Black Belt Course
18
5) Untrained users This is an issue for the technicians handling telephone calls. It is not a cause of hardware failure. The user security and software prevent the users from damaging the data. This topic will be analyzed for a future project.
6) Dirt After reviewing the maintenance notes, this jumped out as the most likely cause of issues in the operating locations. All of the maintenance logs for operations locations referenced an abnormal amount of dirt in the machines. Based on that, the installation specialist, technician and systems engineer from the vendor went to a field location – specifically to a site of a failed DVR. The following outlines their statements: Technician – “No wonder the machine failed, this place is a mess”
Systems engineer - “We test machines in extreme environments, but this is beyond what we would expect to see normally.”
Installation specialist after hearing their comments – “This is normal for operations”
The operating locations tend to be temporary buildings much like you would find on a construction site. One of the external customers is actually a construction firm. Generally, there is a lot of dirt movement with large equipment causing a dusty environment all around. The other observation was that the machines were not being carefully placed. Basically, the site personnel tend to direct where the machines will be placed and thinking of the machine’s health is not high on their list.
Based on the maintenance notes and site observations, we needed to find a way to protect the equipment. Once again, the team headed into a brain storming session. Unfortunately, this time did not produce a lot of results; however, Google™ saved the day. There are several inexpensive covers and keyboard skins on the market. The resulting action plan followed:
1) All DVRs (and servers after a discussion with the IT group) in operations will have dust covers.
2) All keyboards will have keyboard skins 3) IT staff will perform routine cleaning of the DVRs and servers when on site.
Compressed air cans are now standard issue. Further checks will be needed to see if the above recommendations are necessary at transfer stations and office locations. To pilot the process, we pulled two machines at the same site (some sites have multiple DVRs) from service for a quick preventive maintenance routine and cleaned them thoroughly. As they went back into service, one was covered with a dust cover and key board skin and the other was not. After a week, the team went back to the site to review the state of the machines. While it is not conclusive, there was a noticeable difference in the machines. Only a longer period of time and close review of the control charts of outages will clarify the understanding.
Aveta Six Sigma Online Black Belt Course
19
Control Phase
As noted throughout this document, there were several places where we discovered minor deviations from best practices. As part of the ongoing control, there are several steps that are in place (or soon will be) to control and monitor the production environment:
1) A configuration check list has been standardized 2) Each technician completes the check list and signs it. 3) Someone other than the technician reviews the checklist for completeness and prepares the
machines for shipping. The same person did step 2 and 3 previously. 4) Field IT will now handle preventive maintenance when at the sites working normal support.
They will note any issues and report to the Security technicians. 5) Control charts like the one in Figure 6 are now posted for the top 4 items creating DVR outages.
Any time complete failures happen once per week, the staff reviews the machine to look for issues.
6) Routine comparisons of equipment trouble tickets will be compared to what the IT group is experiencing to watch for similarities. As a side note, the IT group has implemented several of the things the Security department has found and while they have not published expected savings, it is easily going to be more the $100,000 annually when combined with the Security Department.
As you can tell from the brevity of the control section, we have not had time to confirm the results. This has literally just been completed. Early indications from Security and IT are things are definitely on the right track. Continued observation will tell the tale. Conclusion The net result of the project was more than anticipated in hard dollar savings. While we have not proven that we are down to one outage every other week, the early indications are good. In addition, the IT group is also seeing a reduced hardware outage issue, but not quite to the same degree. They had a better preventive maintenance program in place, but protecting the machines and discussing the issue with site management has proven productive. At this point, the estimated savings from this project is in excess of $70,000 hard dollars. As stated many times above, there is no estimate on soft dollars. In addition to that, we have also implemented new policies and practices. Some of those lessons learned include the following:
1) Virus protection & device drivers updates and maintenance 2) Proper site preparations (wiring) 3) Surge protection 4) Servers were having similar issues
By using a systematic approach and truly analyzing the problem, we improved customer service, proved to be responsive and saved money.
Aveta Six Sigma Online Black Belt Course
20
APPENDIX A – Cost Structures & Support Hour Samples
The following are the cost and implementation elements used for estimating outage impacts: Function Cost Contract labor – on site camera work $75/hour DVR Shipping $50 each way Central Technical support $20/hour Network connection (Frame) $1000/month/site DVRs in production 508 Sites with DVRs 432 The following are the labor estimates for configuring, installing and troubleshooting (many of these are subject to future study for improvement via Six Sigma): Function Responsible Party Labor Estimate DVR setup (from un-boxing to repackaging
Central Technical support 2 hours
Telephone Support Central Technical support 15 minutes/call Software download & configuration Central support 30 minutes each On-site setup & configuration of DVR Contract Labor 2 hours
Aveta Six Sigma Online Black Belt Course
21
APPENDIX B – DVR Maintenance Log Sheet
The following is a blank maintenance log sheet as generated by the maintenance ticket system. The process has the technician manually complete the form. Eventually, the data is entered into the tracking system.
MAINTENANCE LOG SHEET Date Received: Repair Ticket Number: auto-generated DVR Location: Address used for return shipment (Y/N):
Configuration: Manufacturer: ____________________ Model: _____________________ Disk Drive type/size: ________________ Operating System: __________________
Equipment Status on Receipt:
Repair Comments:
Software Load: Date Returned: Shipment Tracking Number:
Technician:
Aveta Six Sigma Online Black Belt Course
22
APPENDIX C – Data Summaries Summary of outages by weeks of the sample: Sum of Outage Duration in hoursWeek of Outage Total
1 1862 2813 1064 2005 2176 1457 1698 2659 290
10 17011 7312 20413 9714 24115 9716 145
Grand Total 2883 Summary of Outage Hours by outage type Sum of Outage Duration in hoursCause of Outage TotalDVR Hardware 2736Network 77Network Hardware 36Software 25User issue 10Grand Total 2883 Summary of Technician Hours by outage type Sum of Technician HoursCause of Outage TotalDVR Hardware 312Network Hardware 36Software 25User issue 9.75Network 0Grand Total 382.75
Aveta Six Sigma Online Black Belt Course
23
Count of Outages by Type Count of Cause of OutageCause of Outage TotalNetwork 142DVR Hardware 39User issue 39Software 5Network Hardware 2Grand Total 227 DVR Failure counts by week and location type Count of Cause of OutageWeek of Outage Location Type Cause of Outage Total
1 Operations DVR Hardware 22 Operations DVR Hardware 3
Transfer DVR Hardware 13 Operations DVR Hardware 14 Operations DVR Hardware 25 Operations DVR Hardware 2
Transfer DVR Hardware 16 Operations DVR Hardware 27 Operations DVR Hardware 38 Office DVR Hardware 1
Operations DVR Hardware 29 Operations DVR Hardware 4
10 Operations DVR Hardware 211 Operations DVR Hardware 112 Operations DVR Hardware 2
Transfer DVR Hardware 113 Operations DVR Hardware 214 Office DVR Hardware 1
Operations DVR Hardware 315 Operations DVR Hardware 116 Operations DVR Hardware 2