lean data center, a telco experience by andrea pinnola, telecom italia
Post on 09-May-2015
4.202 Views
Preview:
DESCRIPTION
TRANSCRIPT
Copyright © Institut Lean France 2012
3 & 4 October, 2013 Paris, France
Lean Data Center: A Telco Experience
Andrea Pinnola - Telecom Italia
European Lean IT Summit 2013
Copyright © Institut Lean France 2012
• TI Background
• Bologna Data Center Operations Pilot
• Extending to the Delivery Value Streams
• Lessons Learned
Agenda
Lean Data Center: A Telco Experience - Andrea Pinnola 2
Copyright © Institut Lean France 2012
Telecom Italia is the leading Italian Telecommunications and ICT
provider, one of the most important Telcos players in LATIN
AMERICA and one of the Top Ten global telcos players
As of March 31th, 2013
Telecom Italia Group Figures and
Organization (*)
Lean Data Center: A Telco Experience - Andrea Pinnola 3
Copyright © Institut Lean France 2012
• 7 National Data Center Sites
• 4 Local Service Centers
• 24.000 Servers
• 21.000 TB Storage
• 20.000 TB Backup
• 12.000 TB Backup
Transferred/month
Domestic IT Infrastructure
4 Lean Data Center: A Telco Experience - Andrea Pinnola
Organization
Engineering & Delivery coordination
plus 3 Data Centers Areas (North
East, North West & Center South) Local Service
Centers
National Data
Center Sites
IDC Cesano Maderno
IDC Rozzano
DC Bologna
DC Padova
DC Bari
IDC Pomezia
Palermo
Firenze
Torino
Napoli
DC Oriolo Romano
Milano
(Virtual
Data Center)
2,5 Gbps
155 Mbps
VDCN Network
Copyright © Institut Lean France 2012
The case
• Difficulty to answer to all the incoming requests
• Feeling overwhelmed
• Not clear work loads
• Customers complains
Lean DC Bologna Pilot
Lean Data Center: A Telco Experience - Andrea Pinnola 5
Focus on Operations activities: Reducing
Problems and Improve Effectiveness
Copyright © Institut Lean France 2012
• Data Management: Storage
Backup
• Systems: Windows e Linux
UNIX
• Networking
• Plant & Logistics
Bologna Data Center Delivery &
Operations Teams
Lean Data Center: A Telco Experience - Andrea Pinnola 6
Sponsorship from the Head of Infrastructure
Head of DC Bologna willing to improve
Lean Data Center: A Telco Experience - Andrea Pinnola 7
The Improvement Cycle
Process
Identification
Activity
Tracing
Data
Analysis
Improvement
Identification
and Proposal
Improvement
Execution and
Check
Copyright © Institut Lean France 2012
Make Process & Problems Visible:
e.g. Management of Backup Policies
Lean Data Center: A Telco Experience - Andrea Pinnola 8
Over 1200 Backup Policies for Systems and Applications
Problems Request
Documentation
Rework
Requests to know
Policies in place
No access to the
Policies DB
Difficult Password
Retrieval
OOPS … No Data ? Problem!
Make process visible: tracking requests
Operation requests Tracking
Lean Data Center: A Telco Experience - Andrea Pinnola 9
Track every Backup Policy
Request received by Mail, IT
System, Phone call, coffee chat …
From 7-8 main activities / month
to 31 activities tracked / week !
0
2
4
6
8
10
12
14
Fault
Modific
a
SW
Info
rmazio
ni
Impl. P
olicy
Nu
mero
sit
à
0%
20%
40%
60%
80%
100%
120%
Perc
en
tuale
cu
mu
lata
Review of backup controls in order
to prevent faults incidence
Make process visible: tracking problems
Lean Data Center: A Telco Experience - Andrea Pinnola 10
• Backup Faults: highlighted
recurring errors on 4 drives
• New (type of) Fault not
described within the Fault
Management procedures
Disk Drivers Problems
Copyright © Institut Lean France 2012
Gemba Walk: Critical cabling
Lean Data Center: A Telco Experience - Andrea Pinnola 11
In one room:
• Longer cable plug in
• Old cables not removed
• Difficulty and risk of faults
during interventions
Why ?
• Continue escalating requests of
new connections done in a
hurry
Copyright © Institut Lean France 2012
Flow of improvements proposals
Lean Data Center: A Telco Experience - Andrea Pinnola 12
0
10
20
30
40
0
10
20
30
40
• All teams involved in a 5 months
timeframe
• Created one sharing point for
problems, countermeasures
proposal and approval
• Weekly review meeting with the
head of DC, Team Leaders and
Bologna Facilitator
106
By Team
By Type
Copyright © Institut Lean France 2012
Actions and Results (Examples)
Actions Results
• Storage size controls contemporary to storage assignment
• Effectiveness of storage assignment and reduced monthly reporting (4d to 2h)
• Semi-automated procedure to OS patching
• Reduction of OS patch time and of global delivery time of an updated server
• Simplification of systems /servers password management
• Eliminated password renewals delay • Reduced time to interventions
• Backup policy requests simplified templates
• Reliable policies update and reduction of info request
• Enriched scripts to better signaling backup failures
• Reduced errors and increased reliable operations
People feeling change and the effects of
their improvements
13
Copyright © Institut Lean France 2012
Results: Critical cabling recovery
Lean Data Center: A Telco Experience - Andrea Pinnola 14
• Clean situation restored
• New interventions simplified
• Recovered fibers for 1 year
of consumption (variations)
Copyright © Institut Lean France 2012
• Lean DCs Steering Committee
• Biweekly meetings to share improvements
• 3 Lean DCs Facilitators appointed
• Involved all the DCs Team Leaders
• … move to a wider Lean initiative and joint DCs
responsibility
Moving Forward: involving the other DCs
Lean Data Center: A Telco Experience - Andrea Pinnola 15
Copyright © Institut Lean France 2012
• Motivation
Growing number of installations & requests
Long time to deliver
Need for better delivered quality
• Three Cases on:
Infrastructure Creation
Virtual Machines Delivery
Market Infrastructure Delivery
Moving Forward: the Delivery case
Lean Data Center: A Telco Experience - Andrea Pinnola 16
Workshops on Lean Basics and VSM
Copyright © Institut Lean France 2012
Teamworking & Common Knowledge: the
actual Value in Value Stream Mapping
Lean Data Center: A Telco Experience - Andrea Pinnola 17
Infrastructure
Creation
Virtual
Infrastructure
Delivery to External
Market
Main Goals: Lead Time Reduction and reliable end date
Rome
Copyright © Institut Lean France 2012
5S in physical spaces
Lean Data Center: A Telco Experience - Andrea Pinnola 18
Before
• Time spent to look for materials
• Disordered Workspace
• No space to act
• Access forbidden to the customer
After
• Phase out of obsolete and not used
• Space to mount racks & shelfs
• Visual control of quantities
• Better Customer perception
5S on IT Shared Servers
• - 50% of the “space quota” assigned to the Shared Server
• - 83% of documents space, complete and accurate
information mantained
Lean Data Center: A Telco Experience - Andrea Pinnola 19
Copyright © Institut Lean France 2012
OPL: standardization and knowledge transfer
People able to manage the application since their first access
Lean Data Center: A Telco Experience - Andrea Pinnola 20
Lean Data Center: A Telco Experience - Andrea Pinnola 21
•Physical Flow
•Information Flow
• Organization ‘Flow’
Lead Time reduced by 15%
Effort reduced by 5%
Results: Visual Mng & Server Farm Creation
Lean Data Center: A Telco Experience - Andrea Pinnola 22
SERVICE REQUEST BACKLOG TREND SERVICE REQUEST AGING TREND
Results: Virtual Server Delivery
Improvement
Lead Time - 40%
Savings 4 hours for each activity
Quality - 50 % reworks on docs
Copyright © Institut Lean France 2012
Results: External Market
Lean Data Center: A Telco Experience - Andrea Pinnola 23
Improvement
Lead Time - 50%
Operation Tests Definition - 50% Elapsed Time
Rework on project docs - 75% Effort Spent
Organization One single person as DC reference assigned for each delivery
Copyright © Institut Lean France 2012
• Make Process and problems visible
• Adapt Lean to IT (e.g. 5S in physical and ‘virtual’ spaces)
• 80% of the knowledge already there (people are experts)
• Act readily on problems to get people involvement
• Management commitment from the start
• Customers involvement after a while
Lessons Learned
Lean Data Center: A Telco Experience - Andrea Pinnola 24
Copyright © Institut Lean France 2012
• To all the colleagues of the IT Infrastructure
Department that initiated and participated to the
Lean DC initiative
Acknowledgements
Lean Data Center: A Telco Experience - Andrea Pinnola 25
Copyright © Institut Lean France 2012
Lean Data Center: A Telco Experience - Andrea Pinnola 26
andrea.pinnola@telecomitalia.it
Copyright © Institut Lean France 2012
More Lean IT videos and presentations on www.lean-it-summit.com
top related