capacity and performance management ... - cio summits
TRANSCRIPT
MUFG Capacity Management TransformationMUFG Capacity
APRIL 15, 2015
2 MUFG Union Bank, N.A. | Integrated Services for the Americas
Ellen Friedman (Director) – MUFG UB Leader of Capacity, Performance & Availability Management
Roger Lee (Assistant Vice President) – MUFG Lead Capacity & Performance Management Lead
Engineer
• Joined the bank in March 2009 and developed the Capacity Management and Performance Engineering Practice
• In the past 6 years, bank has grown 3-fold, huge investments in new technology, new application platforms,
Program Hudson, New Core Banking Program, New Market Systems initiatives
• Performance Engineering/Testing for 25 Major New Applications in the past 18 months
• Joined the bank in August 2013 and serves as Lead Engineer responsible for Teamquest strategy, architecture,
capacity management reporting framework leads capacity management initiatives particularly for batch
optimization (Data warehouse, Market Systems, Core Banking)
• Prior financial experience (Credit Suisse, Merrill Lynch) and software engineering (Lockheed Martin, PepsiCo),
degree in Computer Science, special interest in statistical modeling and operating systems
MUFG Overview
3 MUFG Union Bank, N.A. | Integrated Services for the Americas
MUFG operates in over 40 countries with more than 1,100 offices. In the Americas we service our
corporate and investment banking customers under the MUFG brand, and we go to market with our
retail, wealth, and commercial banking customers under the Union Bank® brand
Americas #of locations: 31
Hirano
Head
Office
Tokyo,
Japan
Countries
of
Operation
40+
Our vision is to “be the world’s most trusted financial group”
Agenda
BTMU Capacity and Performance Management Services
History & Environment Overview
Realized & Future Benefits
MUFG Union Bank, N.A. | Integrated Services for the Americas5
• Business Driven Requirements & Volume Analysis
• Select vendor, platform, architecture
Planning Phase
Design Phase
• Performance Testing
• Necessary Changes to System Infrastructure/Capacity/Priorities to meet SLA
• Necessary Changes to Application (e.g., code optimization, heap size, application tuning)
Testing Phase
• Performance Problem Resolution
MUFG Union Bank, N.A. | Integrated Services for the Americas6
• After joining the bank, we leveraged TQ for major new VMware virtualization
effort. (June 2009)
• Purchased licenses for VMware only and “20-30 floating licenses” to
evaluate if application met performance/capacity criteria and ROI criteria
for virtualization
(vSphere 2.5- Now vSphere 5.1)
• Increased the VMware footprint and continued to purchase (project basis)
• Within past 18 months we have standardized on TQ as our “Go-To Solution”
for capacity/performance monitoring
• Installation on all Citrix, VDI infrastructure, Exchange, File Server
(physical and virtual)
• New AIX deployment initiative- rolling out to 400 AIX LPARs
7%
(200)
54%
(1,500)
AIX
• Standard tool set with standard
data collection to automate/build
application/server/storage array
bottlenecks” (past minute) for
storage, server and application
• Citrix Session level data
MUFG Union Bank, N.A. | Integrated Services for the Americas8
Report Automation: VMware Capacity
• Realized $ Savings through the automation of monthly management and exception reports
Report Automation: Storage Showback
TeamQuest Deployment and overhead costs
• Demonstrated that “TeamQuest footprint” is “lightweight”
Software Licensing Requirements
• Leveraged TeamQuest to identify “actual CPU usage” to reduce software license costs
Leveraging TeamQuest during
Leveraging TeamQuest for
• Able to identify bottlenecks more quickly for problem determination
MUFG Union Bank, N.A. | Integrated Services for the Americas9
Benefit: Automated Monthly Report Generation: VMware Monthly
Capacity Report: Automatically Generated with Surveyor
Narrative: 1. What changed from last
month
add
more hardware?
MUFG Union Bank, N.A. | Integrated Services for the Americas10
Benefit: Automated Monthly Report Generation:
Showback: Storage Usage by Application
Application: Record Keeping
Benefit: “Light-weight footprint” for deploying on the servers
COMPARISON: TEAMQUEST VS. OTHER TOOLS (PROGNOSIS & TIVOLI)
Approach: • Deploy in Development, SIT, UAT prior to installing in Prod
• Show the overhead after 2 weeks of installation in Test environments
• A week of performance measurement data was reviewed. Presented a representative subset to
the teams
Results: • Broker LPARs-Sensitive to CPU overhead and Memory Overhead
In all cases TeamQuest used less processor than Prognosis, and averaged half as much
as Tivoli
In all cases TeamQuest used less memory than Prognosis, and averaged less than Tivoli
MUFG Union Bank, N.A. | Integrated Services for the Americas12
Benefit: Demonstrate Software Licensing Requirement Savings in
Dev: Implement Virtual Pools for IBM Software Licensing (Broker)
License Rules: • For Capped Partitions: The highest actual level of Entitled Capacity (EC)
• EC was formerly referred to as PrU (Processing units)
• For uncapped Partitions: The Highest actual number of Online VP ( Virtual Processor
Cores)
• Shared Pool Capping Rule: the lower of the sum of each partition for a product or the
processor core capacity of the shared pool
• Aggregate fractional processor cores, apply shared pool capping rules, and round up at
the server level to the next whole processor core
• The greater of what the partition starts with or the result of a DLPAR operation
Purchased
8
Required
5
Savings
3
Benefit: Leveraging TeamQuest for Performance Testing/
Engineering: Analyzing Broker Performance/Resource Footprint
• Broker CPU and Memory Usage is balanced across both LPARs- 2 VCPUs and 32 GB memory each
• Showing Memory Decomposition Total (Computational, Non-computational, unused)
Activity
ramped
Benefit: Leveraging TeamQuest for Performance Testing/
Engineering: Drilldown: Who is using the resources?
Top 5 Processes/Workloads for CPU and Memory
Top CPU Consumers:
1. EPP Common
Benefit: Leveraging TeamQuest for Production Support:
Root Cause Analysis: Memory Leak
• Chart shows memory utilization increases over month
• Spike in memory usage on 3/18 Memory Demand exceeds capacity
(24GB)
• The “CULPRIT” process suddenly increased its memory demand
causing total memory to be “exhausted”.
• “CULPRIT” Process has a “ Memory LEAK”
Memory demands exceed
Total system memory
Our future-state vision
• Extract data from TeamQuest to generate Service-Based capacity reports through a common
capacity reporting database to drive
• Leverage Surveyor to automate manual reports (exception, storage frame, etc.)
• Tie storage capacity to application consumption
Capacity Management Maturity
Future Report Definitions
TeamQuest Infrastructure
Thank you!
APRIL 15, 2015
2 MUFG Union Bank, N.A. | Integrated Services for the Americas
Ellen Friedman (Director) – MUFG UB Leader of Capacity, Performance & Availability Management
Roger Lee (Assistant Vice President) – MUFG Lead Capacity & Performance Management Lead
Engineer
• Joined the bank in March 2009 and developed the Capacity Management and Performance Engineering Practice
• In the past 6 years, bank has grown 3-fold, huge investments in new technology, new application platforms,
Program Hudson, New Core Banking Program, New Market Systems initiatives
• Performance Engineering/Testing for 25 Major New Applications in the past 18 months
• Joined the bank in August 2013 and serves as Lead Engineer responsible for Teamquest strategy, architecture,
capacity management reporting framework leads capacity management initiatives particularly for batch
optimization (Data warehouse, Market Systems, Core Banking)
• Prior financial experience (Credit Suisse, Merrill Lynch) and software engineering (Lockheed Martin, PepsiCo),
degree in Computer Science, special interest in statistical modeling and operating systems
MUFG Overview
3 MUFG Union Bank, N.A. | Integrated Services for the Americas
MUFG operates in over 40 countries with more than 1,100 offices. In the Americas we service our
corporate and investment banking customers under the MUFG brand, and we go to market with our
retail, wealth, and commercial banking customers under the Union Bank® brand
Americas #of locations: 31
Hirano
Head
Office
Tokyo,
Japan
Countries
of
Operation
40+
Our vision is to “be the world’s most trusted financial group”
Agenda
BTMU Capacity and Performance Management Services
History & Environment Overview
Realized & Future Benefits
MUFG Union Bank, N.A. | Integrated Services for the Americas5
• Business Driven Requirements & Volume Analysis
• Select vendor, platform, architecture
Planning Phase
Design Phase
• Performance Testing
• Necessary Changes to System Infrastructure/Capacity/Priorities to meet SLA
• Necessary Changes to Application (e.g., code optimization, heap size, application tuning)
Testing Phase
• Performance Problem Resolution
MUFG Union Bank, N.A. | Integrated Services for the Americas6
• After joining the bank, we leveraged TQ for major new VMware virtualization
effort. (June 2009)
• Purchased licenses for VMware only and “20-30 floating licenses” to
evaluate if application met performance/capacity criteria and ROI criteria
for virtualization
(vSphere 2.5- Now vSphere 5.1)
• Increased the VMware footprint and continued to purchase (project basis)
• Within past 18 months we have standardized on TQ as our “Go-To Solution”
for capacity/performance monitoring
• Installation on all Citrix, VDI infrastructure, Exchange, File Server
(physical and virtual)
• New AIX deployment initiative- rolling out to 400 AIX LPARs
7%
(200)
54%
(1,500)
AIX
• Standard tool set with standard
data collection to automate/build
application/server/storage array
bottlenecks” (past minute) for
storage, server and application
• Citrix Session level data
MUFG Union Bank, N.A. | Integrated Services for the Americas8
Report Automation: VMware Capacity
• Realized $ Savings through the automation of monthly management and exception reports
Report Automation: Storage Showback
TeamQuest Deployment and overhead costs
• Demonstrated that “TeamQuest footprint” is “lightweight”
Software Licensing Requirements
• Leveraged TeamQuest to identify “actual CPU usage” to reduce software license costs
Leveraging TeamQuest during
Leveraging TeamQuest for
• Able to identify bottlenecks more quickly for problem determination
MUFG Union Bank, N.A. | Integrated Services for the Americas9
Benefit: Automated Monthly Report Generation: VMware Monthly
Capacity Report: Automatically Generated with Surveyor
Narrative: 1. What changed from last
month
add
more hardware?
MUFG Union Bank, N.A. | Integrated Services for the Americas10
Benefit: Automated Monthly Report Generation:
Showback: Storage Usage by Application
Application: Record Keeping
Benefit: “Light-weight footprint” for deploying on the servers
COMPARISON: TEAMQUEST VS. OTHER TOOLS (PROGNOSIS & TIVOLI)
Approach: • Deploy in Development, SIT, UAT prior to installing in Prod
• Show the overhead after 2 weeks of installation in Test environments
• A week of performance measurement data was reviewed. Presented a representative subset to
the teams
Results: • Broker LPARs-Sensitive to CPU overhead and Memory Overhead
In all cases TeamQuest used less processor than Prognosis, and averaged half as much
as Tivoli
In all cases TeamQuest used less memory than Prognosis, and averaged less than Tivoli
MUFG Union Bank, N.A. | Integrated Services for the Americas12
Benefit: Demonstrate Software Licensing Requirement Savings in
Dev: Implement Virtual Pools for IBM Software Licensing (Broker)
License Rules: • For Capped Partitions: The highest actual level of Entitled Capacity (EC)
• EC was formerly referred to as PrU (Processing units)
• For uncapped Partitions: The Highest actual number of Online VP ( Virtual Processor
Cores)
• Shared Pool Capping Rule: the lower of the sum of each partition for a product or the
processor core capacity of the shared pool
• Aggregate fractional processor cores, apply shared pool capping rules, and round up at
the server level to the next whole processor core
• The greater of what the partition starts with or the result of a DLPAR operation
Purchased
8
Required
5
Savings
3
Benefit: Leveraging TeamQuest for Performance Testing/
Engineering: Analyzing Broker Performance/Resource Footprint
• Broker CPU and Memory Usage is balanced across both LPARs- 2 VCPUs and 32 GB memory each
• Showing Memory Decomposition Total (Computational, Non-computational, unused)
Activity
ramped
Benefit: Leveraging TeamQuest for Performance Testing/
Engineering: Drilldown: Who is using the resources?
Top 5 Processes/Workloads for CPU and Memory
Top CPU Consumers:
1. EPP Common
Benefit: Leveraging TeamQuest for Production Support:
Root Cause Analysis: Memory Leak
• Chart shows memory utilization increases over month
• Spike in memory usage on 3/18 Memory Demand exceeds capacity
(24GB)
• The “CULPRIT” process suddenly increased its memory demand
causing total memory to be “exhausted”.
• “CULPRIT” Process has a “ Memory LEAK”
Memory demands exceed
Total system memory
Our future-state vision
• Extract data from TeamQuest to generate Service-Based capacity reports through a common
capacity reporting database to drive
• Leverage Surveyor to automate manual reports (exception, storage frame, etc.)
• Tie storage capacity to application consumption
Capacity Management Maturity
Future Report Definitions
TeamQuest Infrastructure
Thank you!