exadata customer case studies - regist-event. · pdf filedatabase consolidation 2 years in...
TRANSCRIPT
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Exadata customer case studies Regional and Global
Subhash Guddad Senior Director, Systems Sales Consulting APAC
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Customer Case Studies
Critical revenue generation applications at large telco
Extreme consolidation of Oracle and non-Oracle applications
High volume processing on Exadata
Extending Information Management with Exadata and BigData
1
2
3
2
4
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Database Consolidation with Oracle Exadata Large Telco, Philippines
Oracle Engineered Systems powered by Intel Xeon
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Large telco in Asia
4
• Telco in top 2 in Philippines( By Subscriber Base)
• Revenues of around PHP126 Billion ( USD 2.5+ Billion)
• Supported by approximately 6,800 employees; with 1.08 million retailers, distributors, suppliers, and business partners nationwide, serving more than 55 million customers.
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Project Environment – Project Good VIBES
April 6, 2017 5
OLTP Systems for Pre-Paid Promotions (NF), Service Management (SMP)
& SMS Gateway Processes (RT & AS )
Part of Tier 1 applications and supports Revenue Generation
Interfaces with Telco Network elements as well as IT Systems
Transactions : 30 million transactions per day
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Project Objectives – Current Challenges
April 6, 2017 6
• Hardware End of Life • Limited Connection Distribution • No robust Disaster Recovery • Limited Real-time SMS Gateway Performance • Limitations on Number of Promos and Other Variables • Occasional Instability of the API Module due to performance issues
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Project Objectives – Additional requirements
April 6, 2017 7
• Consolidate the databases(NF,RT,SMP,AS)
• Support 50% increase in transactions with enough buffer for spikes
• Deliver a robust DR Strategy
• Deliver a robust backup strategy
• Increase supporting service levels
• Connection pooling enhancements
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Existing Infrastructure Landscape 2009-2015:
Current Database Server Specs ( 4 Databases) Prod DB
Quantity 11 Core Count(total) 56 Speed 2.0Ghz RAM(MB) 256GB Storage(External and Internal) 4TB CPU Utilization 73% Average Response Time 69 ms
Write IOPS 2878 Read IOPS 3717
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Exadata Infrastructure Footprint
Exadata X5-2 Eighth Rack with 1 DB instance
Prod DB Quantity 1 Core Count(total) 18 Licenses None Speed 2.3Ghz RAM(MB) 512GB Storage(External and Internal)
22 TB
CPU 40% Response Time 0.4 ms Write IOPS 4854 Read IOPS 7386
Key Components as Part of Discovery Process • AWR Snapshots & AWR • Workload Information • DR Requirements • Migration Strategy • Backup Strategy • Operations Requirements
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Exadata Deployment
10
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
What did we Learn during the Deployment
Sizing Parameters ( as on 2nd March 2015)
Expected Provided
DB Cores 18 Enabled 18 (Available 36 )
Storage (TB) 12 30
IOPS 20,000 Flash IOPS: 518,000/288,000 (Read/Write) Disk IOPS: 3500
Flash Storage (TB) 0 9.6
Workload Analysis During Stress Testing • Flash Enabled for Reads (Write Through by default) based on database workload
(Read:Write @ 80:20 or 70:30) • AWR Miner showed Read:Write @ 30:70 – Write intensive • First volume testing did not show any IO issues – May not be a correct
representation of Production • Enabling Flash Write Back will ensure Flash are also used for Writes • Application VMs were causing connection & Performance Bottlenecks
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Exadata For OLTP
April 6, 2017 12
Schema Consolidation of 4 Databases: NF,RT,SMP,AS • Production : ~ Schema Consolidation of 4 Databases to a single
database
• DR: ~ Created a DR environment using Active Data Guard • OLTP databases
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Leveraging Exadata for performance
April 6, 2017 13
Performance gains of 2x – 2.5x
50% increase in supporting transaction volumes
Use Write-Back Flash Cache to handle write intensive operations
Faster backup to ZFS through Infiniband
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Migration
April 6, 2017 14
• Consolidation planning • Database load compatibility analysis • Setup appropriate resources – parameters, limits
• Migration to Exadata using Golden Gate
• Leveraged on ZFS for storing the Golden Gate trail files
• Test it out – connectivity, test with application and volume testing for
performance validation
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Considerations
April 6, 2017 15
• Coordinate downtime – Maximum 6 hours of downtime for cut-over
• Risk Mitigation: Parallel run before cut-over • Security isolation
• DC-DR Drill before go-live
Database Consolidation 2 Years in Production with
Oracle Exadata
Valeri Minkov 22.09.2016
We are going strong.
•CHF 8,224 m
•operating income (sales)
•CHF 823 m
•operating profit (EBIT)
•CHF 645 m
•Group profit
•CHF 437 m
•investments
(fully self-financed)
•2,172 m
•addressed letters (15 million consignments/day)
•115.2 m
•parcels
•CHF 114.9 bn
•PostFinance customer assets
•145.6 m
•PostBus passengers •(Switzerland)
Post CH AG | 22.09.2016 | v01.00 | public | Database Consolidation. 2 Years in Production with Oracle Exadata. | Valeri Minkov IT234 17
By employing top-quality staff, we can excel.
Post CH AG | 22.09.2016 | v01.00 | public | Database Consolidation. 2 Years in Production with Oracle Exadata. | Valeri Minkov IT234
• More than 62,000 people from over 140 nations work for Swiss Post in more than 100 different professions.
• Swiss Post trains over 2,000 apprentices in 15 professions and is one of the most popular training companies in Switzerland.
• Employees benefit from: …talent management …a wide range of training options in Switzerland …flexible working time models …workplace health management
18
nonSAP and SAP Database Consolidation with Oracle Exadata
– Target Plattform
– Configuration
– Migration
Contents
2 Years in Production with Oracle Exadata
– Organisation
– Conclusion and Benefit
Post CH AG | 22.09.2016 | v01.00 | public | Database Consolidation. 2 Years in Production with Oracle Exadata. | Valeri Minkov IT234 19
3 x Half Rack Oracle Exadata Database Machine X4-2
Configuration per Half Rack for nonSAP Platform:
– 4 Computing Nodes, 8 x 12 CPU Cores and 2 TB RAM
– 4 Cell Nodes active, 12.8 TB Flash Cache
– 4 Nodes RAC One “admin managed”
– Data Guard for HA and DR
Expansion Configuration per Half Rack for SAP Platform:
– 2 Computing Nodes X5-2, 4 x 18 Cores and 1.5 TB RAM
– 3 Cell Nodes active, 9.6 TB Flash Cache
– 2 Nodes RAC “admin managed” for HA
nonSAP and SAP Database Consolidation with Oracle Exadata Target Platform
Post CH AG | 22.09.2016 | v01.00 | public | Database Consolidation. 2 Years in Production with Oracle Exadata. | Valeri Minkov IT234 20
nonSAP and SAP Database Consolidation with Oracle Exadata Target Platform
DC1 Room2 DC2 Room1DC1 Room1
19 DB nonSAP Active Data GuardMaxPerformance &
MaxProtection
High AvailabilityHZ01HE02HE01
65 DB nonSAP Active Data GuardMaxPerformance &
MaxProtection
Disaster Recovery
150m 120km
185 DB nonSAP 187 DB nonSAP34 DB SAP
201 DB nonSAP51 DB SAP
252 DBs Consolidated
252 Databases Consolidated
Post CH AG | 22.09.2016 | v01.00 | public | Database Consolidation. 2 Years in Production with Oracle Exadata. | Valeri Minkov IT234 21
– CPU Resource Management, Instance Caging
• CPU Thread Overbooking 1 : 3.5
• CPU Utilization 55% - 65% on all Computing Nodes
– Cell Parameter
• flashCacheMode: WriteBack
• flashCacheCompress: TRUE
• iormplan: basic
– HCC for DSS ( Compression Rate 1 : 17 )
– Databases Dev, Test, Int, Prod, DSS, OLTP and GP
nonSAP and SAP Database Consolidation with Oracle Exadata Configuration
Post CH AG | 22.09.2016 | v01.00 | public | Database Consolidation. 2 Years in Production with Oracle Exadata. | Valeri Minkov IT234 22
nonSAP and SAP Database Migration in Numbers
– Migration 1 : 1 of 452 nonSAP Databases 11gR2
– Migration 1 : 1 of 85 SAP Databases 11gR2
– Coordination and Change of 146 Services
– Reduction of 92 dedicated and 30 virtualized Linux Servers
– Reduction of 39 dedicated and 12 virtualized Windows Servers
– Reduction of 300 TB SAN Storage
nonSAP and SAP Database Consolidation with Oracle Exadata Migration
Post CH AG | 22.09.2016 | v01.00 | public | Database Consolidation. 2 Years in Production with Oracle Exadata. | Valeri Minkov IT234 23
Migration Project nonSAP Databases
nonSAP and SAP Database Consolidation with Oracle Exadata Migration
Q4 2013
Q1 2014
Q2 2014
Q3 2014
Q4 2014
Q1 2015
Q2 2015
POC Exadata X3-2
Purchase and Setup 3 x Half Rack Exadata X4-2
Start Migration to Exadata
End Migration to Exadata
Post CH AG | 22.09.2016 | v01.00 | public | Database Consolidation. 2 Years in Production with Oracle Exadata. | Valeri Minkov IT234 24
Migration Project SAP Databases
nonSAP and SAP Database Consolidation with Oracle Exadata Migration
Q4 2015
Q1 2016
Q2 2016
Q3 2016
Q4 2016
POC Exadata X4-2 for SAP
Purchase and Setup 4 x X5-2 Computing Nodes
Start Migration to Exadata
End Migration to Exadata
Post CH AG | 22.09.2016 | v01.00 | public | Database Consolidation. 2 Years in Production with Oracle Exadata. | Valeri Minkov IT234 25
Unified Engineered Systems and Database Admin Team with Skills:
– 60% DBA, 15% Linux OS, 15% Network, 10% Storage
Team Tasks
– Operation of Oracle ES and Exadata Platform
– Operation of Oracle Databases on Exadata, ODA, Linux and Windows
• 658 Databases on ES and Linux
• 217 Databases on Windows
– Backup, Monitoring and Capacity Management
2 Years in Production with Oracle Exadata Organization
Post CH AG | 22.09.2016 | v01.00 | public | Database Consolidation. 2 Years in Production with Oracle Exadata. | Valeri Minkov IT234 26
2 Years in Production with Oracle Exadata Conclusion and Benefit
Before After Benefit
Infrastructure Infrastructure Infrastructure
480 nonSAP databases on 100 phys. and 30 virt. servers 573 nonSAP databases on 3 Exadata x4-2 Half Rack Higher consolidation density resulting in higher manageability
96 SAP databases on 47 dedicated and 15 virtualized servers 85 SAP databases on 2 Exadata x5-2 Quarter Rack Higher consolidation density resulting in higher manageability
Database platform “ self engineered” Full engineered, HA and performant Exadata DB Platform Cost savings for engineering
More then 10 hardware and software vendors One vendor Significan savings of time and effort due to one vendor model
300 TB of SAN storage No need of additional storage Cost savings for additional storage
Operations Operations Operations
Several database service failures, heterogeneous infrastructure Standardized environment resulting in reduction of failures Better Stability
Many teams involved for troubleshooting One OS, one vendor model allows rapid resolution to issues Single point of accountability
Expensive patch and upgrade procedures Consistent patch and update process Lower maintenance efforts, fewer outages
Operation of 85 databases per DBA Operation of more then 125 Databases per DBA Cost savings
Post CH AG | 22.09.2016 | v01.00 | public | Database Consolidation. 2 Years in Production with Oracle Exadata. | Valeri Minkov IT234 27
New York Stock Exchange Oracle Exadata – Our Journey
Ajit Singh, Director, Data Technology Platforms
New York stock exchange (NYSE) and Intercontinental exchange (ICE)
NYSE - world’s largest stock exchange by market capitalization ($28 trillion)
NYSE is owned by Intercontinental Exchange (ICE) - Architects of the world’s markets
11 exchanges, 7 clearing houses 12,000 + listed contracts and securities
Diverse markets span futures & options on interest rates, commodities, indexes and FX, as well as equities and equity options.
We transact on average in <200 mics.
30
DATA ENVIRONMENT
April 6, 2017 31
Data volume: ~50-60 billion transactions per day (at peak)
Data size: ~ 15 TB of data per day (at peak)
All data for equities, ETFs, options markets and the feed to the street
Data for trading system latency, other markets
Data services responsible for managing all the data and data platforms including Exadata
31
Oracle database environment on Exadata
32
Oracle - primary transactional database platform supporting all NYSE businesses
OLTP and Data Mart environments - 250+ databases, over 500 instances
Total database size - 350+ TB
Daily transaction volume - ~1000 million transactions
Large transactional databases: ~10-20 TB databases generating about ~ 300 GB of log per day
Large Data Mart: >100 TB in size (after HCC compression for historical data)
CONSOLIDATION TO EXADATA
2004: 2006: 2014:
17 Exadata – 12 for NYSE
33
34
NYSE Exadata footprint
35
EXADATA FOR CONSOLIDATION
April 6, 2017 36
Some examples: Dev: ~60 databases consolidated on a full rack of X3-2 with an average size of 1.6 TB QA: ~70 QA consolidated on to a full rack of X3-2
with an average size of 2 TB
Prod: ~ 50 databases consolidated on a full rack of X3-2 with an average size of 9.5 TB Mix of OLTP and data marts
36
Benefits - Leveraging Exadata for performance
37
Performance gains of 2x – 25x
Trade Reports database – 180,000 messages/sec - ~ 50% increase in processing speed
For the same application, completion time decreased by 66%
Data Mart - daily transactions of 2 TB applied to standby in real-time
Data Loads – processing times decreased by 50%
Data Mart – hybrid columnar compression - 4X compression gain vs ACO. 15-16X compression achieved on raw data using HCC
Full Backup time of a ~150 TB data mart reduced by 85% (over Infiniband switches)
Utilize SmartScan capabilities
CONSIDERATIONS
April 6, 2017 38
Coordinate downtime
Risk Mitigation: Manage capacity Oracle Homes – 2 Load Isolation Instance caging, IORM deployed to manage resources consolidated
Security isolation
Rolling upgrades of databases
Rolling Exadata bundle patches – cell servers, components, compute nodes and clusterware
Observe utilization Re-evaluate database parameters
Reduced time to market
Platform as a service
Database as a service
Reduced cost Hardware, licensing operations
BENEFITS OF CONSOLIDATION
Gurinder SIDHU, I&A CoE Leader, EnergyAustralia
Gaurav SINGH, Solution Architect, EnergyAustralia
Analytics at EnergyAustralia: Building an IM Platform in the Age of Big Data
ORACLE OPEN WORLD2016
Agenda
Who is EnergyAustralia
Transforming our Business
Strategic Analysis
Strategy to Execution
Technology Stack
Architecture
Benefits & Lessons Learnt
Who is EnergyAustralia EnergyAustralia invests in energy generation and retailing
Own and operate a multibillion dollar portfolio supplying the world’s longest interconnected power grid
One of Australia's largest and most experienced energy retailers, servicing over 2.6 million households and businesses across Australia
Transforming our Business Energy Australia needs to implement effective Information Management (IM)
This IM strategy defines the capabilities,
governance, operating models, people, technology and processes for IM to support the business
Implementation of strong enterprise IM is an essential and central prerequisite for meeting current and future business goals
Business strategy execution requires much
better information management
Current information solutions need to be
enhanced
Market disruption is driving change
Information demand is
increasing in scope, detail and
importance
Complexity Information Management is complex and hard to manage
Data domains
Multiple technologies
Operational and Analytical Architecture
Business needs
Strategic Analysis EnergyAustralia required enterprise governance, capabilities, solutions and technology that support the business effectively
Enterprise governance
Information & analytics capability
Architecture & technology
…relate to three areas of change Current capability…
Information Management Investment
The bar for adequate IM capability has been
raised
Strategy to Execution – Energy Australia IM Program To deliver on the big idea an enterprise wide program, one which would provide measurable benefits to the business needed to be
commissioned.
Enterprise wide foundation
Capability Uplift
Business Outcomes
Measure the impact Maximise the return on investment
Build architecture and technology engine
Enable business insight generation
Establish enterprise information governance
Enable I&A CoE driven collaboration
Esta
blis
h t
he
fou
nd
ati
on
s, b
uild
ou
t ca
pa
bili
ty a
nd
d
eliv
er b
usi
nes
s o
utc
om
es
Energy Australia IM – Information Governance A unified governance approach was implemented to give confidence around the data we hold as a company
Information Working Group - Meter
Information Working Group - Customer
System Owners Information Stewards
Executive Sponsorship
Executive Sponsor
Information Governance Council
Information Architect
Information Governance Leads
Information Owners
Customer Meter
Customer Meter
Information Governance components
Energy Australia IM – Execution Methodology The Information & Analytics Centre of Excellence consolidates, standardises and greatly improves the IM capability for EA. It
becomes the supplier of choice for quality information content and delivery.
EA IT
Working groups
Information and Analytics Centre of Excellence
Business Units
Information Governance Council
GOVERNANCE AND CONTROL
CAPABILITY AND SOLUTION
IMPROVEMENT AND KNOWLEDGE
DEMAND AND USAGE
Energy Australia IM – Technology Stack A Hadoop Augmented Data Warehouse architecture was proposed after evaluating 6 different alternatives
Metadata Management
High Performance Data Warehouse RDBMS
Hadoop Distribution
Data Quality Platform
Source Systems
Reporting & Analytic
Tools
Data Integration Platform
Platform Selection A cross functional project team was formed to determine the IM requirements landscape to ultimately drive the RFP selection
process.
Vendor Qualifications Experience, Financial Data, Compliance & Lawful Operations, Credentials, Quality, Risk Management &
Assurance, Partnerships with other vendors, Product Offerings & References, Product R&D, Product Development, Agreement
Lifecycle Costs Pricing, Software licensing, Annual Support costs,
Training costs, Implementation costs, other project related costs, Professional services labour rates
Support Support model, KPIs/SLAs, Software
Maintenance
Solution Delivery Implementation approach, Training
Technology Ingestion, Storage, Processing, Access, Deployment & Administration mechanisms required for diverse content.
Accessibility, Availability, Integrity, Maintainability, Performance, Reliability, Security
Innovate and Execute The ability to incubate and discover and provide a robust platform for critical analytics and reporting
EXECUTION
INNOVATIONIngestionEngines
EnterpriseModelled Data
Raw Data FunctionalModelled Data
Enterprise Data Hub
Data Discovery Hub
Batch Script SQL NoSQL Stream Search Others
INSIGHTS
CertifiedStructured & Unstructured Data
Batch Script SQL NoSQL Stream Search Others
OPPORTUNITIES
Uncertified Structured & Unstructured Data
Technical Architecture The Oracle Big Data Solution provided the ideal architecture to realise insights and drive business opportunities
High Availability Architecture The solution has is required to service a number of applications and users and is setup as an Active/Passive system with fast
failover
Database Architecture Energy Australia’s database design is very closely aligned to the Oracle Reference Architecture
Integration Architecture The Oracle Big Data Solution integrates into EnergyAustralia’s existing systems
INFORMATION MANAGEMENT PLATFORM
MM CustomerOnline Request
HTTPS
LegendLegend
WebService SOAP Call
Secure FTPFile Transfer
Employee Access
TCP/IP ChangeCapture
EXADATA X5-2
DA
TA
BA
SE
IN-M
EM
OR
YE
XA
DA
TA
BIG DATA APPLIANCE
AP
PL
IAN
CE
BIG
DA
TA
GOLDEN
GATE
DIGITAL PLATFORM
ENERGY
AUSTRALIA
WEBSITE
HTTPS
HANSEN KINETIC
AXWAY
SFTP
SFTP
INFRA
TCP/IP
TCP/IP
TCP/IP
CAMPAIGN
MANAGEMENT
SDR GATE
KEEPER
CC&B
ENERGY
AUSTRALIA
STAFF
AXWAY
SFTP
SFTP
SFTP SFTP
ORACLE
CRM on
DEMAND
iDB
ODI
API
HTTPS
MASS
MARKET
CUSTOMERS
API
SOAP
SFTP
Tools and Techniques The tools available in the Oracle Big Data Solution will be vital for delivering the analytical use cases
Use Case Technology Used
Information Governance Oracle Enterprise Data Quality (EDQ) Oracle Enterprise Metadata Manager (OEMM)
Campaign Management Oracle Enterprise Data Quality (EDQ) Address Validation Server Oracle Data Integrator (ODI)
Customer Bill Forecasting APACHE Spark APACHE Hbase APACHE Flume Oracle Big Data Discovery (BDD) Oracle Big Data SQL (BDSQL) Oracle Golden Gate
Energy Market Analysis R MATLAB Oracle Big Data Discovery (BDD) Oracle R Advanced Analytics for Hadoop (ORAAH)
Power of Choice APACHE HIVE Oracle Data Integrator (ODI) Oracle Big Data SQL (BDSQL)
Business Benefits Implementing the Oracle Big Data Solution allows EnergyAustralia to reap real benefits