rac best practices
DESCRIPTION
Oracle Real Application Clusters (Oracle RAC) & Oracle E-Business SuiteTRANSCRIPT
© 2010 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks contained herein are the property of their respective owners.
Best Practices at AT&TOracle Real Application Clusters (Oracle RAC) & Oracle E-Business Suite
Bryon Rickey
Ravi Kayarthodi
September 20th, 2010
Agenda Welcome and Introductions
Supply Chain Management (SCM) environment
Overview/highlights
Architecture
Partnership with Oracle to transform our Business
Key project highlights
Future initiatives
Solution Support Center (SSC) and on-site support
Q&A
About AT&T
AT&T is one of the world’s largest Telecommunications companies
AT&T Global Headquarters is based in Dallas, TX
AT&T has approximately 290,000 employees
AT&T has more than 90 million wireless customers
AT&T has more than 16 million high speed internet subscribers
AT&T has more than 2.5 million uverse subscribers
Exclusive U.S. provider of iPhone 3G / 3GS / 4
AT&T SCM: Architecture highlights
Oracle E-Business Suite Centric Scale-up with cluster database on very large
infrastructure High availability with Oracle Data Guard, Oracle
GoldenGate, and site level redundancy Central inventory/financial management; utilizes
product hub, planning/forecast with ASCP/Demantra Order orchestration and brokering with in-house
developed Oracle based application system Internet deployment for interaction with external
suppliers Management reporting/profitability analysis;
procurement and spend analytics Industry leading warehouse/distribution
management systems
AT&T SCM Platform: Financial Integration Both Network and Retail SCM - financial
counterparts are integrated in the same platform: Shared AP, AR, PA, FA, and GL
Supply Chain transactions leverage same referential data and accounting rules
Financial integration of supply chain transactions are seamless and are recorded real-time in the general ledger
No need for reconciliation between sub ledgers Ability to forecast financial implications based on
SCM forecasts Online financial budgeting validation on
purchasing and requisitions Analytical GL Cube and Inventory Cube
Results
AT&T has saved more than $500 million and has improved its purchasing power by combining accounts with its vendors.
Estimated to save $1B in inventory costs, and $300M per year in reduced operating costs.
Warehouses have been able to keep inventories low without missing SLA, resulting reduced overall inventory costs.
Ability to identify sold merchandise, resulting in fewer fraudulent returns
Industry Recognition
2010 IDG’s CIO AwardRecognizes organizations around the world that
exemplify the highest level of operational and strategic excellence in information technology
Systems Architecture
AT&T SCM: E-Business Suite Centric
Oracle E-Business Suite 11i
Other Billing Systems
HR - Employee
EDI
Warehouse/Distribution System
Asset Tracking
Direct Fulfillment
Spreadsheet
Lease Management
Retail Planning
Construction
General Ledger Receivables
Accounts Payable Purchasing
Inventory
Order Management Assets
AT&T Corporate Systems
Tax
Projects
Archive Returns
Procurement partners
COR Stores
Planning/ Forecasting -
Network
Order Broker/Orchestrator
Legacy Procurement
Provisioning
Other Channels
External Facing; Suppliers and
Auctions
Technician Ordering
SO
AM
idd
lew
are
Procurement
3PL portal
TMS
Demand Planning
Profitability Analysis
Procurement & Spend Analytics
AT&T SCM: Key Systems Architecture
Oracle E-Business Suite 11i
Inventory Planning Purchasing Financials
Corporate Asset TrackingDistribution Center
Warehouse Mgmt Inventory Tracking
ReceiptsReceipts
Shipments
EDI EDI
Audits
Repairs
APPOODP eAuction GL
FAPAASCPOM RNF
Moves
OpusOpus
Oracle 11i E-Business Environment
Dealers/iStore
Dealers/iStore
Store Replenishment
Store Replenishment Direct FulfilmentDirect Fulfilment
National Retailers
National Retailers
DSL and uVerse HSIA
DSL and uVerse HSIA
StoreStore XBM/InsuranceXBM/Insurance Dealer/NRDealer/NRService DepotsService Depots SalesSales ReturnsReturns
Order Management
Order Management
Forward Orders
UpfrontUpfront
Reverse Company Owned Retail
416 CPU Cores & 1.7TB of Memory in DB Cluster
14TB of storage for transaction DB
3TB of storage for Archived DB
PlanningPlanning PurchasingPurchasing InventoryInventory ReceivablesReceivables FinancialsFinancials
Oracle E-Business Suite 11i Apps
11i DB
EPLEPL
RU30RU30
AT&T SCM Platform: Oracle10g RAC Infrastructure: VCS+ CRS
AT&T SCM Platform: Transaction Volume
Approximately 3.35 million logons to the production areas per day
2.06 billion SQL executions per day Approximately 136.5 Billion logical reads and 41TB
of physical reads performed per day Approximately 7 TB of writes performed per day Approximately 1.05 billion user calls and 52
million transactions completed per day 200,000 concurrent request per day
AT&T SCM Platform: Operational architecture
Decoupled month-end book closing preparation Statistics gathered on a clone
Separate reporting environment – refreshed nightly; Inventory and GL Cubes for reporting
Revenue share and allocation on a separate system
Direct supplier interaction with Internet-deployed application modules
Utilizes market making mechanisms (sourcing)
AT&T SCM: Oracle 11i Dev/Test Configurations
PRODL-TRA
RPT
R-TRA
DEV
QA
DEV
DEVQA
DEV
DEVQA
DEVDEV
QA
DEV
Data Guard
Data Guard
Flex Clone
FTP
HDS – USP-V HDS – USP-VBothell, WA
MET1
Shadow Image
NetAppGOLD
Data Guard/
Custom Shipping
NetAppMaster
Flex Clone
MET2
Shadow Image
NetAppGOLD
NetApp
Master
Data Guard/Custom Shipping
Flex Clone
Flex Clone
Shadow Image
SnapMirror
Alpharetta, GA
FTP
Storage Efficiency with NetApp de-duplication Tools Performance test environment on tier-1 storage
AT&T SCM: Oracle Automatic Storage Management Local and Remote DR databases and also
Reporting databases have been migrated to ASM storage to meet storage and capacity needs Avoided SFRAC costs Performance
SCMP DR/BC database converted to RAC database Saved SFRAC license cost on E25K frames
Comply with AT&T standard for cluster database (ASM is required)
Expected to save cost in the future when we upgrade other Corporate Systems’ databases
AT&T SCM: Archive and Purge
Archive Stats 16 custom and core modules
including (GL, INV, OM, AR, WSH, PO, FA)
Executed weekly, monthly, quarterly, and yearly
History instance Informatica Data Archive
application used Schedule is monitored real-time
to ensure adherence
Purge Stats 70 jobs via Oracle concurrent
programs Executed daily and weekly Schedule is monitored real-time
to ensure adherence
AT&T Supply ChainGeographic Distribution – DB Flow to Test and Dev
XX
XX
Prod
Dev/Test
DR Site
Test Site 1
Development Site 1
Primary Site
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XXXX
XX
XXXX XX
Development Site 2
XX
XXXX
XX
Test Site 2
XX
XXXX
XX
AT&T SCM: Spend Analytics
Top 5 Categories
Top 5 Suppliers
Line Of Business
Spend
Contract Leakage Off Contract Spend Payables Leakage
Oracle Business Intelligence Enterprise Edition (Oracle OBIEE) Powered
Sourcing from Oracle E-Business Suite and Legacy Applications
AT&T SCM: Database Maintenance/Performance Management
Monthly month-end close of books stats validation – database statistics preparation on production clone
Table Re-organization – routine and ad-hoc Periodic index rebuild Daily performance call Daily/weekly/monthly/yearly purge/archive
schedules – vigilance in adhering to them Extensive performance monitoring on all levels
of the system
Partnering with Oracle Advanced Customer Service
Oracle Solution Service Center (SSC)
SSC: Project Spotlight: Patch Recommendations
23
Project Activities Description
o Identification of available latest required patches for AT&T environment
An initiative to incorporate E-Business Suite recommended patches, such as, Rollup (RUP) in AT&T’s release schedules for 2010o Participation in meetings to review
recommendations
Value Addition SSC and AT&T worked together to create a new process to efficiently identify required
patches with the goal to minimize the impact of rolling them out in our production environment and to current/future projects: Quarterly document for AT&T environments with the latest recommended patches. Participation on AT&T IT meetings to clarify recommendations and questions Provide detailed patch analysis on a ad-hoc basis
Successful roll out of AP.O patchset along with other Financials patches in February 2010: stabilized production environment and brought environment to the latest code base.
Successful completion of the ASCP Rup 37, Collections Rollup (RUP) 32 and Application Technology Group (ATG) RUP-7 for the April 2010 production rollout.
SSC: Project Spotlight: Performance Improvement
Project Activities Description
o Understand AT&T batch window requirements and transaction volumes
An initiative to address various performance issues related to Month-End-Close processes and other batch processes after Oracle 10g database upgrade.
o Identification and root cause analysis of problem areas
o Implementation of resolutions
Value Addition Active participation on Conference bridge calls and OWC sessions. Engage Oracle development groups (Financials and Apps Performance) to provide SQL
tuning improvements. Improve month end close activities like AR close, depreciation and asset retirement,
optimizer, Invoice Validation processes. Maximize performance during month end, preventing delay; AT&T was able to successfully
meet the SLAs for the Month End Close Processes month after month. Provide recommendations for major upgrades (DB, Applications and Tools)
SSC: Critical SR Spotlight: Other Key areas of focusProject Activities Description
o Workflow background process
WF_DEFERRED queue growing exponentially; AT&T engages SSC to evaluate and address the issue.upgrade.
o RAC – CRS Configuration
Collect diagnostic and configuration dataTroubleshoot frequent RAC node reboot/performance issue post upgradeDiscover unforeseen problemsValidate AT&T configuration with RAC Best Practices
o Device Life Cycle Database UpgradeProvided upgrade expertise and support (pre and post deployment for 11g R2 upgrade.
Value Addition Active participation on conference bridge calls and OWC sessions Review current setups and schedules of workflow Background Process Engage on Severity-1 SR and work collaboratively with Oracle development to provide workaround (eg.
skip script) Work with at&t and Oracle development organization to provide scripts to clear workflow DEFERRED
queue Monitor the implementation of recommended suggestions Engage Oracle Development and RAC COE team Engage on Architectural design discussions Stand-by support for DB/RAC upgrade deployment activities Ensure successful implementation of RAC upgrade Rollout
SSC: AT&T Targeted Services: Operations The Services Yield to the Customer targeted for the next 6-month planning period is comprised of key Support activities, aligned with key customer objectives/activities towards achievement of AT&T’s Account Goals
Supply Chain Transformation (Streamline Operations for newly deployed modules)
Work with ACS sales to determine appropriate support for upcoming SC rollout. Determine/secure funding for new modules (Demantra and Transportation Management Suite)
Develop, Enhance and Secure development sponsors Participate in scheduled project “Deep Dive” calls with PMO and project team Site visit to Memphis to understand warehouse and logistic operations – understand
business functions. Supply Chain Business – develop key relationships with Retail and Network business users
11i Financials and Supply Chain Production Supportability (System) Next generation iPhone launch and 2010 Holiday Retail Season readiness “Top Ten” Prioritization
SSC: AT&T Targeted Services: Transportation Manager OneATT- New Horizon to transform Network and Asset Management: Logistics TMS
Key Secondary Activities Participate in scheduled project “Deep Dive” calls with PMO and project
team Work with PMO/SSC to determine appropriate support model for New Horizon’s Work closely with OCS /PMO on project milestones – develop relationships Continue to maintain Dev OTM Network Supply Chain Business – develop key relationships Site visit to Warehouses SSC walk-thru’s
Key Deliverables Enhance Supply Chain SSC presence and understand usage
and architecture for OTM Site visit to understand implementation, document in
configuration guide. Performance testing as requested. Amend SSC contract to include OTM products
SSC: Oracle E-Business Suite Release 12/Database 11g R2 Upgrade Upgrade existing Oracle 11i.10 SCM and Planning environments to E-
Business Suite R12. This will enhances supportability for our complex environment and provide additional needed functionalities
Three stage upgrade (2 – 3 year journey) Supply Chain planning to release 12 RDBMS for 10g to 11G Supply chain platform to release 12
Objective: System Availability, Supportability, and Software Currency
Key Secondary Activities SSC “Top Ten” Prioritization: Service Request review meetings with Supply Chain,
Financials, and DBA support teams Patching/Release Management strategy for Release 12 interoperability; Leverage Tools and Templates Daily SSC Performance Calls
Key Deliverables Proactive Diagnostics (Configuration/Performance) Contracted deliverables Semi-Annual Account Review
Service Highlights Focused Engineering Team of Oracle Subject Matter Experts Fast track solutions based on intimate Knowledge of customer
Environment and business needs Proactive Software Advice and Knowledge Transfers Performance and Patch Assessments Single point of contact for all technical issues - Technical Lead Single point of contact for all escalations – SDM 24/7 coverage
Solution Support Center
Value Mitigate Upgrade Risks substantially All project milestones are met Ensured optimal usage of Oracle features and functionality Reduced overall Maintenance/upgrade/ training costs
Focused Engineering Team Single Point Accountability –
SDM Exclusive 1 800 number
24x7x365 Review of Critical Issues as
Needed Patch Analysis
Stand-by Support for Critical Go-lives
Proactive Alerts / Patch Notifications
Project Planning / Migration Assistance
Periodic Review Sessions Applied Research
Key Delivered Features
Solution Support Center
Questions
For More Information
On this presentation, e-mail Bryon Rickey ([email protected]) or
Ravi Kayarthodi ([email protected]).
For general information on Advanced Customer Services, go to
oracle.com/acs
or e-mail