upgrade best practices 1273758143
TRANSCRIPT
<Insert Picture Here>
Upgrading to E-Business Suite R12.1 – Best Practices
Udayan Parvate, Director, EBS Release Engineering Lester Gutierrez, Senior Architect, EBS Applications Performance
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Objectives
• Provide an overview of the R12.1 release, the technology stack and supported upgrade paths
• Outline best practices for the R12.1 technical upgrade, with a focus on downtime reduction
• Review runtime performance monitoring and tuning tips
• Provide sample customer upgrade profiles and share Oracle’s GSI upgrade experience
Agenda
• R12.1 Upgrade Overview• Upgrade Best Practices• Customer Upgrade Snapshots• References• Additional Resources• Q&A
<Insert Picture Here>
R12.1 Upgrade Overview
R12.1 Rapid Install (RI)R12.1 Maintenance Pack (MP)
• R12.1 was generally available (GA) in May 2009– Via Oracle Electronic Product Delivery (EPD) and Oracle
Store– For all the supported platforms and languages
• Can be used by new and upgrading customers (11.5.9 and above) to go directly to R12.1– If you are on R11i, use R12.1 RI from EPD. Follow
instructions from the “Upgrade Guide: 11i to 12.1”– If you are on R12.0.X, use R12.1 MP (7303030) from My
Oracle Support (MOS). Follow instructions from “R12.1 Maintenance Pack Install Instructions (752619.1)”
R12.1 Technology Stack
9.2.0.6/9.2.0.6
1.0.2.2/ 1.4.2
6.0.8.25
Version included 11.5.10.2
10.2.0.3
10.1.3.0/1.5
10.1.2.2
Version included 12.0.4 RI
10.2.0.4/11.1.0.7
10.1.3.4/1.6.0
10.1.2.3
Minimum required version
for upgrade
11.2.0.1
10.1.3.5
-
Version certified with
10.1.2.3Apps Mid tier -Forms/Reports
10.1.3.4/1.6.0Apps Mid tier- Java Oracle Home / Apps Mid tier-JDK
11.1.0.7/11.1.0.7
Version included R12.1 RI
Database/JDBC drivers
Technology Component
Complete list at “Oracle E-Business Suite Release 12.1.1 Maintenance Pack Installation Instructions” (752619.1)
R12.1 Upgrade Paths
• Minimum EBS suite level for direct upgrade to R12.1– 11i9, 11i9CU1, 11i9CU2 or above – 11i10, 11i10CU1, 11i10CU2 or above– R12.0.0 and above
• Minimum EBS suite level required for database to be at – 10gr2 (10.2.0.4) : 11i9CU2, 11i10CU2– 11gr1 (11.1.0.7) : 11i10CU2, R12.0.4– 11gr2 (11.2.0.1) : 11i10CU2, R12.0.4
• One or more of the following choices apply based on your current configuration and your target configurationA. Upgrade database and EBS level in a single downtimeB. Upgrade database and EBS level in separate downtimesC. Apply interoperability patches and then upgrade apps level
R12.1 Upgrade Paths Continued…
NANANANANA11.2.0.1
NANANANANA11.1.0.611.1.0.7
6633NC10.2.0.4
5522110.2.0.110.2.0.210.2.0.3
NCNC2219iR2
12.0.4 or above
12.0.3 or below
11i10CU211i9CU211i9/CU111i10/CU1
12.1.1 @ 10.2.0.4
NC Not CertifiedNA Not Applicable
R12.1 DB guide (761570.1)
1. Single Downtime,Use RI2. Separate downtime, Use RI3. Apply interop patches, Use RI4. Single downtime, Use MP5. Separate downtime, Use MP6. Apply interop patches, Use MP
R12.1 Upgrade Paths Continued…
NC Not CertifiedNA Not Applicable
R12.1 DB guide (761570.1) NANANANANA11.2.0.1
6NC3NCNC11.1.0.611.1.0.7
542NCNC10.2.0.4
NCNC4
NCNC4
2NCNC10.2.0.110.2.0.210.2.0.3
NCNC2NCNC9iR2
12.0.4 and
above
12.0.3 and
below
11i10CU211i9CU211i9/CU111i10/CU1
[email protected] 1. Single Downtime,Use
RI2. Separate downtime, Use RI3. Apply interop patches, Use RI4. Single downtime, Use MP5. Separate downtime, Use MP6. Apply interop patches, Use MP
R12.1 Upgrade Paths Continued…
NC Not CertifiedNA Not Applicable
R12.1 DB guide (761570.1) 6NC3NCNC11.2.0.1
5NC2NCNC11.1.0.611.1.0.7
542NCNC10.2.0.4
NCNC4
NCNC4
2NCNC10.2.0.110.2.0.210.2.0.3
NCNC2NCNC9iR2
12.0.4 or above
12.0.3 or below
11i10CU211i9CU211i9/CU111i10/CU1
[email protected] 1. Single Downtime,Use
RI2. Separate downtime, Use RI3. Apply interop patches, Use RI4. Single downtime, Use MP5. Separate downtime, Use MP6. Apply interop patches, Use MP
<Insert Picture Here>
Upgrade Best Practices
Upgrade Best Practices
• Identification of required patches– Prepare a complete list of pre and post patches and
recommended code levels• Keep the system current on AD/ATG/OAM code e.g.
latest AD/ATG RUPs on 11i/R12.0 and once on R12.1• High priority patches from MOS• Consolidated Upgrade Patches (CUP) and required one-
offs that need to be applied in pre-install mode. R12.1 CUP1 (7303029:12.1.0) is available now
• Review “Known-issues” sections from key documents to determine additional pre or post upgrade patches : Release notes, MP Install Instructions
• Patch merging and sequencing– Take advantage of patch merge, hot patching and apply
patches in the correct sequence• Merge patches• Merge NLS patches per language. Apply NLS patches as
“hot” patches • Perform uptime maintenance when possible e.g. hot
patching of iHelp or NLS patches, upload patch history• HRGLOBAL step before NLS ( Document 145837.1 &
414434.1 )
Upgrade Best Practices Continued…
• Adpatch options and other reporting tools– Use applicable adpatch options to reduce downtime and
track key upgrade aspects using in-built tools• Use non-interactive patching• Use adpatch options such as nomaintainmrc, phtofile,
nocompiledb, nolink, nogenform, nogenrep, nocompile jsp, noautoconfig, novalidate
• Use optimal values for batch size and #workers • Use OAM reports and Patch Wizard functionality to track
patch timing, files applied, functionality patched, impacted customized files etc
Upgrade Best Practices Continued...
• Downtime reduction procedures– Employ generic downtime reduction techniques
• Use “Shared APPL_TOP” with “Distributed AD” for upgrades and regular maintenance for multi-node instances– Shared file system avoids the need to apply the same patch
multiple times– Distributed AD adds to the degree of parallelism by
distributing AD workers across application tier nodes and improves D/G portion of the patch driver.
• Use “Staged APPL_TOP” for regular maintenance– Saves time to patch the file system (C/G portion) by using a
patched up copy of production instance file system– NOT certified/supported for 11i => R12.1 upgrade– Can use for R12.0.X => R12.1 upgrade and once on R12.1
Upgrade Best Practices Continued..
• Pre-upgrade activities– Identify and execute tasks that could be completed in a
separate downtime period, prior to the production upgrade• Use applicable steps mentioned in the "Downtime
reduction" and “Upgrade By Request” appendices E and G of the R12.1 upgrade guide
• Upgrade RDBMS version to latest certified for the current APPS level ( 11.2.0.1 / 11.1.0.7 / 10.2.0.4 )
• Convert to Oracle Applications Tablespace Model (OATM)
– Compacts database, optimizes storage settings and reduces I/O
• Other planned HW or OS upgrades
Upgrade Best Practices Continued...
• Pre-upgrade activities continued...• Use TUMS (“The Upgrade Manual Script”) to avoid
running tasks that are not relevant for your system • Convert to multiple organization architecture (Document
210193.1) • Drop MRC schema ( 11.5.10 and above )• Gather stats with GATHER_AUTO option for all schemas• Assign post upgrade jobs to specialized CM queue (by
request_type, see Document 399362.1)
Upgrade Best Practices Continued...
• Pre-upgrade activities continued...• Pre-create new large indexes created by upgrade
(constrained to indexes on existing columns)• Minimize historical data to be upgraded as per business
requirements – “Upgrade By Request”• Purge old and/or transient data before upgrading
– Over 260 standard purge programs in R12– Use new “Purge Portal” in OAM to administrate
Upgrade Best Practices Continued...
• Purging– Use OAM to configure, initiate and monitor purge programs
• Set the execution frequency and view program history– R12 purge programs tagged with the “Purge” program type– Access via: System Administrator > Oracle Applications Manager
>Purging/Critical Activities
Upgrade Best Practices Continued...
• Key recommended fixes – Affecting Upgrade Performance
• AD-Parallel improved scalability: 8917381 (Part of 9179588 AD CUP)
• Forms/reports generation regression with 11g: 8557019– General
• Review documents 244040.1 and 950896.1 for latest EBS recommended performance fixes
Upgrade Best Practices Continued...
• Database tier configuration– Maximize SGA and PGA sizing
• An upgrade only involves 10's of concurrent sessions, starting rules of thumb ...– log buffer = 30 to 100 Mb– shared pool = 1 to 4 Gb– pga target = 3 to 20 Gb– buffer cache = multi Gb, be generous without causing
excessive paging or swapping• Adjust with help from AWR pool advisories
Upgrade Best Practices Continued…
• Database tier configuration– Other upgrade specific init.ora changes
• If specified, remove db_file_multiblock_read_count – Maximize multiblock I/O sizes
• Set job_queue_processes = # of CPUS– adobjcmp.sql (Phases : plb+90 and last+63)
• Set parallel_max_servers = 2 X CPUs– Helps with large index creation, stats gathering and
some large upg+ phase jobs– Need to test with production-like DB server and I/O
subsystem– Shutdown other RAC instances
Upgrade Best Practices Continued...
• Batch size and #workers– Batch size
• 10K is suitable for most installs, you can test other values from 1K up to 100K if time allows
– #Workers• Starting rule-of-thumb is between 1 and 1.5 x #CPUs
– It is critical to do multiple rounds of testing, adjusting above settings to maximize server utilization, but constrained by factors such as
• Memory utilization (no swapping/ excessive paging)• CPU utilization (scale down if at 100%)• I/O response times (scale down if averages > 20 ms)
Upgrade Best Practices Continued...
• Performance testing, monitoring and some unconventional optimizations...
– Analyze long runners via timing report, ad_task_timing analysis
– Mine ad_task_timing to identify low worker utilization due to phasing waits and review responsible culprits
– Review targeted AWR Instance and SQL reports– awrrpt.sql and awrsqrpt.sql
– Use My Oracle Support to check for known issues and workarounds for the longest running jobs
Upgrade Best Practices Continued...
• Performance testing, monitoring and some unconventional optimizations continued...
– Outside-the-box optimizations• Only consider after all previously mentioned optimizations
* Requires Oracle Support approval *• Long running index creation and stats gathering
– As these tasks run with little or no concurrent tasks, consider using alter system commands to increase parallel_max_servers and pga_aggregate_target
– Need to test so as to not exhaust machine resources• For indexes: Pre-create, but do not alter column definitions• For stats: On RAC use all nodes when gathering stats
Upgrade Best Practices Continued...
• Performance testing, monitoring and some unconventional optimizations continued...
– Outside-the-box optimizations• Long running adsstats.sql (Phase: last+63) • Long running mv related xdf’s (“en” phases)
– Cleanup and truncate large MV logs (requires subsequent MV complete refresh)
• Look more closely at long running jobs that you suspect may not be required for your system
– Thousands of jobs, thousands of customer product/data/setup configurations
– Some of the heavy lifting work may not be required for a specific install
– Check with Oracle Support to validate– A fix maybe available or possible to optimize for your case
Upgrade Best Practices Continued...
• Runtime Performance Testing Tips– Use Automated, scripted tools
• EBS Test Started Kits (Winrunner/QTP)– Bundled QA based automated scripts for EBS testing -
Patch 8408886• Oracle Applications Testing Suite (Accelerators for EBS)
– Web and Forms based flows– Complement with user participation tests and batch load tests
with frequent and critical jobs– References
http://blogs.oracle.com/stevenChan/2009/10/oats_ebs_certified.html
http://blogs.oracle.com/stevenChan/2009/08/evolutionary_steps_automated_testing_ebs.html
Upgrade Best Practices Continued...
• Runtime Performance Testing Tips– Review the white paper: Upgrade to Oracle 11g -
Performance Best Practiceshttp://www.oracle.com/apps_benchmark/html/white-papers-e-business.html
• Points to note include11g features for – Testing runtime performance of 11g or 10g
workloads --Real Application Testing– Managing SQL Execution Plans and minimizing
performance degradation -11g SQL Plan Mgmt– Review presentation: Tuning the Oracle E-Business Suite
Environmenthttp://blogs.oracle.com/stevenChan/2009/11/ebs_tuning_oow09.html
– Review EBS Benchmark Publications http://www.oracle.com/apps_benchmark/html/white-papers-e-business.html
Upgrade Best Practices Continued...
<Insert Picture Here>
Customer Upgrade Snapshots
Oracle E-Business Suite Release 12.1 Live Customers
Customer Upgrade Snapshots
• CPS (Chicago Public Schools)– Release: 11.5.10.2 to 12.1.1– DB size: 900GB– #Workers and batch size: 32, 10000– #CPUs on DB server: 2 node RAC, 8 CPUs per node– Downtime reduction measures
• Distributed AD with shared APPL_TOP• Upgrade RDBMS to 10.2.0.4 in a separate downtime prior
to production– #hrs for the D driver: 22 hrs– Customer snapshot
http://www.oracle.com/customers/snapshots/chicago-public-schools-ebs-snapshot.pdf
Customer Upgrade Snapshots Continued...
• Zebra Technologies Corporation– Release: 12.0.6 to 12.1.1– DB Size: 106GB– #Workers and batch size: 32, 10000– #CPUs on DB server: 8– Downtime reduction measures
• Staged APPL_TOP– #hrs for the U driver: 12 hrs– Customer snapshot
http://www.oracle.com/customers/snapshots/zebra-technologies-corporation-ebs-snapshot.pdf
Customer Upgrade Snapshots Continued...
• Cisco– Release: 11i to R12– DB size: 600GB– #Workers and batch size: 32, 20000– #CPUs on DB server: 16– Downtime reduction measures
• Distributed APPL_TOP– #hrs for the D driver: 5.5 hrs
Customer Upgrade Snapshots Continued...
• Oracle GSI– Release: R11i10CU2 to 12.0.3– DB size: 12TB– #Workers and batch size: 60, 10000 – #CPUs on DB server: 88 processors– Downtime reduction measures
• Ran data fixes for problems found in test upgrades prior to production upgrade to minimize stoppages
• Distributed APPL_TOP (4 servers - 15 workers each)• Added plsql_nocompile option in the upgrade driver
– #hrs for the D driver: 80 hrs (8 yrs of SLA data)
Customer Upgrade Snapshots Continued...
• Oracle GSI– Release: 12.0.3+ to R12.1– DB size: 17TB– #Workers and batch size: 60, 10000 – #CPUs on DB server: 88 processors– Downtime reduction measures
• Staged APPL_TOP for US and ten languages • Ran data fixes for problems found in test upgrades
prior to production upgrade to minimize stoppages• Distributed APPL_TOP (4 servers - 15 workers each)• Pre-create large indexes (6 hrs savings)
– #hrs for the D driver: 14 hrs
<Insert Picture Here>
References
References
• R12.1 documentation roadmap (790942.1)• “Oracle E-Business Suite Release 12.1 Info center” (806593.1)• Database preparation guidelines for R12.1 upgrade (761570.1)• Patching FAQs (459156.1, 225165.1)• Using staged or shared APPL_TOP and distributed AD
(734025.1, 384248.1, 236469.1)• OAM “Patch Wizard” overview and FAQ (976188.1, 976688.1)• AD Command Line Options for Release R12 (1078973.1)• Recommended Performance Fixes (244040.1, 950896.1)• R12 Upgrade Sizing & Best Practices (399362.1)
<Insert Picture Here>
Additional Resources
Additional Resources
• White paper– Planning Your Oracle E-Business Suite Upgrade from Release 11i
to Release 12.1 (987516.1)– R12 Upgrade considerations by product: Financials (889733.1)
• Have upgrade questions ? – Post on OTN R12 upgrade forumhttp://forums.oracle.com/forums/forum.jspa?forumID=395&start=0
Join us in San Francisco for Oracle OpenWorld and JavaOne + Develop
• September 19-23, 2010• San Francisco, CA • For more information or to register go to
www.oracle.com/openworld
Oracle Products Available Online
Oracle Store
Buy Oracle license and support online today at
oracle.com/store
Q & A