Copyright © 2012, Oracle and/or its affiliates. All rights reserved.1
Fusion Apps System Administration
Ulrich JankeOracle Fusion Applications Architects (A-Team)
DOAG Business Apps Conference 2013 in Berlin11.10.2013
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.3
• Introduction
• Fusion Apps System Monitoring
• Fusion Apps Lifecycle ManagementInstallationUpgradeCloningPatching
Agenda
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.4
Fusion Applications: A Single Code Line
Remote Management
Hosted & Managed
Cloud / SaaSCombinations
On Premise
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.5
Oracle Fusion Applications MiddlewareIntegrated, Best-of-Breed
Infrastructure & Management
Database
Middleware
Applications
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.6
• Introduction
• Fusion Apps System Monitoring
• Fusion Apps Lifecycle ManagementInstallationUpgradeCloningPatching
Agenda
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.7
Sample TopologyHere: Fusion CRM
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.8
Fusion Apps Full Installation Components
Fusion Apps in Full Installation• 9 Domains ( + 1 IDM Domain)
9 Admin Servers (one each domain)39 Application Managed Servers28 Middleware Managed Servers
• BI Tech StackBI PublisherInformatica IRPresentation Service
• Oracle HTTP Server InstancesGlobal Order PromisingCommon WebtierBI Webtier
• More components may appear in future releases
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.9
Fusion Apps MonitoringFusion Apps Start Script Output
Question: Did all services start accordingly?
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.10
Fusion Apps MonitoringCheckpoint #1: WLS Console
Checklist• For each of 9 domains
Are Managed Servers up and running?
• No status information about Oracle HTTP Server
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.11
Fusion Apps MonitoringCheckpoint #2: Domain Enterprise Manager
Checklist• For each of 9
domains:Detailed Status about all components in a domain
• Control functionality (start, stop etc) for dedicated components (WLS servers, OHS etc)
• Overview of deployed Apps, ESS, SOA etc
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.12
Fusion Apps MonitoringProduct Overview and Relationship
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.13
Fusion Apps MonitoringUltimate monitoring solution: Enterprise Manager Cloud Control 12c
Benefits• All domains at a
glance
• Existing Fusion Apps Adapter
• Straight forward FA registration
• Complete collection of FA relevant data
• Invocation of domain WLS or EM console if required
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.14
Fusion Apps MonitoringEnterprise Manager Cloud Control 12c
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.15
Fusion Apps MonitoringEnterprise Manager Cloud Control 12c
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.16
• Introduction
• Fusion Apps System Monitoring
• Fusion Apps Lifecycle ManagementInstallationUpgradeCloningPatching
Agenda
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.17
Maintain Fusion Applications (PS/RUP/Patch)
Enable languages
Fusion Applications Lifecycle
Installation1. Host Preparation2. DB Installation3. Provision IDM4. Provision FA
Scale Out and High Availability
Maintain Fusion Applications (PB/Patch)
Enable languages
Cloning
Upgrade Fusion Applications
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.18
• Introduction
• Fusion Apps System Monitoring
• Fusion Apps Lifecycle ManagementInstallationUpgradeCloningPatching
Agenda
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.19
Fusion Applications Provisioning Flow (R7)
• Hosts• Network• Storage• Security
Configure
• Databases(IDM, FA)
• DB Patches• Repository
- Middleware- Fusion Apps- Provisioning
DB Install
• Plan• Provisioning• Test
IDM Provisioning
• Plan• Pre-verify• Install• Pre-configure• Configure• Post Configure• Startup
FA Provisioning
• Post-Install• Languages • Validate / Test
Post -Provisioning
• Setup • Patching • Upgrades• Cloning
Operations
Configuration Prerequisites Installation Post
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.20
ProvisioningPlan Generation
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.21
ProvisioningFinalization Plan Generation
Next Steps• Start Provisioning on Primordial Host
• In case multiple FA hosts exist: Run Single Tasks on other hosts when prompted
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.22
• Introduction
• Fusion Apps System Monitoring
• Fusion Apps Lifecycle ManagementInstallationUpgradeCloningPatching
Agenda
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.23
Fusion Apps UpgradesPrinciples
• Fusion Apps Upgrades are major steps in lifecycle of a system• Move Fusion Apps V1 to new releases
...R4 → R5 (11.1.5)R5 → R6 (11.1.6)R6 → R7 (11.1.7)...
• Complete set of actionsTech Stack Upgrade to new FMW releases like 11.1.1.6
- FA Middleware (WLS, SOA, ADF runtime, WC ...)- IDM - BI
Fusion Apps functional patching (cumulative)Exception: No DB patching
• Once a CY (max twice in past)• Careful preparation required and planned downtime• Chained Upgrades!
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.24
Fusion Apps UpgradesR7 Upgrade Flow
• New Flow established with R7 upgrade
• Upgrade Flow has downtimes and breaks for backups
• Mix of manual tasks and RUP installer invocations for dedicated components
• At dedicated points in flow some component specific Installers have to be executed
- RUP Installer
- Health Checker Utility
- RUP Lite for OVM Utility (in OVM
environments)
- RUP Lite for OHS Utility
- RUP Lite for BI Utility
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.25
Fusion Apps UpgradesR7 Upgrade Actions
• Upgrade preparation incl. backups and pre-upgrade patching• Manual tasks according to system topology and configuration• FA patches are invoked by RUP Installer (new features and bug fixes)• Installers are invoked by Middleware Patch Sets configuration assistant (tech stack):
Oracle Business IntelligenceOracle CommonOracle Data Integrator (ODI)Oracle Database ClientOracle Enterprise Content ManagementOracle HTTP Server (OHSOracle Fusion Middleware Extensions for ApplicationsOracle Global Order PromisingOracle Identity Management (IDMUTIL)Oracle Secure Enterprise Search (SES)Oracle SOA SuiteOracle Social Networking (OSN)Oracle WebCenter SuiteOracle WebLogic ServerOracle Web Tier
• More Info: Fusion Apps R7 Upgrade Doc
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.26
• Introduction
• Fusion Apps System Monitoring
• Fusion Apps Lifecycle ManagementInstallationUpgradeCloningPatching
Agenda
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.27
Fusion Apps Cloning and Data MovementPrinciples
Major activity in lifecycle of a Fusion Apps implementation - two features available:1. Production-to-Test (P2T)
Not a „true“ clone – data movement between instancesFeeding an installation with data from another installation
2. Cloning„True“ clone of an FA Installation completelyDuplication of an imageHelps preserving a certain status of an image for testing
- Master Image → Test (CRP, specific purposes etc)- CRP → UAT- Master Image → Dev- Master Image → Prod- Prod → Test- ...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.28
Fusion Apps Data MovementProduction-to-Test (P2T)
• Movement of application data from source to target Fusion Applications installation• Common use case is the refreshing of a test database with production data• P2T can also be used to move data between any two environments (production, staging, testing, etc.)• Two phases in moving data in a Fusion Applications installation:
1) moving the Identity Management Identity and Policy Store data, and 2) moving data from the Fusion Applications transaction databases
Identity Management Policy Store data (application and system policies, but not credentials and keys)Identity Management Identity Store data (not including AppID and user passwords)Fusion Applications transaction data and the crawl index stored in SESFile attachments stored in UCM (such as orders, agreements)ADF Customizations (such as Flex Fields), SOA and ESS customizations stored in MDSBusiness Intelligence (BI) Web Catalog and RPDODI repositoryWebCenter contents
• P2T movement replaces most of the target database with production data• just a small category of data on the test/target system is preserved, as required by the system. • target environment is reconfigured and rewired at the end• All long-running processes on the target are stopped and purged, in order to prevent the non-production
system from sending emails and alerts to real users, as if it were the production system.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.29
Fusion Apps CloningCloning Principles
• Cloning solution has three major activities 1. creating master image copies of all Fusion Applications components in an instance2. mounting them to a matching array of destination hardware3. running the cloning tools (faclone.sh) to "rewire" the destination environment.
• Script faclone.sh executes a set of scripts and tools • These scripts and tools make changes in destination environment:
Password changesRewiring for new database configurationCleanup of in-flight processesRewiring for new Fusion Applications external URLs
• ScopingDuplication 1:1 – no topology changesDatabase activities not handled by Cloning ToolsUsage of abstract and virtual hostnames
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.30
Fusion Apps CloningCloning Activities 1/2
• DiscoveryFeeding Discovery Workbook (Excel) with configuration dataGenerate Response File (.rsp) out of these dataFuture: Use Discovery Scripts
• Creating a Master Image of the SourceManual process to create a .tar image of entire FA Source treeIf using a virtualized environment then entire VM can be copied
• Preparing Target Environment Out of sequence activity till herePreparation of an identical target host
• Making Master Images Accessible to TargetCopy inside or to shared file systemMust be accessible by Cloning Tools and target environment
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.31
Fusion Apps CloningCloning Activities 2/2
• Duplicating the Database(s) from Source to TargetManual activity to duplicate a source DB to a new target DBRAC supportedTopology changes not supported – manual activity out of scope if required (RAC → Non-RAC etc)
• Running the faclone.sh Script• The tool as is to perform cloning steps• Following script targets to run in that order
1. IDM Middle Tier (IDMMT)2. IDM Web Tier (IDMWT)3. FA Middle Tier (FAMT)4. FA Web Tier (FAWT)
• Performing Validation Steps• Performing Post-Clone Cleanup
More Information: Fusion Apps R7 Cloning and Content Movement Administrator's Guide
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.32
• Introduction
• Fusion Apps System Monitoring
• Fusion Apps Lifecycle ManagementInstallationUpgradeCloningPatching
Agenda
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.33
Frequency Semi-annual to annual Monthly to quarterly On-demand or as needed
Content Entire suite Product family Product family
Delivery mechanism
OUI FA Patch Mgr FA Patch Mgr
Translation Released concurrently Released concurrently
Need not be concurrent (partial translation supported)
Restriction None Fixes within one product family
Complex changes not supported (new ear/changed wiring, etc.)
Fusion Applications Patch Types - Overview
In general, apart from laying out the bits, patching framework is expected to accomplish configactivities. For complex changes like new ears & wiring changes, patching framework will not automate those config activities. So these changes must only be done in Patchsets or RUPs
With respect to shared code & contracts, there are limitations on code changes if product family code is going to be maintained independently; With respect to complex change like new ears and changed wirings, Fusion Apps will be asked to provide custom code to enable provisioning of complex changes
Cross product family RUP
Product family Patchset Patches (Standard or One-off)
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.34
Fusion Apps Snowball DefinitionPatches are cumulative at the .jar level e.g. adf-lib including dependencies
• P1 – fix in version 1 of lib A • P1 contains version 1 of lib A
• P2 – fix in version 1 of lib B • P2 contains version 1 of lib B
• P3 – fix in version 2 of lib A and version 1 of lib C – Libs A and C now linked for all future patches to either
• P3 contains version 2 of lib A and version 1 of lib C
• P4 – fix in version 2 of lib C • P4 contains version 2 of lib A and version 2 of lib C – Even though P4 only changed lib C, lib A pulled in because of the
dependency created in P3
• P5 – fix in version 2 of lib B • P5 contains version 2 of lib B
• P6 – fix in version 1 of lib D and version 3 of lib B • P6 contains version 1 of lib D and version 3 of lib B
Sequence of fixes Contents of the patch
Note that some patches pull in other fixes. For instance, if a customer asked for the fix contained in P4, they are also getting fixes for P1 and P3
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.35
FA Patch Manager Leverages Various Existing Tools for Unified Patching Experience
FA PatchBuild/ADE
ARU
Package and addpatch metadata
MOS
Oracle Customer
File ArtifactsC ArtifactsJava ArtifactsBI, JPS
Opatch
FA Layer(Owrapper)
DB Artifacts
Autopatch
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.36
Fusion Apps Patching at Customer
• Fusion Apps Patching must coordinate:– Database artifacts –- Autopatch– Middleware artifacts produced by
Fusion Apps – FA Layer/Opatch– And be lightweight
Opatch
DB Artifacts
File ArtifactsC ArtifactsJava Artifacts
Autopatch
BI, JPS
FA Layer
FA Patch Mgr
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.37
Use Case: Mixed FMW and Database PatchCustomer wants to apply a patch for a CRM specific ADF library and a seed data file for the affected LBA
Patching simple middleware and database artifacts– Patch is downloaded from ARU/MOS and loaded into patch staging area and
submitted
– FA Patch Mgr interprets patch content
– Patch content routed to Autopatch and FA LayerAutopatch calls Seed Data Framework (SDF)
Seed data updated by SDF
FA Layer creates patch plan and calls Opatch
Opatch applies jar file change
– Each patching tool reports back to FA Patch Mgr
– Combined results returned to user
Opatch
DB Artifacts
ADF Library
Autopatch
FA Layer
FA Patch Mgr
Apps Patching UI
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.38
Questions
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.39
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.40