1 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
2 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
15785 - Advanced Administration and Management of Oracle SOA Suite 11g Samrat Ray, Ramkumar Menon, Srimant Misra Oracle SOA Suite Product Management and Development
ORACLE PRODUCT
LOGO
3 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
3 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
4 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Program Agenda
• Overview
• Provisioning and Configuration Management
• Managing a System in Production
• Transactions and Fault Handling
• Troubleshooting
• Q&A
5 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Overview
EJB
Optimized Service & Eventing
Infrastructure
Service Infrastructure Policy Manager
Policy Management
JCA HTTP B2B
Common Connectivity Infrastructure
…
… SOAP
BPMN BPEL Human Task
SCA Composite
SOA Composite Editor
BPEL
Human Task
Rules Mediation
BPEL orchestration
Business Rules
Human Tasks
Web services
and adapters
6 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Program Agenda
• Overview
• Provisioning and Configuration Management
• Managing a System in Production
• Transactions and Fault Handling
• Troubleshooting
• Q&A
7 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Overview SOA Domain
BAM Managed Server
SOA
EM
Managed Server
Administration Server
BAM
… Managed Server
Managed Server
EM
SOA
… Managed Server
Managed Server
Cluster 2: BAM
Cluster 1: SOA
Simple, one-machine install Typical SOA HA install
Cluster_Domain SOA_Domain
Oracle home (binaries)
Domain home (configuration)
Administration Server
8 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
• WLS Properties – Persisted in domain configuration files, changes are propagated across the
cluster by the Adminserver.
• Service Infrastructure (soa-infra) Properties – 11gR1 PS1: Properties are persisted in MDS (database), changes are
automatically propagated across a cluster.
• Composite Properties – Meta-data associated with a composite is stored in MDS (database) – These properties can be customized using a “Configuration Plan” at the time
of deployment. – Composite properties are automatically propagated across a cluster as part
of the composite deployment process.
3 Types of Properties SOA Configuration
9 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
• Configuration: – Stored under “config” directory – The config directory contains a file called config.xml which is
the primary source of all configuration information within a domain.
– SOA configuration is stored in MDS since PS1.
• Scripts: – Stored under “bin” directory. – Start / stop Admin & Managed server – Set environment and JVM properties
• Logs: – Server logs are placed under servers/<server_name>/logs
Directory Structure SOA Domain
configuration root
sub-directory for each managed server (logs / cache)
start/stop & env scripts
domain home
local cache of deployed composites
10 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
SOA 11g MDS Store
• Runtime Store – Composite meta-data – Security policies – Error policies – B2B Agreements – Shared WSDLs and XSDs – ADF / Human Workflow
• Can be accessed from JDeveloper and Ant.
• Configuration customization is stored in MDS.
11 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Configuration Propagation in a Cluster
WLS Configuration • File based • AdminServer has “gold” copy • Configuration propagated by WLS JMX
framework. • During startup, a Managed Server copies
configuration from AdminServer and overwrites local copy.
SOA Configuration • DB (MDS) based. • Database has “gold” copy • Configuration propagated by soa-infra
using the Coherence cluster. • During startup, a SOA server loads its
configuration from MDS.
SOA Cluster
SOA 1
SOA 2
Admin MDS
WLS Configuration SOA Configuration
WLS + Coherence Clusters
12 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
2 Levels of Clustering SOA 11g HA Architecture
• WLS Cluster – Defined in config.xml – Can be viewed from WLS Console and
EM
• SOA Cluster – Coherence based cluster – Defined as system parameters in startup
scripts – Can be validated through deployment /
configuration changes.
13 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
• SOA container configuration done through EM console / WLST
• Endpoint properties modified during deployment through a Configuration Plan – Created – Adapter properties
• WLS Console / WLST for managing WLS configuration – Domain creation, adding / changing resources within a domain.
Tools for Configuring SOA Applications
14 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
• Primary focus is green field movement.
• No virtualization required. • What is moved?
– Installed binaries and patches – Configuration and metadata – Security config e.g. OPSS policies
• Supports MDS meta-data, JCA connection factories
• MovePlan – Rewiring external references to match target – Data-sources, endpoint addresses
Minimize Manual Steps Test to Production (T2P) Utility
15 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Program Agenda
• Overview
• Provisioning and Configuration Management
• Managing a System in Production
• Transactions and Fault Handling
• Troubleshooting
• Q&A
16 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Production Readiness
Sign off on Qualifying Pre-Production Tests
Clearly defined and documented processes
Trained personnel lined up for Go-live and Post-Production
Availability of a Production Replica Environment
Identify SLAs and Key Transactions
The Readiness Checklist
17 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Tools for Middleware Management
EM FMW Control
EM Grid Control 11g
WLS Admin Console
JR Mission Control
WLDF
WLST
BAM
For Farm with a single WLS Domain. Composite lifecycle management, Instance Tracking, Performance Metrics, Fault and Message Recovery
Extends FMW Control for multiple farms. SOA/WLS Management Pack. Historical data, Alerts and Notifications
Monitor and manage the WLS domain and Managed Servers. Configure JDBC, JMS, JTA, Security, Deploy Applications etc.
JVM monitoring and management. Flight recorder for troubleshooting and diagnosis.
Generate and Analyze diagnostic data from WLS instances and your applications. Watches and Notifications. Integrate with JFR for troubleshooting.
Create/manage/monitor WLS Domains via Jython based WLI. WLS Admin Console/Configuration Wizard on a command line.
Real-Time Business dashboards. Correlate Business Indicators with System Metrics by integrating with EM GC 11g
18 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
EM FMW Control
BPEL
Human Workflow
Routing
Business Rules
External Service Drill-down
into component
s
•SOA Composite Lifecycle Management
•Transaction Search and Drill Down
• Fault and Message Recovery
• Service Engine Configuration
•Live In-flight DMS Performance Metrics
•System MBean Browser for advanced configuration and metrics
•Use FMW Control for searching and tracking your composite instances
19 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
FMW Control - Key Performance Tuning Tasks • Configure SOA-Infra Properties
• Audit/Instance Tracking(CPU,DB), Payload Validation (CPU)
• Configuring Service Engine threads • BPEL - Threads(Concurrency), Audit (DB),
Validation (CPU) • Mediator - Parallel/Resequencer Threads
(Concurrency), Audit (DB), Sleep Intervals/Metrics Level (CPU)
• Configure Adapter Properties • MaxRaise/TxnSize(DB), Number of
Threads (Concurrency)
• All this Accessible as MBeans - Update via WLST
20 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
EM Grid Control - SOA Management Pack EE • Central management of multiple SOA domains
• Historical data and trend analysis of SOA metrics
• Configure thresholds with alerts – Integrated with SNMP traps
• Scale monitoring and alert configuration using templates.
• Correlated view between system metrics and SOA composite instances.
• Extensible through shell / SQL scripts.
• Monitor custom rroups [apps, DBs] in one place.
• Use GC11g to track your composite execution flows ,monitor your application and system metrics, alert Ops or perform corrective action on critical issues.
Link to correlated JVM Diagnostics
Detected SOA component name
21 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
JRockit Mission Control • Negligible performance overhead. • Comprised of three toolkits.
– Management Console - for managing and monitoring one or more JVMs.
– Runtime Analyzer – Application and VM Profiler. Record and Playback for Root-cause analysis. • Analyze patterns around GC, CPU, Latency, Locks etc
– Memory Leak Detector
• Integrates with WLDF for analysis of diagnostic dumps
• Use JRMC to root-cause performance issues by analyzing memory, CPU, thread contention, locks etc via Flight Recordings.
22 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
WLST WLST
Online
Server Lifecycle
Composite Deployment and Lifecycle
MBean access and Updates
DomainMBean DomainRuntimeMBean
ServerRuntimeMBean
Custom(For SOA)
WLDF Integration to export
Diagnostic Data
Offline
Create Domain
Templates/Domains from Templates,
Extend Domains etc
.Jython based CLI Create, monitor and manage WLS Domains Runs in Interactive,Batch or Embedded Mode
• Command line automation .
• Hierarchical navigation via domain and server Mbeans: • retrieve and set domain metrics and properties • start/stop servers etc
23 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Purging
• Parallel
• Useful for medium [ ~500 GB total to ~10 GB daily] to large installations
• Resource intensive – hence run during off hours.DOP indicates number of parallel threads.
• For large installs [>~500 GB total or > ~10 GB daily], recommended to run a hybrid of parallel purging + dropping partitions, using optimal thread counts.
• Partitioning • Dropping partitions is the fastest way
to purge.
• Time-Range Partitioning Done at the component level.
• Decide time-range based on average and max composite lifetime.
• Run this in conjunction with parallel purging for a mixed load.
Via EM Console [For Targeted Deletion], or via Purge Scripts and/or dropping Partitions[For Bulk] Scripts located at RCU_HOME/rcu/integration/soainfra/sql/soa_purge
• Looping
• Useful for small installations [~100k rows]
• Loops and runs until max_runtime loop parameter is exceeded.
• Not resource intensive, but slow performance.
• All purgeable instances are deleted.
24 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Program Agenda
• Overview
• Provisioning and Configuration Management
• Managing a System in Production
• Transactions and Fault Handling
• Troubleshooting
• Q&A
25 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Fault Handling – Key Factors
• Handled Faults – Handled programmatically – Component / Policy – Can specify manual intervention through EM – Principal transaction not rolled back
• Un-handled Faults – Not handled programmatically – Typically leads to transaction rollback – Messages can be re-tried when they are retrievable
• JMS queue, DB table, Dehydration storeetc.
26 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
BPEL Service Engine Threads and Persistence
Receive
…
Receive
… sync process invoke executed by caller thread,
e.g. servlet
async process invoke executed by BPEL engine invoker thread
Msg persisted (DB)
Receive
…
After hydration (e.g mid-process receive activity for async callback) process executed by BPEL engine engine thread
(2nd) callback msg persisted (DB), triggers dehydration
(1st) process hydration
Reply
Sync Process Async Process
De/hydration
27 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
BPEL Service Engine Faults and Transactions
Receive
…
Receive
… Receive
…
exception or system crash rolls message back into DB, available to BPEL SE for manual (callback) recovery. composite and BPEL instance running
exception or system crash
returns exception (or timeout) to
client
exception or system crash rolls message back into DB, available to BPEL SE for manual (invoke) recovery. composite running but BPEL instance faulted; new instance initiated on recovery.
msg persisted (DB)
(2nd) callback msg persisted (DB), triggers dehydration
(1st) process hydration Sync Process Async Process De/hydration
* – A 3rd type of recovery is Activity recovery. See next slide.
28 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
BPEL Service Engine Faults and Transactions
<Activity>
…
Receive
Other types of activities can also trigger process dehydration
• Wait • bpelx:exec checkpoint(); • onMessage
exception or system crash rolls message back into DB, available to BPEL SE for manual (activity) recovery
29 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Mediator Service Engine Sequential vs. Parallel Routing
M
sequential (aka sync) routing rules are
executed by caller thread
M
(1) parallel (aka async) routing rules
persist messages into an internal DB table (executed by
caller thread)
(2) A single locker thread retrieves message batch from DB table into memory
Tune the locker thread batch size and # of worker threads for parallel routing rules
…
(3) separate worker threads process each message
30 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Mediator Service Engine Sequential vs. Parallel Routing
M
sequential (aka sync) routing rules are
executed by caller thread
M
(1) parallel (aka async) routing rules
persist messages into an internal DB table (executed by
caller thread) Recover Mediator parallel instances via EM console – Faults page for soa-infra, composite or Mediator SE
…
exception or system crash
returns exception (or
timeout) to caller
Exceptions during processing will cause message to roll back
into DB table, available for manual instance recovery
(3) separate worker thread processes each message
(2) A single locker thread retrieves message batch from DB table into memory
31 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Programmatic Fault Handling
• Fault Policies – Fault handling rules centralized in policy files (XML files) – Associated with specific a Reference, Component or Composite – Can define a conditions chain of recovery actions
• Recovery Actions – Retry – Human Intervention – recover from EM – Abort process – Replay scope – Re-throw fault – Java action – custom Java code called
Fault Policies
32 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Manual / Scripted Fault Handling - Recovery
• Recovery Options – Fault policies or BPEL fault
handlers for programmatic handling of faults • Can delegate to administrator
– BPEL message recovery for unhandled faults • Auto-recovery (recommended) / EM
based • Recover from BPEL recovery
console. – Rejected messages from inbound
adapters
Recovery Action
Recovery screen
Edit payload
Faults & exceptions
log
33 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Program Agenda
• Overview
• Provisioning and Configuration Management
• Managing a System in Production
• Transactions and Fault Handling
• Troubleshooting
• Q&A
34 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Troubleshooting SOA Oracle Fusion Middleware Diagnostic Framework
Diagnostic data
WLDF Thread Dumps
Automatic Diagnostic Repository
Command Line Utilities
Log Files
kill -3 pid WLST
Weblogic Admin Utilility
35
Configuring WLDF Watches in WLS Console • In-built mapping between Watch code
and dumps to create in ADR
36 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Troubleshooting SOA
[2011-09-22T06:50:07.986-07:00] [AdminServer] [NOTIFICATION] [] [oracle.integration.platform.blocks.deploy] [tid: [ACTIVE].ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 3c584fd020bdf1ba:-65df75b4:132916143aa:-8000-0000000000000166,1:23265:2] [APP: soa-infra] [Deployment] Component: SendMail was successfully deployed to service engine:implementation.bpel, for composite:SendMailApp
Anatomy of a Log File 1 2 3
4 5 6 7
8 9
1. Timestamp 2. Server Name 3. Message Type 4. Component ID
5. Thread ID 6. User ID 7. Execution Context ID 8. Supplemental Attribute
9. Message Data
37 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Troubleshooting SOA Editing Log Configuration
38 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Troubleshooting SOA Setting Logging Levels
Active Loggers Saved Loggers
Logger for SOA Adapters
Logger for BPEL
Logging Levels
39 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Troubleshooting SOA Viewing Log Files in EM
Message Pane
40 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Common Problems and Solutions
• Problem: ClassNotFound exceptions when SOA server is started by a Node Manager
• Solution: Add property StartScriptEnabled=true to nodemanager.properties.
Resolving common startup problems with SOA server • Problem: SOA Server (soa_infra)
does not start • Solutions:
• Check port conflicts • Make sure MDS schema is
reachable • Check coherence configuration
in clustered setup. Make sure that you’re using command line instead of active cache configuration
41 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Common Problems and Solutions
• Problem: ‘Too many open files’ during compilation or runtime.
• Solution: /etc/security/limits.conf to increase the value of both file descriptors to at least 4096.
Resolving “too many open files”
• Problem: Transaction or connection timeouts
• Solution: Increase JTA timeout in weblogic console
42 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Common Problems and Solutions
• Problem: Changes to connection factories JNDI (Configuration -> Outbound Connection Pool) for SOA adapters do not take effect .
Common application-related Problems
Hit ‘Enter’ Key after changing
Click ‘Save’ after the connection factory properties
have been updated
Click ‘Update’ after selecting the adapter
43 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Common Problems and Solutions
• Problem: Mis-configured Queues/Datasource in JMSAdapter and DBAdapter.
• Solution: Make sure that the runtime resources such as JMS connection factories, JMS Queues, DB datasources are configured properly in the WLS console
Common adapter-related Problems • Problem: Duplicate message
processing for non-transactional adapters (File/Ftp) in clustered setup.
• Solution: Use ‘eis/HAFileAdapter’ or ‘eis/ftp/HAFtpAdapter’ in jca configuration. Alternatively configure active-passive failover in the composite.xml file by configuring “standalone” property with ”true”
44 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
EM Performance Issue
• Issue: EM dashboard loads slowly / hangs on login.
• Step 1: Set default time duration for aggregate queries.
• Step 2: Disable auto-fetch of aggregated metrics.
45 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Q&A
46 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Latin America 2011 December 6–8, 2011
Tokyo 2012 April 4–6, 2012
47 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
48 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
49 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
APPENDIX