1 copyright © 2011, oracle and/or its affiliates. all ... · • monitor custom rroups [apps, dbs]...


15785 - Advanced Administration and Management of Oracle SOA Suite 11g Samrat Ray, Ramkumar Menon, Srimant Misra Oracle SOA Suite Product Management and Development

ORACLE PRODUCT

LOGO



Insert Information Protection Policy Classification from Slide 8

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Program Agenda

• Overview

• Provisioning and Configuration Management

• Managing a System in Production

• Transactions and Fault Handling

• Troubleshooting

• Q&A


Overview

EJB

Optimized Service & Eventing

Infrastructure

Service Infrastructure Policy Manager

Policy Management

JCA HTTP B2B

Common Connectivity Infrastructure

…

… SOAP

BPMN BPEL Human Task

SCA Composite

SOA Composite Editor

BPEL

Human Task

Rules Mediation

BPEL orchestration

Business Rules

Human Tasks

Web services

and adapters


Program Agenda

• Overview




• Troubleshooting

• Q&A


Overview SOA Domain

BAM Managed Server

SOA

EM

Managed Server

Administration Server

BAM

… Managed Server

Managed Server

EM

SOA

… Managed Server

Managed Server

Cluster 2: BAM

Cluster 1: SOA

Simple, one-machine install Typical SOA HA install

Cluster_Domain SOA_Domain

Oracle home (binaries)

Domain home (configuration)

Administration Server


• WLS Properties – Persisted in domain configuration files, changes are propagated across the

cluster by the Adminserver.

• Service Infrastructure (soa-infra) Properties – 11gR1 PS1: Properties are persisted in MDS (database), changes are

automatically propagated across a cluster.

• Composite Properties – Meta-data associated with a composite is stored in MDS (database) – These properties can be customized using a “Configuration Plan” at the time

of deployment. – Composite properties are automatically propagated across a cluster as part

of the composite deployment process.

3 Types of Properties SOA Configuration


• Configuration: – Stored under “config” directory – The config directory contains a file called config.xml which is

the primary source of all configuration information within a domain.

– SOA configuration is stored in MDS since PS1.

• Scripts: – Stored under “bin” directory. – Start / stop Admin & Managed server – Set environment and JVM properties

• Logs: – Server logs are placed under servers/<server_name>/logs

Directory Structure SOA Domain

configuration root

sub-directory for each managed server (logs / cache)

start/stop & env scripts

domain home

local cache of deployed composites


SOA 11g MDS Store

• Runtime Store – Composite meta-data – Security policies – Error policies – B2B Agreements – Shared WSDLs and XSDs – ADF / Human Workflow

• Can be accessed from JDeveloper and Ant.

• Configuration customization is stored in MDS.


Configuration Propagation in a Cluster

WLS Configuration • File based • AdminServer has “gold” copy • Configuration propagated by WLS JMX

framework. • During startup, a Managed Server copies

configuration from AdminServer and overwrites local copy.

SOA Configuration • DB (MDS) based. • Database has “gold” copy • Configuration propagated by soa-infra

using the Coherence cluster. • During startup, a SOA server loads its

configuration from MDS.

SOA Cluster

SOA 1

SOA 2

Admin MDS

WLS Configuration SOA Configuration

WLS + Coherence Clusters

Presenter

Presentation Notes

WLS Properties Persisted in domain configuration files, changes are propagated across the cluster by the Adminserver. Service Infrastructure (soa-infra) Properties Properties are persisted in MDS (database), changes are automatically propagated across a cluster. Composite Properties Meta-data associated with a composite is stored in MDS (database) These properties can be customized using a “Configuration Plan” at the time of deployment. Composite properties are automatically propagated across a cluster as part of the composite deployment process.


2 Levels of Clustering SOA 11g HA Architecture

• WLS Cluster – Defined in config.xml – Can be viewed from WLS Console and

EM

• SOA Cluster – Coherence based cluster – Defined as system parameters in startup

scripts – Can be validated through deployment /

configuration changes.


• SOA container configuration done through EM console / WLST

• Endpoint properties modified during deployment through a Configuration Plan – Created – Adapter properties

• WLS Console / WLST for managing WLS configuration – Domain creation, adding / changing resources within a domain.

Tools for Configuring SOA Applications


• Primary focus is green field movement.

• No virtualization required. • What is moved?

– Installed binaries and patches – Configuration and metadata – Security config e.g. OPSS policies

• Supports MDS meta-data, JCA connection factories

• MovePlan – Rewiring external references to match target – Data-sources, endpoint addresses

Minimize Manual Steps Test to Production (T2P) Utility


Program Agenda

• Overview




• Troubleshooting

• Q&A



Production Readiness

Sign off on Qualifying Pre-Production Tests

Clearly defined and documented processes

Trained personnel lined up for Go-live and Post-Production

Availability of a Production Replica Environment

Identify SLAs and Key Transactions

The Readiness Checklist

Presenter

Presentation Notes

Completion of all qualifying pre-production tests such as Functional, UAT, Load tests. Clearly defined and documented processes for administrative tasks such as deployment/rollback, purge, message recovery, patching, server maintenance, server restarts, backup/restore, hardware and software upgrades, capacity management, Scale-ups. Personnel “trained” and readily lined-up for go-live transition and post-production support, including DBAs, Network Administrators, System Administrators etc, with appropriate access control. Have a Production Replica Environment for playback, debugging and analysis. Also familiarity with diagnostics and related utilities/scripts. Know the key transactions/SLA that impact revenue and/or are time-sensitive in nature. Make sure monitoring is in place for these ones.



Tools for Middleware Management

EM FMW Control

EM Grid Control 11g

WLS Admin Console

JR Mission Control

WLDF

WLST

BAM

For Farm with a single WLS Domain. Composite lifecycle management, Instance Tracking, Performance Metrics, Fault and Message Recovery

Extends FMW Control for multiple farms. SOA/WLS Management Pack. Historical data, Alerts and Notifications

Monitor and manage the WLS domain and Managed Servers. Configure JDBC, JMS, JTA, Security, Deploy Applications etc.

JVM monitoring and management. Flight recorder for troubleshooting and diagnosis.

Generate and Analyze diagnostic data from WLS instances and your applications. Watches and Notifications. Integrate with JFR for troubleshooting.

Create/manage/monitor WLS Domains via Jython based WLI. WLS Admin Console/Configuration Wizard on a command line.

Real-Time Business dashboards. Correlate Business Indicators with System Metrics by integrating with EM GC 11g

Presenter

Presentation Notes

Monitor/administer a farm with a single WLS Domain. Used for composite lifecycle management, Instance Tracking, System Performance Monitoring (DMS), message and fault recovery, log configuration and more. �Extends FMW Control for centralized monitoring/management of multiple farms. End-to-End monitoring/management with SOA/WLS Management Pack. Also provides historical data, alerts and notifications Web based interface to monitor and manage the WLS domain. Runs on the Admin Server. Use to start/stop managed servers, configure JDBC, JMS, JTA, Security, Deploy Applications etc. Excellent for JVM diagnostics and as a flight recorder for troubleshooting and diagnosing issues. Also a JMX Client, and hence can consume and display WLS and SOA MBean attributes. Used to generate and analyze diagnostic data from WLS instances and your applications. Configure watches and send notifications to signal key diagnostic states in your system. Integrate with JFR for troubleshooting. Used to generate and analyze diagnostic data from WLS instances and your applications. Configure watches and send notifications to signal key diagnostic states in your system. Integrate with JFR for troubleshooting. Jython based CLI environment to create/manage/monitor WLS Domains. Uses in offline mode for creating/extending domains etc, and in online mode for managing active domain configurations, view performance data etc. WLST online is a JMX Client – can retrieve WLS and SOA MBean attributes. Typically used for real-time business activity monitoring via dashboards that display real-time data-flow. Integrates with EM GC 11g to display metric/alert data that can be correlated with business indicators, and for better reporting, event notification and corrective actions.



EM FMW Control

BPEL

Human Workflow

Routing

Business Rules

External Service Drill-down

into component

s

•SOA Composite Lifecycle Management

•Transaction Search and Drill Down

• Fault and Message Recovery

• Service Engine Configuration

•Live In-flight DMS Performance Metrics

•System MBean Browser for advanced configuration and metrics

•Use FMW Control for searching and tracking your composite instances

Presenter

Presentation Notes

SOA Composite Lifecycle Management Search and find transactions based on time-span or composite sensors. Drill down into Audits of Composite Execution to review Payloads and Fault details. View and recover failed transactions or attempt re-delivery of failed messages. Configure Audit and Log Levels Live In-flight DMS Performance Metrics Use System MBean Browser for advanced configuration and metrics. Use FMW Control for searching and tracking your composite instances



FMW Control - Key Performance Tuning Tasks • Configure SOA-Infra Properties

• Audit/Instance Tracking(CPU,DB), Payload Validation (CPU)

• Configuring Service Engine threads • BPEL - Threads(Concurrency), Audit (DB),

Validation (CPU) • Mediator - Parallel/Resequencer Threads

(Concurrency), Audit (DB), Sleep Intervals/Metrics Level (CPU)

• Configure Adapter Properties • MaxRaise/TxnSize(DB), Number of

Threads (Concurrency)

• All this Accessible as MBeans - Update via WLST



EM Grid Control - SOA Management Pack EE • Central management of multiple SOA domains

• Historical data and trend analysis of SOA metrics

• Configure thresholds with alerts – Integrated with SNMP traps

• Scale monitoring and alert configuration using templates.

• Correlated view between system metrics and SOA composite instances.

• Extensible through shell / SQL scripts.

• Monitor custom rroups [apps, DBs] in one place.

• Use GC11g to track your composite execution flows ,monitor your application and system metrics, alert Ops or perform corrective action on critical issues.

Link to correlated JVM Diagnostics

Detected SOA component name



JRockit Mission Control • Negligible performance overhead. • Comprised of three toolkits.

– Management Console - for managing and monitoring one or more JVMs.

– Runtime Analyzer – Application and VM Profiler. Record and Playback for Root-cause analysis. • Analyze patterns around GC, CPU, Latency, Locks etc

– Memory Leak Detector

• Integrates with WLDF for analysis of diagnostic dumps

• Use JRMC to root-cause performance issues by analyzing memory, CPU, thread contention, locks etc via Flight Recordings.



WLST WLST

Online

Server Lifecycle

Composite Deployment and Lifecycle

MBean access and Updates

DomainMBean DomainRuntimeMBean

ServerRuntimeMBean

Custom(For SOA)

WLDF Integration to export

Diagnostic Data

Offline

Create Domain

Templates/Domains from Templates,

Extend Domains etc

.Jython based CLI Create, monitor and manage WLS Domains Runs in Interactive,Batch or Embedded Mode

• Command line automation .

• Hierarchical navigation via domain and server Mbeans: • retrieve and set domain metrics and properties • start/stop servers etc



Purging

• Parallel

• Useful for medium [ ~500 GB total to ~10 GB daily] to large installations

• Resource intensive – hence run during off hours.DOP indicates number of parallel threads.

• For large installs [>~500 GB total or > ~10 GB daily], recommended to run a hybrid of parallel purging + dropping partitions, using optimal thread counts.

• Partitioning • Dropping partitions is the fastest way

to purge.

• Time-Range Partitioning Done at the component level.

• Decide time-range based on average and max composite lifetime.

• Run this in conjunction with parallel purging for a mixed load.

Via EM Console [For Targeted Deletion], or via Purge Scripts and/or dropping Partitions[For Bulk] Scripts located at RCU_HOME/rcu/integration/soainfra/sql/soa_purge

• Looping

• Useful for small installations [~100k rows]

• Loops and runs until max_runtime loop parameter is exceeded.

• Not resource intensive, but slow performance.

• All purgeable instances are deleted.



Program Agenda

• Overview




• Troubleshooting

• Q&A



Fault Handling – Key Factors

• Handled Faults – Handled programmatically – Component / Policy – Can specify manual intervention through EM – Principal transaction not rolled back

• Un-handled Faults – Not handled programmatically – Typically leads to transaction rollback – Messages can be re-tried when they are retrievable

• JMS queue, DB table, Dehydration storeetc.



BPEL Service Engine Threads and Persistence

Receive

…

Receive

… sync process invoke executed by caller thread,

e.g. servlet

async process invoke executed by BPEL engine invoker thread

Msg persisted (DB)

Receive

…

After hydration (e.g mid-process receive activity for async callback) process executed by BPEL engine engine thread

(2nd) callback msg persisted (DB), triggers dehydration

(1st) process hydration

Reply

Sync Process Async Process

De/hydration



BPEL Service Engine Faults and Transactions

Receive

…

Receive

… Receive

…

exception or system crash rolls message back into DB, available to BPEL SE for manual (callback) recovery. composite and BPEL instance running

exception or system crash

returns exception (or timeout) to

client

exception or system crash rolls message back into DB, available to BPEL SE for manual (invoke) recovery. composite running but BPEL instance faulted; new instance initiated on recovery.

msg persisted (DB)

(2nd) callback msg persisted (DB), triggers dehydration

(1st) process hydration Sync Process Async Process De/hydration

* – A 3rd type of recovery is Activity recovery. See next slide.



BPEL Service Engine Faults and Transactions

<Activity>

…

Receive

Other types of activities can also trigger process dehydration

• Wait • bpelx:exec checkpoint(); • onMessage

exception or system crash rolls message back into DB, available to BPEL SE for manual (activity) recovery



Mediator Service Engine Sequential vs. Parallel Routing

M

sequential (aka sync) routing rules are

executed by caller thread

M

(1) parallel (aka async) routing rules

persist messages into an internal DB table (executed by

caller thread)

(2) A single locker thread retrieves message batch from DB table into memory

Tune the locker thread batch size and # of worker threads for parallel routing rules

…

(3) separate worker threads process each message



Mediator Service Engine Sequential vs. Parallel Routing

M

sequential (aka sync) routing rules are

executed by caller thread

M

(1) parallel (aka async) routing rules

persist messages into an internal DB table (executed by

caller thread) Recover Mediator parallel instances via EM console – Faults page for soa-infra, composite or Mediator SE

…

exception or system crash

returns exception (or

timeout) to caller

Exceptions during processing will cause message to roll back

into DB table, available for manual instance recovery

(3) separate worker thread processes each message

(2) A single locker thread retrieves message batch from DB table into memory



Programmatic Fault Handling

• Fault Policies – Fault handling rules centralized in policy files (XML files) – Associated with specific a Reference, Component or Composite – Can define a conditions chain of recovery actions

• Recovery Actions – Retry – Human Intervention – recover from EM – Abort process – Replay scope – Re-throw fault – Java action – custom Java code called

Fault Policies



Manual / Scripted Fault Handling - Recovery

• Recovery Options – Fault policies or BPEL fault

handlers for programmatic handling of faults • Can delegate to administrator

– BPEL message recovery for unhandled faults • Auto-recovery (recommended) / EM

based • Recover from BPEL recovery

console. – Rejected messages from inbound

adapters

Recovery Action

Recovery screen

Edit payload

Faults & exceptions

log



Program Agenda

• Overview




• Troubleshooting

• Q&A



Troubleshooting SOA Oracle Fusion Middleware Diagnostic Framework

Diagnostic data

WLDF Thread Dumps

Automatic Diagnostic Repository

Command Line Utilities

Log Files

kill -3 pid WLST

Weblogic Admin Utilility

Presenter

Presentation Notes

Taking Thread Dumps 1) On Windows : Cntrl + Break 2) On Unix : a) kill -3 pid (where pid is the process id for the managed server JVM instance) b) Using Weblogic Admin Utility java weblogic.Admin -url AdminUrl -username <username> -password <pass> THREAD_DUMP c) Using WLST: i) java weblogic.WLST ii) connect("weblogic","weblogic","t3:\\localhost:7001") iii) threadDump() Posted by Anshul: BusyNut

35

Configuring WLDF Watches in WLS Console • In-built mapping between Watch code

and dumps to create in ADR



Troubleshooting SOA

[2011-09-22T06:50:07.986-07:00] [AdminServer] [NOTIFICATION] [] [oracle.integration.platform.blocks.deploy] [tid: [ACTIVE].ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 3c584fd020bdf1ba:-65df75b4:132916143aa:-8000-0000000000000166,1:23265:2] [APP: soa-infra] [Deployment] Component: SendMail was successfully deployed to service engine:implementation.bpel, for composite:SendMailApp

Anatomy of a Log File 1 2 3

4 5 6 7

8 9

1. Timestamp 2. Server Name 3. Message Type 4. Component ID

5. Thread ID 6. User ID 7. Execution Context ID 8. Supplemental Attribute

9. Message Data



Troubleshooting SOA Editing Log Configuration



Troubleshooting SOA Setting Logging Levels

Active Loggers Saved Loggers

Logger for SOA Adapters

Logger for BPEL

Logging Levels



Troubleshooting SOA Viewing Log Files in EM

Message Pane



Common Problems and Solutions

• Problem: ClassNotFound exceptions when SOA server is started by a Node Manager

• Solution: Add property StartScriptEnabled=true to nodemanager.properties.

Resolving common startup problems with SOA server • Problem: SOA Server (soa_infra)

does not start • Solutions:

• Check port conflicts • Make sure MDS schema is

reachable • Check coherence configuration

in clustered setup. Make sure that you’re using command line instead of active cache configuration




• Problem: ‘Too many open files’ during compilation or runtime.

• Solution: /etc/security/limits.conf to increase the value of both file descriptors to at least 4096.

Resolving “too many open files”

• Problem: Transaction or connection timeouts

• Solution: Increase JTA timeout in weblogic console




• Problem: Changes to connection factories JNDI (Configuration -> Outbound Connection Pool) for SOA adapters do not take effect .

Common application-related Problems

Hit ‘Enter’ Key after changing

Click ‘Save’ after the connection factory properties

have been updated

Click ‘Update’ after selecting the adapter




• Problem: Mis-configured Queues/Datasource in JMSAdapter and DBAdapter.

• Solution: Make sure that the runtime resources such as JMS connection factories, JMS Queues, DB datasources are configured properly in the WLS console

Common adapter-related Problems • Problem: Duplicate message

processing for non-transactional adapters (File/Ftp) in clustered setup.

• Solution: Use ‘eis/HAFileAdapter’ or ‘eis/ftp/HAFtpAdapter’ in jca configuration. Alternatively configure active-passive failover in the composite.xml file by configuring “standalone” property with ”true”



EM Performance Issue

• Issue: EM dashboard loads slowly / hangs on login.

• Step 1: Set default time duration for aggregate queries.

• Step 2: Disable auto-fetch of aggregated metrics.


Q&A


Latin America 2011 December 6–8, 2011

Tokyo 2012 April 4–6, 2012


APPENDIX

1 copyright © 2011, oracle and/or its affiliates. all ... · • monitor custom rroups [apps, dbs]...

Documents