diagframeworkdevadminv2-1883022


Diagnostic Frameworks for Administrators and Developers

Natascha Schnfeld

OCS Bug Diagnostics and Escalations

Maria Salzberger

OCS Proactive Support Team


THE FOLLOWING IS INTENDED TO OUTLINE OUR GENERAL

PRODUCT DIRECTION. IT IS INTENDED FOR INFORMATION

PURPOSES ONLY, AND MAY NOT BE INCORPORATED INTO

ANY CONTRACT. IT IS NOT A COMMITMENT TO DELIVER ANY

MATERIAL, CODE, OR FUNCTIONALITY, AND SHOULD NOT

BE RELIED UPON IN MAKING PURCHASING DECISION. THE

DEVELOPMENT, RELEASE, AND TIMING OF ANY FEATURES

OR FUNCTIONALITY DESCRIBED FOR ORACLE'S

PRODUCTS REMAINS AT THE SOLE DISCRETION OF

ORACLE


Agenda

Motivation

Diagnostic Frameworks and the Resolution Process

Use Cases

Demo References

Q&A


Motivation


The Problem

Missing diagnostic

context on First

Failure

Loss of productivity

Maintenance overhead

Many iterations between Customer and Support


On the way to the best solution... Black Box principle: Gain diagnostic context

information on first failure.

Flexibility: Configure amount and type of

diagnostic information needed for analysis

Integration: Standardize tools and use the same

diagnostic methods for all products

SOA Diagnostics and Instrumentation Project - Principles


Overview: Diagnostic Frameworks Buzzwords

DFW

WLDF

IPS

DMS

ADRCI

RDA

ODL

ST

SWB

ECID


Resolution Process

Detect

Record context information on first failure Monitor resources proactively and provide notification when defined resource limits are

exceeded i.e. Health Checks

Collect

Rules-controlled diagnostic collections Diagnostic information catalogue (Repository) Diagnostic information quality

Analyze Efficient provisioning of diagnostics to all parties involved in the resolution process

Solve

Quicker bug resolution and patch provisioning Discover root cause and apply configuration changes

Phases and Requirements


Diagnostic Frameworks Architectural Overview

Rules

Custom Rules

SOA Rules

WLS Rules

WLDF

DMS

ODL

Incidents

Diagnostic

Dumps

zip

RDA

IPS

Logs

ST

DFW Rules

ADR Repository

Detect

Collect

Analyze

Solve


Diagnostic Frameworks and the Resolution

Process

Detect

detect collect analyze solve


Diagnostic Frameworks Architecture

Rules

Custom Rules

SOA Rules

WLS Rules

WLDF

DMS

ODL

Incidents

Diagnostic

Dumps

zip

RDA

IPS

Logs

ST

DFW Rules

ADR Repository

Detect


DMS: Dynamic Monitoring Service

The Oracle Dynamic Monitoring Service (DMS) provides Runtime

Statistics about the JVM and deployed Applications

Applications utilize the DMS APIs to implement Nouns

Nouns have associated Sensors for runtime statistics like Request

Count, Free Heap Size, Processing Time, etc

Collected Metrics can be viewed through DMS Spy, WLST

(WebLogic Scripting Tool) or Fusion Middleware Control

Functionality


DMS: Dynamic Monitoring Service SOA Suite Sensors


WLDF: WebLogic Diagnostic Framework

Included with WLS

Watches and Notifications to monitor runtime conditions and notify in case of failure

Rules

Sources (server logs, DMS Sensors,WLS MBeans)

Notifications

Generate Diagnostic Dumps containing relevant diagnostic information

i.e. thread dumps, diagnostic images, JRockit flight recordings,

etc

Administration through the WLS Console or WLST

Functionality


WLDF: WebLogic Diagnostic Framework Functionality

3 1

2

4

3


Out-of-the-Box WLDF Watches PS5+: Predefined Watches


ODL: Oracle Diagnostic Logging

Primary logging facility used by most FMW applications to write

diagnostics into the Diagnostic Logs

ODL Structure (name/value pairs)

Mandatory and Optional attributes

Order of attributes

ODL attribute values can be used for defining Conditions

Definition of Logging Levels

Administration through WLST or Fusion Middleware Control

Functionality


ODL: Oracle Diagnostic Logging

[2011-09-22T06:50:07.986-07:00] [AdminServer] [NOTIFICATION] []

[oracle.integration.platform.blocks.deploy] [tid: [ACTIVE].ExecuteThread: '3' for queue:

'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 3c584fd020bdf1ba:-

65df75b4:132916143aa:-8000-0000000000000166,1:23265:2] [APP: soa-infra]

[Deployment] Component: SendMail was successfully deployed to service

engine:implementation.bpel, for composite:SendMailApp

Anatomy of an ODL Entry

1 2 3

4 5

6 7

8 9

1. Timestamp

2. Server Name

3. Message Type

4. Component ID

5. Thread ID

6. User ID

7. Execution Context ID

8. Supplemental Attribute

9. Message Data


ODL: Oracle Diagnostic Logging Functionality

Saved

Loggers

Active Loggers

Logging Levels


ST :Selective Tracing

Selective Tracing adds an additional Attribute: ODL_TRACE_ID

It can limit the TRACE to:

Composite

Application

User

Definition of time boundaries for a logging session

Administration through WLST and Fusion Middleware Control

(start/stop session)

Available in release 11.1.1.5 (11gR1 PS4) and above

Functionality


ST :Selective Tracing

WebLogic

Domain

Context: Composite Name, Client Address etc

Trace Level

Loggers


Diagnostic Frameworks and the

Resolution Process

Collect




Rules

Custom Rules

SOA Rules

WLS Rules

WLDF

DMS

ODL

Incidents

Diagnostic

Dumps

zip

RDA

IPS

Logs

ST

DFW Rules

ADR Repository

Collect


DFW : Diagnostic Framework History / Background

DFW was introduced with Oracle 11g Database (Fault

Diagnosability Infrastructure)

DFW was adopted by Oracle Fusion Middleware SOA starting

with patchset 11.1.1.6 (11gR1 PS5)

Standardization of the diagnostic collection mechanisms

through the whole product stack

Documentation of every failure in a common repository


DFW : Diagnostic Framework Terminology

Problem

Inc_2 Inc_n Inc_1

Problem code bug or configuration issue that causes execution failures

Incident a single occurrence of a problem

Problem Key it associates incidents to problems (e.g. OWS-04086 (SOAP

Fault))

Diagnostic Dump a collection of diagnostic information (i.e. thread

dump, configuration, etc)

Pro

ble

m K

ey


DFW : Diagnostic Framework

DFW Components

Rules as XML Files

Rules validation services

Automatic Diagnostics Repository (ADR) for creating Incident /

problems catalogue

The DFW Rules define:

which ODL events or WLDF Watches MUST trigger an incident

Which type of information MUST BE collected on incident creation

(dump execution)

Functionality


DFW: Diagnostic Framework Rules

LogDetectionCondition Condition based on ODL Attribute values as manifested in Diagnostic

Logs that triggers the execution of

dumps

Rule Condition

Diagnostic Dump Action

Rule Condition

Diagnostic Dump Action

Condition based on WLDF Watches Diagnostic Dump Actions list the

specific DDs to take when the condition is met. They can be scoped by

arguments

Default Actions are taken for any Log or Rule Condition that is triggered Diagnostic Dump Action


DFW: Overview SOA Dumps

Dump description Name

Environment (SOA Version/Label, Patch Inventory, Topology) soa.env

Configuration (soa-infra, service engine config, deployed-

composites.xml)

soa.config

DB Information (Data shape - counts by state, growth patterns for SOA

schema, MDS schema)

soa.db

Composite Metadata (current composite, MDS artifacts referenced) soa.composite

EDN Dump (EDN event bus state variables, EDN subscriber/publisher

information for all deployed composites)

soa.edn

WSDL/Schema Cache (All Composites) soa.wsdl

Composite Instance Trail soa.composite.trail


WLST: Dump Output SOA Composite Instance Trail Dump


DFW: Diagnostic Framework

ADR (Automatic Diagnostic

Repository) is a file-based

repository

Incidents are stored in ADR with a

unique ID

Diagnostic Dumps are stored in

the related Incident Directory

ADR Catalogue of Incidents Managed Server

HOME

Application Incidents

Server Incidents


DFW: Diagnostic Framework Administration


ADRCI/IPS: Incident Packaging System

IPS is used to gather and package Incident data

This happens in three steps:

Create a logical package (Metadata)

Copy diagnostic information

Create physical package (.zip) containing the dumps

Functionality


DFW: Diagnostic Framework Summary Component Interaction

WLDF

DMS

ODL

zip

RDA

IPS

Logs

ST

Rules

Custom Rules

SOA Rules

WLS Rules

Incidents

Diagnostic

Dumps

DFW Rules

ADR Repository



Resolution Process

Analyze




Rules

Custom Rules

SOA Rules

WLS Rules

WLDF

DMS

ODL

Incidents

Diagnostic

Dumps

zip

RDA

IPS

Logs

ST

DFW Rules

ADR Repository

Analyze


EM Cloud Control 12c Incident Manager

Manage reported problems (lifecycle,

history, etc) through a GUI

Present patches and workarounds

when available

Create Service Requests

Package data using IPS

Upload Incident information and test

cases to Oracle

Automatically track and synchronize

Service Request status


RDA: Remote Diagnostic Agent

Log Files

Incidents

Configuration

Diagnostic Information within a SOA Collection


WLST: WebLogic Scripting Tool

List and Execute Diagnostic Dumps

List and Manage Incidents

Create Incidents Manually


Important Steps When you suspect a problem...

Check if an Incident is already registered in ADR

listProblems(), listIncidents().

View details of the specific Incident

showIncident(), getIncidentFile().

If no Incident is created, create an Incident

manually and collect specific Diagnostic Dumps

createIncident()

Package Incidents using IPS or create an

RDA Collection and upload it to the SR



Resolution Process

Solve




Rules

Custom Rules

SOA Rules

WLS Rules

WLDF

DMS

ODL

Incidents

Diagnostic

Dumps

zip

RDA

IPS

Logs

ST

DFW Rules

ADR Repository

Solve


Problem Resolution

Sufficient diagnostics point to a configuration change OR

Diagnostics provide evidence for a bug that should be fixed in

cooperation with Oracle Teams


Use Cases: Production or Development

Environments


Example Analyzing and Solving a Performance Problem

detect collect

End-Users report slow response times

RDA SOA Collection module will collect the existing Incidents

RDA Collection uploaded to the SR for further analysis



The last Incidents in RDA indicate several OutOfMemoryErrors Incident creation was triggered by an out-of-the-box WLDF Watch

analyze solve



Support can use JRMC (JRockit Mission Control) because there is

a JRockit Flight Recording included with the Diagnostic Dumps.

This points to a specific object causing the OOM.

analyze solve


Cooperation with Support

RDA SOA Collection

RDA is used to collect

configuration and

runtime information from

the environment

RDA guarantees that

the analysis can start

immediately w/o

additional iterations

Provision of Diagnostic Information


Problem Key Description

DFW-99996 java.sql.SQLException

DFW-99997 java.lang.OutOfMemoryError creates a jvm.classhistogram dump

DFW-99998 java.lang.NullPointerException, java.io.IOException,

java.lang.StringIndexOutOfBoundsException,

java.lang.ClassNotFoundException

SOA-900000 SOA Runtime problems

SOA-900001 SOA Deployment problems (Deployment hangs)

SOA-900002 SOA DataSource Watch (database and JDBC)

Predefined Problem Keys Uncaught Exception Problem Keys and SOA-90000x Watches


Scenario Action

SOA Data

Source

Problems

Data Source is

suspended Detect the root cause

Setup a WLDF Watch for SOA-900002.

Heap /

Memory

Problems

OutOfMemoryErrors in

the logs - Detect the root

cause

Setup a WLDF Watch for SOA-900000

SOA

Deployment

- SOA Server Start takes

a long time

- Deployment takes a

long time

Setup a WLDF Watch for SOA-900001

Use Selective Tracing for oracle.integration.platform.blocks.deploy.*

SOA Use Cases SOA Watches and Dumps


Beispiel Aktion

Errors in

Diagnostic

Log for

specific

composites

Newly deployed

composite versions cause

Errors in the Diagnostic

Logs

Use Selective Tracing for the specific composite by setting all needed

Loggers to TRACE:32 for a timely

limited session

Random

occurrence

of error

messages in

Diagnostic

Logs

Need to get the context

as the problem happens

sporadically

Configure Custom Rules in DFW to trigger specific SOA Diagnostic Dumps

Setup Selective Tracing on the user level if you have evidence that the

problem is user related

SOA Use Cases


Summary


What can we do TODAY?

We can use WLDF Watches, DMS Metrics and SOA MBeans to

trigger Incident generation thru DFW and automatic collection of

diagnostics. That helps to optimize the resolution time and

increases efficiency.

We can always check if the pre-defined Watches have already

created an Incident in a critical situation (First Failure).

We can use the SOA Modules of RDA for optimizing the workflow

between Support and customers

We have everything in place to start familiarizing ourselves with

the frameworks TODAY!


What can we do TOMORROW?

We can use a variety of dumps for ALL components (including

OSB, Mediator and Adapters)

We can use the pre-configured Health Checks (included in RDA

version 4.30) to proactively identify potential problems


Demos References Your Feedback is IMPORTANT!!!

@SOAProactive

Twitter

SOAProactive

Blog


References Overview of SOA Diagnostics in 11.1.1.6

FMW Diagnostic Framework : Automatic Capture of Diagnostic Data

Upon First Failure!

Customer WebCasts: Advisor Webcast Current Schedule

SOA Suite Information Center


Questions?

diagframeworkdevadminv2-1883022

Documents

oracle andor

diagnostic methods

diagnostic context information

support copyright

motivation copyright

type of diagnostic information

requirements copyright

information purposes