![Page 1: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/1.jpg)
![Page 2: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/2.jpg)
Diagnostics and Verification of Oracle Real Application Clusters 10g
Jack Cai
Principal Product Manager
Oracle Corporation
40248
![Page 3: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/3.jpg)
Agenda
Real Application Clusters 10g is for Everyone
Benefits of RAC Diagnostics and Verification
Diagnostics Framework Cluster Verification Framework Q&A
![Page 4: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/4.jpg)
Oracle RAC 10g for Everyone Foundation of Oracle grid computing Much easier to use and manage
– Full stack reduces complication Clusterware Cluster filesystem Automatic storage manager
– Integral to Enterprise Manager– Workload management with policy based
service– Cluster diagnostics and verification
![Page 5: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/5.jpg)
Diagnostics & Verification Benefits• Improve problem resolution time
Reduce need to reproduce problems Eliminate problems related to incorrect
cluster configuration Improve quality of services through a
more systematic approach
![Page 6: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/6.jpg)
Diagnostics Framework
Trace instrumentation of Database Server Clusterization of oradebug, hang analyzer Trace loader Navigator Equally useful on single instance database For database server internal diagnostics ONLY
– Not for diagnosis of application problems/tuning– Use them ONLY under Oracle support supervisions
![Page 7: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/7.jpg)
Diagnostics Framework Goals Generate enough diagnostic data for
first pass analysis Minimal overhead to database server Easy to use
– Single system view of a cluster– Tool for data gathering, conversion and
repository– Tool for display, navigate and analysis
![Page 8: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/8.jpg)
DIAG Process
Instance 1
SGA
Diagnostics Architecture (Cluster View)
Off-line Tools
On-line Tools
Trace Files
OradebugX$VIEWs
Trace Navigation ToolTrace Loader
DIAG Process
Instance 2
SGA
DIAG Process
Instance 3
SGA
![Page 9: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/9.jpg)
Diagnostics Architecture (Instance)
Off-line Tools
On-line Tools
X$ Views
Trace Buffer Pi
Trace Buffer Pj
Process Pi
Process Pj
DIAG Process
SGA
Trace Files
Instance
Trace Navigational ToolTrace Navigational Tool
Trace LoaderTrace Loader
OradebugOradebug
![Page 10: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/10.jpg)
Flexible Trace Instrumentation Focus on issues encountered by customers
– Instrument trace on events Oracle support handles most frequently
– Foundation laid in 9i, more component instrumentation in 10g
Trace control– Trace is event based, events can be anything within a
program– Trace is controllable on a per process basis– 1000 event id (10000-10999), 256 opcodes, 256 levels– Trace is generated to SGA buffer rather than directly to
files– X$Trace, X$Trace_events can be queried during run time
“Always-on” minimal tracing Optional trace archiving
![Page 11: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/11.jpg)
Trace Control Example
Dynamic control– ALTER TRACING <cmd>
ON OFF ENABLE <event-string> DISABLE <event-spec> FLUSH <proc-spec>
– Examples alter tracing enable <event>12345:<level>5:<pid>32 alter tracing disable 12345:5:BGS alter tracing flush 32
![Page 12: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/12.jpg)
oradebug Is Cluster Aware Single system image of the cluster Commands are clusterized DIAG based implementation
– Damen process– Minimal overhead
HANGANALYZE analyzes the entire cluster
ExamplesSQL> oradebug -g <inst_list>/ALL <cmd>SQL> oradebug setinst “3 4”
![Page 13: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/13.jpg)
Offline Trace Loader
• Offline trace loader converts binary trace files into text format, and loads files into Oracle databases for archiving
• Better data analysis through power of database queries
• Build a repository for problem tracking
• Facilitate trace data exchange in cross-platform environments
![Page 14: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/14.jpg)
Offline Trace Loader Extracts trace data from archived trace
files Converts binary trace data to text Loads trace data into database as trace
repository Simple command line interface
– trcldr [option,...]– trcldr mode=append ifile=f1.trw userid=scott/tiger
Raw Trace Files(in binary or text format)
Converted Trace Files(in text format)
Target Database
Data Conversion Data Loading
TraceRepository
TraceFile Trace
LoaderTrace
Loader
TraceFile
![Page 15: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/15.jpg)
Trace Navigation Tool
Navigate traces with links and highlights across cluster
– send/recc, malloc/free– User Defined Protocol Navigation
Coloring of traces based upon attributes Well-defined queries against keywords Cluster wide time-ordered event
diagram
![Page 16: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/16.jpg)
Navigation Tool Protocol Trace
![Page 17: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/17.jpg)
Trace Navigation
![Page 18: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/18.jpg)
Navigation Tool Event Diagram
![Page 19: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/19.jpg)
Summary of Diagnostics
Diagnostics facilities allow support to resolve issues more quickly
– First pass failure diagnosis
Allows developers to find bugs faster Tools are not meant for end users End users should only use the tools
under guidance from Oracle support personnel
![Page 20: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/20.jpg)
Cluster Verification Goal
Eliminate errors through: – Pre/post validation of installation steps
and/or configuration changes– Ongoing cluster validation
Flexible– API invoked by other programs
OUI, OEM, DBCA, ASM– Command line interface– Can be run anytime
To be released post Oracle Database 10g
![Page 21: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/21.jpg)
Cluster Verification Framework Verification categories
– Stage verification– Component verification
Diagnostic mode API for vendor verification tools
– Utilizing vendor platforms’ verification capabilities
![Page 22: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/22.jpg)
Stage Verification RAC deployment process is divided into stages
– Stages are steps of installation/configuration changes
– All RAC deployment stages are verified– Verification performs pre-stage and post-stage
checks
Example stages– Cluster filesystem setup– Clusterware setup– Database installation– Cluster node addition– Database configuration
![Page 23: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/23.jpg)
Component Verification
RAC is divided into key components– Each component can be individually
verified
Examples– Node connectivity– Shared storage accessibility– Cluster integrity– System requirements– CFS integrity
![Page 24: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/24.jpg)
Diagnostic Mode
Attempt to pinpoint root cause of a verification failure in diagnostic mode
Applicable to stage and component verification
Specific testings are performed when a verification fails
Only triggered by verification failure
![Page 25: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/25.jpg)
How Verification is done
Command line tool: cluvfy– cluvfy stage –post HWOS (verifies
hardware and OS stage)– cluvfy comp sys –p RDBMS (verifies
system components for RDBMS installation)
APIs by other relevant tools
![Page 26: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/26.jpg)
Summary• Diagnostics speed up problem resolutions • Cluster verification eliminates incorrect
configuration• Ultimately reducing unscheduled system
down time and improving system availability
Diagnostics tools should ONLY be used under support guidance
End Users can use cluvfy to perform verifications
![Page 27: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/27.jpg)
Next Steps…. Recommended sessions
– 40131: Oracle Real Application Clusters 10g: The Fourth Generation
– 40132: Oracle Real Application Clusters 10g: Manage your cluster Without it Managing You
Recommended demos and hands-on labs– Visit DEMOgrounds under Real Application Clusters
See Your Business in Our Software– Visit the DEMOgrounds for more information.
Relevant web sites to visit for more information– http://otn.oracle.com/products/database/clustering/
content.html
![Page 28: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/28.jpg)
Reminder – please complete the OracleWorld online session survey
Thank you.
![Page 29: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/29.jpg)
AQ&Q U E S T I O N SQ U E S T I O N SA N S W E R SA N S W E R S
![Page 30: Diagnostics and Verification of Oracle Real Application Clusters 10 g](https://reader031.vdocuments.us/reader031/viewer/2022013101/5681449a550346895db146c3/html5/thumbnails/30.jpg)