velikanovs - performance tuning and troubleshooting for oracle oc4j (slides - long version)
TRANSCRIPT
![Page 1: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/1.jpg)
Performance Tuning and Troubleshooting for Oracle OC4J
Jurijs Velikanovs7+ years Oracle DBA, OCP 7/8/8i/9i, OCA 9iAS R2
![Page 2: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/2.jpg)
Topics:
IntroductionUsage considerationsOC4J InternalsOC4J TroubleshootingOC4J Performance tuningBC4J
![Page 3: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/3.jpg)
Introduction
![Page 4: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/4.jpg)
Who am I? And what are my responsibilities?
Who am I?I am a DBA team leaderI’m neither a Java fan nor a Java Developer nor a Java GuruMy team has spent a lot of time researching and struggling to stabilize the production environments based on OC4J
ResponsibilitiesTo support production and development environmentsTo ensure production availabilityTo troubleshoot production systemTo provide all the necessary information to the developers or support organizations to solve a problem
![Page 5: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/5.jpg)
How were we involved?
Local Oracle Marketing team has pushed customers to demand the use of OC4J in their tenders’ conditionsIn 2002 the company management decided to switch the whole Development from Oracle Forms to OC4J/BC4J/JDeveloper technologyWe had
100 forms developers without any Java experienceA few Java fansOracle 9iAS R2 (9.0.2), later 9.0.3.1In 2003 the first phase of the first application was deployed
![Page 6: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/6.jpg)
Example of the Applications we are working with
Healthcare financial systemEnterprise application2+ years in the production and still under developmentOracle 9iAS 9.0.3.1 OC4J, Oracle 9.2.0.5 DBJSP = 1448 files, CLASS = 3283 files965 tables, 593 views, 251 types, 220 packages, DB Size 50 GB, REDO 2-8 GB Daily800 heavy users per day
10x5 hour availability
![Page 7: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/7.jpg)
![Page 8: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/8.jpg)
Why is it so important?
OC4J is the present strategic direction for Oracle JDeveloper, Forms, Reports, Discoverer, OEM, etc.
Difficult to troubleshootComplex technology Many componentsJava on Java
![Page 9: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/9.jpg)
![Page 10: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/10.jpg)
Why is it so important?
OC4J is a strategic direction for Oracle JDeveloper, Forms, Reports, Discoverer, OEM
Difficult to troubleshootComplex technology Many componentsJava on Java
There is not much information about this productPoor vendor supportMany Oracle customers use it and have troubles
![Page 11: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/11.jpg)
I will not tell you how to develop in Java …
I’m not going to tell you aboutWeb and EJB application technology VMC etc.
The technical layer is of great interest
![Page 12: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/12.jpg)
I’m going to speak about OC4J internals …
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
Java native
ApplicationServerThread
ApplicationServerThread
JMSServerPORT: 3201
RMI ServerRMI ServerPORT: 3101
TaskManagerTaskManager
JMSServerJMSServerJMSServerJMSServerThreadThread
ONS
OC4JMonitor Thread
Thread
Thread
AJPConnection
ListenerPO
RT:
300
1
VM Thread VM Periodic Task Thread
Signal Dispatcher
Suspend Checker Thread
Reference Handler
Finalizer
Java native
VM Thread VM Periodic Task Thread
Signal Dispatcher
Suspend Checker Thread
Reference Handler
Finalizer
Memory Structures
Http Session List JDBC Connection pool
Application objects
...
ApplicationServerThread
...
ApplicationServerThread
AJP 1.3 ONet
new req
3101
3201
![Page 13: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/13.jpg)
Usage considerations or
how to avoid troubles
![Page 14: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/14.jpg)
Oracle AS Architecture Considerations
You may make the configuration complicated, with many components highly dependent on each other
![Page 15: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/15.jpg)
Oracle AS Architecture Considerations
BUT:Keep it as simple as possibleAt first you should be sure that the system works properly in a simple configurationMake components as independent as possibleDon’t use the infrastructure if there is no strong need for it (i.e. Single-Signon, Discoverer, Clustering)
![Page 16: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/16.jpg)
Oracle AS Typical Architecture
PO
RT:
80,
778
0
OHS (c)
Child ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild Server
Child ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild Server
Child ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild Server
OC4J
DB
WAN
ApacheChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild Server
JVM
AJPConnection
Listener
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
TaskManagerOC4JMonitor Thread
JMSServerJMSServerThreadThread
Java native
VM Thread VM Periodic Task Thread
Signal Dispatcher
Suspend Checker Thread
Reference Handler
Finalizer
TaskManager
JVM(java)
AJPConnection
Listener
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
JMSServerPORT: 3201
POR
T: 3
001
TaskManagerOC4JMonitor Thread (dms)
JMSServerJMSServerThreadThread
Java native
VM Thread VM Periodic Task Thread
Signal Dispatcher
Suspend Checker Thread
Reference Handler
Finalizer
TaskManager
RMI ServerRMI ServerPORT: 3101
HTTP AJP 1.3
PM, ONS (c)strat, stop, ping,
notification
OPMNDCM (java)
configuration management
DCM-daemonDMS.JAR (java)
statistics reflecting
DMS-metrics
opmn.xml
httpd.conf server.xml
DCM repository
![Page 17: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/17.jpg)
Installation Considerations
Use the latest version of Oracle AS depending on your circumstancesUse the latest versions of Oracle Forms, Reports, Discoverer, Portal. Install them in the OH separated from your custom development OC4J OHInstall the latest patches for OS, JDK, OC4J, JDBC componentsInstall the version of BC4J which was used by the Development. Don’t use the default version of BC4J/ADF/JHeadstart (See. JDeveloper install.html)
![Page 18: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/18.jpg)
Operational Considerations
Keep running only the components you needDon’t startup in the production environment:
Oracle Enterprise Manager 10g Application Server ControlOC4J_Demos component“home” component
Don’t deploy your applications to the “home”componentCreate your own component and use it to deploy and run your applicationsUse the opmnctl, dcmctl, dmstool utilities to maintain your AS environment
![Page 19: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/19.jpg)
Deployment Considerations
Get a commitment from the Development to use industry standards for the Application development
Use EAR files to deploy any application changesUse Ant tool (http://ant.apache.org) to make EAR/WAR filesIn emergency cases use WAR files to deploy small changes to application
Use undeployApplication/deployApplication combination rather than redeployApplication dcmctl commandUse dcmctl saveInstance before deploying to backup current configurationDeploy BC4J.EAR to your OC4J component to keep “home” down
![Page 20: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/20.jpg)
Versions Considerations
Application
OC4Jc.c.c.x9.0.3.1
JDK c.c.x_x, 1.3.1_15
OS 2.4.9-e.38enterprise
JDBCc.c.c.x9.2.0.5
BC4J 9.0.3.10.7
APP Frame Work
RecommendationsHave a dedicated test environment and verify all the changes there before implementing in productionIf you have the automatic stresstestingscenarios, run them after each change in the test environmentMonitor the impact of changes and if the situation becomes worse rollback them
![Page 21: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/21.jpg)
Versions Considerations
Application
OC4Jc.c.c.x9.0.3.1
JDK c.c.x_x, 1.3.1_15
OS 2.4.9-e.38enterprise
JDBCc.c.c.x9.2.0.5
BC4J 9.0.3.10.7
APP Frame Work
OSIt is important to install the latest patches from an OS vendor (Kernel, Network, etc.)Usually the patches can be rollbacked easilyThe change of OS main release requires more testing and efforts than patching
JDKInstall the latest versions of JDK within c.c. boundariesUse the JDK from Sun ww.JavaSoft.comTo install it replace the OH/jdkdirectory. These changes can be rollbacked easilyThe change of c.c. version requires the Development involvement and more testing
![Page 22: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/22.jpg)
Versions Considerations
Application
OC4Jc.c.c.x9.0.3.1
JDK c.c.x_x, 1.3.1_15
OS 2.4.9-e.38enterprise
JDBCc.c.c.x9.2.0.5
BC4J 9.0.3.10.7
APP Frame Work
JDBCInstall the latest version of JDBC within c.c.c. versionTo install it replace OH/jdbcdirectory. These changes can be rollbacked easily It requires the minimal functionality testing
OC4JCan be easily upgraded within a c.c.c. version boundariesIt requires the minimal functionality testingFollow the patch installation instructions (9iAS 9.0.3.1)There are 50+ one-off patches (*.class)The change of c.c.c. version requires the Development involvement and more testing
![Page 23: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/23.jpg)
Versions Considerations
Application
OC4Jc.c.c.x9.0.3.1
JDK c.c.x_x, 1.3.1_15
OS 2.4.9-e.38enterprise
JDBCc.c.c.x9.2.0.5
BC4J 9.0.3.10.7
APP Frame Work
BC4JA BC4J version sets the requirements for OC4J versionIt can’t be changed without Development involvement even within x.x.x.x. version boundariesOften there are needs for application changesIt requires a lot of testingNew version can have negative impact to your applicationThe change of c.c.c. version requires more application code changes, testing, and other efforts
![Page 24: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/24.jpg)
Versions Considerations
How to Determine the VersionsOS: uname -r
$ uname -r2.4.9-e.38enterprise
JRE: java –version$ $ORACLE_HOME/jdk/bin/java -versionjava version "1.3.1_13"Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_13-b03)Java HotSpot(TM) Client VM (build 1.3.1_13-b03, mixed mode)
OC4J: java -jar oc4j.jar –version$ $ORACLE_HOME/jdk/bin/java -jar $ORACLE_HOME/j2ee/home/oc4j.jar -versionOracle9iAS (9.0.3.1.0) Containers for J2EE (build 031214.BP1.2075)
JDBC: a program with a call to meta.getDriverVersion(location $OH/jdbc/lib/classes12dms.jar, Notes:73629.1/244074.1)
$ $ORACLE_HOME/jdk/bin/java JDBCVersionOracle JDBC driver 9.2.0.5.0
BC4J: a program with a call to meta.getDriverVersion(location $BC4J/lib/bc4jct.jar)
$ $ORACLE_HOME/jdk/bin/java BC4JVersion 9.0.3.10.7
![Page 25: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/25.jpg)
Other Considerations
Have at least one senior level developer, who understands used technology well, is capable to make technical decisions and to lead the framework development, maintenance and troubleshootingSynchronize OC4J, BC4J versions with DevelopmentUse an Automatic testing tool for the application stresstestingA commitment from Oracle is desirable. If you can manage a connection to the Oracle JDeveloper development team to support your project, it will be an invaluable help during troubleshooting and decision makingHave a skilled DBA for environments maintenance, monitoring and troubleshooting. The DBA provides all possible information to the development for application troubleshootingHave a monitoring framework
![Page 26: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/26.jpg)
Before going to life
!!! Stresstest your application !!!(or ask for results)
![Page 27: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/27.jpg)
OC4J Internals
![Page 28: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/28.jpg)
OC4J Internals
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
Java native
ApplicationServerThread
ApplicationServerThread
JMSServerPORT: 3201
RMI ServerRMI ServerPORT: 3101
TaskManagerTaskManager
JMSServerJMSServerJMSServerJMSServerThreadThread
ONS
OC4JMonitor Thread
Thread
Thread
AJPConnection
ListenerPO
RT:
300
1
VM Thread VM Periodic Task Thread
Signal Dispatcher
Suspend Checker Thread
Reference Handler
Finalizer
Java native
VM Thread VM Periodic Task Thread
Signal Dispatcher
Suspend Checker Thread
Reference Handler
Finalizer
Memory Structures
Http Session List JDBC Connection pool
Application objects
...
ApplicationServerThread
...
ApplicationServerThread
AJP 1.3 ONet
new req
3101
3201
![Page 29: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/29.jpg)
Processes (threads) within OC4J
Java native threadsVM Thread, VM Periodic Task Thread, Signal Dispatcher, Suspend Checker Thread, Finalizer, Reference HandlerExist in any Java based application
OC4J Listener - threadsAJP/HttpConnectionListener, RMIServer (2), JMSServerAccepting clients requests, assigning them to the worker threadsThe configuration of listened ports is in opmn.xml file
Set of processes for communicating with ONSOC4JMonitorThread, Thread(2)These processes make connection to the OPMN immediately after the startup of OC4JThey send event messages to the OPMN to be broadcasted to subscribers
![Page 30: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/30.jpg)
Processes (threads) within OC4J
TaskManagerResponsible for a periodic tasks executing (housekeeping)
server.http.SessionTimeoutTask (drops expired session)server.ApplicationServerTask (clear expired threads)sql.DriverManagerXADataSource…
You can set an executing frequency in server.xml filetaskmanager-granularity=1000 (default 1 sec)
Use the dmstool to monitor the statistics of the processdmstool -table oc4j_task
ApplicationServerThreads (Global Thread Pool)Execute users requests, run an application code, communicate with the Apache processes and the databaseInitial count of the processes can be set in server.xml fileglobal-thread-pool min=“200" max="200" queue="20“
![Page 31: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/31.jpg)
OC4J Troubleshooting
![Page 32: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/32.jpg)
Classification of Problem AreasAp
ache
OC4J
DB
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
TaskManager
JMSServerJMSServer
JVM(java) AJP
ConnectionListener
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
POR
T: 3
001
New Request
assign
1.2.3.
JDK
OS
4.
1. OC4J container inside (infinitive cycles, waits on other threads, deadlocks)2. Interface to the Apache processes3. Interface to the database4. 500 Internal Error
![Page 33: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/33.jpg)
Possible Approaches & Information Sources
Java Full thread dumpOther
Log filesOS Monitoring commandsOS Tracing commandsApache side informationJava code decompilationDocumentation, External searches and Metalink
Troubleshooting information gathering (example)
![Page 34: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/34.jpg)
Java “Full thread dump”
How to get a Full thread dump?The Full thread dump interpretation
Different types of the threads within OC4J containerDifferent states of the threadsSome threads examples
Useful java runtime parameterJava DEADLOCK problem
![Page 35: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/35.jpg)
How to get a “Full thread dump”?
Get pid of the OC4J processUse : ps, pstree or opmnadmin
$ opmnadmin debug comp=pmHTTP/1.1 200 OK Content-Length: 677Content-Type: text/htmlResponse:
======== PM ========
PM requests processed: 5
PROCESS TABLE
UID PID FLAGS TYPE STATUS REF HTTP AJP RMI JMS---------- ----- -------- ------- ------- ------ ----- ----- ----- -----
456236 13512 00000040 OC4J Alive 1 0 3003 3104 3204390700 13511 00000040 OC4J Alive 1 0 3002 3103 3203521772 15874 00000040 OC4J Alive 1 0 3001 3102 3202259628 13415 00000040 OC4J Alive 1 0 3000 3101 3201194092 13338 00000040 Apache Alive 1 7780 0 0 0128556 13281 00000040 Generic Alive 1 0 0 0 0
![Page 36: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/36.jpg)
How to get a “Full thread dump”?
Find out the output file name (OH/opmn/logs/*)$ ps -fw --width 4000 -p 16114UID PID PPID C STIME TTY TIME CMDoracle 15874 15902 0 23:40 ? 00:00:00
/u01/app/oracle/product/J2EE01/jdk/bin/i386/native_threads/java -Xms512M -Xmx512M -Doracle.ons.oraclehome=/u01/app/oracle/product/J2EE01 -Doracle.ons.oracleconfighome=/u01/app/oracle/product/J2EE01 -Doracle.ons.clusterid=1 -Doracle.ons.instanceid=10.130.2.9.20dcd9.fdc4ac6949.-8000 -Doracle.ons.indexid=PANDA.default_island.1 -DOPMN=true -jar oc4j.jar -config /u01/app/oracle/product/J2EE01/j2ee/PANDA/config/server.xml -properties -instance -vmId C_10.130.2.9.20dcd9.fdc4ac6949.-8000#J2EE01.ap1.voava.lv#521772#PANDA#PANDA#default_island -ajp 3001 -rmi 3102 -jms 3202
$ ps -wo args --width 4000 -p 15874 | grep -v COMMAND | awk -F"oracle.ons.indexid=" '{print $2}' | awk '{print $1}'
PANDA.default_island.1
Send SIGQUIT (Quit from keyboard) signal$ kill -3 15874
![Page 37: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/37.jpg)
How to get a “Full thread dump”?
Check the output fileOH/opmn/logs/PANDA.default_island.1Full thread dump:
"Thread-13" daemon prio=1 tid=0x82c8690 nid=0x3f15 waiting on monitor [0xa51ff000..0xa51ff89c]
at java.lang.Object.wait(Native Method)at java.util.TimerThread.mainLoop(Timer.java:427)at java.util.TimerThread.run(Timer.java:380)
"Thread-12" prio=1 tid=0x67b0cce0 nid=0x3e02 waiting on monitor [0..0xbffeb62c]
"OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor [0xa53ff000..0xa53ff89c]
at java.lang.Object.wait(Native Method)at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:251)at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:224)at oracle.ons.Subscriber.receive(Subscriber.java:136)at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.java:419)at java.lang.Thread.run(Thread.java:479)
"Thread-10" daemon prio=1 tid=0x676930f0 nid=0x3ef1 waiting on monitor [0xa55ff000..0xa55ff89c]
at java.lang.Object.wait(Native Method)at java.lang.Object.wait(Object.java:415)at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:253)at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:213)at oracle.ons.SenderThread.run(SenderThread.java:81)
![Page 38: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/38.jpg)
The “Full thread dump” Interpretation
====================================================="OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor
[0xa53ff000..0xa53ff89c]at java.lang.Object.wait(Native Method)at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:251)at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:224)at oracle.ons.Subscriber.receive(Subscriber.java:136)at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.java:419)at java.lang.Thread.run(Thread.java:479)
=====================================================
[OC4JMonitorThread] – name (type) of the thread OC4JMonitorThread.javaThread.currentThread().setName("OC4JMonitorThread");
[nid=0x3ef2] – Thread ID (in decimal = 16114)
[waiting on monitor] – State of the thread at the snap time
[oracle.ons.NotificationQueue.internalDequeue] – java stack
![Page 39: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/39.jpg)
TYPES of the Threads
OC4J SpecificAJPConnectionListener [0.0.0.0/0.0.0.0:3001] 1RMIServer [0.0.0.0/0.0.0.0:3102] count:1 1RMIServer [0.0.0.0/0.0.0.0:3102] count:2 1JMSServer 1OC4JMonitorThread 1TaskManager 1Thread 5ApplicationServerThread 195JAVA GENERICVM Thread 1VM Periodic Task Thread 1Reference Handler 1Signal Dispatcher 1Suspend Checker Thread 1Finalizer 1
1
![Page 40: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/40.jpg)
STATES of the Threads
Waiting on monitorIn most of cases it is an idle state of the thread. The thread is waiting for a task to execute or voluntary sleepsAll the available ApplicationServerThreads are in this state
RunnableThe thread is executing application code or reading from the socketAll listeners must be in this state
Waiting for monitor entryWaiting to lock an object (other thread is holding the lock). This happens if two or more threads are trying to execute synchronized java codeNone of threads must be in this state for a long timeIf the system thread stays in this state for a long time you are in troubleIf many threads are in this state you have the situation of serialization
![Page 41: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/41.jpg)
Threads Examples
1. OC4JMonitorThread is waiting for a work. Usual state for most of the threads
"OC4JMonitorThread" daemon prio=1 tid=0x679a3518 nid=0x573 waiting on monitor [0xa53ff000..0xa53ff89c]at java.lang.Object.wait(Native Method)at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:251)at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:224)at oracle.ons.Subscriber.receive(Subscriber.java:136)at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.java:419)at java.lang.Thread.run(Thread.java:479)
2. "ApplicationServerThread-5" – idle worker thread, is waiting for a task"ApplicationServerThread-5" prio=1 tid=0x6794da88 nid=0x43f waiting on monitor [0xb15ff000..0xb15ff89c]
at java.lang.Object.wait(Native Method)at EDU.oswego.cs.dl.util.concurrent.BoundedBuffer.poll(BoundedBuffer.java:170)at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:768)at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:802)at java.lang.Thread.run(Thread.java:479)
3. "Thread-9" is reading from the socket (connected to the ONS process, port 7100). This thread most of time is waiting for a data to be read from the socket
"Thread-9" daemon prio=1 tid=0x67a02920 nid=0x571 runnable [0xa57ff000..0xa57ff89c]at java.net.SocketInputStream.socketRead(Native Method)at java.net.SocketInputStream.read(SocketInputStream.java:85)at oracle.ons.InputBuffer.readMoreData(InputBuffer.java:267)at oracle.ons.InputBuffer.getNextString(InputBuffer.java:222)at oracle.ons.ReceiverThread.run(ReceiverThread.java:225)
![Page 42: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/42.jpg)
Threads Examples
4. "AJPConnectionListener” is waiting for new connections from the Apache processes
"AJPConnectionListener [0.0.0.0/0.0.0.0:3001]" prio=1 tid=0x67911828 nid=0x439 runnable[0xb19ff000..0xb19ff89c]at java.net.PlainSocketImpl.socketAccept(Native Method)at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)at java.net.ServerSocket.implAccept(ServerSocket.java:238)at java.net.ServerSocket.accept(ServerSocket.java:217)at com.evermind.server.http.AJPConnectionListener.run(AJPConnectionListener.java:58)at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804)at java.lang.Thread.run(Thread.java:479)
5. "ApplicationServerThread-5" is reading the request from the Apache process
"ApplicationServerThread-77" prio=1 tid=0x8391048 nid=0x4a75 runnable [0xb4dff000..0xb4dff89c]at java.net.SocketInputStream.socketRead(Native Method)at java.net.SocketInputStream.read(SocketInputStream.java:85)at com.evermind.io.SingleReadBufferInputStream.readChunk(SingleReadBufferInputStream.java:116)at com.evermind.io.SingleReadBufferInputStream.read(SingleReadBufferInputStream.java:35)at com.evermind.server.http.AJPRequestHandler.initRequest(AJPRequestHandler.java:345)at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:131)at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:72)at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804)at java.lang.Thread.run(Thread.java:479)
Apache
![Page 43: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/43.jpg)
Threads Examples6. "ApplicationServerThread-71" is executing database query, waiting for the results to be retrieved"ApplicationServerThread-71" prio=1 tid=0x8379130 nid=0x4a6f runnable [0xb59fe000..0xb59ff89c]
at java.net.SocketInputStream.socketRead(Native Method)at java.net.SocketInputStream.read(SocketInputStream.java:85)at oracle.net.ns.Packet.receive(Unknown Source)at oracle.net.ns.DataPacket.receive(Unknown Source)at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)at oracle.net.ns.NetInputStream.read(Unknown Source)at oracle.net.ns.NetInputStream.read(Unknown Source)at oracle.net.ns.NetInputStream.read(Unknown Source)at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:931)at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:893)at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:375)at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1986)at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:1144)at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2484)at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2850)at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:609)at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:537)at oracle.jbo.server.ViewObjectImpl.getQueryHitCount(ViewObjectImpl.java:1522)at oracle.jbo.server.QueryCollection.getEstimatedRowCount(QueryCollection.java:1530)at oracle.jbo.server.ViewRowSetImpl.getEstimatedRowCount(ViewRowSetImpl.java:1482)at oracle.jbo.server.ViewObjectImpl.getEstimatedRowCount(ViewObjectImpl.java:4596)at _common._comp._CMN__DataScrollerComponent._jspService(_CMN__DataScrollerComponent.java:115)at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56)at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317)at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476)at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390)at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283)at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566)at com.evermind.server.http.ServletRequestDispatcher.include(ServletRequestDispatcher.java:119)at com.evermind.server.http.GetParametersRequestDispatcher.include(GetParametersRequestDispatcher.java:95)at com.evermind.server.http.EvermindPageContext.include(EvermindPageContext.java:280)at oracle.jbo.html.jsp.datatags.ComponentTag.doStartTag(ComponentTag.java:70)at _skd._d001._D2260F__Browse._jspService(_D2260F__Browse.java:249)at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56)at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317)at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476)at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390)at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283)at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566)at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:306)at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:767)at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:166)at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:72)at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804)at java.lang.Thread.run(Thread.java:479)
DB
![Page 44: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/44.jpg)
Useful Java Runtime Parameter
-XX:+JavaMonitorsInStackTrace (opmn.xml)<java-option value="-server -Xms512M -Xmx512M -verbose:gc -XX:+JavaMonitorsInStackTrace"/>
The parameter wasn’t set"ApplicationServerThread-18" prio=1 tid=0x692c40e0 nid=0x2f92 waiting for monitor entry [0xbc1ff000..0xbc1ff89c]
at oracle.jbo.server.ViewRowSetIteratorImpl.setRangeSize(ViewRowSetIteratorImpl.java:376)at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.java:1659)at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.java:4626)at lv.vovaa.panda.common.datatags.DataSourceCmn.doStartTag(DataSourceCmn.java:189)at _nsk._nsk001._M2801F__Edit._jspService(_M2801F__Edit.java:150)at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56)at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317)at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476)at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390)at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
...
The parameter was set"ApplicationServerThread-96" prio=1 tid=0x69d6c8c0 nid=0xdce waiting for monitor entry [0xb25ff000..0xb25ff89c]
at oracle.jbo.server.ViewRowSetIteratorImpl.setRangeSize(ViewRowSetIteratorImpl.java:376)- waiting to lock <460e0338> (a oracle.jbo.JboSyncLock)at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.java:1659)at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.java:4626)at lv.vovaa.panda.common.datatags.DataSourceCmn.doStartTag(DataSourceCmn.java:189)- locked <45f722d0> (a lv.vovaa.panda.common.datatags.DataSourceCmn)at _nsk._nsk001._AP3220F__Tickets__browse._jspService(_AP3220F__Tickets__browse.java:105)at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56)at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317)at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476)at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390)at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
...
OC4J
![Page 45: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/45.jpg)
Useful Java Runtime Parameter
It is possible to determine the blocking thread easily"ApplicationServerThread-55" prio=1 tid=0x69d512b8 nid=0xda4 runnable [0xb77fe000..0xb77ff89c]
at java.net.SocketInputStream.socketRead(Native Method)at java.net.SocketInputStream.read(SocketInputStream.java:85)at oracle.net.ns.Packet.receive(Unknown Source)
...at oracle.jbo.server.QueryCollection.executeQuery(QueryCollection.java:549)at oracle.jbo.server.ViewObjectImpl.executeQueryForCollection(ViewObjectImpl.java:2600)- locked <460e0338> (a oracle.jbo.JboSyncLock)at lv.vovaa.panda.common.pbo.PboViewObjectImpl.pboExecuteQueryForCollection(PboViewObject.java:324)at lv.vovaa.panda.common.pbo.PboViewObjectImpl.executeQueryForCollection(PboViewObjectImp.java:309)at oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:521)
...
Then we can locate the source code for troubleshooting purposesViewObjectImpl.java
protected void executeQueryForCollection(Object obj, Object aobj[], int i){
synchronized(getSyncLock()){
int j = 0;if(InstrumentedEvent.isActive)
j = InstrumentedEvent.startEvent(EventGroup.EXECUTE_QUERY, "ViewObjectexecuteQueryForCollection " + getName());
((QueryCollection)obj).executeQuery(aobj, i);if(InstrumentedEvent.isActive)
InstrumentedEvent.endEvent(j);}
}
![Page 46: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/46.jpg)
Java DEADLOCK
Information from the OC4J process “Full thread dump”Full thread dump:"ApplicationServerThread-126" prio=1 tid=0x881f668 nid=0x5d59 waiting for monitor
entry [0xa19ff000..0xa19ff89c]......FOUND A JAVA LEVEL DEADLOCK:----------------------------"ApplicationServerThread-58":waiting to lock monitor 0x80b9804 (object 0x4595b408, a java.lang.Object),which is locked by "ApplicationServerThread-51"
"ApplicationServerThread-51":waiting to lock monitor 0x80b9564 (object 0x45955750, a com.evermind.server.http.EvermindHttpSession),
which is locked by "ApplicationServerThread-58"JAVA STACK INFORMATION FOR THREADS LISTED ABOVE:------------------------------------------------Java Stack for "ApplicationServerThread-58":==========
at oracle.jbo.http.HttpContainer.setValue(HttpContainer.java:352)- waiting to lock <4595b408> (a java.lang.Object)
JDK/JRE (before 1.5.X) is unable to solve DEADLOCK situationsOPMN doesn’t detect such kind of situationsYou need to take action from outside
Gather troubleshooting information and restart OC4J process
![Page 47: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/47.jpg)
Classification of Problem AreasAp
ache
OC4J
DB
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
TaskManager
JMSServerJMSServer
JVM(java) AJP
ConnectionListener
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
POR
T: 3
001
New Request
assign
1.2.3.
JDK
OS
4.
1. OC4J container inside (infinitive cycles, waits on other threads, deadlocks)2. Interface to the Apache processes3. Interface to the database4. 500 Internal Error
![Page 48: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/48.jpg)
Java “Full thread dump” (conclusion)
It is an extremely useful source of information for understanding OC4J internals and troubleshooting
Take a few snaps to see dynamics
You need to analyze whole OC4J process “Full thread dump” to make conclusions
Java on Java. The ideal person for such analysis is the person who has a good knowledge of java, used technologies, application framework as well as application running on the container
DBA/Administrator should ensure all necessary information to be available to application troubleshooting person
![Page 49: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/49.jpg)
Log Files
OC4J output fileLocated at OH/opmn/log/<comp_name>.default_island.<proc_num> (example PANDA.default_island.1)Store standard output of OC4J process (JAVA DUMP, task manager info, debug info -Dajp.io.debug=true)It can be useful to add time stamp information
* * * * * echo "Timestamp: `date`" >> /u01/app/oracle/product/J2EE01/opmn/logs/PANDA.default_island.1
OC4J default-web-access.logLocated at OH/ j2ee/PANDA/log/PANDA_default_island_1/default-web-access.log
It is an analogue of access_log for Apache
![Page 50: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/50.jpg)
OS Monitoring Commands
System wide: vmstat, iostat, uptime, pstree
Top resources consumers:top, ps -eo pcpu,pid,args | sort -k1n | tail -10
Particular process information$ while [ 1 ] ; do ps -p 13285 -o pcpu | grep -v CPU ; sleep 1 ;
done25.325.325.3$ while [ 1 ] ; do ps -p 19273 -o vsz,rss | grep -v VSZ ; sleep 1 ;
done1095372 2582441095372 2582441095372 258244$ ps -wo args --width 4000 -p 6018
![Page 51: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/51.jpg)
OS Tracing Commands
strace - trace system calls and signals for particular process$ strace –s1000 -f -p 7495read(19,$ strace –s1000 –f -p 27021--- SIGSTOP (Stopped (signal)) ---read(104, "\0224\0\2\0\0", 4096) = 6send(104, "A", 1, 0) = 1send(104, "B", 1, 0) = 1send(104, "\0", 1, 0) = 1send(104, "\3", 1, 0) = 1send(104, "\6", 1, 0) = 1send(104, "\37", 1, 0) = 1send(104, "\372", 1, 0) = 1read(104, "\0224\0\2\0\0", 4096) = 6send(104, "A", 1, 0) = 1send(104, "B", 1, 0) = 1send(104, "\0", 1, 0) = 1send(104, "\3", 1, 0) = 1send(104, "\6", 1, 0) = 1send(104, "\37", 1, 0) = 1send(104, "\372", 1, 0) = 1read(104, "\0224\0\2\0\0", 4096) = 6...
pstack - print a stack trace of running processes
![Page 52: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/52.jpg)
Apache Side Information
Apache Server Status pagehttp://ap1.voava.lv:7780/server-status?refresh=15
log filesApache log files $OH/Apache/Apache/logsSometimes information can mislead
access_log192.168.2.12 - - [01/Feb/2005:16:43:34 +0200] "GET
/PANDA/NSK/NSK003/NS0291F_Browse.jsp HTTP/1.1" 500 0erorr_log[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0080: After calling
Apache's ap_bwrite(), got an return value: -1 and the client could have aborted in the middle of receiving the chunk data.
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0058: Failed to handle response chunk data passed in from oc4j via ajp13.
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0035: After hdndlingajp13 response message, got an invalid value: -2.
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0121: Failed to service request with network worker: PANDA_15 and it is not recoverable.
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0013: Failed to call destination: PANDA's service() to service the request.
![Page 53: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/53.jpg)
Java Code Decompilation
Extremely useful for troubleshooting purposesUnzip archive (jar/zip)Decompile *.class files (http://www.kpdus.com/jad.html)
OC4JOH/j2ee/home/oc4j.jar
JDBCOH/jdbc/lib/classes12.zipOH/jdbc/lib/classes12dms.jar
DCMOH/dcm/lib/dcm.jarOH/dcm/lib/oc4j_deploy_tools.jarOH/dcm/lib/oc4j_remote_deploy.jar
DMSOH/lib/dms.jar
BC4JOH/BC4J/redist/bc4j.earOH/BC4J/jlib/*OH/BC4J/lib/*
![Page 54: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/54.jpg)
Java Code Decompilation Usage Example
Decompile all oc4j.jar classesGrep through *.java files for “System.getProperty”
9.0.4 oc4j.jar…opmnPingIntervalopmn.query.debugoracle.arraylist.deepCopyoracle.aurora.jem.scheduling.threadsoracle.dms.console.DMSConsoleoracle.dms.gateoracle.dms.sensorsoracle.dms.transtrace.ecidenabledoracle.homeoracle.ias.jcache
![Page 55: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/55.jpg)
Debugging Switches
Debug switches are documented inContainers for J2EE User's Guide 10g Release 2 (10.1.2)Part No. B14011-01B Additional InformationB.8 OC4J Command-Line Options and System Properties
Debug information example (-Dajp.io.debug=true)PANDA.default_island.1
1107387907360:ThrP[50] ThrQ[0]1107387907360:CxP[20] CxQ[0] - WebC[0] RmiC[0] AJP: Input request length is: 1AJP: AJPOutputStream::write() of 1AJP: AJPOutputStream::writeChunk(): 11107387908370:ThrP[50] ThrQ[0]1107387908370:CxP[20] CxQ[0] - WebC[0] RmiC[0]
Some of available categories are : ajp, http, jms, rmi, jdbc
![Page 56: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/56.jpg)
Documentation and Internet
The information in 9.0.X documentation is very limitedOracle AS Documentation Library 10g Release 2 (10.1.2)Highly Recommended books
Containers for J2EE User's Guide (144)Containers for J2EE Standalone User's Guide (134)Server Administrator's Guide (542)OPMN Administrator's Guide (158)DCM Administrator's Guide (116)Server Performance Guide (234)
10g Best Practices 10g (9.0.4) Part No. B12223-01BC4J JDeveloper Online Help system.www.google.comhttp://otn.oracle.comhttp://metalink.oracle.com
![Page 57: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/57.jpg)
Metalink Patches
Look through available patches.
………………………………………………………………..
![Page 58: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/58.jpg)
Metalink Patches
Strongly recommended in case of 9.0.3 OC4J
![Page 59: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/59.jpg)
Metalink PatchesMonitor for new patches, which can be relative to your environment
![Page 60: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/60.jpg)
Troubleshooting information gathering (example)
![Page 61: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/61.jpg)
![Page 62: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/62.jpg)
Performance tuning
![Page 63: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/63.jpg)
Our Findings (Java Workers)
Preset java workers threads for OC4J processOH/j2ee/<comp_name>/config/server.xml<global-thread-pool
min="100" max="300" queue="30" keepAlive="-1" cx-min="100" cx-max="300" cx-queue="30" cx-keepAlive="-1"
debug="true"/>
Set min to high value. It allows OC4J not to create the additional threads on the fly
You don’t have to take care on max, keepAlive and queue parameters thenOC4J doesn’t need to manage queue
debug="true" – Allows to see statistics in the OC4J process standard output file
It is useful as troubleshooting information as well. Record appears in the output file at the end of each Task manager cycle.
<PANDA.default_island.1>1107390315893:ThrP[100] ThrQ[0]1107390315893:CxP[100] CxQ[0] - WebC[0] RmiC[0] 1107390316903:ThrP[100] ThrQ[0]1107390316903:CxP[100] CxQ[0] - WebC[0] RmiC[0] ...
![Page 64: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/64.jpg)
Our Findings (OC4J Processes)
Use many OC4J processes without enabled state replicationOH/opmn/conf/opmn.xml<oc4j maxRetry="3" instanceName="PANDA" gid="PANDA" numProcs="1">
<config-file path="/u01/app/oracle/product/J2EE01/j2ee/PANDA/config/server.xml"/><java-option value="-server -Xms512M -Xmx512M"/><oc4j-option value="-properties "/><port ajp="3001-3100" jms="3201-3300" rmi="3101-3200"/><island id="default_island" numProcs="3"/>
</oc4j>
Initially it was recommended by Oracle support while TAR resolution OC4J container is more stable on a lower users loadmod_oc4j distributes requests through 3 OC4J identical containers1/3 of users load is handled by each OC4J processDo not use state replication
It can add significant load on CPUIt complicates configurationIt can lead to opposite effect
Be carefulYou have to test this configuration for your application. It can lead to unpredictable DB connection behaviorMemory/threads configuration is for each process of the OC4J instance
![Page 65: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/65.jpg)
Our Findings (Memory)
OH/opmn/conf/opmn.xml<oc4j …
<java-option value="-server -Xms512M -Xmx512M"/></oc4j>
Set max=minJVM doesn’t need to manage memory allocation dynamicallyYou have reserved memory on the server
Java Garbage collectorThere is a lot of information around
http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/http://www.cons.org/cmucl/doc/gc-tuning.html
-verbose:gcJava Profilers
Add significant overhead to be used in production environmentsExtremely useful in test/dev environments for troubleshooting
![Page 66: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/66.jpg)
Our Findings (Memory)
Monitor memory usage in productiongrep log files for “out of memory” erroruse OS utilitiesuse dmstool
$ dmstool -table JVM | grep 3001 | grep Memory/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.value 495796 kbytes/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.minValue 495796.0 kbytes/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.maxValue 522468.0 kbytes/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.value 524032 kbytes/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.minValue 524032.0 kbytes/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.maxValue 524032.0 kbytesap1.voava.lv*J2EE01-/u01/app/oracle/product$
![Page 67: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/67.jpg)
Our Findings (Stresstesting)
The best way to tune performance is a stresstestingAccording to our experience
After Development had performed the performance testing, a lot of problems were solvedAn automatic testing tool has been used (Rational Robot)Different modules were testedDifferent load profiles were testedTesting was repeated several times until acceptable results were achieved
![Page 68: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/68.jpg)
Response Time Monitoring by DMS
OHS [c]
OC4J
DB
ApacheJVM [java]
ApplicationServerThread
DMS.JAR (java)statistics reflecting
DMS-client
mod_oc4j handle
mod_plsql handle
mod_perlmod_php
...
Request B
Request E
Handle B
Handle EHandle BHandle E
ReadResolve
Process B
Process E
JSP ProcessingServlet Processing
DMS Sensors DMS Sensors
Recive
Send
![Page 69: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/69.jpg)
Response Time Breakdown Example
$ dmstool -table ohs_server | grep time.../Apache/connection.time 48123608767 usecs.../Apache/request.time 15916131942 usecs.../Apache/handle.time 15913988395 usecsrhoracle.voava.lv*J2EE01-/u01/app/oracle/product/J2EE01/Apache/Apache/conf$$ dmstool -table ohs_module | grep time | sort -k2n | tail -2.../Apache/Modules/mod_dms.c/handle.time 8081655 usecs.../Apache/Modules/mod_oc4j.c/handle.time 15905072698 usecs$ $ dmstool -table oc4j_jspExec | grep time .../oc4j/PANDA/WEBs/PANDA/JSPs/processRequest.time 17 563 793 msecs$$ dmstool -table "oc4j_jsp(threadsafe=true)" | grep time | sort -k2n | tail -5.../oc4j/PANDA/WEBs/PANDA/JSPs/NSK/NSK003/NS0285F_searchForm.jsp/service.time 1 072 570 msecs.../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2261F_PersonsBrowse.jsp/service.time 1 407 663 msecs.../oc4j/PANDA/WEBs/PANDA/JSPs/NSK/NSK003/NS0291F_Browse.jsp/service.time 2 942 655 msecs.../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2261F_Edit.jsp/service.time 3 151 184 msecs.../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2260F_Browse.jsp/service.time 4 847 408 msecsrhoracle.voava.lv*J2EE01-/u01/app/oracle/product/J2EE01/Apache/Apache/conf$
![Page 70: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/70.jpg)
DMS Sensors Classification
Informative sensors – reflect a configuration information and some system wide statistics
ohs_child, ohs_responses, JVM, oc4j_ear, oc4j_opmn, oc4j_task, JDBC_Driver , JDBC_Connection
Average performance sensors – reflect response time totals by modules, applications, etc.
ohs_server , ohs_module, ohs_virtualHost, oc4j_context, oc4j_web_module, oc4j_jspExec
Units performance sensors – reflect response time statistics for each unit
oc4j_servletoc4j_jsp(threadsafe=true)
![Page 71: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/71.jpg)
Collecting and Reflecting DMS Statistics
DMS Sensors statistics is incrementalMake periodic snaps of statistics for further analysisDevelop interface to access statistics easily
Graphical interface for system wide statistics
Textual interface for units (JSP/Servlet) response time analysis
![Page 72: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/72.jpg)
Collecting and Reflecting DMS StatisticsSYS:PROD> @sp_jspCompleted Snapshots
Snap Id Snap Started
------- -------------------530 2005-02-03 12:33:26531 2005-02-03 13:03:24532 2005-02-03 13:33:27549 2005-02-03 14:03:32
Specify the Begin and End Snapshot Ids~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Enter value for begin_snap: 532Enter value for end_snap: 549Saved in c:\temp\532_549.log
Snap Id Snap Time ------- ------------------
Begin Snap: 532 2005-02-03 13:33:27End Snap: 549 2005-02-03 14:03:32Elapsed: 30.08 (mins)
JSP ordered by Response time
JSP Name Response Execution Average(sec) Time(sec) Count Response ------------------------------------------------------------------ --------- ---------- ----------../3001/../NSK/NSK001/AP3220F_TicketImportProcessOneByOne.jsp 956.439 2 478.22../3002/../NSK/NSK001/AP3220F_Tickets_browse.jsp 708.967 71 9.99../3001/../NSK/NSK001/M2801F_Browse.jsp 533.389 74 7.21../3003/../NSK/NSK001/P2600F_XMLImportProcess.jsp 471.106 1 471.11../3003/../LRG/l001/L3000F_Browse.jsp 293.775 92 3.19../3002/../NSK/NSK001/AP3221F_ApnAccountingDocFilter.jsp 291.382 6 48.56../3002/../common/comp/CMN_DataFilterComponent.jsp 238.75 35 6.82../3001/../NSK/NSK001/SP0301F.jsp 216.499 32 6.77../3001/../SKD/d001/D2262F_Browse.jsp 208.709 7 29.82../3001/../NSK/NSK001/SP0301F_filter.jsp 168.957 6 28.16../3002/../SKD/d001/D2260F_Browse.jsp 138.453 45 3.08../3003/../SKD/d001/D2260F_Browse.jsp 81.773 24 3.41../3001/../NSK/NSK001/Z3601F_Save.jsp 80.127 30 2.67../3001/../common/comp/CMN_LOVcomp.jsp 73.544 52 1.41../3003/../NSK/NSK001/SP0301F.jsp 72.548 29 2.5../3001/../SKD/d001/D2260F_Browse.jsp 71.642 62 1.16../3002/../NSK/NSK001/AP3221F_Save.jsp 68.509 91 .75../3001/../common/comp/CMN_DataHandlerComponent.jsp 68.46 765 .09../3002/../common/comp/CMN_DataHandlerComponent.jsp 57.144 722 .08
SYS:PROD>
![Page 73: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/73.jpg)
Monitor All the Tiers of the System
Do not forget to monitorOS and HW resources utilizationNetworkDatabase
![Page 74: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/74.jpg)
Instrument Your Code
AskTom.oracle.comhttp://asktom.oracle.com/pls/ask/f?p=4950:8:16675873436862412624::NO::F4950_P8_DISPLAYID,F4950_P8_CRITERIA:12836314571537
…Another example is asktom.oracle.com. Just go there and click on any article from the home page. You'll see a URL similar to:
http://asktom.oracle.com/pls/ask/f?p=...::NO::
If you simply type over that URL and replace the word NO with YES - you'll see the same page but with lots and lots of state/timing information dumped into it.…
![Page 75: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/75.jpg)
BC4J
![Page 76: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/76.jpg)
OC4J Internals & BC4J
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
Java native
ApplicationServerThread
ApplicationServerThread
JMSServerPORT: 3201
RMI ServerRMI ServerPORT: 3101
TaskManagerTaskManager
JMSServerJMSServerJMSServerJMSServerThreadThread
ONS
OC4JMonitor Thread
Thread
Thread
AJPConnection
ListenerPO
RT:
300
1
VM Thread VM Periodic Task Thread
Signal Dispatcher
Suspend Checker Thread
Reference Handler
Finalizer
Java native
VM Thread VM Periodic Task Thread
Signal Dispatcher
Suspend Checker Thread
Reference Handler
Finalizer
Memory Structures
Http Session List ...
Application objects
...
ApplicationServerThread
...
ApplicationServerThread
AJP 1.3 ONet
new req
3101
3201
JBO Connection poolJBO Application Module PoolJBO Application Module PoolJBO Application Module PoolJBO Application Module Pool
JBO View Objects
![Page 77: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/77.jpg)
Usage Considerations
Most information is available under JDevelopercoverUse development version of BC4JDeploy BC4J.EAR to your OC4J component (do not use home component)There is the BC4J monitoring page
![Page 78: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/78.jpg)
![Page 79: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/79.jpg)
Useful Information Sources
0. Metalink Note:273350.1 BC4J configuration tuning tips1. Metalink Note:283476.1 BC4J (ADF BC) Configuration Tuning Tips2. Metalink Note:238634.1 JBO / BC4J Parameters And Defaults3. Metalink Note:231141.1 BC4J Row Spillover and tuning4. How to Performance Tune an ADF Business Components (ADF BC) Application
September 23, 2004Steve Anderson and Michael Gantmanhttp://www.oracle.com/technology/products/jdev/tips/muench/ampooling/index.html5. Understanding Application Module Pooling Concepts and Configuration Parameters
Author: Steve Muench, BC4J Development TeamDate: August 5, 2004http://www.oracle.com/technology/products/jdev/tips/muench/ampooling/index.html6. Getting Application Module Pool Statistics to Aid with Size TuningBy Steve Muenchhttp://radio.weblogs.com/0118231/stories/2004/05/10/gettingApplicationModulePoolStatisticsToAidWithSizeTuning.html7. Overview of Temporary Tables Created By BC4JAn Oracle technical white paperApril 2002http://www.oracle.com/technology/products/jdev/htdocs/bc4j/bc4j_temp_tables.html8. How To Support Dynamic JDBC Credentials
An Oracle BC4J technoteVersion 3: April 23rd 2002
http://www.oracle.com/technology/products/jdev/howtos/bc4j/howto_dynamic_jdbc.html9. BC4J/JClient Performance StudyWritten by Steve Muench, Oracle CorporationJanuary, 2004http://www.oracle.com/technology/products/jdev/tips/muench/jclientperf/index.html10. Dive into BC4J and ADFTips and tricks from Steve Muench on using Oracle's J2EE frameworks, JDeveloper Java IDE, and XML technologies...http://radio.weblogs.com/0118231/
![Page 80: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/80.jpg)
Performance Tuning Parametersjbo.pers.max.active.nodes=1000jbo.pers.max.rows.per.node=1000jbo.doconnectionpooling=truejbo.poolminavailablesize=0jbo.poolmaxavailablesize=600jbo.poolmonitorsleepinterval=600000jbo.poolmaxinactiveage=600000jbo.ampool.monitorsleepinterval=300000jbo.ampool.maxinactiveage=600000
JBO Parameter lookup order:- Set programmatically or configuration in bc4j.xcfg file)- Applet tags- -D flags (or oc4j.properties file) - BC4J.properties file in current working directory of Java VM- /oracle/jbo/BC4J.properties resource in the runtime class path- /oracle/jbo/server/jboserver.properties resource (by default found inside bc4jmt.jar)- /oracle/jbo/common/Diagnostic.properties resource (by default found inside bc4jmt.jar)- System defined default as a fallback
![Page 81: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/81.jpg)
Summary
![Page 82: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/82.jpg)
Summary
IntroductionUsage considerationsOC4J InternalsOC4J TroubleshootingOC4J Performance tuningBC4J
![Page 83: Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)](https://reader031.vdocuments.us/reader031/viewer/2022012401/5466a3ffb4af9fc6288b46ed/html5/thumbnails/83.jpg)
Q&A