PeopleTools Application Server & BEA Tuxedo [email protected] Consultancy Ltd.
http://www.go-faster.co.uk
23 January 20022
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Who am I?
Independent Consultant
Performance Tuning
PeopleSoft / Oracle
23 January 20023
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Rules of Engagement
Interrupt me:— if you think I have got something wrong— if you have a question— if you can’t hear me
23 January 20024
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
BEA Tuxedo
Historical Overview
Tuxedo Internal Architecture
PeopleSoft 3-tier client implementation
How does PeopleSoft Internet Architecture (PIA) fit in
Configuration issues
Monitoring
23 January 20025
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Brief Historical Background
In the beginning— monolithic
PresentationBusiness
Logic
Data Access (DBMS)
23 January 20026
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Client Server
Client Server
LAN
Presentation Business Logic
Data Access (DBMS)
SQLRaw Data
Two-tier client-server
23 January 20027
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
3-Tier Client Server
GUI+
ApplicationLogic
Data Access (DBMS)
Business Logic
Client Server
LAN
Request
Response
“1st Generation”
23 January 20028
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
3-Tier Client Server
Business Logic Data Management
TUXEDO Servers
PresentationLogic
TUXEDO Clients
Presentation
DBMS 1...DBMS n
Database Servers
Service 1Service 2...Service nTUXEDO
MessagingSQL
“2nd Generation” - Tuxedo
23 January 20029
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Where did Tuxedo come from?
Bell Labs - 1993
Transactions for
UniX
Extended for
Distributed
Operations
23 January 200210
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
What is Tuxedo?
It is the knots on the ends of the string!
23 January 200211
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
PresentationLogic
PanelLoad
PanelSave
FieldChange
etc...
PeopleSoft 2-Tier Client
23 January 200212
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
PeopleSoft 3-Tier Client
PresentationLogic
PanelLoad
PanelSave
FieldChange
etc...
Server SideClient Side
Tuxedo C
lient
Tuxedo S
erver
23 January 200213
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
PeopleSoft 2-Tier Client
DATABASE
Client Connects Directly to the Database— 1 database connection per window— inactivity timeout
23 January 200214
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
PeopleSoft 2-Tier Client
DATABASE
Client Connects Directly to the Database— 1 database connection per window— inactivity timeout
What happens in 3-Tier?
23 January 200215
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
PS/Tuxedo Internal Structure
DATABASE
BBL
BB
1st Application Server Process Started— Bulletin Board Liaison (BBL)— Reads PSTUXCFG configuration file
Shared Memory Segment— Bulletin Board (BB)
PSTUXCFG
23 January 200216
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
PS/Tuxedo Internal Structure
DATABASE
WSL
BBL
BB
Workstation Listener (WSL)— Listens for connection request from clients on a specified port— Client initially contacts the WSL
23 January 200217
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
PS/Tuxedo Internal Structure
WSL
BBL
BB
Workstation Listener (WSL)— Client needs to know where to find the WSL— Machine and port specified in Configuration Manager
23 January 200218
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
PS/Tuxedo Internal Structure
DATABASE
WSL WSH
BBL
BB
Workstation Handler (WSH)— handles traffic from client after connection— Client is assigned to a port on the WSH— WSH port range can be constrained to match a firewall— Additional WSH can be spawned if necessary— Compress Messages
23 January 200219
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
PS/Tuxedo Internal Structure
DATABASE
WSL WSH
BBL
BB
PSAUTHAUTHQ
PSAPPSRVAPPQ
PSQCKSRVQCKQ
PSSAMSRVSAMQ
PSQRYSRVQRYQ
PSAPISRVAPIQ
23 January 200220
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
How does it work?
DATABASE
WSL WSH
BBL
BB
PSAUTHAUTHQ
PSAPPSRVAPPQ
PSQCKSRVQCKQ
PSSAMSRVSAMQ
PSQRYSRVQRYQ
PSAPISRVAPIQWSHQ
23 January 200221
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
How does PIA work?
Java Client & PIA use Jolt— Java Class Library— Jolt Listener maps Java functions to Tuxedo Messages
23 January 200222
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Jolt Internal Structure
DATABASE
WSL WSH
BBL
BB
PSAUTHAUTHQ
PSAPPSRVAPPQ
PSQCKSRVQCKQ
PSSAMSRVSAMQ
PSQRYSRVQRYQ
PSAPISRVAPIQ
JSL JSH JREPSRV
JREPOSITORY
JSHQ
23 January 200223
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Java
JVM
New structure for PIA (Apache Webserver)
HTTPD ClientClient
JSL JSH
Servlet EngineServlet Container
Servlet
ServletThread
ServletThread
ServletThread
JSH
Cookie
Disk(static
objects) JSH
http 1.1address+cookie
(stateless)
Servlet thread
IS stateful
mod_jserv
Sessionid = “xy1k…vm8010
Port 8010
Webserver
Tuxedo Application
Server
23 January 200224
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
What is the difference between the web servers?
Apache— Separate Servlet Engine — Separate encryption module (Raven)
Weblogic— Contains an integral servlet container
23 January 200225
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Progressive Connection Concentration
Typical picture
35,000 users
1000 concurrent connections— 1000 users active in the last n minutes— where n is the Java + JSH timeout— therefore need 1000 threads— but maybe only 4 JVMs
100 JSH— assumes 10 clients per JSH
10-20 PSAPPSRV processes— each PSAPPSRV connects to database
1 database
35000 users
1000 connections4 JVMs
1000 Java Threads
100 JSHs
10-20 PSAPPSRVs
1 database
23 January 200226
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
psappsrv.val
psappsrv.cfg
psappsrv.ubb
PSTUXCFG
psappsrv.ubx
psadmin(configuration
dialogue)
ubbgen(PS Utility)
tmloadcf(tuxedo
process)
BBL(domainstartup)
Configuration
23 January 200227
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Configuration Files
psappsrv.cfg— contains configurable parameters
psappsrv.val— validation file for configurable values
psappsrv.ubx— configuration template for Tuxedo domain
psappsrv.ubb— configuration file for Tuxedo domain
PSTUXCFG — compiled configuration file— read by BBL at domain startup
23 January 200228
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
psappsrv.ubx -v- psappsrv.ubb
psappsrv.ubx & psappsrv.cfg— PeopleSoft file— make changes to these files— difficult to read
psappsrv.ubb— Tuxedo file— never make changes to this file— easier to read— this file is compiled by tmloadcf
to generate domain configuration file PSTUXCFG
— variables fully resolved
23 January 200229
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
psappsrv.ubb
Five (or Six) Sections— Resources
– resource limits -> BB sizing
— Machine– Resource limits. Environment
— (Network– Connection between Machines)
— Groups– Logical Split (BASE, APPSRV, JREPGRP, JSLGRP)
— Servers– Number of servers, Command line
— Services– Timeout, Load, Priority
23 January 200230
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Application Server Configuration Issue
Move quick PSAPPSRV services into a second server (PSQCKSRV) (y/n)?
— <=PT7.58 this question is misleading– services are additionally advertised on PSQCKSRV as well as
PSAPPSRV– so they are advertised on more than one queue
— >=PT7.59 and PT8.1– services are de-advertised from PSAPPSRV– quick services not used in PIA, only IC* services
– so not a problem
23 January 200231
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
DATABASE
WSL WSH
BBL
BB
PSAPPSRVAPPQ
PSQCKSRVQCKQ
PeopleTools <=7.58
Quick Service
23 January 200232
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Inside the Bulletin Board
PSAPPSRV
PSAPPSRV
PSAPPSRV
PSQCKSRV
PSQCKSRV
PSQCKSRV
PSAPPSRV
Without Load Balancing (default)
— Table of servers process where each service is advertised
— Read and populated as a stack— WSL looks for first free server
where request can be enqueued— This can cause problems
23 January 200233
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
PSAPPSRVAPPQ
PSQCKSRVQCKQ
Load Balancing
Load Balancing is about choosing the queue upon which to place the message
— by default all requests have the same load
5050 50 50
5050 50 505050 50 50
50
23 January 200234
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Load Balancing Trace
Add -r to application server command line
— CLOPT="-r -e APPQ.stderr -A -- -C psappsrv.cfg -D GP81O81 -S PSAPPSRV”
— -r enables trace to file ‘stderr’ (not channel 2)— -e overrides the output filename
— process trace file with Tuxedo utility ‘txrpt’— obtain average service time
— make load proportional to average service time
23 January 200235
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Load Balancing Trace
— Output from TXRPT
SVCNAME 18p-19p TOTALS
Num/Avg Num/Avg
--------------- -------- -------
SqlRequest 16/0.10 16/0.10
MgrClear 5/0.04 5/0.04
RamList 1/0.43 1/0.43
SamGetParmsSvc 1/0.50 1/0.50
--------------- ------- -------
TOTALS 23/0.12 23/0.12
23 January 200236
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
PSAPPSRVAPPQ
PSQCKSRVQCKQ
Load Balancing
load = execution time (seconds) x 100
7878 386 135
6363 3 766 30 3
50599
116
23 January 200237
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Multiple Queues on Busy System
You can have many server processes on a single queue
BEA recommend no more that 10 server processes per queue
If you have queue contention it is likely to impede the WSH/JSH processes enqueuing requests. You are unlikely to observe queuing with PQ.
So, if you want more than 10 PSAPPSRV processes you should use two APPQ queues
In, which case you should to enable load balancing and you must calculate the load.
— Less significant with PIA because services different.
23 January 200238
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
DATABASE
WSL WSH
BBL
BB
PSAPPSRVAPPQ1
PSAPPSRVAPPQ2
Multiple Queues on Busy System
23 January 200239
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Multiple Queues on Busy System
psadmin will not correctly calculate— MAXACCESSERS— MAXSERVICES
tmadmin->bbs— will report maximum number of services
– count of all services advertised on each server process
and if you have multiple work station listeners— MAXWSCLIENTS
23 January 200240
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
PSQCKSRVQCKQ
Service Priority
Priority is about queue jumping— servers de-queue the request— 9 out of 10 de-queued on the basis of priority— 1 out 10 is taken from the front of the queue
8080 5050 50 5050 50
23 January 200241
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Service Priority
All PeopleSoft Services have same priority— specified in SERVICES section in psappsrv.ubb— therefore there is no queue jumping
— no reason to believe that it would be advantageous
23 January 200242
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Tuning Tip
Financials— On-line journal edit & post - remote call— Service initiates a Cobol process— Process can take minutes— RemoteCall services can block APPQ— Move RemoteCall service to a separate queue
23 January 200243
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
1 domain on many nodes
DATABASE
PSAPPSRVAPPQWSL WSH
BBL
BB
TLISTEN
BRIDGE TLISTEN
BRIDGE
WSL WSH
BBL
BB PSAPPSRVAPPQ
Multi-Node Domain
DBBL
23 January 200244
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Multi-Node Domain
Distinct Bulletin Board Liaison process— Master Node
TLISTEN process— Listens for incoming requests
BRIDGE— Sends & Receives messages between nodes— in a load balancing environment
– configurable bridge load add to remote queues– limiting number of messages sent across the bridge
23 January 200245
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Multi-Node Domain
DO ATTEMPT THIS WITHOUT BEA CONSULTANCY
At UBS— PT 7.5x— susceptible to network errors
— integrated into HP service guard— ultimately this was removed from production because of
transient network errors between the nodes causing the domain to ‘partition’
23 January 200246
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
So what is going on?
tmadmin— Tuxedo command line interface— also available within psadmin
23 January 200247
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
tmadmin commands
pq - printqueues— amount of work queued and waiting— not always accurate
23 January 200248
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
tmadmin commands
psr - printservers— reports activity of individual server processes
23 January 200249
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
tmadmin commands
psc - printservices— each service on each server
23 January 200250
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
tmadmin commands
Lots of other tmadmin commands
documented in the Tuxedo documentation
Use them to— dynamically reconfigure the domain— dynamically resize the domain
– start/shutdown extra servers— monitoring scripts
– tmadmin -r for read only mode
23 January 200251
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Java Administrative Applet
23 January 200252
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Conclusion
So now you know— BEA Tuxedo/Jolt is a messaging protocol— How the messages are moved around— How PeopleSoft has used it— How the PIA has been constructed— Options for Tuning and Sizing— Load Balancing— Multi-Node domains— Monitoring Scripts
BEA Documentation— is on your CD— http://e-docs.bea.com
23 January 200253
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
More Information about PT8 Application Server
PeopleSoft 2000 Conference: Americas— 8006: PeopleTools 8.1 Application Server— 8010: Inside the Application Server
– Buzz Voelker– PeopleTools Application Server Team
23 January 200254
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Final Caveat
PeopleSoft will not support any of the changes that I have described in this presentation
Everything described in this presentation has been done, in production, at UBS
If you want to use a multi-node domain you are strongly recommended to arrange support/consultancy direct from BEA.
23 January 200255
BEA Tuxedo InternalsGo-Faster Consultancy Ltd.
Questions?
PeopleTools Application Server & BEA Tuxedo [email protected] Consultancy Ltd.
http://www.go-faster.co.uk