lessons learnt implementing high-performance integration using sap pi
DESCRIPTION
Describes some of the lessons learned while implementing high-performance, near-real time integration using SAP PI. This was presented at the Mastering SAP Technologies 2011 conference in Sydney, AustraliaTRANSCRIPT
Premier Partners: Supported by: Produced by:
Sascha Wenninger - Australia Post
Recipes For the Perfect PI - Simple Ingredients for Complex Requirements
Mastering SAP Technologies 2011
Bottom-Up Approach
Photo by Gidzy
Mastering SAP Technologies 2011
You’ll Hear About:
Basis
Adapter Framework
Interface Design
Questions
Key Points to Take Home
Mastering SAP Technologies 2011
30,000 Foot View
SAP POS DM
SAP ERP
SAP SCEM
…JMS Queue
XMLXMLXMLXML
SAP PI
Routi
ng
Mapping
Mapping
Mapping
Mapping
Mapping…
Mastering SAP Technologies 2011
Some Figures…
Retail Transactions from 8,000+ terminals in 3,400 stores:
500,000 – 750,000 messages per dayPeaks of 45/second into PI, then split by receiver6 receiving systems, 1-3 receivers per messageMessage size <10kB
Roll-out: January – June 2011
Mastering SAP Technologies 2011
Objectives
Near-real time: <15 minutes end to endScalable to 2,000,000 messages/day and 14 receiver systemsFuture peaks of 90/second into PINo impact on other critical PI interfaces
e.g. Parcel Tracking interface moving 2m+ events per dayAs simple as possibleFuture-proof (no ccBPM, no ABAP mapping, etc.)
Mastering SAP Technologies 2011
The Solution?
Many small steps enabled us to get there.
Mastering SAP Technologies 2011
Key Topic
Basis
Mastering SAP Technologies 2011
Basis
• System Memory• Database Stuff• Java Server Nodes• The JVM• Java Heap Space• Wily Introscope
Mastering SAP Technologies 2011
System Memory
One of the main foundations of the system
Needs to accommodate all aspects of the system:Java + ABAP + ICM + DBMS + OS + caching/buffers + other apps
Need to avoid swapping to disk at all times – causes huge GC delays!
Mastering SAP Technologies 2011
The Database
DB StatisticsWithout DB stats, our throughput dropped to 1/6th of the normal! Run daily in production, but be careful in non-prod systems
DB ReorgsPI DB tables churn very frequently and fragment after a few weeksTest your system/DBMS before and after a reorg to evaluate value
Mastering SAP Technologies 2011
Java Server Nodes
More Java Server Nodes (Java processes) provide:• Better system memory utilisation• Increased throughput via parallelisation
…but also incur costs in cluster synchronisation.
This can be significant for high volumes of EOIO interfaces with more than a few Java Server Nodes
Mastering SAP Technologies 2011
The JVM
• PI 7.1 and later use SAP’s own JVM– SAPJVM 4.1 for PI 7.0 coming soon (Notes 1495160, 1522198)
• Bug fixes and performance improvements delivered continually
Several JVM Upgrades over the past 12 months significantly reduced OOM crashes and Garbage Collection times for our systems.
This enables the use of larger heap spaces!
Patching via JSPM takes 30 minutes and 1 restart, plus some testing
Mastering SAP Technologies 2011
Java Heap Space
More Heap = process larger messages, fewer full GCs, fewer OOMs
• SAP’s recommendation has historically been 2GB per Java process. – Recently changed to 4GB (for SAP JVM)
• AusPost has been using 4GB for 9 months now. Main criteria: Full GCs should take less than 10 seconds.
Mastering SAP Technologies 2011
Java Memory Analysis
This is in SolMan!
Mastering SAP Technologies 2011
Wily Introscope
• Monitoring for the Java Stack• Basic version is free with NetWeaver Java• Provides functionality normally accessed via ABAP transactions
Mastering SAP Technologies 2011
Wily Dashboards
Mastering SAP Technologies 2011
More Dashboards
Mastering SAP Technologies 2011
And More…
Mastering SAP Technologies 2011
Tons of Dashboards!
…and this is only the free version!
Mastering SAP Technologies 2011
Our PI System
PI 7.11 SP5JVM 5.1 Patch level 59 (64 in test systems)3 Java server nodes, 4GB heap space each.AIX 6.1 on IBM POWER6, 60GB RAM
Mastering SAP Technologies 2011
Basis
System Memory Database Stuff Java Server Nodes The JVM Java Heap Space Wily Introscope
Many aspects which require special PI-Basis skills!
Mastering SAP Technologies 2011
Key Topic
Adapter Framework
Mastering SAP Technologies 2011
Adapter Framework
• Thread Behaviour• Messaging System
Mastering SAP Technologies 2011
Threads
Messages are processed using a number of adapter-specific queues.Each queue has its own thread pool.
e.g. for JMS:
Queue Purpose Default Threads
JMS Send Async Sender 5
JMS Recv Async Receiver 5
JMS Call Sync Sender 5
JMS Rqst Sync Receiver 5
Mastering SAP Technologies 2011
Threads
Threads are assigned to process messages from their respective poolsUse Wily Introscope to monitor utilisation and backlogs
Mastering SAP Technologies 2011
Threads
In ‘traditional’ scenarios, both Sender and Receiver queues and thread pools are used.
For Integrated Scenarios in the AAE, only the Sender-side queues and thread pools are used!
Threads from the Sender pools do all the work!
Mastering SAP Technologies 2011
Caveat
Some Adapters process messages serially!
e.g. JMS: 1 Adapter Thread per Communication Channel reads messages from the JMS queue
1 JMS Comm Channel3 Java Server Nodes{
Mastering SAP Technologies 2011
Threads in the AAE
Adapter Thread picks up a message from the JMS queue, then:• Executes Adapter Modules• Performs Receiver Determination• Persists 1 message per Receiver in Messaging System queue.• Confirms message and removes from the JMS queue
The longer this takes, the lower throughput will be.
Mastering SAP Technologies 2011
What does this mean?
Mastering SAP Technologies 2011
Sender-Side Performance is Really Important
Mastering SAP Technologies 2011
Implications
Throughput depends on Sender Adapter performance
Try to avoid adapter modules in Sender adapters. We achieved a 15% improvement by moving an XSLT mapping from
the JMS adapter to the Operation Mapping.
Mastering SAP Technologies 2011
Implications
Throughput depends on Receiver Determination
The initial “Adapter” thread evaluates conditions and copies the message for each Receiver.
Optimise XPath conditionsDB performance: more Receivers = more I/O
Mastering SAP Technologies 2011
Implications
Throughput depends on parallelisation
Increase the Sender-side Thread Pools and Application Threads. We use 15 JMS Sender Threads and 350 Application Threads
(also see Note 937159)
Mastering SAP Technologies 2011
2 Options
In our system, each thread can process 10-15 messages/sec
In order to consistently process 45 messages/sec from JMS, we need 4-5 concurrent JMS Sender Adapter threads.
Option 1: 5 Java server nodes
Option 2: Multiple JMS Communication Channels.
Mastering SAP Technologies 2011
Multiple Comm. Channels
We cloned the Integrated Config Object and Sender Comm Channel2 JMS Sender Adapter Threads per Java Server node
Better thread pool utilisation No increased memory requirements More scalable
This is not needed for SOAP since it’s truly parallel.Image Source: http://t.co/ZOBdlH0
Mastering SAP Technologies 2011
Adapter Framework
Thread Behaviour Messaging System
Increased importance for Integrated Configuration scenarios!
Mastering SAP Technologies 2011
Key Topic
Interface Design
Mastering SAP Technologies 2011
Integrated Configuration
Receiver Determination+ Interface Determination+ Receiver AgreementsIntegrated Configuration Object
Causes an interface to be executed in the Advanced Adapter Engine
Mastering SAP Technologies 2011
Integrated Configuration
• Message Processing entirely in Java (AAE)• Introduced in PI 7.1; more features in 7.11 and 7.3• Improved performance by:
– Reducing database I/O– Eliminating ‘stack jumping’ between ABAP and Java
• 7-10 times throughput possible• Not available for all adapters and scenarios.
Mastering SAP Technologies 2011
Example
Mastering SAP Technologies 2011
Example
Mastering SAP Technologies 2011
Constraints
There are a few things you can’t do with Integrated Configuration:
IDocs*
Multi-Mappings*
ccBPMs
But this is an Opportunity to be Creative!
*features added in PI 7.3
Mastering SAP Technologies 2011
Mapping Programs
XSLT• Easier to debug and tune• Better performance from PI 7.1 onwards• Easier to support
Consolidate mapping stepsSame performance, but less GC!• Reuse still possible via XSLT imports.
Mastering SAP Technologies 2011
Communication Channels
ABAP Proxies/Enterprise Services Use SOAP Adapter in XI 3.0 mode
JMS Senders Clone comm. Channels for greater parallelisation
Non-SAP Systems Ensure inbound queuing is available to avoid backlog in PI
Mastering SAP Technologies 2011
Design-Time Governance
Don’t Overdo It!
Find a balance which facilitates reuse but doesn’t impose too much up-front work.
Image Source: http://geekandpoke.typepad.com
Mastering SAP Technologies 2011
Questions
Mastering SAP Technologies 2011
Key Points to Take Home
Wily IntroscopeEssential for understanding performance of the Java stack
Use Integrated Configuration Objects20 minutes to migrate an existing interface
Keep your JVM CurrentContinuous improvements without much fuss
Most of the other points can be quick wins too if you don’t do them all at once.
Mastering SAP Technologies 2011
Sascha WenningerAustralia Post
@sufw
+61 403 933 472
Mastering SAP Technologies 2011
Further Reading
• PI Performance Check• SOA MiddlewareArticles on SDN
– Incl. the Performance section
Premier Partners: Supported by: Produced by: