sap 2-tier vs 3-tier comparison nov13

IBM SAP International Competence Center

© Copyright IBM Corp. 2013

1

Comparison of SAP Application Performance

on Centralized versus Distributed Server Topologies

This document can be found on the web, www.ibm.com/support/techdocs

Date: November 2013

Walter Orb Matthias Köchl Fabrice Moyen

Hans-Jürgen Reiss

IBM SAP International Competence Center Walldorf, Germany

http://www.ibm.com/support/techdocs



2

Table of contents

TABLE OF CONTENTS ..................................................................................................................................... 2

FIGURES ........................................................................................................................................................ 3

TABLES .......................................................................................................................................................... 3

1. INTRODUCTION ..................................................................................................................................... 4

ACKNOWLEDGEMENTS ........................................................................................................................................... 4

2. TRADITIONAL SAP BUSINESS SUITE TOPOLOGIES .................................................................................. 5

COMMUNICATION BETWEEN APPLICATION SERVER AND DATABASE ................................................................................. 5

3. DESCRIPTION OF TEST ENVIRONMENT .................................................................................................. 6

HARDWARE AND NETWORK SETUP ........................................................................................................................... 6 TESTED TOPOLOGIES .............................................................................................................................................. 7

SAP 2-tier ...................................................................................................................................................... 7 Consolidated “Para 3-tier” / 3-tier-in-a-box ................................................................................................. 7 SAP 3-tier ...................................................................................................................................................... 8 3-tier Campus/Cloud Simulation................................................................................................................... 8

SAP WORKLOAD SCENARIOS ................................................................................................................................... 9 NIPING – Network Round-Trip Time (RTT) .................................................................................................... 9 SAP ERP Workload Simulation .................................................................................................................... 10 DB ROW-Select ........................................................................................................................................... 11 SAP Client Copy ........................................................................................................................................... 11

4. RESULT ANALYSIS ................................................................................................................................ 13

NIPING NETWORK ROUND-TRIP TIMES .................................................................................................................. 13 INTERACTIVE SAP WORKLOAD ............................................................................................................................... 17 SAP BACKGROUND PROCESSING ............................................................................................................................ 19

DB-Select Simulation [SINGLE_MULTI_READ] by SAP................................................................................. 19 SAP Client Copy ........................................................................................................................................... 20

WIDE AREA NETWORK SIMULATION ....................................................................................................................... 23 NIPING Network Round-Trip Times ............................................................................................................ 23 Interactive SAP Workload ........................................................................................................................... 24 SAP Background Processing ....................................................................................................................... 25 SAP Client Copy ........................................................................................................................................... 26 SAP - SD Queries [DB_READ_UPDATE] ....................................................................................................... 27

5. POWER SYSTEMS (AIX) SPECIFIC OBSERVATIONS ................................................................................ 29

ADAPTER / OS / NETWORK SETTINGS ..................................................................................................................... 29 VIO UTILIZATION ................................................................................................................................................ 31

6. SUMMARY ........................................................................................................................................... 33

7. TECHNICAL APPENDIX ......................................................................................................................... 34

DETAILED TEST LANDSCAPE NETWORK LAYOUT AT IBM CLIENT CENTER IN MONTPELLIER ................................................. 34 ANUE NETWORK LATENCY SIMULATOR .................................................................................................................. 36

8. ABOUT THE AUTHORS ......................................................................................................................... 38

9. TRADEMARKS AND SPECIAL NOTICES .................................................................................................. 39



3

Figures

FIGURE 1 SAP APPLICATION TIERS ............................................................................................................................... 5 FIGURE 2 SCHEMATICS OF TEST-ENVIRONMENT .............................................................................................................. 6 FIGURE 3 TESTED TOPOLOGIES..................................................................................................................................... 7 FIGURE 4 NIPING ROUND-TRIP TIMES ........................................................................................................................ 15 FIGURE 5 LARGE PACKETS BANDWIDTH DB<->APP-SERVER ............................................................................................. 16 FIGURE 6 SAP ERP DB-DIALOG TIMES ....................................................................................................................... 17 FIGURE 7 INCREASE OF SAP ERP SD DB-REQUEST TIMES VS. 2-TIER ................................................................................ 17 FIGURE 8 ROW SELECT TIME OVER QUERY RESULT VOLUME .............................................................................................. 19 FIGURE 9 SAP CLIENT COPY ELAPSED RUNTIME ............................................................................................................. 20 FIGURE 10 INCREASE OF SAP CLIENT COPY PROCESSING TIME VS. 2-TIER ........................................................................... 21 FIGURE 11 PARALLEL CLIENT COPY PROCESSES .............................................................................................................. 22 FIGURE 12 ANUE DELAY VERIFICATION BY NIPING ......................................................................................................... 23 FIGURE 13 ERP DB-WAN RESPONSE TIME ................................................................................................................. 24 FIGURE 14 EXPONENTIAL INCREASES IN GUI RESPONSE TIME ........................................................................................... 25 FIGURE 15 DB-BACKGROUND PROCESSING SELECT TIMES (ZTEST-ABAP) ........................................................................ 26 FIGURE 16 3-TIER CLIENT COPY RUNTIME OVER APP-SERVER. LATENCY ............................................................................. 27 FIGURE 17 DB SELECT MIX REPORT ........................................................................................................................... 28 FIGURE 18 IMPACT OF ETHERNET ADAPTER TUNING ON ROUND-TRIP TIMES (IN MS) ............................................................. 30 FIGURE 19 NETWORK ROUND-TRIP TIMES (IN MS) USING DIFFERENT PROCESSOR MODES FOR VIO SERVERS .............................. 31

Tables

TABLE 1 CATEGORIZED RESULT SERIES.......................................................................................................................... 13 TABLE 2 LAN ROUND-TRIP TIMES ............................................................................................................................... 14 TABLE 3 NIPING ROUND-TRIP TIMES ............................................................................................................................ 15 TABLE 4 INCREMENTAL DB-REQUEST TIME VS. 2-TIER .................................................................................................... 18 TABLE 5 INCREMENTAL CLIENT COPY RUNTIME VS. 2-TIER ............................................................................................... 21 TABLE 6 PARALLELIZATION GAINS OF CLIENT COPY ......................................................................................................... 22 TABLE 7 INCREASE OF DB-REQUEST TIME IN WAN VS. 2-TIER ......................................................................................... 25



4

1. Introduction

Today we see a continued trend towards 3-tier client/server implementations at SAP

customers. In combination with virtualized platforms, this provides a high degree in

flexibility and agility.

However, 3-tier topologies introduce a new level of network complexity and

performance impact (latency, bandwidth) compared to a centralized 2-tier SAP system

setup. This is even more important when we compare the speedup of CPUs, I/Os (SSD,

Flash) versus network evolution during the past years.

Furthermore, SAP customers start accommodating the cloud computing paradigm and

start using “Cloud” based application server resources for special purposes (e.g. peak

load processing, SAP upgrades). This introduces wide-area-network effects to their SAP

applications. Obviously, the influence of the network can become the dominating factor

for response and processing time.

The IBM SAP International Competence Center in Walldorf together with the IBM Client

Center in Montpellier conducted a comprehensive series of measurements to quantify

the impact of 2-tier versus 3-tier topologies for different SAP workload characteristics.

The results of this study are represented and commented in this paper.

The SAP AG performance team contributed with know-how and test reports.

Acknowledgements

Special Thanks To:

Thomas Glaser, IBM Power Systems Technical Sales Manager Europe, for his

overall project sponsorship.

Dr. Ulrich Marquard, Senior Vice President and Head of Performance, Data

Management & Scalability at SAP AG Germany, and his team for providing

valuable test cases, consulting, and contribution while performing the tests and

editing this document.



5

2. Traditional SAP Business Suite Topologies

The SAP Business Suite backend architecture comprises of three layers, a database, an

application, and a presentation layer. These layers can be deployed in either 2-tier

mode (DB and application server processes run within a single OS image), or as a 3-tier

configuration where the database and application layers run in separate OS images and

have to communicate via a logical or physical network connection. The first tier is the

presentation layer.

Figure 1 SAP Application Tiers

Each topology has different characteristics in regard to complexity, flexibility and

resiliency.

Communication between Application Server and Database

The introduction of an additional TCP/IP network stack between the database and

application servers for 3-tier configuration has an impact on SAP transaction response

and the runtime of background jobs. In 2-tier implementations (dependent on the used

database version) the database client to server communication path can be optimized

to use an inter-process communication method. In a 3-tier configuration, database

accesses always have to pass through the TCP/IP software stack as well as through

some virtual (hypervisor) or physical (network interface adapter, LAN switches, WAN)

communication layer.

Each database access from the application server needs to pass through the complete

TCP/IP layer twice: first issuing a DB-request (select, update etc.) and then receiving

the results, either in form of selected business data or a transaction commit.



6

3. Description of test environment

Hardware and Network Setup

The infrastructure was set up and operated at the Montpellier IBM Client Center. The

SAP related tests were executed remotely by members of the IBM SAP Competence

Center in Walldorf via VPN access.

The following components were used to build the test landscape:

2x IBM Power 750 servers (model 8408-E8D)

o 32-core POWER7+ CPUs at 4GHz with 1TB of RAM

o Dual Virtual I/O servers at level 2.2.2.2 (with efix IV37111m2a,

IV38225s2a, IV39725m2a)

o All logical partitions at AIX level 7.1 TL2 SP2

IBM Storwize V7000 Storage Subsystems (model 2076-324)

o FC-attached

Dedicated network and SAN infrastructure

o 1Gb and 10Gb Ethernet LAN adapters and switches

o 8Gb fibre SAN adapters and switches

ANUE Network Latency Simulator

o 2 x 1Gb Ethernet ports

Figure 2 Schematics of Test-Environment

A symmetric EtherChannel connection between the two servers was chosen to resemble

realistic customer setups. Configuration details are shown on page 34 in the Technical

Appendix section.

The main focus of the test scenarios was to evaluate the impact of different network

topologies on database request times and subsequently the transaction response and



7

runtimes of background jobs. To avoid any other load dependent influencing factors,

the workloads and the partitions were sized so that CPU and memory utilization did not

create any bottlenecks. The maximum CPU utilization on each partition was kept in the

30-40% range.

Tested Topologies

SAP 2-tier

SAP workload is processed within a single partition (beaci01).

No external network traffic between database and application

server processes is required for transaction processing.

For all test series in this paper the database and SAP Central

Services remained on this partition. Only the application

workload was executed and measured on other partitions.

Consolidated “Para 3-tier” / 3-tier-in-a-box

An additional SAP application server instance is hosted within a

second partition (beaas12) on the same physical server.

Communication via virtualized Ethernet adapters is provided by

the PowerVM hypervisor using the integrated virtual Ethernet

capabilities.

Figure 3 Tested Topologies



8

SAP 3-tier

The application server resides on

another partition (beaas21) on a

second physical server.

The partition setup was identical to

beaas12. TCP/IP traffic had to

pass through the VIO server

partitions and physical network

segments.

For the dedicated adapter tests,

the Ethernet adapters were

assigned directly to partitions

beaci01 and beaas21. Thus any

additional latency induced by the

VIO servers was eliminated for

these test series.

3-tier Campus/Cloud Simulation

Same as the SAP 3-tier scenario

with a latency simulation device

(ANUE) added to the external

communication path. Although a

single physical device, simulated

latency time was split 50:50

across the two network paths

(inbound and outbound) during

our test series. The time delays on

both network paths were set

between 0 and 125 ms, the

resulting round-trip delay

experienced by the applications

was two times this time delay.

See page 36 for more details about the ANUE device.



9

SAP Workload Scenarios

A number of different workloads were used for simulating load on the test systems.

These workloads are taken from typical SAP applications and the measured results

(focussing on database request time) can be adapted to other SAP application load

scenarios.

NIPING – Network Round-Trip Time (RTT)

SAP delivers the NIPING tool to test the SAP NI (Network Interface) layer. This layer is

used for inter application server, RFC, and GUI communication. As the name suggests

the tool works similar as ping and can be used to test the network connectivity. SAP

note 500235 (Network Diagnosis with NIPING) provides a detailed description of the

tool.

When using NIPING, at least two sessions must be started. The first one provides the

server role and it will just receive and immediately send back data packages that are

send by NIPING client sessions. The NIPING client can be controlled by command line

parameters to focus on different aspects of network connectivity (round-trip time

versus throughput).

In our test landscape, we used the following procedure:

1. Start NIPING server on beaci01 (database server)

# niping –s –I 15

This command starts the NIPING tool in a server role and automatically stops after an

idle time of 15 seconds (-I 15).

2. Start NIPING clients on all application server partitions:

# niping –c –H beaci01 -B 1 -L 10000

# niping –c –H beaci01 -B 100000 -L 1000

The first test uses a data-buffer size of one (-B 1) and sends the data packages 10000

times (-L 10000). This test measures the network round-trip time. The second test uses

a large data buffer (100000 bytes) to measure the network throughput.

The following is a sample output of the tool:

Thu Aug 29 09:29:54 2013

connect to server o.k.

Thu Aug 29 09:29:56 2013

send and receive 10000 messages (len 1)

------- times -----

avg 0.192 ms



10

max 9.754 ms

min 0.155 ms

tr 10.197 kB/s

excluding max and min:

av2 0.191 ms

tr2 10.248 kB/s

For the round-trip times we look at the av2 value, which is the average response time

for the number of messages sent excluding the maximum and minimum values. The tr2

value provides the network throughput figure.

Another tool to check network round-trip times is the standard ping utility. However,

the lowest time resolution of the AIX implementation of ping is in milliseconds. The

typical round-trip times in our test environment were in the microsecond range;

therefore, the standard ping command was not adequate for most of the test scenarios.

To validate the results of NIPING, we used another ping like program called fping (see

http://fping.org). This tool uses the Internet Control Message Protocol (ICMP) to test

the network connectivity to a target host and provides the desired round-trip time

resolution in microseconds. It has to be downloaded and installed on each server. We

collected fping results in all our test scenarios. As the results matched closely and

consistently with the output of the NIPING tests, we will provide only the NIPING

results in this paper.

SAP NIPING and fping are basic tests tools to verify the network (typically routing,

firewall settings), but can also be used for basic performance tests with no other

components included but the SAP fundamental network layer.

SAP ERP Workload Simulation

We chose the standard SAP ERP Sales and Distribution (SD) benchmark to check the

impact of network latency on online transactions. The SD benchmark reflects a typical

SAP application where the SAP application server is communicating with the database.

This benchmark is also used as a reference for all SAP sizing (SAPS) calculations (for

details please see: http://www.sap.com/sizing).

The transaction load driver was installed together with the database/central instance

partition (beaci01) and a fixed number of users were simulated on all three application

servers simultaneously to produce a steady work-load of online ERP transactions. All

users executed the same set of predefined transactions in a fixed number of loops. With

this setup a single test run produced the comparison data for all three network

topologies (2-tier, para-3-tier, and 3-tier).

The key metric that is impacted by the network latency from an application point of

view is the database request time for dialog and update work processes. We focused on

these statistics in the result analysis.

We modified the test procedure for the wide-area-network (WAN) runs. To produce a

reliable set of results, the SAP SD benchmark toolset requires that all tested application

http://fping.org/

http://global.sap.com/campaigns/benchmark/appbm_sd.epx

http://www.sap.com/sizing



11

servers start and finish the high-load interval (the period of time where all users are

logged on and generate a steady work-load) on each server at about the same time.

The WAN simulation has a heavy impact on end-user response times, so the 3-tier

application server connected via the delayed network path experiences a significantly

longer high-load interval.

To avoid this problem, we performed a baseline measurement simulating work-load

only on central instance application server running on beaci01. For the subsequent runs

with an increasing number of network round-trip delays, the work-load was then

executed on the 3-tier application server instance beaas21 only. This also means that -

compared to the earlier runs without simulated network delays - only one third of the

overall total workload was executed.

DB ROW-Select

This special load scenario executes a number of simple DB calls. This scenario is typical

for network I/O intensive background processing with many database accesses like

running analysis reports or end of year activities, etc.

We distinguish between two flavors:

1. [SINGLE_MULTI_READ] Single and multiple database reads within one

statement. This is typically found for analysis reports and mass data reads.

2. [DB_READ_UPDATE] Here database reads and writes are executed in a typical

SAP application manner. This scenario reflects heavy network I/O load SAP

applications, like mass data updates.

SAP Client Copy

We used local SAP client copies as another method to simulate the impact of network

latency on background jobs with a significant number of database accesses. This

workload shows both effects of high network I/O load combined with optimized data

access strategies.

A client copy has to read all client specific data of the specified source client from the

database and then insert the same data again using the new client number.

We selected the SAP_ALL client copy profile, which copies “all client specific data

without change documents”.

To ensure that we always copied the same amount of data, we first created a new client

100 as a copy of one of the SD benchmark clients. This client was not used for any

other tests, so after the initial copy, the client data was static.

Next we copied this client to a new target client 200. The first client copy to a new

client just inserts the data in the database. Any subsequent client copy to the same

client will delete existing data before inserting the copied data in the new client. So

after the initial copy to the target client 200, all following client copies used the same

amount or read/delete/insert statements on the database.

In our test system a client copy had to copy 62.544 tables with about 4 GB of data.



12

As test result we noted the runtime in seconds as reported at the bottom of the detailed

file log output of the Client Copy/Transport Log Analysis tool (transaction SCC3):

…

Exit program USERBUF_RESET successfully executed 13:41:15

Selected tables : 62.544

Copied data in kBytes : 4.098.311

Deleted data in kBytes : 4.098.329

Program ran successfully

Runtime (seconds) : 2.017

End of processing: 13:41:15

For each test scenario, we ran the client copy on all three application servers. The client

copies were scheduled as background jobs using the desired application server as

background server without any parallel processing.

Parallel Process Client Copy

To check the effects of an increased network load on the client copies, we also ran a

number of tests with various degrees of parallel processing.

The Client Copy tool (transaction SCCL) contains a “Parameters for Parallel Processes”

pushbutton that allows configuring the maximum number of processes and a RFC

server group to be used for the processing. The client copy process will run as a single

process during the analysis and post-processing phase, but will distribute work

packages to parallel processes to handle the actual copy operations.

We created three RFC server groups, each group containing only a single application

server. The client copies were scheduled similar to the previous scenarios on all three

application server. However, during the background scheduling, the parallel processes

option was used to specify the desired amount of parallel processing and the

appropriate RFC group for the tested application server.

We tested three scenarios with three, six, and nine parallel processes on each

application server.



13

4. Result Analysis

For analysis, the test series were categorized according to the topologies and physical

network characteristics. The provided charts will contain the following measurement

series:

Table 1 Categorized result series

Where applicable, differences within a category caused by (intentional) changes of

network parameters are discussed in the chapter “Adapter / OS / Network Settings” of

this paper.

Often results are normalized to the 2-tier setup which allows for a quick assessment of

the relative effects of different setups. Results are sorted ascending, while the sequence

of series can vary per test. Absolute times will be specified to allow quantified

extrapolations (at one’s own risk).

NIPING Network Round-Trip Times

Network latency in Ethernet networks is often measured and documented as one-way

latency (the amount of time it takes from the source sending the packet until the target

receiving it). From an application point of view, the more important value is the round-

trip latency, which is the one-way latency from the source to the destination plus the

one-way latency from the destination back to the source. This round-trip network

latency does not include the amount of time that is spent in an application on the

destination system processing the packet. Any references to network latency in

this document will refer to round-trip latencies (unless explicitly stated

otherwise).

Before starting to analyse database request times on SAP application servers in 3-tier

configurations, we recommend to perform a quick sanity check to verify the network

round-trip times between application servers and the database server. We recommend

using the SAP NIPING tool, as it is already available in the SAP instance binary

directory1.

1 Please note that according to the definition above, network round-trip latency does not include packet processing time on

the destination. Nevertheless, NIPING provides reasonable good approximations of this network round-trip latency, as the server process just receives the packets and immediately sends them back again.

Topology / Series

2-tier

Para 3-tier

3-tier virtualized 1 Gbit Ethernet

3-tier virtualized 10 Gbit Ethernet

3-tier dedicated 1 Gbit Ethernet

3-tier dedicated 10 Gbit Ethernet

3-tier WAN virtualized 1 Gbit Ethernet



14

The following are some typical values for expected NIPING round-trip times with small

packages in current customer LAN networks:

Round-trip time in

ms

Rating

less than 0.3 Very Good

between 0.3 and 0.7 Average

larger than 0.7 Network configuration should be

included in the performance analysis of

database response time problems

Table 2 LAN round-trip times

The chart below shows the NIPING round-trip times measured with the different

topologies. We measured round-trip times of about 30 microseconds for the 2-tier

configuration. From an application perspective this means that each database request

would have a network delay of at least 30 microseconds contributing to the response

time2.

Moving to the para-3-tier configuration, where both partitions reside on the same

physical server and communicate via the PowerVM hypervisor, we saw that the round-

trip time doubled to about 60 microseconds.

For the 3-tier configurations the round-trip times increased again to about 100

microseconds with dedicated Ethernet adapters.

In a fully virtualized 3-tier setup, all network packets have to pass through a VIO server

partition on both machines. Therefore the network round-trip times increased again to

about 200 microseconds.

For small network packets, there’s little difference between the 10 Gbit and 1 Gbit

Ethernet adapters results. For network transfers with very small payloads, most of the

time is actually spend in processing the TCP/IP protocol stack on the servers and the

advantage in physical network speed does not play a significant role. However, when

looking at network transfers with larger payloads, the advantage of 10 Gbit networks

becomes apparent as they provide significantly improved network round-trip times

leading to better application response times.

2 Database clients in 2-tier configurations that can be configured to use an IPC connection for local communication might

achieve slightly better values for this communication delay.



15

Figure 4 NIPING round-trip times

niping round-trip time/ms

small packets (10.000x 1Byte msgs.)

large packets (1.000x 100.000 Byte msgs.)

2-tier 0,03 0,30

para 3-tier 0,06 0,66

3-tier ded. 10Gb 0,10 0,77

3-tier ded. 1Gb 0,11 1,82

3-tier virt. 10Gb 0,19 1,41

3-tier virt. 1Gb 0,21 2,14

Table 3 niping round-trip times

Although the main focus of this paper is on network round-trip delays, we’ve included

the following chart documenting the network throughput rates of the various scenarios

for completeness. The throughput rates become more relevant for administrative tasks

like database backups or for database queries that transfer a large amount of data per

request.

The test with small (one byte) data packets does not really test the network throughput

but the latency and the results show that there is very little difference between the 1Gb

and 10Gb Ethernet numbers. The additional processing required on the VIO servers

actually lead to lower throughput numbers for the 3-tier virtual 10 Gbit compared to the

3-tier dedicated 1Gbit scenario.



16

The picture changes for the real throughput test with large packets. The faster speed of

the 10 Gbit setup provides a significant improvement in network throughput and that

even a fully virtualized 10 Gbit Ethernet scenario provides better throughput rates than

a scenario with dedicated 1 Gbit Ethernet adapters and switches.

Figure 5 large packets bandwidth DB<->App-Server



17

Interactive SAP Workload

The chart in Figure 6 shows the average database request times for the SAP SD

benchmark transactions for the dialog and update tasks.

Figure 6 SAP ERP DB-Dialog times

The next chart and Table 4 show the relative increase in database request times of the

various scenarios compared to the 2-tier reference measurement.

Figure 7 Increase of SAP ERP SD DB-Request times vs. 2-tier



18

Incremental DB-Request times vs. 2-tier

Query Update

para 3-tier +16% +18%

3-tier ded. 10Gb +27% +34%

3-tier ded. 1Gb +57% +64%

3-tier virt. 10Gb +70% +81%

3-tier virt. 1Gb +78% +88%

Table 4 Incremental DB-Request time vs. 2-tier

The SAP SD benchmark transactions and their database access patterns have been

highly optimized in the past. With the increase of processing power over the last

number of years, the transactions have now become more or less light-weight

transactions.

From an end-user perspective, the more interesting statistic is the database request

time for the task type dialog as it is part of the transaction response time. Comparing a

dialog database request time of little more than 5 milliseconds in the 2-tier setup with

about 10 milliseconds for the 3-tier configuration does not sound that much. An end-

user would certainly not notice the difference for this particular set of transactions.

However, the relative increase in database request time in our test scenarios was more

than 70%.

In customer production systems, there are many business critical transactions that are

substantially more heavyweight and often have to fit in common service level

agreements (SLA) for end-user response times below one second. Let us assume a

transaction with a response time of just under one second, where the database request

time contributes 40% (or 400 milliseconds) of that time. If the database request time

increases by 70% on a 3-tier application server instance, the end-user response time

for this transaction would increase to 1.28 seconds and suddenly violate the SLA. An

end-user will probably complain about a nearly 30% longer dialog response time in this

case.

In general the network part is not dominating the overall response time for SAP

standard transactions and they work well in 3-tier scenarios. Nevertheless, if a

customer has long-running business critical transactions with many database accesses,

one should carefully analyze the potential impact of additional network delays before

moving to a 3-tier configuration.



19

SAP Background Processing

DB-Select Simulation [SINGLE_MULTI_READ] by SAP

The chart below shows the differences in database response times for simple select

statements using the different topologies. The report was parameterized to perform

2000 select statements for each select variant (select single, select 2 rows, select 3

rows …). The database request time in this chart was normalized to an average time

per selected row to provide a better comparison of the results for the different variants.

Figure 8 Row select time over query result volume

As expected an increase in the network delay has a much worse impact on database

response times for applications doing a lot of single record reads compared to optimized

select statements where more rows are retrieved with a single query. This is also

described in the SAP Performance Standard. The test series clearly show the potential

improvements by optimizing the application.

The worst case scenarios obviously are application reports that perform millions of

database accesses retrieving one record only for each access. Running in a 2-tier setup,

each database request would take about 94 microseconds. Comparing this to a common

customer setup with a fully virtualized environment and a 10 Gigabit Ethernet

infrastructure, the database request time for 3-tier scenarios would increase to about

260 microseconds, which is a factor of 2.75 slower or a difference of 166 microseconds

per access.

Let’s assume a hypothetical background job that performs ten million such database

accesses. The total database request time for this job would be about 15 minutes for

the 2-tier scenario and 43 minutes with the 3-tier configuration.

A significant portion of that time could be compensated by rewriting the application to

fetch more than one record with each database access. If an application rewrite is not

an option, then we clearly recommend scheduling such background jobs on a 2-tier

application server instance only.



20

Please note that this 3-tier number was measured in an environment with very good

NIPING network round-trip times of less than 200 microseconds. Customer

environments with NIPING round-trip times of more than 500 microseconds are not

uncommon and this additional network latency would substantially increase the

difference in database request time for the 3-tier scenario.

SAP Client Copy

The SAP client copy process already exploits optimized database statements. It is a

typical example for how to minimize the impacts of network delays by using optimal

database access strategies.

Despite this optimization, there is still a significant impact of the 3-tier scenarios on the

overall runtime. The absolute runtime increased from about 55 minutes for the 2-tier

scenario to 60 minutes for the para 3-tier configuration.

For the fully virtualized setups, the runtime was 72 minutes with the 10 Gbit Ethernet

and 75 minutes with 1 Gbit Ethernet network.

Figure 9 SAP Client Copy elapsed runtime

The following Figure 10 and Table 5 show the increase in runtimes compared to the

2-tier reference run.



21

Figure 10 Increase of SAP Client Copy processing time vs. 2-tier

Incremental SAP Client Copy time vs. 2-tier

para 3-tier 9%

3-tier ded. 10Gb 14%

3-tier ded. 1Gb 21%

3-tier virt. 10Gb 31%

3-tier virt. 1Gb 36%

Table 5 Incremental Client Copy runtime vs. 2-tier

Optimized database access strategies help to reduce the impact of network round-trip

delays in 3-tier configurations, but even then the overall runtime of our client copy

scenarios increased by more than 30%.

Compared to the physical separation of servers, a virtualized 3-tier setup on a single

Power System showed only a relatively small increase in elapsed processing time.



22

Effects of Process Parallelization for Client Copy

One way to mitigate the problems with long running background jobs is to exploit

parallel processing. This is not always an option, but when it is available, it can be used

to reduce the processing time to acceptable levels at the expense of an increased CPU

usage.

We’ve compared the runtime of SAP client copies for all three tested scenarios with

various levels of parallel processing to the reference number of the single process client

copy on the 2-tier application server instance.

The chart below shows that the speed-up with parallel processing was pretty constant

across all topologies.

The second purpose of this test scenario was to check, whether introducing additional

network load (by running multiple processes in parallel) would have noticeable

performance impacts. The slope of the curves for the various scenarios is about the

same, which shows that the chosen workload was not high enough to reach any

network limitation.

Figure 11 Parallel Client Copy processes

This table shows the average speed-up for all three topologies:

#of parallel processes 3 6 9

Runtime acceleration 33% 47% 53%

Table 6 Parallelization gains of Client Copy



23

Wide Area Network Simulation

We used the workloads described in section SAP Workload Scenarios to measure the

impact of wide area network latencies on SAP database request times.

The network latency simulator (ANUE) was used to add a delay to the communication

path between beaci01 and beaas21. Therefore, the test procedure was modified slightly

to run the longer running workloads (ERP benchmark, simulated background jobs, client

copies) only on the remote application server instance beaas21.

We measured the performance impact at various simulated network delays between 1

and 125 milliseconds. After the test sequence with 5 milliseconds network latency

completed, it was obvious that it would not make sense trying to run the ERP SD

benchmark with an even higher network delay and a SAP client copy would have taken

days. Therefore we decided to reduce the test cases to NIPING and the simulated

background jobs for the remaining two latency tests (50 ms and 125 ms).

The measured round-trip time for the LAN tests with dedicated 10 Gb Ethernet adapters

(without the ANUE network delay) was about 0.1 ms (100 microseconds) and 0.2 ms in

a fully virtualized environment.

As the ANUE device provided us the opportunity to simulate LAN networks which do not

provide such good round-trip times as our test landscape, we decided to perform also a

few tests with network latency delays in the typical LAN range (latency delays of 0.1,

0.25, and 0.5 ms which correspond to round-trip delays of 0.2, 0.5, and 1 ms).

NIPING Network Round-Trip Times

The following chart shows the NIPING round-trip times with various simulated network

delays. The results are essentially exactly the same as in the 3-tier measurements

before plus the simulated round-trip delay added on top. This is no surprise as the

NIPING processes basically do nothing else other than sending and receiving network

packets.

We used these NIPING tests mainly as a verification that ANUE network delay was

configured as intended before the subsequent test scenario runs.

Figure 12 ANUE delay verification by niping



24

Interactive SAP Workload

While the end-user response time for the SAP SD benchmark transactions increased

only slightly for the 3-tier application server instance in a controlled LAN environment,

the picture changes completely when extending the tests into a WAN network scenario.

The results show that the application response time gets unacceptable pretty fast.

The impact is biggest on update processing. As mentioned before, update processing

happens asynchronously and does not directly influence end-user response time.

However, the longer running update processing will block the SAP work-process that

executes the update task. Eventually, all available update work-processes will be busy

leading to additional queuing effects.

At a certain point, long running updates will effect dialog transactions too, as some

subsequent transactions expect that previously created documents are already stored in

the various business tables on the database.

What the charts do not show is that we had to change the work-process configuration a

number of times to compensate for the longer database processing. Otherwise,

transactions would have aborted (because of update being too slow) and the dialog und

update response times would have been much higher, as incoming requests would have

to wait for free work-processes.

We ran the LAN tests with a configuration of 12 dialog and 3 update work-processes. To

achieve a successful run for the 10 ms round-trip delay test, we had to double the

number of dialog work-processes to 24 and increase the number of update work-

processes to 18.

Figure 13 ERP DB-WAN Response time



25

Simulated round-trip delay/ms

0 0,2 0,5 1 2 6 10

Increase DB dialog request time vs. 2-tier

+98% 168% 295% 535% 1004% 2851% 4665%

Increase DB update request time vs. 2-tier

+148% 250% 430% 764% 1407% 4213% 8374%

Table 7 Increase of DB-Request time in WAN vs. 2-tier

Figure 14 Exponential increases in GUI response time

SAP Background Processing

The next chart shows the average database request time per row for different network

delays and should be compared with Figure 8 on page 19. The scale on the y-axis is

logarithmic to allow for the wide variance of simulated network round-trip delays.

As in the NIPING case, the results are virtually the same as in the 3-tier measurements

shown in Figure 8 plus the simulated round-trip delay added on top.

Once again, using optimized database access patterns help to mitigate the impact of

the additional network delay.

The average select time per row in the 2-tier scenario was about 25 microseconds when

selecting 9 rows with each query. The respective number for the one millisecond round-

trip delay was about 160 microseconds.

Our hypothetical background job fetching 10 million rows with 9 rows per select would

run a little longer than 4 minutes on a 2-tier application server instance. The same

background job running with only one millisecond network delay would already need

more than 26 minutes.



26

This makes it obvious that applications performing a large number of database access

would not perform well in WAN environments.

Figure 15 DB-Background Processing Select times (ZTEST-ABAP)

SAP Client Copy

For this set of tests we scheduled the client copy with 9 parallel processes. The runtime

of the 2-tier reference measurement was about 28 minutes. This increased to 39

minutes for the 3-tier configuration.

The runtime increased to 70 minutes already with a 1 ms round-trip delay, which is

actually a delay one might experience in a somewhat problematic LAN configuration.

Moving to the higher delays, even with a network round-trip delay of 10 ms only, the

client copy runtime already increased to more than 5 hours.

This clearly shows that it does not make sense trying to run background jobs with a

major database component in a WAN setup.



27

Figure 16 3-tier Client Copy runtime over App-Server. Latency

SAP - SD Queries [DB_READ_UPDATE]

In this test we used the load scenario [DB_READ_UPDATE], which simulates SAP

transactions with heavy network I/O. The test report executed several sets of database

selects in a sequence.

We measured the total runtime and the average runtime for selecting a single database

row in milliseconds. Increasing the network round-trip delay once again shows a rise in

total response time caused by the increased request time for fetching a single database

row. With higher network round-trip times, the total test response time rises up

extremely.

Please note that the chart uses a logarithmic scale for the y-axis, because of the huge

differences in average request time per row (from about 500 microseconds in the 2-tier

case climbing up to almost 100 milliseconds for the test scenario with a simulated

network round-trip delay of 50 milliseconds).



28

Figure 17 DB Select Mix Report



29

5. Power Systems (AIX) Specific Observations

Adapter / OS / Network Settings

During the SAP tests, we tested various AIX system parameters, in particular at

network level, and analysed the impact on SAP results.

Network tuning (referred as “tuned” in tests):

During network traffic, interrupt coalescing is introduced to avoid flooding the host

with too many interrupts. Consider a typical situation for a 1-Gbps Ethernet: if the

average package size is 1000 bytes, to achieve the full receiving bandwidth, there will

be 1250 packets in each processor tick (10ms). Thus, if there is no interrupt coalescing,

there will be 1250 interrupts in each processor tick, wasting processor time with all the

interrupt processing.

Interrupt coalescing is aimed at reducing the interrupt overhead with minimum latency.

There are two typical types of interrupt coalescing in AIX network adapters.

Most 1-Gbps Ethernet adapters, except the HEA (Host Ethernet Adapter) adapter, use

the interrupt throttling rate method, which generates interrupts at fixed frequencies,

allowing the bunching of packets based on time. The default interrupt rate is controlled

by the intr_rate parameter, which is 10000 times per second by default.

Most 10-Gb Ethernet adapters and HEA adapters use an advanced interrupt coalescing

feature. A timer starts when the first packet arrives, and then the interrupt is delayed

for n microseconds or until m packets arrive.

For the 10-Gb Ethernet adapter, the n value corresponds to intr_coalesce, which is 5

microseconds by default. The m value corresponds to receive_chain, which is 16

packets by default. Note the attribute name for earlier adapters might be different.

Today’s Power7+ processors are much faster than the processors that were dominant

when the 1G/10G interrupt coalescing AIX default parameters were chosen.

Additionally, the potential overhead of the processor is also linked to the type of

workload the processor needs to deal with. So we did several tests deactivating the

interrupt coalescing parameters on the 1G and 10G physical adapters.

1G adapters: Changing intr_rate parameter of the physical adapter from 10000 to 0.

10G adapters: Changing intr_coalesce parameter of the physical adapter from 5 to 0

and changing receive_chain of the physical adapter from 16 to 1.

Test conclusion

1G : intr_rate parameter=0 definitely improves the network latency. A system

measurement tool as well as SAP NIPING test demonstrate the latency is dropping

about 0.1ms thanks to this tuning (for example, from 0.28 to 0.2 ms for a system test

with VIO servers)

10G: The system measurement tool as well as SAP tests do not show sensible

improvement using these tuning.



30

Figure 18 Impact of Ethernet adapter tuning on round-trip times (in ms)

Throughput tuning (referred as “tuned_2” in tests):

Using the experience of other benchmarks done in Montpellier that focused on network

throughput, we did some tests (referred as "tuned_2") with specific network

throughput parameters:

LPAR virtual adapters:

mtu=65390 (previously 1500)

mtu_bypass=on (previously off)

VIO Physical adapters:

flow_ctrl=yes

jumbo_frames=yes

large_send=yes

VIO Etherchannel adapters:

use_jumbo_frame=yes

hash_mode=src_dst_port

mode=standard

VIO SEA adapters:

jumbo_frame=yes (previously no)

large_receive=yes (previously no)

largesend=1 (previously 0)

Nevertheless, the SAP tests which were more latency driven did not show any

significant improvement using these specific throughput parameters.



31

VIO Utilization

The PowerVM virtualization platform offers several options to give processor capacities

to a logical partition:

Dedicated processors: simply assign real physical processors (cores) to the

logical partition.

Shared processors (micro-partitioning): create a processor shared pool

with physical processors (cores) and then assign a virtual fraction of this pool to

the logical partition.

Donating (shared dedicating) processors: Compromise between the

dedicated mode and the shared mode. Donating mode offers the simplicity of

the dedicated mode with approaching performance, and almost the flexibility of

the shared mode.

These options are available not only for general-purpose logical partitions but also for

specific logical partitions such as Virtual I/O servers.

When doing network tests, the best compromise regarding performance is achieved

with VIO servers in donating mode. This is also a recommendation from SAP specialists

and is the mode we used in all SAP test scenarios.

Figure 19 Network round-trip times (in ms) using different processor modes for VIO servers

Note:

The AIX nmon monitoring tool provides several reports about processors consumption

such as “LPAR” and “CPU_ALL” (but there are others), but all are not fully accurate

according to the partition’s processor mode:

LPAR is more adapted to shared mode

CPU_ALL is more adapted to dedicated mode and donating mode

When using dedicated partition, the nmon tool does not even provide the “LPAR” report

as it is definitely not adapted to this mode. But as donating mode is a dedicated mode

with some shared mode inside, the nmon tool provides both LPAR and CPU_ALL reports

in such case, even if the LPAR report is definitely not accurate.



32

Hereunder an example in our case, using VIO servers in donating mode:

The LPAR report seems to show the VIO server is working harder before the test starts

and after the test stops!

While the CPU_ALL report shows accurate data without this side effect before and after

the test begins.



33

6. Summary

The measurement results documented in this paper show that the network round-trip

time for database accesses can have a substantial impact on SAP background job

processing times and transaction response times. An increase in processing time of

more than 30% compared to a 2-tier scenario is quite common, even for applications

which make use of optimized database access patterns.

Less optimized applications that perform a large number of database requests – i.e.,

where each request returns only a few records - are especially problematic. Their

runtime can easily increase by 100% or more.

Before moving an existing 2-tier SAP landscape to a 3-tier configuration, one should

carefully examine the database times and access patterns of affected critical business

transactions and background jobs to ensure that the additional network latency does

not result in SLA violations.

Reversely, for existing 3-tier implementations a good method to improve processing

times for problematic applications with a significant amount of database request times

is to run such applications on a 2-tier application server instance (using background job

scheduling or special logon groups for interactive transactions).

The implementation of a 3-tier-in a box setup (multiple partitions within a single

physical server, in this document referred to as “para 3-tier”) using PowerVM

connectivity provide an alternative with relatively low (<10%) incremental network

delays while maintaining a high degree in resource and administration flexibility.

An often advertised use case for hybrid cloud scenarios is to buy incremental

application server capacity for certain periods of time only, for example for month-end

or year-end processing.

According to our test results this temporary increase of processing capacity is only

feasible for applications with a rather small database component and thus low network

traffic between DB-server and remote (cloud) application servers.

Most month-end and year-end processing jobs perform a large number of database

accesses. Running them in a cloud would most likely fail, unless a cloud provider can

guarantee network round-trip times equivalent to Gbit Ethernet LAN networks.

In many cases, PowerVM capabilities for non-disruptive resource adjustment will

provide a better option delivering constant processing times (given a flexible billing

model is established).



34

7. Technical Appendix

Detailed Test Landscape Network Layout at IBM Client Center in Montpellier



35



36

ANUE Network Latency Simulator

The ANUE was configured in the 1Gbit network, as it provides only two 1Gbit interfaces

for external connections.

One can modify the latency at blade1 level and at blade2 level. During out tests, we

kept both values symmetric.

Looking at server YO88WU, all outbound traffic is delayed by blade1 and passed

through by blade 2. All inbound traffic is delayed by blade2 and passed through by

blade1. The overall round-trip time referred to during this document is defined by the

aggregate delay time of both ANUE blades plus the native network latencies in the LAN

segments described above.

Some screenshots from ANUE configuration GUI:



37

Note that the “Latency per Packet” output in this terminal window is actually the round-

trip time measured in the “netlatency.ksh” script. That means a network delay of 4 ms

configured on each blade results in a round-trip time of about 8.2 ms.

ANUE Network statistics panel:



38

8. About the authors

Walter Orb – [email protected]

Walter Orb is a technical consultant working at the IBM SAP International

Competence Center in Walldorf. He has more than twenty years’ experience with

SAP on AIX and Power Systems, with a major focus on system performance,

benchmarks, and large-scale system tests.

Matthias Köchl – [email protected]

Matthias Köchl is a member of the IBM SAP International Competence Center in

Walldorf. As a certified Senior Architect and CIM certified Marketer he is in charge for

field enablement, marketing and education around SAP Solutions running on the IBM

POWER and PureFlex platforms.

Fabrice Moyen – [email protected]

Fabrice Moyen is a benchmark manager at the EMEA IBM Client Center in

Montpellier, France, working in the IBM Power benchmark team, and with more

specific competencies on PowerHA, IBM Systems Director and PureFlex. He is also a

member of the new IBM Power Linux Center recently inaugurated in Montpellier (in

addition to three other IBM Power Centers in Austin, New-York and Beijing).

Hans-Jürgen Reiss - [email protected]

Hans-Jürgen Reiss is a member of the Performance & Scalability team from SAP

Walldorf, Germany. He is a specialist in network sizing and analysis for SAP

applications. He is working with SAP development on architectural design and

optimization of SAP applications for SAP cloud solutions.

mailto:[email protected]

mailto:[email protected]?subject=Whitepaper%20Server%20Topologies%20for%20SAP%20Applications

mailto:[email protected]



39

9. Trademarks and special notices

© Copyright IBM Corporation 2013.

References in this document to IBM products or services do not imply that IBM intends to make them available

in every country.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines

Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked

on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S.

registered or common law trademarks owned by IBM at the time this information was published. Such

trademarks may also be registered or common law trademarks in other countries. A current list of IBM

trademarks is available on the Web at "Copyright and trademark information" at

www.ibm.com/legal/copytrade.shtml.

SAP and other SAP products and services mentioned herein, as well as their respective logos, are trademarks

or registered trademarks of SAP AG in Germany and in several other countries all over the world.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its

affiliates.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United

States, other countries, or both.

Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other

countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used IBM

products and the results they may have achieved. Actual environmental costs and performance characteristics

may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published

announcement material, or other publicly available sources and does not constitute an endorsement of such

products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available

information, including vendor announcements and vendor worldwide homepages. IBM has not tested these

products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM

products. Questions on the capability of non-IBM products should be addressed to the supplier of those

products.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and

represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of

the specific Statement of Direction.

Some information addresses anticipated future capabilities. Such information is not intended as a definitive

statement of a commitment to specific levels of performance, function or delivery schedules with respect to any

future products. Such commitments are only made in IBM product announcements. The information is

presented here to communicate IBM's current investment and development activities as a good faith effort to

help with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled

environment. The actual throughput or performance that any user will experience will vary depending upon

considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the

storage configuration, and the workload processed. Therefore, no assurance can be given that an individual

user will achieve throughput or performance improvements equivalent to the ratios stated here.

Photographs shown are of engineering prototypes. Changes may be incorporated in production models.

Any references in this information to non-IBM websites are provided for convenience only and do not in any

manner serve as an endorsement of those websites. The materials at those websites are not part of the

materials for this IBM product and use of those websites is at your own risk.

http://www.ibm.com/legal/copytrade.shtml

sap 2-tier vs 3-tier comparison nov13

Documents

sap workload scenarios

sap client copy

sap erp workload simulation

tier campuscloud simulation

application server

distributed server topologies

tested topologies

niping network