how linux schedulers and other i/o related setups ... · how linux schedulers and other i/o related...

How Linux Schedulers and other I/O related setups can influence the performance of a SAP system

How Linux schedulers and other I/O related setups influence the performance of an SAP system

Applies to:All SAP products that run with an underlying MaxDB Database. In general this article should also be valid for other databases on Linux like Oracle or IBM LUW. The software used in this article is a Unicode SAP ERP 2005 SR1 with MaxDB.

SummaryThe wrong setup of a disk subsystem for a SAP system has a big impact on the performance of the system, especially the database. Thus using the best disk setup should be the main goal in the setup phase of a server. This article gives you an overview of different possibilities how a setup of a database can be realized and how the performance of these setup are in comparison.

Author(s): Hannes Kuehnemund

Company: SAP AG

Created on: 5 January 2007

Author BioHannes Kuehnemund joined SAP in late 2000 where he finished his diploma theses. He works in the SAP LinuxLab since Sep 2003 and is the SDBenchmark and Linux SAP Certification specialist there.

SAP DEVELOPER NETWORK | sdn.sap.com BUSINESS PROCESS EXPERT COMMUNITY | bpx.sap.com

© 2006 SAP AG 1


Table of Contents

How Linux schedulers and other I/O related setups influence the performance of an SAP system..1

Applies to: ............................................................................................................................. ............ 1

Summary ................................................................................................................... ....................... 1

Author Bio ............................................................................................................................. ............ 1

Table of Contents................................................................................................... ..........................2

Motivation ................................................................................................................. ........................ 3

Test scenario description ...................................................................................................... ............ 3

Software ........................................................................................................................... ............. 3

Hardware ..................................................................................................................................... .. 3

LVM ......................................................................................................................... .................. 4

Native ................................................................................................................ ........................ 4

Database load scenario ................................................................................................... ................. 4

Create database ........................................................................................................................ .... 6

Load database .................................................................................................... .......................... 6

Update statistics .................................................................................................................. .......... 6

System Monitoring ............................................................................................................. ............... 6

Results ........................................................................................................................................ ...... 7

Overall runtimes ............................................................................................................. ............... 7

CPU utilization ........................................................................................................................... .... 10

I/O throughput ........................................................................................................... .................... 11

Related Content ....................................................................................................................... ......... 13

Copyright ................................................................................................................... ....................... 14


© 2006 SAP AG 2


Test scenario descriptionFirst of all the used soft and hardware configuration is described, which means the operating system, processors, storage, etc. Furthermore the way the benchmark is executed and in particular what steps are executed and measured are explained. After these introductory explanations the results will be presented. Probably the most interesting pages in the article.

Software

The operating system used during the tests is SuSE Linux Enterprise Server 10, SLES10. It is installed with the standard settings and no extra updates. The kernel version is 2.6.16.210.8smp, glibc version is 2.431.2. The SAP software is an ERP2005 SR1 system using SAP kernel 7.00 PL 75. The database is MaxDB 7.6.00 Build 034123134685. The database is configured to have one log volume with 2GB and six data volumes with 15GB each, providing 90GB of database space which will later be used to 75%.

Hardware

The machine used for testing has 4 CPU sockets, each holding an Intel DualCore Xeon MP with 3.0Ghz. The architecture is x86_64. The machine has two internal 36GB SCSI disks attached to a Adaptec AACRAID, configured as RAID0 providing a single 72GB volume for the operating system. Normally RAID0 should not be used as it is very insecure, but for the benchmarks the machine was taken the way it was already installed. A total of 16GB of memory is installed and 2GB of SWAP (/dev/sda2) is configured. Having only 2GB of SWAP configured is not sufficient according to SAP note 171356. Nevertheless the system layout was taken the way it was installed. The root filesystem is /dev/sda1.

The machine has one Fibre Channel controller, a QLogic Corp. QLA2312, running with driver version 8.01.04k. The following boot parameters were passed to the kernel via /etc/grub/menu.lst: “qlport_down_retry=1 ql2xfailover=0 ql2xretrycount=5 verbose=1 showopts”. These parameters were needed to make operating system see the configured devices on the storage.

The Fibre Channel controller is attached to a DS4700 storage with two shelves and two controllers having 1.7GB of cache memory each. Between the server and the storage a Fibre Channel switch with 2Gbit/sec is used. The storage is connected with 2 HBAs to the switch, the server only with one HBA. The theoretical maximum transfer rate between the server and the storage thus is 2GBit/sec. The storage has a total amount of 32 disks, each having 73GB and 15K rpm. Four of these disks are configured as spare disks which leaves 28 disks for free use. There are a total of four RAID5 volumes configured, each one having 6+1 disks. To be very flexible with the exported devices the size of a device is set to 36GB. Every device is striped over one of the four configured volumes in a alternating way. They are exported also by alternating the controller on the storage for each device, which means, devices with even numbers use the first controller and the other ones use the second controller. This setup is meant to be very efficient, since it provides a kind of hard wired load balancing over the controllers.

A total of five devices is exported to the test machine, /dev/sdb, /dev/sdc, /dev/sdd, /dev/sde and /dev/sdf. The last device, /dev/sdf, has one partition, formated with ext3, mounted to /TestSuite,


© 2006 SAP AG 3


which holds the SAP TestSuite. The other four devices left are the targets where the database will be installed. Two different disk layouts are used.

LVM

Every device only provides 36GB free space but a total of at least 100GB is needed for the database. Therefore the first four devices are put together in a big LVM volume group with a total size of approx. 140GB. This volume group comprised only one logical volume which is formated with ext3 and mounted to /usr/sap, where database volumes and the SAP binaries are installed to. There are no special mount options used for ext3.

Native

The disk layout for using the devices directly is setup the following way. Every device holds one single partition, formated with ext3. The partition /dev/sdb1 is mounted to /devspace1, /dev/sdc1 is mounted to /devspace2 and /dev/sdd1 is mounted to /devspace3. Each partition holds two MaxDB data volumes with each having 15GB. Partition /dev/sde1 is mounted to /usr/sap where the SAP binaries, the SAP developer trace files and the MaxDB log volume are located.

Database load scenarioThe database load scenario is a task which every customer has to perform at least once the installation of the SAP system. After the installation the system may run for years without any change to the system landscape infrastructure. But after several years of running the database on one server, the hardware could reach its limit by adding too many new users to the system. One way to avoid this upcoming bottleneck maybe changing the hardware of the database server either by migrating to a faster system or by upgrading the hardware directly. Upgrading the hardware may be difficult after several years, because the CPU sockets or the memory module layout changed in a way that a migration to a new server is the only solution left to speed up the database throughput.

A migration of an existing SAP system is done by exporting the current database content to a (database and operating system) independent format, a so called database export. After creating a new database on a new host, this database export is loaded into the new database. During this time the SAP system has to be in a offline state, because the database is unavailable during exporting and importing the content. Such export and import scenarios are normally performed during a weekend or a holiday when the SAP system is not needed. The important and essential part of the migration is, that is has to be done as fast as possible, as on Mondays the normal users want to connect to the SAP system to start their work. Every hour additional downtime is worth a lot of money!

The export which is used during the database load test was created by exporting an almost freshly installed SAP ERP 2005 SR1 system. The export was created by calling the r3load.pl script provided by Alexander Krauth from Realtech, which first calculates the sizes of all tables, creates the export task files and then launches 33 R3load processes simultaneously to extract the data out of the database. Several hours later you have the complete database content available in database and operating system independent format. Using this script makes exporting system much easier. Officially one would use MIGMON to create an official export.


© 2006 SAP AG 4


An export consists of different files for each SAP content (e.g. application content is in a different file than documentation content). Furthermore each content file is separated into four different types of files. These four types are TOC, STR, EXT and data files. The first three types are command files with no real application data content. The data files hold the compressed content of the database. This content is distributed over 66677 tables. The size distribution between the 33 datafiles is as follows:

Job Tables Size/MB Vardata/MB

--------------- ------ --------- ----------

SAP0000 1 4.5 0.0

SAPAPPL0 17143 2744.4 2.2

SAPAPPL1 11537 1193.6 52.3

SAPAPPL2 8855 500.0 74.8

SAPAPPL3 5522 500.0 7.4

SAPAPPL4 1053 500.0 0.0

SAPAPPL5 18885 500.0 0.0

SAPAPPL6 1994 500.0 18.4

SAPAPPL7 298 24.2 1.2

SAPCLUST 44 40.7 40.7

SAPDDIM 1 0.0 0.0

SAPDFACT 2 0.0 0.0

SAPDODS 5 0.0 0.0

SAPPOOL 122 102.1 102.1

SAPSDIC 360 921.9 60.8

SAPSDOCU 100 32.7 0.0

SAPSLEXC 10 18.8 0.2

SAPSLOAD 17 0.4 0.1

SAPSPROT 215 33.0 0.3

SAPSSEX0 100 700.0 215.6

SAPSSEX1 1 732.4 0.0

SAPSSEX2 59 700.0 4.4

SAPSSEX3 27 700.0 0.0

SAPSSEX4 1 1083.0 0.0

SAPSSEX5 29 700.0 1.3

SAPSSEX6 27 700.0 0.0

SAPSSEX7 1 2796.6 0.0

SAPSSEX8 9 700.0 0.0

SAPSSEX9 37 653.1 0.0

SAPSSEXC 39 700.0 0.0

SAPSSRC 169 581.2 2.7

SAPUSER 13 0.1 0.0

SAPUSER1 1 0.0 0.0

SAPVIEW 0 0.0 0.0


© 2006 SAP AG 5


The time from pressing the enter button until the script finishes is measured. This overall runtime will give a rough summary how the scheduler perform. Additionally the duration of three main steps is also measured creating the volumes (writes only), loading the database (writes and reads) and updating database statistics (reads only). Since every phase has different requirements for the I/O subsystem a separated measurement is needed. This (https://www.sdn.sap.com/irj/sdn/thread?threadID=289018) SDN Forum blog gives you additional information how MaxDB handles the issued write requests.

Create database

The first part is creating the the log volume having 2GB and afterwards, creating the six data volumes with each having 15GB of size concurrently. This leads into six MaxDB kernel threads doing write operations on the physical devices at one time. This first phase is plain I/O write performance.

Load database

The second part is the database load phase (the main import) with lots of writes for the data and also reads needed for index creation. The import of the data in our scenario is as follows. During the script based installation, a script (/usr/sap/BEN/export/import.sh) is executed. This file checks if all export files are available. If so, it generates the R3load command files which contain the instructions for the import. Afterwards it starts a R3load process for each different command file (e.g. documentation content, application content). The R3load process imports the data into the database. The number of concurrent R3load processes is determined by the number of available CPU's. The number of concurrent R3load processes can reach the double amount of available CPU's. But in our case we only use nine R3load processes run at a time (CPU's plus 1). After a job is imported, the script starts a new R3load process which imports the next one. During the last minutes of the load it may happen that only one R3load process is running, because only one job is left. During the import a few computing resources are used to unpack the content data and to modify this unpacked data to fit into the database. Besides the small overhead of computing the I/O throughput is still the main factor to determine the time of the load phase.

Update statistics

Updating the database statistics which are database reads only is the last of the three sub measurements. The MaxDB command used is "sql_updatestat ${DB_SCHEMA}.* ESTIMATE SAMPLE 1000 ROWS". This command updates the statistics for the SQL optimizer.

One might argue, that the some of the operation are done using the database cache instead of accessing the data volumes. This would invalidate the argument that loading a database with SAP content is a good I/O benchmark. But the way in which the data is imported is very I/O intensive because after a table with its content is imported a MaxDB savepoint is triggered. Such a savepoint writes the whole cache content physically onto the disk. Furthermore the log writer writes down every action into the log volume which is set to auto overwrite to prevent a log full situation. Furthermore the reads for creating the indexes and updating the statistics cannot be fulfilled by the cache. This gives us at least many physical writes and reads.

System MonitoringThe system monitoring tool sar was used to keep an eye on the machine. The standard interval used by sar is ten minutes. To get more detailed results the interval was reduced to two minutes. Additionally it is not


© 2006 SAP AG 6

https://www.sdn.sap.com/irj/sdn/thread?threadID=289018


enough to have the overall runtime being the only metric for the performance of the scheduler. How many CPU or I/O bandwidth used is also very interesting to know. Bottlenecks in the system can only be pointed out having reliable data about CPU and I/O usage.

ResultsWith four available schedulers on Linux – cfq, as, deadline and noop – the import of the database has to be done four times (see SAP Network Blog: I/OSchedulers on Linux for a scheduler overview). Additionally, having two different disk layouts – LVM logical volumes and native partitions – doubles the number of imports to eight. The testing took four days. The sar data of four days with an interval of two minutes is far to much to put it into this document. Instead, using graphs is the approach we use in this article. The overall runtimes are displayed first (including a small table with numbers), followed by the overall CPU usage during the testing. Last but not least, the I/O statistics will be provided, shown in issued transfers per seconds to the physical devices.

Overall runtimes


© 2006 SAP AG 7

Illustration 1: Runtimes using LVM

Statistics

Load DB

Create DB

Overall

00 1800 3600 5400 7200 9000 10800 12600 14400

Overall runtimes (using LVM)

CFQ

AS

DEADLINE

NOOP

Seconds (less=better)

https://weblogs.sdn.sap.com/pub/wlg/4790


Table 1 displays the four different schedulers used – cfq, as, deadline and noop – with their corresponding overall runtime. Additionally the three different phases – create DB, load DB, and create statistics – are shown. Seen vertically, the cfq scheduler is a little bit ahead of the deadline scheduler. For all three major phases these two have similar runtimes. Seen horizontally, the noop schedules takes almost twice as much time creating the database compared to the as scheduler, the fastest during heavy I/O. The Load phase again is dominated by the fastest overall schedulers, cfq and deadline, whereas the schedulers in the update database statistics phase almost perform equally fast. The runtimes using native partitions is as follows:


© 2006 SAP AG 8

Table 1: Runtimes using LVM

Runtime [sec] CFQ AS DEADLINE NOOPOverall 9735 10733 9864 12748Create DB 1006 939 978 1920Load DB 3425 4564 3567 5308Statistics 3701 3677 3664 3764

Illustration 2: Runtimes using native partitions

Statistics

Load DB

Create DB

Overall

00 1800 3600 5400 7200 9000 10800

Overall runtimes (native)

CFQ

ASDEADLINE

NOOP

seconds (less=better)


As we can see, using native partitions shortens the runtime of the as and noop schedulers, while the cfq and deadline only gain up to two minutes only. Creating the database gains between 15.44% using the as scheduler and 26.2% using the noop scheduler on native partitions. A completely different picture shows the database statistics update. It is 3.66% slower using the deadline scheduler and 1.59% faster then the noop scheduler. Although the noop scheduler wins the statistics creation it is still slower then the other schedulers. The following Illustration depicts the runtimes of all schedulers with both disk setups used. Two bars with the same color are meant to use the same scheduler. The upper bar is the LVM setup, the lower bar the native partition setup:


© 2006 SAP AG 9

Table 2: Runtimes using native partitions

Runtime [sec] CFQ AS DEADLINE NOOPOverall 9698 9996 9766 10338Create DB 792 794 790 1417Load DB 3549 3653 3551 3573Statistics 3727 3739 3798 3704

Illustration 3: Runtimes Complete Comparison

Statistics

Load DB

Create DB

Overall

00 1800 3600 5400 7200 9000 10800 12600 14400

Runtimes Complete ComparisonCFQ lvmCFQ nativeAS lvmAS nativeDEADLINE lvmDEADLINE nativeNOOP lvmNOOP native

seconds (less=better)


CPU utilization

The CPU utilization is as follows:


© 2006 SAP AG 10

Illustration 4: CPU utilization using LVM

0

10

20

30

40

50

60

70

80

90

100

CFQAS

Deadline

Noop

CPU Utilization usign lvm (percent)

CFQASDeadlineNoop

Illustration 5: CPU utilization using native partitions

0

10

20

30

40

50

60

70

80

90

100

CFQAS

Deadline

Noop

CPU Utilization using native partitions (percent)

CFQ

ASDeadline

Noop


The xAxis (which has no label in these graphs) is the time line. Due to clarity the grid lines and seconds are not shown in the graph. Having the first look onto these two illustrations one can clearly see that using native partitions the CPU utilization only in seldom cases exceeds the 70% mark, while using logical volumes the maximum CPU usage exceeds 90% while creating the database. The additional layer of complexity which LVM puts into the environment may be the cause of this. Nevertheless using native partitions results in a much smoother CPU usage compared to LVM. The as scheduler, or even the noop scheduler, for example has a jumping CPU usage on LVM where it is very constant on native partitions.

As mentioned earlier, the number of concurrent R3load processes should be twice the amount of available CPU's. In our testings only CPU's plus 1 R3load processes were used. A CPU utilization of only 60% to 70% during the load phase shows that there was room for more. At least the I/O system was not the bottleneck as the next chapter shows.

I/O throughput


© 2006 SAP AG 11

Illustration 6: I/O Transfers per second

0.00

5000.00

10000.00

15000.00

20000.00

25000.00

30000.00

35000.00

CFQ native

AS native

DEADLINE native

NOOP native

CFQ LVM

AS LVM

DEADLINE LVM

NOOP LVM

Transfers per second

CFQ nativeAS nativeDEADLINE native

NOOP nativeCFQ LVM

AS LVMDEADLINE LVMNOOP LVM

trans

fers


Again, the xAxis (which has no label) is the time line. Due to clarity the grid lines and seconds are not shown in the graph. This is the most interesting illustration. It shows the amount of transfers which where issued to the physical devices. In general, a transfer is a I/O request issued to a physical device. A transfer is of indeterminate size1. Transfers means write and read transfers. As both, read and write transfers are used during the benchmark, the cumulated value is used in this illustration.

Excluding the noop scheduler, using logical volumes issues up to 100 times more transfers to or from the storage compared to native partitions. Assuming that a storage can only handle an limited amount of transfers per second, using logical volumes might lead into a bottleneck if too many writes or reads on logical volumes are happening concurrently. The average writing speed in MB/sec while creating the database (writes only) is as follows:

The first three schedulers are far ahead of the noop scheduler while creating the log and data volumes. Even when using a storage the noop scheduler cannot compete with the other ones. Furthermore, when it comes to plain I/O performance, native partitions can write down around 25% more data in the same time. A comparison between the first three schedulers shows, that their performance is almost the same, either when using logical volumes or native partitions. The read and write performance in MB/sec is as follows:

The values shown here are the average written and read megabytes during the complete load phase. Of course in the first phase of the load the throughput is much higher as there are nice concurrent R3load processes running. In the later phase of the load when no more export files are do be imported the amount of concurrent R3load processes drops by each import that finishes. In the last minutes only 5 to 10 MB/sec were written. Comparing the writes on LVM, only the cfq and deadline scheduler show the same performance as the writes on the native partitions. The as and noop schedulers again mark the bottom line of performance here. This only applies using logical volumes as having a look at the writes on the native partitions all schedulers show almost the same performance. Coming to the read performance again the

1 According to 'man sar'


© 2006 SAP AG 12

Table 3: I/O throughput create database

MB/sec CFQ AS DEAD NOOPLVM 95.78 100.98 99.83 50.49Native 125.12 124.42 124.90 68.53

Table 4: I/O throughput load database

MB/sec CFQ AS DEAD NOOPLVM write 38.31 27.50 36.89 24.08Native write 36.44 35.65 36.11 36.14LVM read 3.99 2.50 3.93 3.45Native read 4.86 4.70 4.60 4.68


same picture. Using LVM, the as and noop scheduler are last, cfq and deadline on top. Comparing LVM to native, the native performance again is better.

Taking a look at the read only performance is not worth another table as it would show the same results. Nevertheless, one additional benchmark run using LVM, the cfq scheduler and a physical extend size of 128MB instead of 4MB was made to figure out if a different physical extend size has an effect. The outcome is: No performance increase was seen using a physical extend size of 128MB instead of 4MB in this test environment. But anyway, the LVM topic is still under investigation and may be topic of one of the upcoming articles featuring best practices on Linux.

A general suggestion which scheduler is the best for your environment, cannot be given for sure. The cfq and the deadline schedulers perform equally whereas the as and noop scheduler mark the bottom line of the field. Nevertheless using LVM may lead into a bottleneck if the amount of transfers issued to a physical device exceeds the maximum transfer capabilities, but using LVM gives you the possibility of altering the disk layout without rebooting the machine, which would lead to additional downtime. With this summary I will close this article and look forward to the next ones.

Related Content• SAP on Linux

• I/OSchedulers on Linux

• Forum: SAP on Linux


© 2006 SAP AG 13

https://www.sdn.sap.com/irj/sdn/forum?forumID=189

https://www.sdn.sap.com/irj/sdn/weblogs?blog=/pub/wlg/4790

https://www.sdn.sap.com/irj/sdn/linux


Copyright© Copyright 2007 SAP AG. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.

IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries, pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, Informix, i5/OS, POWER, POWER5, OpenPower and PowerPC are trademarks or registered trademarks of IBM Corporation.

Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries.

Oracle is a registered trademark of Oracle Corporation.

UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.

Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc.

HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.

Java is a registered trademark of Sun Microsystems, Inc.

JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape.

MaxDB is a trademark of MySQL AB, Sweden.

SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.

These materials are provided “as is” without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or noninfringement.

SAP shall not be liable for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials.

SAP does not warrant the accuracy or completeness of the information, text, graphics, links or other items contained within these materials. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third party web pages nor provide any warranty whatsoever relating to third party web pages.

Any software coding and/or code lines/strings (“Code”) included in this documentation are only examples and are not intended to be used in a productive system environment. The Code is only intended better explain and visualize the syntax and phrasing rules of certain coding. SAP does not warrant the correctness and completeness of the Code given herein, and SAP shall not be liable for errors or damages caused by the usage of the Code, except if such damages were caused by SAP intentionally or grossly negligent.


© 2006 SAP AG 14

how linux schedulers and other i/o related setups ... · how linux schedulers and other i/o related...

Documents