DRAFTDaily Operations and Administration of Your HANA environments

By Dr. Bjarne Berg, VP SAP Business Intelligence, Comerit and Professor at SAP University Alliance at Lenoir Rhyne University

There are many ways to install and operate your HANA environments on premise. However, a creating an daily, weekly and monthly Standard Operating Procedure (SOP) is a good way to make sure that the system stays well-tuned, and that potential issues are avoided. This is also known as the daily operations handbook. In this article we look at what the different landscape options are and how you can start creating your own SOP for your data center.

The first decision you have to make when setting up your data center for your HANA environments is to decide if you are going to place it on-premise, as part of an outsourcing agreement, or on the cloud. The on premise approach is currently most common and it basically means that you will have to integrate the hardware into you current data center and possibly an off-site data center if you are implementing a high-availability (HA) solution.

A major consideration for the on-premise approach would be to make sure that your hardware fits into your existing chassis, racks, power outlets, cooling plan and the outlay of your data center. For example, many are not aware that some of the larger HANA systems like for example Lenovo’s x3850 x6 requires a 4U height in a data center, but if you are using the Lenovo’s x3950 x6, you will need to double that size requirement (since that is basically two stacked 3850s). While other vendors such as Cisco’s C880 M4 Server requires 10U height. So, it is very important to decide what hardware deployment options you are going with. As of September 2015, the most common forms of certified hardware can be summarized as:

Figure 1: Common HANA Hardware Platforms for On-Premise Deployment

1

DRAFT

Table 1: Certified HANA Hardware options as of September 2015.

Scale-out

Scale-up

Business Suite

bullion S2 512-1536 GB 2 x 8890v3 x 2.5 Ghz x xbullion S4 1024-3072 GB 4 x 8890v3 x 2.5 Ghz x xbullion S8 2048-6144 GB 8 x 8890v3 x 2.5 Ghz x x

UCS B260 M4 512-1536 GB 2 x 4890v2 x x x xUCS C460 M4 128-3072 GB 4 x 8890v3 x 2.5 Ghz x x xUCS C880 M4 2048-6144GB 8 x 8890v2 x 2.8 Ghz x x x

Dell PowerEdge R930 128-3072 GB 4 x 8890v3 x 2.5 Ghz x x

PQ 2400 E/S/L 128-1024 GB 4 x 8890v3 x 2.5 Ghz x xPQ 2800B2/E2 128-6144 GB 8 x 8890v3 x 2.5 Ghz x xRX4770M2 128-3072 GB 4 x 8880v3 x 2.3 Ghz x x

CS-500 128-3072 GB 4 x 8880v3 x 2.3 Ghz x x xCS-900 1024-12288 GB 8 x 2890v2 x 2.8 GHz x x x

Hitachi CB520X B2 256-6144 GB 8 x 8880v3 x 2.3 Ghz x x x

FusionCube E9000 512-1024 GB 4 x 4890v2 x 2.8 GHz xRH5885H V3 128-3072 GB 4 x 8880v3 x 2.3 Ghz x x xRH8100 V3 128-6144 GB 8 x 8880v2 x 2.5 Ghz x x x

Flex x880 X6 128-6144 GB 8 x 8890v2 x 2.8 Ghz x x xx3850 X6 128-3072 GB 4 x 8880v3 x 2.3 Ghz x x xx3950 X6 512-6144 GB 8 x 8880v3 x 2.3 Ghz x x x

NEC Exp. 5800/A2040b 128-2048 GB 4 x 4890v2 x 2.8 GHz x x

Silicon Graphics UV 300H 256-6144 GB 8 x 8890v2 x 2.8 Ghz x xUnisys Forward! 4150-B 128-3072 GB 4 x 4880v2 x 2.5 GHz x xVCE UCS B460 M4 1024 GB 4 x 4890v2 x 2.8 GHz x

Target usage

HP

Huawei

Lenovo

Vendor System Memory(RAM)

Bull SAS

Cisco

Fujitsu

CPUIntel Ivy Bridge EX E7 15 Cores

(2014)

Max Number and Type

CPU Speed

Intel Haswell EX E7 18 Cores

(2015)

Customer HANA Admin and Support Responsibilities

As you start with your plan to write an SOP, it is important that you are aware of the normal support, install, monitoring roles and the responsibilities of SAP, your hardware vendor and your own team. The normal support responsibilities can be summarized as in Table 2 below.

2

DRAFTTable 2: Summary of Key Responsibilities

Area Task Hardware vendor Customer SAP

Hardware installation and health check xLinux OS installation xHANA platform installation xData source connectivity xAdding DB instances (MCOS) xSMD agent installation xHANA DB admin xThird party software installations xHANA system monitoring xHANA DB monitoring xBackup and recovery x"Bare metal" recovery xFirmware patching (x)* xLinux OS upgrades and patching (x)* xPeripheral components patching xHANA platform components updates & patching x

Support Issue resolution process (x)* x x* depending on support contract

Initial Setup

Operations

Maintenance

It is important to note that the responsibilities as outlined in Table 3, is based on an on-premise installation of SAP HANA and that no other support agreements are made with the hardware vendor, a cloud vendor, or an outsourcing partners. Depending on how you write your support agreement with these vendors, some or all of the customer responsibilities may be assumed by these partners. The trick to make sure on what you are responsible for is to specify these activities in a Service Level Agreement (SLA) if you are using other vendors to support your systems and landscapes.

There are also different cloud options that some companies might consider. For each or these options the responsibilities of the customer are significantly different. First, you can have your HANA system and applications delivered as a “Software as a Service” (SaaS). Under this offering you can get software applications such as SAP Business Suite, SAP Business Warehouse (BW), and SAP Rapid Deployment solutions (RDS) as SaaS HANA cloud solutions from several vendors who then take over all customer responsibilities for daily monitoring, support and maintenance.

Another option is the “Platform as a Service” (PaaS). This is normally provided as a solution where the database, operating system, connectivity and hardware is supported by a cloud vendor, but where daily operations and monitoring of the application is the customer’s responsibilities. Finally, the lowest level of cloud offerings is known as “Infrastructure as a Service: (IaaS). As the name implies, you are normally responsible for all tasks as illustrated in Table 3, except the hardware maintenance which is then hosted in the cloud.

However, in this article we are going to assume that the support is for an on-premise implementation and that the customer is assuming the normal support, maintenance and monitoring roles and that a cloud solution is not in place.

System vs. Landscape Administration

There are several different tools and procedures that should be developed that are different based on a system or landscape administration perspective. For example, for system administration you should leverage SAP’s HANA

3

DRAFTAdministration guide that can be downloaded on help.sap.com. This guide is maintained and updated by SAP on a release basis. It shows you how to use the HANA cockpit (a Fiori LaunchPad application) and HANA studio for the main system administration, the core functions of high-availability and disaster recovery, scalability (up and out), security administration, as well as how to manage and monitor applications for data provisioning and custom applications built in the extended services (XS) framework.

We can also monitor the system through the DBA Cockpit in Solution Manager and leverage the Trouble Shooting and Performance Tuning guide from SAP when issues arise. However, from a landscape administration perspective we leverage the Technical Operational manual from SAP and the DB control center, as well as any respective application support for the systems you might be running. So, when you start developing your SOP/Daily Operating handbook, you should start by familiarize yourself with these very important documents and tools and think about system admin as different from landscape administration.

Table 3: Key SAP resources for HANA System and Landscape Administration

Area Tool Purpose Web Resource

HANA Administration guide

- How to use the HANA cockpit and HANA studio for system admin.- Core functions of high-availability, disaster recovery & scalability- Security administration- How to manage and monitor applications for data provisioning and custom applications built in the extended services (XS) framework.

tinyurl.com/AdminHana

DBA Cockpit for HANATool to manage system landscape connections and central management of DB configurations

tinyurl.com/DBACockpit

HANA Troubleshooting and Performance Analysis Guide

How to trouble shoot and fix DB performance issues and guidance on general optimization.

tinyurl.com/TroubleGuide

Multitenant DB GuideHow to monitor, setup and manage systems that have HANA multitenant DBs

tinyurl.com/HanaDBs

Technical Operations Manual How to operate and administrate a HANA landscape. tinyurl.com/TechOperationsSAP DB Control Center (DCC) Guide on how to use DCC to monitor HANA and other databases tinyurl.com/databaseCC

Landscape Admin

System Admin

HANA System Monitoring tools and Education

You can also choose one or more ways to perform you system monitoring. For example, you can monitor system databases and also tenant data bases (in MCOD/MCOS) by directly connecting to a database using the HANA cockpit, the DBA Cockpit in Solution Manager, or through regular SAP HANA studio.

In HANA Studio in the administration perspective, you get access to most database and system information. There are several tabs that displays landscape, alerts (automatic scheduled monitoring jobs), performance statistics, disk volume information, configuration settings, overall system information, diagnostic files and configuration of traces and trace files.

4

DRAFTFigure 2: The SAP HANA Studio Administration Console Perspective

Also, since SPS9 of HANA in late 2014, the enhanced HANA Cockpit is a now very interesting way to get access to a simple web based monitoring application that shows you key statuses of your HANA systems and databases. As mentioned before, the HANA Cockpit is basically a Fiori LaunchPad site that you can also customize to show only the items you are interested in for daily operation monitoring.

5

DRAFTFigure 3: Administration with HANA Cockpit in Fiori

The customization of this application is a simple click-and drag of the tiles (much like on you cell phone). You can also choose the refresh rate of the information in HANA Cockpit and the application can run on a web browser and is therefore mobile and simple to deploy. The HANA Cockpit also have a ‘Manage Databases’ app that allows you to monitor single and multi-tenant databases in HANA.

As you click on each of these tiles, a vast array of detail information is provided for your in-depth analysis and system monitoring. However, it is important to note that while the HANA Cockpit supports core administration of tenant databases (i.e. MCOS), HANA Studio and some command-line tools may still be required for key tasks for tenant databases. Frankly, the only minor drawback with the HANA Cockpit is that it may require additional licenses depending on what you bought with the initial license package.

At a higher level the SAP Database Control Center (DCC) is also a Fiori application that allows you to monitor both HANA and other type databases from a central application. As you get more familiar with these tools, you probably will find it useful to start with one or two of these as choose the others as alternatives when you get stuck on a certain task. Most system administrators include HANA studio and either the DBA or the HANA cockpit for daily monitoring.

To get started to learn about these tools, first download and study the guides outlined in Table 1 and then take the 5-day SAP Course called HA-200 “SAP HANA - Installation & Operations”. This course is strongly recommended for experienced support staff as well as for beginners and should be taken before you start writing your own SOP.

Solution Manager and LVM Tools

Many of the tools used for system monitoring is also used for database monitoring. First, you can conduct many of the individual database admin functions through HANA studio and the HANA cockpit from a Web browser. From here you can make changes to the database system settings and also add users and privileges and most standard database admin tasks.

6

DRAFTAlso, like all SAP software, you can use Solution Manager (SolMan) for core monitoring and admin of multiple systems in your landscape and as the backbone for CTS+ integration of transports between the systems in your landscape. Solution Manager can also be used to generate EarlyWatch reports on a periodical basis that shows growth, usage trends and technical support information. You will also find the DBA Cockpit in Solution Manager. This tool allows you to monitor the HANA database and exposes almost all of the technical information you would otherwise find in the administrator console perspective in HANA Studio.

Figure 4: HANA Admin and monitoring with the DBA Cockpit in Solution Manager

Solution Manager and the DBA Cockpit also support trace analysis, workload analysis and exception analysis of HANA databases. Most customers therefore find this tool invaluable when monitoring and managing SAP landscapes with both HANA and other types of databases.

In addition to these tools the Landscape and Virtualization Manager (LVM) from SAP is also supported for HANA. This allows you to conduct core operation of complex landscapes that may have HANA and non-HANA based servers. There is a standard edition of LVM that can be downloaded from SAP for free, and there is an Enterprise Edition that has more features that require a license before you can use it.

7

DRAFTFigure 5: SAP Landscape and Virtualization Manager (LVM)

When any of these administration and management tools is deployed, it is important that your support staff, that is monitoring, maintaining and operating a HANA landscape, has a good understanding of the capabilities of each of these. This can be obtained through regular SAP training classes. If you are new to these tools, you might start with one or two, and integrate them into your support landscape and add more as your experience level increase.

Daily Operations HANA Checklist

Once you have decided on you monitoring tool, downloaded and studied the available support documents in Table 1 and completed the 5-day HA-200 HANA - Installation & Operations class from SAP, you are ready to start writing your Standard Operating Procedure. The SOP should consist of daily operations, weekly jobs and periodic upgrades and patches as supplied by SAP. In this section we take a look at the most common daily operations tasks that you will be doing.

While many prefer to have active or ‘passive’ monitoring of systems, best-practices are to have a combination of these. Passive monitoring usually means activating and scheduling some of the alerts available in HANA Studio. You can place thresholds on the alerts (i.e. when memory consumed exceeds a certain number of GB), and you can schedule how often the checks are performed on the database. These alerts when triggered show up in the HANA cockpit, DBA Cockpit and in HANA studio in both detail and overview pages. Today there are 74 standard alerts that come with the HANA system.

8

DRAFTFigure 6: SAP HANA Alerts in HANA studio

You can also setup email alerts if you have the system privilege CATALOG READ as well as SELECT privilege on the _SYS_STATISTICS schema, and also the system privilege INIFILE ADMIN. The first of these privileges is included in the standard SAP HANA role called MONITORING. This role can be assigned to non-system admin users and allow other technical resources access to see what is happening inside the HANA system without the ability to change anything.

There is also a list of historically executed alerts in HANA studio, but be aware that this list is restricted to the last 1,000 occurrences from the last 30 days. Also, when an alert is triggered, a priority is assigned by the system. In general, there are 4 different priorities with different timing when action is recommended:

Table 4: Alert Priorities in SAP HANA

Alert Priority Action Reccomended

High An Immediate action is required to reduce risk of corrupt or lost data and outages

Medium An action is required in the next hours/days to reduce downtime risk

LowAn action required in the next days/weeks to reduce downtime risks

Information An action can be taken to improve stability and performance of the system

There are also 10 different categories of alerts relating to Availability, Backup, Configuration, CPU, Diagnosis Files, Disk, Memory, Security, Sessions and System. So, when deciding when to schedule these alerts and when to monitor them is a critical task of the HANA administrator. Activating and monitoring the recommended daily, and intra-day, alerts through any of the tools outlined previously will allow you to early detect any performance issues.

To get started, take a look at table 5 as the first step of your own tailored daily HANA admin SOP. In this table you will find all of the available HANA automated alerts, alert IDs (so that you can find them in HANA Studio), the suggested frequency when these alerts should be activated/monitored, descriptions and SAP’s official recommendation on how to resolve any issues.

9

DRAFTTable 5: Admin Monitoring Frequency, Alert IDs and SAP recommended actions

Check Type ID Time Description SAP Recommended Admin Action

Availa-bility

0 Intra-day Identifies internal statistics server problem. Resolve the problem. For more information, see the trace files. You may need to activate tracing first.

3 Intra-day Identifies inactive services. Investigate why the service is inactive, for example, by checking the service's trace files.

4 Intra-dayRestarted Services- services that have restarted since the last time of the check.

Investigate why the service had to restart or be restarted, for example, by checking service's trace files.

21 Daily Identifies internal DB events.Resolve the event and then mark it as resolved by executing the SQL statement ALTER SYSTEM SET EVENT HANDLED '<host>:<port>' <id>.

22 Intra-day Notification of all alerts- if any alerts since the last check is triggered

These alerts can trigger email blasts to specified recipients. Investigate the alerts.23 Intra-day

Notification of medium and high priority alerts- since the last check is triggered

24 Intra-day Notification of high priority alerts- since the last check is triggered

31 DailyLicense expiry-If the disks to which data and log files are written are full. A disk-full event causes DB to stop

Obtain a valid license and install it. For the expiration date, see the monitoring view M_LICENSE.

41 DailyIn-memory DataStore activation- If a problem with the activation of an in-memory DataStore object exists

For more information, see the table _SYS_STATISTICS.GLOBAL_DEC_EXTRACTOR_STATUS and SAP Note 1665553.

70 PeriodicConsistency of internal system components after system upgrade Contact SAP support.

78 DailyConnection between systems in system replication setup- closed connections between primary/ secondary system.

If connections are closed, the primary system is no longer being replicated. Investigate why connections are closed (i.e., network problem) and resolve the issue.

80As

needed

Availability of asynchronous table replication- Monitors error messages related to asynch table replication.

Determine which tables encountered the table replication error using system view M_ASYNCHRONOUS_TABLE_REPLICAS, and check the corresponding indexserver alert traces.

Back-up

28 Periodic

Most recent savepoint operation- How long ago the last savepoint was defined, that is, how long ago a complete, consistent image of the DB was persisted to disk.

Investigate why there was a delay defining the last savepoint and consider triggering the operation manually by executing the SQL statement ALTER SYSTEM SAVEPOINT.

32 Periodic

Log mode LEGACY- If the DB is running in log mode "legacy". Log mode "legacy" does not support point-in-recovery and is not recommended for productive systems.

If you need point-in-time recovery, reconfigure the log mode of your system to "normal". In the "persistence" section of the global.ini configuration file, set the parameter "log_mode" to "normal" for the System layer. When you change the log mode, you must restart the DB system to activate the changes. It is also recommended that you perform a full data backup.

33 Periodic

Log mode OVERWRITE- If the DB is running in log mode "overwrite". Log mode "overwrite" does not support point-in-recovery (only recovery to data backup) and is not recommended for prod systems.

35 Daily Existence of data backup Perform a data backup as soon as possible.

36 Daily Status of most recent data backup Investigate why failed, resolve the problem, and perform a new data backup as soon as possible.

37 Daily Age of most recent successful data backup Perform a data backup as soon as possible.

10

DRAFT

38 DailyStatus of most recent log backups- If the most recent log backups for services and volumes were successful.

Investigate why the log backup failed and resolve the problem.

54 PeriodicSavepoint duration- Identifies long-running savepoint operations. Check disk I/O performance.

65As

needed

Runtime of the log backups currently running- If the most recent log backup terminates in the given time.

Investigate why the log backup runs for too long, and resolve the issue.

66As

needed

Storage snapshot is prepared- if the period, during the DB is prepared for a storage snapshot, exceeds threshold.

Investigate why the storage snapshot was not confirmed or abandoned, and resolve the issue.

69 PeriodicEnablement of automatic log backup- if automatic log backup is enabled.

Enable automatic log backup. For more details please see SAP HANA Administration Guide.

72 Daily

Number of log segments- segments in the log volume of each service Check for number of log segments. Make sure that log backups are being auto created and that there is enough space

Check whether the system has been frequently and unusually restarting services. If it has, then resolve the root cause of this issue and create log backups as soon as possible.

Configuration

3As

neededDiscrepancy between host server times- discrepancies in a scale-out system. Check operating system time settings.

10 Periodic

Delta merge (mergdog) configuration- If the 'active' parameter in the 'mergedog' section of system configuration file(s) is 'yes'.

mergedog is the system process that periodically checks column tables to determine if a delta merge operation needs to be executed. Change in SYSTEM layer the parameter active in section(s) mergedog to yes

16 Periodic

Lock wait timeout configuration- if 'lock_waittimeout' parameter in 'transaction' section of indexserver.ini file is between 100,000 and 7,200,000.

In the 'transaction' section of the indexserver.ini file, set the 'lock_wait_timeout' parameter to a value between 100,000 and 7,200,000 for the System layer.

26 PeriodicUnassigned volumes- Identifies volumes that are not assigned a service.

Investigate why the volume is not assigned a service. I.e.., assigned service is not active, the removal of a host failed, or the service removal was performed incorrectly.

34 Daily If all volumes are available. Investigate why the volume is not available.

79 Periodic

Configuration consistency of systems in system replication setup- Identifies configuration parameters that do not have the same value on the primary system and a secondary system.

The identified configuration parameter(s) should have the same value in both systems, adjust the configuration. If different values are acceptable, add the parameter(s) as an exception in global.ini/[inifile_checker].

CPU 5 Intra-dayHost CPU Usage- Determines the % CPU idle time on the host and therefore if CPU resources are running low.

Investigate CPU usage

Diag-nosis

Files

46As

needed

RTEdump files- Identifies new runtime dump files (*rtedump*) have been generated in the trace directory.

These files These contain information about, for example, build, loaded modules, running threads, CPU, etc..Check contents of the dump files.

50 Periodic Number of diagnosis files- written by the system (excluding zip-files).

A large number of files can indicate a problem with the DB (i.e., problem with trace file rotation or a high number of crashes). Investigate the diagnosis files.

51 Daily Size of diagnosis files- very large file sizes can indicate a problem with DB.

Check the diagnosis files in the SAP HANA studio for details.

52 Daily Crashdump files- new files that have been generated in the trace directory

Check the contents of the dump files.53 Daily

Pagedump files- new files that have been generated in the trace directory

11

DRAFT56 Periodic Python trace activity- If trace is active and

for how long. Trace affects performance.If no longer required, deactivate the python trace in the relevant configuration file.

Disk

2 Intra-day

Disk Usage- Determines what % of each disk containing data, log, and trace files is used. This includes space used by non-SAP HANA files.

Investigate disk usage of processes. Increase disk space, for example by shrinking volumes, deleting diagnosis files, or adding additional storage.

30 Intra-day

Check internal disk full event- If the disks to which data and log files are written are full. A disk-full event causes your DB to stop and must be resolved.

Resolve the disk-full event: In the Admin Editor on the Overview tab, choose the \"Disk Full Events\" link and mark the event as handled. Alternatively, execute the SQL statements ALTER SYSTEM SET EVENT ACKNOWLEDGED '<host>:<port>' <id> and ALTER SYSTEM SET

EVENT HANDLED '<host>:<port>'<id>.

60 PeriodicSync/Async read ratio- Identifies a bad trigger asynchronous read ratio.

This means that asynchronous reads are blocking and behave almost like synchronous reads. This might have negative impact on SAP HANA I/O performance in certain scenarios. Note 1930979.61 Periodic

Sync/Async write ratio- Identifies a bad trigger asynchronous write ratio.

77 Intra-dayDB disk usage- The total used disk space of the DB. All data, logs, traces and backups are considered.

Investigate the disk usage of the DB. See system view M_DISK_USAGE for more details.

Mem-ory

1 Intra-dayHost physical memory usage- The % of total physical memory available on the host

All processes consuming memory are considered, including non-SAP HANA processes. Investigate memory usage of processes.

3 Periodic Row store fragmentation Implement SAP Note 1813245.

12 Intra-dayMemory usage of name server- Determines what % of allocated shared memory is being used by the name server on a host.

Increase the shared memory size of the name server. In the 'topology' section of the nameserver.ini file, increase the value of the 'size' parameter.

17 PeriodicRecord count of non-partitioned column-store tables- Current table size is not critical.

Partitioning need only be considered if tables are expected to grow rapidly. A non-partitioned table cannot contain more than 2,000,000,000 (2 billion) rows). Consider partitioning the table only if you expect it to grow rapidly.

20 Periodic Table growth rate of non-partitioned column-store table

27 PeriodicRecord count of column-store table partitions

29 Periodic Size of delta storage of column-store tablesInvestigate the delta merge history in the monitoring view M_DELTA_MERGE_STATISTICS. Consider merging the table delta manually.

40 Daily

Total memory usage of column-store tables- The % of the effective alloc limit being consumed by individual column-store tables as a whole

This is the cumulative size of all of a table's columns and internal structures. Consider partitioning or repartitioning the table.

43 Daily Memory usage of services- % of effective alloc limit a service is using. Check for services that consume a lot of memory.

44 Periodic Licensed memory usage- % used.Increase licensed amount of main memory. See the peak memory allocation since installation in the system view M_LICENSE, column PRODUCT_USAGE

45 PeriodicMemory usage of main storage of column-store tables- % of effective alloc limit consumed by column-store tables.

Consider partitioning or repartitioning the table.

55 PeriodicColumnstore unloads- # of columns that have been unloaded from memory.

Can indicate performance issues. Check sizing with respect to data distribution.

58As

neededPlan cache size- if the plan cache is too small.

Increase the size of the plan cache. In the 'sql' section of the indexserver.ini file, increase the value of the 'plan_cache_size' parameter.

12

DRAFT67 Periodic Table growth of rowstore tables Reduce the size by removing unused data

68 Periodic Total memory usage of row store used by a service

Investigate memory usage by row store tables and consider cleanup of unused data

73 Periodic Overflow ratio of rowstore version space.Identify the connection or transaction that is blocking version garbage collection. You can do this in the SAP HANA studio by executing the "MVCC Blocker Connection" and "MVCC Blocker Transaction" statements available on the System Information tab of the Administration editor. If possible, kill the blocking connection or transaction.

74 Periodic Overflow ratio of metadata version space.

75 Periodic Rowstore version space skew- if rowstore version chain is too long.

81 Periodic Cached view size- how much memory is occupied by cached view

Increase size of the cached view. In the "view_cache" section of the indexserver.ini file, increase the value of the "total_size" parameter.

Secur-ity

57 Daily Secure store file system (SSFS) consistency regarding the DB

Check and make sure that the secure storage file system (SSFS) is accessible and consistent regarding the DB.

62 Daily

User passwords- Identifies DB users whose password is due to expire with the PW policy. If it expires, the user will be locked. This may impact application availability.

Change password of the DB user. It is recommended that you disable the password lifetime check of technical users so that their password never expires (ALTER USER <username> DISABLE PASSWORD LIFETIME).

63 DailyGranting of SAP_INTERNAL_HANA_SUPPORT role- if the internal support role is currently granted to any DB users.

Check if the corresponding users still need the role. If not, revoke the role from them.

64 Periodic

Total memory usage of table-based audit log- % of the effective allocation limit is being consumed by the DB table used for table-based audit logging.

Consider exporting the content of the table and then truncating the table.

Sess-ions

25 DailyOpen connections- % of the max number of permitted SQL connections open.

The max number of permitted connections is configured in the "session" section of the indexserver.ini file.Investigate why max number is being approached.

Sess-ion &

Transa-ctions

39 Daily Long-running SQL statements Investigate the statement. For more info, see table _SYS_STATISTICS.HOST_LONG_RUNNING_STATEMENTS.

42As

needed Long-idling cursors Close cursor, uncommitted transaction, or the serializable transaction in the application, kill connection, or by executing the SQL statement ALTER SYSTEM DISCONNECT SESSION <LOGICAL_CONNECTION_ID>. For more information, see the tables HOST_LONG_IDLE_CURSOR, HOST_LONG_SERIALIZABLE_TRANSACTION andHOST_UNCOMMITTED_WRITE_TRANSACTION (_SYS_STATISTICS).

47 Periodic Long-running serializable transactions

48 Periodic Long-running uncommitted write transactions

49 Periodic Long-running blocking situations Investigate the blocking and blocked transactions and if appropriate cancel one of them.59 Daily Percentage of blocked transactions

System 83 DailyTable consistency- the number of table consistency errors and affected tables Contact SAP support

Periodic and Active monitoring

In addition to these passive alerts, you can also add active monitoring by the system administrator to starts to predict when performance issues might arise. This include tracking weekly and/or monthly increases in data files, CPU utilization, planned

13

DRAFTprojects, memory consumption and user activity. This will allow you to plan and budget resources for system upgrades, data archiving, NLS implementations and hardware changes.

In addition, it is imperative that you also monitor new security patches and software fixes available from SAP and determine if this is something that you should consider for rapid implementation, or can bundle them into periodic upgrades, service packs and patches on a monthly or quarterly basis. The list of tasks in Table 5 should get you started with the automatic checks, but you can also build your own additional monitoring process by accessing the system information in SAP HANA. This is available in the administrator perspective in HANA studio under the ‘System Information’ tab page and is also mostly exposed in the HANA cockpit and in the DBA Cockpit.

Figure 5: The System Information tab in HANA Studio

14

DRAFT

Periodic SOP Tasks for Keeping a HANA System Smaller

In addition to the daily and periodic standard monitoring tasks, you might want to do an active monitoring of a HANA system during a go-live of a new project, as well as during a ‘hyper care’ period shortly after new functionality has been added to the system. In this section we look at some useful tips that helps you do active monitoring using available transactions in SAP. Since most HANA systems are currently using SAP BW as an application (this will most likely change over time), we are focusing on keeping BW 7.3 and 7.4 on HANA systems as small as possible. However, some of the tasks in this section also apply to a BusinesSuite on HANA system as well,

First, if you will want to quickly get access to see the largest HANA tables (and monitor their growth). This can be done by using the transaction code DB02. Here you will find the largest tables in memory and also their respective record counts.

Figure 6: View Large HANA Tables using DB02

In a SAP BW system there are also a set of tables that are likely to grow faster than others. These include the application log tables BALHDR, BALHDRP, BALM, BALMP, BALDAT, BALC, and BAL_INDX. The tables for linking IDocs (IDOCREL, SRRELROLES) and the Short dump table SNAP.

Other request administration data in BW that you also want to monitor include data in the RSBMLOGPAR, RSBMLOGPAR_DTP, RSBMNODES, RSBMONMESS, RSBMONMESS_DTP, RSBMREQ_DTP, RSCRTDONE, RSDELDONE, RSHIEDONE, RSLDTDONE, RSMONFACT, RSMONICTAB, RSMONMESS, RSMONRQTAB, RSREQDONE, RSRULEDONE, RSSELDONE, RSTCPDONE, RSUICDONE tables, as well as the Dictionary logs found in DDPRS. In this section we will take a look on how to best manage these tables to keep the HANA system as small as possible.

In addition to these tables you should also monitor the size of the BW workbook table RSRWBSTORE and the BW statistics data found in RSDDSTATAGGR, RSDDSTATAGGRDEF, RSDDSTATCOND, RSDDSTATDELE, RSDDSTATDM, RSDDSTATEVDATA, RSDDSTATHEADER, RSDDSTATINFO, RSDDSTATLOGGING, and RSDDSTATDTP. All of these tables tend to grow rapidly in

15

DRAFTactive systems, but keeping them small is rather simple. For example, to clean up the stats entries in these tables, you can run the program RSDDSTAT_DATA_DELETE in SE38 and then schedule the job to run periodically in SM36. In general, it may be useful to keep 12 months of stats data, so you don’t want to remove it all.

Figure 7: Deleting Statistics Entries over 365 days old using the RSDDSTAT_DATA_DELETE program

You should also monitoring the size of the DTP error log in RSBERRORLOG, the Process Chain logs (RSPCINSTANCE), and the BW batch runtime data (RSBATCHDATA). These logs contains numerous warnings and errors in transformation recordings. You can delete these on a periodic basis using RSBM_ERRORLOG_DELETE and schedule it to run to remove entries over 60/90 days old.

Furthermore, you can also keep your system small by periodically remove data in the RSPCLOGCHAIN, RSPCPROCESSLOG, and the RSPCINSTANCET table. This is done by using the RSPC_LOG_DELETE report in SE38.

Figure 8: RSPC_LOG_DELETE of Process Chain Data in BW.

16

DRAFT

In addition to the tips and tricks mentioned above, to keep the application log tables small, you should use SM36 to schedule a job to run periodically. In the screen, click on the ‘Step’ button to define which program will be run by the job (pick SBAL_DELETE). After that, you can also choose a variant to decide how much data you want to keep. For example, in Figure 9 we delete case logs older than 60 days.

Figure 9: Cleaning BW Application Log tables in HANA

17

DRAFT

After this setup, you can then schedule the job to run on periodic basis. This job will proactively help keep the memory utilization of the HANA system smaller than it would be if all these application logs are not periodically cleaned.

Figure 10: Scheduling Archiving Job to run Periodically

You should also periodically archive IDocs to keep the HANA system smaller. IDOCs are used for communication between BW and source system. When BW executes an InfoPackage for data extraction, a request IDocs, RSREQUEST, is sent to the source system’s Application Link Enabler (ALE) inbox. The source system acknowledges the receipt of this IDoc by sending an info IDoc (RSINFO) back to the BW system.

In addition the source system sends an IDoc with all the requested data using the message type (RSSEND). Therefore, these tables can grow fast over time. You should therefore consider archiving IDocs older than 3-6 month. You can do this using the transaction “SARA” and then click on “write” as illustrated in Figure 11.

Figure 11: Keeping HANA Small by Archiving IDocs using the SARA Transaction

18

DRAFT

Figure 12: Selecting tables for IDocs archiving Using SARA

Figure 13: Maintaining “Created On” Filter Dynamically to Archive all Entries Older than 30, 60 or 90 Days

Then you simply give the job a description and schedule the job to run periodically. However, it is important to note that your Basis team will need to periodically backup and delete the archiving file in directory folder. This is typically done on an annual basis. The SARA transaction is also available in the BusinesSuite on HANA system as well.

In addition to the jobs and programs outlined above, you should also consider cleaning up obsolete IDocs links from on a periodic basis. Links are written in the ALE and IDoc environment, resulting in entries in the IDOCREL and SRRELROLES tables. The links are required for IDoc document trace and ALE audit monitoring. You can delete links in IDC8 and IDCA regularly (they are not required after the IDocs are posted). To do delete the links, you can go to SE38 and the report RSRLDREL. Under the "Selection mode", pick “Select using relationship type" and create a variant for the link types IDC8 and

19

DRAFTIDCA. Then, under the "Deletion Criterion", select “Without existence check" and finally, as we did in Figure 11 (the regular IDoc removal), you can now create a dynamic end date variable, and use SM36 to schedule the job to run on a periodic basis. This will help you maintain a small HANA system and keep the BW system much cleaner.

Furthermore, you can make the Request Administration data in HANA smaller. While you should never delete entries in theses tables, you can archive them and re-load old entries if necessary. Before you start this, you should make sure that the reports RSSTATMAN_CHECK_CONVERT_DTA and RSSTATMAN_CHECK_CONVERT_PSA have been executed in SE38 at least once for all objects. You start by using SARA and select the archiving object called BWREQARCH and then schedule this to run to archive entries that are more than 90 days old (see figure 14).

Figure 14: Periodic Maintenance by Archiving Request Administration Data in BW

In addition to these archiving jobs, you can also remove data in the Dictionary Logs. Basically, the DDPRS dictionary log table contains all activities that do any change on the data dictionary objects in SAP BW. This can grow quite large over time, and you might want to remove some of these log entries on an annual basis. To clean up these log entries go to SE38 and run the RADPROTA report. Just make sure you execute the job as a background job (see Figure 15).

Figure 15: Removing Dictionary Logs in SAP BW to reduce HANA Size

20

DRAFT

You also want to make sure that non reportable tables, that have no daily loads going into them, have their memory cleansed. You do that with transaction RSHDBMON using the load/unload the data options, or flag it for ‘Early Unload’. The last option allows HANA to decide when the data should be unloaded (i.e. when the data is not accessed, or data is not loaded). This will allow you to keep your HANA memory usage smaller, while still have access to the data when needed.

Figure 16: Unload Non-Active Data from HANA Memory with RSHDBMON

Also, occasionally you need to clean up short dump table “SNAP” in BW. To do this simply go to transaction ST22 and select “reorg” (see Figure 17).

Figure 17: Periodic Cleaning up of the Short-Dump table in SAP BW

21

DRAFTAnd delete short dumps older than 90 days

Finally, from a HANA monitoring standpoint you may want to ensure that the delta merge is performed regularly. You can monitor this in the HANA Admin perspective of Eclipse in HANA Studio.

Figure 17: Monitoring the Delta Merge Processing in HANA using HANA Studio

Summary

Administrating HANA on a daily basis is a very interesting challenge. There are many technical aspects from hardware health, system connectivity, database performance, security, file management, backup, table management, memory consumption, disk utilization and much more. It is therefore very important that you plan a very structured support approach to SAP HANA and that tasks are scheduled as automated as possible. Using checklists can assures that you are not surprised by activities that could easily be addressed if only monitored in a very organized fashion.

Writing a daily tasks list, or SOP, and also create periodic tasks lists for other activities can assure that you have a very high degree of business continuity. HANA is a highly scalable platform with a substantial amount of built-in fault tolerance at the hardware and software level. Frankly, system failures are rather unusual. However, like all databases, no amount of ‘cleverness’ can totally prevent issues if the utilization is not monitored on a regular basis, and preventive measures are taken well in advance.

22

DRAFTIn this article we explored some of the new, and existing admin tools for SAP HANA and provided an informed opinion on how to best leverage the available system and admin alerts available. We also looked at some of the most common system admin tasks you should periodically perform on a SAP BW system and how to automate them as much as possible,

HANA is now almost 5 years old and best practices are starting to get solidified. So, as you start writing your own tailored SOP, I recommend you leverage access to the SAP community, SAP employees and skilled HANA consultant resources with real hands-on experience to guide you through the process. However, the items in this article should at least get you started.

Dr. Berg

23