">

RAC and Oracle Clusterware Best Practices and Starter Kit (Solaris) [ID 811280.1]

Modified 12-OCT-2011 Type BULLETIN Status PUBLISHED

In this Document Purpose Scope and Application RAC and Oracle Clusterware Best Practices and Starter Kit (Solaris) RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent) RAC Platform Specific Best Practices and Starter Kits

RAC on Solaris Step by Step Installation Instructions

RAC on Solaris Best Practices Hardware/Vendor Specific Considerations Oracle Software Considerations References

Applies to:

Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.2.0.1 - Release: 10.2 to 11.2 Oracle Solaris on SPARC (64-bit)

Purpose

The goal of the Oracle Real Application Clusters (RAC) series of Best Practice and Starter Kit notes provides customers with quick knowledge transfer of generic and platform specific best practices for implementing, upgrading and maintaining an Oracle RAC system. This document is compiled and maintained based on Oracle's experience with its global RAC customer base.

This Starter Kit is not meant to replace or supplant the Oracle Documentation set, but rather, it is meant as a supplement to the same. It is imperative that the Oracle Documentation be read, understood, and referenced to provide answers to any questions that may not be clearly addressed by this Starter Kit.

All recommendations should be carefully reviewed by your own operations group and should only be implemented if the potential gain as measured against the associated risk warrants implementation. Risk assessments can only be made with a detailed knowledge of the system, application, and business environment.

As every customer environment is unique, the success of any Oracle Database implementation, including implementations of Oracle RAC, is predicated on a successful test environment. It is thus imperative that any recommendations from this Starter Kit are thoroughly tested and validated using a testing environment that is a replica of the target production environment before being implemented in the production environment to ensure that there is no negative impact associated with the recommendations that are made.

Scope and Application

This article applies to all new and existing RAC implementations as well as RAC upgrades.

RAC and Oracle Clusterware Best Practices and Starter Kit (Solaris)

RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent)The following document focuses on RAC and Oracle Clusterware Best Practices that are applicable to all platforms including a white paper on available RAC System Load Testing Tools and RAC System Test Plan outlines for 10gR2 & 11gR1 and 11gR2:

Document 810394.1 RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent)

RAC Platform Specific Best Practices and Starter Kits

The following notes contain detailed platform specific best practices including Step-By-Step installation cookbooks (downloadable in PDF format):

Document 811306.1 RAC and Oracle Clusterware Best Practices and Starter Kit (Linux) Document 811280.1 RAC and Oracle Clusterware Best Practices and Starter Kit (Solaris) Document 811271.1 RAC and Oracle Clusterware Best Practices and Starter Kit (Windows) Document 811293.1 RAC and Oracle Clusterware Best Practices and Starter Kit (AIX) Document 811303.1 RAC and Oracle Clusterware Best Practices and Starter Kit (HP-UX)

Page 1 of 4

12/19/2011https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=BULLET...

RAC on Solaris Step by Step Installation Instructions

Click here for a Step By Step guide for installing Oracle RAC 10gR2 on Solaris. Click here for a Step By Step guide for installing Oracle RAC 11gR1 on Solaris. Click here for a Step By Step guide for installing Oracle RAC 11gR2 on Solaris.

RAC on Solaris Best Practices

The Best Practices in this section are specific to the Solaris Platform. That said, it is essential that the Platform Independent Best Practices found in Document 810394.1 be reviewed in addition to the content provided in this Document.

OS Configuration Considerations

Validate your hardware/software configuration against the RAC Technologies Matrix for Unix.

11gR2 requires Solaris 10 update 6 or greater. Reference Document 971464.1.

Ensure all required OS packages are installed and system prerequisites have been properly implemented for your particular release of Oracle. This information is documented in Document 169706.1 as well as the install guides for your particular release.

In pre-11gR2 clusters, system times are to be synchronized across cluster nodes using NTPD and NTPD should be configured to slew time to prevent false reboots. Configure NTP client as per Document 759143.1 to take corrective action on this issue.

With 11gR2, Cluster Time Synchronization Daemon (CTSSD) can be used in place of NTPD. CTSSD will synchronize time with a reference node in the cluster when an NTPD is not found to be configured. Should you require synchronization from an external time source you must use NTPD which will cause CTSSD to run in "observer" mode. However, if NTP is running, then it must to be configured with the slewing option as documented in Document 551704.1.

Ensure SunOS 5.10 kernel patch 127111-03 or higher is installed in order to address a condition with a deadlock in the Solaris 10 paging algorithm. Reference Document 460424.1.

There is an interoperability issue between Solaris 10's memory management and Oracle's Automatic PGA Memory Management in a Solaris 10 environment with a large number of CPUs. The issue causes CPU spikes happening at random intervals and increases system cpu time. Corrective action can be found in Document 460424.1.

Solaris Local Containers are supported, refer to Supported Virtualization Matrix for details.

Storage Considerations

Skip the first 1Mb when creating raw device to avoid overwriting the disk VTOC. See Document 367715.1 for details.

When using NFS, ensure that the correct mount options are used. The proper mount options for NFS are defined in Document 359515.1.

Use Solaris MPxIO or other 3rd party IO multi-pathing to protect against HBA or SAN switch failures.

Make sure Solaris SAN Foundation Kit (SFK) patches are installed for Solaris 8 and Solaris 9. Reference Document 392639.1

Network Considerations

For 11.2.0.2 (GI and RDBMS) and above, it is highly recommended that Oracle Redundant Interconnect/HAIP be used for interconnect redundancy. See Document 1210883.1 for details.

For pre-11.2.0.2 environments use IPMP NIC redundancy for the private interconnect. Configuration details can be found in the following notes:

For 11.2.0.1 see Document 1069584.1 - Solaris IPMP and Trunking for the cluster interconnect in Oracle Grid Infrastructure 11g Rel. 2

For pre-11gR2 see Document 368464.1 - How to Setup IPMP as Cluster Interconnect

Page 2 of 4

12/19/2011https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=BULLET...

For all releases use IPMP for public NIC redundancy. Configuration details can be found in Document 730732.1.

The Solaris OS default setting for udp_recv_hiwat and udp_xmit_hiwat are too low. For heavy cluster interconnect traffic, Oracle has increased the recommended value of both kernel parameters to at least 65536 to improve UDP throughput. See the Oracle Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Solaris Operating System (same applies to pre-11gR2 installations).

For Solaris versions earlier than Solaris 10, tune the sq_max_size to protect against dropped packets on the interconnect. See Document 1004755.1 for details.

Add set ce:ce_taskq_disable=1 to /etc/system if the ce network interface is used for cluster interconnect to prevent frequent node reboots. Reference Document 437420.1

E1000 NIC can cause kernel panic or CSS packet corruption on Solaris 10 and patch 118833-30 or later for SPARC Platform is required.

Jumbo frames with MTU 9000 may cause problems with HP NC7170 drivers.

Hardware/Vendor Specific Considerations

Ensure minimum BIOS version 2.35.3.3 is used for SUN V40Z DUAL CORE machines, for ECC memory checking.

Ensure SUN V40Z 2.6V memory management voltage regulator issues. A SUN CE can identify if the voltage regulator is beginning to fail. The new VRM (Voltage Regulator Module) revision board from rev 1.0 to rev 2.0.

T-Series machines on SUN use a threading model to make it look like there are more CPUs then the machine actually has. On these Cool Threads machines, (the example here is a machine with 32 cores), it appears to Oracle that there are actually 8 times as many CPUs (8 threads per core) - so we automatically set CPU_COUNT to 256, and then the DB will not open. The workaround here is to manually set the CPU_COUNT to a reasonable value (i.e. 16, or 32, etc, depending on the # of actual cores). See Document 781763.1 and Document 1181315.1.

Ensure the network cards don't use the shared PCI-X bus slots 2 and 3 on a SUN V40Z.

Oracle Software Considerations

The Software Considerations in this section are specific to the Solaris Platform. That said, it is highly recommended that the Platform Independent Best Practices found in Document 810394.1 be reviewed.

For 11.2.0.1 where IPMP is used for public and/or cluster interconnect, critical merge Patch 10094017 should be applied to both Grid Infrastructure and RDBMS Oracle homes. See Document 1069254.1 for details.

If using IPMP for the cluster interconnect in an 11.2.0.2 environment be sure to take corrective action on Bug 10357258 (Many HAIP created after active NIC fails in IPMP) by applying Patch 12666373 (includes the 11.2.0.2.2 GI PSU). This issue is fixed in 11.2.0.2.3 (GI PSU3).

For 11.2.0.2 when using Jumbo Frames for the interconnect HAIP may not start with the proper MTU size, see Document 1290585.1. It is recommended to apply Patch 12666373 (includes the 11.2.0.2.2 GI PSU) which corrects the following issues (in addition to the 11.2.0.2.2 fixes):

Bug 10357258 - [IPMP] HUNDREDS OF DUP IP AFTER INTRA-NODE FAILOVER Bug 9795321 - MTU SIZE FOR VIP UNDER 11GR2 GRID INFRASTRUCTURE

Note: Bug 10357258 is corrected in GI PSU3 (11.2.0.2.3), Bug 9795321 is not corrected in 11.2.0.2.3 at the time of this writing (workaround by disabling Jumbo Frames). Both bugs are to be fixed in the upcoming 11.2.0.2.4 GI PSU.

References

Attachments

RACGuides_Rac10gR2OnSolaris.pdf (23,209.51 KB) RACGuides_Rac11gR1OnSolaris.pdf (21,805.84 KB) RACGuides_Rac11gR2OnSolaris.pdf (9,461 KB)

Page 3 of 4

12/19/2011https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=BULLET...

Related

Products

Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Enterprise Edition

Keywords

CONFIGURATION; CRS; KIT; RAC; SOLARIS; VIPErrors

ENOMEM; RFC-952; ORA-27102; ORA-7445

Back to top

Copyright (c) 2007, 2010, Oracle. All rights reserved. Legal Notices and Terms of Use | Privacy Statement

Page 4 of 4

12/19/2011https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=BULLET...