deploying an oracle rac 10g release 2 database with asm on ...€¦ · deploying oracle rac 10g...

Deploying an Oracle RAC 10g Release 2 Database with ASM on a Red Hat Enterprise Linux 5 or a SUSE Linux Enterprise Server 10 with an IBM D3200 Storage Subsystem

Installation and Configuration Guide35831-00, Rev. BDecember 2008© Copyright 2008, IBM Corporation

Contents

Chapter 1: DEPLOYING ORACLE 10G DATABASE ON LINUX RED HAT 5 WITH AN IBM DS3200 STORAGE SUBSYSTEM

Introduction .......................................................................................................................................................1-1

Benefits ................................................................................................................................................................1-2

Disclaimers..........................................................................................................................................................1-4

Test Bed................................................................................................................................................................1-5

Test Bed Overview .....................................................................................................................................1-5

Test Bed Details ..........................................................................................................................................1-6

Software .................................................................................................................................................1-6

Two Servers ...........................................................................................................................................1-6

One IBM Storage Subsystem ..........................................................................................................1-7

SAS HBAs ................................................................................................................................................1-7

Configuring the Host and the Storage Subsystem ..............................................................................1-7

Configuring the Host ................................................................................................................................1-7

Configuring an IBM Storage Subsystem............................................................................................1-8

Confirming the Host and Storage Subsystem Setup ................................................................. 1-11

System Mount Points............................................................................................................................. 1-11

Pre-Installation Tasks ................................................................................................................................... 1-12

Configuring the Operating System......................................................................................................... 1-17

Installing Oracle ............................................................................................................................................. 1-20

Installing OCFS2....................................................................................................................................... 1-20

Downloading OCFS2 ...................................................................................................................... 1-20

Configuring OCFS2 ......................................................................................................................... 1-20

Initializing OCFS2 ............................................................................................................................. 1-21

Installing ASM with ASMlib 2.0 .......................................................................................................... 1-22

Installing the Oracle Clusterware ...................................................................................................... 1-23

Installing the Oracle Database Software ........................................................................................ 1-26

Creating a Database with DBCA ........................................................................................................ 1-27

Conclusion ....................................................................................................................................................... 1-29

Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem iii© Copyright 2008, IBM Corporation

Contact Information..................................................................................................................................... 1-30

Appendix A: EXAMPLE ORACLE USER PROFILE

Appendix B: EXAMPLE /etc/fstab FILE

Example mount command output on host Host2.........................................................................B-1

Appendix C: EXAMPLE /etc/sysctl.conf FILE

Appendix D: ORACLE RAC TROUBLESHOOTING

One or both hosts restart or lock up ..................................................................................................D-1

Kernel panic under heavy I/O loads ...................................................................................................D-1

Reinstalling the Oracle Clusterware...................................................................................................D-2

All hosts do not show up in DBCA or ASM creation .....................................................................D-2

Oracle Clusterware complains that “nodes are not clusterable”.............................................D-2

ASM library install error ..........................................................................................................................D-2

Cannot communicate with Clusterware services..........................................................................D-2

Error messages during the Oracle Clusterware installation ......................................................D-3

Appendix E: REFERENCES

iv Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem© Copyright 2008, IBM Corporation

Chapter 1
Deploying Oracle 10g Database on Linux Red Hat 5 with an IBM DS3200 Storage Subsystem
IntroductionIT departments that already run an Oracle RAC 10g Release 2 database with ASM on Red Hat 5 might consider upgrading their storage subsystem to an IBM storage subsystem. Any proposed upgrade raises the question of interoperability with the existing system.

Using a test setup, this document shows that you can successfully install Oracle 10g Clusterware and the Oracle 10g database application with an IBM DS3200 storage subsystem.

This document shows how to overcome the following key obstacles to a successful installation:

• Incorrect configuration of Red Hat 5 kernel parameters for Oracle. The key challenge of this configuration is tuning the Oracle kernel parameters on the hosts. Kernel parameters must fit the needs of your database environment to achieve the best performance optimization. Only certain minimum requirements for kernel tuning must be met to install correctly. Theoretically, you could wait to adjust the kernel parameters until after installation. However, kernel tuning needs to be understood in the beginning as much as possible. Kernel turning’s biggest impact is on performance.

• Incorrect file permissions for the various components. When the Oracle Cluster Register OCR file permissions are set incorrectly, installation of Oracle Cluster might fail, or even if the installation does complete, Clusterware might not run correctly. The same rules apply to the voting disk drives.

• Not getting disk drive access to the storage subsystem from all shared devices. If you do not map all shared logical drives to the host group, not all hosts see the correct LUNs. Also, if the Host Bus Adapter (HBA) driver that you installed is not compatible with the BIOS and firmware or does not use the recommended settings, the host might have problems detecting the LUNs or might experience unexpected

Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem 1-1© Copyright 2008, IBM Corporation

Deploying Oracle 10g Database on Linux Red Hat 5 with an IBM DS3200 Storage Subsystem . . .

errors. Furthermore, if you forget to disable auto failback when you install the RDAC failover driver, LUN thrashing could result in the event of a path failure on a single host.

• Incorrect partition of the IBM storage subsystem. Map all shared LUNs through storage partitioning to the host group to which all hosts in the cluster belong. If a shared LUN is mapped only to a single host, the other hosts cannot access this LUN, causing problems during installation.

• Problems installing Oracle Clusterware and Oracle Database software. Errors during configuration of a host setup and storage partitioning cause various problems (see Appendix D, Oracle RAC Troubleshooting) during installation of Oracle Clusterware and Oracle Database software. As mentioned in previous bullets, disk drive access to the correct hosts is crucial. File permissions and ownership are just as important. Kernel parameters must meet the minimum levels that Oracle recommends. An experienced database administrator can identify the correct kernel parameter settings for your environment.

BenefitsThis method of deploying an Oracle 10g database with ASM on Red Hat 5 lets you use an DS3200 storage subsystem and thus reap its many benefits. The DS3200 storage subsystem, combined with Oracle RAC, provide a reliable solution for the high availability and high performance needs of the small-to-medium business installation:

• Combines next-generation Serial-attached SCSI (SAS) technology with IBM time-proven designs for external storage to create a fully-featured storage subsystem ideally-suited for SMB requirements.

• Offers SAS technology, the next evolution of SCSI. SAS has leveraged SCSI technology and improved upon SCSI technology to offer these features:

■ Increased aggregate performance by implementing full-duplexing, and wider bandwidth

■ Greater scalability by breaking free of the 15-disk drive-barrier-per-channel limitation of parallel SCSI technology

■ Simple storage consolidation without the need of a storage network

■ Improved reliability through use of a point-to-point topology and dual-ported architecture

• Features modular design. Modular design creates an affordable entry-point without sacrificing future scalability, enabling customers to start small and grow larger when they are ready. The DS3200 storage subsystem features SAS host interface technology with up to 12 SAS disk drives per enclosure, supporting a total of 4 enclosures and 48 disk drives.

1-2 Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem© Copyright 2008, IBM Corporation

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Benefits

The DS3200 storage subsystem offers these storage capacities:

■ 7 TB of total storage, when configured with 146 GB SAS disk drives

■ 14.4 TB of total storage, when configured with 300 GB SAS disk drives

■ 24.0 TB of total storage, when configured with 500 GB SATA disk drives

■ 36.0 TB of total storage, when configured with 750 GB SATA disk drives

■ 48.0 TB of total storage, when configured with 1TB SATA disk drives

• Provides three servers with redundant connections and six host-side SAS interfaces. This configuration provides capacity-efficient storage consolidation without the need of a storage network.

• Offers solid performance as a critical component of any Oracle Database installation. The DS3200 storage subsystem is highly scalable to provide outstanding performance with these rates:

■ A maximum burst I/O rate from cache of 91,000 I/Os per second

■ A maximum sustained I/O rate from disk drive of 22,000 I/Os per second

■ A maximum sustained transfer rate from disk drive of 900 MB per second

• Provides high availability, not possible with internal storage, ensuring that data is accessible at all times. High availability offers these features:

■ Dual-active controllers with automated I/O path failover and online administration

■ RAID levels 0, 1, 3, 5 and 1+0, redundant, hot-swappable storage controllers, disk drives, and power/cooling units

■ Automatic disk drive failure detection and rebuild using global hot spare disk drives

■ Mirrored cache with battery backup

■ Configuration metadata stored on every configured disk drive

Included with the DS3200 storage subsystem, Simplicity Storage Manager provides a friendly user interface from set-up to general administration. Simplicity Storage Manager is designed for the part-time administrator and significantly reduces the complexity of installation, configuration, management and diagnostic tasks. Simplicity Storage Manager has these features:

• Online capacity expansion, logical drive creation, and host-to-logical drive mappings, offers the user control of their storage subsystem and the ability to make quick changes when necessary.



• Email diagnostic alerts and Recovery Guru provide valuable trouble shooting assistance by diagnosing the system, alerting the administrator if a problem occurs, and goes a step further by determining the appropriate recovery procedure to be taken.

Premium key-enabled features provide additional functionality and replication to an SMB's storage subsystem only when and if their storage demands call for additional functionality.

• FlashCopy® creates near-instantaneous, capacity-efficient, point-in-time logical drive images, providing a logical logical drive for such uses as decreasing file restoration or backup times, or for application testing.

• VolumeCopy creates a complete physical copy - or clone - of a logical drive within a storage subsystem. These unique copies can be assigned to any host and used for application testing, data analysis/mining, or logical drive relocation.

• Storage partitioning logically divides a single system into multiple systems by defining which host or host group can access each logical drive in your storage subsystem. This partitioning provides a range of hosts with different capacity, performance, or data protection demands to effectively share a single DS3200 storage subsystem.

The Oracle Database application is supported across all operating systems for a given controller firmware. Refer to the compatibility matrix for the supported operating system and the controller firmware you are running. Applications do not manage or manipulate the data path. Oracle RAC is not considered an application and therefore is supported on only the operating systems that were tested. Tested operating systems include the following:

■ Windows 2003

■ Redhat Linux Advanced Server 3, 4, and 5

■ SUSE Linux Enterprise Server 8 and 9

■ Solaris Sparc 8, 9, and 10

DisclaimersThis document serves as an example only, not as a best practices guide or as a configuration guide. The steps in this document were performed on a test setup. There was no live data involved. This document implies no warranties. These instructions are not guaranteed to work in every situation. For the latest information and instructions on installing Oracle products, refer to Oracle installation guides. Go to:

http://otn.oracle.com



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Test Bed

This document does not cover the following situations:

• An existing non-RAC Oracle installation with an IBM storage subsystem

(This situation requires support from Oracle to convert the database to a RAC database.)

• An existing Oracle RAC or Oracle install with IBM storage subsystem

The complexity of the installation described in this document requires that the installer have considerable prior expertise. Do not attempt to follow this procedure unless you are correctly qualified. Installing this type of configuration for production requires special expertise for tuning an IBM storage subsystem, the Oracle Database parameters, and the host server to best fit the needs of your configuration. This type of installation requires expertise on both data migration and on Oracle. The installer must have worked with Oracle Databases, Oracle Clusterware, Redhat Linux Advanced Server 5, and theIBM DS3200 storage subsystem.

Test BedThe operating system used in this test setup was Red Hat Enterprise Linux (RHEL) Advanced Server 5. The installer must be familiar with all aspects of the hardware and software configuration before attempting to follow the instructions in this document.

Test Bed OverviewThe test setup consists of RHEL AS 5. However, the instructions in this document might also be applicable to SLES10 and later. For the latest updates, go to:


Oracle Cluster File System 2 (OCFS2) and Oracle Automatic Storage Management (ASM) were used to configure the shared devices.

• OCFS2 – A general purpose cluster file system developed by Oracle for Oracle database products, OCFS2 was used for voting and registry files. For documentation on OCFS2, go to:

http://www.oracle.com.

• ASM – A feature in Oracle Database 10g that provides the database administrator with a simple storage management interface that is consistent across all server and storage platforms. As a vertically integrated file system and a logical drive manager, purpose-built for Oracle database files, ASM provides the performance of asynchronous I/O with the easy management of a file system. ASM provides capability that saves the database administrator time and provides flexibility to manage a dynamic database environment with increased efficiency. File systems and logical drive managers use the disk drive that is served up from the storage


http://otn.Oracle.com

http://www.oracle.com


subsystem. The disk drive is then filtered through the multi-path disk driver. As long as the operating system, the HBA, and the multi-path disk driver are supported by IBM, Oracle ASM works.

■ For supported HBAs, go to:

http://www.ibm.com/systems/support/storage/config/ssic/displayesssearchwithoutjs.wss?start_over=yes

■ For documentation on ASM go to:

http://www.oracle.com/technology/products/database/asm/index.html

NOTE The test setup uses the cluster software supplied with Oracle 10gR2. In previous releases of Oracle, this cluster software was called Cluster Ready Services (CRS). However, the cluster software was renamed to Oracle Clusterware. Therefore, the term Clusterware refers to CRS in this document.

Test Bed Details

Software

Oracle 10g R2 DB – 10.2.0.3

Oracle 10g R2 Clusterware – 10.2.0.3

ASMlib driver 2.0

OCFS2 – Version 1.2.6

ocfs2console-1.2.6

ocfs2tools-1.2.6

Two Servers

Server 1 – has a single 36.4 GB SCSI hard disk drive

Server 2 – has a single 36.4 GB SCSI hard disk drive

Both servers have RHEL5

MPP 99.01.C5.13


http://www.Oracle.com/technology/products/database/asm/index.html


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuring the Host and the Storage Subsystem

One IBM Storage Subsystem

Firmware – 06.70.35.00

NVSRAM – N133x670800-000

DS4000 Storage Manager – 02.70.G5.11

SAS HBAs

Vendor/Model – IBM 1068E

Two single-port HBA in each host

Driver – MPP-4.00.13.04

Firmware – 01170200h

NOTE Testing was performed on the specified test bed. Check the IBM Compatibility Matrix for supported HBAs. See step 2 in the following procedure.

Configuring the Host and the Storage Subsystem

Configuring the HostTo completely configure the host, you must install and configure the operating system, the HBA driver, and the DS4000 Storage Manager software.

1 Install the operating system. Refer to the operating system documentation. Check with Oracle that the operating system is certified. Make sure the kernel that you installed is SMP-capable.

2 Install the HBA driver. For supported HBAs and drivers, refer to the IBM Compatibility Matrix. Go to:


For the HBA Settings document, contact Support. See “Contact Information” on page 30.

3 Install the DS4000 Storage Manager software. Also known as SMinstaller, this software contains the following individual packages:

■ SMclient – Manages the storage subsystem from a single management station

■ SMutils – Collection of utilities to confirm what the host sees from the storage subsystem. Installing SMutils is highly recommended, and SMutils can be installed independently of the other packages.




■ SMruntime – Java runtime environment

■ SMagent – Required for inband management of the storage subsystem

4 Download the RDAC driver from the following site and follow the instructions provided in the README for installation:

http://www-304.ibm.com/jct01004c/systems/support/supportsite.wss/allproducts?taskind=2&brandind=5345868#st_software

5 Make sure that the RDAC is loaded after a restart:

lsmod | grep mpp

Example output:

[root@Host2 /]# lsmod | grep mpp

mpp_Vhba84096 4

mpp_Upper 74356 0 [mpp_Vhba]

scsi_mod 115496 6 [mpp_Vhba qla2300 aic7xxx mpp_Upper sg sd_mod]

Configuring an IBM Storage Subsystem1 Using the IBM Storage Manager Client, create logical drives on the IBM storage

subsystem.

a Oracle home – Install the Oracle Clusterware and the Oracle Database software on a logical drive or logical drives from the storage subsystem. Each host must separate file systems, one for the Oracle Clusterware and one for the Oracle Database software. You also can choose to install these softwares on the host disk drive itself.

b Oracle system and data files – Create separate logical drives for the voting, registry, and ASM disk drives. Configurations for production vary depending on the following factors:

- Percentage of reads and writes

- Size of the database

- Expected growth

- Purpose of the database, that is, data warehousing, OLTP and so forth.

- Number of simultaneous connections

- Server configuration (memory, number of processors, speed of the processors)

- Network infrastructure


http://www-304.ibm.com/jct-1004c/systems/support/supportsite.wss/allproducts?taskind=2&brandind=5345868#st_software


- Whether queries are optimized to use indexes

IMPORTANT Consult with an experienced Oracle database administrator to determine the best configuration.

2 In the DS4000 Storage Manager main window, right-click the Unconfigured Space icon, and choose Create Logical Drive to open the Create Logical Drives wizard. Follow the prompts to create the logical drives for this installation of Oracle RAC. Table 1-1 shows the options that you must choose on each page of the wizard.

3 Create host port partitions on the storage subsystem to map the logical drives to a single host or to a group of hosts for Oracle RAC.

Figure 1-1 shows how the mappings would be in a setup with only one storage subsystem.

Table 1-1 Create Logical Drives Wizard

Page Action

Introduction Click Next.

Specify Array Choose the RAID Level and the disk drive selection on this screen. The recommended disk drive selection is Automatic.

NOTE Interpret the term Array to mean a storage subsystem.

Specify Capacity Name Specify the capacity and the name for the logical drive.

Specify Logical Drive-to-LUN Mapping Specify the host type and choose to map the logical drives to LUNs at a later time. In the test setup, “Linux” is the host type.



Figure 1-1 A Setup with Only One Storage Subsystem

Host (1)

DS3300Controller Left Ctlr BRight Ctlr A

123

86123-11

ASM 1LUN 6

Host 2 Oracle HomeMapped only to Host 2

ASM 2LUN 7

ASM-spfileLUN8

Voting Diskvote 3LUN 5



Oracle ClusterRegistry(OCR2)LUN 2

Oracle ClusterRegistry(OCR1)LUN 1

Host 1 Oracle HomeMapped only to Host 1

ASM DG-1

Host (2)

456

789

101112

Test Datebase

NOTES:

All OCR, Voting and ASM LUNs are mappedto both hosts via SANtricity Storage Partioning.Each host will connect to the Test Database.

DS3200Controller



The test setup can be summarized as follows:

• The host group Host1-Host2 contains two hosts, Host1 and Host2.

• Each individual host has a logical drive that contains a file system for Oracle Database home and Oracle Clusterware home, to which only that host has access, as shown in the Accessible By column.

• Each host can see the logical drives mapped to the host group, as shown in the Accessible By column.

IMPORTANT The LUN numbering for the logical drives that each host sees must always start with 0 (zero) on at least one storage subsystem. In addition, LUNs must be sequential. Therefore, do not skip LUN numbers when mapping. Problems can occur with hosts not seeing LUNs if these conditions are not met.

Confirming the Host and Storage Subsystem SetupConfirm the storage subsystem access by making sure that the hosts have access to the mapped logical drives.

The /usr/sbin/SMdevices command shows what the host sees. After the initial mapping of the logical drives, you can either restart the server, or run the hot_add command. Restarting is preferred.

The devnode disk designation in the SMdevices listings need not be identical on both hosts. Even so, make sure that the two hosts are formatting and working with the same shared device across hosts if they have different /dev/<disk> names. If using OCFS2, resolve the issue by running the ocfs2console command to initialize and mount the clustered file systems on each host.

System Mount PointsSee Appendix B, Example /etc/fstab File for an example of what mount points are used in this test setup. Compare Appendix B, Example /etc/fstab File with the SMdevices listing shown from the /usr/sbin/SMdevices command.



Pre-Installation Tasks1 Check the Oracle installation manual for hardware and software requirements.

For the latest certified list of hardware and software, go to:

http://metalink.oracle.com

2 Synchronize the clocks.

Make sure that the clocks on all of the hosts are synchronized. You can perform this task either manually or by using the NTP (network time protocol) servers. If the clocks do not match, the Oracle Clusterware installation gives errors.

3 Set up the host network.

a Configure three static IP addresses (public, private, and virtual) for each host. In the test setup, the eth0 interface is public and the eth1 interface is private. Configure interfaces through the system-config-network command in RHEL5.

b Specify the virtual IP in /etc/hosts, but do not setup up the virtual IP manually through system-config-network. Wait to set up the virtual IP until directed later.

c Make sure that the virtual IP is on the same subnet as the public IP address.

d Make sure that the selection of interfaces is the same on all hosts. The eth0 should be public and eth1 should be private on all hosts.

e Make sure that the loopback line in the /etc/hosts file does not contain the host name.

f Enter all IP addresses configured from each host’s /etc/hosts file.

Table 1-2 Sample /etc/hosts File

IP Address Host Name Abbreviated Host Name

127.0.0.1 localhost.localdomain localhost

172.22.215.95 Host1.ks.lsil.com Host1

172.22.215.25 Host2.ks.lsil.com Host2

172.22.215.197 Host1vip

172.22.215.198 Host2vip

192.168.1.95 Host1prv

192.168.1.25 Host2prv


http://metalink.oracle.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pre-Installation Tasks

4 Get the packages from Oracle.

a Use the four main packages:

- Oracle Clusterware – Go to: http://otn.oracle.com, then go to Downloads.

- Oracle Database – Go to: http://otn.oracle.com, then go to Downloads.

- OCFS2 – Go to: http://oss.oracle.com

- ASM – Go to: http://oss.oracle.com

b Place these packages in the /home/oracle directory with oracle:oinstall owning them. This directory becomes the oracle user’s home directory.

5 Set up oracle users and groups.

a Create oinstall and dba groups on the first host, if the groups do not exist.

- /usr/sbin/groupadd oinstall

- /usr/sbin/groupadd dba

b Create the Oracle Database software owner user (Oracle).

- /usr/sbin/useradd –u 200 –g oinstall –G dba –d /home/oracle –s /bin/bash oracle

- Set the password for this user using: passwd oracle

c Make sure that the user “nobody” exists: id nobody. If the user does not exist, add this user by typing the following command.

/usr/sbin/useradd nobody

d On the first host, type the following command:

id oracle.

This command gives you the oracle user’s details.

Example:

uid=200(oracle) gid=501(oinstall) groups=501(oinstall),502(dba)

e On the second host, create users and groups that are identical to those on the first host.

- /usr/sbin/groupadd –g 500 oinstall

- /usr/sbin/groupadd –g 501 dba

- /usr/sbin/useradd –u 200 –g oinstall –G dba –d /home/oracle –s /bin/bash oracle




http://oss.Oracle.com



- passwd oracle

6 On both hosts, create an Oracle user home, Clusterware and Oracle Database homes, and voting and registry directories. This setup attempts to adhere to the Oracle Optimal Flexible Architecture (OFA) guidelines for setting up directory structures. See The Oracle Database Installation Guide 10gR2 for additional details.

a Make the Oracle user home directory:

mkdir /home/oracle

chown oracle:oinstall /home/oracle

chmod 700 /home/oracle

b Create the Oracle base directory:

mkdir -p /u01/app/oracle

chown -R oracle oinstall /u01/app

chmod -R 775 /u01/app/oracle

c Create the Clusterware home directory:

mkdir /u01/crs

chown -R root oinstall /u01/crs

chmod -R 775 /u01/crs

7 Configure the hosts to establish user equivalency so that the oracle user can access one host from the other without a password. User equivalency means that a user can log on to any host, and from that host then access all other hosts without a password. To establish user equivalency, configure both hosts to accept either the SSH command or the RSH command.

■ SSH is the preferred type of user equivalency. To establish user equivalency using SSH, go to step 8.

■ Use RSH only if SSH does not work. To establish user equivalency using RSH, go to step 9.

IMPORTANT Because installations perform remote copying of files, Oracle Clusterware and Oracle Database software installations can fail unless user equivalency is established as described in step 7.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pre-Installation Tasks

8 To establish user equivalency using the SSH command, complete the following steps.

a Log on as the oracle user: ssh oracle@host1.

b Make an .ssh directory in Oracle’s home directory if one does not exist.

c Type the following command: chmod 700 .ssh

d In the .ssh directory, type the following commands. Accept all defaults, and enter no passwords.

/usr/bin/ssh-keygen –t rsa.

/usr/bin/ssh-keygen –t dsa.

e Repeat step a through step d on the second host.

f Run the following commands on Host1.

touch ~/.ssh/authorized_keys

chmod 600 ~/.ssh/authorized_keys

g Go to .ssh directory on Host1. Run the following commands, entering the appropriate password when prompted.

cat id_rsa.pub >> authorized_keys

cat id_dsa.pub >> authorized_keys

ssh Host2 cat /home/oracle/.ssh/id_rsa.pub >> authorized_keys

ssh Host2 cat /home/oracle/.ssh/id_dsa.pub >> authorized_keys

h Also in the .ssh directory on Host 1, run the scp authorized_keys Host2:/home/oracle/.ssh command. This command does not require a password.

i Run the following commands on both hosts.

$ exec /usr/bin/ssh-agent $SHELL

$ /usr/bin/ssh-add

j Make sure that you can use the SSH command to access one host from another without using a password. For example from Host2: ssh Host1 date should show the date without having to use a password. Make sure that you can use the SSH command to access the same host. For example, from host 1, use ssh to access host 1 data. Oracle fails the installation if Oracle cannot gain ssh access to local and remote hosts. The database might need more than one try to achieve this task.



k Was user equivalency was established?

- Yes – Go to step 10.

- No – Go to step 9.

9 To establish user equivalency using the RSH command, do the following tasks on both hosts. Depending on your configuration this example might not work because later releases have implemented various security fixes to RSH command such as Red Hat 5. Refer to Red Hat 5 documentation for implementation.

a Enable RSH and rlogin.

As the root user, enter the following commands:

chkconfig rsh on

chkconfig rlogin on

service xinetd reload

b Create an /etc/hosts.equiv file on each host.

1 Create the file with the touch /etc/hosts.equiv command.

2 Set the file permissions with the following commands:

chmod 600 /etc/hosts.equivchown root:root /etc/hosts.equiv

3 Add in all hosts to this file.

Example:

4 In the /etc/securetty file, add in rsh and rlogin at the bottom.

c Make sure that both hosts have access to one another without a password. Do the following from each host as the oracle user:

rsh <public_nodename> datersh <private_nodename> date

10 On each host, create a partition and ext2/ext3 file system for the Oracle_home logical drive and the crs_home logical drive. Add an entry to /etc/fstab to both hosts to mount on startup.

+Host1 oracle #Public hostname for node 1

+Host1prv oracle #Private hostname for node 1

+Host2 oracle #Public hostname for node 2

+Host2prv oracle #Private hostname for node 2


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuring the Operating System

Example of /etc/fstab:

/dev/sdb /u01 ext2 defaults 0 0

11 Prevent stty command errors.

From a terminal, if using the Bourne, bash, or Korn shell, run the following commands.

if [ -t 0 ]; then

stay intr ^C

fi

12 Configure the oracle user’s environment.

a For each host in the oracle user’s home directory, configure a .bash_profile such as the one shown in Appendix A, Example Oracle User Profile.

b Make sure that the ORACLE_SID variable in the .bash_profile is unique. The variable is usually the same string followed by unique numbers. Make the string your intended database name.

c Run the .bash_profile by: . ./.bash_profile.

Configuring the Operating SystemPerform the following procedure on each host of the operating system.

1 Make sure that the swap space is at least 512 MB or equal to the amount of RAM in the server. If needed, add temporary swap space by creating a temporary swap file. For instructions, refer to the documentation for the operating system.

2 Refer to the Oracle manual for additional hardware and software requirements.

3 Adjust the kernel parameters as specified in Table 1-3.

.Table 1-3 Kernel Parameters (1 of 2)

Parameter Value File

semmsl 250 /proc/sys/kernel/sem

32000

100

128

shmmax Half the size of physical memory.

(in bytes)

/proc/sys/kernel/shmmax



a Are the current values equal to or greater than the recommended values listed previously in Table 1-3, “Kernel Parameters”?

- Yes – Go to step c.

- No – Go to step b.

b If not, change the settings in /etc/sysctl.conf. See Appendix C, Example /etc/sysctl.conf File.

c Apply the settings by running the command appropriate for your operating system.

- For Redhat, run the /sbin/sysctl –p command.

- For SUSE, run the /sbin/chkconfig boot.sysctl on and reboot commands.

4 Add in /etc/security/limits.conf.

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536

5 Add to /etc/pam.d/login:

session required /lib/security/pam_limits.so

6 Add to /etc/profile

Bourne, Bash, or Korn shell:

if [ $USER = "oracle" ]; then

shmmni 4096 /proc/sys/kernel/shmmni

shmall 2097152 /proc/sys/kernel/shmall

file-max 65536 /proc/sys/fs/file-max

ip_local_port_range Minimum: 1024

Maximum: 65000

/proc/sys/net/ipv4/ip_local_port_range

rmem_default 1048576 /proc/sys/net/core/rmem_default

rmem_max 1048576 /proc/sys/net/core/rmem_max

wmem_default 262144 /proc/sys/net/core/wmem_default

wmem_max 262144 /proc/sys/net/core/wmem_max

Table 1-3 Kernel Parameters (2 of 2)

Parameter Value File


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuring the Operating System

if [ $SHELL = "/bin/ksh" ]; then

ulimit -p 16384

ulimit -n 65536

else

ulimit -u 16384 -n 65536

fi

fi

7 Configure the hangcheck-timer module. Oracle Clusterware uses this module to fence (isolate and restart) an unresponsive system.

a Make sure that the hangcheck-timer module is running: lsmod | grep hang

b If the hangcheck-time module is not running, run the following command to start the hangcheck-timer module.

insmod /lib/modules/<kernel version>/kernel/drivers/char/hangcheck-timer.ko hangcheck_tick=30 hangcheck_margin=180

c Insert the command from step b into the file appropriate for your operating system.

- For Redhat, insert the command into the /etc/rc.d/rc.local file.

- For SUSE, insert the command into the /etc/init.d/boot.local file.

Inserting the command into the appropriate file causes the hangcheck-timer module to be loaded at startup.

8 Run the rootpre.sh script located on the Oracle Clusterware installation media at $mount_point/clusterware/rootpre.



Installing OracleThe following sections describe how to install Oracle by performing these steps.

• Install OCFS2

• Install ASM with ASMlib 2.0

• Install Oracle Clusterware

• Install Oracle Database software

• Create a database with DBCA

Installing OCFS2The following sections describe how to install OCFS2 by performing these steps:

• Download OCFS2

• Configure OCFS2

• Initialize OCFS2

Downloading OCFS2

Install the following OCFS2 packages on both hosts. To download the packages, go to:

http://oss.oracle.com

• OCFS2 driver for your kernel version

• OCFS2 Tools

• OCFS2 console

These packages might have other dependencies that you need to install.

Configuring OCFS2

1 Disable SELinux if you are running OCFS2 on RHEL4 U2. SELinux has known issues with OCFS2.

2 Run the ocfs2console command to bring up the GUI.

3 Select Cluster >> Configure Nodes.

NOTE A node is synonymous with a host.

4 In the Node Configuration dialog, add your host names and IP addresses, and then save the changes.



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installing Oracle

5 Select Cluster >> Propagate Configuration. Make sure that each host has an identical /etc/ocfs2/cluster.conf file containing the host information.

6 Bring up the o2cb cluster stack on all hosts. Type these commands to shut down o2cb if o2cb is already up.

/etc/init.d/o2cb offline ocfs2/etc/init.d/o2cb unload/etc/init.d/o2cb configure

The last command lets you configure o2cb to load on boot. The name of the cluster to start on boot is by default ocfs2.

Initializing OCFS2

Do step 1 through step 4 from only one host.

1 Run the ocfs2console command to bring up the GUI.

2 Initialize all voting and registry logical drives.

3 To start to initialize OCFS2, select Tasks >> Initialize. From the disk drive list, choose the correct disk drives to initialize. See SMdevices output for that host to select the appropriate disk drives.

4 Mount the initialized disk drives to their mount points. Perform this task by selecting each disk drive and clicking the Mount button. Each ocfs2 logical drive must be mounted with “_netdev,datalogical drive,nointr” options.

The test setup uses the following mount points:

Voting_disk = /dev/sdde mounted on /vote1

Registry_disk = /dev/sded on /ocr1

Backup_voting = /dev/sdfh on /vote2

Backup_voting=/dev/sdk on /vote3

Backup_registry = /dev/sdli on /ocr2

a Go to the remaining hosts and open ocfs2console. The initialized disk drives already should appear in list. Mount them to the same directories as the first host.

b On all hosts, add the ocfs2 mount point information to the /etc/fstab file. See Appendix B, Example /etc/fstab File for an example /etc/fstab file with Oracle mount points specified.

c On one host, create empty voting files in the directories created in step 4. Create three voting files to achieve normal redundancy. Create only one voting file if you intend to use external redundancy.



In the test setup, the locations and filenames of the three voting files are as follows.

- /vote1/votefile1

- /vote2/votefile2

- /vote3/votefile3

The voting files must be empty during install, or Oracle Clusterware does not function correctly.

d On the same host as step c, create two registry files. The registry files must be empty during install, or Oracle Clusterware does not function correctly. In the test setup, the location and filename of the registry file is /ocr1/registry_file1 , /ocr2/registry_file2.

5 Restart all hosts and make sure that o2cb is loaded and all ocfs2 logical drives are mounted on boot.

6 Go to “Installing ASM with ASMlib 2.0” on page 22.

Installing ASM with ASMlib 2.01 Download the ASMlib driver, oracleasmlib, and oracleasm-support

packages for your kernel version. For details, go to:

http://oss.oracle.com.

2 Install all packages on both hosts.

3 On both hosts, configure the oracleasm to load on boot by running the /etc/init.d/oracleasm configure command. Choose the following options:

■ For Default user, choose oracle.

■ For Default group, choose dba.

■ Answer Yes to all other prompts.

4 On only one host, create the ASM disk drives to store database files (data, control, redo log files, and so forth). The test setup uses three ASM disk drives.

a Using fdisk, create partitions that uses the whole disk drive on all ddisk drives you want to use as ASM disk drives. For example, /dev/sdf1, /dev/sdg1.




b Run the following command to create all ASM disk drives.

/etc/init.d/oracleasm createdisk <name> <device name>

c Run the following command to confirm disk drive creation.

/etc/init.d/oracleasm listdisks

5 Run the following commands in all other hosts.

■ To detect disk drives, run the /etc/init.d/oracleasm scandisks command.

■ To confirm disk drive discovery, run the /etc/init.d/oracleasm listdisks command.

6 Go to “Installing the Oracle Clusterware.”

Installing the Oracle ClusterwareBefore you install Oracle Clusterware, you must make sure that all of your servers are seeing the same devices. Because of multiple factors, sometimes different servers do not recognize the same LUN as the same device. For Oracle RAC to operate correctly, the servers must recognize each device through the same discovery path. To prevent this problem, you must create links to each device on each server. These links must be the same on all servers.

Table 1-4 shows the devices that Oracle uses to configure the voting disk drives and the registry disk drives for RAC, with the links that you must create. LUN 1 and LUN 4 have different names for each server. However, because the same link is used by each server, the situation remains workable. Oracle RAC references the devices under the Link column.

Device persistence is another issue. After you restart a single host, devices sometimes do not remain the same. To make sure that the underlying devices do not change upon a restart or when new devices are added, you must use bindings at the host adapter

Table 1-4 Oracle Devices

Storage Subsystem

LUN #

Devices Seen by Server A

Devices Seen by Server B Link

0 /dev/sdea /dev/sdea ocr1

1 /dev/sdlb /dev/sdlc ocr2

2 /dev/sdd /dev/sdd vote1

3 /dev/sdfe /dev/sdfe vote2

4 /dev/sdkf /dev/sdkf vote3



level. Refer to either Red Hat documentation or HBA documentation for information about persistent binding. In addition, use the appropriate one of the following methods:

• Use these links as OCFS2 mount points, if you are using OCFS2.

• Create links at the operating system level from the correct device to the link location.

After you make sure that the essential devices are linked and are persistent, install the Oracle Clusterware according to the following instructions.

1 Make sure that all logical drives are mounted and that all directories are owned by oracle:oinstall and have the right permissions.

2 Store the Oracle Clusterware and Oracle Database images in the oracle user’s home directory and have them be owned by oracle:oinstall.

3 Log on as the oracle user. Do not use su – oracle, use ssh oracle@nodename.

4 Run the unset command to clear the following environment variables:

■ unset ORA_CRS_HOME

■ unset ORACLE_HOME

■ unset ORA_NLS10

■ unset TNS_ADMIN

5 Make sure that ORACLE_SID is unique on each host and that ORACLE_BASE is set to the correct location.

6 Go to the Oracle Clusterware image’s location, and run runInstaller to start the Oracle Universal Installer wizard. Table 1-5 shows the options that you must choose on each page of the wizard.



7 After the database software is installed, download the 10.2.0.3 update and any other updates required from the Oracle web site (http://oracle.com) and install according to the instructions there.

8 Update both CRS and DB homes. Before loading the update, stop the listener and the ASM.

Table 1-5 Oracle Universal Installer Wizard

Page Action

initial screen Click Next.

Specify Inventory Directory and Credentials Accept the default inventory location. This location is shared between the Oracle Clusterware home and the Oracle Database home. The location shown should be $ORACLE_BASE/oraInventory.

Specify Home Details Specify the Oracle Clusterware mount point (/crs in the test setup).

Product-Specific Prerequisite Checks All checks pass if everything is configured correctly. If not, fix the issue and click Retry.

Specify Cluster Configuration Specify the hosts. Click Use Cluster Configuration File if you have many hosts. A “nodes are not clusterable” error appears if SSH or RSH are not configured correctly.

Specify Network Interface Usage Specify the Interface Type appropriately as Public or Private.

Specify Oracle Cluster Registry (OCR) Location Choose Normal Redundancy.

In the Specify OCR Location field, type the pathway and filename to the registry mount point.*

In the Specify OCR Mirror Location field, type the pathway and filename to the backup registry mount points.*

*These mount points have files in them that you created earlier. See step 4 on page 21.

Specify Voting Disk Location Specify the location of the voting disk drive files created in step 4c of “Installing OCFS2” on page 21.

Execute Configuration Scripts Follow the on-screen directions to run scripts on both servers. Scripts might take up to ten minutes to run completely. After the scripts have run, do not click OK yet.



9 Run the vipca application manually. Running vipca manually is necessary because Oracle 10.2.0.1 has a known bug (4437727) at the end of running the scripts. The last host that you run the scripts on provides the following message:

The given interface(s), “eth0” is not public. Public interfaces should be used to configure virtual IPs.

NOTE Before running the Vipca utility, see mealink doc-id: 414163.1 regarding modification to srvctl.

a As root user on the last host that the scripts were run on (Host2 in the test setup) run $ORA_CRS_HOME/bin/vipca and click Next on the initial screen.

b Use the VIP Configuration Assistant wizard to configure virtual IP addresses through vipca. Table 1-6 shows the options that you must choose on each page of the wizard.

10 Check the Oracle Clusterware status by running the $ORA_CRS_HOME/bin/crs_stat -t command on either host.

11 Go to “Installing the Oracle Database Software.”

Installing the Oracle Database Software1 Make sure that you unset the environment variables as you did before you ran the

Oracle Clusterware installer. The Oracle Database software image or folder should be owned by oracle:oinstall.

Table 1-6 VIP Configuration Assistant Wizard

Page Action

Network Interface Select both interfaces, and then click Next.

Virtual IPs for Cluster Nodes Configure the IP Alias Name and the virtual IP address as you specified in /etc/hosts. See step 3 of “Pre-Installation Tasks” on page 12.

Summary Confirm your actions.

Progress Dialog Watch as Vipca creates the appropriate application resources.

final screen shows configuration results

Click Finish.

Execute Configuration Scripts Click OK. The configuration assistants confirm correct configuration.



2 As the oracle user, run runInstaller to open the Oracle Universal Installer wizard. Table 1-7 shows the options that you must choose on each page of the wizard.

.

3 Go go “Creating a Database with DBCA.”

Creating a Database with DBCABefore creating a database, you must create a listener. Log on again as the oracle user and make sure that all environment variables are correctly set. Run $ORACLE_HOME/bin/netca to start the Oracle Configuration Assistant for Real Application Clusters. Table 1-8 shows the options that you must choose on each page of the wizard.

Table 1-7 Oracle Universal Installer Wizard

Page Action

initial screen Click Installed Products to make sure that the Oracle Clusterware is installed successfully.

Select Installation Type Select Enterprise Edition.

Specify Home Details Specify $ORACLE_BASE/<any folder> as the location of the SW home directory for your database.

Specify Hardware Cluster InstallationMode

Select Cluster Installation and select all hosts.

Product-Specific Prerequisite Checks All checks pass if everything is configured correctly. If not, fix the issue and click Retry.

Select Configuration Option Choose Install database software only. You can configure the database later with DBCA.

Execute Configuration Scripts Follow the on-screen directions.

Table 1-8 Oracle Configuration Assistant Wizard for Real Application Clusters (1 of 2)

Page Action

Configuration Choose Cluster configuration.

Active Nodes Click Select all nodes.

Welcome Select Listener configuration.

Listener Name Name the listener.

Select Protocols Select TCP as the protocol.



Netca creates the listener on both hosts. You can check listener status using lsnrctl status on both hosts.

For the test setup, the Database Configuration Assistant (DBCA) was used to create the clustered database. As the oracle user, run dbca ($ORACLE_HOME/bin/dbca) to open the Oracle Database Configuration Assistant, which guides you through the process of the creating the database. Table 1-9 shows the options that you must choose on each page of the wizard.

TCP/IP Protocol Specify the port number to use.

Welcome Select Naming Methods configuration.

Select Name Select Local Naming.

Table 1-9 Database Configuration Assistant Wizard (1 of 2)

Page Action

Welcome Select Oracle Real Application Clusters database.

Operations Select Configure Automatic Storage Management. You must set up ASM before setting up the database.

Node Selection Click Select All to select all hosts. If all hosts do not show, see Appendix D, Oracle RAC Troubleshooting to resolve the problem.

Create ASM Instance Specify a password. The test setup uses an SPFILE (Oracle recommended) for the ASM instance. You must specify a shared OCFS2 location for the SPFILE. In the test setup, the location is /vote/dbs, where /vote is the OCFS2 mount point for the voting disk drive.

Create Disk Group Configure ASM disk drives into arrays. Recall that the test setup creates ASM disk drives using /etc/init.d/oracleasm. DBCA recognizes these disk drives. Choose a Redundancy option:

• If you choose External, create at least 2-3 arrays, dividing the disk drives as you want.

• If you choose High redundancy or Normal redundancy, you must specify failover groups (see ASM documentation). Create at least one failover group for each storage subsystem that you use.

ASM Disk Groups The test setup uses two ASM arrays. Click Finish to mount and use.

Operations Select Create a Database.

Node Selection Click Select All to select all hosts.

Database Templates Select the database type to create. The test setup uses the General Purpose database type.

Table 1-8 Oracle Configuration Assistant Wizard for Real Application Clusters (2 of 2)

Page Action


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion

ConclusionBy following the instructions and the test setup provided in this paper, a system administrator can correctly configure and install Oracle RAC using an IBM storage subsystem as the primary storage. Careful planning and execution of the prerequisites help to guarantee a successful installation of this solution. Confirm the following items on the host setup:

• Linux 2.6 kernel parameters for Oracle

• File permissions for all Oracle devices, such as the Oracle Cluster Register (OCR) disk drives, voting disk drives

DatabaseIdentification

Name the database and SID. The SID prefix must be exactly the same string that you specified in the $ORACLE_SID environment variable without the numbers.

Management Options Choose management options to configure.

Database Credentials Specify passwords.

Storage Options Click Automatic Storage Management (ASM) to select ASM to manage storage.

ASM Disk Groups Select one of the ASM arrays that you created to use as main storage for database files. You can create and modify new disk drives and groups at this point as well.

Database FileLocations

Make sure that the array you selected in the ASM Disk Groups page is listed. Select any location you want. The test setup uses Oracle-managed files. You also can choose a common location.

RecoveryConfiguration

Select Specify Flash Recovery Area for additional redundancy. The test setup uses Diskgroup2 as the Flash Recovery Area.

Database Content Select any samples to include and any custom scripts you intend to run at the time of database creation.

Database Services Create any database services that you want. The test setup uses none.

InitializationParameters

Choose initialization parameters. The test setup uses defaults.

Database Storage Review the options you have selected.

Copying databasefiles

Watch as the database is created.

Database creationcomplete

See when the database is successfully created.

Table 1-9 Database Configuration Assistant Wizard (1 of 2)

Page Action



• Storage subsystem disk drive access for all shared devices including the correct HBA driver installation and RDAC failover driver installation.

Provisioning the IBM storage subsystem is also a vital step in the configuration. Make sure that the storage partitioning of each storage subsystem is configured to grant access to all shared LUNs by all hosts in the cluster. Storage Partitioning is a premium feature of an IBM storage subsystem that permits mapping of LUNs to a specific host or hosts.

The final step is installing the Oracle Clusterware and the Oracle Database software. If the host and the storage subsystem setup were completed correctly, the installation goes smoothly without errors. Typically, errors encountered during this step of the process are a result of errors during host setup or storage subsystem provisioning. The key challenge of this configuration is tuning the Oracle kernel parameters on the hosts to fit the needs of your database environment to achieve the optimal performance.

Use this document to configure the host and the storage subsystem so that Oracle installation becomes simple. Plan to upgrade to an IBM storage subsystem so that you can take advantage of fast performance and premium features. If you have a complex system with multiple hosts and multiple operating systems, you can use the Storage Partitioning premium feature to map them to a single IBM storage subsystem.

Contact InformationFor more information and sales office locations, visit the IBM web site at:

http://www.ibm.com


http://www.ibm.com

Appendix A
Example Oracle User Profile
# .bash_profileumask 022

# Get the aliases and functionsif [ -f ~/.bashrc ]; then . ~/.bashrcfi

# User specific environment and startup programs

PATH=/sbin:/usr/bin:/usr/sbin:/usr/ucb:/etc/:/u01/crs:/u01/appPATH=$PATH:/bin:/usr/local/bin:/u01/crs/oracle/product/10.2.0/app/bin:.:

export PATHunset USERNAMEexport ORACLE_SID=orcl2export ORA_CRS_HOME=/u01/crs/oracle/product/10.2.0/appexport ORACLE_BASE=/u01/app/oracleexport ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1export TNS_ADMIN=/u01/app/oracle/product/10.2.0/db_1/network/adminexport ORACLE_ADMIN=$ORACLE_BASE/adminexport ORACLE_PFILE=$ORACLE_ADMIN/$ORACLE_SID/pfileexport ORACLE_ADHOC=$ORACLE_ADMIN/adhocexport ORACLE_ARCH=$ORACLE_ADMIN/archexport ORACLE_BKUP=$ORACLE_ADMIN/bkupexport ORACLE_EXP=$ORACLE_ADMN/expexport ORACLE_TERM=vt100

PS1='$HOSTNAME:$ORACLE_SID:[$LOGNAME]:>'if [[ ! -d $ORACLE_HOME ]]; thenecho "\n>> * * * * * * * WARNING * * * * * * * <<"echo "\n>> ORACLE_HOME DOES NOT EXIST '$ORACLE_HOME'"echo "\n>> * * * * * * * WARNING * * * * * * * <<"fiPATH=$PATH:$ORACLE_HOME/bin:

Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem A-1© Copyright 2008, IBM Corporation

Example Oracle User Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A-2 Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem© Copyright 2008, IBM Corporation

Appendix B
Example /etc/fstab File
#

#ORACLE_MOUNTS

Example mount command output on host Host2# /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)/dev/sda1 on /boot type ext3 (rw)tmpfs on /dev/shm type tmpfs (rw)/dev/sdb on /u01 type ext2 (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

/dev/VolGroup00/LogVol00 / ext3 defaults 1 1

LABEL=/boot /boot ext3 defaults 1 2

devpts /dev/pts devpts gid=5,mode=620 0 0

tmpfs /dev/shm tmpfs defaults 0 0

proc /proc proc defaults 0 0

sysfs /sys sysfs defaults 0 0

/dev/sdb1 swap swap defaults 0 0

/dev/VolGroup00/LogVol01 swap swap defaults 0 0

/dev/sdb /u01 ext2 defaults 0 0

/dev/sdd /vote1 ocfs2 _netdev,datavolume,nointr 0 0

/dev/sde /ocr1 ocfs2 _netdev,datavolume,nointr 0 0

/dev/sdf /vote2 ocfs2 _netdev,datavolume,nointr 0 0

/dev/sdk /vote3 ocfs2 _netdev,datavolume,nointr 0 0

/dev/sdl /ocr2 ocfs2 _netdev,datavolume,nointr 0 0

/dev/sdc /ASM-spfile ocfs2 _netdev,datavolume,nointr 0 0

Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem B-1© Copyright 2008, IBM Corporation

Example /etc/fstab File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

configfs on /sys/kernel/config type configfs (rw)ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)/dev/sdd on /vote1 type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)/dev/sde on /ocr1 type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)/dev/sdf on /vote2 type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)/dev/sdk on /vote3 type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)/dev/sdl on /ocr2 type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)/dev/sdc on /ASM-spfile type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)oracleasmfs on /dev/oracleasm type oracleasmfs (rw)

B-2 Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem© Copyright 2008, IBM Corporation

Appendix C
Example /etc/sysctl.conf File
# Kernel sysctl configuration file for Red Hat Linuxnet.ipv4.ip_forward = 0net.ipv4.conf.default.rp_filter = 1net.ipv4.conf.default.accept_source_route = 0kernel.sysrq = 0kernel.core_uses_pid = 1kernel.sem = 250 32000 100 128kernel.shmmax = 536870912net.ipv4.ip_local_port_range = 1024 65000net.core.rmem_default = 1048576net.core.rmem_max = 1048576net.core.wmem_default = 262144net.core.wmem_max = 262144

Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem C-1© Copyright 2008, IBM Corporation

Example /etc/sysctl.conf File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C-2 Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem© Copyright 2008, IBM Corporation

Appendix D
Oracle RAC Troubleshooting
This appendix lists known issues and their resolutions.

One or both hosts restart or lock upOne possible reason for hosts hanging or locking up is that the voting disk drive access time is greater than the o2cb (cluster stack) heartbeat threshold. To resolve this issue, adjust the heartbeat time to a greater value.

1 Observe the current heartbeat threshold by running the following command:

cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold

2 Calculate the value according to this formula:

[fence time in seconds] = (O2CB_HEARTBEAT_THRESHOLD - 1) * 2

3 Set the o2cb heartbeat in /etc/sysconfig/o2cb to a higher value. The test setup uses 200.

4 Restart the hosts (recommended) or unmount ocfs2 logical drives, unload and reload the o2cb stack, and remount logical drives.

5 Make sure that the heartbeat value was changed by repeating step 1.

Kernel panic under heavy I/O loadsUnder heavy I/O loads, the system reports the following error:

Kernel panic - not syncing: ocfs2 is very sorry to be fencing this system by panicking

For instructions for resolving this issue, go to:

http://oss.oracle.com/bugzilla/show_bug.cgi?id=671.

Use deadline I/O scheduler—instead of cfq—until the bug is fixed. To invoke the deadline I/O scheduler, run the applicable command for your operating system.

• On Redhat, run edit /boot/grub/menu.lst.

• On SUSE, run /boot/grub/grub.conf.

Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem D-1© Copyright 2008, IBM Corporation

http://oss.Oracle.com/bugzilla/show_bug.cgi?id=671

Oracle RAC Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Under the title you are loading with, add the elevator=deadline option somewhere on the kernel line.

Reinstalling the Oracle ClusterwareMake sure that before the new installation you delete and then recreate the following items that you are using: vote disk drives, registry disk drives, and any backup vote files or backup registry files. Otherwise, the Clusterware service and the CSS service fail to start and other unpredictable problems occur.

All hosts do not show up in DBCA or ASM creationThe hosts might not appear for two reasons:

• The oraInventory location is being pointed to more than one location. To avoid this issue, always accept the default oraInventory location ($ORACLE_BASE/oraInventory) during Oracle Clusterware installation. You can check the oraInventory location through /etc/oraInst.loc.

• The vipca was not correctly run before clicking OK in the Script Execution window of the Oracle Clusterware installation. During the Oracle Database software installation, choose the Cluster Install option and select All Nodes instead of individually installing each host.

Oracle Clusterware complains that “nodes are not clusterable”

This error message means that either the SSH command or the RSH command (depending on which one you configured) is not configured correctly. Make sure that the SSH command or the RSH command is working correctly for access between all hosts without the use of a password.

ASM library install errorThe ASM driver requires an SMP-enabled kernel. Make sure that your kernel complies with this requirement.

Cannot communicate with Clusterware servicesClusterware takes a few minutes to start up correctly. If the problem persists, stop the Clusterware service and then restart Clusterware service using the following commands:

/etc/init.d/init.crs stop

/etc/init.d/init.crs start

D-2 Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem© Copyright 2008, IBM Corporation

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

If these commands do not resolve this issue, other problems exist with either the pre-installation steps or the installation steps. Check the Oracle documentation. See Appendix E, References.

Error messages during the Oracle Clusterware installationIf messages appear indicating that the commands failed to run on the remote host or that permission was denied, a problem exists with the ownership and permissions on the cluster directories. Make sure that all directories used for RAC are owned by oracle:oinstall and have permissions set to 775.

Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem D-3© Copyright 2008, IBM Corporation

Oracle RAC Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D-4 Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem© Copyright 2008, IBM Corporation

Appendix E
References
http://www.oracle.com/pls/db102/homepage

Oracle Clusterware and Oracle Real Application Clusters Installation Guide, RAC 10g Release 2 (10.2) for Linux

Oracle Database Installation Guide, RAC 10g Release 2 (10.2) for Linux x64

OCFS2 User’s Guide

http://www.oracle.com/technology/tech/linux/asmlib/install.html

http://www.oracle.com/technology/products/database/asm/index.html

http://www.oracle.com/technology/pub/articles/hunter_rac10gr2.html

The IBM Compatibility matrix:

http://www.ibm.com/systems/support/storage/config/ssic/displayesssearchwithoutjs.wss?start_over=yes.

The HBA Settings document is not available on an external link. Customers who want the HBA Settings document should contact Support. To contact Support, see Contact Informationon page 1-30.

Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem E-1© Copyright 2008, IBM Corporation

http://www.Oracle.com/pls/db102/homepage

http://www.Oracle.com/technology/tech/linux/asmlib/install.html

http://www.ibm.com/systems/support/storage/config/ssic/displayesssearchwithoutjs.wss?start_over=yes.



References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

E-2 Deploying Oracle RAC 10g Release 2 on a Linux Server with an IBM Storage Subsystem© Copyright 2008, IBM Corporation

Trademarks and Special Notices © Copyright IBM Corporation 2008. All rights reserved.

References in this document to IBM products or services do not imply that IBM intends to make them available in every country.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.

Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

www.ibm.com/legal/copytrade.shtml

deploying an oracle rac 10g release 2 database with asm on ...€¦ · deploying oracle rac 10g...

Documents