deploying oracle rac 10g r2 and 11g r1 with asm on rhel with ds8000

© Copyright IBM Corporation, 2008. All Rights Reserved. All trademarks or registered trademarks mentioned herein are the property of their respective holders

Deploying Oracle RAC 10g and Oracle RAC 11g with ASM on Red Hat Enterprise

Linux Using IBM System Storage DS8000

Pearl Valiton IBM Systems and Technology Group

Open Systems Lab, San Jose, California

August 2008

Deploying Oracle RAC 10g and 11g with ASM on Red Hat Linux with the IBM System Storage DS8000 Version August 26, 2008 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2008

Table of contents Abstract........................................................................................................................................1 Executive summary ....................................................................................................................1 Introduction .................................................................................................................................2

IBM System Storage DS8000.................................................................................................................. 2 Copy Services.......................................................................................................................................... 2

FlashCopy........................................................................................................................................... 3 Metro Mirror and Global Mirror ........................................................................................................... 3

Oracle technologies....................................................................................................................4 Oracle RAC 10g Release 2 and Oracle Clusterware .............................................................................. 4 Oracle Cluster Synchronization Services ................................................................................................ 4 Oracle Automatic Storage Management (ASM) ...................................................................................... 4

Advantages of ASM technology ......................................................................................................... 5 ASM disk groups................................................................................................................................. 5 ASM failure groups ............................................................................................................................. 5

Linux Device Mapper Multipath .................................................................................................6 Lab test environment..................................................................................................................6

Oracle hardware topology........................................................................................................................ 6 Oracle RAC servers ................................................................................................................................. 7 Storage hardware .................................................................................................................................... 7

Configuration by example ..........................................................................................................8 Configuring DS8000 storage by using DSCLI ......................................................................................... 8 Setting up the storage pool (extent pool)................................................................................................. 8 Creating Storage Volume Group and LUNs ............................................................................................ 9 Creating the host connection and LUN assignments to nodes ............................................................. 10 Configuring Device Mapper Multipath.................................................................................................... 11 Establishing LUN persistence with Multipath......................................................................................... 12 Creating raw devices ............................................................................................................................. 13

Creating raw devices for Oracle RAC 10g Release 2 in RHEL 4..................................................... 13 Creating raw devices for Oracle RAC 10g Release 2 in RHEL 5..................................................... 14

Changing the default permission on the raw and block device ............................................................. 16 Changing the default permission on raw devices for Oracle RAC 10g Release 2 in RHEL 4 ......... 16 Changing the default permission on raw devices for Oracle RAC 10g Release 2 in RHEL 5 ......... 17 Changing the default permission on block devices for Oracle RAC 11g Release 1 in RHEL 5....... 18

Configuring ASM.................................................................................................................................... 19 Understanding Oracle RAC 10g Release 2 installation requirements on RHEL 5................................ 21 Understanding Oracle RAC 11g Release 1 installation requirements on RHEL 5................................ 21

Best practice recommendations..............................................................................................22 Summary....................................................................................................................................22 Resources..................................................................................................................................23 Trademarks and special notices..............................................................................................25


1

Abstract This white paper details the configuration of the IBM System Storage DS8000 series disk storage system in a software environment consisting of Oracle Database 10g or 11g with Oracle Real Application Clusters (RAC) and Oracle Automatic Storage Manager (ASM) on a four-node Red Hat Enterprise Linux cluster.

Executive summary This white paper details the step-by-step configuration of the IBM® System Storage™ DS8000™ with Oracle Database 10g or Database 11g, Oracle Real Application Clusters (RAC), and Oracle Automatic Storage Management (ASM) on the Red Hat Enterprise Linux operating system. The test exercise demonstrates that it is easy to configure the DS8000 in this environment by using the guidelines that are laid out in this document.

The IBM System Storage DS8000 disk storage system is a high-performance storage system that offers expandable data open storage and data-protective systems to correspond with specific business requirements.

The IBM Advanced Copy Services features that are available on the DS8000 include FlashCopy, Metro Mirror, Global Copy, and Global Mirror.

Oracle Database 10g and 11g, and Oracle Clusterware technologies offer several features that work in single instance database and Oracle RAC database environments. Oracle Automatic Storage Management (ASM) technology is a feature that is designed to distribute information uniformly across all storage disk groups and has unique mirroring capabilities.

The Linux® operating system’s Device Mapper Multipath (DM) driver works with the DS8000 disk-storage system in a multipath I/O (MPIO) environment that runs on servers with the x86-64 architecture, including the IBM System x™ platform. If an active path fails, Device Mapper Multipath reroutes the I/O through one of the other active paths.


2

Introduction Enterprise customers running Oracle Real Application Clusters (RAC) database can confidently use the IBM System Storage DS8000 to meet their critical business needs. The DS8000 and the Oracle RAC database help to deliver robust, flexible, highly available and cost-effective disk storage to support continuous operations for mission-critical transactions.

This white paper explains the interoperability of the DS8000, Red Hat Enterprise Linux (RHEL) on x86-64 servers, and Linux Device Mapper Multipath, combined with Oracle RAC features, Oracle Automatic Storage Management (ASM), and Oracle Clusterware. Oracle ASM was introduced in Oracle 10g and provides an integrated file system and volume manager for Oracle database files.

This white paper documents one configuration of Oracle RAC with ASM with DS8000 on a four-node cluster that runs the Red Hat Linux operating system on x86-64 servers.

IBM System Storage DS8000

The DS8000 is member of the IBM System Storage Disk Systems family and provides high capacity and performance using IBM POWER5™ technology. The DS8000 series is a flexible and extendable disk-storage subsystem that is designed to add and adapt to new technologies as they become available. Management tools that are provided with the DS8000 series include DS Storage Manager and the DS command line interface (DS CLI), which allow for management and configuration. The DS8000 series supports high availability environments and provide remote mirror and copy functions to support business continuity. The DS8000 can be configured as RAID-5, RAID-10, or a combination of both. RAID-5 offers excellent performance for many applications, although RAID-10 can offer better performance for selected applications.

Note: For more information about DS8000, visit www.redbooks.ibm.com/abstracts/sg246786.html?Open.

Copy Services

Copy Services are a collection of functions that support disaster recovery, data migration, and data-replication operations. Copy Services run on the DS8000 storage unit and support the UNIX®, Linux, Microsoft® Windows® and System z operating system environments. Many design characteristics and advanced functions of the DS8000 contribute to the protection of data. The DS8000 has a number of advanced Copy Services functions that are part of the IBM System Storage Resiliency family. Copy Services include the following types of functions:

• FlashCopy, a point-in-time copy function

• Remote mirror and copy functions which include: Metro Mirror, Global Copy, and Global Mirror

As mentioned, management of the Copy Services functions is done through the IBM System Storage DS8000 command-line interface (DSCLI) and Web-based interface, called IBM System Storage DS Storage Manager. The DS Storage Manager allows set up and management of data-copy functions from any point where network access is available.


3

FlashCopy

The FlashCopy feature copies the contents of a source disk to a target disk. Any data that exists on the target disk is lost and replaced by the copied data. After the copy operation completes, the target disks contain the contents of the source disks as they existed at a point in time. Both source and target disks can then be updated independently. The FlashCopy feature is sometimes described as an example of a point-in-time copy technology. Although the FlashCopy operation takes time to complete, the resulting data on the target disk is presented so that the copy appears to occur immediately.

FlashCopy operations can occur on multiple source and target disks. FlashCopy management operations are coordinated to allow a common, single point in time for copying target disks from their respective source disks. This allows a consistent copy of data that spans multiple disks.

Metro Mirror and Global Mirror

Metro Mirror and Global Mirror are continuous-replication functions. They operate between DS8000 systems at different locations to help create copies of data for use in the event of a catastrophic data-center event. Metro Mirror is designed to maintain a fully synchronized copy at metropolitan distances (up to 300 km), whereas Global Mirror operates asynchronously and, thus, helps maintain a copy at much greater distances. Because Global Mirror is an asynchronous function, the remote copy is always lagging slightly behind the local copy and some updates are lost in the event of a disaster.

Note: For more details about Copy Services, see the IBM Redbooks® Web site (www.redbooks.ibm.com) to find the following two books: IBM System Storage DS8000 Series: Architecture and Implementation (www.redbooks.ibm.com/abstracts/sg246786.html?Open) and IBM System Storage DS8000: Copy Services in Open Environments (www.redbooks.ibm.com/abstracts/sg246788.html).


4

Oracle technologies This section describes the Oracle RAC 10g Release 2 software and technologies.

Oracle RAC 10g Release 2 and Oracle Clusterware

Oracle RAC 10g Release 2 provides features that can help ease complex Oracle administrative tasks. Some of the key features are Oracle ASM, Oracle Clusterware (including Cluster Ready Services [CRS]), enhanced Recovery Manager (RMAN) features, Flashback technology, performance-tuning enhancements, job scheduler, Data Pump, and Automatic Work Load Repository.

Oracle Cluster Synchronization Services

ASM is designed to work with both single database and Oracle RAC environments. ASM requires that Oracle Cluster Synchronization Services (CSS) be installed and started prior to ASM becoming available. In a single-instance database environment, CSS maintains synchronization between the ASM and database instances. Oracle Cluster Ready Services (CRS) includes the CSS component, which is automatically installed on each node that runs ASM and is started when each node boots.

CSS monitors ASM and the shared-disk storage components, providing cluster and node-monitoring management. Mounted ASM disk groups and ASM itself are registered with CSS upon startup, thereby keeping disk-group metadata in synchronization across all Oracle RAC nodes. Thereby, CSS dynamically registers any new ASM disk groups that are created and broadcast to the other cluster nodes.

Activities in both the database and ASM instances are synchronized using inter-node communication. CSS verifies the health of those ASM instances. Structural changes that require synchronization, such as adding or deleting a disk, initiate the inter-node messages. Efficient synchronization of both ASM and the database are thereby accomplished with the same integrated lock-management infrastructure.

Oracle Automatic Storage Management (ASM)

Automatic Storage Management (ASM) is a feature introduced in Oracle Database 10g. ASM provides a solution for storage-management challenges. ASM has an integrated storage management interface that maintains consistent volumes across all servers and storage platforms. ASM virtualizes storage into disk groups. ASM distributes data evenly across storage resources within disk groups to optimize performance and utilization. ASM provides three mirroring options for protection against disk failure, as shown in Table 1:

Mirror option Mirror description External Defers redundancy to storage device Normal Two-way mirroring High Three-way mirroring

Table 1. Three mirroring options for protection against disk failure

ASM uses a unique mirroring algorithm. ASM does not mirror disks, but rather it mirrors extents. As a result, a hot-spare disk is not required. Only spare capacity is required within the disk group.

If a disk fails, ASM automatically reconstructs the contents of the failed disk on the surviving disks. Reading the mirrored contents from the surviving disks, the I/O content that was hit from a disk failure is spread across several disks, rather than on the single disk that mirrors the failed drive. When ASM allocates a


5

primary extent of a file to one disk in a disk group, it allocates a mirror copy of that extent to another disk in the disk group. Primary extents on a given disk have their respective mirror extents on one of several partner disks in the disk group. Each disk in a disk group has the same ratio of primary and mirror extents.

Advantages of ASM technology

There are many benefits to using ASM technology, including the following:

• Reduction of administration tasks

• Reduction in volume management tasks

• Automation of I/O tuning for workloads

• Simplification of the process for adding and removing disks

• Reduction in downtime as there is no file-system interface for file management

• Automatic balancing of load across disks in the ASM disk group — with the database up and running after the changes to storage capacity

ASM disk groups

To reduce the complexity of managing ASM disk groups, Oracle recommends that generally no more than two disk groups be maintained and managed per Oracle RAC cluster.

• Database Area: The active-database files, control files, online-redo logs and change tracking files that are used in incremental backups are stored in this area

• Flash Recovery Area: Recovery-related files, multiplexed copies of current control files and redo logs, archive logs, backup sets and flashback-log files are stored in this area

ASM failure groups

Defining a failure group within ASM is an additional level of security against disk failure. A failure group is a set of disks in a disk group that share a common resource. ASM ensures that a primary extent and its mirror copy never reside in the same failure group. If you define failure groups for your disk group, ASM can tolerate the simultaneous failure of multiple disks in a single failure group. This allows ASM to mirror across disks on separate storage units to protect against the failure of an entire storage unit. Due to its tight integration with the database, ASM is able to take advantage of database features to recover from I/O failures.


6

Linux Device Mapper Multipath Device Mapper Multipath allows hosts to route I/O over multiple paths that are accessible to the storage unit. A path refers to the connection from a host bus adapter (HBA) port to a storage controller port. If an active path fails, Device Mapper Multipath reroutes the I/O through one of the other active paths. When there are multiple paths to a storage controller, each path appears as a separate block device and, therefore, multiple block devices are created for a single physical LUN. Device Mapper Multipath creates one multiple-block device for that same LUN worldwide name (WWN). For example: Imagine that a host with two HBAs is connected to a two-port storage controller through a single unzoned fibre channel (FC) switch. In this example, four block devices appear on the host: /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd. Device Mapper Multipath creates a single block device, /dev/mpath/mpath1, that reroutes I/O tasks through those four underlying block devices.

Lab test environment The following sections provide the details related to the topology, configuration and software levels for the servers that are used for the example that is discussed in this white paper.

Oracle hardware topology

Figure 1 provides a visual understanding of the Oracle hardware topology that is used for the example in this white paper.

Figure 1. Oracle hardware topology

SAN

eth0 eth0

DS8000

eth0

eth1

eth1

eth1

eth1eth0

eth0

public network

private network

node1

node2 node3 node4


7

Oracle RAC servers

Table 2 shows the Oracle RAC servers that are used for the example presented in this white paper.

Four IBM System x3550 Servers with 4 processors each were configured as Oracle RAC nodes. The Table below summarizes the Red Hat Enterprise Linux (RHEL) version 4 and 5 test environments:

Features Red Hat 4/10g Release 2 Red Hat 5/10g Release 2 Red Hat 5/11g Release 1 Server model IBM System x3550 IBM System x3550 IBM System x3550 Processors Four Four Four RAM 4 GB 4 GB 4 GB Internal disk 2 x 36 GB 2 x 36 GB 2 x 36 GB I/O adapter 2 x 4 Gb Dual Port QLE2462

Firmware version 4.00.23 Driver version 8.01.06

1 x 4 Gb Dual Port QLE2462 Firmware version 1.18 Driver version 8.01.07-k7

1 x 4 Gb Dual Port QLE2462 Firmware version 1.18 Driver version 8.01.07-k7

Operating-system version

Red Hat Linux 2.6.9-42.ELsmp

Red Hat Linux 2.6.18-53.el5

Red Hat Linux 2.6.18-53.1.14.el5

Oracle version

Oracle 10.2.0.3 Database, Oracle Clusterware, Oracle ASM

Oracle 10.2.0.3 Database, Oracle Clusterware, Oracle ASM

Oracle 11.1.0.6 Database , Oracle Clusterware, Oracle ASM

Table 2. Oracle RAC servers

Storage hardware

Table 3 shows that one System Storage DS8000 was used for the Oracle RAC database.

Features Storage model IBM DS8000 2107-921 Storage manager 6.1.2.20070618.1 Storage image IBM.2107-1300351 License machine code 5.2.420.636 RAID type 5 (7+P)

Table 3. Storage hardware


8

Configuration by example This section provides an example of the storage-configuration scenario that is explained in this white paper.

Configuring DS8000 storage by using DSCLI

It is necessary to create 15 devices from the DS8000, IBM.2107-1300351:

• Ten for the Oracle RAC database

• Three for Oracle Clusterware Registry (OCR)

• Two for the Oracle Voting devices

All volumes are shared within the Oracle RAC nodes. Table 4 summarizes the storage infrastructure.

Description LUN size Number of LUNs Volume type Node name

Oracle Database 10 GB 10 ASM node1, node2, node3, node4

Oracle Cluster Registry 1 GB 2 Raw or block node1, node2, node3, node4

Oracle Voting Disk 1 GB 3 Raw or block node1, node2, node3, node4

Table 4. Storage allocation for Oracle RAC

Notes:

• Use the storage DSCLI command to set the storage configuration.

• The raw devices are used for the Oracle CRS and Voting devices with Oracle RAC 10g Release 2.

• The block devices are used for the Oracle CRS and Voting devices with Oracle RAC 11g Release 1.

Setting up the storage pool (extent pool)

Perform the following steps to create the extent pool:

1. Create an array.

dscli> mkarray -dev IBM.2107-1300351-raidtype 5 -arsite IBM.1750-1300351 /S1

2. Create one fixed block from one array.

dscli> mkrank -dev IBM.2107-1300351-array A0 -stgtype fb

3. Create a fixed block storage type extent pool.

dscli> mkextpool –rankgrp 0 -stgtype fb

4. Assign an unassigned rank to an extent pool.

dscli> chrank -extpool p0 r0


9

5. Display a list of array sites and status information. dscli> lsarrysite arsite DA Pair dkcap (10^9B) State Array ================================================================================= IBM.2107-1300351/S1 IBM.2107-1300351/2 146.0 Assigned IBM.2107-1300351/A0

6. Display a list of defined ranks in a storage image and rank status information.

dscli> lsrank ID Group State datastate Array RAIDtype extpoolID stgtype ============================================================================ IBM.2107-1300351/R0 0 Normal Normal IBM.2107-1300351/A0 5 IBM.2107-1300351/P0 fb

7. List the extent pool.

dscli> lsextpool Name ID stgtype rankgrp status availstor (2^30B) %allocated available reserved numvols ================================================================================= Extent Pool IBM.2107-1300351/P0 fb 0 below 48 81 48 0 28

Creating Storage Volume Group and LUNs

Perform the following steps to create a volume group and the LUNs:

1. Create a volume group in a storage image.

dscli> mkvolgrp -type scsimap25 x3550_ora_rhel_ds8k

Note: This creates a volume group named x3550_ora_rhel_ds8k and assigned an identifier as v26.

2. Create an open-systems fixed-block volume in a storage image.

dscli> mkfbvol -extpool p5 -cap 10 -volgrp v26 -name x3550_#h 5100-5109

Note: Allocates ten 10 GB LUNs for the ASM disks.

dscli> mkfbvol -extpool p6 -cap 1 -volgrp v26 -name x3550_#h 1401-1405

Note: This allocates five 1 GB LUNs for the Voting and CRS disks.

3. List the resulting volume group and its members.

dscli> showvolgrp v26 Type SCSI Map 256

Vols 5100 5101 5102 5103 5104 5105 5106 5107 5108 5109 1401 1402 1403 1404 1405

Note: This displays a total of fifteen fixed block volumes in the volume group, v26.


10

Creating the host connection and LUN assignments to nodes

Each node has two dual-port HBAs. Therefore, there are four host connections per node. Perform the following steps to make the host connections:

1. Make an I/O port and host-connect configuration.

dscli> mkhostconnect –wwname 2100001b321753f9 -profile "Intel - Linux RHEL" -volgrp v26 node1_h1 dscli> mkhostconnect –wwname 2101001b323753f9 –profile "Intel - Linux RHEL" -volgrp v26 noed1_h2

:

:

:

: dscli> mkhostconnect -wwname 210000e08b86399c -profile "Intel - Linux RHEL" -volgrp v26 node4_h3 dscli> mkhostconnect -wwname 210100e08ba6399c -profile "Intel - Linux RHEL" -volgrp v26 node4_h4

2. Display a list of host connections.

dscli> lshostconnect node1_h1 IBM.2107-1300351/000C 2100001B321753F9 - Intel - Linux RHEL 0 IBM.2107-1300351/V26 all

node1_h2 IBM.2107-1300351/000D 2101001B323753F9 - Intel - Linux RHEL 0 IBM.2107-1300351/V26 all

:

:

:

:

noed4_h3 IBM.2107-1300351/0033 210000E08B86399C - Intel - Linux RHEL

0 IBM.2107-1300351/V26 all

gxvf9_h4 IBM.2107-1300351/0034 210100E08BA6399C - Intel - Linux RHEL 0 IBM.2107-1300351/V26 all

Note: For details of the DSCLI command, see ibm.com/support/docview.wss?uid=ssg1S7001162&rs=555.


11

Configuring Device Mapper Multipath

Beginning with the Linux 2.6 kernel, Device Mapper and Device Mapper Multipath packages ship with the release. Those packages do not need to be installed separately on the host system. Perform the following steps for the Device Mapper Multipath configuration on all Oracle RAC nodes:

1. Edit the /etc/multipath.conf file to configure device-mapper multipathing.

2. Add a devices section at the end of /etc/multipath.conf. In the devices section, specify the DS8000 storage-array product so that it is recognized in the database of known multipathing devices.

devices { # DS8000 device { vendor “IBM” product “2107900” path_grouping_policy group_by_serial } }

3. Start the Device Mapper Multipath.

$ modprobe dm-multipath $ modprobe dm-round-robin $ service multipathd start $ multipathd –F $ multipathd –v2

4. Run the multipathd command to list which devices are configured as multipath.

$ multipath –ll 36005076303ffc0200000000000005104 [size=10 GB][features="0"][hwhandler="0"] \_ round-robin 0 [active] \_ 2:0:0:4 sdaj 66:48 [active][ready] \_ 2:0:1:4 sday 67:32 [active][ready] \_ 3:0:0:4 sdbn 68:16 [active][ready] \_ 3:0:1:4 sdcc 69:0 [active][ready] \_ 4:0:0:4 sdcr 69:240[active][ready] \_ 4:0:1:4 sddg 70:224[active][ready] \_ 1:0:0:4 sdf 8:80 [active][ready] \_ 1:0:1:4 sdu 65:64 [active][ready] : : : 36005076303ffc0200000000000005108 [size=10 GB][features="1 queue_if_no_path"][hwhandler="0"] \_ round-robin 0 [active] \_ 2:0:0:8 sdan 66:112[active][ready] \_ 2:0:1:8 sdbc 67:96 [active][ready] \_ 3:0:0:8 sdbr 68:80 [active][ready] \_ 3:0:1:8 sdcg 69:64 [active][ready] \_ 4:0:0:8 sdcv 70:48 [active][ready] \_ 4:0:1:8 sddk 71:32 [active][ready] \_ 1:0:0:8 sdj 8:144 [active][ready] \_ 1:0:1:8 sdy 65:128 [active][ready]

Note: For details on Device Mapper Multipath, see http://source.redhat.com/dm/. Documentation is located in /usr/share/doc/device-mapper-multipath-X.X.X (where X.X.X is the version level).


12

Establishing LUN persistence with Multipath

The /dev/mapper directory holds all the multipath device files. They can be used for any file- or block-level I/O operations, such as file system creation and are essentially the same as standard block devices.

New multipath-device names are created by adding multipath subsections to the /etc/multipath.conf file. The file’s alias statement associates a new name to a specific worldwide ID (WWID). Because the aliased name is associated with a WWID, not the /dev/sdX device name, the new name persists after host-system reboots. Table 1 shows the mapping of Oracle Clusterware Registry, Voting and ASM LUNs to their specific alias names.

Description LUN size Number of LUNs Alias name Node name

Oracle ASM Database 10 GB 10 ASM1 - ASM10 node1, node2, node3, node4

Oracle Cluster Registry 1 GB 2 OCR1 - OCR2 node1, node2, node3, node4

Oracle Voting Disk 1 GB 3 VOTE1 - VOTE3 node1, node2, node3, node4

Table 5. New alias names for Oracle RAC devices

Perform the following steps to discover all WWIDs, determine their LUN size, and map the WWID to the new aliases on all Oracle RAC nodes:

1. Run the multipath command to list which devices are configured as multipath.

$ multipath –ll 36005076303ffc0200000000000005104 [size=10 GB][features="0"][hwhandler="0"] \_ round-robin 0 [active] \_ 2:0:0:4 sdaj 66:48 [active][ready] \_ 2:0:1:4 sday 67:32 [active][ready] \_ 3:0:0:4 sdbn 68:16 [active][ready] \_ 3:0:1:4 sdcc 69:0 [active][ready] \_ 4:0:0:4 sdcr 69:240[active][ready] \_ 4:0:1:4 sddg 70:224[active][ready] \_ 1:0:0:4 sdf 8:80 [active][ready] \_ 1:0:1:4 sdu 65:64 [active][ready] : : 36005076303ffc0200000000000005108 [size=10 GB][features="1 queue_if_no_path"][hwhandler="0"] \_ round-robin 0 [active] \_ 2:0:0:8 sdan 66:112[active][ready] \_ 2:0:1:8 sdbc 67:96 [active][ready] \_ 3:0:0:8 sdbr 68:80 [active][ready] \_ 3:0:1:8 sdcg 69:64 [active][ready] \_ 4:0:0:8 sdcv 70:48 [active][ready] \_ 4:0:1:8 sddk 71:32 [active][ready] \_ 1:0:0:8 sdj 8:144 [active][ready] \_ 1:0:1:8 sdy 65:128 [active][ready]


13

2. Use the output from step 1 that determines which WWID is associated with a specific LUN size to assign a new alias name to the specific WWID for the Oracle Clusterware Registry, Voting and ASM LUNs based on Table 5. The following example associates a new name, OCR_1, to a LUN with a WWID of 36005076303ffc0200000000000001401:

multipaths { multipath { wwid 36005076303ffc0200000000000001401 alias OCR_1 path_grouping_policy group_by_serial path_checker tur path_selector “round-robin 0” failback immediate rr_weight priorities no_path_retry 5 } multipath { ….. For other target } multipath { ….. For other target } }

3. Start the Device Mapper Multipath daemon.

$ service multipathd stop $ service multipathd start $ multipathd –F $ multipathd –v2

Creating raw devices

Note: Skip this section if you are installing Oracle RAC 11g.

Beginning with Oracle RAC 11g, the Oracle installer supports the block device for the Oracle Registry and Voting devices. The block devices are used for the Oracle RAC 11g installation.

Creating raw devices for Oracle RAC 10g Release 2 in RHEL 4

Skip this section if the operating system is RHEL 5. LUN persistence with multipath for the raw devices is implemented by mapping them in the /etc/sysconfig/rawdevices file to files with a character major number (currently 162) under the /dev/raw directory. The raw devices are used for the Oracle CRS and Voting devices. All Oracle RAC nodes must use the same raw devices. Table 6 shows the raw device mappings for the Oracle CRS and Voting devices.

Description Alias name Raw Node name

Oracle Cluster Registry OCR_1 raw101 node1, node2, node3, node4


Oracle Voting Disk VOTE_1 raw103 node1, node2, node3, node4



Table 6. Raw device for Oracle RAC devices in RHEL 4


14

Perform the following steps for the raw-device configuration on all Oracle RAC nodes:

1. Create the raw devices and add the following lines to the /etc/sysconfig/rawdevices file:

/dev/raw/raw101 /dev/mapper/OCR_1 /dev/raw/raw102 /dev/mapper/OCR_2 /dev/raw/raw103 /dev/mapper/VOTE_1 /dev/raw/raw104 /dev/mapper/VOTE_2 /dev/raw/raw105 /dev/mapper/VOTE_3

2. Activate the rawdevices file without a reboot.

$ service rawdevices restart

3. Query every raw device to ensure all Oracle RAC Clusterware Registry and Voting raw devices are bound properly.

$ /usr/bin/raw –qa /dev/raw/raw101: bound to major 253, minor 2 /dev/raw/raw102: bound to major 253, minor 3 /dev/raw/raw103: bound to major 253, minor 4 /dev/raw/raw104: bound to major 253, minor 5 /dev/raw/raw105: bound to major 253, minor 6

4. List the new raw devices.

$ ll /dev/raw crw-r--r-- 1 root root 162, 101 Aug 8 14:22 /dev/raw/raw101 crw-r--r-- 1 root root 162, 102 Aug 8 14:22 /dev/raw/raw102 crw-r--r-- 1 root root 162, 103 Aug 8 14:22 /dev/raw/raw103 crw-r--r-- 1 root root 162, 104 Aug 8 14:22 /dev/raw/raw104 crw-r--r-- 1 root root 162, 105 Aug 8 14:22 /dev/raw/raw105

Creating raw devices for Oracle RAC 10g Release 2 in RHEL 5

Beginning with RHEL 5, the method to create raw devices has changed. In RHEL 5, the method to create raw devices uses udev to add custom rules. The udev facility is a standard method to provide persistent file naming for creating character special files and raw devices. The raw devices are used for the Oracle Cluster Registry and Voting devices. All RAC nodes must use the same raw devices. Table 7 lists the devices that need to be created as raw devices.

Description Alias name Raw Node name






Table 7. Raw device for Oracle RAC devices in Red Hat 5

Note: All devices that are listed in Table 7 need to be partitioned by using the fdisk command prior to the binding of raw devices for Oracle 10g Release 2 Clusterware installation.


15

Perform the following steps for the raw-device configuration on all Oracle RAC nodes:

1. List the OCR and VOTE partition-device names. (Those names will be bound to raw devices.)

% /sbin/dmsetup ls --target linear --exec /bin/basename | egrep “OCR|VOTE”

VOTE_1p1 VOTE_2p1 VOTE_3p1 OCR_1p1 OCR_2p1

2. Create the /etc/udev/rules.d/10-oracle-raw.rules file and add the following lines from step 1:

# Raw bind to Oracle Clusterware devices ACTION=="add", KERNEL=="dm-[0-9]*", RUN+="/bin/bash -c '/sbin/dmsetup info -c --noheadings -o name -j %M -m %m | grep ^OCR_1p1 && /bin/raw /dev/raw/raw501 /dev/%k'" ACTION=="add", KERNEL=="dm-[0-9]*", RUN+="/bin/bash -c '/sbin/dmsetup info -c --noheadings -o name -j %M -m %m | grep ^OCR_2p1 && /bin/raw /dev/raw/raw502 /dev/%k'" ACTION=="add", KERNEL=="dm-[0-9]*", RUN+="/bin/bash -c '/sbin/dmsetup info -c --noheadings -o name -j %M -m %m | grep ^VOTE_1p1 && /bin/raw /dev/raw/raw503 /dev/%k'" ACTION=="add", KERNEL=="dm-[0-9]*", RUN+="/bin/bash -c '/sbin/dmsetup info -c --noheadings -o name -j %M -m %m | grep ^VOTE_2p1 && /bin/raw /dev/raw/raw504 /dev/%k'" ACTION=="add", KERNEL=="dm-[0-9]*", RUN+="/bin/bash -c '/sbin/dmsetup info -c --noheadings -o name -j %M -m %m | grep ^VOTE_3p1 && /bin/raw /dev/raw/raw505 /dev/%k'"

3. Reactivate the udev facility without a reboot.

% /sbin/start_udev

4. Query all raw devices to ensure that all Oracle RAC Clusterware Registry and Voting raw devices are bound properly.

% /bin/raw –qa /dev/raw/raw501: bound to major 253, minor 48 /dev/raw/raw502: bound to major 253, minor 80 /dev/raw/raw503: bound to major 253, minor 67 /dev/raw/raw504: bound to major 253, minor 63 /dev/raw/raw505: bound to major 253, minor 60

5. List all the new raw devices.

% ll /dev/raw crw-r----- 1 root root 162, 501 Mar 13 06:46 raw501 crw-r----- 1 root root 162, 502 Mar 13 06:46 raw502 crw-r----- 1 root root 162, 503 Mar 13 06:46 raw503 crw-r----- 1 root root 162, 504 Mar 13 06:46 raw504 crw-r----- 1 root root 162, 505 Mar 13 06:46 raw505


16

Changing the default permission on the raw and block device

This section of the paper explains how to change the default permissions on raw and block devices.

Changing the default permission on raw devices for Oracle RAC 10g Release 2 in RHEL 4

Note: Skip this section if the operating system is Red Hat 5 Linux.

The udev facility dynamically creates device files under the /dev directory in Red Hat 4. Manual changes to the permission, user or group IDs to a device file under /dev are lost after a reboot. To prevent this, edit the /etc/udev/permissions.d/50-udev.permissions file to change the permission, user or group IDs. Perform the following steps to change the default permission of a device on all Oracle RAC nodes:

1. List all raw devices.

$ ll /dev/raw crw-r--r-- 1 root root 162, 101 Aug 8 14:22 /dev/raw/raw101 crw-r--r-- 1 root root 162, 102 Aug 8 14:22 /dev/raw/raw102 crw-r--r-- 1 root root 162, 103 Aug 8 14:22 /dev/raw/raw103 crw-r--r-- 1 root root 162, 104 Aug 8 14:22 /dev/raw/raw104 crw-r--r-- 1 root root 162, 105 Aug 8 14:22 /dev/raw/raw105

2. Add the Oracle Registry and Voting raw devices under the raw devices section in the /etc/udev/permissions.d/50-udev.permissions file.

# raw devices ram*:root:disk:0660 raw/raw101:root:g502:0640 raw/raw102:root:g502:0640 raw/raw103:oracle:g502:0660 raw/raw104:oracle:g502:0660 raw/raw105:oracle:g502:0660 raw/*:root:disk:0660

3. Activate the 50-udev.permissions file without a reboot.

% /sbin/start_udev


$ ll /dev/raw crw-r----- 1 root g502 162, 101 Aug 8 14:25 /dev/raw/raw101 crw-r----- 1 root g502 162, 102 Aug 8 14:25 /dev/raw/raw102 crw-rw---- 1 oracle g502 162, 103 Aug 8 14:25 /dev/raw/raw103 crw-rw---- 1 oracle g502 162, 104 Aug 8 14:25 /dev/raw/raw104 crw-rw---- 1 oracle g502 162, 105 Aug 8 14:25 /dev/raw/raw105


17

Changing the default permission on raw devices for Oracle RAC 10g Release 2 in RHEL 5

Device files under the /dev directory are created dynamically by udev in Red Hat 5. Manual changes to the permission, user and group IDs to a device file under /dev are lost after a reboot. To prevent this, create the /etc/udev/rules.d/65-raw-permissions.rules file to change the permission, user and group IDs. Perform the following steps to change the default permission of the devices on all Oracle RAC nodes:


% ll /dev/raw crw-r----- 1 root root 162, 501 Mar 13 06:46 raw501 crw-r----- 1 root root 162, 502 Mar 13 06:46 raw502 crw-rw---- 1 root root 162, 503 Mar 13 06:46 raw503 crw-rw---- 1 root root 162, 504 Mar 13 06:46 raw504 crw-rw---- 1 root root 162, 505 Mar 13 06:46 raw505

2. Create the /etc/udev/rules.d/65-raw-permissions.rules file and add the Oracle Registry and Voting raw devices.

# Set permissions of raw bindings to Oracle Clusterware devices KERNEL=="raw501", OWNER="root", GROUP="oinstall", MODE="640" KERNEL=="raw502", OWNER="root", GROUP="oinstall", MODE="640" KERNEL=="raw503", OWNER="oracle", GROUP="oinstall", MODE="660" KERNEL=="raw504", OWNER="oracle", GROUP="oinstall", MODE="660" KERNEL=="raw505", OWNER="oracle", GROUP="oinstall", MODE="660"

3. Reactivate the udev without a reboot.

$ /sbin/start_udev


$ ll /dev/raw crw-r----- 1 root oinstall 162, 501 Mar 13 06:46 raw501 crw-r----- 1 root oinstall 162, 502 Mar 13 06:46 raw502 crw-rw---- 1 oracle oinstall 162, 503 Mar 13 06:46 raw503 crw-rw---- 1 oracle oinstall 162, 504 Mar 13 06:46 raw504 crw-rw---- 1 oracle oinstall 162, 505 Mar 13 06:46 raw505


18

Changing the default permission on block devices for Oracle RAC 11g Release 1 in RHEL 5

The udev facility dynamically creates device files in Red Hat 5. Manual changes to the permission, user and group IDs to a device file under /dev are lost after a reboot. To prevent this, create the /etc/udev/rules.d/99-oracle-permissions.rules file to change the permission, user and group IDs. Perform the following steps to change the default permission of the devices on all Oracle RAC nodes:

1. List the OCR and VOTE block devices.

% ll /dev/mapper | egrep “OCR*|VOTE*” brw-rw---- 1 root disk 253, 21 May 9 16:27 OCR_1 brw-rw---- 1 root disk 253, 28 May 9 16:27 OCR_2 brw-rw---- 1 root disk 253, 32 May 9 16:27 VOTE_1 brw-rw---- 1 root disk 253, 30 May 9 16:27 VOTE_2 brw-rw---- 1 root disk 253, 29 May 9 16:27 VOTE_3

2. Create the /etc/udev/rules.d/99-oracle-permissions.rules file and add the Oracle Registry and Voting block devices.

# Set permissions of block bindings to Oracle Clusterware devices PROGRAM="/bin/chown root:oinstall /dev/mapper/OCR_1" PROGRAM="/bin/chown root:oinstall /dev/mapper/OCR_2" PROGRAM="/bin/chown oracle:oinstall /dev/mapper/VOTE_1" PROGRAM="/bin/chown oracle:oinstall /dev/mapper/VOTE_2" PROGRAM="/bin/chown oracle:oinstall /dev/mapper/VOTE_3" PROGRAM="/bin/chmod 640 /dev/mapper/OCR_1" PROGRAM="/bin/chmod 640 /dev/mapper/OCR_2" PROGRAM="/bin/chmod 660 /dev/mapper/VOTE_1" PROGRAM="/bin/chmod 660 /dev/mapper/VOTE_2" PROGRAM="/bin/chmod 660 /dev/mapper/VOTE_3"

3. Reactivate the udev without a reboot.

$ /sbin/start_udev

4. List the OCR and VOTE block devices.

$ ll /dev/mapper | egrep “OCR*|VOTE*” brw-r----- 1 root oinstall 253, 21 May 9 16:27 OCR_1 brw-r----- 1 root oinstall 253, 22 May 9 16:27 OCR_2 brw-rw---- 1 oracle oinstall 253, 23 May 9 16:27 VOTE_1 brw-rw---- 1 oracle oinstall 253, 24 May 9 16:27 VOTE_2 brw-rw---- 1 oracle oinstall 253, 26 May 9 16:27 VOTE_3


19

Configuring ASM

The Oracle ASM Library software is installed on all Oracle RAC nodes. Therefore, you need to perform the following steps for the ASM configuration on all Oracle RAC nodes.

All ASM devices, ASM1 - ASM9, need to be partitioned by using the fdisk command prior to the ASMLib installation.

1. Download the Oracle ASMLib-related installation files that correspond to the proper Ret Hat version from the URL www.oracle.com/technology/tech/linux/asmlib/index.html.

2. Install the ASMLib related files.

3. Configure the ASM Library.

$ /etc/init.d/oracleasm configure

4. List the ASM partition-device names.

% /sbin/dmsetup ls --target linear --exec /bin/basename | grep ASM ASM1p1 ASM2p1 ASM3p1 ASM4p1 ASM5p1 ASM6p1 ASM7p1 ASM8p1 ASM9p1

5. Configure the ASM disk group for the database. The ASM disk group uses the ASM1 – ASM9 devices, as shown below. Perform these steps on only one node.

$ /etc/init.d/oracleasm createdisk ASM1p1 /dev/mapper/ASM1p1 Marking disk “/dev/mapper/ASM_1” as an ASM disk: [ OK ]

$ /etc/init.d/oracleasm createdisk ASM2p1 /dev/mapper/ASM2p1

Marking disk “/dev/mapper/ASM_2” as an ASM disk: [ OK ] : : :

$ /etc/init.d/oracleasm createdisk ASM8p1 /dev/mapper/ASM8p1 Marking disk “/dev/mapper/ASM8p1” as an ASM disk: [ OK ]

$ /etc/init.d/oracleasm createdisk ASM9p1 /dev/mapper/ASM9p1

Marking disk “/dev/mapper/ASM9p1” as an ASM disk: [ OK ]


20

6. Activate, scan and list the ASM disk devices to ensure that the disks in the ASM disk group are available on all Oracle RAC nodes.

$ /etc/init.d/oracleasm disable $ /etc/init.d/oracleasm enable $ /etc/init.d/oracleasm scandisks $ /etc/init.d/oracleasm listdisks

Note: For details about installing ASMLib, see www.oracle.com/technology/tech/linux/asmlib/install.html.

7. Modify the /etc/sysconfig/oracleasm file to configure ASMLib so that it discovers multipath devices and excludes all single-path devices on every Oracle RAC node. (See Table 8.)

Oracle ASM parameters

ORACLEASM_ENABLED=true

ORACLEASM_UID=oracle

ORACLEASM_GID=oinstall

ORACLEASM_CLEARBOOT=true

ORACLEASM_SCANBOOT=true

ORACLEASM_SCANORDER=dm

ORACLEASM_SCANEXCLUDE=sd

Table 8. Oracle ASM parameters

Note: For information about configuring Oracle ASMLib on Multipath Devices, see www.oracle.com/technology/tech/linux/asmlib/multipath.html.


21

Understanding Oracle RAC 10g Release 2 installation requirements on RHEL 5

This section only applies to the Red Hat Enterprise Linux 5 operating system. All users must review the following lists from http://metalink.oracle.com prior to installing Oracle 10g Release 2, version 10.2.0.1.0.

1. Verify the required RPM packages are installed on the system prior to installing Oracle RAC 10g Release 2.

Note: For information about required RPM packages, visit: http://metalink.oracle.com and read NOTE:421308.1.

2. The prerequisite check fails when installing Oracle RAC 10g Release 2 on RHEL 5.

Note: For information about checking the prerequisites when installing Oracle RAC 10g Release 2, see http://metalink.oracle.com, NOTE:456634.1.

3. Running the root.sh script on the first Oracle RAC node results in the following error: Failed to Upgrade Oracle Cluster Registry Configuration.

Note: For information about failing to upgrade the Oracle Cluster Registry when installing Oracle RAC 10g Release 2 Clusterware, see http://metalink.oracle.com, NOTE:466673.1.

4. The vipca execution fails at the end of the root.sh script on the last Oracle RAC node.

Note: For information about failing to run vipca when installing Oracle RAC 10g Release 2 Clusterware, see http://metalink.oracle.com, NOTE:414163.1.

All users must review the following list prior to installing Oracle 10g Release 2, version 10.2.0.3.0.

5. The Enterprise manager configuration fails, resulting in the following error: oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-06502: PL/SQL: numeric or value error: character string buffer too small.

Note: For information about the character string buffer too small error when configuring Enterprise manager, see www.oakdene.net/blog/2008/02/13/some-issues-installing-oem-for-oracle-102 and http://metalink.oracle.com, NOTE:278100.1.

Understanding Oracle RAC 11g Release 1 installation requirements on RHEL 5

This section only applies to the RHEL 5 operating system. All users must review the following lists from http://metalink.oracle.com prior to installing Oracle 11g Release 1, version 11.1.0.6.0.

1. Verify that the required RPM packages are installed on the system prior to installing Oracle 11g Release 1.

Note: For information about required RPM packages, visit http://metalink.oracle.com, NOTE:438766.1.


22

Best practice recommendations Here is a list of best practices that you should follow:

• Use multiple LUNs with equal capacity, performance and characteristics within an ASM disk group

• Use multiple LUNs per ASM disk group to evenly distribute database read and write tasks across members of the disk group

• Use LUNs with equal capacity and performance when adding LUNs to increase the disk space for an ASM disk group

• Configure ASM disk groups to use external redundancy

• Place data files and log files in same disk group

• Configure ASM disk groups to use external redundancy

• Use Device Mapper Multipathing for high availability

ASM simplifies database file management on System Storage disk products. External redundancy allows the host to use more processor resources for data management and less processor resources for storage management. Oracle ASM and IBM DS8000 work together to enhance database administrator and system administrator productivity.

Summary The use of Oracle ASM with the IBM System Storage DS8000 can help greatly simplify storage administration. ASM manages the workload of balancing I/O within the ASM disk group, making the addition or removal of physical storage possible without having to shut down the Oracle database. ASM also takes advantage of existing DS8000 fault-tolerance mechanisms for increased data protection.


23

Resources These Web sites provide useful references to supplement the information contained in this document:

• IBM Publications Center www.elink.ibmlink.ibm.com/public/applications/publications/cgibin/pbi.cgi?CTY=US

• IBM Redbooks www.redbooks.ibm.com

DS8000

• The IBM System Storage DS8000 Series: Architecture and Implementation: www.redbooks.ibm.com/abstracts/sg246786.html?Open

• IBM System Storage DS8000 Series: Copy Services in Open Environments: www.redbooks.ibm.com/abstracts/sg246788.html

• IBM TotalStorage DS8000 Series: Performance Monitoring and Tuning: www.redbooks.ibm.com/redbooks/pdfs/sg247146.pdf

• IBM System Storage DS8000 Command-line Interface User’s Guide: ibm.com/support/docview.wss?uid=ssg1S7001162&rs=555

• The IBM System Storage DS8000 Information Center: http://publib.boulder.ibm.com/infocenter/dsichelp/ds8000ic/index.jsp

Device Mapper

• For information about Device Mapper Multipath http://source.redhat.com/dm

• QLogic Driver Downloads and Manuals http://support.qlogic.com/support/drivers_software.aspx

Oracle

• Oracle Database Oracle Clusterware and Oracle Real Application Clusters Installation Guide 10g Release 2 (B14201-04) http://download.oracle.com/docs/cd/B19306_01/install.102/b14203/toc.htm

• Oracle® Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide 10g Release 2 (10.2) Part Number B14197-08 http://download.oracle.com/docs/cd/B19306_01/rac.102/b14197.pdf

• Oracle Database10g Release 2 New Features www.oracle.com/technology/products/database/asm/pdf/asm%20r2%20new%20features.pdf

• ASM Best practices www.oracle.com/technology/products/database/asm/pdf/asm_10gr2_bestpractices%2005-07.pdf

• Oracle Database 10g Release 2 Automatic Storage Management Overview www.oracle.com/technology/products/database/asm/pdf/asm_10gr2_bptwp_sept05.pdf


24

• Overview of Automatic Storage Management www.oracle.com/technology/products/manageability/database/pdf/asmov.pdf

• Automatic Storage Management Technical Overview: www.oracle.com/technology/products/manageability/database/pdf/asmwp.pdf

• Oracle Clusterware Installation Guide 11g Release 1 for Linux (B28263-04) http://download.oracle.com/docs/cd/B28359_01/install.111/b28263.pdf

• Oracle Real Application Clusters Installation Guide 11g Release 1 for Linux and UNIX (B28264-04) http://download.oracle.com/docs/cd/B28359_01/install.111/b28264.pdf

• Oracle Database 2 Day + Real Application Clusters Guide 11g Release 1 (B28252-03) http://download.oracle.com/docs/cd/B28359_01/rac.111/b28252.pdf

• Oracle Real Application Clusters Administration and Deployment Guide 11g Release 1 (B28254-04) http://download.oracle.com/docs/cd/B28359_01/rac.111/b28254.pdf

• Oracle Database11g Automatic Storage Management New Features Overview www.oracle.com/technology/products/database/asm/pdf/11gr1%20asm%20new%20features%20wp%2005-2007.pdf

• Oracle ASMLib www.oracle.com/technology/tech/linux/asmlib/index.html

• Taking the Guesswork out of database tuning www.oracle.com/technology/products/database/asm/pdf/take%20the%20guesswork%20out%20of%20db%20tuning%2001-06.pdf


25

Trademarks and special notices © Copyright IBM Corporation 2008. All rights Reserved.

References in this document to IBM products or services do not imply that IBM intends to make them available in every country.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

deploying oracle rac 10g r2 and 11g r1 with asm on rhel with ds8000

Documents