2512680 cookbook 11grac r1 asm aix5l san storage installation guide

8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide

1/393

DRAFT

COOKBOOK

I ns ta l l a t i on G u i deOracle 11g RAC Release 1with Oracle AutomaticStorage Management (ASM)on IBM System p


2/393

This document is based on our experiences.

This is not an official (Oracle or IBM) documentation.

This document will be constantly updated

and were open to any add-on

or feedback from your own experiences,on same or different storage solution !!!

Document history :

Version Date Update Who Validated by

1.0 January 2008 - Creation Frederic MichiaraThierry Plumeau

Alain RoyPaul Bramy

April 2008 - Review Dider WojciechowskiPaul Bramy

C t ib t


3/393

1 The aim of this document ............................................... ......................... 62 About Oracle clusterware ................................................. ....................... 8

3 About Oracle Automatic storage management ........................................ 154 About Real Application Cluster ............................................................. .. 175 About 11G RAC / ASM on AIX .......................................................... ........ 196 Whats new with 11g RAC implementation on AIX ................................... 207 infrastructure requirements .................................... .............................. 20

7.1 General Requirements ..................................................................................................... ...............227.1.1 About Servers and processors ................................................................................. ................227.1.2 About RAC on IBM System p ............................................................................ ........................23

7.1.3 About Network ........................................................................................................... ..............247.1.4 About SAN Storage ................................................................................................. .................267.1.5 Proposed infrastructure with 2 servers ................................................................... .................287.1.6 What do we protect ? ....................................................................................................... ........297.1.7 About IBM Advanced Power Virtualization and RAC .................................................... .............30

7.2 Cookbook infrastructure ...................................................................................... ...........................357.2.1 IBM Sytem p servers .................................................................................................... ............367.2.2 Operating System ............................................................................................................ ........447.2.3 Multi-pathing and ASM ............................................................................ ................................45

7.2.4 IBM storage and multi-pathing ........................................................................ ........................467.2.5 EMC storage and multi-pathing ........................................................................... ....................537.2.6 HITACHI storage and multi-pathing ........................................................... ..............................557.2.7 Others, StorageTek, HP EVA storage and multi-pathing ....................................................... ....57

8 Specific cOnsiderations for RAC/ASM setup with HACMP installed ........... 599 Installation steps ........................................................ .......................... 6210 Preparing the system ............................................... ........................... 64

10.1 Network configuration ........................................................................................ ..........................65

10 1 1 Define Networks layout Public Virtual and Private Hostnames 65


4/393

11.2.9 Formating the virtual devices (zeroing) .................................................................... ...........12211.3 ASM disks .................................................................................................... ...............................123

11.3.1 Required LUNs ......................................................................................... ...........................12511.3.2 How to Identify if a LUN is used or not ? ........................................................... ...................12511.3.3 Register LUNs at AIX level.............................................................................................. .....12711.3.4 Identify LUNs and corresponding hdisk on each node ...................................... ..................12911.3.5 Removing reserve lock policy on hdisks from each node ................................. ...................13211.3.6 Identify Major and Minor number of hdisk on each node .................................................... .13411.3.7 Create Unique Virtual Device to access same LUN from each node ....................................13611.3.8 Set Ownership / Permissions on Virtual Devices ........................................................ ..........13811.3.9 Formating the virtual devices (zeroing) .................................................................... ...........141

11.3.10 Removing assigned PVID on hdisk ................................................................................. ....14211.4 Checking Shared Devices ..................................................................................... ......................14311.5 Recommandations, hints and tips ................................................................ ..............................145

11.5.1 OCR / Voting disks .................................................................................. .............................14511.5.2 ASM disks ................................................................................................ ............................148

12 Oracle Clusterware (CRS) Installation ................................................ 15112.1 Cluster VeriFication utility ............................................................................................ ...............152

12.1.1 Understanding and Using Cluster Verification Utility ................................................... ........15212.1.2 Using CVU to Determine if Installation Prerequisites are Complete ............................ .........152

12.2 Installation ....................................................................................................................... ...........16312.3 Post Installation operations ............................................................................. ...............................4

12.3.1 Update the Clusterware unix user .profile ........................................................ .......................412.3.2 Verify parameter CSS misscount .................................................................. ...........................512.3.3 Cluster Ready Services Health Check ....................................................................... ...............612.3.4 Adding enhanced crsstat script ....................................................................... .......................912.3.5 Interconnect Network configuration Checkup ................................................................ ........1112.3.6 Oracle CLuster Registry content Check and Backup .............................................. ................12

12.4 Some usefull commands ................................................................................. .............................19

12 5 Accessing CRS logs 21


5/393

15.7 About DBCONSOLE ........................................................................................... ..........................16715.7.1 Checking DB CONSOLE ........................................................................................... .............16715.7.2 Moving from dbconsole to Grid Control.......................................................... .....................171

16 asm advanced tools ............................................................. .............. 17216.1 ftp and http access .................................................................................................. ...................172

17 Some usefull commands ............................. ....................................... 17317.1 Oracle CLuster Registry content Check and Backup ................................................. ..................174

18 Appendix A : Oracle / IBM technical documents ................................... 17519 Appendix B : Oracle technical notes .................................................. .. 176

19.1 CRS and 10g Real Application Clusters ................................................................................... ....17619.2 About RAC ....................................................................................................................... ...........186

19.3 About CRS .......................................................................................................................... ........18619.4 About VIP ............................................................................................................. ......................18719.5 About manual database cration ................................................................................................ .18719.6 About Grid Control .................................................................................................. ...................18719.7 About TAF .................................................................................................... ..............................18719.8 About Adding/Removing Node ..................................................................................... ..............18719.9 About ASM .......................................................................................................... .......................18719.10 Metalink note to use in case of problem with CRS .................................................... ..............188

20 Appendix C : Usefull commands ...................................................... .... 18921 Appendix D : Empty tables to use for installatioN ................................ 19921.1 Network document to ease your installation ......................................................... .....................19921.2 Steps Checkout .................................................................................................................. .........20121.3 Disks document to ease disks preparation in your implementation .............................. .............202

22 Documents, BOOKS to look at .. ........................................................ .. 203


6/393

1 THE AIM OF THIS DOCUMENT

This document is written to provide help installingOracle11gReal Application Clusters (11.1) release 1 withOracle Automated Storage Management on IBM System p

and i servers with AIX.

Within the Oracle/IBM Joint Solutions Center, we are receiving a lot of request about RAC and ASM, this isthe reason why we decided to deliver the first 11gRAC / AIX cookbook with Oracle ASM (AutomatedStorage Management). AIX6 is not covered as 11gRAC is not yet supported with AIX6, but certificationshould be available soon. Check on metalink for update, the cookbook will be updated with AIX6 whenofficial certification will be released.

We will describe step by step the architecture Oracle CRS (Cluster Ready Service) on raw disks and database on ASM(Automatic Storage Management).

For the architecture using Oracle 11gRAC with IBM GPFS as cluster files system, cookbook named InstallationGuideOracle 11gRAC Release 1 and IBM GPFS on IBM eServer System p and i running AIX with SAN Storage will be released soon.

M t li k (htt // t li k l / t li k/ l l/ l2 i t t )
http://metalink.oracle.com/metalink/plsql/ml2_gui.startuphttp://metalink.oracle.com/metalink/plsql/ml2_gui.startuphttp://metalink.oracle.com/metalink/plsql/ml2_gui.startup


7/393

The information contained in this paper resulted from :

-Oracle and IBM documentations

-Workshop experiences done in the Oracle/IBM Joint Solutions Center

-Benchmarks and POC implementations for customers performed by PSSC Montpellier

-This documentation is a joint effort from Oracle and IBM specialists.

Please also refer to Oracle & IBM online documentation for more information :

http://docs.oracle.com

o Oracle Database 11gRelease 1 (11.1) Documentationhttp://www.oracle.com/technology/documentation/database.html

http://tahiti.oracle.com

Oracle RAC home page : http://www.oracle.com/database/rac_home.html

For more information IBM System p : http://www-03.ibm.com/systems/p/

Your comments are important for us, and we thanks the ones who send us their feedback about previousrelease, and about how this document did help them in their implementation. We want our technical papers tobe as helpful as possible.

Please send us your comments about this document to the Oracle/IBM Joint Solutions Center.

U il dd O h b
http://docs.oracle.com/http://www.oracle.com/technology/documentation/database.htmlhttp://www.oracle.com/database/rac_home.htmlhttp://www-03.ibm.com/systems/p/http://docs.oracle.com/http://www.oracle.com/technology/documentation/database.htmlhttp://www.oracle.com/database/rac_home.htmlhttp://www-03.ibm.com/systems/p/


8/393

2 ABOUT ORACLE CLUSTERWARE

Extract from : Oracle Database 2 Day + Real Application Clusters Guide 11gRelease 1 (11.1)Part Number B28252-02http://www.oracle.com/pls/db111/homepage?remark=tahiti

Oracle Real Application Clusters (Oracle RAC) uses Oracle Clusterware as the infrastructure that binds togethermultiple nodes that then operate as a single server. Oracle Clusterware is a portable cluster management solution thatis integrated with Oracle Database. In an Oracle RAC environment, Oracle Clusterware monitors all Oraclecomponents (such as instances and listeners). If a failure occurs, Oracle Clusterware automatically attempts to restartthe failed component and also redirects operations to a surviving component.

Oracle Clusterware includes a high availability framework for managing any application that runs on your cluster.Oracle Clusterware manages applications to ensure they start when the system starts. Oracle Clusterware alsomonitors the applications to make sure that they are always available. For example, if an application process fails, thenOracle Clusterware attempts to restart the process based on scripts that you customize. If a node in the cluster fails,then youcan program application processes that typically run on the failed node to restart on another node in thecluster.

Oracle Clusterware includes two important components: the voting disk and the OCR. The voting disk is a file thatmanages information about node membership, and the OCR is a file that manages cluster and Oracle RAC database

configuration information.

The Oracle Clusterware installation process creates the voting disk and the OCR on shared storage. If you select theoption for normal redundant copies during the installation process, then Oracle Clusterware automatically maintainsredundant copies of these files to prevent the files from becoming single points of failure. The normal redundancyfeature also eliminates the need for third-party storage redundancy solutions. When you use normal redundancy,Oracle Clusterware automatically maintains two copies of the OCR file and three copies of the voting disk file.

Y ill l b t th ti f th O l l t h t b ild th l t d th t t f
http://www.oracle.com/pls/db111/homepage?remark=tahitihttp://www.oracle.com/pls/db111/homepage?remark=tahiti


9/393

Oracle Clusterware, Group Membership and Heartbeats

Cluster needs to know who is a member at all times

Oracle Clusterware has 2 heartbeats

Network heartbeat

If a node does not send a heartbeat forMissCount (time in seconds), then node isevicted from cluster.

Disk heartbeat

If disk heartbeat is not updated in I/Otimeout, then node is evicted from cluster

Oracle Clusterware, OCR and Voting disks

Oracle Cluster Registry contains allinformation on the cluster.

Since 10gRAC R2 OCR can bemirrored

Voting is the cluster heartbeat.

Since 10gRAC R2 voting can have

multiple copies in odd number formajority.

11gRAC/ASM/AIX [email protected] 9 of 393


10/393

Oracle Clusterware and its virtual IPs

- 1 -

Each node participating to the Oracle Cluster has :

1 Static IP address (10.3.25.81)

1 Virtual IP address, each virtual address(10.3.25.181) has its home node

- 2 -

If one node fails :

Static IP (10.3.25.81) from failed node wil notbe anymore reachable

Virtual IP (10.3.25.181) from failed node willswitch to one of the remaining node, and willremain reachable

- 3 -

When failed node return to normal operation :

Static IP (10.3.25.81) will be back andreachable

Virtual IP (10.3.25.181) from failed node,hosted on one of the remaining nodes willswitch back to its home node



11/393

Protecting Single Instance Database and Thrid application tier thru Oracle Clusterware

- 1 -

Using Oacle clusterware, we can have Cluster Databases, butalso non cluster database. In our exemple, we have one singleinstance database on one node.

Each node has its own node VIP, and we can create anapplication VIP (APPS VIP) on wich different resources aredependents :

ASM Single instance,

HR Single Database Instance,

Listener

HR application tier

/apps and /oracle local files systems

- 2 -

If first node where APPS VIP is hosted failed THEN :

Node VIP (10.3.25.81) will switch to an other node

APPS VIP (10.3.25.200) will switch to a preffered node (if configured, inj our caseon the third node. If third node was not available, APPS VIP will switch to one available node).



12/393

- 3 -

APPS VIPS has switched to preferred 3rd node. THEN, All resourcesdependant of APPS VIP will be restarted on third node.

- 4 -

When node 1 come back to normal operation, node VIP (10.3.25.81) will switch back

automatically to its home node, meaning first node. BUT APPS VIPS will not switchback !

- 5 -

To switch back APPS VIP and its dependant resources to first node, administrator wil l have to relocate the APPS VIP to first node thru clusterware commands.

Sending command on cluster events thru Oracle Clusterware



13/393

- 1 -

With Oracle clusterware, all cluster events(node, database, instance, etc )information (up, down, etc ) can beobtained and used to associate an action.

In our example case, we have :

3 nodes, with 3 LPARs each (RAC, APPS,TEST/DEV)

IBM HMC to administrate the IBM POWERSystem p and LPARs (cores, memory)

In normal operation, all LPARs are usingtheir defined cores and memory, butcould also use micro-partitionningbetween LPARs.

- 2 -

In our example case, if first node fails, THEN :

Node VIP (10.3.25.81) from first node will switchon one available node.

Oracle clusterware will have a FAN (FastApplication Notification) Event : node1 down

A unix shell script (Call out script) will analyze theinformation and send a request to the IBM HMC.

o Request to remove resources from

TEST/DEV LPARs from second and thirdnode.

o Request to assign new resources to APPSand RAC LPARs.

- 3 -

THEN, the HMC will execute the requests, and willresize tle LPARs as requested on each availablenodes.



14/393

For more information :

On http://otn.oracle.comhttp://www.oracle.com/technology/products/database/clusterware/index.html

Oracle Clusterware

Oracle Clusterware is a portable cluster software that allows clustering of single servers so that they cooperate as a

single system. Oracle Clusterware also provides the required infrastructure for Oracle Real Application Clusters (RAC).In addition Oracle Clusterware enables the protection of any Oracle application or any other kind of application within acluster. In any case Oracle Clusterware is the intelligence in those systems that ensures the required cooperationbetween the cluster nodes.

Oracle Clusterware Oracle Clusterware Whitepaper (PDF)

Oracle Clusterware Technical Articles Using standard NFS to support a third voting disk on an Extended Distance

cluster configuration on Linux, AIX, HP-UX, or Solaris (PDF) February 2008 Oracle Homes in an Oracle Real Application Clusters Environment (PDF) February 2008

Using Oracle Clusterware to protect any kind of application Using Oracle Clusterware to Protect 3rd Party Applications (PDF) February 2008

Using Oracle Clusterware to Protect 3rd Party Applications (PDF) February 2008

Using Oracle Clusterware to Protect Oracle Application Server (PDF) November 2005

U i O l Cl t t P t t O l D t b 10 ith
http://otn.oracle.com/http://www.oracle.com/technology/products/database/clusterware/index.htmlhttp://www.oracle.com/technology/products/database/clusterware/pdf/TWP_Clusterware_11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/oh_rac.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/TWP-Oracle-Clusterware-3rd-party.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/TWP-Oracle-Clusterware-3rd-party.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/Using-Clusterware-to-protect-Appserver.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/EM-DB-Failover.pdfhttp://otn.oracle.com/http://www.oracle.com/technology/products/database/clusterware/index.htmlhttp://www.oracle.com/technology/products/database/clusterware/pdf/TWP_Clusterware_11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/oh_rac.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/TWP-Oracle-Clusterware-3rd-party.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/TWP-Oracle-Clusterware-3rd-party.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/Using-Clusterware-to-protect-Appserver.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/EM-DB-Failover.pdf


15/393

3 ABOUT ORACLE AUTOMATIC STORAGE MANAGEMENT


With Oracle RAC, each instance must have access to the datafiles and recovery files for the Oracle RAC database.Using Automatic Storage Management (ASM) is an easy way to satisfy this requirement.

ASM is an integrated, high-performance database file system and disk manager. ASM is based on the principle that thedatabase should manage storage instead of requiring an administrator to do it. ASM eliminates the need for you todirectly manage potentially thousands of Oracle database files.

ASM groups the disks in your storage system into one or more disk groups. You manage a small set of disk groups andASM automates the placement of the database files within those disk groups.

ASM provides the following benefits:

StripingASM spreads data evenly across all disks in a disk group to optimize performance and utilization.This even distribution of database files eliminates the need for regular monitoring and I/O performance tuning.

MirroringASM can increase data availability by optionally mirroring any file. ASM mirrors at the file level,unlike operating system mirroring, which mirrors at the disk level. Mirroring means keeping redundant copies,or mirrored copies, of each extent of the file, to help avoid data loss caused by disk failures. The mirrored copyof each file extent is always kept on a different disk from the original copy. If a disk fails, ASM can continue toaccess affected files by accessing mirrored copies on the surviving disks in the disk group.

Online storage reconfiguration and dynamic rebalancingASM permits you to add or remove disks fromyour disk storage system while the database is operating. When you add a disk to a disk group, ASMautomatically redistributes the data so that it is evenly spread across all disks in the disk group, including thenew disk. The process of redistributing data so that it is also spread across the newly added disks is known as

b l i It i d i th b k d d ith i i l i t t d t b f


16/393


On http://otn.oracle.com

http://www.oracle.com/technology/products/database/asm/index.html

Automatic Storage Management

Automatic Storage Management (ASM) is a feature in Oracle Database 10g/11g that provides the databaseadministrator with a simple storage management interface that is consistent across all server and storage platforms. Asa vertically integrated file system and volume manager, purpose-built for Oracle database files, ASM provides theperformance of async I/O with the easy management of a file system. ASM provides capability that saves the DBAstime and provides flexibility to manage a dynamic database environment with increased efficiency.

What's New for ASM in Oracle Database 11g

11g ASM New Features Technical White PaperOracle Database 11g ASM new features technical white paper

ASM Overview and Technical Papers ASM and Multipathing Best Practices and Information Matrix (PDF)

Multipathing software and ASM information matrix and best practices guide Oracle Database 10gRelease 2 ASM - New Features White Paper(PDF)

This paper discusses the enhancements to ASM which are new in Release 2 of Oracle Database 10g. ASM Overview and Technical Best Practices White Paper(PDF)

This paper discusses the basics of ASM such as the steps to add disks, create a diskgroup, and createa database within ASM, emphasizing best practices.

T k th G k O t f D t b IO T i Whit P (PDF)
http://otn.oracle.com/http://www.oracle.com/technology/products/database/asm/index.htmlhttp://www.oracle.com/technology/products/database/asm/pdf/11gr1%20asm%20new%20features%20wp%2005-2007.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm%20and%20multipathing%20best%20practices%20info%20matrix%203-5-08.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm%20r2%20new%20features.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm%20r2%20new%20features.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm%20r2%20new%20features.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm_10gr2_bestpractices%2009-07.pdfhttp://www.oracle.com/pls/wocprod/url/ITEM/0A94991092E93070E040149032B17CE1http://otn.oracle.com/http://www.oracle.com/technology/products/database/asm/index.htmlhttp://www.oracle.com/technology/products/database/asm/pdf/11gr1%20asm%20new%20features%20wp%2005-2007.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm%20and%20multipathing%20best%20practices%20info%20matrix%203-5-08.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm%20r2%20new%20features.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm_10gr2_bestpractices%2009-07.pdfhttp://www.oracle.com/pls/wocprod/url/ITEM/0A94991092E93070E040149032B17CE1


17/393

4 ABOUT REAL APPLICATION CLUSTER


Oracle RAC extends Oracle Database so that you can store, update, and efficiently retrieve data using multipledatabase instances on different servers at the same time. Oracle RAC provides the software that facilitates serversworking together in what is called a cluster. The data files that make up the database must reside on shared storagethat is accessible from all servers that are part of the cluster. Each server in the cluster runs the Oracle RAC software.

An Oracle Database database has a one-to-one relationship between datafiles and the instance. An Oracle RACdatabase, however, has a one-to-many relationship between datafiles and instances. In an Oracle RAC database,multiple instances access a single set of database files. The instances can be on different servers, referred to as hostsor nodes. The combined processing power of the multiple servers provides greater availability, throughput, andscalability than is available from a single server.

Each database instance in an Oracle RAC database uses its own memory structures and background processes.Oracle RAC uses Cache Fusion to synchronize the data stored in the buffer cache of each database instance. CacheFusion moves current data blocks (which reside in memory) between database instances, rather than having onedatabase instance write the data blocks to disk and requiring another database instance to reread the data blocks from

disk. When a data block located in the buffer cache of one instance is required by another instance, Cache Fusiontransfers the data block directly between the instances using the interconnect, enabling the Oracle RAC database toaccess and modify data as if the data resided in a single buffer cache.

Oracle RAC is also a key component for implementing the Oracle enterprise grid computing architecture. Having ult ipledatabase instances accessing a single set of datafiles prevents the server from being a single point of failure. Anypackaged or custom application that ran well on a Oracle Database will perform well on Oracle RAC without requiringcode changes.

You will learn more about the operation of the Oracle RAC database in a cluster, how to build the cluster, and the

t t f O l RAC d t b i th ti f thi id


18/393


On http://otn.oracle.com

http://www.oracle.com/technology/products/database/clustering/index.html

Oracle Real Application Clusters

Oracle Real Application Clusters (RAC) is an option to the award-winning Oracle Database Enterprise Edition. Oracle

RAC is a cluster database with a shared cache architecture that overcomes the limitations of traditional shared-nothingand shared-disk approaches to provide highly scalable and available database solutions for all your businessapplications.Oracle Database Standard Edition includes Real Application Clusters support for higher levels of system uptime.

Real Application Clusters 11g Oracle Real Application Clusters Datasheet (PDF) July 2007

Oracle Real Application Clusters 11gTechnical Overview(PDF) July 2007

Workload Management with Oracle Real Application Clusters (FAN, FCF, Load Balancing)(PDF) July

2007 NEW Oracle Homes in an Oracle Real Application Clusters Environment(PDF) January 2008

UPDATED Using standard NFS to support a third voting disk on an Extended Distance clusterconfiguration on Linux,AIX, HP, or Solaris (PDF) February 2008
http://otn.oracle.com/http://www.oracle.com/technology/products/database/clustering/index.htmlhttp://www.oracle.com/technology/products/database/clustering/pdf/ds_rac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/awmrac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/oh_rac.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://otn.oracle.com/http://www.oracle.com/technology/products/database/clustering/index.htmlhttp://www.oracle.com/technology/products/database/clustering/pdf/ds_rac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/awmrac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/oh_rac.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdf


19/393

5 ABOUT 11G RAC / ASM ON AIX

11gRAC with OCR (Oracle Cluster Registry) disk(s), and Voting (Heartbeat) disk(s) on raw disks, and databaseon Oracle ASM (Automated Storage Management).


20/393

6 WHATS NEW WITH 11G RAC IMPLEMENTATION ON AIX

With Oracle 1 1g RAC on IBM POWER System p, i running AIX :

HACMP is not any more necessary as clusterware software since10gRAC Release 1.

Oracle provide with 11gRAC its own clusterware , named as Oracle

Clusterware, or CRS (Oracle Cluster Ready Service). Oracle Clusterware can cohabitate with HACMP under some

conditions.

With 11gRAC, Oracle Clusterware is mandatory as clusterware, evenso other vendors clusterware software are installed on the samesystems. You MUST check that third party clusterware can cohabitate

with Oracle clusterware.Subject: Using Oracle Clusterware with Vendor Clusterware FAQ

Doc ID:Note:332257.1

With Oracle Clusterware 11g Release 1 :
https://metalink.oracle.com/help/usaeng/Search/search.html#filehttps://metalink.oracle.com/help/usaeng/Search/search.html#filehttps://metalink.oracle.com/help/usaeng/Search/search.html#file


21/393

Oracle Real Application Cluster intend to provide :

High Availability of users services to maintain continuity of the business

Scale-in within each node, adding resources when possible, as your business grow

Scale out when scale in is not anymore possible, responding to your business grow

Workload balancing and affinity

Etc


22/393

7.1 General Requirements

This chapter will list the general requirements to look at for an Oracle RAC implementation on IBM system p.

Topics covered are :

Servers and processors

Network

Storage

Storage attachements

VIO Server

7.1.1 About Servers and processors


23/393

7.1.2 About RAC on IBM System p

Oracle Real Application cluster could be implemented on LPARs/DLPARs from Separated physical IBM System pservers

For production ortesting to achieve :

High Availability

(protecting loss ofa physical server)

Scalability (addingcores and memory,or adding newserver)

Database

Workload affinityacross servers

OR implemented between LPARs, DLPARs from a same physical IBM System p server


24/393

7.1.3 About Network


25/393

For PRIVATE Network (Cluster/RAC Interconnect) !!!Gigabit switch is mandatory for production

implementation, even for only 2 nodes architecture.

(Cross-over cable can be used only for test purpose, and its not supported by Oracle Support,please read RAC FAQ on http://metalink.oracle.com).

A second gigabit ethernet interconnect, with a different network mask, can be setup for securitypurposes or performance issues.

Network cards for public network must have same name on each participating node in the RAC cluster(For example en0 on all nodes).

Network cards for Interconnect Network (Private) must have same Name on each participating Node in the

RAC cluster (For example en1 on all nodes).1 virtual IP per node must be reserved, and not used on the network prior to Oracle clusterwareinstallation. Dont set IP allias at AIX level, Oracle clusterware will take charge of it.
http://metalink.oracle.com/http://metalink.oracle.com/


26/393

7.1.4 About SAN Storage

When implementing RAC, you must be carrefull on the SAN storage to use. The SAN Storage must becapable thru its drivers of read/write concurrency at same time from any member of the RAC cluster, whichmeans that reserve_policy attribute from disks (hdisk, hdiskpower, dlmfdrv, etc ) discovered must be ableto be set to no_reserve or no_lock values.


27/393


28/393

7.1.5 Proposed infrastructure with 2 servers

Example of infrastructure as it should be with 2 servers, for network and storage.


29/393

7.1.6 What do we protect ?

Oracle RAC protect against node failures, the infrastructure should cover a maximum of failure case as :

For the storage

loss of 1 HBA on one node

loss of 1 SAN switch

For the Network

loss of 1 interconnect Network card on one node

loss of 1 Interconnect Network switch


30/393

7.1.7 About IBM Advanced Power Virtualization and RAC

Extract from Certify : RAC for Unix On IBM AIX based Systems (RAC only) (27/02/2008, check for last update)https://metalink.oracle.com/metalink/plsql/f?p=140:1:12020374767153922780:::::

Oracle products are certif ied for AIX5L on all servers that IBM supports with AIX5L. This includes IBM System iand System p models that use POWER5 and POWER6 processors. Please visit IBM's website at this URL formore information on AIX5L support for System i details.

IBM power based systems that support AIX5L includes machines branded as RS6000, pSeries, iSeries, Systemp and System i.

The minimum AIX levels for POWER 6 based models are AIX5L 5.2 TL10 and AIX5L 5.3 TL06

AIX5L certifications include AIX5L versions 5.2 and 5.3. 5.1 was desupported on 16-JUN-2006.

o Customers should review MetaLink Note 282036.1

64-bit hardware is required to run Real Application Clusters.

AIX 64-bit kernel is required for 10gR2 RAC. AIX 32-bit kernel and AIX 64-bit kernel are supported with 9.2 and10g.

Extract from RAC Technologies Matrix for UNIX Platforms (11/04/2008, check for last update)http://www.oracle.com/technology/products/database/clustering/certify/tech_generic_unix_new.html

PlatformTechnology

CategoryTechnology Notes
https://metalink.oracle.com/metalink/plsql/f?p=140:1:12020374767153922780:::::http://www-03.ibm.com/systems/i/os/aix/https://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=282036.1http://www.oracle.com/technology/products/database/clustering/certify/tech_generic_unix_new.htmlhttps://metalink.oracle.com/metalink/plsql/f?p=140:1:12020374767153922780:::::http://www-03.ibm.com/systems/i/os/aix/https://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=282036.1http://www.oracle.com/technology/products/database/clustering/certify/tech_generic_unix_new.html


31/393

7.1.7.1 Network and VIO Server

Implementing Etherchannel links thru VIO servers for public and private network between nodes :


32/393


33/393

7.1.7.2 Storage and VIO Server

Following implementation is now supported !! !


34/393

Following implementation also supported :


35/393

7.2 Cookbook infrastructure

For our infrastructure, we used a cluster which is composed of three partitions (IBM LPAR) on an IBM systemp 570 using AIX 5L.

BUT in the real world, to achieve true high availability its necessary to have at least two IBM Systems p / i servers asshown bellow :


36/393

7.2.1 IBM Sytem p servers

This is the IBM SytempServer we used forour installation

http://www-03.ibm.com/servers/eserver/pseries/hardware/highend/

http://www-03.ibm.com/systems/p/

??????
http://www-03.ibm.com/servers/eserver/pseries/hardware/highend/http://www-03.ibm.com/systems/p/http://www-03.ibm.com/servers/eserver/pseries/hardware/highend/http://www-03.ibm.com/systems/p/


37/393

Commands to print the config for IBM System p on AIX5L (node1) :


38/393

{node1:root}/ # prtconf

System Model: IBM,9117-MMAMachine Serial Number: 651A260Processor Type: PowerPC_POWER6Number Of Processors: 1Processor Clock Speed: 3504 MHzCPU Type: 64-bitKernel Type: 64-bitLPAR Info: 1 JSC-11g-node1Memory Size: 3072 MBGood Memory Size: 3072 MBPlatform Firmware level: EM310_048

Firmware Version: IBM,EM310_048Console Login: enableAuto Restart: trueFull Core: false

Network InformationHost Name: node1IP Address: 10.3.25.81Sub Netmask: 255.255.255.0Gateway: 10.3.25.254

Name Server:Domain Name:

Paging Space InformationTotal Paging Space: 512MBPercent Used: 7%

Volume Groups Information==============================================================================rootvg:


39/393

Commands to print the config for IBM System p on AIX5L (node2) :


40/393

{node2:root}/ # prtconf

System Model: IBM,9117-MMAMachine Serial Number: 651A260Processor Type: PowerPC_POWER6Number Of Processors: 1Processor Clock Speed: 3504 MHzCPU Type: 64-bitKernel Type: 64-bitLPAR Info: 2 JSC-11g-node2Memory Size: 3072 MBGood Memory Size: 3072 MBPlatform Firmware level: EM310_048

Firmware Version: IBM,EM310_048Console Login: enableAuto Restart: trueFull Core: false

Network InformationHost Name: node2IP Address: 10.3.25.82Sub Netmask: 255.255.255.0Gateway: 10.3.25.254

Name Server:Domain Name:

Paging Space InformationTotal Paging Space: 512MBPercent Used: 2%

Volume Groups Information==============================================================================rootvg:


41/393


42/393

Command to get information on the LPAR :

{node1:root}/ # lparstat -iNode Name : node1Partition Name : JSC-11g-node1Partition Number : 1Type : Shared-SMTMode : UncappedEntitled Capacity : 0.50Partition Group-ID : 32769Shared Pool ID : 0

Online Virtual CPUs : 1Maximum Virtual CPUs : 1Minimum Virtual CPUs : 1Online Memory : 3072 MBMaximum Memory : 6144 MBMinimum Memory : 1024 MBVariable Capacity Weight : 128Minimum Capacity : 0.10Maximum Capacity : 1.00

Capacity Increment : 0.01Maximum Physical CPUs in system : 16Active Physical CPUs in system : 16Active CPUs in Pool : 16Shared Physical CPUs in system : -Maximum Capacity of Pool : -Entitled Capacity of Pool : -Unallocated Capacity : 0.00Physical CPU Percentage : 50.00%Unallocated Weight : 0


43/393


44/393

7.2.2 Operating System

Operating system must be installed the same way on each LPAR, with the same maintenance level, same APARand FILESETS level.

Check PREPARING THE SYSTEM chapter for Operating System requirements on AIX5L


45/393

7.2.3 Multi-pathing and ASM

Please check Metalink note Oracle ASM and Multi-Pathing Technologies Doc ID: Note:294869.1

Note, that Oracle Corporation does not certify ASM against multipathing utilities. The MP util ities listed below are onesthat known working solutions. As we do more testing, additional MP utilities will be listed here, thus, this document is anactive document.

Multi-pathing allow SAN access failover, and load balancing accros SAN Fiber Channel attachements.

OSPlatfor

m

Multi-pathingtool

ASM DeviceUsage

Notes

AIX EMC PowerPath Use raw partitionsthru the pseudodevice/dev/rhdiskpowerx

IBM SDD (Vpath)

NOTSUPPORTED forRAC/ASM onAIX !!!

/dev/vpathx As of this writing, SDD-AIX is known to cause discovery anddevice handling problems for ASM, and thus is not viablesolution.ASM needs to access disks/vpath thru non root user, which is notallowed by SDD as for today 27 March 2007.See SDDPCM section below for an alternative solution to SDDfor AIX

IBM SDDPCM Use /dev/rhdiskxdevice

You must install SDDPCM filesets and enable SDDPCM..SDDPCM cannot co-exist w/ SDD.
https://metalink.oracle.com/help/usaeng/Search/search.html#filehttps://metalink.oracle.com/help/usaeng/Search/search.html#file


46/393

7.2.4 IBM storage and multi-pathing

With IBM, please refer to IBM to confirm which IBM storage is supported with RAC, if not specified in our document.

IBM TotalStorage products for IBM Sytem p

IBM DS4000, DS6000 and DS8000 series are supported with 10gRAC.

IBM Storage DS300 and DS400 are not, and will not be supported with 10gRAC.As for today March 27, 2007 IBM Storage DS3200 and DS3400 are not yetsupported with 10gRAC.

IBM System Storage and TotalStorage products

http://www-03.ibm.com/servers/storage/product/products_pseries.html

??????

There are 2 cases when using IBM storage :

IBM MPIO (Multi-Path I/O).MPIO driver is supported with IBM Total Storage ESS, DS6000 and DS8000 series onlyAnd with IBM SVC (SAN Volume Controler).

IBM RDAC (Redundant Disk Array Controller) for IBM Total Storage DS4000.RDAC driver is supported with IBM Total Storage DS4000 series only, and former FasTt.
http://www-03.ibm.com/servers/storage/product/products_pseries.htmlhttp://www-03.ibm.com/servers/storage/product/products_pseries.html


47/393

7.2.4.1 IBM MPIO (Multi Path I/O) Setup Procedure

AIX Packages needed to installon all nodes :

devices.sddpcm.53.2.1.0.7.bffdevices.sddpcm.53.rtedevices.fcp.disk.ibm.mpio.rte

devices.fcp.disk.ibm.mpio.rte download page :http://www-

1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=en

MPIO for AIX 5.3 download page :http://www-1.ibm.com/support/docview.wss?uid=ssg1S4000201

On node 1,and node 2

Installingthe filesets :

smitty install

Install and Update SoftwareInstall Software

* INPUT device / directory for software[/mydir_with_my_filesets]

SOFTWARE to install []Press F4

Select devices.fcp.disk.ibm.mpio

Install Software
http://www-1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=enhttp://www-1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=enhttp://www-1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=enhttp://www-1.ibm.com/support/docview.wss?uid=ssg1S4000201http://www-1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=enhttp://www-1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=enhttp://www-1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=enhttp://www-1.ibm.com/support/docview.wss?uid=ssg1S4000201


48/393

Check the insatllationsucced and the

installation summarymessage :

Installation Summary--------------------Name Level Part Event Result-------------------------------------------------------------------------------devices.fcp.disk.ibm.mpio.r 1.0.0.0 USR APPLY SUCCESS

Install devices.sddpcm.53

Select :

devices.sddpcm.53

ALL + 2.1.0.0 IBM SDD PCM for AIX V53 + 2.1.0.7 IBM SDD PCM for AIX V53

Install Software

Type or select values in entry fields.Press Enter AFTER making all desired changes.

* INPUT device / directory for software .* SOFTWARE to install [devices.sddpcm.53

> +PREVIEW only? (install operation will NOT occur) no

+COMMIT software updates? yes

+SAVE replaced files? no

+AUTOMATICALLY install requisite software? yes

+


49/393

Commandsto know AIX WWPN

{node1:root}/ #pcmpath query wwpn

Adapter Name PortWWNfscsi0 10000000C935A7E7fscsi1 10000000C93A1BF3

Commands

to check disks :

{node1:root}/ # lsdev -Cc disk -t 2107

hdisk2 Available 0A-08-02 IBM MPIO FC 2107hdisk3 Available 0A-08-02 IBM MPIO FC 2107hdisk4 Available 0A-08-02 IBM MPIO FC 2107hdisk5 Available 0A-08-02 IBM MPIO FC 2107hdisk6 Available 0A-08-02 IBM MPIO FC 2107hdisk7 Available 0A-08-02 IBM MPIO FC 2107hdisk8 Available 0A-08-02 IBM MPIO FC 2107hdisk9 Available 0A-08-02 IBM MPIO FC 2107hdisk10 Available 0A-08-02 IBM MPIO FC 2107hdisk11 Available 0A-08-02 IBM MPIO FC 2107

Commands

to check disks :

{node1:root}/ #pcmpath query device

DEV#: 2 DEVICE NAME: hdisk2 TYPE: 2107900 ALGORITHM: Load BalanceSERIAL: 75271812000==========================================================================Path# Adapter/Path Name State Mode Select Errors

0 fscsi0/path0 CLOSE NORMAL 0 01 fscsi1/path1 CLOSE NORMAL 0 0

DEV#: 3 DEVICE NAME: hdisk3 TYPE: 2107900 ALGORITHM: Load BalanceSERIAL: 75271812001


50/393

7.2.4.2 IBM AIX RDAC (FCP.ARRAY filesets) Setup Procedure

This ONLY apply to use of of DS4000 storage series. NOT to DS6000, DS8000 and ES800.

RDAC is installed by default on AIX5L.Each node must have 2 HBA cards, for multi-pathing. With ONLY 1 HBA per node, it will works but path to SAN will notbe protected. THEN in production, 2 HBA per node must be used.

All AIX hosts in your storage subsystem must have the RDAC multipath driver installed.In a single server environment, AIX allows load sharing (also called load balancing). You can set the load balancingparameter to yes. In case of heavy workload onone path the driver will move other LUNs to the controller with less

workload and, if theworkload reduces back to the preferred controller. Problem that can occur is disk thrashing.Thatmeans that the driver moves the LUN back and forth from one controller to the other. As a result the controller is moreoccupied by moving disks around than servicing I/O. Therecommendation is to NOT load balance on an AIX system.The performance increase isminimal (or performance could actually get worse).

RDAC (fcp.array filesets) for AIX support round-robin load-balancing

Setting the attributes of the RDAC driver for AIXThe AIX RDAC driver files are not included on the DS4000 installation CD.

Either install them from the AIX Operating Systems CD, if the correct version is included, or download them from thefollowing Web site: http://techsupport.services.ibm.com/server/fixesorhttp://www-304.ibm.com/jct01004c/systems/support/

Commands

to check thatnecessary filesets

{node1:root}/ # lslpp -L devices.fcp.disk.array.rteFileset Level State Type Description (Uninstaller)----------------------------------------------------------------------------devices.fcp.disk.array.rte

5.3.0.52 A F FC SCSI RAIDiant Array DeviceSupport Software
http://techsupport.services.ibm.com/server/fixeshttp://www-304.ibm.com/jct01004c/systems/support/http://techsupport.services.ibm.com/server/fixeshttp://www-304.ibm.com/jct01004c/systems/support/


51/393

Commands

to check the RDAC configuration and HBA path to hdisk

On node1

{node1:root}/ # fget_config -v -A

---dar0---

User array name = 'DS4000_JSC'dac0 ACTIVE dac3 ACTIVE

Disk DAC LUN Logical Drivehdisk1 dac0 0 G8_spfilehdisk2 dac3 1 G8_OCR1hdisk3 dac0 2 G8_OCR2hdisk4 dac3 3 G8_Vote1hdisk5 dac0 4 G8_Vote2

hdisk6 dac3 5 G8_Vote3hdisk7 dac0 6 G8_Data1hdisk8 dac3 7 G8_Data2hdisk9 dac0 8 G8_Data3hdisk10 dac3 9 G8_Data4hdisk11 dac0 10 G8_Data5hdisk12 dac3 11 G8_Data6hdisk13 dac0 12 G8_tie{node1:root}/ #

On node2

{node2:root}/ # fget_config -v -A

---dar0---

User array name = 'DS4000_JSC'

dac0 ACTIVE dac3 ACTIVE

Disk DAC LUN Logical Drivehdisk0 dac0 0 G8_spfilehdisk1 dac3 1 G8_OCR1hdisk2 dac0 2 G8_OCR2hdisk3 dac3 3 G8_Vote1hdisk4 dac0 4 G8_Vote2hdisk5 dac3 5 G8_Vote3

hdisk6 dac0 6 G8_Data1hdisk7 dac3 7 G8_Data2hdisk8 dac0 8 G8_Data3hdisk9 dac3 9 G8_Data4hdisk10 dac0 10 G8_Data5hdisk12 dac3 11 G8_Data6hdisk13 dac0 12 G8_tie{node2:root}/ #


52/393

Fcs0 is one of the HBA.

Commands

To see the HBA fiberchannel statistics :

{node1:root}/ # fcstat fcs0

FIBRE CHANNEL STATISTICS REPORT: fcs0

Device Type: FC Adapter (df1080f9)Serial Number: 1F41709923Option ROM Version: 02E01871Firmware Version: H1D1.81X1World Wide Node Name: 0x20000000C93F8E29World Wide Port Name: 0x10000000C93F8E29

FC-4 TYPES:Supported:0x0000012000000000000000000000000000000000000000000000000000000000

Active:0x0000010000000000000000000000000000000000000000000000000000000000Class of Service: 3Port Speed (supported): 2 GBITPort Speed (running): 2 GBITPort FC ID: 0x650B00Port Type: Fabric

Seconds Since Last Reset: 2795

Transmit Statistics Receive Statistics------------------- ------------------

Frames: 41615 96207Words: 1537024 12497408

LIP Count: 0NOS Count: 0Error Frames: 0


53/393

7.2.5 EMC storage and multi-pathing

With EMC, please refer to Hitachi to see which EMC storage is supported with RAC.

There are 2 cases when using EMC storage :

case 1: Luns provided by the EMC storage with IBM MPIO installed as multi-pathing driver.

Disks (LUNs) will be seen as hdisk at AIX level using lspv command.

On node 1 {node1:root}/ # lspvhdisk0 00ced22cf79098ff rootvg activehdisk1 none Nonehdisk2 none Nonehdisk3 none Nonehdisk4 none None

Then for disks to be used for ASM, and on all nodes :

1. Install MPIO on all nodes, attach the LUN to each node, dicover LUNs with cfgmgr.

2. Identify hdisk names on each nodes, for a given LUN ID.

3. remove PVID from hdisk and change the reserve policy to no reserve using :

chdev -l hdisk -a pv=clear

chdev -l hdisk -a reserve_policy=no_reserve

4. set ownerchip to oracle:dba to the /dev/rhdisk

5. set read/write permissions to 660 to the /dev/rhdisk


54/393

7.2.5.1 EMC PowerPath Setup Procedure

See PowerPath for AIX version 4.3 Installation & Administration Guide, P/N 300-001-683 for details

On node 1, and node 2

1. Install EMC ODM drivers and necessary filesets 5.2.0.1 from ftp://ftp.emc.com/pub/elab/aix/ODM_DEFINITIONS/EMC.AIX.5.2.0.1.tar.Z

install using smit install

2. remove any existing devices attached to the EMC{node1:root}/ # rmdev dl hdiskX

3. run /usr/lpp/EMC/Symmetrix/bin/emc_cfgmgr to detect devices

4. Install PowerPath version 4.3.0 minimum using smit install

5. register PowerPath{node1:root}/ # emcpreg install

6. initialize PowerPath devices{node1:root}/ # powermt config

7. verify that all PowerPath devices are named consistently across all cluster nodes{node1:root}/ # /usr/lpp/EMC/Symmetrix/bin/inq.aix64 | grep hdiskpower

compare results. Consistent naming is not required for ASM devices, but LUNs usedfor the OCR and VOTE functions must have the same device names on all rac systems.


55/393

7.2.6 HITACHI storage and multi-pathing

With Hitachi, please refer to Hitachi to see which HDS storage is supported with RAC.There are 3 cases when using Hitachi HDS storage :

case 1: Luns provided by the HDS storage with IBM MPIO installed as multi-pathing driver.

*** NOT SUPPORTED with all HDS storage, check with Hitachi ***

Disks (LUNs) will be seen as hdisk at AIX level using lspv command.

On node 1 {node1:root}/ # lspvhdisk0 00ced22cf79098ff rootvg activehdisk1 none Nonehdisk2 none Nonehdisk3 none Nonehdisk4 none None


1. Install MPIO on all nodes, attach the LUN to each node, dicover LUNs with cfgmgr.

2. Identify hdisk names on each nodes, for a given LUN ID.

3. remove PVID from hdisk and change the reserve policy to no reserve using :

chdev -l hdisk -a pv=clear


56/393

Case 3 : Luns provided by the HDS storage with HDLM as multi-pathing driver.

Disks will be seen as dlmfdrv at AIX level using lspv command, and part of a HDLM VG(volume groups).

On node 1 {node1:root}/ # lspvdlmfdrv0 00ced22cf79098ff rootvg activedlmfdrv1 none vg_asm activedlmfdrv2 none vg_asm activedlmfdrv3 none vg_asm activedlmfdrv4 none vg_asm activedlmfdrv5 none vg_asm active

or

{node1:root}/ # lspvdlmfdrv0 00ced22cf79098ff rootvg activedlmfdrv1 none vg_ocr_disk1 activedlmfdrv2 none vg_voting_disk1 activedlmfdrv3 none vg_asm_disk1 activedlmfdrv4 none vg_asm_disk2 activedlmfdrv5 none vg_asm_disk3 active


1. Install HDLM on all nodes, attach the LUN to each node, dicover LUNs with cfgmgr.

2. Turn off reserve locking on all nodes dlnkmgr set rsv on 0 s

3. Create VGs (Volume Groups) and LVs (Logical Volumes)

2 options :


57/393

dlmvaryoffvg vg_asm

On node 2

6) Identify which dlmfdrv correspond to dlmfdrv1 fromnode1.

dlmfdrv1 on node1 dlmfdrv0 on node2

7) import the volume groups on node2

This will copy the vg/lv configuration that was made onnode1

dlmimportvg -V 101 -y vg_asm dlmfdrv0

7) enable the volume groups on node2

dlmvaryonvg vg_asm

8) this will ensure the vg will not get varyon'd atboot

dlmchvg -a n vg_asm

On node 1

9) enable the volume groups on node1

dlmvaryonvg vg_asm

Check for document :

Hitachi Dynamic Link Manager (HDLM) for IBM AIXSystems Users Guide:

chmod 660 /dev/rlv_voting_disk1chmod 660 /dev/rlv_asm_disk1chmod 660 /dev/rlv_asm_disk2chmod 660 /dev/rlv_asm_disk3

5) disable the volume groups

dlmvaryoffvg vg_ocr1dlmvaryoffvg vg_vot1dlmvaryoffvg vg_asm1dlmvaryoffvg vg_asm2dlmvaryoffvg vg_asm3

On node 2

6) Identify which dlmfdrv correspond to dlmfdrv1 fromnode1, and so on with other dlmfdrv

dlmfdrv1 on node1 dlmfdrv0 on node2dlmfdrv2 on node1 dlmfdrv1 on node2dlmfdrv3 on node1 dlmfdrv2 on node2dlmfdrv4 on node1 dlmfdrv3 on node2dlmfdrv5 on node1 dlmfdrv4 on node2

7) import the volume groups on node2

This will copy the vg/lv configuration that was made onnode1

dlmimportvg -V 101 -y vg_ocr1 dlmfdrv0dlmimportvg -V 101 -y vg_vot1 dlmfdrv1dlmimportvg -V 101 -y vg_asm1 dlmfdrv2dlmimportvg -V 101 -y vg_asm2 dlmfdrv3dlmimportvg -V 101 -y vg_asm3 dlmfdrv4
http://sys-admin.net/uploads/SAN/hldm_admin_guide.pdfhttp://sys-admin.net/uploads/SAN/hldm_admin_guide.pdfhttp://sys-admin.net/uploads/SAN/hldm_admin_guide.pdfhttp://sys-admin.net/uploads/SAN/hldm_admin_guide.pdfhttp://sys-admin.net/uploads/SAN/hldm_admin_guide.pdf


58/393


59/393

8 SPECIFIC CONSIDERATIONS FOR RAC/ASM SETUP WITH HACMPINSTALLED

Oracle Clusterware does not requires HACMP to work, but some customers may need to have HACMPinstalled on the RAC node cluster to protect third party products, or ressources. Oracle clusterware couldreplace HACMP for most operations as cold failover for 11g, 10g and 9i single database, applications servers or anyapplications.

Please check following documents for details onhttp://www.oracle.com/technology/products/database/clustering/index.html :

Comparing Oracle Real Applicaton Clusters to Failover Clusters for Oracle Database(PDF) December 2006 Workload Management with Oracle Real Application Clusters (FAN, FCF, Load Balancing) (PDF) May 2005

Using Oracle Clusterware to Protect 3rd Party Applications (PDF) May 2005

Using Oracle Clusterware to Protect a Single Instance Oracle Database (PDF) November 2006

Using Oracle Clusterware to protect Oracle Application Server (PDF) November 2005

How to Build End to End Recovery and Workload Balancing for Your Applications 10g Release 1(PDF) Dec2004

Oracle Database 10gServices (PDF) Nov 2003

Oracle Database 10gRelease 2 Best Practices: Optimizing Availability During Unplanned Outages UsingOracle Clusterware and RACNew!

HOWEVER, if customer still need to have HACMP, Oracle Clusterware can cohabitate with HACMP.

Please check for last status of certification on Oracle Metalink (certify)

Extract from metalink in date of March 3rd, 2008.
http://www.oracle.com/technology/products/database/clustering/index.htmlhttp://www.oracle.com/technology/products/database/clustering/pdf/failover_cluster.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twpracwkldmgmt.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_oracleclusterware3rdparty%5B1%5D.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/ocsingleinstance.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/Using_Oracle_Clusterware_to_protect_Oracle_Application_Server.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/oowfan.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac_services_10gr1_112503.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac_services_10gr1_112503.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac_services_10gr1_112503.pdfhttp://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdfhttp://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdfhttp://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdfhttp://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdfhttp://www.oracle.com/technology/products/database/clustering/index.htmlhttp://www.oracle.com/technology/products/database/clustering/pdf/failover_cluster.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twpracwkldmgmt.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_oracleclusterware3rdparty%5B1%5D.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/ocsingleinstance.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/Using_Oracle_Clusterware_to_protect_Oracle_Application_Server.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/oowfan.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac_services_10gr1_112503.pdfhttp://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdfhttp://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdf


60/393

Detailed information for HACMP 5.3 and 11gRAC :

Certify - Additional Info RAC for Unix Version 11gR1 64-bit On IBM AIX based Systems (RAC only)

Operating System: IBM AIX based Systems (RAC only) Version 5.3RAC for Unix Version 11gR1 64-bitHACMP Version 5.3Status: Certified

Product Version Note:None available for this product.

Certification Note:

Use of HACMP 5.3 requires minimum service levels of the following:

o AIX5L 5.3 TL 6 or later, specifically bos.rte.lvm must be at least 5.3.0.60

o HACMP V5.3 with PTF5, cluster.es.clvm installed and ifix for APAR IZ01809

o RSCT (rsct.basic.rte) version 2.4.7.3 and ifix for APAR IZ01838

Detailed information for HACMP 5.4.1 and 11gRAC :

Certify - Additional Info RAC for Unix Version 11gR1 64-bit On IBM AIX based Systems (RAC only)

Operating System: IBM AIX based Systems (RAC only) Version 5.3RAC for Unix Version 11gR1 64-bitHACMP Version 5.4.1Status: Certified

Product Version Note:


61/393

Following rules have to be applied :

1. HACMP must not take-over/failover the Oracle Clusterware ressources (VIP, database, etc )

2. HACMP VIP must not be configured on IP from Public node name used by RAC (hostname), or OracleClusterware VIP

3. With 11g, its not necessary to declare the RAC interconnect in HACMP

4. Its not mandatory to declare hdisks used for ASM in HACMP as logical volumes (LV) from Volume Groups(VG). In this case follow the cookbook to prepare the disks for OCR, Voting and ASM disks.

5. If the choice is to declare hdisks used by ASM in HACMP Volume Groups, THEN youll have to prepare thedisks for OCR, Voting, ASM spfile and ASM disks as describe in official Oracle document available onhttp://tahiti.oracle.com

Please check :

Oracle Real Application Clusters Installation Guide11g Release 1 (11.1) for Linux and UNIXPart Number B28264-03http://download.oracle.com/docs/cd/B28359_01/install.111/b28264/toc.htm

And Metalink note 404474.1

Status of Certification of Oracle Clusterware with HACMP 5.3 & 5.4STATUS of IBM HACMP 5.3, 5.4 Certifications with Oracle RAC 10g5.4 Certifications with Oracle RAC 10g What do you need to do? Configuring HACMP 5.3 orHACMP 5.4.1,and 10gR2 with Multi-Node Disk Heartbeat (MNDHB)

Even so this note is written for 10gRAC R2, its also applicable to 11g RAC R1.
http://tahiti.oracle.com/http://download.oracle.com/docs/cd/B28359_01/install.111/b28264/toc.htmhttps://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=404474.1http://tahiti.oracle.com/http://download.oracle.com/docs/cd/B28359_01/install.111/b28264/toc.htmhttps://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=404474.1


62/393

9 INSTALLATION STEPS

Prioir to install and use Oracle 11g Real Application Cluster, you must :

1. Prepare the infrastructurea. Hardwareb. Storagec. Networkd. San and Network connectivitye. Operating system

2. Prepare the systems

a. Defining Network Layout, Public, Virtual andPrivate hostnames

b. AIX Operating system level, required APARsand filsets

c. System Requirements (Swap, temp, memory,internal disks)

d. Users And Groupse. Kernel and Shell Limitsf. User equivalencesg. Time Server Synchronization


63/393

When all that is done !!!, you can process the installation in the following order :

1. Install Oracle 11g Clusterwarea. Apply necessary patchset and patchesb. Check that Oracle clusterware is working

properly

2. Install Oracle 11g Automated Storage Managementa. Apply necessary patchset and patches

b. Create default nodes listenersc. Create ASM instances on each noded. Configure ASM instances Local and remote

listeners if requirede. Change necessary ASM instances parameters

(process, etc )f. Create ASM diskgroup(s)

3. Inst all Oracle 11g Real Application Cluster or/and

Oracle 10g Real Application clustera. Apply necessary patchset and patches


64/393

10 PREPARING THE SYSTEM

Preparing the system is to be done on all servers which are planned to participate in the Oracle Cluster.All servers MUST be as clones, with ONLY different HOSTNAME and IP addresses !!!

IMPORTANT !!!

For ALL servers, you MUST apply the Oracle pre-requisites, those prerequisites are notoptional, BUT MANDATORY !!!

For some parameters as tuning settings, values specified are the minimum required, andmigh be increase depending of your needs !!!

PLEASE check Oracle documentation for last update, and MOSTLY :Oracle METALINK Note

Preparing the system is about :

Defining Network Layout, Public, Virtual and Private hostnameso Hostname and RAC Public Node nameo Network interface identificationo Default Gateway


65/393

10.1 Network configuration

10.1.1 Define Networks layout, Public, Virtual and Private Hostnames

We need 2 differents networks, with associated network interfaces on each node participating to the RAC cluster :

Public Network to be used as Users network or reserved network for Application and Databases servers.

Private Network to be used as a Reserved network for Oracle clusterware, and RAC.

MANDATORY : Network interfaces must have same name, same subnet and same usage !!!

For each node, We have to define :


66/393

Let identify hostnames, and network interfaces

Please make a table as follow to have a clear view of your RAC network architecture :PUBLIC NODE NAME MUST BE NAME RETURNED BY hostname AIX COMMAND

Public Hostname VIP Hostname

(Virtual IP)

Private Hostname

(RAC Interconnect)

Not Used

en ? (Public Network) en ? (Private Network) en ?

NodeName

IP NodeName

IP NodeName

IP Node Name IP


67/393

For second server ...

{node2:root}/ # hostnamenode2

{node2:root}/ #ping node2PING node2: (10.3.25.82): 56 data bytes64 bytes from 10.3.25.82: icmp_seq=0 ttl=255 time=0 ms64 bytes from 10.3.25.82: icmp_seq=1 ttl=255 time=0 ms64 bytes from 10.3.25.82: icmp_seq=2 ttl=255 time=0 ms^C

----node2 PING Statistics----3 packets transmitted, 3 packets received, 0% packet lossround-trip min/avg/max = 0/0/0 ms{node2:root}/ #

Other method to check the default hostname :

{node1:root}/crs/11.1.0/bin # lsattr -El inet0authm 65536 Authentication Methods Truebootup_option no Use BSD-style Network Configuration Truegateway Gateway Truehostname node1 Host Name Truerout6 IPv6 Route Trueroute net,-hopcount,0,,0,10.3.25.254 Route True{node1:root}/crs/11.1.0/bin #


68/393

Oracle clusterware VIPs IP adress and corresponding nodes names must not be used on the network prior toOracle Clusterware installation. Dont make any AIX allias on the public network interface, the clusterware installationwill do it. Just reserve 1 VIP and its hostname per RAC node.


(Virtual IP)

Private Hostname

(RAC Interconnect)

Not Used

en ? (Public Network) en ? (Private Network) en ?

NodeName

IP NodeName

IP NodeName

IP Node Name IP

node1 10.3.25.81 node1-vip 10.3.25.181

node2 10. 3.25.82 node2-vip 10. 3.25.182

Oracle Clusterware VIPs IP and corresponding nodes names can be declared in the DNS, or at minimum in the localhosts file.

10.1.2 Identify Network Interfaces cards


69/393

As root, Issue the following shell to get necessary information from network interfaces on each node :

Result from node1 :

{node1:root}/ # for i in en0 en1 en2do

echo $ifor attribut in netaddr netmask broadcast

statedo

lsattr -El $i -a $attributdone

doneen0netaddr 10.3.25.81 Internet Address Truenetmask 255.255.255.0 Subnet Mask Truebroadcast Broadcast Address Truestate up Current Interface Status Trueen1netaddr 10.10.25.81 Internet Address True

netmask 255.255.255.0 Subnet Mask Truebroadcast Broadcast Address Truestate up Current Interface Status Trueen2netaddr 20.20.25.81 Internet Address Truenetmask 255.255.255.0 Subnet Mask Truebroadcast Broadcast Address Truestate up Current Interface Status True{node1:root}/ #

Result from node2 :

{node2:root}/ # for i in en0 en1 en2do

echo $ifor attribut in netaddr netmask broadcast

statedo

lsattr -El $i -a $attributdone

doneen0netaddr 10.3.25.82 Internet Address Truenetmask 255.255.255.0 Subnet Mask Truebroadcast Broadcast Address Truestate up Current Interface Status Trueen1netaddr 10.10.25.82 Internet Address True

netmask 255.255.255.0 Subnet Mask Truebroadcast Broadcast Address Truestate up Current Interface Status Trueen2netaddr 20.20.25.82 Internet Address Truenetmask 255.255.255.0 Subnet Mask Truebroadcast Broadcast Address Truestate up Current Interface Status True{node2:root}/ #


70/393

Now, we have identified the following network interfaces :

en0 is set to be the Public Network Interface on all nodes.

en0 is set to be the VIP Network Interface on all nodes.

en1 is set to be the Private Network Interface, also named as RAC Interconnect on all nodes.

en2 is set as not used

THEN, the table looks as follow :


(Virtual IP)

Private Hostname

(RAC Interconnect)

Not Used

en0 (Public Network) en1 (Private Network) en2


71/393

THEN, we will get the following table with our system :

Public VIP RAC Interconnect (PrivateNetwork)

Not Used

en0 en1 en2

NodeName

IP NodeName

IP NodeName

IP NodeName

IP

node1 10.3.25.81 node1-vip 10.3.25.181 node1-rac 10.10.25.81

node2 10. 3.25.82 node2-vip 10. 3.25.182 node1-rac 10.10.25.82

Within our infrastructure for the cookbook, we have the following layout :


72/393

10.1.4 Defining Default gateway on public network interface


73/393

may have if you have already a default gateway set on a different network interface, and multiple networkinterfaces not used for Oracle clusterware and RAC purpose.

First check if default gateway is set :Use netstat -r On both nodes.

{node1:root}/ # netstat -rRouting tablesDestination Gateway Flags Refs Use If Exp Groups

Route Tree for Protocol Family 2 (Internet):

default 10.3.25.254 UG 1 73348 en0 - - =>default 9.212.131.254 UG 0 435 en0 - -10.3.25.0 node1 UHSb 0 0 en0 - - =>10.3.25/24 node1 U 11 6113590 en0 - -node1 loopback UGHS 37 1034401 lo0 - -node1-vip loopback UGHS 8 80831 lo0 - -10.3.25.255 node1 UHSb 0 4 en0 - -10.10.25.0 node1-rac UHSb 0 0 en1 - - =>10.10.25/24 node1-rac U 25 350557 en1 - -

node1-rac loopback UGHS 16 481 lo0 - -10.10.25.255 node1-rac UHSb 0 4 en1 - -20.20.25.0 node1-rac-b UHSb 0 0 en2 - - =>20.20.25/24 node1-rac-b U 16 176379 en2 - -node1-rac-b loopback UGHS 5 392 lo0 - -20.20.25.255 node1-rac-b UHSb 0 4 en2 - -127/8 loopback U 49 187105 lo0 - -

Route Tree for Protocol Family 24 (Internet v6):


74/393

10.1.5 Configure Network Tuning Parameters

Verify that the network tuning parameters shown in the following table are set to the values shown or higher values.The procedure following the table describes how to verify and set the values.

Network TuningParameter Recommended minimum Value on all nodes

ipqmaxlen 512

rfc1323 1

sb_max 1310720

tcp_recvspace 65536

tcp_sendspace 65536

udp_recvspace 655360

Note: The recommended value of this parameter is 10 times the value of the udp_sendspaceparameter. The value must be less than the value of the sb_max parameter.

udp_sendspace 65536

Note: This value is suitable for a default database installation. For production databases, theminimum value for this parameter is 4 KB plus the value of the database DB_BLOCK_SIZEinitialization parameter mult iplied by the value of the DB_MULTIBLOCK_READ_COUNTinitialization parameter:

(DB_BLOCK_SIZE * DB_MULTIBLOCK_READ_COUNT) + 4 KB

To check values


75/393

To change thecurrent values torequired ones, ifnecessary,

follow these steps :

1. If you must change the value of any parameter, enter the following command todetermine whether the system is running in compatibility mode:

# /usr/sbin/lsattr -E -l sys0 -a pre520tune

If the system is running in compatibility mode, the output is similar to thefollowing, showing that the value of the pre520tune attribute is enable:

pre520tune enable Pre-520 tuning compatibility mode True

By default, with AIX5L, compatibility mode is set to false !!!Change it to true ONLY if necessary !!!

** if you want to enable the compatibility mode, issue the following command :# chdev -l sys0 -a pre520tune=enable

2. If the system is running in compatibility mode,

THEN

follow these steps to change the parametervalues:

Enter commands similar to the following tochange the value of each parameter:

# /usr/sbin/no oparameter_name=value

For example:

ELSE

# /usr/sbin/lsattr -E -l sys0 -a pre520tune

Will give the following :

pre520tune disable Pre-520 tuningcompatibility mode True

The system is not running in compatibility mode, entercommands similar to the following to change the


76/393

10.2 AIX Operating system level, required APARs and filsets


77/393

10.2.1 Filesets Requirements for 11g RAC R1 / ASM (NO HACMP)

AIX filesets required on ALL nodes for 11gRAC Release 1 implementation withASM !!!

Check that therequired filsets areinstalled on thesystem.

(Note: If the PTF isnot downloadable,customers shouldrequest an efixthrough AIXcustomer support.)

AIX 5L version 5.3,Maintenance Level 6 or later

AIX6 version ...Maintenance Level ... or later

Filesets

bos.adt.base bos.adt.lib

bos.adt.libm

bos.perf.libperfstat

bos.perf.perfstat

bos.perf.proctools

rsct.basic.rte

rsct.compat.clients.rte

xlC.aix50.rte 8.0.0.5

xlC.rte 8.0.0.5

Specific Filesets

For EMC Symmetrix : EMC.Symmetrix.aix.rte.5.2.0.1 or higher

For EMC CLARiiON : EMC.CLAR11ON.fcp.rte.5.2.0.1 or

higher

NOT YETSUPPORTEDCheck Metalink and certify for last update oncertification status.


78/393

10.2.2 APARs Requirements for 11g RAC R1 / ASM (NO HACMP)

Check that the required APARs are installed on the system.

AIX Patches (APAR) required on ALL nodes for 11gRAC R1 implementation withASM !!!

(Note: If the PTF is not downloadable, customers should request an efix through AIX customer support.)

AIX 5L version 5.3,Maintenance Level 6 or later

AIX6 version ...Maintenance Level ... or later

APARs

IY89080

IY92037

IY94343

IZ01060 or efix for IZ01060

IZ03260 or efix for IZ03260

JDK : (Not mandatory for the installation), thes APAR arerequired only if you are using the associated JDK version

IY58350 Patch for SDK 1.3.1.16 (32-bit)



IBM JDK 1.5.0.06 (IA64 - mixed mode) is installed

NOT YETSUPPORTEDCheck Metalink and certify for last update on certificationstatus.


79/393

10.3 System Requirements (Swap, temp, memory, internal disks)

Requirements to meet on ALL nodes !!!

RAM >= 512 MB minimum. Command to check the physical memory : lsattr El sys0 a realmem

{node1:root}/ # lsattr -El sys0 -a realmemrealmem 3145728 Amount of usable physical memory in Kbytes False{node1:root}/ #

{node2:root}/home # lsattr -El sys0 -a realmemrealmem 3145728 Amount of usable physical memory in Kbytes False{node2:root}/home #

Internal disk >= 12 GB for the oracle code (CRS_HOME, ASM_HOME, ORACLE_HOME)

This part will be detailed in chapter Required local disks (Oracle Clusterware, ASM and RAC software) !!!

{node1:root}/ # df -kFilesystem 1024-blocks Free %Used Iused %Iused Mounted on/dev/hd4 262144 207816 21% 13591 23% //dev/hd2 4718592 2697520 43% 46201 7% /usr/dev/hd9var 262144 233768 11% 565 2% /var/dev/hd3 1310720 1248576 5% 255 1% /tmp/dev/hd1 262144 261760 1% 5 1% /home


80/393

Paging space = 2 x RAM, with a minimum of 400 MB and a maximum of 2 GB. To check the paging space configured : lsps a

{node1:root}/ # lsps -aPage Space Physical Volume Volume Group Size %Used Active Auto Typehd6 hdisk0 rootvg 512MB 7 yes yes lv{node1:root}/ #

{node2:root}/home # lsps -aPage Space Physical Volume Volume Group Size %Used Active Auto Typehd6 hdisk1 rootvg 512MB 2 yes yes lv{node2:root}/home #

Temporary Disk Space : The Oracle Universal Installer requires up to 400 MB of free space in the /tmp directory. To check the free temporary space available: df k /tmp

{node1:root}/ # df -k /tmp

Filesystem 1024-blocks Free %Used Iused %Iused Mounted on/dev/hd3 1310720 1248576 5% 255 1% /tmp{node1:root}/ #

{node2:root}/home # df -k /tmpFilesystem 1024-blocks Free %Used Iused %Iused Mounted on/dev/hd3 2097152 29636 99% 4812 35% /tmp{node2:root}/home #


81/393

10.4 Users And Groups

2 options possibles

OPTION 1 OPTION 2

1 user for each installation, for example :

unix crs user for CRS_HOME installation

unix asm user for ASM_HOME installation

unix rdbms user for ORACLE_HOMEinstallation

1 user for all installations,for example oracle unix user for :

CRS_HOME installation

ASM_HOME installation

ORACLE_HOME installation


82/393

This setup has to be done on all the nodes of the cluster.

Default user home for crs, asm and rdbms users MUST be in /home, otherwise you may have troublewith ssh setup (user equivalence).


83/393

The crs, asm and rdbms users must have oinstallas primary group, dba as