wp o10g rac config win 303805

7/29/2019 Wp o10g Rac Config Win 303805

1/70

1/70

SAP Netweaver / Oracle Database 10gR2 RAC

on Windows 2003

A Best Practices Guide

An Oracle White Paper

June 2007


2/70

2/70

Preface...........................................................................................................................4Naming Conventions and Terms ...............................................................................4

Implementation Roadmap..........................................................................................5

Technical Knowledge ............................................................................................5

Evaluation ..............................................................................................................5Pre-Installation.......................................................................................................5

Installation..............................................................................................................5

Testing....................................................................................................................5

Prerequisites...............................................................................................................6

New Cluster Hardware...........................................................................................6

Storage ...............................................................................................................6

Hardware Configuration ............................................................................................7

Storage ...................................................................................................................7

Network..................................................................................................................8

Network Interfaces.............................................................................................8

System Configuration ..............................................................................................10Operating System.................................................................................................10

Support of Windows 2003 Server........................................................................10

Microsoft Cluster Software and Heartbeat ..........................................................11

Windows 2003 64-bit or 32-bit............................................................................11

Windows 2003 64-bit on x86_64 platforms ....................................................11

Windows 2003 32-bit on x86_32 platforms ....................................................11

Storage, Disks ......................................................................................................12

Network Interfaces...............................................................................................14

Installation before Migration?..................................................................................14

Software Installation & Configuration ....................................................................16Overview..................................................................................................................16

Oracle Clusterware (OCW)......................................................................................17

Installation............................................................................................................17

Upgrade................................................................................................................29

Oracle CFS...............................................................................................................31

Choosing the right NIC for OCFS .......................................................................31

Setting up OCFS volumes....................................................................................31

OCFS IO-Modes..................................................................................................31

Rules for Formatting Volumes with OCFS .....................................................31

Formatting Volumes ............................................................................................32

Migration..................................................................................................................33Oracle RDBMS Software ....................................................................................33

Installation........................................................................................................33

Upgrade............................................................................................................34

Database Conversion to RAC ..............................................................................34

Enabling RAC..........................................................................................................34

Undo Tablespaces................................................................................................34

Online Redo Logs ................................................................................................35

Init.ora and Spfile.ora ..........................................................................................35

Oracle Administrator Groups...............................................................................36

Oracle Instance Creation......................................................................................37

Environment Variables ........................................................................................37SQL*Net Configuration and Service Creation ....................................................37


3/70

3/70

Major Differences between Oracle 9i and Oracle 10g Configurations............37

Oracle Net Configuration Assistant (netca) .....................................................38

Listener.ora ......................................................................................................42

Tnsnames.ora ...................................................................................................43

Sqlnet.ora .........................................................................................................45

Putting the Database and Instances under CRS control ...................................45Services............................................................................................................46

Checking the Setup ..............................................................................................47

Srvctl and crsctl........................................................................................................48

Windows 2003 and NTBackup................................................................................49

Backing up OCR......................................................................................................49

Backup .....................................................................................................................49

Testing......................................................................................................................50

Failures.................................................................................................................50

Hardware Failures............................................................................................50

Software Failures .............................................................................................51

SAP Application Workload .................................................................................52Additional Performance Recommendations ............................................................52

Automated Segment Space Management (ASSM)..............................................52

Local Update Dispatching....................................................................................52

Automatic Offlining of Undo Segments by SMON ............................................53

Appendix A: Oracle Cluster File System for Windows (OCFS).............................53

Introduction..........................................................................................................53

Mount and Volume Recognition Process ........................................................53

Differences to NTFS and FAT.............................................................................54

Security ............................................................................................................54

Multi-Host Concurrent Metadata Changes......................................................54

Predictable Hangs.........................................................................................54

Mount Points, Drive Letters.............................................................................54

Internal Metadata for Multiple Cluster Nodes .................................................54

OCFS Administration ..........................................................................................55

Setting Volumes Offline ..................................................................................55

Setting Volumes Online ...................................................................................55

Adding a new OCFS Volume ..........................................................................55

Removing a Volume ........................................................................................55

Split-Mirror Backup and Restore under OCFS....................................................56

Appendix B: FAQ....................................................................................................56

Appendix C: BR*Tools Sample Configuration for Legato Networker................58initT011.sap: ........................................................................................................58

initT012.sap: ........................................................................................................58

initT01arch.sap: ...................................................................................................58

initT01arch.utl......................................................................................................64


4/70

4/70

PrefaceOracle Real Application Clusters (RAC) is a supported solution for the database

backend in an SAP installation. Using SAP with Oracle RAC provides high

availability and scalability. The purpose of this document is to provide best practices

on how to migrate an existing single-node SAP installation to a multiple-instance

Oracle 10gR2 RAC cluster database configuration.

This document covers the installation of Oracle Clusterware, Oracle Cluster

Filesystem (OCFS) and Oracle RDBMS software. It describes the necessary changes

to the database structures on the disk, as well as to database and SAP system

configurations.

The pre-installation steps for setting up the underlying cluster hardware are also

discussed as a preliminary requirement. However, due to the fact that the cluster

technology from different hardware vendors is evolving quickly, this paper just gives

some guidelines on what hardware is required and how this hardware has to beconfigured.

It is highly recommended that you be familiar with installing and configuring SAP

and Oracle RDBMS software before starting a new SAP and Oracle RAC integration!

Please refer also to the white paper

Configuration of SAP NetWeaver for Oracle 10g Release 2 Real Application

Clusters.

Naming Conventions and Terms

The following naming conventions and terms are used in this document:

Name or Synonym Description / Purpose

Disk, physical disk A single physical disk from a Windows

point of view. We create extended

partitions on physical disks.

Extended partition An extended partition can span part of a

disk or a whole disk. Logical volumes are

created inside an extended partition.

Logical volume Logical volumes are created in an

extended partition and can either be usedas raw devices or formatted for use with a

filesystem like NTFS or OCFS.

OCFS Oracle Cluster Filesystem

OCFS volume A volume formatted with ocfsformat and

used by OCFS

OCW or CRS Oracle ClusterWare that consists of a

whole stack of clustering software

components. OCFS also belongs to

OCW. This was formerly called Cluster

Ready Services and is often used as a

synonym for Oracle Clusterware.OCR Oracle Cluster Repository


5/70

5/70

Implementation Roadmap

Technical Knowledge

Clusters are different to single server systems and require specific skills in terms of

implementation and administration. This is also true for Oracle 10gR2 RAC Clusters

running on Windows. Many customers run single instance systems for many yearsand know their systems very well. But most do not have enough technical knowledge

to implement and run a RAC system. It is essential to build up that knowledge and

make yourself familiar with that new technology. This will take time, so start as early

as possible.

Evaluation

In Oracle RAC environments with two or more nodes the hardware configuration is

different compared with single server systems. The most important differences are the

requirement of a shared storage system and local disks. Also the number of network

interfaces and switches is usually higher than in single server systems. Take time toevaluate the hardware components and check if they meet all requirements for

running RAC. This will avoid misunderstandings and trouble later on.

Pre-Installation

Before starting an installation of the whole software stack to run Oracle RAC ensure

you have a detailed plan of how your configuration should look like. E.g. required IP

addresses, sub net masks, default gateway, bind order of network interface cards,

DNS, configuration of network switches and vlans, disk layout, volume sizes and

usage, drive letters or symbolic links that have to be used, swap space configuration

and much more.

Installation

An installation of RAC on a Windows cluster consists of multiple steps. Each step of

the installation should be finished, checked and tested meticulously before proceeding

with the next step. Overlooking wrong or missing configuration steps will very likely

cause problems during the later steps.

Testing

When the installation of the whole Windows cluster with Oracle RAC is done it is

necessary to test your configuration before going live. Create a detailed test plan. This

plan should include tests for hardware failures, software failures, application and loadtests and backup and recovery scenarios. Test examples are documented in a later

chapter Testing. Going live tests should take weeks not days!


6/70

6/70

Prerequisites

New Cluster Hardware

New cluster hardware must be provided in order to set up a new RAC environment.

Storage

Each node of the cluster must have its own local disks attached for OperatingSystem and Oracle Clustersoftware.

The cluster must have shared disks attached to each node for the Database and

Oracle RDBMS software.

Each node requires at least 2 NICs:

In an Oracle RAC environment, database instances communicate with each

other using a dedicated private network called Cluster Interconnect. This

private network requires at least Gigabit Ethernet, and a dedicated Ethernet-

Switch must be used to connect each node to this private network. OCFS will

also use this private network for synchronization purposes.

Two teamed NICs are recommended in order to ensure that the clusterinterconnect still works even if one connection fails. This requires two

switches, which must be connected with each other in order to detect a failed

connection.

The second NIC is required for the public network from the RAC point of

view. It should be used for all other networking needs like connecting SAP

application servers to the DB in 3-tier configurations or connecting front-end

users to the application servers in 2-tier configurations.

One or more additional NICs may be required when MSCS is used to provide

high availability for the SAP central instance.

Shared

Storage

Node 1 Node 2 Node N

Fiberchannel Switch


7/70

7/70

Hardware Configuration

Storage

The required number and size of shared disks must be discussed with the hardware

vendor. This depends on customers requirements.

The storage devices should be dual attached to each server in order to lower the riskof a single path failure. A high available connection to the storage devices can beprovided in conjunction with appropriate drivers (e.g. EMC Powerpath).

As a general rule there should be:

Three extended partitions (each >=500MB) for the Oracle Cluster Ready

Service (OCW) voting disks. These should be distributed over multiple

physical disks.

Two extended partitions (each >=500MB) for the Oracle Cluster Repository

(OCR). These should be distributed over multiple physical disks.

One extended partition for Oracle HOME (>=4GB)

At least one extended partition for the database files (depends on the size ofthe database)

Node 1 Node 2

Gigabit Ethernet

Private Network Switch

(Interconnect)

Node N

Public Network Switch


8/70

8/70

Network

Network Interfaces

Proper configuration of the network is one of the most important and mostdifficult tasks of the whole cluster configuration process. Extreme care and

attention must be taken when selecting and configuring network interface cards

and configuring network protocols and services. Furthermore, newly configured

clusters must be extensively tested and verified in order to ensure that all

components are working properly with each other. Incorrect or incomplete

network configuration can be very hard to diagnose and might cause serious

issues like hardware crashes, hangs or data loss!

General Rules for Network Configuration

All server NICs must be from the same vendor and must use the same

firmware and driver versions.

All servers should use the latest recommended NIC device drivers. NIC

settings must be re-checked after updating, installing or reinstalling a driver

because the new driver might override previously configured settings. If

available, vendor-specific instructions should be followed explicitly.

All NICs must use the same explicit speed and duplex settings. They should be

configured manually on both the switch ports and the NICs. Auto negotiating,

when enabled, can result in lost packets.

Public and private connects must be on separate subnets and network address

ranges. All switch ports must belong to the same subnet and/or VLAN!

The OS on each server node must have properly configured NIC priorities(network connections must specify public, then private, then unused adapters.

See Network Connections Advanced Advanced Settings-Connections

(priority order is top to bottom)

E.g.


9/70

9/70

For Windows 2003 Server, Microsoft recommends disabling IP media sensepolicy. Please refer to Microsoft Knowledge Base article # 239924 for more

information.

Although using teamed NICs is recommended in order to ensure a reliablecluster interconnect, extreme care should be taken when using teamed NIC

configurations. Not all NIC teaming solutions might be as stable and well-

tested as you might expect! Only solutions of proven reliability (e.g. NICs of

the Intel PROSet family) should be used for teaming NICs. Even then, the

implementation can be finicky. If any network problems arise, disable network

teaming and make sure the network is working properly without it.

If firewall software is installed on the cluster nodes, it should be disabled.

Care should be taken when installing MS Windows Service Packs. Service

packs might add new security features, such as an integrated firewall, that

might prevent cluster communications from working properly, thus causing

serious problems! It is strongly recommended that you use a dedicated server

as a firewall between the cluster nodes and the outside world instead of a

personal firewall solution.

If there is antivirus software installed on the cluster nodes, ensure that it does

not affect network communications. Many modern antivirus solutions alsoinstall network filter drivers that can interfere with proper networking. It is


10/70

10/70

strongly recommended that you do not install any antivirus software on the

cluster nodes and that you scan all software to be installed on another PC

dedicated for virus scanning.

Use human-readable names for your network interface like Public,

Private, Interconnect etc. Never use non-alphanumeric characters orspecial characters in their names. Allowed are A-Z, a-z and 0-9. If

this rule is ignored, you cannot install Oracle Clusterware successfully!

Please also refer to "Server Clusters: Network Configuration Best Practices for

Windows 2000 and Windows Server 2003" at

http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/clus

tering/clstntbp.mspx

TCP/IP Configuration

The private network to be used for the cluster interconnect should be a dedicated

switched (and if possible redundant) Gigabit Ethernet network. A sample network

configuration is shown below:

Node / NIC IP-Address Subnet Mask

1 / 1 1.1.1.1 255.255.255.0

1 / 2 IP for public network

2 / 1 1.1.1.2 255.255.255.0

2 / 2 IP for public network

Do not specify a default gateway for private interconnect. Only one defaultgateway must be configured - usually on the public network.

System Configuration

Operating System

Windows 2003 Server Standard or Enterprise Edition must be used on all nodes.

Make sure that the latest Windows service packs and patches have been installed on

all nodes.

Use the same users and passwords on all cluster nodes. This is required in order to be

able to copy files from one node to another during software installation. It is highly recommended that you use domain users only and that you add them to

the required local groups. See Oracle Administrator Groups.

Support of Windows 2003 Server

Please note that when creating partitions for use with OCFS, only Basic MBR Disks

are supported under Windows 2003.


11/70

11/70

Microsoft Cluster Software and Heartbeat

Microsoft Cluster Software can be used on the same cluster in order to provide high

availability for the SAP central instance. In this case, a dedicated network interface

must be used for the heartbeat.

When using MS Cluster for Windows 2003 in a three or more node cluster

configuration with heartbeat communication over a switch, it is important that the

switch is able to forward multi-broadcast messages. Otherwise, nodes may

disappear and re-appear in the cluster causing node evictions and other problems.

If the switch used does not support multi-broadcasts, you have to use single

broadcasts. Please refer to Microsofts Knowledgebase on how to do this.

Windows 2003 64-bit or 32-bit

Windows 2003 64-bit on x86_64 platforms

64-bit x86_64 systems are getting cheaper and cheaper. If there are no compelling

reasons why you should use a 32-bit based operating system it is strongly

recommended to use a 64-bit OS. There are many advantages to this, such as bigger

Oracle SGA, faster large memory access, larger OS memory pool sizes etc.

Windows 2003 32-bit on x86_32 platforms

Customers often use Windows special boot.ini switches like/3GB /PAE /AME to

support more than 2GB of memory for user applications. If these switches have to be

used on RAC cluster nodes, then Windows 2003 Server Enterprise Edition must beused instead of Windows 2000 Server as long Microsoft does not support the/userva

switch for Windows 2000 Server.

Please see next paragraph Tuning OS Memory for further details.

Tuning OS Memory

On Windows 2003 Server operating systems, the memory pool sizes differ greatly

when using /3GB /PAE /AME switches in boot.ini compared to when these switches

are not used.

Kernel Memory without the /3GB switch:

Non-paged Pool Memory Max =~ 256MB

Paged Pool Memory Max =~ 332MB

Free System PTEs =~ 450MB

Kernel Memory with the /3GB switch:Non-paged Pool Memory Max =~ 128MB

Paged Pool Memory Max =~ 256MB

Free System PTEs =~ 64MB


12/70

12/70

As you can see, the pool sizes are significantly reduced when using the /3GB

switches. The free system PTEs in particular are reduced dramatically. This can cause

several problems in RAC environments.

For systems using the /3GB switches, you should monitor free system PTEs using

perfmon to ensure that the number never drops below 20000. If the number of freesystem PTEs drops below 20000, you may have to deal with database instance crashes

due to insufficient free system PTEs.

It is therefore recommended that you use Windows 2003 Server, which supports the

/userva switch in boot.ini. Please refer to Microsofts support page and Knowledge

Base to find out if it is supported or not.

The /userva switch reduces the 3072MB available to user processes to the given value

and provides the memory gained to the free system PTE pool.

The recommended boot.ini parameters for such installations are:

/3GB /PAE /AME /Userva=3008

Please note that this reduces the available memory for user processes by 64MB and

gives the memory to the free system PTEs. The free system PTEs should still be

monitored to ensure that this reduction was enough, especially if other PTE-

consuming applications are also running on the same cluster node.

Please also refer to the following articles and notes for further information:

MS Knowledge Base#316739 How to use the /userva switch with the /3GB switch to tune the User-mode

space to a value between 2 GB and 3 GB

#810371 Using the /Userva switch on Windows Server 2003-based computers that

are running Exchange Server

Oracle Metalink#297498.1 Resolving Instance Evictions on Windows Platforms Due to OS 10055

Errors

Storage, Disks

OCW for Windows is installed together with OCFS and must be installed in aseparate Oracle Home on local disks, e.g. C:\oracle\ohcrs102.

OCW requires Voting disk volumes and Cluster Repository disk volumes.

Three voting disk volumes and two volumes are recommended for the cluster

repository (one is used as a mirror). These must be placed on OCFS volumes

as described later. All volumes must be created in extended partitions on a

basic disk. Dynamic disks are not supported! All disk volumes used by

Oracle Clusterware require drive letters!

For disks storing Oracle binaries and database files, extended partitions must

be created on a basic volume (not a dynamic volume) on each disk. All of


13/70

13/70

these partitions must be formatted using the ocfsformat tool, discussed later.

OCFS also supports mount points for OCFS volumes. They can also be mixed

with volumes with drive letters. It is NOT recommended that you create

any OCFS volumes manually before OCW is installed properly!

Before starting to install OCW, create the required volumes in the Disk

Administrator of Windows. Do NOT assign drive letters or mount points for

them and do NOT format the volumes. Drive letters are assigned during OCW

installation. The volumes will be formatted during OCW installation. Drive

letters that have already been assigned before OCW installation will not be

available for use as voting disks or home/mirror of the cluster repository. After

the volumes have been created, it is highly recommended that you reboot all the

cluster nodes to ensure that every node will see the changes made by the Disk

Administrator from one of the nodes.

E.g.:Purpose Disk/Partition Location FS-Type*

OCW Home Physicaldisk0/0 C:\oracle\ohcrs102 NTFS

Oracle HOME Physicaldisk1/0 M:\oracle\\102 OCFS

Oracle Database Physicaldisk2/0 N:\oracle\\sapdata. OCFS

Oracle Database Physicaldisk3/0 O:\oracle\\origlog. OCFS

Oracle Database Physicaldisk4/0 P:\oracle\\mirrlog. OCFS

OCW voting

disk volume 1

Physicaldisk1/1 Q:\ OCFS

OCW voting

disk volume 2

Physicaldisk2/1 R:\ OCFS

OCW voting

disk volume 3

Physicaldisk3/1 S:\ OCFS

OCW cluster

repository disk

volume

Physicaldisk3/1 T:\ OCFS

OCW cluster

repository disk

volume (mirror)

Physicaldisk4/1 U:\ OCFS

Oracle Database Physicaldisk5/0 C:\oracle\\sapdatax OCFS Volumemounted into an

NTFS directorySAP Physicaldisk6/0 W:\usr\ NTFS (it is not

supported to run

SAP from OCFS)

* If the SAP system and the database are to be installed on a cluster node from scratch

and not copied from a different server to one of the cluster nodes, some additional

installation steps apply.

In order to avoid problems with incompatibilities between sapinst and OCFS, you

must switch file systems at the relevant installation step by backing up the volume,

dropping the volume and then reformatting it with the appropriate file system.


14/70

14/70

Network Interfaces

Use static IP addresses for

the cluster interconnect (private network),

the public network,

the VIP,

and make sure that each cluster node can be reached from the other nodes by a

public hostname,

a private hostname,

a VIP hostname.

The hosts file under c:\windows\system32\drivers\etc should resolve the hostnames.

E.g.

127.0.0.1 localhost

192.168.228.1 oracx3

192.168.228.50 w2kracna w2kracna.vmware

192.168.228.51 w2kracnb w2kracnb.vmware

192.168.228.52 w2kracna-vip w2kracna-vip.vmware

192.168.228.53 w2kracnb-vip w2kracnb-vip.vmware

192.168.228.54 w2kracnc w2kracnc.vmware

192.168.228.55 w2kracnc-vip w2kracnc-vip.vmware

192.168.22.50 w2kracna-priv

192.168.22.51 w2kracnb-priv192.168.22.52 w2kracnc-priv

192.168.228.60 w2kdom w2kdom.vmware

192.168.22.60 w2kdom-priv w2kdom-priv.vmware

Do not start OCW software installation until all nodes have been properly

configured.

For production systems, you must use teamed, fault-tolerant NICs and redundant

switches to ensure that the RAC interconnect is highly available. Otherwise, a simple

switch failure might cause one or more nodes to fail.

If the network interface cards support interrupt throttling, you should keep the

hardware default. This will usually prevent very high interrupt rates during high

network loads in terms of many but small packets, as well as reduce the load on PCI

buses while not significantly delaying the delivery of the data to be transferred (small

latency).

Installation before Migration?

The next sections assume that there is already an existing SAP System with a single-

instance Oracle database to be migrated to an SAP System with multiple RACinstances.


15/70

15/70

These assumptions have been made because there is no standard installation

procedure for all of these components.

There are some obstacles that will cause the installation to fail, e.g. SAP installation

tools do not know anything about OCFS. They all assume NTFS as the underlying file

system to be used and try to set NTFS file system permissions (ACLs) for someOracle directories. As OCFS does not support ACLs, the whole setup procedure will

fail.

To avoid this and other problems, there is no direct installation path. You have to

install an SAP System and a single-instance Oracle database on NTFS and move all

Oracle files (Oracle Software and Oracle Database) to OCFS volumes later.

For a completely new system, first install the SAP System with the Oracle database as

described in SAP installation guides, and then continue with the migration steps

described in this document. If the source system is initially installed on one of the

cluster nodes, proceed as follows in order to avoid these obstacles:

1. Check all prerequisites.

2. Install Oracle Clusterware (OCW) while ALL (!!!) nodes are up and running

on a local disk, e.g. C:\

3. Create an OCFS volume and install the Oracle database software on this

volume while all nodes are up and running. This will ensure that the Oracle

Universal Installer makes important registry and environment settings.

4. Backup the whole volume.

5. Drop the OCFS volume using ocfsutil while all nodes are up and running.

Check that all nodes dropped it properly by trying to access it.

6. Shut down all the cluster nodes, except the one where the installation will be

performed.

7. Reformat the volume with NTFS and create the volumes needed for the

database and the SAP system.

8. Restore the backup from step 3.

9. Perform a standard SAP installation.

10. Back up all volumes (Oracle database software, Oracle database, SAP System)

and drop all NTFS partitions you created in step 7 except the one where the

SAP System was installed.11. Create OCFS volumes for all NTFS volumes dropped in step 10.

12. Restore all files from the previous backup(s).

Keep the drive letters assigned during the whole setup! Never boot another node

while one of the shared disk volumes contains an NTFS file system. This would cause

data corruption!

To switch an NTFS volume to an OCFS volume, proceed as follows:

1. Delete the volume using Disk Administrator

2. Create the volume using Disk Administrator without formatting it

3. Reboot all nodes and check that drive letters have been assigned correctly

4. Format the volume on one node using ocfsformat


16/70

16/70

To switch an OCFS volume to a NTFS volume, proceed as follows:

1. Use ocfsutil to delete the volume cluster wide (all nodes should be up and

running) and check that the volume is now inaccessible from all the nodes.

2. Shut down all nodes except one in order to prevent simultaneous access frommore than one node.

3. Format the volume with NTFS.

Software Installation & Configuration

Overview

The complete software stack to run RAC consists of the following components:

Oracle Clusterware (which also has its own software stack) Oracle RDBMS Software

The installation and upgrade of each component are described in the following

sections.


17/70

17/70

Oracle Clusterware (OCW)

Installation

Run setup.exe from your original Oracle 10.2.0.1 Clusterware distribution media.Enter a name for the Oracle Home of CRS and a local path where the Oracle Home

will be located.

The next screen shows the result of some prerequisite checks. Do not continue if any

of the checks fail, otherwise the installation may not work properly.


18/70

18/70

Specify the cluster configuration. Ensure that public, private and virtual hostnames

are correct.


19/70

19/70

Define how to use the installed NICs. If more than two NICs are displayed, e.g. when

using NAS (network attached storage) or SAN (storage area network) solutions, click

Edit and select Do not use for these NICs.

The next screen shows all volumes available for use by OCW. Select a volume and

press Edit to define how this volume shall be used. Please note that this screenshot

was taken on a demo system that uses only one voting disk and two OCR disks. In a

production environment, three voting disks are highly recommended.


20/70

20/70

Choose Format with OCFS, select the drive letter to assign, select Place voting

disk on this partition, and press OK


21/70

21/70

Specify that you want to use this volume as the primary location for the OCR.


22/70

22/70

Specify that you want to use this volume as the mirror location for the OCR.

Press Next after you have assigned all volumes properly.


23/70

23/70

One of the following warning messages will appear if you placed the primary cluster

repository and its mirror on the same physical disk, or if you have configured just one

voting disk. This is not recommended for production systems!


24/70

24/70

Start the installation by pressing, Install.

After all files have been copied, the installer runs a number of configuration

assistants. In some situations Virtual Private IP Configuration Assistant will fail asshown below.

If this is the case, press OK and NEXT to finish the installation. A second error

message will appear saying that one of the configuration assistants had a problem.


25/70

25/70


26/70

26/70

If the Virtual Private IP Configuration Assistant fails, then it should be run

manually. Open a command prompt window and start VIPCA by calling vipca.bat. It

is located in the bin directory of OCWs Oracle Home, e.g.

c:\oracle\ohcrs102\bin\vipca.bat.

Configure VIP as shown below.


27/70

27/70


28/70

28/70


29/70

29/70

Upgrade

After the installation, upgrade the Oracle Clusterware to the latest release shipped

with the latest patch set. Follow the instructions in the release information document

on how to do this.

Usually, you just have to stop the running CRS on each node by calling

crsctl stop crs

before starting the upgrade of OCW components. Note that stopping CRS may take a

few minutes. Check the absence of crsd.exe and evtd.exe in the Windows Task

Manager. If one of these processes did not terminate after 5 minutes it is necessary to

kill them using Windows Task Manager before proceeding with the upgrade.

The screenshots below demonstrate the mandatory upgrade from OCW 10.2.0.1 to

10.2.0.2!!!


30/70

30/70


31/70

31/70

Important: After installing OCW patches run the post-patch batch file

patch102.bat on every node. This may take a while. Do not run the script in parallel

on more than one node. E.g. C:\oracle\crs102\install\patch102.bat

Oracle CFS

As described earlier, CRS voting disks, primary OCR and mirror OCR locations aredefined and created using the OCW installation tool. OCFS volumes for Oracle

database software and all files belonging to the database should be created manually.

Choosing the right NIC for OCFS

You probably have to change the NIC that OCFS uses for synchronization. This is

done with the Ocfschangecommif tool. Run Ocfschangecommif /p on each node in

order to see which NIC is used. If it is not the right one, run Ocfschangecommif /c on

each node to change the NIC. Then reboot the nodes. If configured NICs are different,

OCFS will still work, but for better performance OCFS should be configured to use

the same NIC on all cluster nodes.

Setting up OCFS volumes

OCFS is not an entirely general-purpose file system. It is designed to support RAC

databases in a shared file system environment. It is highly recommended that you read

Appendix A: Oracle Cluster File System for Windows (OCFS) on how to use OCFS

in order to avoid problems due to administration errors.

OCFS IO-Modes

OCFS supports DirectIO-Mode and NondirectIO-Mode for volumes.

Rules for Formatting Volumes with OCFS

Please take care when deciding how to format a volume. There are some restrictions

on what may be stored depending on how a volume was formatted.

Type of data stored on OCFS volume IO-Modes allowed

ORACLE_HOME (Oracle executables etc.) non-direct

Datafiles non-direct

Archivelogfiles (saparch or oraarch) Direct

DB alertfiles and tracefiles (saptrace) Direct

DB controlfiles non-directOnline Redo Logfiles non-direct

To format a volume for DirectIO-Mode, use /d.

To format a volume for NondirectIO-Mode (default), just omit the /d.


32/70

32/70

Formatting Volumes

Ocfsformat is used to format new OCFS volumes.

The table below shows how to use Ocfsformat for different drive letters / links orDirectIO / NondirectIO modes:

Volume configuration Command / Example

Assign a drive letter to a

volume with NondirectIO-Mode

ocfsformat /l : /c


33/70

33/70

Migration

If there is an Oracle Home installed on the target system, this will have to be removed

first. During the installation described in the next section, some registry entries are

made on all of the cluster nodes. They are required to successfully run RAC on all thenodes. It is assumed that OCW is already installed and upgraded to the latest patch

set.

Oracle RDBMS Software

Installation

This step assumes that you have manually created an OCFS volume. The OCFS

volume should be visible from all cluster nodes.

Install Oracle 10.2.0.1 Enterprise Edition in a separate Oracle Home on the shared

OCFS volume. Only install the Oracle software. Do not create a database at this point!

When the Installation asks which nodes to install the software on, chooseCluster Installation and select all cluster nodes.

If you are using Oracle RDBMS CDs shipped by SAP, you will have to execute

setup.exe directly. SAPs documentation says to use a setup script that invokes


34/70

34/70

setup.exe with a special response file. Do not start setup.exe in this way because setup

based on the response file will not install the RAC option.

As already mentioned, it is assumed that a database that can be copied to the

cluster already exists on a source system, or that a new database will be installed on

one of the nodes before the migration starts.

Usually you might want to run the central instance (probably highly available by

using MS-Cluster) on the cluster together with the database. If this is the case, you

will have to install all SAP/CI related files on a separate NTFS volume as a cluster

resource later.

Upgrade

Get the latest patch set for Oracle 10gR2 and install it as described in the patch set

note that comes with it.

Database Conversion to RACIf you are converting an existing database to cluster hardware, it is recommended that

you copy the whole database to the new hardware maintaining the same names for

directories and file locations. If the existing database has not been upgraded to Oracle

10gR2, it should be upgraded to Oracle 10gR2 on the existing hardware before

moving it to the new hardware.

If this is not possible, there are two main ways to move the database:

Issue a shutdown normal on the source database, copy the database to the

target system, and then upgrade it to 10gR2. The advantage is that it is not

necessary to install 9.2 RDBMS software on the cluster. The upgrade can bedone immediately after copying. The disadvantage may be longer downtime

for the additional upgrade step and copying the database to the new cluster

hardware.

Set up a standby database on the target system and stop the source database

just before the upgrade. The advantage is that no downtime is needed for

copying. Disadvantages are: more work, more complex setup, the need to

install 9.2 RDBMS software for the recovery of the physical standby.

If it is not possible to put the database under the same paths as the source system start

the database instance (10gR2) on the target system into mount state and rename all

the datafiles and online redolog files before proceeding with the upgrade.Alternatively issue an alter database backup control file to trace on the source

system, modify the paths to match the new structure, and recreate the control files on

the target system using the same database version as the source system.

Enabling RAC

Undo Tablespaces

Create one undo tablespace for each node on the first instance. The undo tablespaces

should be as large as your current PSAPUNDO tablespace.


35/70

35/70

E.g.create undo tablespace PSAPUNDO1

datafile 'J:\oracle\T01\sapundo1\undo_1\undo1.data1'

size 100M autoextend on;

create undo tablespace PSAPUNDO2

datafile 'J:\oracle\T01\sapundo2\undo_2\undo2.data1'

size 100M autoextend on;

Online Redo Logs

Each RAC instance uses a specific thread for writing redo logs, thread 1 on a single

instance.

In an RAC environment, each instance has its own thread and its own online redo

logs. You therefore have to create new redologs. Its a good idea to implement the

corresponding thread number in their name.

E.g.alter database add logfile thread 2 group 24

('L:\ORACLE\T01\ORIGLOGB\LOG_G24M1_T2.DBF',

'K:\ORACLE\T01\MIRRLOGB\LOG_G24M2_T2.DBF')

size 20M;

You should add the same number of redo log groups and mirrors as in a single node

environment. The redo log sizes should also be the same.

If you have plenty of disk space, its a good idea to use this additional disk space

for more online redologs. This will reduce the chance that the system requires

archived redologs during crash recovery.

For all new redolog groups been added, the corresponding thread has to be enabled:

alter database enable thread 2;

Init.ora and Spfile.ora

1.

If you are using a pfile (init.ora), copy it to a working copy (e.g.init_work.ora) and continue with step 3.

2.

If you are already using a spfile.ora, you will have to transfer it to a pfile in order

to be able to make the necessary changes for RAC:

e.g.create pfile = init_work.ora from spfile;

3.

Modify the init_work.ora" and add the following lines for each instance:


36/70

36/70

.instance_name=''

.instancenumber=

.local_listener=

.thread=

.undo_tablespace=

E.g. assuming that your SID is T01, you have 2 instances:

*.cluster_database=TRUE

T011.instance_name='T011'

T012.instance_name='T012'

T011.instance_number=1

T012.instance_number=2

T011.local_listener='LISTENER_T01_W2KRACNA.world'

T012.local_listener='LISTENER_T01_W2KRACNB.world'

T011.thread=1

T012.thread=2

T011.undo_tablespace='PSAPUNDO1'

T012.undo_tablespace='PSAPUNDO2'

T011.remote_listener='REMOTE_LISTENER_T01_W2KRACNA.world'

T012.remote_listener='REMOTE_LISTENER_T01_W2KRACNB.world'

4.

Enable automatic undo management by setting

undo_management=auto

5.Save the modified init_work.ora and create a spfile.ora from it:

Create spfile from pfile = init_work.ora

Please note that it is not possible to start the database until Oracle Administrator

Groups and the Windows services for the instances have been created and the

SQL*Net and Services configuration is completed. This is described in the following

sections.

Oracle Administrator Groups

In the next step we will remove the installed OracleService on node 1, and then

create new windows services OracleService1 on node 1, OracleService2on node 2 and so on.

In addition, ensure that the Windows 2003 groups ORA__DBA

and ORA__OPER exist on all nodes and that the same users

belong to them.

E.g.

ORA_T011_DBA and ORA_T011_OPER on node 1 and ORA_T012_DBA and

ORA_T012_OPER on node 2 and so on.


37/70

37/70

Oracle Instance Creation

Check if there is already an Oracle Service (OracleService) running on Node 1.

If so, remove it:

Oradim delete sid

Then create a new one:

Oradim new sid

E.g.

Oradim new sid T011

On Node 2 create a new one:

Oradim new sid

E.g.

Oradim new sid T012

Environment Variables

On Node 1:

Set ORACLE_SID to 1

Set NLS_LANG to the right value for your system.

E.g. AMERICAN_AMERICA.WE8DEC for a Non-Unicode system.

Set DBS_ORA_TNSNAME to the service name to which the application server

should connect.E.g. T01_D00 (as defined in tnsnames.ora)

On Node 2:

Set ORACLE_SID to 2

Set NLS_LANG to the right value for your system.

E.g. AMERICAN_AMERICA.WE8DEC for a non-Unicode system.

Set DBS_ORA_TNSNAME to the service name to which the application server

should connect.

E.g. T01_D01 (as defined in tnsnames.ora)

SQL*Net Configuration and Service Creation

Major Differences between Oracle 9i and Oracle 10g Configurations

Unlike the RAC 9i / SAP integration, the TNS_ADMIN environment variable

pointing to node-specific directories for sqlnet.ora, listener.ora and

tnsnames.ora configuration files should not be set for a RAC 10g / SAP

installation. The SQL*Net configurations for all nodes are stored in one single

sqlnet.ora, listener.ora and tnsnames.ora under the

ORACLE_HOME/network/admin directory.


38/70

38/70

Make sure TNS_ADMIN is not set before starting netca!

Application Servers of course may use their own tnsnames.ora and sqlnet.ora

to connect to the database.

While the listener name was OracleTNSListener with Oracle 9i,Oracle 10g uses different node-specific names for the listener on each node.

With Oracle 9i, the server provided high availability through RAC and the

client decided which RAC instance to connect to by having multiple entries in

the address list in tnsnames.ora. Oracle 10g introduces services to which a

client connects. The client does not decide to which database instance it will

connect based on the supplied address list in tnsnames.ora and which instances

are available during connect time. Instead it connects to a service that can run

on one or more nodes. Although it is possible not to use services, it is highly

recommended to do so. A configuration without services is not discussed in

this whitepaper.

Oracle Net Configuration Assistant (netca)

Run netca to create a listener on all cluster nodes. As per convention, the names

should be LISTENER__. Furthermore, Netca not only creates

the required entries in listener.ora, it also creates the corresponding resources in the

OCR.

The screenshots below show a sample configuration:


39/70

39/70


40/70

40/70

As already mentioned above, the convention for listener names is

LISTENER__


41/70

41/70


42/70

42/70

When netca has finished configuring listeners, your listener.ora configuration file

should look like this:

Listener.ora

This is a sample configuration:

SID_LIST_LISTENER_T01_W2KRACNA =

(SID_LIST =

(SID_DESC =

(SID_NAME = PLSExtProc)

(ORACLE_HOME = R:\oracle\T01\102)

(PROGRAM = extproc)

)

)

SID_LIST_LISTENER_T01_W2KRACNB =

(SID_LIST =

(SID_DESC =(SID_NAME = PLSExtProc)

(ORACLE_HOME = R:\oracle\T01\102)

(PROGRAM = extproc)

)

)

LISTENER_T01_W2KRACNA =

(DESCRIPTION_LIST =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = w2kracna-vip)(PORT =

1527)(IP = FIRST))

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.228.50)(PORT =

1527)(IP = FIRST))

)

)


43/70

43/70

LISTENER_T01_W2KRACNB =

(DESCRIPTION_LIST =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = w2kracnb-vip)(PORT =

1527)(IP = FIRST))

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.228.51)(PORT =1527)(IP = FIRST))

)

)

Tnsnames.ora

This is a sample configuration:

### LOCAL LISTENER ENTRIES ###

### only a single address is allowed here

LISTENER_T01_W2KRACNA.WORLD =

(DESCRIPTION =(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = w2kracna-vip)(PORT = 1527))

)

(CONNECT_DATA =

(SERVICE_NAME = LISTENER_T01_W2KRACNA.WORLD)

)

)

### only a single address is allowed here

LISTENER_T01_W2KRACNB.WORLD =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = w2kracnb-vip)(PORT = 1527))

)

(CONNECT_DATA =

(SERVICE_NAME = LISTENER_T01_W2KRACNB.WORLD)

)

)

### REMOTE LISTENER ENTRIES ###

### add more address entries for remote listeners if necessary

REMOTE_LISTENER_T01_W2KRACNA.WORLD =

(DESCRIPTION =

(ADDRESS_LIST =


)

(CONNECT_DATA =

(SERVICE_NAME = REMOTE_LISTENER_T01_W2KRACNA.WORLD)

)

)

REMOTE_LISTENER_T01_W2KRACNB.WORLD =

(DESCRIPTION =

(ADDRESS_LIST =


)

(CONNECT_DATA =

(SERVICE_NAME = REMOTE_LISTENER_T01_W2KRACNB.WORLD)

)

)


44/70

44/70

### SERVICE ENTRY FOR SAP CENTRAL INSTANCE ###

T01_D00.WORLD =

(DESCRIPTION =

(ADDRESS_LIST =



)(CONNECT_DATA =

(SERVICE_NAME = T01_D00)

(GLOBAL_NAME = T01.WORLD)

(FAILOVER_MODE =

(TYPE = SELECT)

(METHOD = BASIC)

)

)

)

### SERVICE ENTRY FOR CONNECT TO INSTANCE T011 ###

T011.WORLD =

(DESCRIPTION =(ADDRESS_LIST =



)

(CONNECT_DATA =

(SERVICE_NAME = T011)


(FAILOVER_MODE =

(TYPE = SELECT)

(METHOD = BASIC)

)

)

)

### SERVICE ENTRY FOR CONNECT TO INSTANCE T012 ###

T012.WORLD =

(DESCRIPTION =

(ADDRESS_LIST =



)

(CONNECT_DATA =

(SERVICE_NAME = T012)


(FAILOVER_MODE =

(TYPE = SELECT)

(METHOD = BASIC)

)

)

)

### DEFAULT SERVICE ENTRY FOR CONNECT TO ANY INSTANCE ###

T01.WORLD =

(DESCRIPTION =

(ADDRESS_LIST =



)

(CONNECT_DATA =

(SERVICE_NAME = T01)(GLOBAL_NAME = T01.WORLD)


45/70

45/70

(FAILOVER_MODE =

(TYPE = SELECT)

(METHOD = BASIC)

)

)

)

EXTPROC_CONNECTION_DATA =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC0))

)

(CONNECT_DATA =

(SID = PLSExtProc)

(PRESENTATION = RO)

)

)

Sqlnet.oraThis is a sample configuration:

AUTOMATIC_IPC = ON

TRACE_LEVEL_CLIENT = OFF

NAMES.DEFAULT_DOMAIN = WORLD

SQLNET.EXPIRE_TIME = 0

SQLNET.AUTHENTICATION_SERVICES = (NTS)

DEFAULT_SDU_SIZE=32768

Putting the Database and Instances under CRS control

After completing the steps described above, the database instances can be startedusing sqlplus. The next step is to put the database and database instances under CRS

control. CRS will then monitor these resources and can take actions if one or more of

these resources fail.

1. Add the database resource to the OCR

e.g. srvctl add database -d T01 -o R:\oracle\T01\102

2. Add the database instances to the OCR

e.g. srvctl add instance -d T01 -i T011 -n w2kracna

srvctl add instance -d T01 -i T012 -n w2kracnb

3. Activate the database resource

e.g. srvctl start database -d TST

4. Check that the database resource and the instances have come online.

e.g. crs_stat t


46/70

46/70

Name Type Target State Host

-----------------------------------------------------------------------------------

ora....11.inst application ONLINE ONLINE w2kracna

ora....12.inst application ONLINE ONLINE w2kracnb

ora.T01.db application ONLINE ONLINE w2kracnaora....NA.lsnr application ONLINE ONLINE w2kracna

ora....cna.gsd application ONLINE ONLINE w2kracna

ora....cna.ons application ONLINE ONLINE w2kracna

ora....cna.vip application ONLINE ONLINE w2kracna

ora....NB.lsnr application ONLINE ONLINE w2kracnb

ora....cnb.gsd application ONLINE ONLINE w2kracnb

ora....cnb.ons application ONLINE ONLINE w2kracnb

ora....cnb.vip application ONLINE ONLINE w2kracnb

ServicesAfter configuring SQL*Net, the services have to be defined in OCR. For services to

which SAP application servers will connect, our naming convention is:

_, e.g. T01_D00 as service name for the central instance of

T01.

Each Oracle RAC instance must have a service named like the instance itself.

If the instance name is T011, then the service name must be T011. This

service must be configured to run on this instance only.

E.g.srvctl add service -d T01 -s T011 -r "T011"

srvctl add service -d T01 -s T012 -r "T012"

Each SAP instance must have its own service for instance T01_D00, for the

central instance of T01, T01_D01, for an additional application instance and

so on. Never configure a service for SAP instances to run on more than

one Oracle instance at the same time! This can cause problems in the SAP

system such as aborted transactions, performance issues etc.

E.g.

Lets assume service T01_D00 should be run preferably on instance T011 butmay also run on instance T012.

srvctl add service d T01 s T01_D00 r T011 a T012

Once you have defined the services, start them with:

Srvctl start service d -s

E.g.

Srvctl start service d T01 s T011

Srvctl start service d T01 s T012Srvctl start service d T01 s T01_D00


47/70

47/70

Checking the Setup

After the above services have been defined and started, check the configuration. Some

of the most important checks are listed below:

Check that all OCR resources are online

R:\oracle\T01\102\NETWORK\ADMIN>crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora....011.srv application ONLINE ONLINE w2kracna

ora....T011.cs application ONLINE ONLINE w2kracna

ora....11.inst application ONLINE ONLINE w2kracna

ora....012.srv application ONLINE ONLINE w2kracnb

ora....T012.cs application ONLINE ONLINE w2kracnb

ora....12.inst application ONLINE ONLINE w2kracnb

ora....011.srv application ONLINE ONLINE w2kracnaora...._D00.cs application ONLINE ONLINE w2kracna

ora.T01.db application ONLINE ONLINE w2kracna

ora....NA.lsnr application ONLINE ONLINE w2kracna

ora....cna.gsd application ONLINE ONLINE w2kracna

ora....cna.ons application ONLINE ONLINE w2kracna

ora....cna.vip application ONLINE ONLINE w2kracna

ora....NB.lsnr application ONLINE ONLINE w2kracnb

ora....cnb.gsd application ONLINE ONLINE w2kracnb

ora....cnb.ons application ONLINE ONLINE w2kracnb

ora....cnb.vip application ONLINE ONLINE w2kracnb

Reboot the cluster nodes and check that all services start properly.

From each node, use the tnsping utility to ping all configured services (e.g.tnsping T01, tnsping T011, tnsping T012, tnsping T01_D00).

Try to connect to every service using sqlplus on each node (e.g. sqlplus

/@T01, sqlplus /@T011, sqlplus /@T012, sqlplus /@T01_D00).

Check that database instances stop and start as expected using srvctl utility,

which is described in the next section.

Check that relocating your services to where SAP connects to works asexpected.

Check the alert logs for errors.


48/70

48/70

Srvctl and crsctl

Most SAP / Oracle administrators use BR*Tools or/and sqlplus to start and stop

Oracle instances. In 10g RAC, OCW controls the state of each cluster resource.

Oracle instances, listeners, VIPs and services are cluster resources controlled byOCW. Because of this, an administrator should not use BR*Tools or sqlplus without

notifying OCW when starting or stopping a database instance.

E.g. If the administrator were to shut down a database instance using sqlplus and

OCWs target were to set this resource online, CRS would start this instance

immediately having recognized its offline state. The same applies for BR*Tools!

Because of this, Oracle recommends that you use OCW tools srvctl, crsctl and

crs_stat to control these resources.

The following table lists the commands most frequently used with OCW:

Command Purpose

Srvctl stop database d -o

immediate

Stops all database instances and takes the

database offline in CRS

Srvctl start service d Starts all services. This implies setting

the database and instance resources

online.

Srvctl start service d -s Starts the specified service

Srvctl stop service d -s Stops the specified service

Srvctl relocate service d -s -i -t

-f

Forces a disconnect of all clientscurrently connected to the instance

providing the specified service and

relocates the specified service to another

instance. Always use f (=force) option toensure all disp+work processes are connected

to the same instance. If you omit -f, only newconnections will connect to the instance

running that service.Crs_stat t Displays a table of all CRS resources

with their current state and their target

state

Refer to CRS documentation for a complete reference.


49/70

49/70

Windows 2003 and NTBackup

Windows 2003s NTFS introduced a new feature called volume shadow copy,

which allows files to be backed up while in use. This feature is not supported by

OCFS and must therefore be disabled in Windows 2003 NT-Backup when backing updata on OCFS file systems using this tool. Otherwise, the backup will fail with an

error.

Backing up OCR

Backing up the Oracle Cluster Repository is very important. Losing OCR will

definitely mean unplanned downtime. Please refer to the standard Oracle

documentation Oracle Real Application Clusters Administrators Guide 10g Release

2 for further information.

Backup

The backup strategy to be implemented may become quite complicated depending on

the customers requirements.

There are currently two tested and supported ways of backing up.

Split-mirror (e.g. by using EMCs BCV technology) for the databasetogether with br*tools and Legato Networker for the archives

BR*Tools with Oracle Recovery Manager (RMAN). This is the

recommended option for most customers where the backup time is not critical

because of the size of the database.

It is very important to understand that not every backup tool for backing up

Oracle databases can work with Oracle databases on OCFS! Most backup tools

assume that the database resides on NTFS and not on OCFS.

A sample BR*Tools configuration for the first option can be found inAppendix

C: BR*Tools Sample Configuration for Legato Networker.

Additional scripts are required to successfully implement a backup method. The

customers storage partner should be involved in the implementation of these scripts.

As a general rule, each backup and restore scenario that is not based on BR*Tools

with RMAN has to be treated as an implementation project which may require

development of scripts and will require testing.


50/70

50/70

Testing

Failures

One of the most important things before going into production with a new clusterenvironment is testing. Only by testing the behaviour of the cluster by simulating

hardware and software failures ensures that it will react as expected when a real

failure occurs. The following two sections describe some basic tests that should be

done. It is highly recommended not to go into production if those tests do not show

the expected results. The list is not complete and customers may think about

additional tests that have to be done for their specific cluster environment and HA

requirements.

Hardware Failures

Power Loss or Complete Failure of Cluster NodeAction

One of the most likely failures that may occur is a crash of a cluster node. Simulate a

crash by issuing a hard power-off for one of the cluster nodes.

Expected Behaviour

One of the remaining nodes should do a crash recovery for the crashed node.

Depending on the services definitions and their failover policy the services may be

activated on one of the remaining nodes. The whole failover may take some minutes.

Check if the failover process is finished with crs_stat t. Check that one of the

remaining nodes finished the crash recovery by viewing the alert.logs. Finally the VIPof the failed cluster node and all services should be available as defined.

Action

Power on the shutdown cluster node.

Expected Behaviour

The VIP of the started cluster node that was serviced by one of the remaining cluster

nodes should fail back to the rebooted cluster node. All defined Oracle instances (if

more than one database was running on the node before) should start automatically.

All services that fell over to other cluster nodes should stay there. They have to be

moved to their corresponding nodes using srvctl relocate service..

Oracle Interconnect Failures

Cluster node has lost its connection to the switch

Action

Disconnect one cluster node from the switch by unplugging all network cables of the

Oracle Interconnect. If you have NIC teaming active (redundant interconnect) also

check what happens when you unplug only one network cable.

Expected Behaviour

In a two node RAC cluster one of both nodes should crash. In a three or more node

RAC cluster the node where you did unplug the network cables should crash. In both

situations the VIP should move to one of the remaining nodes. One of the remaining


51/70

51/70

instances should do the crash recovery for the crashed node. Services should move to

the remaining nodes as defined.

(Multiple) Switch Failures with all nodes loosing their connection

Action

Turn off or disable the network switch of the Oracle Interconnect.Expected Behaviour

Regardless of the number of cluster nodes all nodes except one should crash. The

remaining node should takeover the VIPs of the crashed nodes, perform a crash

recovery and provide the database services as defined.

Storage Failures

One cluster node looses its storage connection

Action

Unplug one of the cluster nodes from the storage. If you have redundant paths to the

storage also check what happens when you unplug only one (fiberchannel-) cable.Expected Behaviour

The node should crash without automatic reboot. Automatic reboot may be a

configuration option in the BIOS of the node.

All cluster nodes lose their storage connection

Action

Unplug all cluster nodes from the storage or turn off all switches to the storage.

Expected Behaviour

All cluster nodes should crash. After reconnecting the nodes to the storage and

rebooting all the nodes the database should be available as expected.

Software Failures

Shutdown abort

Action

Shutdown an Oracle instance with shutdown abort.

Expected Behaviour

The cluster node should stay up and running. One of the remaining Oracle instances

should do the crash recovery. All services of the stopped instance should move to the

previously defined remaining instances. crs_stat utility should display OFFLINE

as the target state for the stopped instance.

Kill oracle.exe

Action

Kill oracle.exe with the Windows Task Manager.

Expected Behaviour

The cluster node should stay up and running. One of the remaining Oracle instances

should do the crash recovery. All services of the killed instance should move to thepreviously defined remaining instances. CRS should start the killed instance


52/70

52/70

automatically. All services that moved to other instances should stay there. They have

to be relocated manually.

SAP Application Workload

The most important tests that have to be run are workload tests where customersshould try to run their SAP applications under simulated business conditions. Testing

the performance of core transactions and batchjobs will help to resolve bottlenecks

and configuration issues before going into production with your new cluster

environment. These tests should also be combined with some of the failure test

scenarios described earlier.

Check if local update dispatching (described in the next chapter) is active and

working as expected with dpmon.exe or SAP Management Console. Both tools

show the number of times an update process was active.

Check if database service relocation works as expected and every SAP

applicationserver can connect to its service.

Additional Performance Recommendations

Use the ASSM tablespace feature!

Use local update dispatching for SAP update-processes

Turn off automatic offlining of undo segments by SMON

Use table-multiplexing for VB-tables

Automated Segment Space Management (ASSM)

In RAC environments it is highly recommended that you use Oracles ASSM feature

for optimal performance of space management functions. Unfortunately, most SAP

systems do not use this feature at all, and only new installations use it as default.

Nevertheless, it is a good idea to move all tables to ASSM tablespaces step by step

(e.g. when doing reorganizations). Please refer to your Oracle documentation to learn

how to use ASSM.

Local Update Dispatching

The SAP application does a lot of the update work on the database asynchronously

through update-processes. A dialog work process inserts an update job into tables

VBHDR, VBDATA, VBMOD and posts the dispatcher on the central instance to

select an update-process to actually perform the update job. This update-process may

be connected to another RAC instance, similarly to the dialog-process. If this is the

case, the update-process would read the data just written by the dialog-process. To

satisfy this read request, all required database blocks have to be shipped over the

interconnect from the instance where the data was inserted to the instance where the

data has to be read. This produces unnecessary additional data load and can be

avoided.

In order to avoid these non-local updates, it is recommended that you:


53/70

53/70

have a number of update-processes on each SAP instance

turn off update-dispatching and use the local update-processes

To turn off update dispatching, set rdisp/vb_dispatching to 0 as described in SAP note

#62203. Also, set rdisp/vb_name to the name of the local instance (e.g. rdisp/vb_name= app1_C11_00).

Automatic Offlining of Undo Segments by SMON

Onlining and offlining of undo segments is an operation performed dynamically by an

Oracle instance. These operations are serialized over all Oracle instances with the US

enqueue lock. This serialization can cause wait situations for other enqueue locks (e.g.

TX locks) when SMON starts offlining unused undo segments while some Oracle

shadow processes are onlining undo segments because of increased workload.

To stop SMON from offlining undo segments, set event 10511 in SPFILE.ORA and

restart your database instances.

e.g. *.event="10511 trace name context forever, level 2"

It is highly recommended that you set this event for production systems.

Appendix A: Oracle Cluster File System for Windows (OCFS)

IntroductionUsually single-hosted file systems like NTFS or FAT assume they are the only onesaccessing a volume once they have mounted it. Therefore they do not have to pay

attention to other hosts that modify data and metadata in the same file system. Using

disks with these file systems on more than one host would corrupt them pretty fast

because neither host would pay attention to what the others are doing.

OCFS provides mechanisms to support concurrent access to shared volumes from

multiple hosts. This section also provides some background information on OCFS

probably not mentioned in other documents.

Mount and Volume Recognition Process

Windows usually loads file system drivers while booting. This does not mean that a

volume is mounted immediately. In fact, mounting is usually delayed until a program

or the OS itself accesses a volume for the first time. When this happens, Windows

sends a message to each file system driver telling it to mount a specific volume. The

file system driver in turn accesses the volume and tries to recognize the file system. If

the file system is recognized, it completes the mount request and further IO requests

are sent to that file system driver.

Mounting can be (and usually is) even more complex. Usually, a file system driver

not only performs file system recognition before responding with Success to the

OS. It may also check whether the volume is dirty (which means the system was not


54/70


55/70


56/70

56/70

1. Set the volume offline on all cluster nodes:

ocfsutil /c OfflineVol /m : or

2. Check that the volume has been offlined properly by trying to access it. Do

this on all nodes.

3. Delete the volume from OCFS configuration:

ocfsutil /c DeleteVol /m : or

Split-Mirror Backup and Restore under OCFS

Split-mirror backup techniques like EMCs BCV technology can be used to create

online backups of an Oracle database. This is also true if the database resides on

OCFS volumes. The procedure for backing up a database is the same as for non-RAC

Oracle databases on NTFS file systems.

Restoring an online backup that was created using a split-mirror technique is

more complex. Some storage systems may not be able to handle the disks in a

way required by OCFS. In this case split-mirror cannot be used!

As already stated, OCFS stores the state of a volume (online or offline) on the volume

itself. When doing a split-mirror backup, this online state is saved like all other data

on the disk. Before a volume can be restored, it is necessary to set the volume in

question offline on all cluster nodes.

When the volume is restored (copied sector by sector!), the volume state (= online!!!)

will also be restored. This may cause OCFS to start mounting the volume in question

before the restore has finished. This can cause data corruption on the volume!

To do a safe restore, the storage device has to support offlining LUNs. On EMCstorage devices this is possible by using a command line tool that can also be called

by a shell script.

The trick is to set the LUN offline after the volume has been set offline with ocfsutil

on all cluster nodes. This will set the device to a not ready state and prevent

automatic mounting until the LUN has been set back online again.

Appendix B: FAQ

The following list of Questions and Answers is a compilation of some questions

from partners and customers that want to implement or already have implementedSAP with Oracle RAC on Windows.

Q: What are the differences between OCFS for Windows and OCFS1/OCFS2 for

Linux ?

A: The filesystems are completely different! While OCFS on Linux is available as

two different versions OCFS1 and OCFS2, OCFS on Windows is available as OCFS

9.2 and OCFS 10.2. All filesystem versions are incompatible with each other, so

basically no volume created with one filesystem type can be mounted using another

filesystem driver. Although OCFS 9.2 can be upgraded to 10.2 this is not

recommended. It is not possible to use different OCFS filesystems at the same time.


57/70

57/70

Q: Is it possible to install SAP (SAP binaries and Oracle database that is created by

sapinst and loaded with R3load) on OCFS for Windows ?

A: No! As described in one of the previous chapters sapinst does not know about any

other filesystem than NTFS. OCFS does not support ACLs and the installation will

fail if one tries to install directly to OCFS volumes.

Q: Is it allowed or possible to share CRS home on an OCFS volume like on Linux ?

A: No! On Windows CRS has to be installed on a local disk (e.g. C:\).

Q: Is it possible to run more than one database on a RAC cluster ?

A: Yes. As long as each database has its own ORACLE_HOME! It is not supported to

share an ORACLE_HOME to run different databases, but multiple nodes have to

share the ORACLE_HOME for the same database.

Q: Is it possible to use symbolic links for those volumes that contain Oracle Cluster

Repository (OCR) and voting disks?

A: No. The standard installation of CRS does only support driveletters.

Q: Is it possible to use symbolic links for ORACLE_HOME and database files ?

A: Yes. This is described in an earlier chapter of this document.

Q: Is it possible to re-install CRS ?

A: Yes, but it is a little bit tricky to do it the right way! Before de-installing CRS

using Oracle Universal Installer it is necessary to delete the OCFS volumes where

CRS stores its OCR and voting files. The steps are: Stop CRS on every node. Use

ocfsutil /c DeleteVol /m to delete the volumes. De-install CRS using Oracle

Universal Installer. It is very important to delete the partitions first because otherwise

the installation procedure will install and start OCFS, OCFS will mount the volumes

and then the installation procedure will format the volumes while the volumes are

already mounted. This would cause data corruption!!!


58/70

58/70

Appendix C: BR*Tools Sample Configuration for Legato Networker

In order to be able to back up the database using BR*Tools from different nodes, each

node needs its own BR*Tools configuration file. These configuration files are only

needed to start BR*Tools. The actual configuration that will be used can come from ashared configuration file. In the examples below, initT011.sap and initT012.sap are

the dummy configuration files needed to start BR*Tools. The actual configuration

file is initT01arch.sap, which should be provided as a command line parameter to

BR*Tools. It also includes a file called initT01arch.utl that contains Legato

Networker specific parameters for backing up the archives. All three *.sap files are

basically the same, so only one is printed below.

initT011.sap:

initT012.sap:

initT01arch.sap:# @(#) $Id: //bas/BIN/src/ccm/rsbr/initNT.sap#1 $ SAP

########################################################################

# #

# SAP backup sample profile. #

# The parameter syntax is the same as for init.ora parameters. #

# Enclose parameter values which consist of more than one symbol in #

# double quotes. #

# After any symbol, parameter definition can be continued on the next #

# line. #

# A parameter value list should be enclosed in parentheses, the list #

# items should be delimited by commas. ## There can be any number of white spaces (blanks, tabs and new lines) #

# between symbols for parameter definition. #

# #

########################################################################

# backup mode [all | all_data | full | incr | sap_dir | ora_dir

# | | | -

# | | ()]

# default: all

backup_mode = all

# restore mode [all | all_data | full | incr | incr_only | incr_full

# | | | -

# | | ()]

# redirection with '=' is not supported here - use option '-m' instead

# default: all

restore_mode = all

# backup type [offline | offline_force | offline_standby | offline_split

# | offline_stop | online | online_cons | online_split]

# default: offline

backup_type = online

# backup device type

# [tape | tape_auto | tape_box | pipe | pipe_auto | pipe_box | disk

# | disk_copy | disk_standby | stage | stage_copy | stage_standby

# | util_file | util_file_online | rman_util | rman_prep]

# default: tape

backup_dev_type = util_file

# backup root directory [ | ()]


59/70

59/70

# default: %SAPDATA_HOME%\sapbackup

backup_root_dir = B:\oracle\T01\sapbackup

# stage root directory [ | ()]

# default: value of the backup_root_dir parameter

stage_root_dir = B:\oracle\T01\sapbackup

# compression flag [yes | no | hardware | only]# default: no

compress = no

# compress command

# first $-character is replaced by the source file name

# second $-character is replaced by the target file name

# = .Z

# for compress command the -c option must be set

# recommended setting for brbackup -k only run:

# "%SAPEXE%\mkszip -l 0 -c $ > $"

# no default

compress_cmd = "G:\usr\sap\T01\sys\exe\run\mkszip -c $ > $"

# uncompress command

# first $-character is replaced by the source file name

# second $-character is replaced by the target file name

# = .Z

# for uncompress command the -c option must be set

# no default

uncompress_cmd = "G:\usr\sap\T01\sys\exe\run\uncompress -c $ > $"

# directory for compression [ | ()]

# default: value of the backup_root_dir parameter

compress_dir = B:\oracle\T01\sapreorg

# brarchive function [save | second_copy | double_save | save_delete

# | second_copy_delete | double_save_delete | copy_save

# | copy_delete_save | delete_saved | delete_copied]

# default: savearchive_function = save_delete

# directory for archive log copies to disk

# default: first value of the backup_root_dir parameter

archive_copy_dir = B:\oracle\T01\sapbackup

# directory for archive log copies to stage

# should contain subdirectory

# default: first value of the stage_root_dir parameter

archive_stage_dir = B:\oracle\T01\sapbackup

# new database home directory for disk_copy | disk_standby

# no default

# new_db_home = H:\oracle\C11

# stage database home directory for stage_copy | stage_standby

# default: value of the new_db_home parameter

# stage_db_home = /oracle/C11

# original database home directory for split mirror disk backup

# no default

# orig_db_home = /oracle/C11

# remote host name

# no default

# remote_host =

# remote user name

# default: current operating system user# remote_user =

wp o10g rac config win 303805

Documents