rac on windows 2003

1 © 2006 Julian Dyke

RAC on Windows 2003

Julian Dyke

Independent Consultant

Web Version

juliandyke.com

2

© 2006 Julian Dykejuliandyke.com

Introduction This presentation describes Windows specific RAC

configuration issues including:

Installation Pre installation steps Installing Oracle Clusterware Installing Oracle Database Software Patches Post installation steps

Administration Services Networking

3


Warning This presentation is intended to supplement information in the

Oracle documentation

It is NOT intended to replace the Oracle documentation

Most of the information contained in this presentation was correct for Oracle 10.2 Some information applies to earlier releases Where appropriate this is highlighted in the slides

4


Overview Oracle RAC is available on Windows in

Standard Edition Enterprise Edition

Standard Edition must use Automatic Storage Management (ASM)

Enterprise Edition can use Automatic Storage Management (ASM) Oracle Cluster File System (OCFS)

5


Shared Oracle Home Oracle Clusterware (CRS) MUST be installed in a local

directory on each node

Oracle Database Software can be installed in a shared Oracle home on the cluster file system

6


Installation

7


Pre-InstallationSteps

8


Pre-Installation Steps Pre-Installation tasks include:

Check public interface is first in bind order Disable Windows Media Sensing for TCP/IP Disable Write Caching on Shared Storage Enable Auto-mounting Local drives must be accessible from all nodes Environment variables System clocks must be synchronized

9


Public Interface Bind Order The public interface must be the first in the bind order

To check this Start Windows Explorer Right click on My Network Places and choose Properties In the Advanced menu click Advanced Settings... In the Adapters and Bindings tab

If the public interface is not the first name listed Click the arrow to move it to the top of the list Click OK to save the setting

10


Windows Media Sensing for TCP/IP Windows Media Sensing for TCP/IP must be disabled To check this

Run REGEDT32.EXE Navigate to HKEY_LOCAL_MACHINE\System\

CurrentControlSet\Services\Tcpip\Parameters

Add the following registry entry: Right click on right hand pane and select

New -> DWORD value

Value Name: DisableDHCPMediaSenseValue: 1

NOTE - It is recommended that you backup the registry before making changes using REDEDT32.EXE / REGEDIT.EXE

11


Write Caching Write Caching must be disabled on shared storage

To disable write caching at operating system level:

Start->Settings->Control Panel->Administrative Tools-> Computer Management->Device Manager->Disk Drives

Expand the disk drives list Double-click the first drive listed

On the Disk Properties tab uncheck the write cache enabled option

Repeat for all remaining disk drives

Not necessary if using HP Array Manager as write caching is automatically disabled at operating system level

12


Disk Manager To start the Disk Management tool:

Start->Administrative Tools->Computer Management->Storage->Disk Management

Alternatively the Disk Management tool can be started from the command line using:

C:> DISKMGMT.MSC

13


Auto-Mounting Auto-mounting must be enabled when using

RAC with raw partitions RAC with cluster file system Oracle Clusterware Logical drives for ASM

Auto-mounting must be enabled on each node in the cluster

To check if auto-mounting is enabled use:

C:> DISKPARTDISKPART> AUTOMOUNT

Automatic mounting of new volumes disabled

14


Auto-Mounting To enable auto-mounting use:

C:> DISKPARTDISKPART> AUTOMOUNT ENABLE

Automatic mounting of new volumes enabledDISKPART> EXIT

Repeat on each node in the cluster

Restart all nodes after configuring this setting

15


Local Drive Accessibility Local drives must be accessible from all nodes

To check local drive accessibility: For each node

For each local drive to be used by the Oracle installation

Use NET USE command to confirm accessibility

For example if Oracle will be installed on E$ in a two-node cluster containing nodes RAC1 and RAC2

On RAC1:

NET USE \\RAC2\E$

NET USE \\RAC1\E$

On RAC2:

16


Environment Variables On each node the following environment variables must be

set TEMP TMP

By default these are set to %USERPROFILE%\Local Settings\Temp

Oracle recommends these are redefined as follows TEMP=C:\TEMP TMP=C:\TMP

These settings should be the same on each node

17


Time Synchronization It is recommended that the system clocks are synchronized

between all nodes in the cluster appears to be less important in Oracle 10g Release 2 than

in previous RAC versions To synchronize the system clocks enable the Windows Time

Service on one node

The current time server for a node can be discovered using:

NET TIME /QUERYSNTPThe current SNTP server is: node2

The current time server for a node can be set using:

NET TIME /SETSNTP:<hostname> For example:

NET TIME /SETSNTP:SGHRAC1

18


Time Synchronization The current time on a node can be discovered using

NET TIME \\NODE1 /SETCurrent time at \\NODE1 is 20/02/2006 10:56

The current local clock is 20/03/2006 10:54Do you want to set the local computer's time to match the time at \\node1? (Y/N) [Y]: yThe command completed successfully.

To initially synchronize the time use

NET TIME \\NODE1 Current time at \\NODE1 is 20/02/2006 10:50

19


Oracle ClusterwareInstallation

20


Oracle Clusterware Installation To install Oracle Clusterware

Login as Local Administrator only

The documentation states that Clusterware can be installed using Domain Administrator Installation currently fails when attempting to start

Clusterware services

21


Oracle Clusterware Installation In Oracle 10.1 Oracle Clusterware was called Cluster Ready

Services (CRS)

In Oracle 10.1 to set up CRS run CRS/SETUP.EXE

DO NOT run CRS/INSTALL/SETUP.exe This will cause the installation to fail. See Metalink Note 277688.1 - CRS Install: Failed to locate

service OracleCSSService on second node

Also make sure that the install path does not include any spaces

In Oracle 10.2 there is only one SETUP.EXE program

22


Oracle Clusterware Installation During installation the OracleClusterPreInstService is

installed on each node to provide a list of available drives

This service uses ORACLECLUSTERPREINSTSERVICE.EXE

23


Oracle Clusterware Installation To manually delete a failed clusterware installation:

Stop and delete all Oracle services. For example:

SC STOP OracleCRServiceSC DELETE OracleCRService

Delete contents of Oracle Clusterware home directory

If required delete Oracle inventory directory C:\Program Files\Oracle

Use REGEDIT to remove Oracle keys from registry HKEY_LOCAL_MACHINE\Software\Oracle

Use Oracle Object Manager to delete links to OCR and Voting disk

24


Oracle Object Manager If using raw devices

Oracle Object Manager is used to create links for OCR Voting Disk

Pathname is %ORACLE_HOME%\bin\GUIObjectOBJManager

Link names are: ocrcfg votedsk1

Executed automatically by the OUI during installation Must be executed manually during manual deinstallation

25


Oracle Object Manager For example

26


Oracle Database Software

Installation

27


Database Installation The script %ORACLE_HOME%\bin\SelectHome.bat must be

executed on all remote nodes to activate the following products: Oracle Data Provider for .NET Oracle Provide for OLE DB Oracle Objects for OLE Oracle Counters for Windows Performance Monitor Oracle Administration Assistant

28


AutomaticStorage

Management

29


Automatic Storage Management ASM disks must be stamped before they can be used by ASM Can be stamped

Using ASMTOOL command line utility Using ASMTOOLG GUI utility During ASM instance creation process in DBCA

30


ASMTOOLG

31


Cluster File System

32


Cluster File System If selected in the Oracle Universal Installer (OUI) session then

the Oracle Cluster File System is installed automatically

It is not necessary to download OCFS separately (as would be the case in Linux for example)

33


Cluster File System To create partitions use OCFSFORMAT Recommended block sizes are:

Software (Shared ORACLE_HOME) 4

Database files 1024

For example:

OCFSFORMAT /l R:/C 1024 /V REDO

34


Patches

35


Patches In addition to normal patch sets additional cumulative patches

are issued for Oracle on Windows platforms

For example 10.1.0.3 Patch 5 10.2.0.1 Patch 3

Install the latest cumulative patch for the platform

For example in Oracle 10.2 available patches include 10.2.0.1.0 Patch 2 (10.2.0.1.2P) Patch 4751342 10.2.0.1.0 Patch 3 (10.2.0.1.3P) Patch 4751539 10.2.0.1.0 Patch 4 (10.2.0.1.4P) Patch 4923768

At the time of writing Patch 4 was recommended

36


Patches Current Windows patch sets are documented on Metalink

For Oracle 10.1 See Metalink Note 276548.1 - 10.1.0.x Oracle Database and

Networking Patches for Microsoft Platforms

For Oracle 10.2 See Metalink Note 342443.1 10.2.0.x Oracle Database and

Networking Patches for Microsoft Platforms

Both reference Note 161549.1 - Oracle Database Server and Networking

Patches for Microsoft Platforms

37


Patch Set Installation Installing Oracle 10.1.0.3 Patch 5 The patch set should be installed in both the Clusterware

home and the Oracle database home Run Setup.exe in the Oracle database home

After installing the CRS Patch on each node stop all CRS services (Service Management)

OracleCRService OracleEVMService OracleCSService OracleClusterVolumeService

Run C:\Oracle\product\10.1\crs\install\patch10103.bat After installing Oracle database software patch on one node,

run $ORACLE_HOME\bin\SelectHome.bat on remaining nodes

38


Patch Set Installation Installing Oracle 10.2.0.1 Patch 3 The patch set should only be installed in the Oracle database

home and does not affect CRS

Patch is installed using OPATCH OPATCH should be upgraded to 10.2.0.1.1 or above

Patch# : 4898608

Stop all services using %ORACLE_HOME% Install 10.2.0.1.3 using

OPATCH APPLY

39


Patch Set Installation Installing Oracle 10.2.0.1 Patch 4 The patch set should be installed in both the Clusterware and

RDBMS home directories Patch 4: 4923768

Patch is installed using OPATCH OPATCH should be upgraded to 10.2.0.1.1 or above in both

Clusterware and RDBMS home directories Patch# : 4898608

Stop all services using %ORACLE_HOME% Install 10.2.0.1 Patch 4 using

OPATCH APPLY

40


Post-InstallationSteps

41


Post Installation Post installation tasks include:

Make files node-specific (Shared Oracle home only) Create ORA_DBA group on remaining nodes Set Log on as a Batch Job privilege for Enterprise Manager

users

42


Make Files Node-Specific If using shared Oracle Home on OCFS file system

After database creation using DBCA the HC files must be made node-specific.

Applies to Oracle 10.1 (at least)

On each host these files have the pathname:

%ORACLE_HOME%\DATABASE\HC_<instance_name>.DAT

For example on instance RAC1

%ORACLE_HOME%\DATABASE\HC_RAC1.DAT

43


Make Files Node-Specific For example on node node1:

srvctl stop instance -d RAC -i RAC1

ocfsutil /c NodeSpecificFile /o create /m h: /p oracle\product\10.1.0\db\database\hc_rac1.dat

srvctl start instance -d RAC -i RAC1

Repeat for remaining instances

44


Log on as a batch job Users must have the Log on as a batch job privilege for

Enterprise Manager to work correctly Start->Administrative Tools->Local Security Policy-> Security Settings->Local Policies-> User Rights Assignment->Log on as a batch job Assign privilege for each administrative users Repeat on each node

There is a description of this problem in Metalink Note 279765.1 "Error Message

'RemoteOperationException: ERROR: Wrong password for users' when trying to startup or shutdown the database from the Enterprise Manager console"

There is a description of the resolution in Metalink Note 109188.1 How to Set "Logon as a Batch Job"

Privileges on Windows 2000 Systems

45


ORA_DBA Group Administrators must be members of this group on each node

in order to use / AS SYSDBA

By default ORA_DBA group is only created on installation node Not created on remaining nodes Must be created manually

See Metalink Notes 177354.1 RAC: Connect AS SYSDBA Fails with ORA-01031 77665.1 - Guide to setup/enable OS authentication to

Connect using OS user account

46


ORA_DBA Group To create new ORA_DBA group and add users

Start->Administrative Tools->Computer Management-> System Tools->Local Users and Groups->Groups Right click and select New Name is ORA_DBA Description is Oracle DBA Group Add user e.g. XYZDOM\abc.admin

47


Administration

48


Environment Variables Environment variables are stored in the registry

Can be set using: Start -> Control Panel -> System -> Advanced-> Environment Variables

49


Registry Keys Oracle registry entries are stored in

HKEY_LOCAL_MACHINE\Software\Oracle

The inventory location is defined in HKEY_LOCAL_MACHINE\Software\Oracle\inst_loc

The default inventory location is: C:\Program Files\Oracle\Inventory

The Oracle Cluster Repository location is stored in HKEY_LOCAL_MACHINE\Software\Oracle\OCR\ocrconfig_loc

50


Registry By default CRS-related keys are stored in

HKEY_LOCAL_MACHINE\Software\Oracle\KEY_OraCr10g_home

For example ORA_CRS_HOME ORACLE_HOME

By default database keys are stored in HKEY_LOCAL_MACHINE\Software\Oracle\KEY_OraDb10g_home

For example ORACLE_BASE ORACLE_HOME ORACLE_SID

51


Scheduler Windows contains a job scheduler

To schedule a job e.g. overnight backup use Start->Control Panel->Scheduled Tasks-> Add Scheduled Task

A wizard will step through the configuration steps

52


Services

53


Services Oracle uses Windows services

Similar to Unix daemons Configured in windows registry

All Oracle services have the "Oracle" prefix

The following services are created during Clusterware installation Oracle Object Service OracleClusterVolumeService OracleCRService OracleCSService OracleEVMService

54


Services The following services are created by DBCA during ASM

instance configuration OracleOraDb10g_home1TNSListenerLISTENER_<node> OracleASMService<ASM_instance_name>

The following services are created by DBCA during database instance configuration OracleJobScheduler<instance_name> OracleService<instance_name> OracleDBConsole<Instance_name>

55


Services Windows services can be controlled in three ways

Using the GUI Using the NET command Using the SC command

56


Services To control services using the GUI

Start->Administrative Tools->Services Click on a service from the list of available services

Alternatively the Service Management tool can be started from the command line using:

C:> SERVICES.MSC

Single click allows you to control the service including: Start / Stop Pause/Continue Restart

57


Services Double click allows you to manage the service including:

Start / Stop / Pause / Continue Specify the startup type

Automatic Manual Disabled

Change the login type Local System Account Named User

Specify recovery parameters Specify dependencies

58


Services Services can be controlled using the NET command To check which services are running use:

Use the MORE command to control the output

NET START | MORE

NET START

Note that this command only prints information about services which are currently running

Use the SC command to list services which are currently stopped

59


Services To start a service use:

NET START <service_name>

For example:

NET START OracleServiceRAC1

NET STOP <service_name>

For example:

NET STOP OracleServiceRAC1

To stop a service use:

60


Services Services can also be controlled using the SC command Available on Windows 2003 and Windows XP To print the help message use:

SC This prints generic help and asks if you wish to see help for the QUERY and QUERYEX options

61


Services To start a service use:

SC START <service_name>

For example:

SC START OracleServiceRAC1

To stop a service use:

SC STOP <service_name>

For example:

SC STOP OracleServiceRAC1

62


Services To query the status of all running services use:

SC QUERY

To query the status of all services use:

SC QUERY STATE= ALL

Note that the syntax for this command is very sensitive You must have

No space between STATE and = Space between = and ALL

Use the MORE command to control the output:

SC QUERY STATE= ALL | MORE

63


Services To query the status of an individual service use:

SC QUERY <service_name>

For example:

C:\> SC QUERY OracleServiceRAC1SERVICE_NAME: OracleServiceRAC1 TYPE : 10 WIN32_OWN_PROCESS STATE : 4 RUNNING (STOPPABLE, PAUSABLE, ACCEPTS_SHUTDOWN) WIN32_EXIT_CODE : 0 (0x0) SERVICE_EXIT_CODE : 0 (0x0) CHECKPOINT : 0x0 WAIT_HINT : 0x0

SC QUERY OracleServiceRAC1

64


Services To delete an individual service use:

SC DELETE <service_name>

For example:

SC DELETE OracleCRService

This may be necessary if you need to: tidy up after a failed installation remove Oracle manually

65


Tasklist To list relationship between processes and services use:

C:\> TASKLIST /SVC

Image Name PID Services--------------------- ---- --------------evmd.exe 2236 OracleEVMServicecrsd.exe 2216 OracleCRService TNSLSNR.EXE 4936 OracleOraDb10g_home1TNSListenerLISTENER_NODE1OracleOBJService.exe 832 Oracle Object Serviceoracle.exe 1964 OracleASMService+ASMOcfsFindVol.exe 1552 OracleClusterVolumeServicenmesrvc.exe 1580 OracleDBConsolePDS1oracle.exe 1628 OracleServicePDS1ocssd.exe 2412 OracleCSService

TASKLIST /SVC

For example:

66


Networking

67


Hosts In Windows the hosts file is typically located in

C:\Windows\System32\Drivers\etc

The format is similar to Unix. For example:

RAC1 10.47.0.101RAC2 10.47.0.102

RAC1-vip 10.47.0.201RAC2-vip 10.47.0.202

RAC1-priv 192.168.0.1RAC2-priv 192.168.0.2

68


HOSTNAME The name of the current node can be obtained at the

command line using the HOSTNAME command For example

C:\> HOSTNAMEnode1

69


PING As in Unix the PING utility can be used to verify network

connections By default only four ICMP probes are performed For example:

C:\> PING NODE2Pinging NODE2 [10.131.60.202] with 32 bytes of data:

Reply from 10.131.60.202: bytes=32 time<1ms TTL=128Reply from 10.131.60.202: bytes=32 time<1ms TTL=128Reply from 10.131.60.202: bytes=32 time<1ms TTL=128Reply from 10.131.60.202: bytes=32 time<1ms TTL=128

Ping statistics for 10.131.60.202: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss).Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms

70


PING By default PING uses a packet size of 32 bytes. The default TCP/IP packet size is around 1400 bytes

Can be increased by configuring jumbo frames Maximum packet size with jumbo frames is about 9000

bytes (network card and switch specific)

To confirm that the network has been correctly configured to use jumbo frames use the ping command:

PING -L <packet_size> -F <hostname>

where <packet_size> is the size in bytes For example

PING -L 1600 -F RAC1

71


PING For example assume the default TCP/IP configuration

C:\> PING -L 1400 -F NODE2Pinging NODE2 [10.131.60.202] with 1400 bytes of data:

Reply from 10.131.60.202: bytes=1400 time<1ms TTL=128Reply from 10.131.60.202: bytes=1400 time<1ms TTL=128Reply from 10.131.60.202: bytes=1400 time<1ms TTL=128Reply from 10.131.60.202: bytes=1400 time<1ms TTL=128

......

C:\> PING -L 1600 -F NODE2Pinging NODE2 [10.131.60.202] with 1600 bytes of data:

Packet needs to be fragmented but DF setPacket needs to be fragmented but DF setPacket needs to be fragmented but DF setPacket needs to be fragmented but DF set

.....

72


IPCONFIG IPCONFIG can be used to verify network configurations Similar to ifconfig on Unix systems For example:

C:\> IPCONFIG

Windows IP Configuration

Ethernet adapter Public Team:

Connection-specific DNS Suffix . : IP Address. . . . . . . . . . . . : 10.131.60.201 Subnet Mask . . . . . . . . . . . : 255.255.0.0 IP Address. . . . . . . . . . . . : 10.131.60.101 Subnet Mask . . . . . . . . . . . : 255.255.0.0 Default Gateway . . . . . . . . . : 10.131.60.254

Ethernet adapter Private Team:

Connection-specific DNS Suffix . : IP Address. . . . . . . . . . . . : 192.168.200.1 Subnet Mask . . . . . . . . . . . : 255.255.255.0 Default Gateway . . . . . . . . . :

Virtual IPAddress

Public IPAddress

Private IPAddress

73


ARP As on Unix systems ARP can be used to view the current

contents of the ARP cache Maps IP addresses to MAC (Ethernet) addresses For example:

C:\> ARP-A

Interface: 10.130.60.230 --- 0x10005 Internet Address Physical Address Type 10.131.60.96 00-0b-cd-3a-19-48 dynamic 10.131.60.202 00-15-60-55-12-81 dynamic 10.131.60.203 00-0b-cd-6d-3c-5b dynamic 10.131.60.207 00-0f-20-f9-c9-ba dynamic 10.131.60.209 00-0f-20-f9-84-65 dynamic 10.131.60.210 00-0b-cd-41-b5-72 dynamic

Interface: 192.168.200.1 --- 0x10006 Internet Address Physical Address Type 192.168.200.2 00-11-0a-5a-42-63 dynamic

74


Acknowledgements Thanks for help in the preparation of this presentation to:

Stephen Bendall Tak Tang John Plowman Lee Cashmore Dirk Schmidt Davy Witvrouwen

Julian Dyke Independent Consultant Web Site : www.juliandyke.com Email: [email protected] Mobile: +44 7917 360777

rac on windows 2003

Documents