11gr2 cluster upgrade

Oracle 11gR2 Clusterware Upgrade on HPUX

A Success Story

Presented by : Syed Jaffer Hussain Slide # 1

Disclaimer

This views/content in this slides are those of the author and do not necessarily reflect that of Oracle Corporation and/or its affiliates/subsidiaries. The material in this document is for informational purposes only and is published with no guarantee or warranty, express or implied..

This material should not be reproduced or used without the authors' written permission.

Foreword

The goal of this presentation is to explain you the best practices that we had adopted/followed/practiced to upgrade successfully our FOUR Oracle 10.2.0.4 clusterware environments to Oracle 11g R2 on HPUX Servers.

What will be covered

o Why upgrade?o Upgrade compatibility matrix.o Upgrade pattern [best practices].o Introduction to our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post upgrade steps.o Upgrade aftermath stories.o References.

Know your presenter

Syed Jaffer HussainDatabase Support Manager20 years IT experience11+ years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i,9i,10g & 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1/R2 Real Application Clusters EssentialsTwitter: @sjaffarhussainhttp://jaffardba.blogspot.com

o WHY upgrade?o Upgrade compatibility matrix.o Upgrade pattern [best practices].o Overview of our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post upgrade steps.o Upgrade aftermath stories.o References.

Why upgrade? – A burning question

Is it mandatory to upgrade an existing environment with every Oracle new release?

Why upgrade?

Situations that perhaps drives you to oppose an upgrade:

Legacy application. Doesn’t want to take the risk. Lack of skills and support.

Why upgrade?

Situations that are likely to drives you to embrace for an upgrade:

Organization standards. Oracle release support policy. Tantalizing new features. Bug fixes.

o Why upgrade?o Upgrade compatibility MATRIX.o Upgrade pattern [best practices].o Overview of our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post upgrade steps.o Upgrade aftermath stories.o References.

Upgrade compatibility matrix

Courtesy from Oracle CorporationThanks to Roy Swonger & Mike Dietrich

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9.2.0.8 or higher 11.2.x10.1.0.5 or higher 11.2.x10.2.0.2 or higher 11.2.x11.1.0.6 or higher 11.2.x

The following database version will require an indirect upgrade path.

Source Database ---> Upgrade Path for Target Database--->Target Database7.3.3 (or lower)-----> 7.3.4 ---> 9.2.0.8 ---->11.2.x8.0.5 (or lower)----> 8.0.6 ---> 9.2.0.8 ---->11.2.x8.1.7 (or lower)----> 8.1.7.4---> 10.2.0.4---->11.2.x9.0.1.3 (or lower)----> 9.0.1.4-- ->10.2.0.4---->11.2.x9.2.0.7(or lower)---->9.2.0.8---->11.2.x

o Why upgrade?o Upgrade compatibility matrix.o Upgrade pattern [BEST PRACTICE].o Overview of our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post upgrade steps.o Upgrade aftermath stories.o References.

Upgrade pattern [BEST PRACTICE]

UPGRADEClusterware

Upgrade pattern [best practice]

UPGRADEClusterware

INSTALLRDBMS

Software

UPGRADEClusterware

INSTALLRDBMS

Software

UPGRADEdatabases

UPGRADEClusterware

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

o Why upgrade?o Upgrade compatibility matrix.o Upgrade pattern [best practices].o Overview of our SETUP.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post upgrade steps.o Upgrade aftermath stories.o References.

Overview of our setup

HP Superdome 1 Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 6.5.6

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

•31 databases•51 instances

•4 logical processors•22GB Memory

V11.23/var/opt

V11.31/var/opt

HP UX upgrade from V11.23 TO v11.31

The framework that we have followed for the upgrade

V11.23/var/opt

V11.31/var/opt

EMC Storage Configurationupdate

V11.23/var/opt

V11.31/var/opt

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

10.2.0.4 11.2.0.2

o Why upgrade?o Upgrade compatibility matrix.o Upgrade pattern [best practices].o Overview of our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post upgrade steps.o Upgrade aftermath stories.o References.

Preparation is the headstone to every successful installation orupgrade projects.

Prepare for upgrade – prerequisites checks

Verify:o OS certification for Oracle 11gR2.o OS kernel, packages.o Additional OS groups.o Oracle base location.o GIH location.

Set GI home different from OB. Configure Single Client Access Name(SCAN)

o Maximum of 3 IP’s (same across nodes).o DNS round robin resolution.o Must be on the subnet as Public and VIP IP’s.o Not recommended to configure in HOSTS file.

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade – prerequisites checks

Space:o Additional space for [/u00] mount point.o Enough /tmp space

Back up:o Back up Oracle homes (CRS, ASM)o Back up Voting disk.o Back up OCR.

CRS version:o Remember, the crs active and software version

must be the same.crsctl query crs activeversioncrsctl query crs softwareversion

Nodes readiness:runcluvfy.sh stage –pre crsinst –n nodes_list –fixup –fixupdir <dirname> -verbose

Clusterware, ASM upgrade – What’s new?

Grid Infrastructure

Out-of-Place andIn-place upgrades

Clusterware

Automatic Storage Management (ASM)

Clusterware, ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together?

Clusterware, ASM upgrade

Clusteware + ASM =

no rolling upgrade

Clusterware and ASM upgrade

ORACLE BASE must be different from the SW

location……

Cause : False warning generated by unpublished CVU bug 10024549Solution: The warning can be ignored

Can I run the rootupgrade.sh script in PARALLEL across nodes?

Always run the script on the LOCAL node first.

Run the script in parallel on all nodes, except the LAST node.

Run the script on the LAST node.

Shuts down the existing clusterware stack.

Brings up the new version clusterware stack.

Up on successfully running the script on the LAST node, the CRS active/release version will set to the new version.

rootupgrade.sh

Average rootupgrade.sh execution time on each node was just about 15 min.

Took more time on couple nodes, mostly while stopping the existing cluster stack.

Look closely for the logs under /u00/app/11.2.0/grid/cfgtoollogs.

Cried for HELP!

Internet

Cried for HELP!

Internet MOS

Cried for HELP!

ASM alert.log, trace files & log files

Internet MOS

Cried for HELP!

Doesn’t help much!

Life Savior

MOS [ID 398374.1]

SQL> ORACLE instance shut down.SQL> Disconnected

[PRSR][1]prsr_get_enabled_key: OCR api procr_open_key failed for key DATABASE.ASM.node01.+asm1.ENABLED with 4[ PRSR][1]OCR error code : 4[ PRSR][1]OCR error msg : PROC-4: The cluster registry key to be operated on does not exist.2009-06-23 16:12:26.332: [ RACG][1] [28830][1][ora.node01.ASM1.asm]: prsr_fetch_asm_config() error, rc = 8

2009-06-23 16:12:27.443: [ RACG][1] [28830][1][ora.node01.ASM1.asm]:SQL*Plus: Release 10.2.0.4.0 - Production on Tue Jun 23 16:12:26 2009

Enter user-name: Connected to an idle instance.

SQL> ORA-01506: missing or illegal database nameSQL> Disconnected

cd $ORA_ASM_HOME/log/hostname/racg/ora.<hostname>.ASM.asm.log

What might have gone wrong???

ASM instance name registered as ASMn instead of +ASMn in the OCR.

Could be due to a manual entry.

GI is very particular about + in contrast to previous cluster version.

Bug 5644904: WRONG ASM KEY PROC-4: THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Remedy

Stop ASM and dependent instances.

Remove dependency on ASM instance.

Remove ASM from OCR.

Add ASM back in with ‘+’ correct name

Add dependency to ASM.

Startup ASM.

Resume the failed step.

In action srvctl stop asm –n <hostname>

srvctl modify instance –d DBNAME –i <instance_name> -f [do it for all active instances on the node]

srvctl remove asm –n <hostname> -i <ASM_INSTANCE1>

srvctl add asm –n <hostname> -i <+ASM_INSTANCE1>-o <ORA_ASM_HOME>

srvctl modify instance –d <DBNAME> -i <instance_name>-s +ASM_INSTANCE1

[do it for all active instances on the node] srvctl start asm –n <hostname>

In action srvctl stop asm –n <hostname>

srvctl modify instance –d DBNAME –i <instance_name> -f [do it for all active instances on the node]

srvctl remove asm –n <hostname> -i <ASM_INSTANCE1>

srvctl add asm –n <hostname> -i <+ASM_INSTANCE1>-o <ORA_ASM_HOME>

srvctl modify instance –d <DBNAME> -i <instance_name>-s +ASM_INSTANCE1

[do it for all active instances on the node] srvctl start asm –n <hostname>

Repeat on all nodes!!!

Post upgrade checks – active/software version

#crsctl query crs activeversionOracle Clusterware active version on the cluster is [11.2.0.2.0] # crsctl query crs releaseversionOracle High Availability Services release version on the local node is [11.2.0.2.0]

Post upgrade checks – cluster stats across nodes

# crsctl check cluster -all**************************************************************node01:CRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online**************************************************************node02:CRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online**************************************************************node03:CRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online**************************************************************

Post upgrade checks – query voting disk

# ocrcheck

Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 1051708 Used space (kbytes) : 27048 Available space (kbytes) : 1024660 ID : 1837247566 Device/File Name : /dev/rdsk/oracle/ocr/ln1/ora_ocr_001 Device/File integrity check succeeded Device/File Name : /dev/rdsk/oracle/ocr/ln1/ora_ocr_002 Device/File integrity check succeeded

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Post upgrade checks – query voting disk

# crsctl query css votedisk

## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 3ee7c55614f24fdcfff130e1abf94384 (/dev/rdsk/oracle/vote/ln1/ora_vote_001) [] 2. ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (/dev/rdsk/oracle/vote/ln1/ora_vote_002) [] 3. ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (/dev/rdsk/oracle/vote/ln1/ora_vote_003) []

Post upgrade checks – b/g and daemon processes#ps -ef |grep d.bin oracle 14458 1 0 15:14:12 ? 0:41 /u00/app/11.2.0/grid/bin/oraagent.bin oracle 14462 1 0 15:14:13 ? 0:00 /u00/app/11.2.0/grid/bin/mdnsd.bin root 14539 1 0 15:14:28 ? 0:24 /u00/app/11.2.0/grid/bin/cssdagent oracle 14549 1 0 15:14:29 ? 0:01 /u00/app/11.2.0/grid/bin/diskmon.bin -d -f root 14541 1 0 15:14:28 ? 4:17 /u00/app/11.2.0/grid/bin/orarootagent.bin root 14732 1 0 15:15:48 ? 7:35 /u00/app/11.2.0/grid/bin/crsd.bin reboot oracle 14469 1 0 15:14:15 ? 0:11 /u00/app/11.2.0/grid/bin/gpnpd.bin oracle 14739 1 0 15:15:49 ? 3:21 /u00/app/11.2.0/grid/bin/evmd.bin oracle 14535 1 0 15:14:25 ? 2:51 /u00/app/11.2.0/grid/bin/gipcd.bin oracle 14545 1 0 15:14:29 ? 4:42 /u00/app/11.2.0/grid/bin/ocssd.bin root 20675 1 0 15:36:03 ? 4:28 /u00/app/11.2.0/grid/bin/orarootagent.bin root 14595 1 0 15:14:50 ? 3:06 /u00/app/11.2.0/grid/bin/octssd.bin root 13999 1 0 15:13:20 ? 2:47 /u00/app/11.2.0/grid/bin/ohasd.bin reboot root 14533 1 0 15:14:24 ? 0:26 /u00/app/11.2.0/grid/bin/cssdmonitor oracle 20671 1 0 15:36:03 ? 0:08 /u00/app/11.2.0/grid/bin/appagent.bin oracle 20685 1 0 15:36:03 ? 0:26 /u00/app/11.2.0/grid/bin/oraagent.bin

#ps -ef |grep init.d

root 66 0 0 Feb 9 ? 0:00 pagetable_init_daemon oracle 8645 15234 0 19:31:02 pts/0 0:00 grep init.d root 13977 1 0 15:13:19 ? 0:00 /bin/sh /sbin/init.d/init.ohasd run

Remember

You can easily resume/downgrade from a failed or successful upgrade.

Can fall back to the pre 11gR2 version.

You hardly need a 3-step procedure to do that.

Remove OLD homes

Removing OLD cluster home

Although it’s a best practice to keep theOLD_HOME for a while after a successfulupgrade, you can use the following to remove the OLD_HOMES.

export ORACLE_HOME=$OLD_HOME

## detach OLD_HOME$OLD_HOME/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME=$OLD_HOME

#confirm $OLD_HOME is removed from central inventory:$OLD_HOME/OPatch/opatch lsinventory -all

## remove files in OLD_HOME manually:/bin/rm -rf $OLD_HOME

unset ORACLE_HOME

11gR2 RDBMS software installation

cluvfy stage -pre dbinst -n node01,node02,node03 -d $ORACLE_HOME –verbose

Verify the READINESS of the nodes for DB software installation

How to manage the pre 11gR2 databases under 11gR2 GIH?

root@node01: /u00/app/11.2.0/grid/bin # ./crsctl pin css -n node01 node02 node03

CRS-4664: Node usiogd01 successfully pinned.CRS-4664: Node usiogd02 successfully pinned.CRS-4664: Node usiogd03 successfully pinned.

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella.

o Why upgrade?o Upgrade compatibility matrix.o Upgrade pattern [best practices].o Overview of our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post DB upgrade steps.o Upgrade aftermath stories.o References.

Database upgrade – Caution

Ensure the database backed up thoroughly BEFORE you kick off the db upgrade procedure.

Database upgrade – upgrade paths

UPGRADE

SQL> @catupgrade.sql

• RAC aware and most recommended for RAC dbs.• Also can be run in –silent mode.

Database upgradeFor a smooth rise, follow the best practice, run @OH_11G/rdbms/admin/utlu112i.sql********************************************************************** Tablespaces: [make adjustments in the current environment] ********************************************************************** --> SYSTEM tablespace is adequate for the upgrade. .... minimum required size: 788 MB --> UNDOTBS1 tablespace is adequate for the upgrade. .... minimum required size: 43 MB --> SYSAUX tablespace is adequate for the upgrade. .... minimum required size: 566 MB --> TEMP tablespace is adequate for the upgrade. .... minimum required size: 61 MB

PURGE DBA_RECYCLEBIN .... must be executed immediately prior to executing your upgrade. . ********************************************************************** Recommendations ********************************************************************** Oracle recommends gathering dictionary statistics prior to upgrading the database. To gather dictionary statistics execute the following command while connected as SYSDBA: EXECUTE dbms_stats.gather_dictionary_stats;

Database upgrade

As a best practice, run @OH_11G/rdbms/admin/utlu112i.sql********************************************************************** Components: [The following database components will be upgraded or installed] ********************************************************************** --> Oracle Catalog Views [upgrade] VALID --> Oracle Packages and Types [upgrade] VALID --> JServer JAVA Virtual Machine [upgrade] VALID --> Oracle XDK for Java [upgrade] VALID --> Real Application Clusters [upgrade] VALID --> Oracle Workspace Manager [upgrade] VALID --> OLAP Analytic Workspace [upgrade] VALID --> OLAP Catalog [upgrade] VALID --> EM Repository [upgrade] VALID --> Oracle Text [upgrade] VALID --> Oracle XML Database [upgrade] VALID --> Oracle Java Packages [upgrade] VALID --> Oracle interMedia [upgrade] VALID --> Spatial [upgrade] VALID --> Data Mining [upgrade] VALID --> Expression Filter [upgrade] VALID --> Rule Manager [upgrade] VALID --> Oracle OLAP API [upgrade] VALID

Database upgrade

Listener will be created

on all nodes

Database upgrade

Note 453903.1 - Enabling and Disabling Oracle Database Vault in UNIX Note 453902.1 - Enabling and Disabling Oracle Database Vault in WINDOWS $ cd $ORACLE_HOME/rdbms/lib$ make -f ins_rdbms.mk dv_off$ cd $ORACLE_HOME/bin$ relink all (For RAC installations, run these commands on all nodes.) #cd /u00/app/oracle/product/11.2.0/db_1/rdbms/lib # make -f ins_rdbms.mk dv_off /usr/ccs/bin/ar d /u00/app/oracle/product/11.2.0/db_1/rdbms/lib/libknlopt.a kzvidv.o /usr/ccs/bin/ar cr /u00/app/oracle/product/11.2.0/db_1/rdbms/lib/libknlopt.a /u00/app/oracle/product/11.2.0/db_1/rdbms/lib/kzvndv.o

Database upgrade

Database post upgrade – list

Run @utlrp.sql to compile ANY invalid objects.

Run @utlu112s.sql to check post upgrade components status.

Adjust time zone data in the database.

Gather statistics for system and fixed objects.

Amend scripts [if any] to affect the new OH settings.

Verify the database upgrade srvctl config database –d <DBNAME>

Back up the database.

Modify the COMPATIBLE init. Parameter to 11gR2.

Database upgrade – measuring upgrade

Database upgrade – trivia

Upgraded over 100 databases successfully using MANUAL and DBUA methods.

Avg. database upgrade time was just about 2 hrs.

Upgrade using DBUA for RAC DB proved a wise decision.

Nearly identical duration between CLI and DBUA upgrade methods.

Duration is typically proportional to the number of installed database components.

Database upgrade – CLI upgrade steps

Upgrade aftermath stories

Snapshot Controlfile must be placed on a shared storage for a RAC database.

ORA-24247: network access denied by access control list (ACL)Solution: UTL_TCP, UTL_HTTP, UTL_SMTP, UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMIN.CREATE_ACL

ADD_PRIVILEGE ASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution: crsctl modify resource “ora.asm” –attr “CHECK_TIMEOUT=132”MOS [ID 1323679.1]

IMPDP fails with ORA-29913, 31011, 19202, LPX-00217[BUG 11877267]Solution: no solution… workaround exists

ORA-07445: exception core dump [evaopn3()]Solution: _simple_view_merge, _complex_view_merging=false

o Why upgrade?o Upgrade compatibility matrix.o Upgrade pattern [best practices].o Overview of our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post DB upgrade steps.o How to proceed from failed upgrade. o Upgrade aftermath stories.o References.

References

Oracle Database Upgrade Path Reference List [ID 730365.1] Complete Checklist for Manual Upgrades to 11gR2 [ID 837570.1] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 870814.1] Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 1096952.1] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on Linux/Unix [ID 969254.1] Oracle Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System? [ID 743649.1]

Oracle 11g R1/R2 Real Application Clusters Essentials

Move to Oracle Database 11g – The whole Story blogs.oracle.com/UPGRADE http://download.oracle.com/docs/cd/E11882_01/server.112/e17222/toc.htm http://www.oracle.com/technetwork/database/upgrade/index.html http://oukc.oracle.com/static09/opn/core09/oracle9i_database/95960/060311_95960/index.htm

A big thank you allfor

listening ...

You can write me at sjaffarhussain@gmail.com

11gr2 cluster upgrade

minimum required

syed jaffer

install rdbms

roy swonger

cluster ready

cluster synchronization

real application

cluster registry

Documents

old home= /d03/oracle/racdb/db/tech st/11.1.0 new · pdf...

upgrade 11gr2 to 12cr1 clusterware

tacc lonestar cluster upgrade to 300 teraflops

using the oracle asm cluster file system (oracle acfs) on...

oracle 10g on solaris to oracle rac 11gr2 on linux...

11gr2 dbua upgrade

11gr2 installation

boise state cluster installer upgrade - computer science

an oracle 11gr2 upgrade war story introduction - jarneil

leveraging oracle asm cluster file system for …...oracle...

upgrade bppm from 11gr2 to 12c with oraclepdb · web...

asm 11gr2 installation & configuration€¦ · asm 11gr2...

mysql cluster - doag.org & nosql combined ... how does mysql...

11gr2 upgrade companion[1]

veritas™ cluster server installation and upgrade guide ·...

rac 11gr1 to 11gr2 upgrade

11i 11gr2 upgrade

veritas cluster server 5.1 sp1 installation and upgrade...

mmsc2.0 cluster to mmsc2.1 cluster with sg upgrade manual

oracle 10g on solaris to oracle rac 11gr2 on linux...