best practices document - ibm developer€¦ · • the oracle database software (version 12c)...

IBM NOI Best Practices Deploying ITNM with Oracle DataGuard

/68- IBM

IBM Network Operations Insights

Best Practices Deploying ITNM DR with Oracle Active Dataguard

Release: Draft 1.2 Updated on: 23/08/2019

Classification: IBM

Author(s): Andrea Smith Anand Badgi Shenisetti Sreekanth

Owner(s): IBM NOI Team

© Copyright 2018 by International Business Machines Corporation All rights reserved


/68- IBM

1. Table of Contents 1. Table of Contents 2

2. Document History 4

2.1. Revision History 4

2.2. Acknowledgements 4

3. Document Distribution Details 5

3.1. Reviewers 5

3.2. Document Copy List 5

4. Introduction 6

4.1. Document use 6

5. Typical Architecture for ITNM Cold standby DR solution with Omnibus and DASH in HA/DR mode 8

6. Assumptions and Scope 9

6.1. Assumptions 9

6.2. In Scope 9

6.3. Out of Scope 10

7. Setup 11

8. Installation of Oracle 12c Enterprise Edition 13

8.1. Preparing the servers ( Primary , Secondary and FarSync ) 13

9. Installing Oracle 12c 15

10. Database Creation 16

11. Preparing Primary, Secondary and FarSync databases for Active DataGuard 17

12. Configuring the Standby Instance 24

13. Configuring the FarSync Instance 31

14. Configuring DataGuard with DGMGRL and Fast-Start Fail over 38

14.1. Configuring BROKER and DATA GUARD on DB server: 38

14.2. Enabling Fast-Start Failover and Starting the Observer 41

15. Installation of ITNM on Primary Instance 45

16. Installing ITNM on physical standby 48

17. Enabling Rysnc between Primary and Secondary ITNM server 49

17.1. Creating rsync files 49 17.1.1. Create rsync files on primno1 as mentioned below 49 17.1.2. Create rsync files on stbyno1 : 52

17.2. Enabling Passwordless Authentication 54


/68- IBM

17.3. Running rsyng scripts to sync Primary ITNM Core with Secondary ITNM core (default set up) 55

17.4. Stopping rsync 55

17.5. 55

18 ITNM Fail over, Fail back and Switch Over Procedure 56

18. 56

18.1. Fail-over Procedure 56

18.2. Fall-back Procedure 57

19. Monitoring Oracle ADG and Rysnc processes 65

19.1. Monitoring ADG FRA space 65

19.2. Monitoring Rysnc processes 67


/68- IBM

2. Document History Ensure that this document is current. Printed and local copies of this document may become obsolete due to changes to the master document.

2.1. Revision History Revision Number Revision Date Summary of Changes Changes Marked Issue 1.0 23/08/2019 Initial issue

2.2. Acknowledgements Thank you to Shenisetti Sreekanth for the initial work on this.


/68- IBM

3. Document Distribution Details

3.1. Reviewers Name Anand Badgi IBM NOI Best Practice Team member Shenisetti. Sreekanth WIPRO ITNM Team member Andrea Smith IBM NOI OA Team member

Satyanarayana Mamidisetti IBM Service Team member

3.2. Document Copy List

Name Title


/68- IBM

4. Introduction

4.1. Document use This Technical Solution Document will describe the steps required to setup IBM Tivoli Network Manager ( ITNM version 4.2 FP 6 ) in Disaster Recovery ( DR ) cold stand by mode using Oracle Active Data Guard ( ADG ) feature of Oracle 12c Enterprise version and Linux based Rysnc tool where users are having 2 ITNM instances in two geo separate Data centres locations ( Primary Site and Secondary/DR Site ) along with respective Netcool DASH ( Dashboard Application Services Hub ) WebGUI and Omnibus components in each site already configured for HA/DR availability. Oracle ADG The Active Data Guard Option is an evolution of Data Guard technology that incorporates significant innovation (multiple patents pending) designed for a specific purpose - to improve production database performance for critical transactions. Active Data Guard enables read-only access to a physical standby database while Redo Apply is active. Queries and reports can be offloaded from the production system to a synchronized physical standby database - all queries at the standby database return up-to-date results. An Active Data Guard Option license must be purchased in addition to Oracle Enterprise Edition in order to utilize these new capabilities. Please refer to following URL to fully understand the licencing requirements of ADG. https://www.oracle.com/database/technologies/faq-active-data-guard.html Linux based Rsync Tool rsync is an open source utility that provides fast incremental file transfer. rsync is freely available under the GNU General Public Licence and is currently being maintained by Wayne Davision. rsync is a utility for efficiently transferring and synchronizing files between a computer and an external hard drive and across networked computers by comparing the modification times and sizes of files. It is commonly found on Unix-like operating systems. The rsync algorithm is a type of delta encoding. This is used for minimizing network usage. Zlib may be used for additional data compression and SSH or stunnel can be used for security. Rsync is typically used for synchronizing files and directories between two different systems. Please refer to following url for more details. https://en.wikipedia.org/wiki/Rsync After successful installation of ITNM DR solution, users should able to • Fail over from Primary to Secondary ITNM , In case of complete failure of Active ITNM

instance from Primary Site or complete failure of Primary Site , User should able to start Standby ITNM instance from Secondary/DR site which shall assume Primary role and act as ‘Network Discovery and Availability Monitoring Tool’ and provide full functionality of ITNM like Network Discovery , Polling , RCA and Topology visualization to end user during fail over period.

• Switch over from Primary to Secondary ITNM, User can optionally turn down Active ITNM instance from Primary Site and force Standby ITNM instance from Secondary/DR site to become active to act as ‘Network Discovery and Availability Monitoring Tool’ and provide full functionality of ITNM like Network Discovery, Polling, RCA and Topology visualization to end user during fail over period.

https://www.oracle.com/database/technologies/faq-active-data-guard.html

https://en.wikipedia.org/wiki/Rsync


/68- IBM

• Fall back from Secondary to Primary ITNM, Once failed Primary site is available, Users can fall back to default original configuration where ITNM in Primary Sie is in active mode and ITNM in Secondary/DR site is in Standby Mode.

This document does not cover installing ITNM or any other products available as part of the Netcool Operations Insights (NOI) solution.


/68- IBM

5. Architecture for ITNM Cold standby DR solution with Omnibus and DASH in HA/DR mode

In order to achive a typical Geo redudenat Distarer Recover ( DR ) capability for ITNM , primary ITNM server instance, ITNM Database along with Omnibus and DASH system should be co located but installed on separate servers in same Site ( Primary Site ) compared to secondary/backup ITNM server instance, ITNM Database, Omnibus and DASH instance that to be co located in separate Site ( DR Site ) , so that if any of the primary active hosts fails, the backup standby hosts are unaffected and ready to take the role primary host/s.

As shown in above diagram Primary Object server in Primary Site and Secondary Object server running on DR site are configured for HA/DR mode using Omnibus bi directional gateway running on DR Site. Primary WebGUI DASH instance and Secondary WebGUI DASH is configured in HA mode. Primary Oracle DB and ITNM server in Primary Site and Secondary Oracle DB and ITNM server in DR site are configured for DR mode as explained in this document.


/68- IBM

6. Assumptions and Scope

6.1. Assumptions The following assumptions govern this document:

• All relevant licenses and permissions have been made available to the installation and setup team completing this work.

• The Oracle database software (version 12c) along with ADG ( Active Data Guard ) component has been obtained and downloaded to the required DB2 servers in Primary and DR site.

• In order to use Oracle ADG feature between Primary and Secondary Database, we need to install a third light weight Oracle instance ( call Farsync Node ) on a separate physical server but it should be physically near to Primary Database ( LAN Proximity ) but ideally in a separate site. In our case we have located Primary Database server and Farsync server in same Primary Site.

• Rysnc tool has been obtained and installed on ITNM servers in Primary and DR site • The ITNM core and GUI component (version 4.2 FP5 or higher ) has been obtained and

downloaded to the required servers in Primary and DR site. • Other Netcool components like IBM Omnibus ( version 8.1.0.17 or higher ) , Omnibus

WebGUI/DASH( version 8.1.0.14 or higher ) have been obtained , installed and configured in Primary and DR site for HA/DR mode.

• Tivoli Netcool/OMNIbus ObjectServer is opted as user authentication method to configure and store ITNM users and roles.

• ITNM core component running in Primary site and DR site is NOT configured for ITNM HA solution as described in following URL https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/concept/ins_failoverarchitecture.html

• The term ‘server’ is used throughout this document and may refer to a physical machine or a VM depending upon your setup.

6.2. In Scope • Installing and configuring Oracle Active Data Guard ( ADG ) with Fast-Start Fail over to

enable synchronization of ITNM Topology and Poller data between Primary and Secondary ITNM.

https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/concept/ins_failoverarchitecture.html



/68- IBM

• Configuring ITNM in Primary and DR Site to use Oracle ADG and Linux based Rysnc feature to synchronize ITNM core components data ( config , model cache , stitchers etc ) to achieve Cold Stand by DR capabilities for ITNM functionality.

• Example verification scenario ( ITNM Fail-Over and Fall-Back in DR situation )

6.3. Out of Scope The following items are out of scope of this document:

• Installation of ITNM in Primary and DR site. Please refer to following URL for ITNM installation guide https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/common/reference/ref_installation_guide.html

• Installation of any other NOI components like Omnbus and WebGUI DASH components in Primary and DR site and configuration of them to operate in failover Mode and integration with them and ITNM . Please refer to following URLs for Omnibus and DASH installation and HA/DR configuration guide. https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/task/ins_confobjectserverfailover.html https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/task/ins_connobjserverfailoverpair.html https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/task/ins_confwebguidatasourcefailover.html https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/task/ins_confobjservauthentication.html https://www.ibm.com/support/knowledgecenter/SSEKCU_1.1.0/com.ibm.psc.doc_1.1.0/psc_ic-homepage.html

• ITNM Primary or/and Secondary DB Configuration with Oracle RAC feature • Configuration and set up of ITNM HA ( High Availability ) feature with two Data

centre/site ( Primary and Secondary ) configuration using ITNM OOTB Fail over configuration ( ncp_virtualdomain ) as described in following url https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/concept/ins_failoverarchitecture.html

• ITNM Switch-Over and Switch-Back during Maintenance period • In depth Installation and Configuration of Oracle product even though we have provided few

steps but then are mainly to assist with ADG configuration procedure.

These may be required as part of the implementation project but are not described within this document.

https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/common/reference/ref_installation_guide.html

https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/common/reference/ref_installation_guide.html

https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/task/ins_confobjectserverfailover.html

https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/task/ins_confobjectserverfailover.html

https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/task/ins_connobjserverfailoverpair.html

https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/task/ins_connobjserverfailoverpair.html

https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/task/ins_confwebguidatasourcefailover.html

https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/task/ins_confwebguidatasourcefailover.html

https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/task/ins_confobjservauthentication.html

https://www.ibm.com/support/knowledgecenter/en/SSSHRK_4.2.0/install/task/ins_confobjservauthentication.html

https://www.ibm.com/support/knowledgecenter/SSEKCU_1.1.0/com.ibm.psc.doc_1.1.0/psc_ic-homepage.html

https://www.ibm.com/support/knowledgecenter/SSEKCU_1.1.0/com.ibm.psc.doc_1.1.0/psc_ic-homepage.html




/68- IBM

7. Setup The following sample Netcool lab environment ( Two site up with Omnibus/DASH in HA mode and ITNM in DR mode ) and corresponding Netcool component server/node hostname, site name, oracle data base server and sid names etc will be referenced throughout this document to describe the steps and procedure to install and configure DR solution. Please substitute for your actual names. Please note that all Netcool components ( ITNM/Omnbus/DASH ) including Oracle server are installed on a single server in each site ( Primary and DR ) but in in case of real life production environment setup , it is recommend to run each components on separate servers/vms as per the sizing requirements.

Netcool Primary Site (Active) Netcool Secondary/DR Site (Standby)

Oracle FarSync Server Site

Hostname primno1.fragglerock.mycom.com stbyno1.fragglerock.mycom.com farsync.fragglerock.mycom.com IP 10.162.64.172 10.45.41.202 10.162.64.137 Geo Location

Chennai DC London DC Chennai DC

OS Linux CentOS Linux CentOS Linux CentOS OS Version

CentOS Linux release 7.6.1810 (Core)



Products installed

Oracle server holding ITNM NCIM DB , ITNM , Omnibus and DASH

Oracle server holding ITNM NCIM DB , ITNM , Omnibus and DASH

Oracle server operating as FarSync server.

Oracle version

Oracle 12.1 R2 (12.1.0.2.0) Oracle 12.1 R2 (12.1.0.2.0) Oracle 12.1 R2 (12.1.0.2.0)

Firewall considerations The following range of ports will be used during the course of ITNM and there after. It is mandatory to allow these ports through firewall to avoid any inconvenience in accessing ITNM services from outside the ITNM server. The ports 16311-16320 are the default ports used by ITNM and the port 1521 will be used by Oracle listener. ( If the ports are not available, or ITNM/Oracle choose another port, those ports should be allowed through firewall)

Example : $ firewall-cmd --zone=public --add-port=16311/tcp --permanent $ firewall-cmd --zone=public --add-port=16312/tcp --permanent $ firewall-cmd --zone=public --add-port=16313/tcp --permanent $ firewall-cmd --zone=public --add-port=16314/tcp --permanent $ firewall-cmd --zone=public --add-port=16315/tcp --permanent


/68- IBM

$ firewall-cmd --zone=public --add-port=16316/tcp --permanent $ firewall-cmd --zone=public --add-port=16317/tcp --permanent $ firewall-cmd --zone=public --add-port=16318/tcp --permanent $ firewall-cmd --zone=public --add-port=16319/tcp --permanent $ firewall-cmd --zone=public --add-port=16320/tcp --permanent $ firewall-cmd --zone=public --add-port=1521/tcp --permanent $ firewall-cmd --reload $ sudo iptables-save

The above ports should be added as exception on all the three instances. Refer to your company’s firewall team for advice on modifying your firewall.

Hostname aliasing

Add the IP addresses and fully qualified domain names of all your servers into the /etc/hosts file on each server. Add the following hostname aliasing in “/etc/hosts” file on all the three instances.

#--------------------------------------------------------------------------------------------------- 10.162.64.172 primno1 primno1.fragglerock.mycom.com 10.162.64.137 farsync farsync.fragglerock.mycom.com 10.45.41.202 stbyno1 stbyno1.fragglerock.mycom.com #---------------------------------------------------------------------------------------------------


/68- IBM

8. Installation of Oracle 12c Enterprise Edition

8.1. Preparing the servers ( Primary , Secondary and FarSync ) It not our intention to provide detailed steps and instructions about how two install Oracle 12c on Primary/Secondary ITNM DB server and FarSync server. Following steps are only for guidance and user are requested to refer to Oracle 12c documentation based on actual sizing requirements.

• Log into Oracle Primary sever ( primno1.fragglerock.mycom.com )as root user

• Create users and groups # groupadd oinstall # groupadd dba # groupadd oper # useradd -G dba,oper -g oinstall -d /home/orcl_user oracle

• Create password for user oracle # passwd oracle

- Enter a password - Verify password

• Assign permissions for the oracle user # chown -R oracle:oinstall /home/orcl_user/ # chmod 775 -R /home/orcl_user/

• Edit file /etc/sysctl.conf # vi /etc/sysctl.conf

-------------------------------------------------------------------------------- fs.file-max = 6815744 kernel.sem = 250 32000 100 128 kernel.shmmni = 4096 kernel.shmall = 1073741824 kernel.shmmax = 4398046511104 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576 fs.aio-max-nr = 1048576 net.ipv4.ip_local_port_range = 9000 65500 --------------------------------------------------------------------------------


/68- IBM

• Execute /sbin/sysctl -p to save changes # /sbin/sysctl -p

• Set the file limits # vi /etc/security/limits.conf

-------------------------------------------------------------------------------------------------- oracle soft nofile 1024 oracle hard nofile 65536 oracle soft nproc 16384 oracle hard nproc 16384 oracle soft stack 10240 oracle hard stack 32768 ---------------------------------------------------------------------------------------------------

• Set Environment Variables for oracle user Switch user to oracle

# su - oracle

Edit bash profile # vi .bash_profile

Add following lines (Give the relevant hostname and unique_name) --------------------------------------------------------------------------------------------------- export TMP=/tmp export TMPDIR=$TMP export ORACLE_HOSTNAME= primno1.fragglerock.mycom.com export ORACLE_UNQNAME=ODB export ORACLE_BASE=/home/orcl_user/app/oracle export ORACLE_HOME=$ORACLE_BASE/product/12.1.0/dbhome_1 export ORACLE_SID=ODB export PATH=/usr/sbin:$PATH export PATH=$ORACLE_HOME/bin:$PATH export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib ---------------------------------------------------------------------------------------------------

• Log in on Oracle Secondary ( stbyno1.fragglerock.mycom.com ) and FarSync server ( farsync.fragglerock.mycom.com ) as root and carry out same steps mentioned above but make sure to use appropriate values to set respective ORACLE_HOSTNAME , ORACLE_UNQNAME and ORACLE_SID environment variables.


/68- IBM

9. Installing Oracle 12c It not our intention to provide detailed steps and instructions about how two install Oracle 12c on Primary/Secondary ITNM DB server and FarSync server. Following steps are only for guidance and user are requested to refer to Oracle 12c documentation based on actual sizing requirements.

On Oracle Primary DB server , Navigate to the directory containing Installation files. Run the installer using install option. # ./runInstaller

Once the installation is done, you’ll be asked to run two scripts as below /home/orcl_user/oraInventory/orainstRoot.sh /home/orcl_user/app/oracle/product/12.2.0/dbhome_1/root.sh

The installation of Oracle 12C Enterprise Edition is completed on Primary instance. Same procedure has to be followed to install Oracle 12c on both Standby and FarSync instances. While configuring .bash_profile, following care should be taken. On Physical Standby Server :

export ORACLE_HOSTNAME= stbyno1.fragglerock.mycom.com export ORACLE_UNQNAME=ODB_STBY

On FarSync Server : export ORACLE_HOSTNAME= farsync.fragglerock.mycom.com export ORACLE_UNQNAME=ODB_FS


/68- IBM

10. Database Creation The database should only be created on the Primary Oracle server instance and NOT on Secondary or FarSync Oracle server instance.

Login on to Primary Oracle Server as root ( primno1.fragglerock.mycom.com ) # su - oracle # $ORACLE_HOME/bin/dbca

You’ll see a pop-up window of Oracle DBCA(Database Configuration Assistance)

• Create a database

Next ->

• Typical configuration

Give relevant details

Next ->

• Finish Note : Make sure that listener is running on port 1521. If listener port is other than 1521, it should be allowed through firewall.


/68- IBM

11. Preparing Primary, Secondary and FarSync databases for Active DataGuard

Configuring the Primary Instance ( primno1.fragglerock.mycom.com )

Logon to primary instance as oracle. # su - oracle

Create the following folders # mkdir - p /home/oracle/app/oracle/oradata/ODB/onlinelog # mkdir - p /home/oracle/app/oracle/fast_recovery_area/ODB/onlinelog

Run the following commands in SQL to make appropriate changes.

# sqlplus sys/<password>@ODB as sysdba.

<password> is the password set at the creation of database

SQL> SELECT log_mode FROM v$database;

LOG_MODE

------------

NOARCHIVELOG

SQL> SHUTDOWN IMMEDIATE;

SQL> STARTUP MOUNT;

SQL> ALTER DATABASE ARCHIVELOG;

SQL> ALTER DATABASE OPEN;

SQL> ALTER DATABASE FORCE LOGGING;

SQL> ALTER SYSTEM SWITCH LOGFILE;

SQL> ALTER DATABASE FLASHBACK ON;

SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;

SQL> ALTER DATABASE ADD STANDBY LOGFILE thread 1 GROUP 4 (

'/home/oracle/app/oracle/oradata/ODB/onlinelog/stdbyrdo41.log',

'/home/oracle/app/oracle/fast_recovery_area/ODB/onlinelog/stdbyrdo42.log'

) SIZE 50M BLOCKSIZE 512;






/68- IBM









Check the groups created :

SQL> select group#,bytes/1024/1024,members from v$log;

SQL> select member,group# from v$logfile;

Create tnsnames.ora file in $ORACLE_HOME/network/admin folder as shown below

# vi $ORACLE_HOME/network/admin/tnsnames.ora

#----------------------------------------------------------------------------------

ODB_FS =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = farsync.fragglerock.mycom.com)(PORT = 1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB_STBY =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = stbyno1.fragglerock.mycom.com)(PORT = 1521))

)


/68- IBM

(CONNECT_DATA =

(SID = ODB)

)

)

ODB =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = primno1.fragglerock.mycom.com)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = ODB)

)

)

LISTENER_ODB =

(DESCRIPTION =


(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = ODB)

)

)

LISTENER_ODB_STBY =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = stbyno1.fragglerock.mycom.com)(PORT = 1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

LISTENER_ODB_FS =

(DESCRIPTION =


/68- IBM

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = farsync.fragglerock.mycom.com)(PORT = 1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

#--------------------------------------------------------------------------------------------------------

Create a file for listener as shown below. # vi $ORACLE_HOME/network/admin/listener.ora

#--------------------------------------------------------------------------------------------------------

LISTENER =

(DESCRIPTION_LIST =

(DESCRIPTION =


(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))

)

)

SID_LIST_LISTENER =

(SID_LIST =

(SID_DESC =

(GLOBAL_DBNAME = ODB)

(ORACLE_HOME = /home/oracle/app/oracle/product/12.1.0/dbhome_1)

(SID_NAME = ODB)

)

)

ADR_BASE_LISTENER = /home/oracle/app/oracle

#-----------------------------------------------------------------------------------------------


/68- IBM

Restart the listener as shown below. # lsnrctl stop #lsnrctl start

Edit $ORACLE_HOME/dbs/initODB.ora file to accommodate all configuration parameters as shown below.

#-----------------------------------------------------------------------------------------------

ODB.__data_transfer_cache_size=0

ODB.__db_cache_size=7918845952

#ODB.__inmemory_ext_roarea=0

#ODB.__inmemory_ext_rwarea=0

ODB.__java_pool_size=67108864

ODB.__large_pool_size=100663296

ODB.__oracle_base='/home/oracle/app/oracle'

#ORACLE_BASE set from environment

ODB.__pga_aggregate_target=3388997632

ODB.__sga_target=10099884032

ODB.__shared_io_pool_size=536870912

ODB.__shared_pool_size=1442840576

ODB.__streams_pool_size=0

*.audit_file_dest='/home/oracle/app/oracle/admin/ODB/adump'

*.audit_trail='db'

*.compatible='12.1.0.2'

*.control_files='/home/oracle/app/oracle/oradata/ODB/control01.ctl','/home/oracle/app/oracle/fast_recovery_area/ODB/control02.ctl'

*.db_block_size=8192

*.db_create_file_dest='/home/oracle/app/oracle/oradata'

*.db_name='ODB'

*.db_unique_name='ODB'

*.db_recovery_file_dest='/home/oracle/app/oracle/fast_recovery_area/ODB'

*.db_recovery_file_dest_size=8016m


/68- IBM

*.diagnostic_dest='/home/oracle/app/oracle'

*.dispatchers='(PROTOCOL=TCP) (SERVICE=ODBXDB)'

#*.enable_pluggable_database=true

*.log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=ODB'

*.log_archive_dest_2='service=ODB_FS SYNC AFFIRM alternate=log_archive_dest_3 valid_for=(online_logfiles,primary_role) db_unique_name=ODB_FS'

*.log_archive_dest_3='service=ODB_STBY ASYNC max_failure=1 alternate=log_archive_dest_2 valid_for=(online_logfiles,primary_role) db_unique_name=ODB_STBY'

*.log_archive_dest_state_3=alternate

*.log_archive_config='DG_CONFIG=(ODB,ODB_FS,ODB_STBY)'

*.local_listener='LISTENER_ODB'

*.open_cursors=300

*.pga_aggregate_target=3201m

*.processes=640

*.remote_login_passwordfile='EXCLUSIVE'

*.sga_target=9603m

*.standby_file_management='AUTO'

*.undo_tablespace='UNDOTBS1'

*.fal_server='ODB_STBY'

*.dg_broker_start=false;

*.db_file_name_convert='/home/oracle/app/oracle/oradata/ODB_STBY/','/home/oracle/app/oracle/oradata/ODB/'

*.log_file_name_convert='/home/oracle/app/oracle/oradata/ODB_STBY/','/home/oracle/app/oracle/oradata/ODB/'

#-----------------------------------------------------------------------------------------------

Restart the database:

SQL> shutdown immediate

SQL> create spfile from pfile;

SQL> startup mount


/68- IBM

SQL> select status,instance_name,database_role,open_mode from v$database,v$Instance

STATUS INSTANCE_NAME DATABASE_ROLE OPEN_MODE

------------ ---------------- ---------------- --------------------

OPEN ODB PRIMARY READ WRITE

Create a password file for Standby and FarSync instances. Use oracle cli command ‘orapwd’ as shown below to create a password file orapwODB with 10 entries.

# orapwd file=$ORACLE_HOME/dbs/orapwODB password=<password> entries=10

Create the Control File for Far SYNC:

SQL> alter database create far sync instance controlfile as '$ORACLE_HOME/control01.ctl';

SQL> create pfile='$ORACLE_HOME/initODB_FS.ora' from spfile;

Create Control File for Physical Standby:

SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS '$ORACLE_HOME/control02.ctl';

Create the folders in far sync. The destination folders in farsync instance should be in place prior to transferring files to it.

Copy control and password files to FarSync instance as shown below # scp $ORACLE_HOME/dbs/orapwODB oracle@farsync:/home/oracle/app/oracle/product/12.1.0/dbhome_1/dbs # scp $ORACLE_HOME/initODB_FS.ora oracle@farsync:/home/oracle/app/oracle/product/12.1.0/dbhome_1/dbs/initODB_FS.ora # scp $ORACLE_HOME/control01.ctl oracle@farsync:/home/oracle/app/oracle/oradata/ODB_FS/controlfile/control01_.ctl # scp $ORACLE_HOME/control01.ctl oracle@farsync:/home/oracle/app/oracle/fast_recovery_area/ODB_FS/controlfile/control02_.ctl

Copy Password and Control Files to Standby server as shown below

Check whether the destination folders are present on Standby instance or not. If not, create them before transferring the files.

# scp $ORACLE_HOME/dbs/orapwODB oracle@stbyno1:/home/oracle/app/oracle/product/12.1.0/dbhome_1/dbs/orapwODB # scp $ORACLE_HOME/control02.ctl oracle@stbyno1:/home/oracle/app/oracle/oradata/ODB_STBY/controlfile/control01_.ctl # scp $ORACLE_HOME/control02.ctl oracle@stbyno1:/home/oracle/app/oracle/fast_recovery_area/ODB_STBY/controlfile/control02_.ctl

mailto:oracle@farsync:/home/oracle/app/oracle/fast_recovery_area/ODB_FS/controlfile/control02_.ctl

mailto:oracle@farsync:/home/oracle/app/oracle/fast_recovery_area/ODB_FS/controlfile/control02_.ctl


/68- IBM

12. Configuring the Standby Instance

Logon to standby server stbyno1 stbyno1.fragglerock.mycom.com as oracle user # su - oracle

Create the following folders # mkdir -p /home/orcl_user/app/oracle/admin/ODB_stby/adump

# mkdir -p /home/orcl_user/app/oracle/oradata/ODB_stby/controlfile

# mkdir -p /home/orcl_user/app/oracle/fast_recovery_area/ODB_stby/controlfile

# mkdir -p /home/orcl_user/app/oracle/admin/ODB/adump

# mkdir -p /home/orcl_user/app/oracle/oradata/ODB/controlfile

# mkdir -p /home/orcl_user/app/oracle/fast_recovery_area/ODB/controlfile

# mkdir -p /home/orcl_user/app/oracle/oradata/ODB_STBY/onlinelog

# mkdir -p /home/orcl_user/app/oracle/oradata/ODB/onlinelog/

# mkdir -p /home/orcl_user/app/oracle/fast_recovery_area/ODB/onlinelog/

Create the $ORACLE_HOME/dbs/initODB_stby.ora file # vi $ORACLE_HOME/dbs/initODB_stby.ora

ODB.__data_transfer_cache_size=0


#ODB.__inmemory_ext_roarea=0

#ODB.__inmemory_ext_rwarea=0



ODB.__oracle_base='/home/orcl_user/app/oracle'#ORACLE_BASE set from environment






*.audit_file_dest='/home/orcl_user/app/oracle/admin/ODB_stby/adump'

*.audit_trail='db'

*.compatible='12.1.0.2'


/68- IBM

*.control_files='/home/orcl_user/app/oracle/oradata/ODB/control01.ctl','/home/orcl_user/app/oracle/fast_recovery_area/ODB/control02.ctl'


*.db_create_file_dest='/home/orcl_user/app/oracle/oradata'

*.db_name='ODB'

*.db_unique_name='ODB_STBY'

*.db_recovery_file_dest='/home/orcl_user/app/oracle/fast_recovery_area/ODB_stby'


*.diagnostic_dest='/home/orcl_user/app/oracle'

*.dispatchers='(PROTOCOL=TCP) (SERVICE=ODB_STBYXDB)'


*.log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=ODB_STBY'

*.log_archive_dest_2='service=ODB ASYNC valid_for=(online_logfiles,primary_role) db_unique_name=ODB'


*.local_listener='LISTENER_ODB_STBY'

*.open_cursors=300


*.processes=640


*.sga_target=9603m



*.dg_broker_start=true;

*.fal_server='ODB_FS','ODB'

*.db_file_name_convert='/home/orcl_user/app/oracle/oradata/ODB/','/home/orcl_user/app/oracle/oradata/ODB_stby/'

*.log_file_name_convert='/home/orcl_user/app/oracle/oradata/ODB/','/home/orcl_user/app/oracle/oradata/ODB_stby/'

Create tnsnames.ora file as shown below


/68- IBM

# vi $ORACLE_HOME/network/admin/tnsnames.ora

LISTENER_ODB =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = ORACLE_PRIMARY)(PORT = 1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB =

(DESCRIPTION =

(ADDRESS_LIST =


)

(CONNECT_DATA =

(SID = ODB)

)

)

LISTENER_ODB_FS =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = ORACLE_FARSYNC)(PORT = 1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

LISTENER_ODB_STBY =

(DESCRIPTION =


/68- IBM

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = ORACLE_STANDBY)(PORT = 1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB_STBY =

(DESCRIPTION =

(ADDRESS_LIST =


)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB_FS =

(DESCRIPTION =

(ADDRESS_LIST =


)

(CONNECT_DATA =

(SID = ODB)

)

)

Create listener file :


/68- IBM

# vi $ORACLE_HOME/network/admin/listener.ora

LISTENER =

(DESCRIPTION_LIST =

(DESCRIPTION =



)

)

SID_LIST_LISTENER =

(SID_LIST =

(SID_DESC =

(GLOBAL_DBNAME = ODB_STBY)

(ORACLE_HOME = /home/orcl_user/app/oracle/product/12.1.0/dbhome_1)

(SID_NAME = ODB)

)

)

ADR_BASE_LISTENER = /home/orcl_user/app/oracle

Restart the listener : # lsnrctl stop

# lsnrctl start

Create the Standby redo log files on Standby database : SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 4 (

'/home/orcl_user/app/oracle/oradata/ODB/onlinelog/stdbyrdo41.log',

'/home/orcl_user/app/oracle/fast_recovery_area/ODB/onlinelog/stdbyrdo42.log'



/68- IBM

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 5 (












# export ORACLE_SID=ODB

Start the database session with created parameter file

# sqlplus / as sysdba

SQL> STARTUP NOMOUNT PFILE;

Check the groups created : SQL> select group#,bytes/1024/1024,members from v$log;

SQL> select member,group# from v$logfile;

Duplicate the Primary database to Standby Database with RMAN using Duplicate command

# rman

rman> connect TARGET sys/<password>@ODB

rman> connect AUXILIARY sys/<password>@ODB_STBY

rman> DUPLICATE TARGET DATABASE

FOR STANDBY

FROM ACTIVE DATABASE

DORECOVER

SPFILE

SET db_unique_name='ODB_STBY' COMMENT 'Is standby'

NOFILENAMECHECK;


/68- IBM

Restart the database : # sqlplus sys/PASSWD as sysdba


SQL> startup mount

SQL> select status,instance_name,database_role from v$database,v$instance;

STATUS INSTANCE_NAME DATABASE_ROLE

--------- ---------------- ---------------- MOUNTED ODB PHYSYCAL STANDBY


/68- IBM

13. Configuring the FarSync Instance Logon to farsync farsync.fragglerock.mycom.com instance as oracle

# su - oracle

Create the following folders : # mkdir -p /home/orcl_user/app/oracle/oradata/ODB_FS/controlfile/ # mkdir -p /home/orcl_user/app/oracle/oradata/ODB_FS/controlfile/ # mkdir -p /home/orcl_user/app/oracle/oradata/ODB_FS/onlinelog/ # mkdir -p /home/orcl_user/app/oracle/oradata/ODB_FS/pdbseed # mkdir -p /home/orcl_user/app/oracle/oradata/ODB_FS/pdb1 # mkdir -p /home/orcl_user/app/oracle/oradata/fastpwd_recovery_area/ODB_FS # mkdir -p /home/orcl_user/app/oracle/admin/ODB_FS/adump # mkdir -p /home/orcl_user/app/oracle/fast_recovery_area/ODB_FS/controlfile/ # mkdir -p /home/orcl_user/app/oracle/admin/ODB/adump # mkdir -p /home/orcl_user/app/oracle/oradata/ODB_FS/onlinelog # mkdir -p /home/orcl_user/app/oracle/fast_recovery_area/ODB_FS/onlinelog

Rename initODB_FS.ora file as initODB.ora in $ORACLE_HOME/dbs # mv initODB_FS.ora initODB.ora

Edit the initODB.ora file on the Far Sync server to change the location of control_files *.control_files='/home/orcl_user/app/oracle/oradata/ODB_FS/controlfile/control01_.ctl','/home/orcl_user/app/oracle/fast_recovery_area/ODB_FS/controlfile/control02_.ctl'

Check the parameters in initODB.ora file and add the following if not present # vi $ORACLE_HOME/dbs/initODB.ora

DB.__data_transfer_cache_size=0




ODB.__oracle_base='/home/orcl_user/app/oracle'#ORACLE_BASE set from environment






*.audit_file_dest='/home/orcl_user/app/oracle/admin/ODB_FS/adump'

*.audit_trail='db'


/68- IBM

*.compatible='12.1.0.2.0'

*.control_files='/home/orcl_user/app/oracle/oradata/ODB_FS/controlfile/control01_.ctl','/home/orcl_user/app/oracle/fast_recovery_area/ODB_FS/controlfile/control02_.ctl'


*.db_domain=''

*.db_create_file_dest='/home/orcl_user/app/oracle/oradata'

*.db_name='ODB'

*.db_unique_name='ODB_FS'

*.db_recovery_file_dest='/home/orcl_user/app/oracle/fast_recovery_area/ODB_FS'


*.diagnostic_dest='/home/orcl_user/app/oracle'

*.dispatchers='(PROTOCOL=TCP) (SERVICE=ODB_FSXDB)'


*.log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=ODB_FS'

*.log_archive_dest_2='service=ODB_STBY ASYNC valid_for=(standby_logfiles,standby_role) db_unique_name=ODB_STBY'


*.local_listener='LISTENER_ODB_FS'

*.open_cursors=300


*.processes=300


*.sga_target=4740m



*.fal_server='ODB'

*.dg_broker_start=true;

*.db_file_name_convert='/home/orcl_user/app/oracle/oradata/ODB/','/home/orcl_user/app/oracle/oradata/ODB_FS/'

*.log_file_name_convert='/home/orcl_user/app/oracle/oradata/ODB/','/home/orcl_user/app/oracle/oradata/ODB_FS/'


/68- IBM

Create tnsnames.ora file : # vi $ORACLE_HOME/network/admin/tnsnames.ora

LISTENER_ODB =

(DESCRIPTION =

(ADDRESS_LIST =


)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB =

(DESCRIPTION =

(ADDRESS_LIST =


)

(CONNECT_DATA =

(SID = ODB)

)

)

LISTENER_ODB_FS =

(DESCRIPTION =

(ADDRESS_LIST =


)

(CONNECT_DATA =

(SID = ODB)

)


/68- IBM

)

LISTENER_ODB_STBY =

(DESCRIPTION =

(ADDRESS_LIST =


)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB_STBY =

(DESCRIPTION =

(ADDRESS_LIST =


)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB_FS =

(DESCRIPTION =

(ADDRESS_LIST =


)

(CONNECT_DATA =

(SID = ODB)

)

)

Create listener file :


/68- IBM

# vi $ORACLE_HOME/network/admin/listener.ora

LISTENER =

(DESCRIPTION_LIST =

(DESCRIPTION =



)

)

SID_LIST_LISTENER =

(SID_LIST =

(SID_DESC =

(GLOBAL_DBNAME = ODB_FS)

(ORACLE_HOME = /home/orcl_user/app/oracle/product/12.1.0/dbhome_1)

(SID_NAME = ODB)

)

)

ADR_BASE_LISTENER = /home/orcl_user/app/oracle

Restart the listener : # lsnrctl stop

# lsnrctl start

Start the database in mount mode

# sqlplus sys/<password>@ODB_FS as sysdba

SQL> startup mount;

SQL> select GROUP#,MEMBER,TYPE from v$logfile;

SQL> select group#,bytes/1024/1024,members from v$log;

Check if the below groups exists and add if not.


/68- IBM

SQL>

ALTER DATABASE ADD STANDBY LOGFILE GROUP 4 (

'/home/orcl_user/app/oracle/oradata/ODB_FS/onlinelog/stdbyrdo41.log',

'/home/orcl_user/app/oracle/fast_recovery_area/ODB_FS/onlinelog/stdbyrdo42.log'


SQL>





SQL>





SQL>





ReStart the database instance : # sqlplus sys/<password>@ODB_FS as sysdba


SQL> create spfile from pfile='$ORACLE_HOME/dbs/initODB.ora';

SQL> startup mount

SQL> select status,instance_name,database_role from v$database,v$instance;

STATUS INSTANCE_NAME DATABASE_ROLE

--------- ---------------- ----------------

MOUNTED ODB FAR SYNC


/68- IBM


/68- IBM

14. Configuring DataGuard with DGMGRL and Fast-Start Fail over

14.1. Configuring BROKER and DATA GUARD on DB server: Enable the broker on all 3 Database server instances primno1 , stbyno1 and farysnc as below

SQL> alter system set dg_broker_start=true;

On Primary Instance prinno1 server

# dgmgrl sys/<password>@ODB

Create the configuration with name ODB_CONF_NEW with primary database ODB DGMGRL> CREATE CONFIGURATION ODB_CONF_NEW AS PRIMARY DATABASE IS ODB CONNECT

IDENTIFIER IS ODB;

( If any error raised with log_archive_dest_n parameter, note down the parameter value

somewhere else and remove the parameter having ‘service’ field in it on all 3 instances. SQL> show parameter log_archive_dest_2 SQL> alter system set log_archive_dest_2=’’ scope=both;

)

Add the Physical Standby database ODB_STBY as ODB_STBY to the configuration DGMGRL> ADD DATABASE ODB_STBY AS CONNECT IDENTIFIER IS ODB_STBY MAINTAINED AS

PHYSICAL;

Add the FarSync instance ODB_FS to the configuration as ODB_FS DGMGRL> ADD FAR_SYNC ODB_FS AS CONNECT IDENTIFIER IS ODB_FS;

Find the status of the configuration

(Set log_archive_dest_n values before enabling the configuration) DGMGRL> DGMGRL> enable configuration

Enabled.

DGMGRL> show configuration

Configuration - odb_conf_new

Protection Mode: MaxPerformance

Members:

odb - Primary database

odb_stby - Physical standby database

odb_fs - Far sync instance

Fast-Start Failover: DISABLED Configuration Status:

SUCCESS (status updated 42 seconds ago)

DGMGRL>


/68- IBM

Fast-start failover allows the broker to automatically fail over to a previously chosen standby database in the event of loss of the primary database. Fast-start failover quickly and reliably fails over the target standby database to the primary database role, without requiring you to perform any manual steps to invoke the failover. Fast-start failover can be used only in a broker configuration and can be configured only through DGMGRL or Enterprise Manager.

• Before Fast-Start Failover: Data Guard is operating in a steady state, with the primary database transmitting redo data to the target standby database and the observer monitoring the state of the entire configuration.

• FastStart Failover Ensues: Disaster strikes the primary database and its network connections to both the observer and the target standby database are lost. Upon detecting the break in communication, the observer attempts to reestablish a connection with the primary database for the amount of time defined by the FastStartFailoverThreshold property before initiating a fast-start failover. If the observer is unable to regain a connection to the primary database within the specified time, and the target standby database is ready for fast-start failover, then fast-start failover ensues.

• After Fast-Start Failover: The fast-start failover has completed and the target standby database is running in the primary database role. After the former primary database has been repaired, the observer reestablishes its connection to that database and reinstates it as a new standby database. The new primary database starts transmitting redo data to the new standby database.

Before Fast-Start Failover enabled : DGMGRL> show configuration verbose


Protection Mode: MaxPerformance

Members:

odb_stby - Primary database

odb - Physical standby database


Properties:

FastStartFailoverThreshold = '30'

OperationTimeout = '30'

TraceLevel = 'USER'

FastStartFailoverLagLimit = '30'

CommunicationTimeout = '180'

ObserverReconnect = '0'

FastStartFailoverAutoReinstate = 'TRUE'

FastStartFailoverPmyShutdown = 'TRUE'

BystandersFollowRoleChange = 'ALL'

ObserverOverride = 'FALSE'

ExternalDestination1 = ''



/68- IBM

PrimaryLostWriteAction = 'CONTINUE'

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS


/68- IBM

14.2. Enabling Fast-Start Failover and Starting the Observer We can enable fast-start failover from any site, including the observer site, while connected to any database in the broker configuration. Enabling fast-start failover does not trigger a failover. Instead, it allows the observer to begin observing the primary and standby databases and initiate a fast-start failover should conditions warrant a failover.

Step 1 Ensure standby redo logs are configured on the primary and target standby databases.

Standby redo logs must be configured on the primary and standby databases. You must stop log apply services prior to configuring standby redo logs.

Step 2 Ensure the LogXptMode Property is set to SYNC.

The LogXptMode configurable database property must be set to SYNC on the primary and target standby databases.

To set the redo transport service that corresponds to the protection mode you plan to set, use the EDIT DATABASE (property) command on the primary and target standby databases. For example, if the protection mode to be set is MAXAVAILABILITY, you must set the LogXptMode property to SYNC on the primary database and on the target standby database, as shown in the following examples: DGMGRL> EDIT DATABASE ODB SET PROPERTY 'LogXptMode'='SYNC';

Property "LogXptMode" updated

DGMGRL> EDIT DATABASE ODB_STBY SET PROPERTY 'LogXptMode'='SYNC';

Property "LogXptMode" updated

The broker does not allow these commands to succeed unless the databases are configured with standby redo log files.

Step 3 Set the FastStartFailoverTarget configuration property.

If you have two or more standby databases, set up the FastStartFailoverTarget configuration property on the primary database to indicate the desired target standby database. For example: DGMGRL> EDIT DATABASE ODB SET PROPERTY FastStartFailoverTarget='ODB_STBY';

Property "FastStartFailoverTarget" updated

DGMGRL> EDIT DATABASE ODB_STBY SET PROPERTY FastStartFailoverTarget='ODB';

Property "FastStartFailoverTarget" updated

Step 4 Upgrade the protection mode to MAXAVAILABILITY, if necessary.

If it is necessary to upgrade the protection mode, use the following DGMGRL EDIT CONFIGURATION command. For example: DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MAXAVAILABILITY;

Step 5 Enable Flashback Database on the primary and target standby databases, if necessary.

If it is not already enabled on both primary and standby databases, enable Flashback Database by issuing the following statements on each database server Primary and Secondary:


/68- IBM

SQL> ALTER SYSTEM SET UNDO_RETENTION=3600 SCOPE=SPFILE;

SQL> ALTER SYSTEM SET UNDO_MANAGEMENT='AUTO' SCOPE=SPFILE;

SQL> SHUTDOWN IMMEDIATE;

SQL> STARTUP MOUNT;

SQL> SHOW PARAMETER UNDO;

SQL> ALTER SYSTEM SET DB_FLASHBACK_RETENTION_TARGET=4320 SCOPE=BOTH;

SQL> ALTER DATABASE ARCHIVELOG;

SQL> ALTER DATABASE FLASHBACK ON;

SQL> ALTER DATABASE OPEN;

On Standby only, start the redo apply processes:

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

Ensure the UNDO_RETENTION and DB_FLASHBACK_RETENTION_TARGET initialization parameters are set to sufficiently large values so that reinstatement is still possible after a prolonged outage.

Step 6 Start the observer.

Start the observer by logging into the observer computer and running DGMGRL. Connect to the configuration as SYS and then issue the START OBSERVER command. Note that the command does not return; that is you will not get DGMGRL prompt after issuing the command. DGMGRL> CONNECT sys@ODB;

Password:

Connected.

DGMGRL> START OBSERVER;

Observer started

Step 7 Enable fast start failover.

You can enable fast-start failover while connected to any database system in the broker configuration. For example: DGMGRL> ENABLE FAST_START FAILOVER;

Enabled.

Step 8 Verify the fast-start failover configuration.

Use the SHOW FAST_START FAILOVER command to display the fast-start failover settings: DGMGRL> SHOW FAST_START FAILOVER;

Fast-Start Failover: ENABLED

Threshold: 30 seconds

Target: odb

Observer: farsync.fragglerock.mycom.com


/68- IBM

Lag Limit: 30 seconds (not in use)

Shutdown Primary: TRUE

Auto-reinstate: TRUE

Observer Reconnect: (none)

Observer Override: FALSE

Configurable Failover Conditions

Health Conditions:

Corrupted Controlfile YES

Corrupted Dictionary YES

Inaccessible Logfile NO

Stuck Archiver NO

Datafile Offline YES

Oracle Error Conditions:

(none)

Configuration : DGMGRL> show configuration verbose


Protection Mode: MaxAvailability

Members:

odb_stby - Primary database

odb - (*) Physical standby database


(*) Fast-Start Failover target

Properties:

FastStartFailoverThreshold = '30'

OperationTimeout = '30'

TraceLevel = 'USER'

FastStartFailoverLagLimit = '30'

CommunicationTimeout = '180'

ObserverReconnect = '0'

FastStartFailoverAutoReinstate = 'TRUE'

FastStartFailoverPmyShutdown = 'TRUE'

BystandersFollowRoleChange = 'ALL'

ObserverOverride = 'FALSE'



PrimaryLostWriteAction = 'CONTINUE'


/68- IBM

Fast-Start Failover: ENABLED

Threshold: 30 seconds

Target: odb

Observer: farsync.fragglerock.mycom.com

Lag Limit: 30 seconds (not in use)

Shutdown Primary: TRUE

Auto-reinstate: TRUE

Observer Reconnect: (none)

Observer Override: FALSE

Configuration Status:

SUCCESS

There is no configuration changes needed on ITNM level for failover process


/68- IBM

15. Installation of ITNM on Primary Instance For detailed instructions on installing ITNM please see https://www.ibm.com/support/knowledgecenter/SSSHRK_4.2.0/install/task/ins_installingitnm.html

Download the latest packages and extract into a temporary directory on your server. Assuming the customer already has the following packages

1. IBM Installation Manager for Linux x86_64 2. ITNM Dependencies (WAS v8.5.5.9, Omnibus v8.1.0.5, Omnibus WebGUI 8.1.0.14,

Jazz v1.1.2.1, IBM DASH v3.1.3.0 ) 3. ITNM v4.2 FP5

Installing IBM Installation Manager : Open the folder containing IBM Installation Manager and run the installer # cd <ibm_install_manager> #./install (./userinst if the user is non-root)

Give the location where IBMIM to be installed.

Next > Install>

Installing ITNM Dependencies : Navigate to location where IBMIM is installed. (default location <userHome> /IBM/InstallationManager) Start the IBMIM

# eclipse/IBMIM

Navigate to File-> Preferences… Add the respective repository in which all the dependencies are present.

Apply> OK> Click on the “Install” option, Next> Select all the dependencies Next> Give appropriate details Next> Install Next> After the installation is completed, select None Finish>

Creating and Populating Oracle database on Primary Instance: Start IBMIM

Navigate to File> Preference Uncheck the repository for dependencies and add another one containing ITNM packages Apply> OK> Click Install>

https://www.ibm.com/support/knowledgecenter/SSSHRK_4.2.0/install/task/ins_installingitnm.html


/68- IBM

Select “Network Manager Topology Database Creation Scripts” package and proceed. Once the package is installed, the database schema are to be created on Primary instance. Login as oracle user,

# su - oracle

Navigate to $NCHOME/precision/scripts/sql/oracle (default NCHOME <userHome>/IBM/netcool/core) Create users for the database

# ./create_oracle_ncadmin_user.sh sys <password>

Create database schema

# ./create_oracle_database.sh system <password>

Create the tables on the database schema

# ./populate_oracle_database.sh ncim ncim

Installing ITNM on Primary Instance : Create Group and user for ITNM

# groupadd mycomsrc # useradd -d /home/itnmadm -g mycomsrc itnm_poller # passwd itnm_poller # id itnm_poller

Start the IBMIM

Install>

Select the Network manager Core and GUI components

Next>

Give the relevant details that are given at the time of installing dependencies. At ObjectServer : ObjectServer Name : AGG_V (Virtual ObjectServer) Host : primno1.fragglerock.mycom.com Port : 4100 ( Port on which AGG_P is running) User : root Password : < root password>


/68- IBM

At the database : Select Oracle database, Service Name : ODB Host : primno1.mycom.mycom.com Port : 1521 (port on which listener is running) User : ncim Password : ncim Next> Install>


/68- IBM

16. Installing ITNM on physical standby Before installing the ITNM on Standby, insure that ADG manual Database switchover has to been performed and Secondary DB server is ACTIVE and in READ/WRITE mode.

Similar procedure as followed to install ITNM on Primary server has to be followed to install ITNM Standby Instance except following

The step/procedure to population of database is not needed while installing ITNM on secondary server as the database schema is already created and populated when ITNM is installed on Primary server primno1 as primary instance.

At the time of installing ITNM core and GUI, while providing details, give the following details…

At ObjectServer : ObjectServer Name : AGG_V (Virtual ObjectServer) Host : stbyno1.fragglerock.mycom.com Port : 4150 ( Port on which AGG_B is running) User : root Password : < root password>

At the database :

Select Oracle database, Service Name : ODB_STBY Host : stbyno1.fragglerock.mycom.com Port : 1521 (port on which listener is running) User : ncim Password : ncim Next> Install>

It is always mandatory to keep ITNM process ideal (not running) at all time when the server is acting as standby instance


/68- IBM

17. Enabling Rysnc between Primary and Secondary ITNM server

Rsync is a utility tool used to sync the files between servers more effectively. It can copy locally/remotely from/to another host over any remote shell. It provides a numerous options that control every aspect of its behavior. It reduces the amount of data sent over the network by sending only the differences between the source files and existing files in the destination.

In Active DataGuard setup, only the NCIM database of primary is highly available for the standby which is connected through broker configuration. The second most important files which are present in the primary ITNM instance (Discovery cache files, other config files) resides and available only on primary ITNM. These are not accessible to standby ITNM. Inorder to facilitate this accessibility and make the High Availability more meaningful, Rsync utility in association with InotifyWait is used to sync these files between primary ITNM and standby ITNM.

inotify is an API utility used to moniter a file/directory and notify the user whenever a particular event is occurred. When a directory/file is monitored, inotify will return the events for the directory itself and for all the files inside the directory.

Setting Up Rsync :

Install the inotifywait and rsync utilities on both Primary and Standby servers

# yum -y install epel-release # yum install inotify-tools # yum install rsync

17.1. Creating rsync files

17.1.1. Create rsync files on primno1 as mentioned below

To Synchronize $NCHOME/etc/precision directory between Primary and Secondary ITNM server

$ vim /home/oracle/bin/rsync_nchome_etc_precision.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/etc/precision; do

rsync -azu --delete --exclude={'DbLogins*','MibDbLogin*','ServiceData.cfg'} /home/oracle/IBM/tivoli/netcool/etc/precision/ oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/etc/precision/


/68- IBM

done

To synchronize $NCHOME/precision/aoc directory between Primary and Secondary ITNM server

$ vim /home/oracle/bin/rsync_nchome_prec_aoc.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/aoc; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/aoc/ oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/precision/aoc/

done

To Synchronize $NCHOME/precision/collectors/perlCollectors directory between Primary and Secondary ITNM server

$ vim /home/oracle/bin/rsync_nchome_prec_coll_perlcol.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/collectors/perlCollectors; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/collectors/perlCollectors/ oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/precision/collectors/perlCollectors/

done

To Synchronize $NCHOME/precision/disco/agents directory between Primary and Secondary ITNM server

$ vim /home/oracle/bin/rsync_nchome_prec_disco_agent.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/disco/agents; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/disco/agents/ oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/precision/disco/agents/

done

To Synchronize $NCHOME/precision/eventGateway/stitchers directory between Primary and Secondary ITNM server


/68- IBM

$ vim /home/oracle/bin/rsync_nchome_prec_evgate_stch.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/eventGateway/stitchers; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/eventGateway/stitchers/ oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/precision/eventGateway/stitchers/

done

To Synchronize $NCHOME/precision/mibs directory between Primary and Secondary ITNM server

$ vim /home/oracle/bin/rsync_nchome_prec_mibs.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/mibs; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/mibs/ oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/precision/mibs/

done

To Synchronize $NCHOME/var/precision directory between Primary and Secondary ITNM server

$ vim /home/oracle/bin/rsync_nchome_var_precision.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/var/precision; do

rsync -azu --delete --exclude='storm' /home/oracle/IBM/tivoli/netcool/var/precision/ oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/var/precision/

done

Give execution permissions for above bash scripts

# chmod +x /home/oracle/bin/rsync_*.bash


/68- IBM

17.1.2. Create rsync files on stbyno1 :

To Synchronize $NCHOME/etc/precision directory between Primary and Secondary ITNM server

$ vim /home/oracle/bin/rsync_nchome_etc_precision.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/etc/precision; do

rsync -azu --delete --exclude={'DbLogins*','MibDbLogin*','ServiceData.cfg'} /home/oracle/IBM/tivoli/netcool/etc/precision/ oracle@primno1:/home/oracle/IBM/tivoli/netcool/etc/precision/

done

To Synchronize $NCHOME/precision/aoc directory between Primary and Secondary ITNM server

$ vim /home/oracle/bin/ rsync_nchome_prec_aoc.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/aoc; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/aoc/ oracle@primno1:/home/oracle/IBM/tivoli/netcool/precision/aoc/

done

To Synchronize $NCHOME/precision/collectors/perlCollectors directory between Primary and Secondary ITNM server

$ vim /home/oracle/bin/rsync_nchome_prec_coll_perlcol.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/collectors/perlCollectors; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/collectors/perlCollectors/ oracle@primno1:/home/oracle/IBM/tivoli/netcool/precision/collectors/perlCollectors/

done

To Synchronize $NCHOME/precision/disco/agents directory between Primary and Secondary ITNM server

$ vim /home/oracle/bin/rsync_nchome_prec_disco_agent.bash

#!/usr/bin/bash


/68- IBM

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/disco/agents; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/disco/agents/ oracle@primno1:/home/oracle/IBM/tivoli/netcool/precision/disco/agents/

done

To Synchronize $NCHOME/precision/eventGateway/stitchers directory between Primary and Secondary ITNM server

$ vim /home/oracle/bin/rsync_nchome_prec_evgate_stch.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/eventGateway/stitchers; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/eventGateway/stitchers/ oracle@primno1:/home/oracle/IBM/tivoli/netcool/precision/eventGateway/stitchers/

done

To Synchronize $NCHOME/precision/mibs directory between Primary and Secondary ITNM server

$ vim /home/oracle/bin/rsync_nchome_prec_mibs.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/mibs; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/mibs/ oracle@primno1:/home/oracle/IBM/tivoli/netcool/precision/mibs/

done

To Synchronize $NCHOME/var/precision directory between Primary and Secondary ITNM server

$ vim /home/oracle/bin/rsync_nchome_var_precision.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/var/precision; do

rsync -azu --delete --exclude='storm' /home/oracle/IBM/tivoli/netcool/var/precision/ oracle@primno1:/home/oracle/IBM/tivoli/netcool/var/precision/

done

Give execution permissions for above bash scripts

# chmod +x /home/oracle/bin/rsync_*.bash


/68- IBM

17.2. Enabling Passwordless Authentication

For the rsync to work, the password is to be provided everytime the files are transferred to remote destination. For this process has to be done automatically, a passwordless authentication must be enabled from primno1 to stbyno1 and as well as from stbyno1 to primno1. Follow the below steps.

On primno1,

Log on to oracle user,

# su - oracle # cd ~/.ssh

Check if id_rsa.pub file is present. If not, create the file

# ssh-keygen -t rsa

Give passphrase as blank(Hit Enter)

Copy the string generated in .ssh/id_rsa.pub in local server

On remote server(stbyno1), go to <user>/.ssh/authorized_keys file (if not present, create the file)

Append the string copied earlier

Similarly, paste the id_rsa.pub file of stbyno1 in the authorized_keys of primno1.

This should enable the rsync to transfer files continuously without any authentication.

Make sure that relevant directories/files have sufficient permissions to them for passwordless ssh to work.

Example :

drwx------. 29 oracle oinstall 4096 Jun 3 04:43 /home/oracle drwx------. 2 oracle oinstall 4096 May 29 05:54 /home/oracle/.ssh -rw-r--r--. 1 oracle oinstall 837 May 29 05:54 /home/oracle/.ssh/authorized_keys


/68- IBM

17.3. Running rsyng scripts to sync Primary ITNM Core with Secondary ITNM core (default set up)

Initiate the rsync on primno1 by running following commands. Please do not run these commands from Secondary ITNM server in default mode.

# nohup /home/oracle/bin/rsync_nchome_etc_precision.bash >> etc_precision.log & # nohup /home/oracle/bin/rsync_nchome_prec_aoc.bash >> prec_aoc.log & # nohup /home/oracle/bin/rsync_nchome_prec_coll_perlcol.bash >> prec_coll_perlcol.log & # nohup /home/oracle/bin/rsync_nchome_prec_disco_agent.bash >> prec_disco_agent.log & # nohup /home/oracle/bin/rsync_nchome_prec_evgate_stch.bash >> prec_egate_stch.log & # nohup /home/oracle/bin/rsync_nchome_prec_mibs.bash >> prec_mibs.log & # nohup /home/oracle/bin/rsync_nchome_var_precision.bash >> var_prec.log &

The above commands append the logs to respective log files.

**Caution : As storm runs continuously, the logs are generated in huge amount. It would be good practice to monitor the size of var_prec.log file from time to time.

17.4. Stopping rsync 17.5. The rsync scripts can be stopped by conventional method of killing rsync But in default mode, we should keep running rysnc scripts. Here is example of killing rsync process.

# ps - ef | grep rsync_var_precision.bash # kill -9 <PID>


/68- IBM

18 ITNM Fail over, Fail back and Switch Over Procedure 18.1. Fail-over Procedure

ITNM Fail Over Procedure ( DR Situation ) You may need to fail over from Primary Active ITNM instance to Secondary Standby ITNM instance when Primary ITNM server become unavailable due to some unforeseen condition like sustained power outage or physical Sever failure to achieve business continuity with some disruption. This is typical Disaster Recovery situation. Please carry out following mentioned steps and procedure to start ITNM from DR site and achieve full ITNM functionality while Primary Site remain unavailable. In case of Disaster Kind of situation, it is expected that full NOI stack (ITNM Core , Omnibus , DASH ) including Primary ITNM DB server running in Primary Site has become unavailable. The loss of Primary Site should be noticed by end users/operators typically through manual actions or external sources. There is no mechanism in built in this solution to notify the user or operator about total failure of Primary Site. Its is presumed that you are running Observer ( Oracle farsync node ) in a separate site than Primary Site and DR Site. On the Observer terminal ( farsync node ), the following status will be logged when Oracle ADG solution notice the failure of Primary Oracle server and trigger the automatic fail over operation to bring Secondary Standby Database in Active mode. 04:46:11.80 Monday, January 28, 2019 Initiating Fast-Start Failover to database "odb_stby"... Performing failover NOW, please wait... Failover succeeded, new primary is "odb_stby" 04:46:27.91 Monday, January 28, 2019

Now run DGMGRL tool from Observer Node and run ‘show configuration’ command to verify that ADG has set up the Standby Database in Active Mode as shown below. DGMGRL> show configuration Configuration - odb_conf_new Protection Mode: MaxAvailability Members: odb_stby - Primary database odb - (*) Physical standby database odb_fs - Far sync instance Fast-Start Failover: ENABLED Configuration Status: SUCCESS (status updated 6 seconds ago)

Once we verified that Oracle DB server in DR site is in Active Mode. Start ITNM Core processes from DR site as below.


/68- IBM

1. Check Omnibus running 1. ps -ef | grep objserv 2. Start all itnm processes 1. navigate to /home/oracle/IBM/tivoli/netcool cd /home/oracle/IBM/tivoli/netcool

2. set the environment variable . env.sh

3. start itnm processes itnm_start

4. check all the processes are running including Apache Storm itnm_status

3. Open Dash, check login 1. check Jazz server running ps -ef | grep Jazz

If not, start manually /home/oracle/IBM/JazzSM/profile/bin/startServer.sh server1

2. open https://stbyno1.fragglerock.mycom.com:16311/ibm/console in firefox or IE 3. login as itnmadmin/******** 4. Start Discovery 1. Discovery->Network Discovery-> Network Discovery Status select the domain, Check what are the devices listed run full discovery. After completion of Discovery, verify that Network is discovered as expected.Also verify rest all ITNM functionality like Network views , polling and RCA is working as expected. 5. Configure Rsync from Secondary to Primary ITNM Server 1. Discovery->Network Discovery-> Network Discovery Status select the domain, Check what are the devices listed run full discovery. After completion of Discovery, verify that Network is discovered as expected. Verify rest all ITNM functionality like Network views, polling and RCA is working as expected. 18.2. Fall-back Procedure

Assuming Primary Site has recovered from failure and all Nodes from Primary site are up and running again. Please follow following procedure to stop ITNM server running on DR site and start ITNM from Primary Site. Carry out following steps from Primary DB server node of Primary Site

• Make sure that network connection is restored between nodes from primary site and other sites ( farsync ,standby)

• Make sure that you are able to ping the other servers with their IP, short-hand hostname, full-fledged hostname

• Make sure that oracle listener is running


/68- IBM

o Listener status : $ORACLE_HOME/bin/lsnrctl status o Start listener : $ORACLE_HOME/bin/lsnrctl start

• Make sure that you can get a response through tnsping o Syntax :

tnsping <hostname> tnsping <DB name>

o Example : tnsping stbyno1.fragglerock.mycom.com tnsping stbyno1 tnsping odb_stby tnsping farsync.fragglerock.mycom.com tnsping farsync tnsping odb_fs

As primary database had been configured with Flashback Database, you can easily reinstate the

failed primary database as a standby database of the new primary database. The failed primary

database will be reinstated as a standby type that matches the old standby database. For example,

if you failed over to a physical standby database, the old primary will be reinstated as a physical

standby database.

To reinstate the failed primary database, start it to the mounted state. Then run DGMGRL, connect

to the secondary database and reinstate the primary database as mentioned below

Restart the Primary Database

[oracle@primno1 ~]$ sqlplus sys/<password> as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Mon Jan 28 04:16:57 2019

Copyright (c) 1982, 2014, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup mount

ORACLE instance started.

Total System Global Area 1.0100E+10 bytes

Fixed Size 3721224 bytes

Variable Size 1879050232 bytes

Database Buffers 8187281408 bytes


/68- IBM

Redo Buffers 29831168 bytes

Database mounted.

SQL>

Reinstate the primary database

Then run DGMGRL from Primary Database Server as shown below to connect to the secondary database and reinstate the Primary database.

[oracle@primno1 netcool] dgmgrl connect sys@ODB_STBY Password: Connected. DGMGRL> REINSTATE DATABASE 'odb'; Reinstating database "odb", please wait... Operation requires shutdown of instance "sales1" on database "odb" Shutting down instance " odb "... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires startup of instance " odb" on database "odb" Starting instance "odb "... ORACLE instance started. Database mounted. Continuing to reinstate database "odb " ... Reinstatement of database "odb " succeeded

After the primary has been reinstated, issue the SHOW CONFIGURATION and SHOW DATABASE commands to confirm that the primary database has been successfully reinstated

Show the Configuration and Databases

DGMGRL> show configuration Configuration - odb_conf_new Protection Mode: MaxAvailability Members: Odb_stby - Primary database odb - (*) Physical standby database odb_fs - Far sync instance Fast-Start Failover: ENABLED Configuration Status: SUCCESS (status updated 50 seconds ago) DGMGRL> show database odb_stby Database – odb_stby Role: PRIMARY Intended State: TRANSPORT-ON Instance(s): ODB Database Status: SUCCESS

DGMGRL> show database odb Database - odb


/68- IBM

Role: PHYSICAL STANDBY Intended State: APPLY-ON Transport Lag: 0 seconds (computed 0 seconds ago) Apply Lag: 0 seconds (computed 1 second ago) Average Apply Rate: 9.00 KByte/s Real Time Query: OFF Instance(s): ODB Database Status: SUCCESS

Carry out the following sql commands from Secondary database server to verify the ADG role status of secondary database.

SQL> select status,instance_name,database_role,open_mode from v$database,v$Instance;


------------ ---------------- ---------------- --------------------


SQL> select switchover_status from v$database;

SWITCHOVER_STATUS

--------------------

TO STANDBY

Carry out the following sql commands from Primary database server to verify the ADG role status of Primary database



------------ ---------------- ---------------- --------------------

MOUNTED ODB PHYSICAL STANDBY MOUNTED

SQL> select switchover_status from v$database;

SWITCHOVER_STATUS

--------------------


/68- IBM

NOT ALLOWED

Add log_archive_dest_2 on Secondary Database server ( primno1 )

SQL> alter system set log_archive_dest_2='service=ODB_FS SYNC AFFIRM alternate=log_archive_dest_3 valid_for=(online_logfiles,primary_role) db_unique_name=ODB_FS';

Add log_archive_dest_2 on Secondary Database server ( stbyno1)

SQL> alter system set log_archive_dest_2='service=ODB ASYNC valid_for=(online_logfiles,primary_role) db_unique_name=ODB';

Add log_archive_dest_2 on Farsync Database server (farsync )

SQL> alter system set log_archive_dest_2='service=ODB_STBY ASYNC valid_for=(standby_logfiles,standby_role) db_unique_name=ODB_STBY';

Verify the database switchover status on DG Broker configuration by running DGMGRL tool commands as mentioned below from any one of Database servers ( Primary , Secondary or FarSync ) DGMGRL> validate database odb_stby Database Role: Primary database Ready for Switchover: Yes Flashback Database Status: odb_stby: Off DGMGRL> validate database odb Database Role: Physical standby database Primary Database: odb_stby Ready for Switchover: Yes Ready for Failover: Yes (Primary Running) Flashback Database Status: odb_stby: Off odb: Off DGMGRL> validate far_sync odb_fs Member Role: Far Sync Instance When Primary Is: odb_stby Active Redo Source: ODB_STBY Redo Destinations: (none) Thread # Online Redo Log Groups Standby Redo Log Groups Status odb_stby odb_fs 1 3 4 Sufficient SRLs

Run following Switch Over verification sql command from Secondary Database server.

SQL> alter database switchover to ODB verify;


/68- IBM

Database altered.

( If response is other than ‘Database altered’ then please refer to error message that will be logged in to alert.xml file )

Run following sql command from Secondary DB server to switch over the DB ADG roles so that Primary Database server become ACTIVE and Secondary Database server assume Stand By role.

SQL> alter database switchover to ODB;

Database altered.

After switching over, verify the status of Primary and Secondary DB by running following commands.



------------ ---------------- ---------------- --------------------

MOUNTED ODB PRIMARY MOUNTED

On Primary Database server run following command to set it in READ WRITE mode as below

SQL> alter database open;

Verify the status of Primary data base by running following command



--------- --- ---------------- ---------------- --------------------


Secondary Database Server (stbyo1) shall go in shutdown mode so run following sql commands on it

SQL> startup nomount

ORACLE instance started.

Total System Global Area 1.0100E+10 bytes

Fixed Size 3721224 bytes

Variable Size 1879050232 bytes

Database Buffers 8187281408 bytes


/68- IBM

Redo Buffers 29831168 bytes

SQL> alter database mount

Database altered.



------------ ---------------- ---------------- --------------------

MOUNTED ODB PHYSICAL STANDBY MOUNTED

Start MRP processes on Secondary Database server as below

SQL> alter database recover managed standby database disconnect;

Database altered.

Re-enable the broker configuration from Primary Database server(primno1)

# dgmgrl sys/<password>@ODB

Disable Fast_Start failover if already enabled

DGMGRL> disable fast_start failover Disabled. DGMGRL> disable configuration Disabled. DGMGRL> enable configuration Enabled. DGMGRL> show configuration Configuration - odb_conf_new Protection Mode: MaxPerformance Members: odb - Primary database odb_stby - Physical standby database odb_fs - Far sync instance Fast-Start Failover: DISABLED Configuration Status: SUCCESS (status updated 46 seconds ago) DGMGRL>enable fast_start failover Enabled. DGMGRL> show configuration Configuration - odb_conf_new Protection Mode: MaxPerformance Members: odb - Primary database odb_stby - Physical standby database odb_fs - Far sync instance Fast-Start Failover: ENABLED Configuration Status: SUCCESS (status updated 46 seconds ago)


/68- IBM

Now we need to re enable Fast Start Failover configuration from Primary Database server. Please refer to steps mentioned in section 14.2 and carry out exact steps mentioned in same section from Primary DB server.

Restart Netcool components from Primary Site as mentioned below.

• Start the Omnibus if not already running # /home/oracle/IBM/tivoli/netcool/omnibus/bin/nco_objserv -name AGG_P &

( AGG_P is the objectserver name that is configured at the time of installation of ITNM)

• If the DB2, that is used to configure HA of DASH/WebGUI is on the same server, start the DB2 database

• Start DASH o <jazz_home>/profile/bin/startServer.sh server1

• Check the DASH HA status o <jazz_home>/ui/bin/consolecli.sh ListHANodes --username smadmin --password netcool

• Set the environment and Check database access # cd /home/oracle/IBM/tivoli/netcool # . env.sh # /home/oracle/IBM/tivoli/netcool/precision/scripts/perl/scripts/ncp_db_access.pl -domain NCOMS (NCOMS is the default domain whose database is configured through dataguard)

• Initiate the rsync from Secondary ITNM server ( stbyno1 ) by running following commands. Let these commands run for some time so that ITNM core files from Primary ITNM server sync with ITNM core files from Secondary ITNM server. Please verify same by checking few files sizes and time stamps. # nohup /home/oracle/bin/rsync_nchome_etc_precision.bash >> etc_precision.log & # nohup /home/oracle/bin/rsync_nchome_prec_aoc.bash >> prec_aoc.log & # nohup /home/oracle/bin/rsync_nchome_prec_coll_perlcol.bash >> prec_coll_perlcol.log & # nohup /home/oracle/bin/rsync_nchome_prec_disco_agent.bash >> prec_disco_agent.log & # nohup /home/oracle/bin/rsync_nchome_prec_evgate_stch.bash >> prec_egate_stch.log & # nohup /home/oracle/bin/rsync_nchome_prec_mibs.bash >> prec_mibs.log & # nohup /home/oracle/bin/rsync_nchome_var_precision.bash >> var_prec.log &

• Stop/Kill all rsync processes running on ITNM secondary server ( stbyno1 ) • Now Initiate the rsync from Primary ITNM server ( primno1 ) by running following

commands. Let these commands run forever so that ITNM core files from Secondary ITNM server remain in sync with ITNM core files from Primary ITNM server.

# nohup /home/oracle/bin/rsync_nchome_etc_precision.bash >> etc_precision.log & # nohup /home/oracle/bin/rsync_nchome_prec_aoc.bash >> prec_aoc.log & # nohup /home/oracle/bin/rsync_nchome_prec_coll_perlcol.bash >> prec_coll_perlcol.log & # nohup /home/oracle/bin/rsync_nchome_prec_disco_agent.bash >> prec_disco_agent.log &


/68- IBM

# nohup /home/oracle/bin/rsync_nchome_prec_evgate_stch.bash >> prec_egate_stch.log & # nohup /home/oracle/bin/rsync_nchome_prec_mibs.bash >> prec_mibs.log & # nohup /home/oracle/bin/rsync_nchome_var_precision.bash >> var_prec.log &

• Now start the ITNM processes from Primary ITNM server ( primno1 ) # itnm_start

19. Monitoring Oracle ADG and Rysnc processes

19.1. Monitoring ADG FRA space

It is important to monitor space usage in the fast recovery area to ensure that it is large enough to contain backups and other recovery-related files.


/68- IBM

Query 1:

SQL >SELECT ROUND((A.SPACE_LIMIT / 1024 / 1024 / 1024), 2) AS FLASH_IN_GB,

ROUND((A.SPACE_USED / 1024 / 1024 / 1024), 2) AS FLASH_USED_IN_GB,

ROUND((A.SPACE_RECLAIMABLE / 1024 / 1024 / 1024), 2) AS FLASH_RECLAIMABLE_GB,

SUM(B.PERCENT_SPACE_USED) AS PERCENT_OF_SPACE_USED

FROM

V$RECOVERY_FILE_DEST A,

V$FLASH_RECOVERY_AREA_USAGE B

GROUP BY

SPACE_LIMIT,

SPACE_USED ,

SPACE_RECLAIMABLE ;

Query 2 :

SQL> SELECT * FROM V$RECOVERY_AREA_USAGE;

Query 3 :

SQL> SELECT trunc(first_time) DAY,

count(*) NB_SWITCHS,

trunc(count(*)*log_size/1024/1024) TOTAL_SIZE_MB,

to_char(count(*)/24,'9999.9') AVG_SWITCHS_PER_HOUR

FROM v$loghist,

(select avg(bytes) log_size from v$log) GROUP BY trunc(first_time),log_size;


/68- IBM

If the available FRA space is nearly 0 in any of the instance, the redo-logs may not be shipped which effects the functionality of ADG. So, Oracle provides 3 methods to tackle this issue.

Method 1 : Increase the FRA space

SQL> ALTER SYSTEM SET db_recovery_file_dest_size=xxG;

Where xx is the new size(should be greater than earlier value) of FRA in GigaBytes.

Method 2 : Relocate the FRA to another location

SQL> SHOW PARAMETER DB_RECOVERY_FILE_DEST

SQL> ALTER SYSTEM SET DB_RECOVERY_FILE_DEST='<New_Location>';

Method 3 : Disable the FRA policy

SQL> ALTER SYSTEM SET DB_RECOVERY_FILE_DEST='' SCOPE=BOTH SID="*";

Removing FRA logs rman target=/ RMAN> run { crosscheck archivelog all; crosscheck foreign archivelog all; list expired archivelog all; delete expired archivelog all; delete noprompt archivelog all; delete noprompt backup of database; delete noprompt copy of database; delete expired foreign archivelog all; delete foreign archivelog all; }

19.2. Monitoring Rysnc processes

Since we are going with one-way rsync, source of rsync should be always ‘primary site’ and the destination should always be ‘standby site’ Scenario 1: ITNM in Primary Site Active and ITNM in DR site in cold standby mode

• On ITNM server primno1, check whether rsync is running # ps -ef | grep rsync


/68- IBM

Expected result: Seven bash scripts like rsync_var_prec.bash , rsync_etc_precision.bash, rsync_nchome_precision.bash must be running.

• Tail the var_prec.log file # tail -f /home/oracle/bin/log/rsync_var_prec.log

Expected result: This should log the files those have been transferred to stbyno1

• On stbyno1, check whether the files are getting updated as shown in the var_prec.log of primno1

Repeat the above test for other 6 rsync scripts.

Scenario 2 : ITNM in DR Site Active and ITNM in Primary site in cold standby mode

• On stbyno1, check whether rsync is running # ps -ef | grep rsync

Expected result: Seven bash scripts like rsync_var_prec.bash rsync_etc_precision.bash, rsync_nchome_precision.bash must be running.

• Tail the var_prec.log file # tail -f /home/oracle/bin/log/rsync_var_prec.log

Expected result: This should log the files those have been transferred to primno1

• On primno1, check whether the files are getting updated as shown in the rsync_var_prec.log of stbyo1

Repeat the above test for other 6 rysnc scripts.

End of Document

best practices document - ibm developer€¦ · • the oracle database software (version 12c)...

Documents